Find the answer to your Linux question:
Results 1 to 4 of 4
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1

    NFS server issue - massive slow down under load


    Hi,

    I am new to these forums; hope I found the right section of the forum to ask this.

    I have recently upgraded and moved a 10-node Scientific Linux 6.5 cluster. I added three nodes and the machines were relocated to a new server hall and put into a new subnet/vlan. All settings, as far as I can tell, were updated appropriately.

    Since the upgrade/move I have been seeing massive drops in NFS performance and I can't seem to nail down the source. I should say that I am self-thaught and really only do this on a need-to-know sort of basis. Essentially, I am at the end of my ropes and am hoping for some help.

    The setup:

    10 * Dell PowerEdge 620 (compute)
    1 * Dell PowerEdge 720XD (storage, NFS server)
    1 * Dell PowerEdge 520 (head node)

    Networking happens over:
    Intel TrueScale Infiniband 4*QDR
    Del PowerConnect 6224 Ethernet (1Gbit)

    OS: Scientific Linuc 6.5
    ----------------
    Currently, the export/mount siutation looks as follows:

    /etc/exports:

    Code:
    # IB shares
    /data0/projects 10.10.10.0/24(rw,async,insecure,no_root_squash,no_subtree_check)
    /data0/sw 10.10.10.0/24(rw,async,insecure,no_root_squash,no_subtree_check)
    
    # Ethernet shares
    
    /data0/home     192.168.1.0/24(rw,async,insecure,no_subtree_check,no_root_squash)
    /opt    192.168.1.0/24(rw,async,insecure,no_subtree_check,no_root_squash)
    /data0/sw       192.168.1.0/24(rw,async,insecure,no_subtree_check,no_root_squash)
    /data0/repos    192.168.1.0/24(rw,async,insecure,no_subtree_check,no_root_squash)
    On the clients, the mounts are:

    Code:
    192.168.1.2:/data0/home /home    nfs4    _netdev,auto,async,intr  0  0
    192.168.1.2:/data0/repos /repos    nfs4    _netdev,auto,async,intr,nolock  0  0
    
    # Infiniband data share
    10.10.10.2:/data0/projects /projects  nfs4    async,_netdev,auto,intr,noatime  0  0
    10.10.10.2:/data0/sw /sw  nfs4    async,_netdev,auto,intr,noatime  0  0
    All the exported mount points reside on the same 16TB , XFS-formatted Raid-6 rray (which may not have been the smartest setup, but oh well..)

    Under no load, write speeds on the ethernet shares are ~95MB/sec
    On the Infiniband shares ~400MB/sec

    Once I start large MPI jobs (40++ cores) of an application that reads and writes lots (hundreds, possibly thousands) of small files (< 1MB in size), the speed on the client-side file system completely plummets and sits at around 20-30MB/sec (still fine on the server tho). Since /home is affected by this, the user experience is pretty poor.

    The question I have is whether this is simply expected or if there is something wrong?

    - I did not see the same issues before migrating and moving everything - but the cluster was a bit smaller then

    - netstat -s on the NFS server shows:
    Code:
     9385225 packets collapsed in receive queue due to low socket buffer
    I've increased the tcp buffer size to max. 12MB, and the counter has largely stopped, but at least there is/was an issue there.

    - iostat -x on the nfs server shows (under load):
    Code:
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.06    0.72   15.56    0.39    0.00   83.27
    
    Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.13    20.01    0.94    2.12   146.37   176.92   105.45     0.00    0.98   0.69   0.21
    sdb               0.80     3.19  140.41   45.74  7967.78  7696.92    84.15     2.22   11.93   3.54  65.88
    I am happy to provide more details, but am a bit at a loss what to monitor and where else to look.

  2. #2
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    13,681
    Please post the contents of your nfs server configuration file here (/etc/nfsmount.conf).
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  3. #3
    Sorry for the late reply, Issue still persists so happy for any input.

    The file in question is completely commented out. I am starting to wonder if I may be dealing with a bad cable? But changing all cables just on a hunch is perhaps a bit overkill, not sure..

  4. $spacer_open
    $spacer_close
  5. #4
    After another enjoyable day of debugging, I may have found a solution to this....

    A comprehensive test of the infiniband fabric revealed mismatched rpm versions between some nodes (damned if I know how that happened). Re-installed the IB stack on the affected nodes and speed stays fairly stable > 300Mb/s for writes to the NFS shares even when running > 100 core MPI jobs.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •