NFS Issues and woes

[majorbytes]

---

I have two systems setup using nfs that are basically just a bunch of drives and I share them to our clients. Recently some issues have appeared that are completely puzzling me. We have a few directories that if any files are written or copied into them, they are deleted. It is happening in a couple of other directories, but not really system wide, just where we happen to be working mostly. The issue first started up around April of this year. The previous three or four years no problems.. We only have around six or seven total users on the system.

I have traced them with WireShark and all of the IP's are internal from the local clients. I am thinking some sort of buffer issue, but not sure.. We are puzzled.

Both servers are quite different in architecture. One is hardware raided and the other one is software raided. The raids are all raid 6. I have tried nfs3, nfs4 and samba and the issue happens on all three..

[Arola1982]

Hi

May i ask where the directories are?

I could understand you having similar issues across nfs3 and nfs4 but samba is a different protocol so it suggests to me that its a problem with the location in which your sharing

[majorbytes]

they are mounted as /mnt/HelmsDeep1/data on the clients.

[Arola1982]

Is that the client side mount on the path on the server that is being shared?

Also what distro is the server? If its redhat,centos based could you run exportfs -a and share the output

Also from a client, could you run

showmount -e servername

mount

Share the outputs of both, obviously only share the relevent paths, unless you have nothing to hide

[majorbytes]

that is the client side mount... they are running centos 6 and the file system is xfs on the nfs servers. i will have to login tomorrow to get the rest..

[majorbytes]

Is that the client side mount on the path on the server that is being shared?

Also what distro is the server? If its redhat,centos based could you run exportfs -a and share the output

Also from a client, could you run

showmount -e servername

mount

Share the outputs of both, obviously only share the relevent paths, unless you have nothing to hide

NFS Server
exportfs -a

exportfs: No host name given with /mnt/DataDrive2 (insecure,rw), suggest *(insecure,rw) to avoid warning
exportfs: No host name given with /mnt/DataDrive1 (insecure,rw), suggest *(insecure,rw) to avoid warning
exportfs: No host name given with /mnt/DataDrive3 (sync,rw), suggest *(sync,rw) to avoid warning

showmount -e servername

/mnt/DataDrive3 *
/mnt/DataDrive1 *
/mnt/DataDrive2 *

Client
/mnt/DataDrive2/ARCHIVES *
/mnt/DataDrive1/data *

If I create files on the nfs server in one of the directories on the physical drive, those files still go away.. Takes about a minute or so..

[Arola1982]

Ok so a couple of things. If you put files on the server and the files vanish I dont think its an nfs issue, seconded by the try with samba and the same problem.

However, in the exportfs dump, the no hostname given error, make sure that in /etc/exports when you specify any ips that you use a CIDR format,so something like 192.168.1.34/24 and re export the shares.

In regards to your main issue, I don't have any experience with xfs filesystems, however I did find this which might be of use.

docs.oracle.com/cd/E37670_01/E37355/html/ol_repair_xfs.html

[Arola1982]

That url is https by the way. I cant post properly

Sent from my HUAWEI CRR-L09 using Tapatalk

[majorbytes]

Ok so a couple of things. If you put files on the server and the files vanish I dont think its an nfs issue, seconded by the try with samba and the same problem.

However, in the exportfs dump, the no hostname given error, make sure that in /etc/exports when you specify any ips that you use a CIDR format,so something like 192.168.1.34/24 and re export the shares.

In regards to your main issue, I don't have any experience with xfs filesystems, however I did find this which might be of use.

docs.oracle.com/cd/E37670_01/E37355/html/ol_repair_xfs.html

I think I am dealing with corrupted cache issues.. I am currently doing a consistency check on the raids, then I am going to run xfs_check to see whats up. I had write back enabled on one of the server's raid card, so I disabled that..

I think with the combination of all of the caches and a gazillion runs hitting those nfs shares it might be confusing that thing..

I just ordered 5 - 6tb wd blacks to stick into a local run server to finish my runs while I sort it out...

I use xfs on most of my servers and just started recently using btrfs on a couple of the CentOS 7 servers, they mostly do local runs that get copied over to the shares..