Slow authentication via FTP

Hello! I have a perplexing problem for which I would be grateful for some assistance.
We are a small predominantly Windows site with a few Linux machines (Red Hat ES 3) and about 70 Windows Servers (mixture of physical and virtual).
When connecting TO a Linux machine via FTP, authentication is taking about 40 seconds (very slow). The username and password prompts come up instantly. You put in the password, press enter and then it hangs for approx 40 seconds before it eventually connects fine. Transferring files via FTP, once connected, is fast. It's just the authemtication that is slow.
To help the diagnosis, I set up a simple shell script executed 24x7 every 5 minutes by cron, which connects to another linux box via FTP. It records the elapsed time to authenticate.
I have established the following facts:
(1) Authenticating via FTP from Windows to a Windows machine is always instantaneous.
(2) Authenticating via FTP from Windows to a Linux machine is almost always very slow.
(3) Authenticating via FTP from Linux to a Linux machine is almost always very slow
(4) We have occasional periods lasting between 10 and 15 minutes each time, where authenticating via FTP from Linux to a Linux machine is instanteneous. There is no apparent pattern to when these periods occur, but they always last 10 to 15 minutes. For example one day we had 7 of the "fast" periods, another day just 2.
(5) We have 3 Linux machines. When we have one of these "fast" periods, it is fast for all the Linux machines. (2 of the Linux machines are physical, one is virtual).
So there we have it. The evidence suggests to me that the problem is caused by something external to the Linux machines because all the Linux machines seem to be affected (fast or slow) at the same time. We have a proxy server, DNS server and firewalls here, but I'm not a network administrator so I'm not well up on those things.
Any ideas gratefully received - this is one wierd problem where Google has failed to help me!
Many Thanks!

Check the authentication mechanisms on the /etc/nsswitch.conf
If the FTP accounts are local then /etc/passwd should authenticate the users....
so on /etc/nsswitch.conf the priority for lookup will be,

passwd files nis ldap (I would remove both nis and ldap and check if this makes a difference)
group files nis ldap
shadow files nis ldap

Also let me know if NIS service is running

Quote:

---

Originally Posted by Legolasthehansy

Check the authentication mechanisms on the /etc/nsswitch.conf
If the FTP accounts are local then /etc/passwd should authenticate the users....
so on /etc/nsswitch.conf the priority for lookup will be,

passwd files nis ldap (I would remove both nis and ldap and check if this makes a difference)
group files nis ldap
shadow files nis ldap

Also let me know if NIS service is running

---

Thank you for your post Legolasthehansy.
I have checked /etc/nsswitch.conf on all the Linux machines and it has the following...
passwd: files
shadow: files
group: files
hosts: files dns
So user lookup is in /etc/passwd, and hosts lookup is first in /etc/hosts and second in DNS. Note that our hosts files do not contain entries for the other machines, so lookup can only be via DNS.
I have check /etc/resolv.conf on all the machines, and the ip address of the primary DNS Server is first in the nameserver list (we use the Domain Controllers as DNS servers).
It's still a mystery!

Oops, forgot to say in the last post in respect of NIS, that I have checked on all of the servers and none of the yp* services are running
(ypbind, yppasswdd, ypserv, ypxfrd).

Strange indeed...
Another test I would run is try running other services from windows to linux.
Try SSH from an ssh client application (like putty) to the linux servers.. do you see the same behavior?
I have seen extremely slow authentication to linux systems from windows but that was because NIS stopped running on the machine.. but since you don't run NIS on these machines that is ruled out as well..
How about trying to run tcpdump on the linux machine and/or wireshark in windows and checking their dumps..
I would need to put my thinking hat on to think of any other tests you can run.. probably some one here might have better ideas..

Hello! Connecting via ssh Putty is always instant. I'm looking at tcpdump now but I'm not familiar with it so it will take a while to work it out. Just to be clear, FTP authentication is slow Linux to Linux, and Windows to Linux. When we get one of the brief occasions when FTP authentication is temporarily fast, it is fast Linux to Linux, and Windows to Linux, for all Linux machines.

I have tried some other stuff....
(1) host 123.123.123.123 (the IP address of one of the connecting IP addresses) - the command completes instantly and comes back with the correct ip address.
(2) I chose a pair of Linux machines, and made sure that both /etc/host files had entries for both hostnames. - FTP authentication is still slow.
(3) ftp localhost - FTP authentication is still slow.
(4) ftp fred (where I am signed on to machine fred) - FTP authentication is still slow.
(5) ftp 127.0.0.1 - FTP authentication is still slow.
(6) I have put an "nslookup" into the 24x7 FTP shell script, immediately before and after the ftp connection. This has shown that it is always using the same DNS server to do the lookup, during the "fast" and "slow" FTP authentication periods.

Hello! I have used tcpdump and wireshark to try to work out what is happening. As far as I can tell. I have established the following facts:
(A) The delay is occurring on the server end, not the client end. Just in case I've got my terminology wrong, by this I mean if I am invoking the FTP application on host FRED and establishing a connection to remote host DELORES, then the delay is happening on DELORES.
(B) On the remote server (DELORES in the example) I get about 30 DNS protocol network packets being sent to the primary DNS server. The "info" column in the Wireshark display shows about 3 different types of similar description
[1] Standard query A kerberos.example.com
[2] Standard query AAAA kerberos.example.com
[3] Standard query AAAA kerberos.example.com.bongo.co.uk (where bongo.co.uk is our domain name, changed here for security reasons).
Now I don't know anything about kerberos, except that on the Linux machines it hasn't been intentionally set up. But the "kerberos.example.com" suggests an unconfigured kerberos setup (kerberos.example.com appears in the /etc/krb5.conf file which apparently is one of the places you configure kerberos).
I have stopped at this point as I don't want to go changing stuff to do with kerberos as I don't know what I'm doing, and I don't want to risk locking myself out of the machine by meddling with authentication from a position of ignorance.
Any ideas? Does kerberos need to be disabled in some way?

Wow - I think it's finally solved!
I had set up all the machines to run gssftp via xinetd. I chose gssftp for no particular reason at the time. Apparently gssftp is "a kerberized xinetd-based FTP daemon which does not pass authentication information over the network". The big clue being the word "kerberized". I have now disabled gssftp on the 2 test servers, and have set up vsftpd to run as a service via xinetd instead of gssftp. This has solved the problem. Connecting by FTP is now fast all the time. So I just need to roll it out to the live machines now.
There's a noticeable difference in the messages that appear when I initiate the ftp connection from the client. Here's what I get when gssftp is running as the server:
# ftp fred
Connected to fred.bongo.co.uk.
220 fred.bongo.co.uk FTP server (Version 5.60) ready.
334 Using authentication type GSSAPI; ADAT must follow
GSSAPI accepted as authentication type
GSSAPI error major: Miscellaneous failure
GSSAPI error minor: No credentials cache found
GSSAPI error: initializing context
GSSAPI authentication failed
334 Using authentication type KERBEROS_V4; ADAT must follow
KERBEROS_V4 accepted as authentication type
Kerberos V4 krb_mk_req failed: You have no tickets cached
Name (fred:root):
And here's what I get when vsftpd is running as the server:
# ftp dolores
Connected to dolores.bongo.co.uk.
220 (vsFTPd 2.0.1)
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (dolores:root):
So hopefully that's it. The step that led me to discover kerberos being involved was using tcpdump and wireshark to notice the packets being sent to a non-existent kerberos server throughout the delay period. Many thanks to legolastlehansy who along with another guy provided the tcpdump suggestion that lead to the breakthrough!

Ooops, I didn't mean to say packets where being sent to a non-existent kerberos server. I should have said packets were being sent to the primary DNS server and the contents of the packets quoted a non-existent DNS server.