X Windows Loading Sluggishly from Server

[HappyDad]

I have 3 servers, all of them are running Red Hat Linux 5.6 on i686 architecture.

I'm having an odd issue with applications being "served" to my windows desktop from the servers (via ssh).

Here's the issue... on all three servers I have eclipse installed locally. I have my DISPLAY set to my Windows desktop and I'm using Cygwin/X to display X windows.

One of the servers works just fine, I ssh into the machine, I call `eclipse` and the splash logo will pop up on my Windows desktop immediately and takes only about 7-10 seconds to completely load.

The problem is with the other two servers. When I try the same thing with either of them, the eclipse splash screen will spluggishly appear after about 2 seconds and it will take 3-4 Minutes before eclipse is fully loaded.

I've ran `top` while doing this and there is no evidence of anything hogging the memory or cpu.

A couple of quirky things I did notice, not sure if they matter.
1) Is that the machine which runs fine shows a `runlevel` of 3... whereas the other two machines have a `runlevel` of unknown.
2) If I do a `netstat -t` the "good" machine seems to exchange packets really quick, whereas the other machines are exchanging packets really slow. Not sure if this would matter though, being that eclipse is being loaded locally and displayed through an X window.


Anyone have any idea what could be causing the issue or maybe some suggestions I can try to solve this issue?

EDIT: Just to clarify, these are more than capable machines. All three of them are Dell PowerEdge Servers T710 (dual quad-core CPUs - 8 cores). There should be no problem loading eclipse in a snap.

Thanks,
- Darin

[Rubberman]

Assuming that the systems are running without a local x server (no GUI or GDM graphic login), then they should all be running at run-level 3. The "unknown" run level seems to indicate that the systems in question are not init'ing properly. Check the log files to see if there is something untoward going on when booting them.

[HappyDad]

I overlooked one thing with regard to the "runlevel bug" I was experiencing.

If I type in `sudo runlevel`, then it returns the correct values of N 3... upon further investigation I determined that this is because the file /var/run/utmp had the permissions of 600. Once I changed the permission of utmp to 644, then running `runlevel` from my local linux account also returned the correct values of N 3.

However, if I do a reboot, the file permissions on utmp get set back to 600... not sure if this causes boot issues when trying to configure the system.

The system that is running "a-ok" has the correct permissions on utmp of 644.

So still trying to figure out what's causing these lag issues... back to investigating.

Let me know if anyone has any ideas or if I'm leaving out any information that might prove fruitful.

EDIT: P.S. The /var/log/boot log has no recent entries to indicate that anything is failing at boot time. I tried reinstalling the kernel to see if that would fix the issue, but still "no go".


- Darin

[HappyDad]

Whelp, 10 hours of labor later, I figured it out... or I should say, I got it working.

I compared the running services on the good working server to those on one of the servers that was having the 'laggy' issues when launching X Windows, and here's a list of the services that weren't running on the 'laggy' server.

acpid
anacron
atd
avahi-daemon
ip6tables
iptables
isdn
mdmonitor
netfs
nfslock
rpcgssd
rpcidmupd
smartd
xinetd
yum-updated

Not knowing exactly which service might be causing the issue, I turned on the services above on the 'laggy' server, rebooted, and now the issue is solved.

A good majority of these services are network related, maybe someone might have an idea as to which of these services would have been causing the behaviour that was being exhibited by not being turned on.

I guess I learned one thing... don't ever trust your IT System Admin to be consistent with the installs they performed.

On a side note, the `runlevel` command, when launched from my local account, still shows unknown, so that had nothing to do with the issue I was experiencing.


- Darin

[Rubberman]

acpid - Advanced Configuration and Power Interface event daemon - notifies user space programs of ACPI events (hardware plug-ins and removal for example).
anacron - the cron daemon - runs cron scripts at appropriate times
atd - run jobs queued for later execution
avahi-daemon - this is what records DNS IP addresses so local programs can avoid network lookups of frequently accessed network resources
ip6tables/iptables - these are your system firewalls effectively for ipv4 and ipv6
isdn - don't have it on my system, but I suspect that it manages isdn network connections. Unlikely to be used these days.
mdmonitor - I suspect this is the same as my mdmon daemon - it keeps user space programs informed when there are updates to system metadata by the kernel for disc arrays such as disk failures, etc.
netfs - Mounts and unmounts Network File System (NFS), Windows (SMB), and Netware (NCP) file systems
nfslock - pretty self-explanatory - it manages file locks on NFS file systems.
rpcgssd - security api for rpc protocol support
rpcidmupd - part of the nfs server software stack
smartd - monitors disc drives with SMART support, such as temperature, read/write failures, number of remapped (bad) sectors, etc.
xinetd - the replacement for the inetd daemon, which starts services such as ftp, telnet, and such on demand when a client attempts to connect to the appropriate ports.
yum-updated - are you sure this isn't yum-updatesd? In any case, it is a daemon to automatically look for and install package updates for system software.

As you can see, many of these probably won't affect your systems performance if they aren't installed. The avahi-daemon not running can adversely affect performance. You can try shutting them down one at a time and testing for which are affecting you.

[HappyDad]

Thanks for the information Rubberman. I went through and disabled each service individually and then ran eclipse, but eclipse has no issues starting up now.

Oh well, I guess whatever service that it was has fixed the issue. Perhaps the service that was missing wrote something to the disk in a one-time fashion once it was started, and now that the data is there, all is good.

Well I'm a little bit smarter in the linux filesystem area... I think I pretty much looked at every file within /etc and all it's sub-directories. Now I can get on with real work.


Thanks,
- Darin

[HappyDad]

Just for clarification on this issue. I experienced the issue again, due to IT performing updates on the server. It turned out to be nfslock being turned off that was causing the slow down. Just wanted to post that here so that this thread is now complete.