Network interface slow

[matcho]

Here's a corker . . . . Can't work it out, and would appreciated more eyes on it.

Cenario: Many servers off a cisco 2950 (100Mb), all of which can transfer (scp,rsync) data at about 10MB/s sustained (testing with 1 gig file created with dd and /dev/zero). 1 server, which happens to be the router/transparent proxy for the network will not sustain any decent throughput. Usually starting off fast, then quite quickly slowing down to a halt or a crawl. This is most visible using scp as it shows the speed until it stalls.

At first I thought it may have been an issue with the switch, so I cleared interface counters, checked for input/output errors - none. Verified that the speed and duplex weren't mismatching, spent some good time ensuring this wasn't some crap half-duplex problem.

I changed the cables, and went through a different switch, not good. The server has three interfaces, 2 of which is the same, 1 different. I situated my laptop on the other side (dmz with the different nic) and tested, still bad. Even with a crossover cable, it's still slow.

There is no bus, cpu, memory or disk, power or any other hardware, unless it is NIC's, but all 3 ? With different 2 vendors ? The server is seriously grunty, and basically doing nothing.

I tcpdumped one of these transfers, thinking there could've been an mtu issue - none. The mtu was dead consistant, but I lowered it to 1400 anyway and began testing. The server has a long firewall on it, it makes no difference when it is loaded or flushed, and policies set to ACCEPT.

A an extreme far out was that ecn messages may have been recieved, but that wasn't the case.

I played around with buffers and memory stuff with sysctl, but left it pretty clean. pretty muck just set rp_filter and accept_source_route.

The box is dual 2.4 Xeon, 4G Ram, sata2 disks, CentOS 4.4 running an out-of-the-box 2.6.9 kernel.

I pretty much have no idea where to take it from here. Most frustrated, as I haven't really ever seen things so stubborn.

I will greatly appreciate any thoughts or ways forward

[matcho]

oops . . . Forgot to mention, it is only in 1 direction - to the server. Anything initiated from the server is just as fast as the other servers.