I have a very strange problem and I hope I can represent the situation
clearly enough.

There is a CentOS 5.0 box, which has 4 network interfaces. Two by two,
they are combined in balance-rr mode bonding interfaces.
The peers on both ends are cisco 3650 layer 3 switches, having a port
group etherchannel setup. The load balance policy of the cisco
switches is src-mac.
The server itself has two dual core Xeon processors. Each core handles
the interrupts from a single network adapter. Thus all cores are used
by the all four network adapters and the load is supposed to be
equally spread to the cpus.

Here is a little ascii chart, which would probably represent the setup
more clearly.

Etherchannel both on balance-rr
Po1 bond0 bond1 Po1
INTERNET - | Cisco 3650 | ======== | Server | ======== | Cisco 3650 | --------- LAN

All interfaces are 10/100/1000Mbps.

When working in src-mac mode the setup works just fine. The only
problem is on the INTERNET site of the setup where only one mac
address is seen by the cisco switch, which makes the downlink traffic
to pass only trough one of the bonding network adapters in bond0.

Here is the average cpu usage in this setup:

Average: CPU %user %nice %system %iowait %steal %idle
Average: all 1.70 0.00 43.46 0.00 0.00 54.85
Average: 0 0.80 0.00 77.40 0.00 0.00 21.80
Average: 1 5.98 0.00 5.38 0.20 0.00 88.45
Average: 2 0.00 0.00 56.89 0.00 0.00 43.11
Average: 3 0.00 0.00 33.93 0.00 0.00 66.07

As you see CPU0 is eating more than 70%, because the downlink traffic
passes only trough one of the network adapters and interrupts are
generated only on cpu0. This is reached with a traffic rate near
200Mbps. At near 300Mbps that percent grows and passes 80%-85%.
To optimize the work of the server, I had to change the port-group
mode at the cisco switch from src-mac to dst-ip. Moving the balancing
from layer2 to layer3 setup. This step made the download traffic to
pass pretty equally between the two network adapters. BUT the very
strange problem occur. Just take a look on the cpu usage.

Average: CPU %user %nice %system %iowait %steal %idle
Average: all 0.00 0.00 79.96 0.00 0.00 20.04
Average: 0 0.00 0.00 78.20 0.00 0.00 21.80
Average: 1 0.00 0.00 81.00 0.00 0.00 19.00
Average: 2 0.00 0.00 82.20 0.00 0.00 17.80
Average: 3 0.00 0.00 78.44 0.00 0.00 21.56

All cores jumped near and over 80% utilization with the same bandwidth usage.

I cannot give any meaningful explanation of this? I am starting to
think that the incoming packets from the dst-ip load balancing mode
are mis ordered and the kernel has to put them in the right order...
but that is only some guess.

Is anyone having the same problem? Or any idea what might be the
problem of this insane cpu load? I did not test with different bonding
modes as this is a live server and I can't just like that experiment
on it.

Thanks for your time guys. Will appreciate your help.
Best wishes,