I have two Red Hat servers in different sites that are running a line of business application requiring frequent replication. Recently we had a situation where the replication traffic was eating up all of our T1 line, and the system became unresponsive. We had to shut down one of the servers to get back to normal.

I am trying to set up queueing discipline on the eth0 interface so that the replication, which is done using scp, is low priority. I also want to make sure that it doesn't consume more than 50% of our bandwidth.

I set up the traffic control accordingly:
Code:
# tc qdisc add dev eth0 root handle 1: prio
# tc qdisc add dev eth0 parent 1:1 handle 10: sfq
# tc qdisc add dev eth0 parent 1:2 handle 20: sfq
# tc qdisc add dev eth0 parent 1:3 handle 30: tbf burst 20480 limit 20480 mtu 1514 rate 768kbit
After setting this up on a test machine, I tried copying a large file with ftp, and the transfer was indeed limited to 768kbps (50% of a T1), which means that ftp data was being put into Band 2 as desired. I have read that scp has similar behavior to ftp, in that it puts control traffic as normal priority 0x0 TOS, and data traffic as 0x8 TOS (maximize-throughput), which gets sent to the lowest priority Band 2.

After sending a 75MB file by ftp, I confirmed it by checking the statistics:
Code:
# tc -s qdisc show
qdisc prio 1: dev eth0 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 79103237 bytes 52298 pkt (dropped 796, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 10: dev eth0 parent 1:1 limit 128p quantum 1514b 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 20: dev eth0 parent 1:2 limit 128p quantum 1514b 
 Sent 3466 bytes 45 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc tbf 30: dev eth0 parent 1:3 rate 6176Kbit burst 20Kb lat 0us 
 Sent 79099771 bytes 52253 pkt (dropped 796, overlimits 92386 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0
The problem is that I cannot figure out WHY it is going to Band 2. The default priomap for a prio qdisc, and also for the default pfifo_fast qdisc, is
Code:
priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
This means that traffic with TOS 0x0 should go to Band 1, right? (first number in the priomap). And traffic with TOS 0x8, like ftp and scp data, should go to the 9th number listed (number 8 starting from 0), which in the default priomap IS ALSO BAND 1.

How the heck is my traffic going to Band 2 with that priomap? I can't put this into production until I really understand how it works!