I make use of a lot of torrent traffic on my network, and so about a year ago I started using tc htb queuing in conjunction with iptables to give priority to non-torrent traffic. The setup that I use to configure tc is as follows:
Code:
tc qdisc add dev eth0 root handle 1: htb default 60
tc class add dev eth0 parent 1: classid 1:1 htb rate 1850kbit

#Setup classes
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 500kbit ceil 1850kbit prio 0
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 500kbit ceil 1850kbit prio 1
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 500kbit ceil 1850kbit prio 2
tc class add dev eth0 parent 1:1 classid 1:40 htb rate 500kbit ceil 1850kbit prio 3
tc class add dev eth0 parent 1:1 classid 1:50 htb rate 500kbit ceil 1850kbit prio 4
tc class add dev eth0 parent 1:1 classid 1:60 htb rate 320kbit ceil 1850kbit prio 5
As I understand it, it sets up a bunch of classes with different priority levels, and then defaults into a class with the least priority. Then in my iptables script I go through and essentially "white list" traffic that I want given a higher priority. I found this more effective than trying to classify the torrent traffic itself because it jumps across all sorts of ports and destination hosts. I am classifying the good traffic as follows:
Code:
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --syn -m length --length 40:68 -j CLASSIFY  --set-class 1:10
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --tcp-flags ALL SYN,ACK -m length --length 40:68  -j CLASSIFY --set-class 1:10
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --tcp-flags ALL ACK -m length --length 40:100  -j CLASSIFY --set-class 1:10
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --tcp-flags ALL RST -j CLASSIFY --set-class 1:10
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --tcp-flags ALL ACK,RST -j CLASSIFY --set-class 1:10
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --tcp-flags ALL ACK,FIN -j CLASSIFY  --set-class 1:10

$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --dport 22 -j CLASSIFY --set-class 1:20
$IPTABLES -t mangle -A POSTROUTING -o $INET_IFACE -p tcp --sport 22 -j CLASSIFY --set-class 1:20
...
This has been extremely effective. I can let torrents run, do a bandwidth speed test and watch it drop down the torrent traffic to almost nothing, while giving me almost my exact bandwidth allocation reading on the successful speed test.

Where I run into problems is that recently I tried to move my torrent client to my router/gateway/firewall and run it from there. Even though the traffic is still exiting out of my internet interface (eth0) and to my understanding still traversing the POSTROUTING chain (as well as the OUTPUT chain) tc doesn't seem to be touching it. Why does all of the torrent traffic generated locally not get defaulted into the 1:60 class like all of the torrent traffic from the LAN? Let me know if there is any more information I can post. I really appreciate your help, this problem is driving me nuts.