TCP RST after duplicated ack
I have a very strange problem with my linux server.
I had set nat using iptables that makes redirection of port 111 from my proxy server to application server.
The problem is when client connected to proxy to port 111 is losing some tcp segments sent from app server through proxy. Then when first DUP ACK arrives to proxy, proxy server is resetting the connection by sending RST to client. The RST is not sent by application server.
Both servers are HP DL380 with kernel 2.6.23-rc2 #5 SMP i686
NICs are Broadcom's (integrated with mainboard) and Intel (e1000). Every two network interfaces are bonded. My version of iptables is 1.2.11.
I've disabled TSO option and eliminated possibility of MTU problems.
Sometimes retransmission is working fine (1-2 for 10 tries). But most of the time proxy is sending resets. In a working case there are about 20 dup ack's and then retransmission occurs. In a problematic case reset is sent immediately after first dup ack is received.
Here are my iptables rules:
/sbin/iptables -t nat -A PREROUTING -p tcp -i bond2 --dport 111 --destination $IP1 -j DNAT --to-destination $IP2
/sbin/iptables -t nat -A POSTROUTING -o bond0 -j SNAT --to $IP1
/sbin/iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
May it be some kind of bug maybe in netfilter used to this kernel build?
Any ideas what can be checked before updating kernel?