UDP socket send fails continuously for 10 minutes and then works

[udaykishore]

Hi,

I have a multi-threaded Java program in which each thread sends out asynchronous SNMP request messages to the IP addresses on the network and each thread uses a dedicated local UDP port to send out the packets. The IP addresses are shared among the threads, such that each thread sends packets to a unique set of IP addresses. What I notice is the following: Around 1020 requests get sent successfully (from all threads put together) in about 100-200 millisec of time, and then for the next 10 minutes, every send request fails with the error message "No buffer space available". And then another ~1020 requests are sent out successfully. And then failure to send any request for 10 minutes. This cycle repeats.

I checked the output of iptables with the command "iptables -t <table> -n -L" for the tables "filter", "nat" and "mangle", and found that iptables does not have any rules configured in it.

I changed the txqueuelength setting on the eth0 from 100 to 1000. That did not help either.

I tried searching for such a problem being reported in this website and did not find any. Any help is greatly appreciated.

With best regards,
Uday

[Rubberman]

Perhaps the problem is on the receiving end. Depending upon how long they take to process the messages received, their receive buffers have likely filled up and until they process enough of the received messages they cannot handle any more, replying with the error you are getting. Just a SWAG since you haven't provided enough information to be definitive about the cause.

[udaykishore]

Thank you for the reply. I will try to provide more information here.

The Linux machine which is sending these SNMP requests is connected to a 1G Ethernet Switch port. And there is no one destination to which these messages are destined to. Each request is sent to a different IP address. The sender is sending one request each for an IP address on the network to determine if that IP address is alive and is SNMP enabled or not.

Should I suspect some problem on the Ethernet Switch port?

[Rubberman]

Thank you for the reply. I will try to provide more information here.

The Linux machine which is sending these SNMP requests is connected to a 1G Ethernet Switch port. And there is no one destination to which these messages are destined to. Each request is sent to a different IP address. The sender is sending one request each for an IP address on the network to determine if that IP address is alive and is SNMP enabled or not.

Should I suspect some problem on the Ethernet Switch port?

So, how many unique addresses are on your network? It's possible that network congestion somewhere between your switch and the receiving end's switch is causing your switch to get a "buffer full" condition. So, how big are these packets? Even at a full 1500 byte packet size and assuming each message is only 1 or 2 packets, 1020 messages is only a couple of megabytes which should not be an issue for all but the most inadequate switches available. So another question is, what is the make+model of the switch at your end?

[udaykishore]

Each packet is very small, hardly a couple of hundreds of bytes in size. The number of IP addresses is approximately 4000. And the switches are Juniper switches.

BTW, this problem does not seem to happen from every sender machine. I tried running the sender application from a few other Linux machines (which are also connected to ports on some other Juniper Ethernet Switches), and I dont see the issue.

Other than iptables, is there a possibility that another kind of firewall is running on the sender Linux machine, which is rate-limiting to 1020 messages per second and then blocking any further outgoing messages from the local port for 10 minutes if the limit is exceeded?

Does the rate-limiting work in a way that after sending 1020 outgoing messages, it drops the remaining outgoing messages from that port within that second? If this were the case, I should be seeing the dropped count going up in the netstat output. But I dont see that.

[Rubberman]

I'd look at the configuration of this particular switch. I don't think it's the Linux system that's the problem. It's just possible that the switch has been configured for rate limiting from individual addresses/ports. Have you tried changing ports on the switch? If that works, then the port is at fault, or the switch has rate-limited that port. If that doesn't work, then either the switch is defective, or it has been misconfigured.