recv appears to return wrong data length

[Micke_N]

Hi,
I am working on a program that receives data from a client and writes it to a log file. The problem I have is that sometimes (seemes to be completely random) a packet with data is missing. The data in a TCP packet is organized by a struct and holds 16 samples and is always 208 bytes. I have verified with Wireshark that all the packets arrive to the server in good shape and after a little investigation I found that the packet is actually read by my program but the data in is just crap. This is how I read the socket:

PKG_MAX_SIZE = 384;

struct pkg_t{
    int len;
    char data[PKG_MAX_SIZE];
};

int main(){
  struct pkg_t pkg;

  // initiate variables and socket...

  for(;;){
    s = accept( ... );

    // sizeof(pkg) = 208
    bytes = recv(client[i].socket, (void*)&pkg, sizeof(pkg), 0);

    // handle data

  } // for loop

}// main

I found that a packet that was "crap" only had 28 bytes in it (at least that is what recv returns) and then I saw that recv thought that the preceding packet was 388 bytes. As all packets are supposed to be 208 B, I assume that recv first returns too much data (208 + 180 B), but that does not cause bad behaviour as the extra data at the end is ignored. It then reads the remaining 28 B the next time the loop comes to the recv and as the data is not organized as expected I get an error.

Does anyone have an explanation to this? Any suggestions on how I can fix it are greatly appreciated.

[gerard4143]

Not sure how you arrive at this value 'sizeof(pkg) = 208' since you have

PKG_MAX_SIZE = 384;

struct pkg_t{
    int len;
    char data[PKG_MAX_SIZE];
};

struct pkg_t pkg;

I get sizeof(pkg) = 388 on my machine or am I missing something here...G4143

[Micke_N]

You are right, of course it shoud be sizeof(pkg)=388, and therefore it also makes sense that recv returns 388 B - it has filled the struct. What I've found is that it is probably two packets ready to be read by recv, and there is no null character or anything else to separate them so recv simply returns the whole lot.
I (kind of) solved it by reading the last 28 B and storing the 180+28 in a buffer that is processed the next time the loop comes around. I suppose the best way to do it is to let recv read the packet(s) into a pkg_t array that is iterated similar.

Thanks for your reply!
Best Regards,
M

[Rubberman]

One thing to remember about TCP connections, is that it is a stream protocol. It assumes no structure to the data, and can/will buffer and concatenate multiple sends into a single packet, so recv() returns as much as was on the wire when you asked for it. The total data buffered can be sent in any number of packets, and your receiving system will concatenate them as received, removing the data from the ring buffer as your program asks for it. So, you need to either know exactly how to segregate each message from the stream (by message size, by some size field such as you use, or by some delimiter), and keep reading until you get a full message. Also remember that TCP (opposed to UDP) is a reliable protocol and you will receive the data in the order sent. It just might arrive in dribs and drabs...

So, in your example, you can receive up to the full package size (388 bytes, including the size element), though you might be expecting only 208 bytes (204 bytes data + 4 bytes size element). You should look at the size element first, in order to know exactly how many bytes are in the data package, then store the excess into a buffer to be concatenated with the next packet received.

[Micke_N]

Thanks Rubberman,
The code now handles the situation correctly if there are two data chunks read by recv, the problem now is that sometimes there are three... Doh!
I think the recv problem above is an expression of an underlying performance issue. With 100 samples per second and 16 samples per paket there will be 6.25 packet/second (6.25*208 = 1300 B/s) to handle which do not seem like much to me but clearly its not up to me to decide what is "much".
I might post the code for the data handling if I cannot solve it myself.

[Micke_N]

I cannot understand how my code is not fast enough.... I have commented just about everything out and it still does not run fast enough to read one packet at a time, see below:

for(;;)
        // Test if there are any new connections
        socklen_t alen = sizeof(a);


        // Check for new connections from clients. If a new connection was
        // found save it to the clients array. If there are no available
        // sockets left in array, then deny the client.
        s = accept(host_socket, (struct sockaddr *)&a, &alen);

        // If a valid socket is returned a new connection was accepted
        if(s != ERR){

            // Set the new socket to non blocking too
            if (fcntl(s, F_SETFL, O_NDELAY) < 0) {
                perror("Error while setting socket to non blocking");
                return ERR;
            }

            // Find first available socket in sockets array
            for(i = 0; i < MAX_CLIENTS; i++){
                if(client[i].socket == 0){
                    client[i].socket = s;
                    client[i].addr = a;
                    printf("Client (%d) with IP %s connected.\n",
                            i, inet_ntoa(a.sin_addr));
                    break;
                }
            }

            // If there were no available sockets in array -
            // deny the client connection
            if(i == MAX_CLIENTS){
                printf("No empty slots for client.\n");
                if(shutdown(s, SHUT_RDWR) < 0){
                    perror("Error while shutting down socket");
                    return ERR;
                }
            }
        } // End of new connection


        // Test if there are any new data from any of the clients connected.
        // Looping through all sockets in list and do recv() at all sockets in
        // use.
        for(i = 0; i < MAX_CLIENTS; i ++){

            // Test if socket is in use
            if(client[i].socket){
            		bytes = recv(client[i].socket, (void*)&pkg, sizeof(pkg), 0); 
            		printf("recv: %d\n", bytes);
   
                	                  sample_pkg = (struct sample_pkg_t *)pkg.data;

                                if(bytes > 0){

                                  switch(sample_pkg->type){

                                  case PKG_SAMPLES:

                                       for(si = 0; si < sample_pkg->nr_samples; si++){
            				samplesinfile++;
                                       }
                                       break;
                                  }//switch

                     }
          }
}

I have set MAX_CLIENTS to 2 (and I only use 1 for testing). I can only get the loop around 1-2 times/second. Very frustrating...
Any suggestions welcome...

[Rubberman]

How much data is delivered between calls to recv() is dependent on many factors, such as congestion on the wire, buffering on both ends, NIC performance, duplex of the connection(s)... the list is almost endless. In any case, ethernet and TCP/IP are not definitely NOT deterministic protocols! So, your programming has to take that into account. Anyway, on a LAN with 100mbps or 1gbps NIC's and switches, you should easily get many, many thousands of packets per second between your systems. You can compute the theoretical maximum for your message size, assuming each gets a packet, which BTW can be from (I think) about 22bytes + header to about 1500 bytes + header, unless your LAN supports jumbo frames (bigger than 1500 bytes). So, multiply 388 bytes x 8 bits == 3104bits / message. 100mbps (fast enet) / 3104 ~= 32000 messages / second. Divide that by 2 for the packet acks and you get about 15000-16000 messages / second. This is just a crude estimate, but certainly in the "ballpark". That's a latency of only 62 microseconds per packet. So, even if each sample was packed into a separate message (100/sec), your network should still be over 2 orders of magnitude faster than that (10ms / sample vs 0.062ms / packet).

Since there is no timeout in your loop, it is probably cycling much faster than the network is passing data. That is why we usually use the select() function with a timer for determining when to read the data from the network or raise an exception because nothing came in as expected.

[Micke_N]

Thanks again Rubberman,
I did some good old fashion troubleshooting by commenting stuff out and I had to eliminate just about the whole program before the loop started spinning at a rate that is expected. I then started to uncomment code to see where the loop was held up (could not really see why when I commented the code) but to my surprise the loop kept its speed.
When I uncommented a line at one point I had some trouble with the { and } not matching, I would expect that the compiler would complain if there was a mismatch but perhaps I had gotten both an opening and closing paranthesis wrong and it resulted in a buildable and runnable program.
I have also rewritten the code handling the recv again, it now puts the data into a buffer whos size is a multiple (currently ten) of the packet size that will prevent partial packets from being fetched by recv. After the recv there is a loop iterating over the packets recieved before going back for another recv call. From my tests so far I can see that the loop(s) are fast enough to handle a pile-up of four packets.

Now that I got it working with 100 Hz at the client side I will keep doubling the sample rate until something breaks again.... All in the spirit of Rubberman's signature!