C: Sending data to multiple sockets at once?

[Orakio]

I would like to know what is the best way to write the same buffer to multiple
fds in a single go.

A little background.

I have a program that listens to a number of connected sockets using epoll.
When some data is recieved it is written to every fd (broadcast) as follows
in pseudo code:

read into buffer

for each file descriptor
write buffer to current file descriptor

Indeed it is actually far more complex than that but basically that's the
important part.

It is likely the buffer is to be reused imediately after and from what I have
read it copies the buffer before returning so it is safe to do so.

The problem is here is that surely it is copying the buffer as many times as
there are file descriptors potentually using a great deal of memory. The buffer
can be rather large. What the program actually does is read from all FDs into
the buffer and at most 100 times a second force flushes it or flushes it if it
is full.

What I would like to know is how I can tell it to use the same buffer for each
call. I would rather not have a version that just doesn't copy the buffer and
uses the one provided. The obvious reason for this is that I really don't want
the hassle of having to make my own mega buffer so old things can be kept or
to have to wait or check with epoll for when the data has been written. I am
using non-blocking pretty much everywhere but really either way it's still
inefficient if it has to copy the same buffer each time.

I might be entirely wrong about the way write works but I can't see how. I want
the server to be able to handle high loads and while the code can handle it this
system call seems to potentially be most unsuitable.

It would just be nice to be able to do something like:


got=read(buffer);
ptr=socket_static_buffer_make(buffer,got);

for(i=0;i<sockets;i++)
static_buffer_write(socket[i],ptr);

socket_static_buffer_free(ptr); // Doesn't actually free it as such, just
// notifies that we are done with it.

I've looked at sendmsg (sockets.h) rather than write but it does not say if it
copies the buffer or not. Also it may return before finishing. All in all
finding out when the kernel is done with it would just be way too much hassle
when it would be easier for userspace to tell the kernel when it's done.

I can't find anything useful for ioctl.

I've had a look at these new tee and splice functions. They're the closest thing
in the direction of what I'm looking for but still no good.

So is there anyway to do it or any I stuck with this horrible mess?

[wje_lf]

If you're willing to go UDP instead of TCP, use multicasting. It's an involved topic, as you can see from the wikipedia article. I've never used multicasting, but I highly recommend that you skip all that web stuff and read the 30 pages or so about multicasting that W. Richard Stevens of happy memory wrote in UNIX Network Programming, Volume 1, published by Addison Wesley. The book is well worth the price for all the other network programming stuff in it, too.

Hope this helps.

[Orakio]

Multicasting did briefly cross my mind along with SO_HDRINCL (which still needs standard IO so doesn't help).

Unfortunately there are some limitations.

1: Has to be TCP.
2: Goes over the internet.
3: Has to be accessible by things like Java applets and Flash. Doing complex things on these can be troublesome enough even without their restricted permissions.

The only multicast system I can think of is to somehow locally set up a multicast IP and registering/unregistering connected IPs to it forcefully and perhaps using some crazy packet mangling/inverse nat. Implementing that and getting it to work would be just too much. It may even not be possible.

[wje_lf]

I can't think of a solution, then. The best you can do is non-blocking I/O everywhere and select().

Unless you want to rewrite the protocol stack. That would be an excellent adventure, though maybe not worth the time and headache.