Semaphores/Mutex IPC

[Marin_]

Hi all!

Well im programming in C, using the suse/ubuntu, and i have multiple processes running. I need to share data between them, and to do that i use shared memory. Well, basically its inter-process-communication. Because i have more than one process read/write the (same) segment of shared memory i use semaphores to synchronize that.
My question is, if i must use sempahores or mutex to synchronize the ipc.
I know that mutex is like a binary semaphore, but i have semaphores that take values of 20, because i need to know how many processes are reading the same segment of shared memory, so apparently i can not use mutex.
I read, that there is an attribute to mutex, pthread_mutexattr_setpshared, that allows mutex been shared between multiple processes. But, i think that, to make this possible i need to load the mutex in a segment of shared memory to make that mutex available in other process (not the one that creates it).

1)Mutex can be used to synchronize segments of shared memory between multiple processes?
2)If it is, how can i do that? I need to save (load) the mutex in a segment of shared memory to be available in all processes?
3)If i can use mutex, what is better? Mutex or semaphores?

I hope you can help me, Thank you

[Marin_]

Anyone? Thank you

[Rubberman]

For situations where you have multiple readers and writers to shared memory, I find that using semaphores to be a very error-prone situation, especially in a C-programming environment. There are frameworks for C++ that help deal with this in a consistent and usable manner, but for C you are pretty much on your own. In any case, to address your questions.

1. I'm pretty sure that pthread mutexes cannot be used between processes, and AFAIK there is no system-level mutex for Linux, unlike Windoze.
2. As per my answer to #1, you cannot do that. You will have to implement mutex behavior with semaphores.
3. My preference for the multiple reader+writer situation is to have a server process that manages the memory. There can be a couple of ways to implement that. One way is that readers and writers send messages to the server to read or write to the memory buffer. If the buffer is big, this has a lot of IPC overhead. Another way is for the server to manage the shared memory. Clients send a read or write request to the server and the server does not reply until that client is at the top of the request queue. Then the client can read/write the shared memory with the assurance that it has exclusive access to the memory. This is effectively turning the server into a mutex server and the IPC between it and the client is of a very low bandwidth (a few bits) so you should be able to get good performance factors out of current hardware. From your description of the problem, I think that the second method is the most viable one - using a server as a mutex to gate access to the shared memory. That method also allows multiple readers mostly simultaneous access.

[Marin_]

Thanks for help. I have a few questions. I hope you can reply me.

I have a lot of processes, and they are implemented using shared memory synchronized with semaphores. So its hard for me remake everything, with your suggestion in 3.
Why you tell that using semaphores is an error? Do you think this implementation is too bad?
Because, for me, its impossible change all the struct of the program as you said in point 3.

[Rubberman]

The biggest problem using semaphores in complex scenarios like this is that simple mistakes can easily result in deadlocks and/or race conditions. This is especially true when you have multiple readers and writers. With a single writer, it becomes more simple. If you are going to do this then I strongly recommend that you model it first with a well-ordered finite state machine, and have as many knowledgeable people review it as you can get. More eyes will uncover pernicious conditions better, for sure.

[Rubberman]

Continued:

One of the issues with semaphores that is often ignored until the system locks up, is to deal with dead/dying processes. In most situations, any semaphores a failed process may have touched will not be restored to their correct state automatically, so you have to monitor the health of all participating processes and possibly have the semaphores reset externally. Coding and testing the monitor adds considerably to the complexity of the system. I have been doing large scale, distributed, real time, high-availability, and self-healing systems with a LOT of IPC activity, both local and cross-network, for over 25 years. I have never encountered a situation such as you describe where semaphores are an optimal solution. There is just way too much violation of the KISS principal, and my experience has shown that is the biggest cause of catastrophic systems failure.

My clients' systems cost about $10M USD per hour of down time, and a semaphore-caused deadlock has the potential to corrupt system data to the point where scrapped production ensues. In these cases, a single lot (semiconductor manufacturers - 25 wafers in a cassette) being scrapped will cost in excess of $1M USD. In the systems that control all the operations in a semiconductor fab we typically have 500+ clients and dozens of servers. I have NEVER used a single semaphore to control access to shared data. However, the design is such that it is not necessary. This is what is called an n-tier, shared-nothing architecture. It supports very high transaction volumes (100's of thousands of transactions per hour), very large data sets (tables with 10M or more rows), very large message sizes (from 100 bytes to 10MB per message) and 24x365 operation with 6-sigma plus uptime.

So, model what you want to do. Employ appropriate tools such as UML state and sequence diagrams, etc. Some tools let you simulate and execute the model to find deadlocks and such before you actually commit to code.

[Marin_]

ok. i use semop to avoid race conditions, i think it works.
in my work i have only one writer in the same shared segment and 2 or 3 readers. Its not a big program, like yours.

Its to late for me, change everything. Maybe later i talk to you again about semaphores.

Thank you for your advices

[Rubberman]

ok. i use semop to avoid race conditions, i think it works.
in my work i have only one writer in the same shared segment and 2 or 3 readers. Its not a big program, like yours.

Its to late for me, change everything. Maybe later i talk to you again about semaphores.

Thank you for your advices

Ok. This is a better scenario than I assumed. I'm eating dinner now, but I'll address this use of semaphores to lock a segment of shared memory as soon as I can. It might be tomorrow, however.