Problem with mutexes

[striker07]

Hi!

I'm using the Posix's mutexes in a project. I have 3 threads and 3 queues. The problem is: I used the mutexes to "protect" the access to the queues. But the first started thread monopolizes the use of the queues. See a example code:

// Global vars
myQueue queue; // is my custom implementation os queue
pthread_mutex_t theMutex= PTHREAD_MUTEX_INITIALIZER;

int main(void)
{
   pthread_t thread1, thread2;

   // Some code... then

   thread1return = pthread_create(&thread1, NULL, thread1func, NULL);
   thread2return = pthread_create(&thread2, NULL, thread2func, NULL);

   // The code continues...
   pthread_join(thread1, NULL);
   pthread_join(thread2, NULL);
}

void *thread1func(void *args)
{
   while (1)
   {   
      pthread_mutex_lock(&theMutex);
      // Use the queue...
      pthread_mutex_unlock(&theMutex);
   }
}

void *thread2func(void *args)
{
   while (1)
   {
      pthread_mutex_lock(&theMutex);
      // Use the queue...
      pthread_mutex_unlock(&theMutex);
   }
}

Anyone can help me?

[tetsujin]

Hi!

I'm using the Posix's mutexes in a project. I have 3 threads and 3 queues. The problem is: I used the mutexes to "protect" the access to the queues. But the first started thread monopolizes the use of the queues.

Hm, I'm not entirely sure what's going on here. If I try the attached code, it works fine:

#include <pthread.h>
#include <stdio.h>

pthread_mutex_t mutex;

void* f1(void* args) {
    while (1) {
        pthread_mutex_lock(&mutex);
        puts("peas");
        pthread_mutex_unlock(&mutex);
    }
    return 0;
}

void* f2(void* args) {
    while (1) {
        pthread_mutex_lock(&mutex);
        puts("carrots");
        pthread_mutex_unlock(&mutex);
    }
    return 0;
}

int main(int argc, char** argv) {
    pthread_t thread1, thread2;

    pthread_mutex_init(&mutex, 0);
    pthread_create(&thread1, NULL, f1, NULL);
    pthread_create(&thread2, NULL, f2, NULL);

    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);

    pthread_mutex_destroy(&mutex);

    return 0;
}

However, if I give each thread a call to sleep(1) or a moderately time-consuming calculation (Calculating the first 10,000 primes) inside the critical region, then all I see is "peas" and never "carrots".

When you have two processes blocking on the same mutex and the mutex becomes available - the prioritization between those threads is an arbitrary function of the scheduler. Based on man pages I found for pthread_mutex_unlock I thought that call would wake one of the processes that's waiting for the semaphore with the result that another thread would get the semaphore before the first could re-lock it... But I guess that's not the case.

I tried a few approaches to try and solve this in the two- or three-thread case. I had hoped sched_yield() after the call to pthread_mutex_unlock() would do the trick, but no dice. I found two approaches that seem to work:

The first is a two-level semaphore:

void* f1(void* args) {
    while (1) {
        pthread_mutex_lock(&mutex2);
        pthread_mutex_lock(&mutex1);
        pthread_mutex_unlock(&mutex2);
        puts("peas");
        primes();
        pthread_mutex_unlock(&mutex1);
    }
    return 0;
}

Basically, a thread has to have the lock on mutex2 before it can even attempt to get mutex1. When thread 1 gets mutex1, it then immediately releases mutex2 - which often means one of the other threads will be able to grab mutex 2 before thread 1 can grab it again. (When thread 1 spends a significant amount of time in the critical region, one of the other threads will apparently always get mutex 2 before thread 1 can grab it again... If there's no significant delay inside the loop (i.e. the call to primes() is removed) then it still seems like the different threads get their fair share of access but here and there you'll see one thread dominating...

The other method that seemed to work is putting a brief usleep() call after each unlock. That puts the active thread to sleep long enough that one of the other threads is almost guaranteed to get a chance to pick it up... It's not a particularly attractive solution and it's probably not super-reliable either.

Another approach would be to explicitly control the scheduling to give each thread a turn in sequence. But if you think about it in those terms, why use threads at all? You could just call your three functions in a loop, since you never have more than one of 'em running at a time anyway...

Out of curiosity, though: what are you doing that requires the multiple threads to compete so viciously? Is there a way, for instance, that each thread could wait until it actually has something to put on the queue before it attempts to lock the semaphore? That might yield better performance.

[striker07]

Great explanation, tetsujin! I'm working in a embedded system programming. This means low power of processor, and low memory. In this project (that I can't tell details, due contracts restrictions), I need to receive data from an ethernet connection (a ppp modem connection), via socket, process the data, and send them to a serial port. At the same time, I need to read the serial port, process the data, and send them to the ethernet connection. The CPU is a ARM9, with low clock.

To solve the question about the interchange of data inside the application, I thinked in three threads and three queues, since I discovered (in another topic here), that I can't share a fd between functions. So, one thread will send and receive data from the socket connection. It will put the received data in a queue of data waiting to be processed, and send the data contained in another queue, with processed data. The second thread, listen the serial port. It takes data from the third queue, that contais data ready to be written in the serial port, and reads data from the serial, adding a header, and putting it in the queue of processed data. The third and last thread will process the data of the queue of "waiting unprocessed data", will process them, and put it or in the queue of processed data, or in the queue of serial data ready to be writed.

I've implemented this "architecture", but I've found the problem described above...

[Rubberman]

I think Tetsujin's solution is a reasonable one for an embedded system. You want to avoid sleeps and polling as much as possible and this will happily "thread" your access to the critical elements in a balanced manner. I also do embedded programming on arm boards so I know the limitations you are facing.