Producer-consumer based multi-threading for image processing - c++

Question

UPDATE: I have provided the reason of problem and its solution in my answer below.
I want to implement multi-threading which is based upon Producer-consumer approach for an image processing task. For my case, the Producer thread should grabs the images and put them into a container whereas the consumer thread should extract the images from the Container thread. I think that I should use queue for the implementation of container.
I want to use the following code as suggested in this SO answer. But I have become quite confused with the implementation of container and putting the incoming image into it in the Producer thread.
PROBLEM: The image displayed by the first consumer thread does not contain the full data. And, the second consumer thread never displays any image. May be, there is some race situation or lock situation due to which the second thread is not able to access the data of queue at all. I have already tried to use Mutex.
#include <vector>
#include <thread>
#include <memory>
#include <queue>
#include <opencv2/highgui.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
Mutex mu;
struct ThreadSafeContainer
{
queue<unsigned char*> safeContainer;
};
struct Producer
{
Producer(std::shared_ptr<ThreadSafeContainer> c) : container(c)
{
}
void run()
{
while(true)
{
// grab image from camera
// store image in container
Mat image(400, 400, CV_8UC3, Scalar(10, 100,180) );
unsigned char *pt_src = image.data;
mu.lock();
container->safeContainer.push(pt_src);
mu.unlock();
}
}
std::shared_ptr<ThreadSafeContainer> container;
};
struct Consumer
{
Consumer(std::shared_ptr<ThreadSafeContainer> c) : container(c)
{
}
~Consumer()
{
}
void run()
{
while(true)
{
// read next image from container
mu.lock();
if (!container->safeContainer.empty())
{
unsigned char *ptr_consumer_Image;
ptr_consumer_Image = container->safeContainer.front(); //The front of the queue contain the pointer to the image data
container->safeContainer.pop();
Mat image(400, 400, CV_8UC3);
image.data = ptr_consumer_Image;
imshow("consumer image", image);
waitKey(33);
}
mu.unlock();
}
}
std::shared_ptr<ThreadSafeContainer> container;
};
int main()
{
//Pointer object to the class containing a "container" which will help "Producer" and "Consumer" to put and take images
auto ptrObject_container = make_shared<ThreadSafeContainer>();
//Pointer object to the Producer...intialize the "container" variable of "Struct Producer" with the above created common "container"
auto ptrObject_producer = make_shared<Producer>(ptrObject_container);
//FIRST Pointer object to the Consumer...intialize the "container" variable of "Struct Consumer" with the above created common "container"
auto first_ptrObject_consumer = make_shared<Consumer>(ptrObject_container);
//SECOND Pointer object to the Consumer...intialize the "container" variable of "Struct Consumer" with the above created common "container"
auto second_ptrObject_consumer = make_shared<Consumer>(ptrObject_container);
//RUN producer thread
thread producerThread(&Producer::run, ptrObject_producer);
//RUN first thread of Consumer
thread first_consumerThread(&Consumer::run, first_ptrObject_consumer);
//RUN second thread of Consumer
thread second_consumerThread(&Consumer::run, second_ptrObject_consumer);
//JOIN all threads
producerThread.join();
first_consumerThread.join();
second_consumerThread.join();
return 0;
}

Answer 1

I don't see an actual question in your original question, so I'll give you the reference material I used to implement producer-consumer in my college course.
http://cs360.byu.edu/static/lectures/winter-2014/semaphores.pdf
Slides 13 and 17 give good examples of producer-consumer
I made use of this in the lab which I have posted on my github here:
https://github.com/qzcx/Internet_Programming/tree/master/ThreadedMessageServer
If you look in my server.cc you can see my implementation of the producer-consumer pattern.
Remember that using this pattern that you can't switch the order of the wait statements or else you can end up in deadlock.
Hope this is helpful.
EDIT:
Okay, so here is a summary of the consumer-producer pattern in my code linked above. The idea behind the producer consumer is to have a thread safe way of passing tasks from a "producer" thread to "consumer" worker threads. In the case of my example, the work to be done is to handle client requests. The producer thread (.serve()) monitors the incoming socket and passes the connection to consumer threads (.handle()) to handle the actual request as they come in. All of the code for this pattern is found in the server.cc file (with some declarations/imports in server.h).
For the sake of being brief, I am leaving out some detail. Be sure to go through each line and understand what is going on. Look up the library functions I am using and what the parameters mean. I'm giving you a lot of help here, but there is still plenty of work for you to do to gain a full understanding.
PRODUCER:
Like I mentioned above, the entire producer thread is found in the .serve() function. It does the following things
Initializes the semaphores. There are two version here because of OS differences. I programmed on a OS X, but had to turn in code on Linux. Since Semaphores are tied to the OS, it is important to understand how to use semaphores in your particular setup.
It sets up the socket for the client to talk to. Not important for your application.
Creates the consumer threads.
Watches the client socket and uses the producer pattern to pass items to the consumers. This code is below
At the bottom of the .serve() function you can see the following code:
while ((client = accept(server_,(struct sockaddr *)&client_addr,&clientlen)) > 0) {
sem_wait(clients_.e); //buffer check
sem_wait(clients_.s);
clients_.q->push(client);
sem_post(clients_.s);
sem_post(clients_.n); //produce
}
First, you check the buffer semaphore "e" to ensure there is room in your queue to place the request. Second, acquire the semaphore "s" for the queue. Then add your task (In this case, a client connection) to the queue. Release the semaphore for the queue. Finally, signal to the consumers using semaphore "n".
Consumer:
In the .handle() method you really only care about the very beginning of the thread.
while(1){
sem_wait(clients_.n); //consume
sem_wait(clients_.s);
client = clients_.q->front();
clients_.q->pop();
sem_post(clients_.s);
sem_post(clients_.e); //buffer free
//Handles the client requests until they disconnect.
}
The consumer does similar actions to the producer, but in opposite fashion. First the consumer waits for the producer to signal on the semaphore "n". Remember since there are multiple consumers it is completely random which consumer might end up acquiring this semaphore. They fight over it, but only one can move passed this point per sem_post of that semaphore. Second, they acquire the queue semaphore like the producer does. Pop the first item off the queue and release the semaphore. Finally, they signal on the buffer semaphore "e" that there is now more room in the buffer.
Disclaimer:
I know the semaphores have terrible names. They match my professor's slides since that's where I learned it. I think they stand for the following:
e for empty : this semaphore stops the producer from pushing more items on the queue if it is full.
s for semaphore : My least favorite. But my professor's style was to have a struct for each shared data struct. In this case "clients_" is the struct including all three semaphores and the queue. Basically this semaphore is there to ensure no two threads touch the same data structure at the same time.
n for number of items in the queue.

Answer 2

Ok, so to make it as simple as possible. You will need 2 threads, mutex, queue and 2 thread processing functions.
Header.h
static DWORD WINAPI ThreadFunc_Prod(LPVOID lpParam);
static DWORD WINAPI ThreadFunc_Con(LPVOID lpParam);
HANDLE m_hThread[2];
queue<int> m_Q;
mutex m_M;
Add all needed stuff, these are just core parts you need
Source.cpp
DWORD dwThreadId;
m_hThread[0] = CreateThread(NULL, 0, this->ThreadFunc_Prod, this, 0, &dwThreadId);
// same for 2nd thread
DWORD WINAPI Server::ThreadFunc_Prod(LPVOID lpParam)
{
cYourClass* o = (cYourClass*) lpParam;
int nData2Q = GetData(); // this is whatever you use to get your data
m_M.lock();
m_Q.push(nData2Q);
m_M.unlock();
}
DWORD WINAPI Server::ThreadFunc_Con(LPVOID lpParam)
{
cYourClass* o = (cYourClass*) lpParam;
int res;
m_M.lock();
if (m_Q.empty())
{
// bad, no data, escape or wait or whatever, don't block context
}
else
{
res = m_Q.front();
m_Q.pop();
}
m_M.unlock();
// do you magic with res here
}
And in the end of main - don't forget to use WaitForMultipleObjects
All possible examples can be found directly in MSDN so there is quite nice commentary about that.
PART2:
ok, so I believe header is self-explainable, so I will give you little bit more description to source. Somewhere in your source (can be even in Constructor) you create threads - the way how to create thread may differ but idea is the same (in win - thread is run right after its creation in posix u have to join). I believe u shall have somewhere a function which starts all your magic, lets call it MagicKicker()
In case of posix, create thread in constructor and join em in your MagicKicker(), win - create in MagicKicker()
Than you would need to declare (in header) two function where you thread function will be implemented ThreadFunc_Prod and ThreadFunc_Prod , important magic here is that you will pass reference to your object to this function (coz thread are basically static) so u can easy access shared resources as queues, mutexes, etc...
These function are actually doing the work. You actually have all u need in you code, just use this as adding routine in Producer:
int nData2Q = GetData(); // this is whatever you use to get your data
m_M.lock(); // locks mutex so nobody cant enter mutex
m_Q.push(nData2Q); // puts data from producer to share queue
m_M.unlock(); // unlock mutex so u can access mutex in your consumer
And add this to your consumer:
int res;
m_M.lock(); // locks mutex so u cant access anything wrapped by mutex in producer
if (m_Q.empty()) // check if there is something in queue
{
// nothing in you queue yet OR already
// skip this thread run, you can i.e. sleep for some time to build queue
Sleep(100);
continue; // in case of while wrap
return; // in case that u r running some framework with threadloop
}
else // there is actually something
{
res = m_Q.front(); // get oldest element of queue
m_Q.pop(); // delete this element from queue
}
m_M.unlock(); // unlock mutex so producer can add new items to queue
// do you magic with res here

Answer 3

The problem mentioned in my question was that the image displayed by the Consumer thread was not containing complete data. The image displayed by the Consumer thread contains several patches which suggest that it could not get the full data produced by Producer thread.
ANSWER The reason behind it is the declaration of Mat image inside the while loop of Consumer thread. The Mat instance created inside the while loop gets deleted once the second round of while loop starts and therefore the Producer thread was never able to access the data of Mat image created in the Consumer thread.
SOLUTION: I should have done it something like this
struct ThreadSafeContainer
{
queue<Mat> safeContainer;
};
struct Producer
{
Producer(std::shared_ptr<ThreadSafeContainer> c) : container(c)
{
}
void run()
{
while(true)
{
// grab image from camera
// store image in container
Mat image(400, 400, CV_8UC3, Scalar(10, 100,180) );
mu.lock();
container->safeContainer.push(Mat);
mu.unlock();
}
}
std::shared_ptr<ThreadSafeContainer> container;
};
struct Consumer
{
Consumer(std::shared_ptr<ThreadSafeContainer> c) : container(c)
{
}
~Consumer()
{
}
void run()
{
while(true)
{
// read next image from container
mu.lock();
if (!container->safeContainer.empty())
{
Mat image= container->safeContainer.front(); //The front of the queue contain the image
container->safeContainer.pop();
imshow("consumer image", image);
waitKey(33);
}
mu.unlock();
}
}
std::shared_ptr<ThreadSafeContainer> container;
};