Unable To Open Two Cameras Using OpenCV - Multithreading Camera Reads - c++

Question

I am trying to use OpenCV to stream video from 2 cameras continuously via separate threads. The following code is displaying Segmentation fault (core dumped)
What is the reason for this and How do I fix this issue?
main.cpp
#include <iostream>
#include <pthread.h>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
using namespace std;
struct thread_data {
string path;
int thread_id;
};
void *capture(void *threadarg)
{
struct thread_data *data;
data = (struct thread_data *) threadarg;
cv::VideoCapture cap(data->path);
if( !cap.isOpened())
{
std::cout<<"Not good, open camera failed"<<std::endl;
}
std::cout<< "Opened IP camera successfully!"<<std::endl;
cv::Mat frame;
string ext = ".jpg";
string result;
while (true) {
cap >> frame;
cv::imshow("Frame",frame);
cv::waitKey(1);
}
pthread_exit(NULL);
}
int main(void) {
pthread_t threads[2];
struct thread_data td[2];
int rc=0;
for( int i = 0; i < 2; i++ ) {
cout <<"main() : creating thread, " << i << endl;
td[i].thread_id = i;
td[0].path = "rtsp://admin:opencv123#192.168.1.23:554/Streaming/Channels/101/";
td[1].path = "rtsp://admin:opencv123#192.168.1.24:554/Streaming/Channels/101/";
rc = pthread_create(&threads[i], NULL, capture, (void *)&td[i]);
if (rc) {
cout << "Error:unable to create thread," << rc << endl;
exit(-1);
}
}
pthread_exit(NULL);
return 0;
}
log:
main() : creating thread, 0 main() : creating thread, 1
Segmentation fault (core dumped)
When I tried running it multiple times, I am able to open only one camera and that too isn't streaming continuously. It starts and stops in a few seconds.
Sometimes I get an error which says
OpenCV Error: Insufficient memory (Failed to allocate 140703464366800 bytes) in OutOfMemoryError
I have gone through various Q&As' on StackOverflow but none helped.

Answer 1

The problem here is that the code is facing race conditions. I was able to replicate the issue on my system and identified the following issues:
OpenCV window titles are not unique.
Spawned threads are not joined.
Race condition while opening the video stream.
Lets look into these issues in detail.
1.
OpenCV windows are uniquely identified by their title. In the current code, the title is a hardcoded string "Frame". So basically, both threads are creating/updating/destroying the same window in unknown order. That is a race condition which can be fixed by adding a string field to the struct thread_data which will serve as unique window identifier.
2.
In the main thread, the child threads are created asynchronously so the for loop will exit immediately after creating the threads and the program will exit pre-maturely without waiting for the spawned thread to complete execution. This problem can be solved by adding function calls to wait for the threads before the program exits. This process is called joining and can be achieved by calling pthread_join for each spawned thread.
3.
This issue was a bit trickier to track down. Due to some reason the backend library for video stream capture used by OpenCV is not behaving in a thread-safe manner. Seemingly, the video capture opening process is prone to race-condition and requires a synchronization lock. The lock can be easily implemented by calling the functions pthread_mutex_lock and pthread_mutex_unlock just before and after opening the VideoCapture object.
Here is the modified code demonstrating the solution for all of the above mentioned issues
#include <iostream>
#include <pthread.h>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
using namespace std;
//Mutex for thread synchronization
static pthread_mutex_t foo_mutex = PTHREAD_MUTEX_INITIALIZER;
struct thread_data
{
string path;
int thread_id;
string window_title; //Unique window title for each thread
};
void *capture(void *threadarg)
{
struct thread_data *data;
data = (struct thread_data *) threadarg;
cv::VideoCapture cap;
//Safely open video stream
pthread_mutex_lock(&foo_mutex);
cap.open(data->path);
pthread_mutex_unlock(&foo_mutex);
if( !cap.isOpened())
{
std::cout<<"Not good, open camera failed"<<std::endl;
}
std::cout<< "Opened IP camera successfully!"<<std::endl;
cv::Mat frame;
string ext = ".jpg";
string result;
//Create window with unique title
cv::namedWindow(data->window_title);
while (true)
{
cap >> frame;
cv::imshow(data->window_title,frame);
cv::waitKey(10);
}
//Release VideoCapture object
cap.release();
//Destroy previously created window
cv::destroyWindow(data->window_title);
//Exit thread
pthread_exit(NULL);
}
int main(void)
{
const int thread_count = 2;
pthread_t threads[thread_count];
struct thread_data td[thread_count];
//Initialize thread data beforehand
td[0].path = "rtsp://admin:opencv123#192.168.1.23:554/Streaming/Channels/101/";
td[0].window_title = "First Window";
td[1].path = "rtsp://admin:opencv123#192.168.1.24:554/Streaming/Channels/101/";
td[1].window_title = "Second Window";
int rc=0;
for( int i = 0; i < thread_count; i++ )
{
cout <<"main() : creating thread, " << i << endl;
td[i].thread_id = i;
rc = pthread_create(&(threads[i]), NULL, capture, (void *)& (td[i]) );
if (rc)
{
cout << "Error:unable to create thread," << rc << endl;
exit(-1);
}
}
//Wait for the previously spawned threads to complete execution
for( int i = 0; i < thread_count; i++ )
pthread_join(threads[i], NULL);
pthread_exit(NULL);
return 0;
}

Answer 2

void writeFrametoDisk(const cv::Mat *frame, std::string path, int frameNum, std::string windowName)
{
cv::imwrite(path.append(std::to_string(frameNum)).append(".png"), *frame);
return;
}
void openCameraStream(int deviceId, std::string dirName)
{
cv::VideoCapture cap;
cap.open(deviceId);
if(!cap.isOpened()){std::cerr << "Unable to open the camera " << std::endl;}
std::cout << cap.get(cv::CAP_PROP_FRAME_WIDTH) << " " << cap.get(cv::CAP_PROP_FRAME_HEIGHT) << std::endl;
std::string windowName = deviceId == 0 ? "Cam 0" : "Cam 1";
std::string outputDir = dirName;
bool frameStFg = false;
while(!frameStFg)
{
cv::Mat frame;
cap.read(frame);
if(frame.empty()){std::cerr << "frame buffer empty " << std::endl;}
else
{
frameStFg = true;
}
}
cv::TickMeter timer;
timer.start();
int frameCount = 0;
while(1)
{
cv::Mat frame;
cap.read(frame);
if(frame.empty()){std::cerr << "frame buffer empty " << std::endl;}
frameCount++;
std::thread th(writeFrametoDisk, &frame, outputDir, frameCount, windowName);
th.join();
//// a simple wayto exit the loop
if(frameCount > 500)
{
break;
}
}
timer.stop();
std::cout << "Device id " << deviceId << " Capture ran for " << timer.getTimeSec() << " seconds" << std::endl;
std::cout << "Device id " << deviceId << " Number of frames to be capture should be " << timer.getTimeSec() * 30 << std::endl;
std::cout << "Device id " << deviceId << " Number of frames captured " << frameCount << std::endl;
cap.release();
}
int main(int argc, char * argv[])
{
std::string outputDir1 = "";
std::string outputDir2 = "";
std::thread camera1Thread(openCameraStream, 0, outputDir1);
std::thread camera2Thread(openCameraStream, 1, outputDir2);
camera1Thread.join();
camera2Thread.join();
}
As the comments mention, streaming without any imshow() of the image streams works well with out any problems. My setup includes, running two threads with two USB camera's and also these two threads launch a new thread each to save the image read from the cameras. I didn't observe any frame drops or errors and also was able to capture and write at approximately 30 fps.
A little debugging of the reason, it is recommended to use any imshow() in the main thread only i.e. main() function. Hope this helps anyone.