Multithreading with opencv and pthreads - c++

I wanted to get video frames from openCV and render it onto a graphics screen with openGL. I got it to work; however, the video drops frames consistently. I believe its because reading the frame and rendering is all on the same thread. Therefore, I implemented multithreading with pthreads but imshow is just a white screen. Also, it seems like the main thread does not run. I implemented a version with conditions and signals, and one with it. Any pointers to what I am doing wrong will be very helpful. Below is my code:
void *updateloop(void *arg)
{
while(1)
{
pthread_mutex_lock(&frameLocker);
while(my_queue.empty())
{
pthread_cond_wait(&condc, &frameLocker);
}
std::cout << "entering update loop" << std::endl;
current_image = my_queue.front();
my_queue.pop();
/*
if(!current_image.empty())
{
std::cout << "rendering to output window" << std::endl;
cv::imshow("Output Window", current_image);
cv::waitKey(30);
}
*/
pthread_mutex_unlock(&frameLocker);
}
}
void *captureloop(void *arg)
{
cv::Mat tempFrame;
while(1)
{
std::cout << "entering capture loop" << std::endl;
cap.read(tempFrame);
pthread_mutex_lock(&frameLocker);
my_queue.push(tempFrame);
pthread_cond_signal(&condc);
pthread_mutex_unlock(&frameLocker);
}
}
int main(int argc, const char * argv[])
{
cap.open(0);
pthread_mutex_init(&frameLocker, NULL);
pthread_t update, webcam;
pthread_cond_init(&condc, NULL);
pthread_cond_init(&condp, NULL);
pthread_create(&update, NULL, updateloop, NULL);
pthread_create(&webcam, NULL, captureloop, NULL);
pthread_join(update, NULL);
pthread_join(webcam, NULL);
cv::Mat tempFrame;
while (1)
{
pthread_mutex_lock(&frameLocker);
std::cout << "in the main thread" << std::endl;
if(current_image.empty()){
printf("recieved empty frame\n");
continue;
}
if(!current_image.empty())
{
std::cout << "rendering to output window" << std::endl;
cv::imshow("Output Window", current_image);
}
cv::waitKey(1);
pthread_mutex_unlock(&frameLocker);
}
return 0;
}

Related

Unable To Open Two Cameras Using OpenCV - Multithreading Camera Reads

I am trying to use OpenCV to stream video from 2 cameras continuously via separate threads. The following code is displaying Segmentation fault (core dumped)
What is the reason for this and How do I fix this issue?
main.cpp
#include <iostream>
#include <pthread.h>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
using namespace std;
struct thread_data {
string path;
int thread_id;
};
void *capture(void *threadarg)
{
struct thread_data *data;
data = (struct thread_data *) threadarg;
cv::VideoCapture cap(data->path);
if( !cap.isOpened())
{
std::cout<<"Not good, open camera failed"<<std::endl;
}
std::cout<< "Opened IP camera successfully!"<<std::endl;
cv::Mat frame;
string ext = ".jpg";
string result;
while (true) {
cap >> frame;
cv::imshow("Frame",frame);
cv::waitKey(1);
}
pthread_exit(NULL);
}
int main(void) {
pthread_t threads[2];
struct thread_data td[2];
int rc=0;
for( int i = 0; i < 2; i++ ) {
cout <<"main() : creating thread, " << i << endl;
td[i].thread_id = i;
td[0].path = "rtsp://admin:opencv123#192.168.1.23:554/Streaming/Channels/101/";
td[1].path = "rtsp://admin:opencv123#192.168.1.24:554/Streaming/Channels/101/";
rc = pthread_create(&threads[i], NULL, capture, (void *)&td[i]);
if (rc) {
cout << "Error:unable to create thread," << rc << endl;
exit(-1);
}
}
pthread_exit(NULL);
return 0;
}
log:
main() : creating thread, 0 main() : creating thread, 1
Segmentation fault (core dumped)
When I tried running it multiple times, I am able to open only one camera and that too isn't streaming continuously. It starts and stops in a few seconds.
Sometimes I get an error which says
OpenCV Error: Insufficient memory (Failed to allocate 140703464366800 bytes) in OutOfMemoryError
I have gone through various Q&As' on StackOverflow but none helped.
The problem here is that the code is facing race conditions. I was able to replicate the issue on my system and identified the following issues:
OpenCV window titles are not unique.
Spawned threads are not joined.
Race condition while opening the video stream.
Lets look into these issues in detail.
1.
OpenCV windows are uniquely identified by their title. In the current code, the title is a hardcoded string "Frame". So basically, both threads are creating/updating/destroying the same window in unknown order. That is a race condition which can be fixed by adding a string field to the struct thread_data which will serve as unique window identifier.
2.
In the main thread, the child threads are created asynchronously so the for loop will exit immediately after creating the threads and the program will exit pre-maturely without waiting for the spawned thread to complete execution. This problem can be solved by adding function calls to wait for the threads before the program exits. This process is called joining and can be achieved by calling pthread_join for each spawned thread.
3.
This issue was a bit trickier to track down. Due to some reason the backend library for video stream capture used by OpenCV is not behaving in a thread-safe manner. Seemingly, the video capture opening process is prone to race-condition and requires a synchronization lock. The lock can be easily implemented by calling the functions pthread_mutex_lock and pthread_mutex_unlock just before and after opening the VideoCapture object.
Here is the modified code demonstrating the solution for all of the above mentioned issues
#include <iostream>
#include <pthread.h>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
using namespace std;
//Mutex for thread synchronization
static pthread_mutex_t foo_mutex = PTHREAD_MUTEX_INITIALIZER;
struct thread_data
{
string path;
int thread_id;
string window_title; //Unique window title for each thread
};
void *capture(void *threadarg)
{
struct thread_data *data;
data = (struct thread_data *) threadarg;
cv::VideoCapture cap;
//Safely open video stream
pthread_mutex_lock(&foo_mutex);
cap.open(data->path);
pthread_mutex_unlock(&foo_mutex);
if( !cap.isOpened())
{
std::cout<<"Not good, open camera failed"<<std::endl;
}
std::cout<< "Opened IP camera successfully!"<<std::endl;
cv::Mat frame;
string ext = ".jpg";
string result;
//Create window with unique title
cv::namedWindow(data->window_title);
while (true)
{
cap >> frame;
cv::imshow(data->window_title,frame);
cv::waitKey(10);
}
//Release VideoCapture object
cap.release();
//Destroy previously created window
cv::destroyWindow(data->window_title);
//Exit thread
pthread_exit(NULL);
}
int main(void)
{
const int thread_count = 2;
pthread_t threads[thread_count];
struct thread_data td[thread_count];
//Initialize thread data beforehand
td[0].path = "rtsp://admin:opencv123#192.168.1.23:554/Streaming/Channels/101/";
td[0].window_title = "First Window";
td[1].path = "rtsp://admin:opencv123#192.168.1.24:554/Streaming/Channels/101/";
td[1].window_title = "Second Window";
int rc=0;
for( int i = 0; i < thread_count; i++ )
{
cout <<"main() : creating thread, " << i << endl;
td[i].thread_id = i;
rc = pthread_create(&(threads[i]), NULL, capture, (void *)& (td[i]) );
if (rc)
{
cout << "Error:unable to create thread," << rc << endl;
exit(-1);
}
}
//Wait for the previously spawned threads to complete execution
for( int i = 0; i < thread_count; i++ )
pthread_join(threads[i], NULL);
pthread_exit(NULL);
return 0;
}
void writeFrametoDisk(const cv::Mat *frame, std::string path, int frameNum, std::string windowName)
{
cv::imwrite(path.append(std::to_string(frameNum)).append(".png"), *frame);
return;
}
void openCameraStream(int deviceId, std::string dirName)
{
cv::VideoCapture cap;
cap.open(deviceId);
if(!cap.isOpened()){std::cerr << "Unable to open the camera " << std::endl;}
std::cout << cap.get(cv::CAP_PROP_FRAME_WIDTH) << " " << cap.get(cv::CAP_PROP_FRAME_HEIGHT) << std::endl;
std::string windowName = deviceId == 0 ? "Cam 0" : "Cam 1";
std::string outputDir = dirName;
bool frameStFg = false;
while(!frameStFg)
{
cv::Mat frame;
cap.read(frame);
if(frame.empty()){std::cerr << "frame buffer empty " << std::endl;}
else
{
frameStFg = true;
}
}
cv::TickMeter timer;
timer.start();
int frameCount = 0;
while(1)
{
cv::Mat frame;
cap.read(frame);
if(frame.empty()){std::cerr << "frame buffer empty " << std::endl;}
frameCount++;
std::thread th(writeFrametoDisk, &frame, outputDir, frameCount, windowName);
th.join();
//// a simple wayto exit the loop
if(frameCount > 500)
{
break;
}
}
timer.stop();
std::cout << "Device id " << deviceId << " Capture ran for " << timer.getTimeSec() << " seconds" << std::endl;
std::cout << "Device id " << deviceId << " Number of frames to be capture should be " << timer.getTimeSec() * 30 << std::endl;
std::cout << "Device id " << deviceId << " Number of frames captured " << frameCount << std::endl;
cap.release();
}
int main(int argc, char * argv[])
{
std::string outputDir1 = "";
std::string outputDir2 = "";
std::thread camera1Thread(openCameraStream, 0, outputDir1);
std::thread camera2Thread(openCameraStream, 1, outputDir2);
camera1Thread.join();
camera2Thread.join();
}
As the comments mention, streaming without any imshow() of the image streams works well with out any problems. My setup includes, running two threads with two USB camera's and also these two threads launch a new thread each to save the image read from the cameras. I didn't observe any frame drops or errors and also was able to capture and write at approximately 30 fps.
A little debugging of the reason, it is recommended to use any imshow() in the main thread only i.e. main() function. Hope this helps anyone.

opencv stereo camera error

i working in a stereo camera project i have two cameras 5megapixels in every one i connected it with my laptop and run my code but when i run it i get this error libv4l2: error turning on stream: No space left on device
im linux os that's my c++ opencv code there are any ideas how to fix it i tried others codes i found it in network but still give me the same error
#include <opencv2/opencv.hpp>
int main()
{
cv::VideoCapture cap1(1);
cv::VideoCapture cap2(2);
if(!cap1.isOpened())
{
std::cout << "Cannot open the video cam [1]" << std::endl;
return -1;
}
if(!cap2.isOpened())
{
std::cout << "Cannot open the video cam [2]" << std::endl;
return -1;
}
cap1.set(CV_CAP_PROP_FPS, 15);
cap2.set(CV_CAP_PROP_FPS, 15);
// Values taken from output of Version 1 and used to setup the exact same parameters with the exact same values!
cap1.set(CV_CAP_PROP_FRAME_WIDTH, 640);
cap1.set(CV_CAP_PROP_FRAME_HEIGHT, 480);
cap2.set(CV_CAP_PROP_FRAME_WIDTH, 640);
cap2.set(CV_CAP_PROP_FRAME_HEIGHT, 480);
cv::namedWindow("cam[1]",CV_WINDOW_AUTOSIZE);
cv::namedWindow("cam[2]",CV_WINDOW_AUTOSIZE);
while(1)
{
cv::Mat frame1, frame2;
bool bSuccess1 = cap1.read(frame1);
bool bSuccess2 = cap2.read(frame2);
if (!bSuccess1)
{
std::cout << "Cannot read a frame from video stream [1]" << std::endl;
break;
}
if (!bSuccess2)
{
std::cout << "Cannot read a frame from video stream [2]" << std::endl;
break;
}
cv::imshow("cam[1]", frame1);
cv::imshow("cam[2]", frame2);
if(cv::waitKey(30) == 27)
{
std::cout << "ESC key is pressed by user" << std::endl;
break;
}
}
return 0;
}

C++ Pthread/SFML Music - Error AL lib: (EE) alc_cleanup: 1 device not closed

I have a project that is to recode the Bomberman and I implemented the sf :: Music for music added to main menu but I come across an error:
AL lib: (EE) alc_cleanup: 1 device not closed
And it's relied to this :
void *pthreadSound(void *);
int main(int, char **){
GameEngine *game = new GameEngine();
pthread_attr_t attr;
pthread_t sound;
void *retval;
try {
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
if (!game->initialize())
throw ErrorBomberman("Game fail initialisation", "main");
game->m_mutex = PTHREAD_MUTEX_INITIALIZER;
game->setLock(true);
pthread_create(&sound, &attr, pthreadSound, (void *)game);
game->ChangeState(IntroState::Instance());
while (game->Running()){
game->HandleEvents();
game->update();
game->draw();
}
game->Cleanup();
pthread_join(sound, &retval);
} catch (ErrorBomberman const &e) {
std::cerr << "Error caught in " << e.getComponent() << ": " << e.what() << std::endl;
}
return (EXIT_SUCCESS);
}
void *pthreadSound(void *game){
sf::Music music1;
if (!music1.OpenFromFile("./ressources/Bomberman_Theme.ogg"))
std::cout << "Erreur : load" << std::endl;
music1.Play();
while (((GameEngine *)game)->getLock()){
}
std::cout << "Stop" << std::endl;
music1.Stop();
pthread_exit(NULL);
return (NULL);
}
Maybe, it's because the Stop doesn't work ? I don't know. If you have an idea, thank you.
According to the docs:
As a sound stream, a music is played in its own thread in order not to block the rest of the program. This means that you can leave the music alone after calling play(), it will manage itself very well.
So you don't really need to create another thread to play the music.

MP3 header missing from IP camera RTSP

I just want to show the IP camera video without sound, but this camera plays video with sound, and I get [mp3 # 0x107808800] Header missing in the console. How do I play without sound? I'm using vstarcam-30s. Here is my code:
int main (int argc, char *argv[])
{
VideoCapture vcap;
Mat image;
string videoStreamAddress =
"rtsp://[IPADDRESS]/profile2/media.smp";
if (!vcap.open(videoStreamAddress)) {
cout << "Error opening video stream or file" << endl;
return -1;
}
for(;;) {
if(!vcap.read(image)) {
cout << "No frame" << endl;
waitKey();
}
imshow("Output Window", image);
if(waitKey(1) >= 0) break;
}
return 0;
}

Using OpenCV, Boost threading and multiple cameras

I am trying to write a program that is able to capture images from two different cameras in two different threads. I want to do this because when I do this in the same thread I have to keep waiting for cvQueryFrame twice the amount of time and so i can not grab images at 30 fps (I get 15 FPS from each camera).
I have taken a look at this SO post, but this only works for one camera. Using cvQueryFrame and boost::thread together
My current program gives varying results, sometimes it gives memory leaks, usually I just don't see anything happening and sometimes it worls for a few seconds but the the image freezes again. The strange thing is that earlier when I didn't call cvShowImage, but had my imageProcessing function do something useful I could see that I was getting real time results from both cameras. I assume this means that it is possible to make this work, but that I made a stupid mistake somewhere. My OS is LINUX and I am using OpenCV 2.4
My code:
#include <iostream>
#include <cstdio>
#include <cv.h>
#include <ml.h>
#include <cvaux.h>
#include <highgui.h>
#include <vector>
#include <stdio.h>
#include "producer_consumer_queue.hpp"
//Camera settings
int cameraWidth = 1280;
int cameraHeight = 720;
int waitKeyValue = 5;
bool threads_should_exit = false;
CvCapture * capture;
CvCapture * capture2;
using namespace std;
using namespace cv;
void grabFrame(concurrent_queue<IplImage* > * frame_queue, int camNumber) {
try {
//Load first frames
cout << "grabFrame: " << camNumber << " init with " << cameraWidth << " x " << cameraHeight << endl;
IplImage* frame;
if (camNumber == 0)frame = cvQueryFrame(capture);
if (camNumber == 1)frame = cvQueryFrame(capture2);
while (frame && !threads_should_exit) {
if (camNumber == 0)frame = cvQueryFrame(capture);
if (camNumber == 1)frame = cvQueryFrame(capture2);
IplImage* frame_copy = NULL;
frame_copy = cvCloneImage(frame);
if (camNumber == 0)cvShowImage("NE", frame);
cout << "grabFrame: " << camNumber << " pushing back to queue" << endl;
frame_queue->push(frame_copy);
int k = cvWaitKey(waitKeyValue);
if (k == 1048603 || k == 27 || k == '\r') {
cout << "grabFrame: Process killed" << endl;
//release memory
threads_should_exit = true;
}
}
} catch (const concurrent_queue<IplImage* >::Canceled & e) {
cout << "grabFrame: Show thread is canceled" << endl;
return;
}
}
void processFrames(concurrent_queue<IplImage* > * frame_queue0, concurrent_queue<IplImage* > * frame_queue1) {
try {
do {
cout << "processFrames: Processing two frames" << endl;
IplImage* frm = NULL;
frame_queue0->wait_and_pop(frm);
IplImage * frm2 = NULL;
frame_queue1->wait_and_pop(frm2);
cvReleaseImage(&frm);
cvReleaseImage(&frm2);
} while (!threads_should_exit);
} catch (const concurrent_queue<IplImage* >::Canceled & e) {
cout << "processFrames: Processing thread is canceled" << endl;
return;
}
}
int main() {
capture = cvCreateCameraCapture(0);
capture2 = cvCreateCameraCapture(1);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, cameraWidth);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, cameraHeight);
cvSetCaptureProperty(capture2, CV_CAP_PROP_FRAME_WIDTH, cameraWidth);
cvSetCaptureProperty(capture2, CV_CAP_PROP_FRAME_HEIGHT, cameraHeight);
boost::thread_group frame_workers;
boost::thread_group frame_workers2;
concurrent_queue<IplImage* > frame_queue(&frame_workers);
concurrent_queue<IplImage* > frame_queue2(&frame_workers2);
boost::thread * query_thread = new boost::thread(processFrames, &frame_queue, &frame_queue2);
boost::thread * cam0_thread = new boost::thread(grabFrame, &frame_queue, 0);
usleep(10000);
boost::thread * cam1_thread = new boost::thread(grabFrame, &frame_queue2, 1);
frame_workers.add_thread(query_thread);
frame_workers.add_thread(cam0_thread);
frame_workers2.add_thread(query_thread);
frame_workers2.add_thread(cam1_thread);
while (true) {
if (threads_should_exit) {
cout << "Main: threads should be killed" << endl;
while (!frame_queue.empty()) {
usleep(10000);
}
frame_workers.remove_thread(query_thread);
frame_workers2.remove_thread(query_thread);
frame_workers.remove_thread(cam0_thread);
frame_workers2.remove_thread(cam1_thread);
frame_workers.join_all();
break;
}
usleep(10000);
}
return 0;
}
EDIT:
I added a simple function to detect a piece of paper to see if everything is working fine when I don't call on cvShowImage(). My program can detect a piece of paper fine if I don't call cvShowImage(). If I do the program has strange behavior again and freezes etc.
There should only be one thread manipulating the GUI (this is true for just about any GUI framework). You should organize your code so that cvShowImage is only invoked from the main "GUI thread".
It seems that the work that's being done in query_thread could just as easily be done inside the main thread.