Visual Studio 2010 OpenCV Multithread Mat does not copy to another Mat - c++

Good afternoon to all!
I've been using Visual Studio 2010 with OpenCV to develop a code for Face recognition. I'm trying to reach the task by two Threads (I need to do it this way, because i´m going to apply it on a bigger project), one (the main) to show the frames and the second to capture (from the Webcam of my laptop) and store the frames on a Mat object (Fast Capture).
The inconvenient here is that the second Thread is capturing the frames, but the main is not showing them. I think there is a problem with copying the Mat from the capture Thread to the Mat on the main thread ("current_frame" seems to be empty after I do the assignation)
Here is the code (I'm using Boost::Thread for Multithreading)
New code with suggestions
Global declarations
#include <iostream>
#include <stdio.h>
#include <boost\thread.hpp>
#include <opencv2/objdetect/objdetect.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
using namespace std;
using namespace cv;
boost::mutex mtx;
The function
void camCapture(VideoCapture& cap, Mat& frame, bool* Capture)
{
while (*Capture==true)
{
mtx.lock();
cap >> frame;
mtx.unlock();
if(frame.empty())
{
cout<<"No hay captura"<<endl;
}
else
{
cout<<"Frame capturado"<<endl;
}
}cout << "camCapture finished\n";
return;}
The main
int main() {
try{
VideoCapture cap(0); // open the default camera
Mat frame,current_frame, SFI, Input;
bool *Capture = new bool;
*Capture = true;
if (!cap.isOpened()) // check if we succeeded
return -1;
//your capture thread has started
boost::thread captureThread(camCapture, cap, frame, Capture);
while(1)
{
if(frame.empty())
{
cout<<"Frame en hilo principal vacio"<<endl;
}
else{
cout<<"Frame en hilo principal capturado"<<endl;
}
mtx.lock();
current_frame = frame.clone();
mtx.unlock();
if(current_frame.empty())
{
cout<<"Current_Frame vacio"<<endl;
}
else{
imshow("Streaming",current_frame);
if(waitKey(10)==27)break;
}
}
//Terminate the thread
captureThread.join();
}
catch(Exception & e)
{
cout<<e.what()<<endl;
}
return 0;}

according to Boost threads - passing parameters by reference you have to use boost::ref(v) if you want to pass a variable by reference to a boost::thread function. But you could use pointers instead.
In addition you should share the mutex to the thread by passing it as pointer or reference variable instead of using it as global:
#include <iostream>
#include <stdio.h>
#include <boost/thread.hpp>
#include <opencv2/objdetect/objdetect.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
using namespace std;
using namespace cv;
boost::mutex mtx;
void camCapture(VideoCapture& cap, Mat& frame, bool* Capture)
{
while (*Capture == true)
{
mtx.lock();
cap >> frame;
mtx.unlock();
if (frame.empty())
{
cout << "No hay captura" << endl;
}
else
{
cout << "Frame capturado" << endl;
}
}cout << "camCapture finished\n";
return;
}
int main() {
try{
VideoCapture cap(0); // open the default camera
Mat frame, current_frame, SFI, Input;
bool *Capture = new bool; // better not use a pointer here, but use a bool and pass the address or by reference.
*Capture = true;
if (!cap.isOpened()) // check if we succeeded
return -1;
//your capture thread has started
boost::thread captureThread(camCapture, boost::ref(cap), boost::ref(frame), Capture);
while (1)
{
mtx.lock();
current_frame = frame.clone();
mtx.unlock();
if (current_frame.empty())
{
cout << "Current_Frame vacio" << endl;
}
else{
imshow("Streaming", current_frame);
if (waitKey(10) == 27)
{
// TODO: you should use a mutex (or an atomic type) here, too, maybe setting a bool is thread-safe, but this can't be guaranteed for each hardware!
*Capture = false;
break;
}
}
}
//Terminate the thread
captureThread.join();
// give memory free:
delete Capture;
}
catch (Exception & e)
{
cout << e.what() << endl;
}
return 0;
}

Related

Playing video by long key press using open cv

I need to make small video player with OpenCV, which have to support the following functionality. Key 'p' on keyboard - pause/unpause, 'q' - exit, left and right arrow keys - play video frame by frame straight and reverse when it is paused. So the problem is when I try to show video with high quality and I hold arrow key for several seconds it does not run, but freeze and then jump to current frame after I release key. I tried to fix this with adding this_thread::sleep after cv::imshow() to give time to draw, but it did not help at all. So here is the code. Also, I have some reasons to use boost instead of C++11, so it is ok.
main.cpp
#include "VideoPlayer.hpp"
#include <iostream>
int main(int argc, char *argv[])
{
if (argc < 2) {
std::cerr << "Video file full name required as argument." << std::endl;
}
VideoPlayer vp(argv[1]);
if (!vp.play())
return 1;
return 0;
}
VideoPlayer.hpp
#pragma once
#include <opencv/cxcore.hpp>
#include <opencv/highgui.h>
#include <string>
class VideoPlayer
{
public:
VideoPlayer(const std::string &video, const std::string &windowName = "Output window",
unsigned int delay = 30);
bool play();
private:
cv::VideoCapture videoCapture_;
std::string windowName_;
unsigned int delay_;
private:
bool processKey(int key);
void setFrame(int frameNum);
};
VideoPlayer.cpp
#include "VideoPlayer.hpp"
#include <iostream>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/thread/thread.hpp>
VideoPlayer::VideoPlayer(const std::string &video, const std::string &windowName,
unsigned int delay)
: videoCapture_(video)
, windowName_(windowName)
, delay_(delay)
{}
bool VideoPlayer::play()
{
if (!videoCapture_.isOpened()) {
std::cerr << "Unable to open video." << std::endl;
return false;
}
cv::namedWindow(windowName_);
for (;;) {
cv::Mat frame;
videoCapture_ >> frame;
cv::imshow(windowName_, frame);
int key = cv::waitKey(delay_);
if (processKey(key))
break;
}
return true;
}
bool VideoPlayer::processKey(int key)
{
if (key == 'p') {
for (;;) {
int pausedKey = cv::waitKey(0);
if (pausedKey == 'p') {
break;
} else if (pausedKey == 'q') {
return true;
} else if (pausedKey == 65363) {
int frameNum = videoCapture_.get(CV_CAP_PROP_POS_FRAMES);
setFrame(frameNum);
} else if (pausedKey == 65361) {
int frameNum = videoCapture_.get(CV_CAP_PROP_POS_FRAMES);
setFrame(frameNum - 2);
}
}
} else if (key == 'q') {
return true;
}
return false;
}
void VideoPlayer::setFrame(int frameNum)
{
if (frameNum > 0 && frameNum < videoCapture_.get(CV_CAP_PROP_FRAME_COUNT)) {
std::cerr << frameNum << std::endl;
videoCapture_.set(CV_CAP_PROP_POS_FRAMES, frameNum);
cv::Mat frame;
videoCapture_ >> frame;
cv::imshow(windowName_, frame);
boost::this_thread::sleep(boost::posix_time::milliseconds(10));
}
}
I also created a multithreading implementation with buffer based on std::queue with lock, but it didn't solve the problem. And I tried to use boost::lockfree::queue, but I could not manage to finish it because of some strange behavior. I will share this code later, if it is necessary.
So, if somebody knows some good practice, how to avoid this problem, help me please.
EDIT:
Replacing boost::this_thread::sleep(boost::posix_time::milliseconds(10)); with cv::waitKey(0) is bad, because is makes me to do two short presses on arrow key to change one frame, and it does not help, because holding key skips it very fast. So the following code helped, but it is too strange and it is necessary to select times each for each video.
void VideoPlayer::setFrame(int frameNum)
{
if (frameNum > 0 && frameNum < videoCapture_.get(CV_CAP_PROP_FRAME_COUNT)) {
std::cerr << frameNum << std::endl;
videoCapture_.set(CV_CAP_PROP_POS_FRAMES, frameNum);
cv::Mat frame;
videoCapture_ >> frame;
cv::imshow(windowName_, frame);
int times = 7;
for (int i = 0; i < times; i++)
cv::waitKey(10);
}
}
Also, I'm unable to use Qt or something else, only C++03 with boost and OpenCV.
I think I need some trick to make cv::waitKey(time) wait fortime whether any key pressed or not.

Pushing & Popping a frame to a Mat vector

I am trying to grab multiple frames from 2 USB cameras using threads. Issue is i need to push frames into frame_queue vector and then pop one frame at a time from the vector for further image processing.
I followed this link
https://putuyuwono.wordpress.com/2015/05/29/multi-thread-multi-camera-capture-using-opencv/
The code in this link has used concurrent_queue.h for queuing up Mat frames but instead i want to use vector Mat to store the incoming frames.
Header file CameraStreamer.h:
#include <iostream>
#include <string>
#include <thread>
#include <vector>
//#include <concurrent_queue.h>
#include <boost/thread.hpp>
#include "opencv2/videoio.hpp"
using namespace std;
using namespace cv;
//using namespace concurrency;
class CameraStreamer{
public:
//this holds camera stream urls
vector<string> camera_source;
//this holds usb camera indices
vector<int> camera_index;
//this holds OpenCV VideoCapture pointers
vector<VideoCapture*> camera_capture;
//this holds queue(s) which hold images from each camera
vector<Mat*> frame_queue;
//this holds thread(s) which run the camera capture process
vector<thread*> camera_thread;
//Constructor for IP Camera capture
CameraStreamer(vector<string> source);
//Constructor for USB Camera capture
CameraStreamer(vector<int> index);
//Destructor for releasing resource(s)
~CameraStreamer();
private:
bool isUSBCamera;
int camera_count;
//initialize and start the camera capturing process(es)
void startMultiCapture();
//release all camera capture resource(s)
void stopMultiCapture();
//main camera capturing process which will be done by the thread(s)
void captureFrame(int index);
};
Here is my CameraStreamer.cpp:
#include "camerastreamer.h"
CameraStreamer::CameraStreamer(vector<string> stream_source)
{
camera_source = stream_source;
camera_count = camera_source.size();
isUSBCamera = false;
startMultiCapture();
}
CameraStreamer::CameraStreamer(vector<int> capture_index)
{
camera_index = capture_index;
camera_count = capture_index.size();
isUSBCamera = true;
startMultiCapture();
}
CameraStreamer::~CameraStreamer()
{
stopMultiCapture();
}
void CameraStreamer::captureFrame(int index)
{
VideoCapture *capture = camera_capture[index];
while (true)
{
Mat frame;
//Grab frame from camera capture
(*capture) >> frame;
//Put frame to the queue
frame_queue[index]->push(frame);
//relase frame resource
frame.release();
}
}
void CameraStreamer::startMultiCapture()
{
VideoCapture *capture;
thread *t;
vector<Mat> *q;
for (int i = 0; i < camera_count; i++)
{
//Make VideoCapture instance
if (!isUSBCamera){
string url = camera_source[i];
capture = new VideoCapture(url);
cout << "Camera Setup: " << url << endl;
}
else{
int idx = camera_index[i];
capture = new VideoCapture(idx);
cout << "Camera Setup: " << to_string(idx) << endl;
}
//Put VideoCapture to the vector
camera_capture.push_back(capture);
//Make thread instance
t = new thread(&CameraStreamer::captureFrame, this, i);
//Put thread to the vector
camera_thread.push_back(t);
//Make a queue instance
q = new vector<Mat>;
//Put queue to the vector
frame_queue.push_back(q);
}
}
void CameraStreamer::stopMultiCapture()
{
VideoCapture *cap;
for (int i = 0; i < camera_count; i++)
{
cap = camera_capture[i];
if (cap->isOpened()){
//Relase VideoCapture resource
cap->release();
cout << "Capture " << i << " released" << endl;
}
}
}
Main.cpp:
int main()
{
//IP camera URLs
vector<string> capture_source = {
"rtsp://192.168.2.100/profile2/media.smp",
"rtsp://192.168.0.100/profile2/media.smp"
};
//USB Camera indices
vector<int> capture_index = { 0, 1 };
//Highgui window titles
vector<string> label;
for (size_t i = 0; i < capture_index.size(); i++)
{
string title = "CCTV " + to_string(i);
label.push_back(title);
}
//Make an instance of CameraStreamer
CameraStreamer cam(capture_index);
while (waitKey(20) != 27)
{
//Retrieve frames from each camera capture thread
for (size_t i = 0; i < capture_index.size(); i++)
{
Mat *frame;
//Pop frame from queue and check if the frame is valid
cam.frame_queue[i].pushback(frame);
//Show frame on Highgui window
imshow(label[i], frame);
}
}
}
i tried using
cam.frame_queue[i].pushback(frame);
but it doesn't work.
So i am not sure how do i push and pop a frame into frame_queue vector with an index.

imshow is not working

I used a for loop for reading 300 frames and for accumulating them.I gave an imshow command inside to print the frames continuously but they are not printed during the for loop is processing but it comes as a single image
Here's my code:
enter code here
#include<iostream>
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include<stdlib.h>
#include<stdio.h>
using namespace cv;
using namespace std;
int main()
{
char k;
int learningframes=300;
VideoCapture cap(0);
if(cap.isOpened()==0)
{
cout<<"ERROR";
return -1;
}
//while(1)
//{
Mat frame;
cap>>frame;
Mat frameaccF,frameacc,framethr32,framehsv,framethr;
frameaccF=Mat::zeros(frame.size(),CV_32FC1);
for(int i=0;i<=learningframes;i++)
{
cap>>frame;
imshow("nn",frame);
cvtColor(frame,framehsv,CV_BGR2HSV);
inRange(framehsv,Scalar(0,30,0),Scalar(50,150,255),framethr);
framethr.convertTo(framethr,CV_32F);
accumulate(framethr,frameaccF);
}
frameaccF=frameaccF/300;
frameaccF.convertTo(frameaccF,CV_8U);
imshow("frame",frame);
imshow("frameacc",frameaccF);
waitKey(0);
//if(k=='q')
//break;
//}
return 0;
}
You need to put the waiKey() inside the for loop
for(int i=0;i<=learningframes;i++)
{
cap>>frame;
imshow("nn",frame);
cvtColor(frame,framehsv,CV_BGR2HSV);
inRange(framehsv,Scalar(0,30,0),Scalar(50,150,255),framethr);
framethr.convertTo(framethr,CV_32F);
accumulate(framethr,frameaccF);
waitKey(0);
}

OpenCV camera capture from within a thread

This is a small part of the code I'm trying to get to work. This is also one of my first times working with C++. I'm used to higher-level languages, like Java or C#.
The main version is meant to be run as a shared object or DLL. The idea is that an external program (in C#) will start the main loops. The frames from the camera will be captured in a thread. Information will processed inside of that thread and copied to an array ("dataArray"). This copy process will be done while a class mutex is locked. Then, another function called externally will copy that saved array ("dataArray") to a second array ("outArray") and return a pointer to the second array. The external program will use the pointer to copy the data from the second Array, which will not be modified until the function is called again.
But for all that to work, I need the frames to constantly be captured. I realized that I needed something to keep my main function going, so I'm keeping an infinite loop in there. In the "real" version, the keepRunning variable will be changed by the external program running the library.
I was recently lectured on StackOverflow about not making global variables, so I'm keeping the one instance of my class in a static member. That's pretty standard in Java. I don't know if it's bad practice in C++. I was also taken by surprise as to how C++ threads start as soon as they're created, without an explicit "start" instructions. That's why I'm putting my only thread in a vector. That seems to be what most people recommend.
I understand that without keepRunning never being actually changed, the threads will never be joined, but I'll dear with that later. I'm running this on a Mac, but I'll need it to eventually run on Windows, Mac and Linux.
Here's my header:
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <thread>
#include <vector>
using namespace cv;
using namespace std;
class MyCap {
public:
MyCap();
VideoCapture cap;
static MyCap * instance;
void run();
static void RunThreads(MyCap * cap);
bool keepRunning = true; // Will be changed by the external program.
vector<thread> capThreads;
private:
Mat frame;
};
And here's my code:
#include "theheader.h"
MyCap * MyCap::instance = NULL;
int main(int argc, char** argv) {
MyCap::instance = new MyCap();
MyCap::instance->capThreads.push_back(thread(MyCap::RunThreads, MyCap::instance));
// Outside loop.
while(MyCap::instance->keepRunning) {
}
for (int i = 0; i < MyCap::instance->capThreads.size(); i++) {
MyCap::instance->capThreads[i].join();
}
}
MyCap::MyCap() {
namedWindow("flow", 1);
cap.open(0);
}
void MyCap::RunThreads(MyCap * cap) {
cap->run();
}
void MyCap::run() {
// Inside loop.
while(keepRunning) {
cap >> frame;
imshow("flow", frame);
if (waitKey(30) >= 0) {
break;
}
}
}
With this code, I get a black screen. If I run cap.open(0) from within the run method, I don't even get that. I'm obviously doing something very wrong. But what really puzzles me is: why does it make a difference where that same code is called from? If I run what is now in run inside of main it will work. If I change the call of cap.open(0) from the constructor to run, that changes what the method does. Also the waitKey condition stops working from within the thread. What big thing am I missing?
Version 2
Based on the suggestions of #darien-pardibas, I made this second version:
Header:
#include <stdio.h>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <thread>
#include <vector>
using namespace cv;
using namespace std;
class MyCap {
public:
MyCap();
void run();
bool keepRunning = true; // Will be changed by the external program.
static void RunThreads(MyCap * cap);
static vector<thread> capThreads;
static MyCap * getInstance();
private:
static MyCap * instance;
};
The main file:
#include "theprogram.h" // I'll admit that, even for a placeholder, it was a bad name.
MyCap * MyCap::instance = NULL;
vector<thread> MyCap::capThreads;
MyCap::MyCap() {
cout << "Instantiate" << endl;
}
MyCap * MyCap::getInstance() {
if (MyCap::instance == NULL) {
MyCap::instance = new MyCap;
}
return MyCap::instance;
}
void MyCap::RunThreads(MyCap * cap) {
cap->run();
}
void MyCap::run() {
cout << "Run" << endl;
namedWindow("flow", 1);
cout << "Window created." << endl;
VideoCapture cap(0); // HANGS HERE!
cout << "Camera open." << endl; // This never gets printed.
// Inside loop.
Mat frame;
while(keepRunning) {
cap >> frame;
imshow("flow", frame);
if (waitKey(30) >= 0) {
break;
}
}
}
int main(int argc, char** argv) {
MyCap::capThreads.push_back(thread(&MyCap::RunThreads, MyCap::getInstance()));
for (int i = 0; i < MyCap::capThreads.size(); i++) {
MyCap::capThreads[i].join();
}
}
This prints:
Instantiate
Run
Window created.
And hangs there.
But if I move the code from run to main and change keepRunning to true, then it works as expected. I think I'm missing something else, and I'm guessing it has something to do with how C++ works.
Okay, without looking at resolving all design patterns issues I can see in your code, I can confirm that the code below works. I think the main problem was that you needed to create the namedWindow in the same thread where you will be capturing the image and remove the while loop you had in your main method.
// "theheader.h"
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <thread>
#include <vector>
class MyCap {
public:
void run();
static void RunThreads(MyCap * cap);
bool keepRunning = true; // Will be changed by the external program.
std::vector<std::thread> capThreads;
private:
cv::Mat frame;
cv::VideoCapture cap;
MyCap() { }
static MyCap * s_instance;
public:
static MyCap *instance();
};
// "theheader.cpp"
#include "theheader.h"
#pragma comment(lib, "opencv_core248d")
#pragma comment(lib, "opencv_highgui248d")
using namespace std;
using namespace cv;
MyCap * MyCap::s_instance = NULL;
MyCap* MyCap::instance() {
if (s_instance == NULL)
s_instance = new MyCap();
return s_instance;
}
void MyCap::RunThreads(MyCap * cap) {
cap->run();
}
void MyCap::run() {
namedWindow("flow", 1);
cap.open(0);
// Inside loop.
while (keepRunning) {
cap >> frame;
imshow("flow", frame);
if (waitKey(30) >= 0) {
break;
}
}
}
int main(int argc, char** argv) {
MyCap::instance()->capThreads.push_back(thread(&MyCap::RunThreads, MyCap::instance()));
for (int i = 0; i < MyCap::instance()->capThreads.size(); i++) {
MyCap::instance()->capThreads[i].join();
}
}

Why does OpenCV emit "Bad flag in unknown function" for this code?

I have been trying to get a real basic video player working using the Open CV API. Everything seems to run smoothly until the end of a video clip, then i get this error:
OpenCV Error: Bad flag in unknown function, file ........\ocv\opencv\modules\core\src\array.cpp
This creates a break in the code at imshow("video", frame), i find this wierd as this is the part that effectively plays the video so why does it only kick up a fuss at the end of the clip?? I found that it seems to give me this error in the last 90% of every video i play so at the moment im working around it by telling it to stop once 90% of the clip has played but this isnt very good programming so can anyone send some suggestions/help?
I have looked at other peoples post on this matter and none of the solution suggested have worked for me as of yet.
Heres my code...its only an experimentation piece so im sorry if its a bit messy:
Thanks for any help in advance
#include <cv.h>
#include <cxcore.h>
#include <highgui.h>
#include <direct.h>
#include <iostream>
using namespace cv;
void onTrackBarSlide(int);
double the_next_play_frame;
VideoCapture video("test.MOV"); // open the video file
int main(int, char**)
{
if(!video.isOpened()) // check if we succeeded
{
return -1;
}
int no_of_frames = video.get(CV_CAP_PROP_FRAME_COUNT); //total number of frames in video
std::cout << no_of_frames << std::endl;
std::cout << video.get(CV_CAP_PROP_FPS) << std::endl;
namedWindow("Video", 1);
cvCreateTrackbar("trackbar", "Video", 0, 40, onTrackBarSlide);
double stop_at = no_of_frames * 0.999;
for(;;){
if(the_next_play_frame >= double(stop_at))
{
break;
}
Mat frame;
video >> frame; // get a new frame from camera
imshow("Video", frame); // <---------- place where break/error occurs
if(waitKey(30) >= 0)
{
break;
}
}
return 0;
}
void onTrackBarSlide(int pos)
{
std::cout << getTrackbarPos("trackbar", "Video") << std::endl;
double frameratio = video.get(CV_CAP_PROP_FRAME_COUNT)/40; //10005 is the maximum value the slider can actual hold
double next_play_frame = frameratio * getTrackbarPos("trackbar", "Video");
video.set(CV_CAP_PROP_POS_FRAMES,next_play_frame);
the_next_play_frame = next_play_frame;
}
VideoCapture video("test.MOV"); // open the video file
int main(int, char**)
{
if(!video.isOpened()) // check if we succeeded
{
return -1;
}
}
Try put the VideoCapture instantiation inside main.
int main(){
VideoCapture video("test.MOV"); // open the video file
...
}