I'm writing a producer/consumer code that receives frame date from an external library.
Each frame is read in a callback function from an external library that runs in a parallel thread and is pushed into a Mat queue. I created another function that runs in a different thread that reads and pops each frame.
The problem is that I'm getting "Access violation reading location" when I'm trying to read the frame data from the queue.
I'm declaring theses variables globally:
queue<Mat> matQ;
OnFrameDataReceivedCB videoCB;
OnDeviceConnectStatusCB connectCB;
guide_usb_video_mode_e videoMode;
int width = 640;
int height = 512;
std::mutex mu;
Here goes the code for the callback function that pushes each frame data:
void OnVideoCallBack(const guide_usb_frame_data_t data) //callback function
{
if (data.frame_rgb_data_length > 0)
{
// Send the displayed data directly
unsigned char* rgbData;
Mat frame;
Size size = Size(width, height);
rgbData = data.frame_rgb_data;
frame = Mat(size, CV_8UC3, rgbData, Mat::AUTO_STEP);
if (mu.try_lock())
{
printf("producing...\n");
matQ.push(frame);
printf("free producing\n");
mu.unlock();
}
}
}
Here is the function that reads from the queue:
void OnHandleVideoData()
{
while (true)
{
try
{
if (matQ.size() <= 0)
{
chrono::milliseconds duration(200);
this_thread::sleep_for(duration);
continue;
}
if (mu.try_lock())
{
if (matQ.size() > 0)
{
printf("consuming...\n");
Size size = Size(width, height);
Mat frame = Mat(size, CV_8UC3);
frame = matQ.front().clone();
matQ.pop();
imwrite("frame.jpg", frame); //the access violation exception is thrown on this line
printf("free consuming\n");
mu.unlock();
}
}
}
catch (...)
{
}
}
}
I also tried to put the unsigned char* rgbData array into a queue instead of the Mat, but I got the same error.
What am I missing?
You should try to clone the frame immediately when receiving it:
frame = Mat(size, CV_8UC3, rgbData, Mat::AUTO_STEP).clone();
instead of there:
frame = matQ.front()/*.clone()*/;
Related
I have a camera application that will display the cameras live stream:
void MainWindow::on_start()
{
if(video.isOpened())
{
video.release();
return;
}
if(!isCameraFound || !video.open(pipeline.trimmed().toStdString(), cv::CAP_GSTREAMER))
{
QMessageBox::critical(this,
"Video Error",
"Must be the correct USB Camera connected!");
return;
}
Mat frame;
while(video.isOpened())
{
video >> frame;
if(!frame.empty())
{
QImage qimg(frame.data,
frame.cols,
frame.rows,
frame.step,
QImage::Format_RGB888);
pixmap.setPixmap( QPixmap::fromImage(qimg.rgbSwapped()) );
ui->graphicsView->fitInView(&pixmap, Qt::KeepAspectRatio);
}
...
And I have a function that does processing and has a loop to inspect the first 250 frames because the camera gives off a specific frame when something is found. But when doing this loop the videostream lags so much it basically freezes until the loop is done. How would I be able to go about removing the lag caused by the frame grabbing loop to be able to go through the loop and have the videostream not lag?
void MainWindow::on_push()
{
Mat test;
for(int i = 0; i < 250; i++){
video >> test;
...
I am having an OpenCV program which works like this:
VideoCapture cap(0);
Mat frame;
while(true) {
cap >> frame;
myprocess(frame);
}
The problem is if myprocess takes a long time which longer than camera's IO interval, the captured frame be delayed, cannot get the frame synchronized with the real time.
So, I think to solve this problem, should make the camera streaming and myprocess run parallelly. One thread does IO operation, another does CPU computing. When the camera finished capture, send to work thread to processing.
Is this idea right? Any better strategy to solve this problem?
Demo:
int main(int argc, char *argv[])
{
cv::Mat buffer;
cv::VideoCapture cap;
std::mutex mutex;
cap.open(0);
std::thread product([](cv::Mat& buffer, cv::VideoCapture cap, std::mutex& mutex){
while (true) { // keep product the new image
cv::Mat tmp;
cap >> tmp;
mutex.lock();
buffer = tmp.clone(); // copy the value
mutex.unlock();
}
}, std::ref(buffer), cap, std::ref(mutex));
product.detach();
while (cv::waitKey(20)) { // process in the main thread
mutex.lock();
cv::Mat tmp = buffer.clone(); // copy the value
mutex.unlock();
if(!tmp.data)
std::cout<<"null"<<std::endl;
else {
std::cout<<"not null"<<std::endl;
cv::imshow("test", tmp);
}
}
return 0;
}
Or use a thread keep clearing the buffer.
int main(int argc, char *argv[])
{
cv::Mat buffer;
cv::VideoCapture cap;
std::mutex mutex;
cap.open(0);
std::thread product([](cv::Mat& buffer, cv::VideoCapture cap, std::mutex& mutex){
while (true) { // keep product the new image
cap.grab();
}
}, std::ref(buffer), cap, std::ref(mutex));
product.detach();
int i;
while (true) { // process in the main thread
cv::Mat tmp;
cap.retrieve(tmp);
if(!tmp.data)
std::cout<<"null"<<i++<<std::endl;
else {
cv::imshow("test", tmp);
}
if(cv::waitKey(30) >= 0) break;
}
return 0;
}
The second demo I thought shall be work base on https://docs.opencv.org/3.0-beta/modules/videoio/doc/reading_and_writing_video.html#videocapture-grab, but it's not...
In project with Multitarget tracking I used 2 buffers for frame (cv::Mat frames[2]) and 2 threads:
One thread for capturing the next frame and detect objects.
Second thread for tracking the detected objects and draw result on frame.
I used index = [0,1] for the buffers swap and this index was protected with mutex. For signalling about the end of work was used 2 condition variables.
First works CatureAndDetect with frames[capture_ind] buffer and Tracking works with previous frames[1-capture_ind] buffer. Next step - switch the buffers: capture_ind = 1 - capture_ind.
Do you can this project here: Multitarget-tracker.
I`m using vlc-qt for decode h264 video stream but I need every frame from video (stream) for further processing. I found this link that describes the solution :
https://discuss.tano.si/t/how-to-get-frame-from-video/253
I made a class that inherits from VlcVideoStream class and re-implement frameUpdated() function as bellow :
void MyVideoStream::frameUpdated() {
qDebug() << "frame" ;
int rows, cols, matType;
// convert to shared pointer to const frame to avoid crash
std::shared_ptr<const VlcYUVVideoFrame> frame = std::dynamic_pointer_cast<const VlcYUVVideoFrame>(renderFrame());
if (!frame) {
return; // LCOV_EXCL_LINE
}
rows = frame->height + frame->height/2;
cols = frame->width;
}
and declared my class as :
MyVideoStream *_stream ;
_stream = new MyVideoStream(Vlc::YUVFormat,ui->video) ;
_stream->init(_player) ;
where _player is a VlcMediaPlayer object reference. but when I ran the program nothing happened. I don`t know what is the problem.
when you subclass the VlcVideoStream and re-implement frameUpdated(), you have access to YUV frame every time it is updated.
if you are familiar with Opencv , just add these code to frameUpdated() function, then you can see the gray image :
void MyVideoStream::frameUpdated()
{
std::shared_ptr<const VlcYUVVideoFrame> frame= std::dynamic_pointer_cast<const VlcYUVVideoFrame>(renderFrame());
if (!frame) {
return; // LCOV_EXCL_LINE
}
int width = frame->width;
int height = frame->height;
cv::Mat result = cv::Mat(height, width, CV_8UC1, (void*)frame->frameBuffer.data());
imshow("result", result);
waitKey(2);
}
I tried using imshow in opencv to display a but the image window is closed and displayed again in every loop. Is there a way to hold the display until the new imshow arrives so the image display doesn't look flickering?
for(int iframe = 0; iframe < 10; iframe++)
{
..some processing code..
cv::imshow("image", a[iframes]);
cv::waitKey(1);
}
check your ...some processing code...:
If you're using namedWindow() in the beginning of each loop and/or using destroyWindow() at the end of the loop, then you're effectively closing your window in each iteration.
Remove these function calls unless you're absolutely sure that you need them.
The two threads doesn't share any parameters except the display window. Here is my code. I didn't include the processing part since it's quite long.
int main(int argc, char *argv[])
{
// Some processings//
// Create the thread
cv::namedWindow("image2D");
std::thread t1(task1, "Start");
t1.join();
}
void task1(std::string msg)
{
std::cout << "Task 1: " << msg << std::endl;
// Some processings to compute img1
// Display img1
float min = -50;
float max = 100;
cv::Mat adjMap = cv::Mat::zeros(height, width, CV_8UC1);
cv::Mat img1;
float scale = 255.0 / (max - min);
zw.convertTo(adjMap, CV_8UC1, scale);
applyColorMap(adjMap, img1, cv::COLORMAP_JET);
cv::imshow("image2D", img1); // The code hangs here
cv::waitKey(1);
}
I have a few vision algorithms which perform well enough on live camera streams; however, it is far from good when I run these algorithms on video files; the stream slows down way too much, although it is fine when not running on a vision algorithm, which are executed by calling VideoAlgoVision->execute( grabbedFrame, &CurrentROI );
Here is how I read video so far:
//////////////////////////////////////////////////////////////////
void VisionUnit_widget::Play()
{
if(IsVideoPause)
{
ui->pausePushButton->setEnabled(true);
ui->playPushButton->setEnabled(false);
IsVideoPause = false;
TimerOpen->start( (int) (1000/FPS_open) );
return;
}
else
{
ui->pausePushButton->setEnabled(true);
ui->stopPushButton ->setEnabled(true);
ui->rewPushButton ->setEnabled(true);
ui->ffdPushButton ->setEnabled(true);
ui->videoSlider ->setEnabled(true);
ui->playPushButton ->setEnabled(false);
if(!VideoCap)
VideoCap = new VideoCapture( FileName.toStdString() );
else
VideoCap->open( FileName.toStdString() );
if( VideoCap->isOpened() )
{
FrameH = (int) VideoCap->get(CV_CAP_PROP_FRAME_HEIGHT);
FrameW = (int) VideoCap->get(CV_CAP_PROP_FRAME_WIDTH);
FPS_open = (int) VideoCap->get(CV_CAP_PROP_FPS);
NumFrames = (int) VideoCap->get(CV_CAP_PROP_FRAME_COUNT);
ui->videoSlider->setMaximum( (int)NumFrames );
ui->videoSlider->setEnabled(true);
READCOUNT = 0;
TimerOpen->start( (int) (1000/FPS_open) );
}
}
}
//////////////////////////////////////////////////////////////////
void VisionUnit_widget::Pause()
{
ui->playPushButton->setEnabled(true);
ui->pausePushButton->setEnabled(false);
TimerOpen->stop();
IsVideoPause = true;
}
//////////////////////////////////////////////////////////////////
void VisionUnit_widget::Stop()
{
ui->stopPushButton->setEnabled(false);
ui->playPushButton->setEnabled(false);
ui->pausePushButton->setEnabled(false);
ui->rewPushButton->setEnabled(false);
ui->ffdPushButton->setEnabled(false);
FileName = "";
TimerOpen->stop();
READCOUNT = 0;
ui->videoSlider->setSliderPosition(0);
ui->videoSlider->setEnabled(false);
ui->frameLabel->setText( "No camera connected" );
delete TimerOpen;
TimerOpen = 0;
if(VideoCap)
{
delete VideoCap;
VideoCap = 0;
}
if(VideoAlgoVision)
{
delete VideoAlgoVision;
VideoAlgoVision = 0;
}
}
//////////////////////////////////////////////////////////////////
void VisionUnit_widget::Read()
{
READCOUNT++;
// Update Video Player's slider
ui->videoSlider->setValue(READCOUNT);
if(READCOUNT >= NumFrames) // if avi ends
{
Pause();
return;
}
Mat grabbedFrame;
// Get next frame
if(VideoCap->isOpened() && VideoCap->read(grabbedFrame))
{
// Execute the vision filter
if(VideoAlgoVision)
VideoAlgoVision->execute( grabbedFrame, &CurrentROI );
// Convert Mat to QImage
QImage frame = MatToQImage( grabbedFrame );
// Update the display
UpdateFrame( frame );
}
}
//////////////////////////////////////////////////////////////////
QImage VisionUnit_widget::MatToQImage(const Mat& mat)
{
// 8-bits unsigned, NO. OF CHANNELS = 1
if(mat.type()==CV_8UC1)
{
// Set the color table (used to translate colour indexes to qRgb values)
QVector<QRgb> colorTable;
for (int i=0; i<256; i++)
{
colorTable.push_back(qRgb(i,i,i));
}
// Copy input Mat
const uchar *qImageBuffer = (const uchar*)mat.data;
// Create QImage with same dimensions as input Mat
QImage img(qImageBuffer, mat.cols, mat.rows, mat.step, QImage::Format_Indexed8);
img.setColorTable(colorTable);
return img;
}
// 8-bits unsigned, NO. OF CHANNELS=3
else if(mat.type()==CV_8UC3)
{
// Copy input Mat
const uchar *qImageBuffer = (const uchar*)mat.data;
// Create QImage with same dimensions as input Mat
QImage img(qImageBuffer, mat.cols, mat.rows, mat.step, QImage::Format_RGB888);
return img.rgbSwapped();
}
else
{
return QImage();
}
}
So, my question is, is there a better way than this to read video files with Qt and OpenCV, including video processing? Should I adapt the QTimer timing at run time? I start it with TimerOpen->start( (int) (1000/FPS_open) ); but obviously the vision algorithm will slow down the whole thing. Any thought?
There may be some optimization to make on the vision algorithm, however my point here is they do well on my webcam and IP cameras, having me think there is something wrong with the way I read/use video files.
thx
You did not provide the whole code however i guess that you have connected TimerOpen's timeout() signal to VisionUnit_widget::Read() slot. If this is the case, you are accumulating 1000/FPS_open and processing time.
Changing your desing to something like the following will solve it :
int fSpace = 1000 / FPS;
QTime timer;
timer.start();
forever {
if(timer.elapsed() < fSpace)
msleep(1);
timer.restart();
..Process the frame..
}
And probably it is better to move this to another thread from main thread.