I am semantically segmenting images within a video sequence and displaying the result within a display window in real-time. However, my code is "dropping frames" by not updating the display window with each call to imshow.
Below is a code extract that demonstrates my problem:
for (int i = 0; i < images->size(); i++)
{
Mat frame = images->getImage(i);
// semantic segmentation code (omitted)
Mat frame_overlay = ...
// problem: the window "seg result" does not always update
imshow("seg result", frame_overlay);
int key = waitKey(1);
if (key == 27)
break;
}
As a work around I can increase the wait time from 1 ms to 10 ms in the call to waitKey, but surely there has to be a better way of doing this.
How can I force the "seg result" window to update with the new frame_overlay image?
Related
I'm running into an odd problem with OpenCV on Linux, Ubuntu 16.04 specifically. If I use usual code to show a webcam stream like this it works fine:
// WebcamTest.cpp
#include <opencv2/opencv.hpp>
#include <iostream>
int main()
{
// declare a VideoCapture object and associate to webcam, 1 => use 2nd webcam, the 0th webcam is the one integral to the TX2 development board
cv::VideoCapture capWebcam(1);
// check if VideoCapture object was associated to webcam successfully, if not, show error message and bail
if (capWebcam.isOpened() == false)
{
std::cout << "error: capWebcam not accessed successfully\n\n";
return (0);
}
cv::Mat imgOriginal; // input image
cv::Mat imgGrayscale; // grayscale of input image
cv::Mat imgBlurred; // intermediate blured image
cv::Mat imgCanny; // Canny edge image
char charCheckForEscKey = 0;
// while the Esc key has not been pressed and the webcam connection is not lost . . .
while (charCheckForEscKey != 27 && capWebcam.isOpened())
{
bool blnFrameReadSuccessfully = capWebcam.read(imgOriginal); // get next frame
// if frame was not read successfully, print error message and jump out of while loop
if (!blnFrameReadSuccessfully || imgOriginal.empty())
{
std::cout << "error: frame not read from webcam\n";
break;
}
// convert to grayscale
cv::cvtColor(imgOriginal, imgGrayscale, CV_BGR2GRAY);
// blur image
cv::GaussianBlur(imgGrayscale, imgBlurred, cv::Size(5, 5), 0);
// get Canny edges
cv::Canny(imgBlurred, imgCanny, 75, 150);
cv::imshow("imgOriginal", imgOriginal);
cv::imshow("imgCanny", imgCanny);
charCheckForEscKey = cv::waitKey(1); // delay (in ms) and get key press, if any
} // end while
return (0);
}
This example shows the webcam stream in one imshow window and a Canny edges image in a second window. Both windows update and show the images as expected with very little if any perceptible flicker.
If you're wondering why I'm using the 1th camera instead of the usual 0th camera, I'm running this on a Jetson TX2 and the 0th camera is the one integral to the development board and I'd prefer to use an additional external webcam. For this same reason I have to use Ubuntu 16.04 but I suspect the result would be the same with Ubuntu 18.04 (have not tested this however).
If instead I have a function that takes significant processing instead of taking simple Canny edges, i.e.:
int main(void)
{
.
.
.
// declare a VideoCapture object and associate to webcam, 1 => use 2nd webcam, the 0th webcam is the one integral to the TX2 development board
cv::VideoCapture capWebcam(1);
// check if VideoCapture object was associated to webcam successfully, if not, show error message and bail
if (capWebcam.isOpened() == false)
{
std::cout << "error: capWebcam not accessed successfully\n\n";
return (0);
}
cv::namedWindow("imgOriginal");
cv::Mat imgOriginal;
char charCheckForEscKey = 0;
// while the Esc key has not been pressed and the webcam connection is not lost . . .
while (charCheckForEscKey != 27 && capWebcam.isOpened())
{
bool blnFrameReadSuccessfully = capWebcam.read(imgOriginal); // get next frame
// if frame was not read successfully, print error message and jump out of while loop
if (!blnFrameReadSuccessfully || imgOriginal.empty())
{
std::cout << "error: frame not read from webcam\n";
break;
}
detectLicensePlate(imgOriginal);
cv::imshow("imgOriginal", imgOriginal);
charCheckForEscKey = cv::waitKey(1); // delay (in ms) and get key press, if any
} // end while
.
.
.
return (0);
}
The detectLicensePlate() function takes about a second to run.
The problem I'm having is, when running this program, the window only appears for the slightest amount of time, usually not long enough to even be perceptible, and never long enough to actually see the result.
The strange thing is, the window disappears, then the second or so day occurs for detectLicensePlate() to do its thing, then the window appears again for a very short time, then disappears again, and so on. It's almost as though just after cv::imshow("imgOriginal", imgOriginal);, cv::destroyAllWindows(); is implicitly being called.
The behavior I'm attempting to achieve is for the window to stay open and continue to show the previous result while processing the next. From what I recall this was the default behavior on Windows.
I should mention that I'm explicitly declaring the windows with cv::namedWindow("imgOriginal"); before the while loop in an attempt to not let it go out of scope but this does not seem to help.
Of course I can make the delay longer, i.e.
charCheckForEscKey = cv::waitKey(1500);
To wait for 1.5 seconds, but then the application gets very unresponsive.
Based on this post c++ opencv image not display inside the boost thread I tried declaring the window outside the while loop and putting detectLicensePlate() and cv::imshow() on a separate thread, as follows:
.
.
.
cv::namedWindow("imgOriginal");
boost::thread myThread;
// while the Esc key has not been pressed and the webcam connection is not lost . . .
while (charCheckForEscKey != 27 && capWebcam.isOpened())
{
// if frame was not read successfully, print error message and jump out of while loop
if (!blnFrameReadSuccessfully || imgOriginal.empty())
{
std::cout << "error: frame not read from webcam\n";
break;
}
myThread = boost::thread(&preDetectLicensePlate, imgOriginal);
myThread.join();
.
.
.
} // end while
// separate function
void preDetectLicensePlate(cv::Mat &imgOriginal)
{
detectLicensePlate(imgOriginal);
cv::imshow("imgOriginal", imgOriginal);
}
I even tried putting detectLicensePlate() on a separate thread but not cv::imshow(), and the other way around, still the same result. No matter how I change the order or use threading I can't get the window to stay open while the next round of processing is going.
I realize I could use an entirely different windowing environment, such as Qt or something else, and that may or may not solve the problem, but I'd really prefer to avoid that for various reasons.
Does anybody have any other suggestions to get an OpenCV imshow window to stay open until the window is next updated or cv::destroyAllWindows() is called explicitly?
I am running a code something like that;
Mat img1 = imread("C:\\input0.png");
namedWindow("original");
imshow("original", img1 );
int lowThreshold = 20;
int ratio = 2;
int kernel_size = 5;
Mat edge_map;
Mat gray_image;
cvtColor(img1, gray_image, CV_BGR2GRAY);
Canny(gray_image, edge_map, lowThreshold, lowThreshold*ratio, kernel_size);
namedWindow("Edge Image");
imshow("Edge Image", edge_map);
Mat result_image = produce_the_result_image(img1, edge_map);
namedWindow("Final Image");
imshow("Final Image", result_image );
int key = 1;
while (key != 'q') {
key = waitKey(5);
}
Till shows the last image (Final Image window), other windows show 'not responding', however, after the last function finishes (produce_the_result_image), which lasts 2-3 minutos and shows the all image windows, the error disappears. Is that normal?
Thanks!
Images in OpenCV will not display or respond until you have called waitKey();. So if after your first two calls to imshow you call waitKey(1);, the image will display (it will wait one millisecond for a key press and then it will become nonresponsive). If you call waitKey(); with no arguments or use a while loop similar to the one at the end of the code, the images will display and respond. Once you press a key, the images will become nonresponsive again.
So yes, this is normal behavior for OpenCV. Note that it applies to all windows at once: either they are all responsive, if waitKey is currently being called, or none of them are responsive.
I've been trying to simultaneously grab frames from two different cameras, but no matter how many times I call VideoCapture::grab(), there seems to be no effect. The frame retrieved using VideoCapture::retrieve() is always the first frame captured from the last VideoCapture::retrieve().
I've tested it on both OpenCV 2.4 and 3.1, with a Logitech C920 camera on windows.
Example:
VideoCapture vCapture;
Mat imgResult;
vCapture.open(0); //at this point, there is a green sheet in front of the camera
Sleep(100000); //change green sheet with red sheet
vCapture.grab(); //returns true
vCapture.retrieve(imgResult); //image with green sheet is retrieved
Sleep(100000); //change red sheet with blue sheet
vCapture.retrieve(imgResult); //red sheet is retrieved
I've also tried:
for(int i = 0; i < 1000; i++){
vCapture.grab();
} //takes almost no processing time, like an empty for
vCapture.retrieve(imgResult); //same as before
Retrieve always returns true and retrieves a frame, even if no grab was called since opening vCapture.
My current solution has been retrieving the frame twice (multi-threaded) to ensure the latest frame, but it isn't reliable enough to sync both cameras. Can anyone shed some light on how to force the camera to grab the current frame?
Thanks!
Edit:
A variation of the first example, for clarity:
VideoCapture vCapture;
Mat imgResult;
vCapture.open(0); //at this point, there is a green sheet in front of the camera
vCapture.retrieve(imgResult); //image with green sheet is retrieved
Sleep(100000); //change green sheet with red sheet
vCapture.grab(); //returns true
vCapture.retrieve(imgResult); //green sheet is retrieved
vCapture.retrieve(imgResult); //red sheet is retrieved
Sleep(100000); //change red sheet with blue sheet
vCapture.retrieve(imgResult); //red sheet is retrieved
vCapture.retrieve(imgResult); //blue sheet is retrieved
Expected behavior:
VideoCapture vCapture;
Mat imgResult;
vCapture.open(0); //at this point, there is a green sheet in front of the camera
vCapture.retrieve(imgResult); //error or image with green sheet is retrieved
Sleep(100000); //change green sheet with red sheet
vCapture.grab(); //returns true
vCapture.retrieve(imgResult); //red sheet is retrieved
Per OpenCV documentation:
VideoCapture::grab: The methods/functions grab the next frame from video file or camera and return true (non-zero) in the case of success.
VideoCapture::retrieve: The methods/functions decode and return the just grabbed frame. If no frames has been grabbed (camera has been disconnected, or there are no more frames in video file), the methods return false and the functions return NULL pointer.
Please try this code with the following instructions:
before and while you start the program, hold a red sheet in front of the camera. That moment, the first .grab will be called.
Once you see the black window popping up, remove the red sheet and hold a blue sheet or something else (except the red or the green sheet) in front of the camera. Then press keyboard key 'q'.
Now you have 5 seconds time to change the scene again. Hold hold the green sheet in front of the camera and wait. The black window will be switched to one of your camera images.
int main(int argc, char* argv[])
{
cv::Mat input = cv::Mat(512,512,CV_8UC1, cv::Scalar(0));
cv::VideoCapture cap(0);
while (cv::waitKey(10) != 'q')
{
cap.grab();
cv::imshow("input", input);
}
cv::waitKey(5000);
cap.retrieve(input);
cv::imshow("input", input);
cv::waitKey(0);
return 0;
}
3 possible results:
you see the red sheet: this means that the first grab was called and fixed the image, until a retrieve was called.
you see blue sheet: this means every .grab call "removes" one image and the camera will capture a new image on the next call of .grab
you see the green sheet: this means your .retrieve doesn't need a .grab at all and just grabs images automatically.
For me, result 1 occurs, so you can't grab and grab and grab and just .retrieve the last image.
Test 2: control about everything:
looks like you are right, on my machine no matter when or how often I call grab, the next image will be the one captured at the time when I called the previous .retrieve and the calls of .grab don't seem to influence the time position of capturing at all.
Would be very interesting whether the same behaviour occurs for different (all) kind of cameras and operating systems.
I've tested on the internal camera of a T450s and Windows 7.
int main(int argc, char* argv[])
{
cv::Mat input = cv::Mat(512,512,CV_8UC1, cv::Scalar(0));
cv::VideoCapture cap(0);
bool grabbed;
bool retrieved;
while (true)
{
char w = cv::waitKey(0);
switch (w)
{
case 'q': return 0;
case 27: return 0;
case ' ': retrieved = cap.retrieve(input); break;
case 'p': grabbed = cap.grab(); break;
}
cv::imshow("input", input);
}
return 0;
}
In addition, this simple code seems to be off 1 frame for my camera (which therefore probably has a buffersize of 2??):
while (true)
{
cap >> input;
cv::imshow("input", input);
cv::waitKey(0);
}
I ran my test and note very strange behavior of .garab and .retrive functions.
This is example:
cv::Mat input = cv::Mat(512, 512, CV_8UC1, cv::Scalar(0));
cv::VideoCapture cap(0);
while (true)
{
cap.grab();
cap.retrieve(input, 5);
cv::imshow("input", input);
cv::waitKey(0);
}
If you press any key slowly, about every 5 seconds, and change something in front of the camera between pressing, the position of the object on the image will change every second image showing, that is, every second call of the .grab and .retrive functions.
If you press any key quickly, about every 1 seconds, and also change something in front of the camera between pressing, the position of the object on the image will be changed every image showing.
This circumstance tell about that this function can be used to sync cameras.
I am trying to substract this building brick.
.
For that I am using the KNN algorithm provided by opencv 3.0.
To initialize the background model I am using 40 frames without the brick.
All in all it works pretty well.
(Brick with Shadow)
The only problem is that the algorithm starts loosing the brick around Frame 58
(Image shows frame 62)
After frame 64 I get only black images. I know this wouldn't happen if the brick would move, but unfortunatly there are long sequences where it doesn`t.
Does somebody know a solution to this?
PS: I tried playing around with the history Paramer of
cv::createBackgroundSubtractorKNN(int history,double Threshold, bool detectShadows= true)
But there is no difference between history = 500 or history = 500000
A easy but slow solution is to reinitialize the background model every five frames.
for (size_t i = 0; i < imageList.size(); i++){
if (i % 5 == 0){
for (auto& it : backgroundList){
string nextFrameFilename(it.string());
frame = cv::imread(nextFrameFilename);
pMOG->apply(frame, fgMaskMOG2);
imshow("Frame", frame);
imshow("FG Mask MOG 2", fgMaskMOG2);
keyboard = cv::waitKey(30);
}
}
}
I am using opencv to show frames from camera. I want to show that frames in to two separation windows. I want show real frame from camera into first window (show frames after every 30 mili-seconds) and show the frames in second window with some delay (that means it will show frames after every 1 seconds). Is it possible to do that task. I tried to do it with my code but it is does not work well. Please give me one solution to do that task using opencv and visual studio 2012. Thanks in advance
This is my code
VideoCapture cap(0);
if (!cap.isOpened())
{
cout << "exit" << endl;
return -1;
}
namedWindow("Window 1", 1);
namedWindow("Window 2", 2);
long count = 0;
Mat face_algin;
while (true)
{
Mat frame;
Mat original;
cap >> frame;
if (!frame.empty()){
original = frame.clone();
cv::imshow("Window 1", original);
}
if (waitKey(30) >= 0) break;// Delay 30ms for first window
}
You could write the loop to display frames in a single function with the video file name as the argument and call them simultaneously by multi-threading.
The pseudo code would look like,
void* play_video(void* frame_rate)
{
// play at specified frame rate
}
main()
{
create_thread(thread1, play_video, normal_frame_rate);
create_thread(thread2, play_video, delayed_frame_rate);
join_thread(thread1);
join_thread(thread2);
}