In OpenCV is there a way to dramatically increase the frame rate of a video (.mp4). I have tried to methods to increase the playback of a video including :
Increasing the frame rate:
cv.SetCaptureProperty(cap, cv.CV_CAP_PROP_FPS, int XFRAMES)
Skipping frames:
for( int i = 0; i < playbackSpeed; i++ ){originalImage = frame.grab();}
&
video.set (CV_CAP_PROP_POS_FRAMES, (double)nextFrameNumber);
is there another way to achieve the desired results? Any suggestions would be greatly appreciated.
Update
Just to clarify, the play back speed is NOT slow, I am just searching for a way to make it much faster.
You're using the old API (cv.CaptureFromFile) to capture from a video file.
If you use the new C++ API, you can grab frames at the rate you want. Here is a very simple sample code :
#include "opencv2/opencv.hpp"
using namespace cv;
int main(int, char**)
{
VideoCapture cap("filename"); // put your filename here
namedWindow("Playback",1);
int delay = 15; // 15 ms between frame acquisitions = 2x fast-forward
while(true)
{
Mat frame;
cap >> frame;
imshow("Playback", frame);
if(waitKey(delay) >= 0) break;
}
return 0;
}
Basically, you just grab a frame at each loop and wait between frames using cvWaitKey(). The number of milliseconds that you wait between each frame will set your speedup. Here, I put 15 ms, so this example will play a 30 fps video at approx. twice the speed (minus the time necessary to grab a frame).
Another option, if you really want control about how and what you grab from video files, is to use the GStreamer API to grab the images, and then convert to OpenCV for image tratment. You can find some info about this on this post : MJPEG streaming and decoding
Related
So I'm writing a C++ program that will take a wav file, generate a visualization, and export the video out alongside the audio using ffmpeg (pipe). I've been able to get output out to ffmpeg just fine and a video with the visualization and audio are created by ffmpeg.
The problem is the video and audio are desyncing. The video is just too fast and ends before the song is completed (the video file is the correct length; the waveform just flatlines and ends, indicating that ffmpeg reached the end of the video and is just using the last frame it received until the audio ends). So I'm not sending enough frames to ffmpeg.
Below is a truncated version of the source code:
// Example code
int main()
{
// LoadAudio();
uint32_t frame_max = audio.sample_rate / 24; // 24 frames per second
uint32_t frame_counter = 0;
// InitializePipe2FFMPEG();
// (Left channel and right channel are always equal in size)
for (uint32_t i = 0; i < audio.left_channel.size(); ++i)
{
// UpdateWaveform4Image();
if (frame_counter % frame_max == 0)
{
// DrawImageAndSend2Pipe();
frame_counter = 1;
}
else
{
++frame_counter;
}
}
// FlushAndClosePipe();
return 0;
}
The commented-out functions are fake and irrelevant. I know this because "UpdateWaveform4Image()" updates the waveform used to generate the image every sample. (I know that's inefficient, but I'll worry about optimization later.) The waveform is a std::vector in which each element stores the y-coordinate of each sample. It has no effect on when the program will generate a new frame for the video.
Also ffmpeg is set to output 24 frames per second--trust me, I thought that was the problem too because by default ffmpeg outputs to 25 fps.
My line of thinking for the modulus check is that frame_counter is incremented every sample. frame_max equals 2000 because 48000 / 24 = 2000. I know the audio is clocked at 48kHz because I created the file myself. So it SHOULD generate a new image every 2000 samples.
Here is a link to the output video: [output]
Any advice would be helpful.
EDIT: Skip to 01:24 to see the waveform flatline.
I have a system that is typically running a scan time of 100 HZ or 10 ms and performing time critical tasks. I'm trying to add a camera with opencv to once a while (depends on when a user interacts with the system so it can be anywhere from 10 seconds pauses to minutes) capture an image for quality control.
Here is what my code is doing:
int main(int, char**)
{
VideoCapture cap(0); // open the default camera
if(!cap.isOpened()) // check if we succeeded
return -1;
UMat frame;
for(;;){
if (timing_variable_100Hz){
cap >> frame; // get a new frame from camera
*Do something time critical*
if(some_criteria_is_met){
if(!frame.empty()) imwrite( "Image.jpg", frame);
}
}
}
return 0;
}
Now the issue I'm having is that cap >> frame takes a lot of time.
My scan time regularly runs around 3ms and now it's at 40ms. Now my question is, are there anyway to open the camera, capture, then not have to capture every frame after until I have to? I tried to move the cap >> frame inside the if(some_criteria_is_met) which allowed me to capture the first image correctly but the second image which was taken a few minutes later was a single frame past the first captured image (I hope that makes sense).
Thanks
The problem is that your camera has a framerate of less than 100fps, probably 30fps (according to the 32ms you measured), so grab wait for a new frame to be available.
Since there is no way to do a non blocking read in opencv, i think that your best option is to do the video grabbing in another thread.
Something like this, if you use c++11 (this is an example, not sure it is entirely correct):
void camera_loop(std::atomic<bool> &capture, std::atomic<bool> &stop)
{
VideoCapture cap(0);
Mat frame;
while(!stop)
{
cap.grab();
if(capture)
{
cap.retrieve(frame);
// do whatever you do with the frame
capture=false;
}
}
}
int main()
{
std::atomic<bool> capture=false, stop=false;
std::thread camera_thread(camera_loop, std::ref(capture), std::ref(stop));
for(;;)
{
// do something time critical
if(some_criteria_is_met)
{
capture=true;
}
}
stop=true;
camera_thread.join();
}
It doesn't answer your question of are there anyway to open the camera, capture, then not have to capture every frame after until I have to?, but a suggestion
You could try and have the cap >> frame in a background thread which is responsible only for capturing the frames.
Once the frame is in memory, push it to some sort of shared cyclic queue to be accessed from the main thread.
I have problem with plying video file, why it is slow motion ?
How can I make it normal speed?
#include"opencv2/opencv.hpp"
using namespace cv;
int main(int, char**)
{
VideoCapture cap("eye.mp4");
// open the default camera
if (!cap.isOpened())
// check if we succeeded
return -1;
namedWindow("Video", 1);
while (1)
{
Mat frame;
cap >> frame;
imshow("Video", frame);
if (waitKey(10) == 'c')
break;
}
return 0;
}
VideoCapture isn't built for playback, it's just a way to grab frames from video file or camera. Other libraries that supports playback, such as GStreamer or Directshow, they set a clock that control the playback, so that it can be configured to play as fastest as possible or use the original framerate.
In your snippet, the interval between frames comes from the time it takes to read a frame and the waitKey(10). Try using waitKey(0), it should at least play faster. Ideally, you could use waitKey(1/fps).
I am capturing video through a webcam which gives a mjpeg stream.
I did the video capture in a worker thread.
I start the capture like this:
const std::string videoStreamAddress = "http://192.168.1.173:80/live/0/mjpeg.jpg?x.mjpeg";
qDebug() << "start";
cap.open(videoStreamAddress);
qDebug() << "really started";
cap.set(CV_CAP_PROP_FRAME_WIDTH, 720);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 576);
the camera is feeding the stream at 20fps.
But if I did the reading in 20fps like this:
if (!cap.isOpened()) return;
Mat frame;
cap >> frame; // get a new frame from camera
mutex.lock();
m_imageFrame = frame;
mutex.unlock();
Then there is a 3+ seconds lag.
The reason is that the captured video is first stored in a buffer.When I first start the camera, the buffer is accumulated but I did not read the frames out. So If I read from the buffer it always gives me the old frames.
The only solutions I have now is to read the buffer at 30fps so it will clean the buffer quickly and there's no more serious lag.
Is there any other possible solution so that I could clean/flush the buffer manually each time I start the camera?
OpenCV Solution
According to this source, you can set the buffersize of a cv::VideoCapture object.
cv::VideoCapture cap;
cap.set(CV_CAP_PROP_BUFFERSIZE, 3); // internal buffer will now store only 3 frames
// rest of your code...
There is an important limitation however:
CV_CAP_PROP_BUFFERSIZE Amount of frames stored in internal buffer memory (note: only supported by DC1394 v 2.x backend currently)
Update from comments. In newer versions of OpenCV (3.4+), the limitation seems to be gone and the code uses scoped enumerations:
cv::VideoCapture cap;
cap.set(cv::CAP_PROP_BUFFERSIZE, 3);
Hackaround 1
If the solution does not work, take a look at this post that explains how to hack around the issue.
In a nutshell: the time needed to query a frame is measured; if it is too low, it means the frame was read from the buffer and can be discarded. Continue querying frames until the time measured exceeds a certain limit. When this happens, the buffer was empty and the returned frame is up to date.
(The answer on the linked post shows: returning a frame from the buffer takes about 1/8th the time of returning an up to date frame. Your mileage may vary, of course!)
Hackaround 2
A different solution, inspired by this post, is to create a third thread that grabs frames continuously at high speed to keep the buffer empty. This thread should use the cv::VideoCapture.grab() to avoid overhead.
You could use a simple spin-lock to synchronize reading frames between the real worker thread and the third thread.
Guys this is pretty stupid and nasty solution, but accepted answer didn't helped me for some reasons. (Code in python but the essence pretty clear)
# vcap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
data = np.zeros((1140, 2560))
image = plt.imshow(data)
while True:
vcap = cv2.VideoCapture("rtsp://admin:#192.168.3.231")
ret, frame = vcap.read()
image.set_data(frame)
plt.pause(0.5) # any other consuming operation
vcap.release()
An implementation of Hackaround 2 in Maarten's answer using Python. It starts a thread and keeps the latest frame from camera.read() as a class attribute. A similar strategy can be done in c++
import threading
import cv2
# Define the thread that will continuously pull frames from the camera
class CameraBufferCleanerThread(threading.Thread):
def __init__(self, camera, name='camera-buffer-cleaner-thread'):
self.camera = camera
self.last_frame = None
super(CameraBufferCleanerThread, self).__init__(name=name)
self.start()
def run(self):
while True:
ret, self.last_frame = self.camera.read()
# Start the camera
camera = cv2.VideoCapture(0)
# Start the cleaning thread
cam_cleaner = CameraBufferCleanerThread(camera)
# Use the frame whenever you want
while True:
if cam_cleaner.last_frame is not None:
cv2.imshow('The last frame', cam_cleaner.last_frame)
cv2.waitKey(10)
You can make sure that grabbing the frame took a bit of time. It is quite simple to code, though a bit unreliable; potentially, this code could lead to a deadlock.
#include <chrono>
using clock = std::chrono::high_resolution_clock;
using duration_float = std::chrono::duration_cast<std::chrono::duration<float>>;
// ...
while (1) {
TimePoint time_start = clock::now();
camera.grab();
if (duration_float(clock::now() - time_start).count() * camera.get(cv::CAP_PROP_FPS) > 0.5) {
break;
}
}
camera.retrieve(dst_image);
The code uses C++11.
There is an option to drop old buffers if you use a GStreamer pipeline. appsink drop=true option "Drops old buffers when the buffer queue is filled". In my particular case, there is a delay (from time to time) during the live stream processing, so it's needed to get the latest frame each VideoCapture.read call.
#include <chrono>
#include <thread>
#include <opencv4/opencv2/highgui.hpp>
static constexpr const char * const WINDOW = "1";
void video_test() {
// It doesn't work properly without `drop=true` option
cv::VideoCapture video("v4l2src device=/dev/video0 ! videoconvert ! videoscale ! videorate ! video/x-raw,width=640 ! appsink drop=true", cv::CAP_GSTREAMER);
if(!video.isOpened()) {
return;
}
cv::namedWindow(
WINDOW,
cv::WINDOW_GUI_NORMAL | cv::WINDOW_NORMAL | cv::WINDOW_KEEPRATIO
);
cv::resizeWindow(WINDOW, 700, 700);
cv::Mat frame;
const std::chrono::seconds sec(1);
while(true) {
if(!video.read(frame)) {
break;
}
std::this_thread::sleep_for(sec);
cv::imshow(WINDOW, frame);
cv::waitKey(1);
}
}
If you know the framerate of your camera you may use this information (i.e. 30 frames per second) to grab the frames until you got a lower frame rate.
It works because if grab function become delayed (i.e. get more time to grab a frame than the standard frame rate), it means that you got every frame inside the buffer and opencv need to wait the next frame to come from camera.
while(True):
prev_time=time.time()
ref=vid.grab()
if (time.time()-prev_time)>0.030:#something around 33 FPS
break
ret,frame = vid.retrieve(ref)
i have a Raspberry Pi and installed on it the OpenCV and Guvcview. When i open Guvcview, i get ~ 17-21 fps but when i run a simple program (only capture from webcam and display frame) in C++ with Opencv, i get only 6 fps.
What is wrong? i need to configure Opencv to use Guvcview's configuration? why guvcview get 20 fps? What can i do?
thanks.
P.D. I've done the same in my computer and i get 29 fps in both cases.
//*********************************this is the code C++ :
#include <iostream>
#include "opencv2/opencv.hpp"
using namespace std;
using namespace cv;
time_t start, end; //variabile di tipo time_t , contiene tempo in sec.
// inizializzo contatore nella dichiarazione
int counter=0;
int main()
{ time(&start);
VideoCapture cap(1);
cap.set(CV_CAP_PROP_FRAME_WIDTH, 640);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 480);
if (!cap.isOpened())
{ cout << "could not capture";
return 0; }
Mat frame;
namedWindow("camera", 1);
char key = 'a';
while(key != 27)
{ cap.read( frame);
imshow("camera", frame);
//##################
//time at the end of 1 show, Stop the clock and show FPS
time(&end);
++counter;
cout <<"fps: "<< counter/ difftime(end,start) <<endl <<endl;
//##################
key = waitKey(3); }
destroyAllWindows();
return 0;
}
OpenCV is a heavy weight API and following tips may introduce minor improvements:
you can disable RGB conversion:
cap.set(CV_CAP_PROP_CONVERT_RGB , false);
you can increase frame rate if its default frame rate is low:
cap.set(CV_CAP_PROP_FPS , 60);
I'd suggest to do direct video capture via V4L, since OpenCV may do YUYV to RGB transformations and other stuff that involves floating point calculations, that are expensive on this kind of hardware. We have done many robotics projects on embedded systems and the rule of the thumb is that you will be always better of either directly using V4L or small 3rd party libraries like CMVision (http://www.cs.cmu.edu/~jbruce/cmvision/) to do image processing on embedded systems.