I am trying to create a video recorder with Qt. What I did so far was taking a screenshot of a rectangle on the screen and save it. At the end I use ffmpeg to get a video file out of the images.
I connected a timer's signal timeout() to my custom slot which takes the snapshot and saves it to my tmp folder. The timer has an intervall of 1000 / 30. That should be 30 times per second. But 1000 / 30 is a little bit more than 33 milliseconds so I cannot really get 30 fps. It's a bit more.
I recorded a youtube video with my recorder and everything was smooth but a little bit faster / slower depending on the intervall.
So my question basically is how do I get really 30 / 40 / 50 / ... fps?
Start a QElapsedTimer when you start the capture.
When you are done capturing a frame (at the end of your snapshot slot), multiply the next frame number by the approximate frame duration (a double precision floating point value) in milliseconds (e.g. for 30 fps, it is ~33.33333333333, but don't write that - write (double)1000/30). Call this value next_timestamp.
Call elapsed() on your QElapsedTimer. Call this value current_timestamp.
Call the static function QTimer::singleShot() to sleep for next_timestamp - current_timestamp. Set the slot argument to your snapshot slot again. Note that if the time to sleep is <= 0, you are falling behind; your system can't keep up with the load of capturing that fast.
When QTimer::singleShot() fires, your snapshot slot will be called again with some error in timing. However, this error is inevitable under non-realtime operating systems such as Windows and Unix (OS X/Linux/etc). This is because you don't get to decide when code executes - the OS kernel does. On average, though, you will end up with exactly 30 frames per second (assuming your computer can keep up with the load!), because the time elapsed reported by the QElapsedTimer will be quite accurate, and if the system falls behind, it will capture frames more quickly, and if it gets ahead, it will capture frames more slowly.
Related
I am about to capture frames from a video grabber card. Those frames are processed and written to the HDD.
The whole setting is in a multithreading environment, so the grabber writes the images to a queue, and in another thread, the images is processed and another one writes to hdd. If the image is good by the definition of the processor, the image gets written to hdd. If 10 images in a row are "bad", the file is completed. If there are 9 images or less "bad", all the images get written with the next good image, so the file writer gets informed.
Here the problem, if I do not do it this way, instead writing each file directly after it is processed, the video file is fine, But 9 "bad" images are written. If I do it the way in my description above, the speed/frame rate of the video is not suitable. This description is a bit weird, so here is just a simplified example, so you can see the problem:
void FrameWriter::writeFrameLoop() {
string path = getPath();
cv::Size2i size(1350, 1080);
cv::VideoWriter videoWriter(path, fourcc, 30, size);
while (this->isRunning) {
while (!this->frames.empty()) {
usleep(100000); // this effects the speed/frame
videoWriter.write(this->pop());
}
std::this_thread::sleep_for(10ms);
}
videoWriter.release();
}
The example is pretty simple, here I "block" the writing process with a sleep, remember this is a different thread. This means after the capturing is stopped, the file writing takes a bit longer.
But I would expect that this does not effect the video itself, because there is a framerate of 30 and the images are still in the same order. But for me it seems to effect the video file, when I call "videoWriter.write" not in time. In this case the video is much faster than expected.
I thought only the configured frames of 30 and the count of written images would effect the video speed, but it is not. Can anyone help me do understand what is going on here?
I am using openCV 4.4.0 with Ubuntu 18.04.
Thank you for your help.
BR Michael
I think I know the reason of fast-playing result videos.
In constructor cv::VideoWriter videoWriter(path, fourcc, 30, size); you set frame rate (FPS) of resulting video to 30. It means that CV library expects exactly 30 frames to be written by write() function for each 1 second of resulting video stream.
Also for CV library there's no difference how fast you call write() with new frame, you may call it 5 times per second or 10 or even 1000 times. The only thing that matters is that you have to provide exactly 30 frames for each second of video, and how fast you provide these 30 frames doesn't matter.
I mean that all your sleep(...) functionality doesn't matter for CV VideoWriter class. And it is always true for all video rendering/conversion libraries. So pausing thread doesn't change anything at all.
But in your case you're saying that you grab 10 frames per second from real-time video data of your grabber's video card. It means the your FPS is really 10 frames per second. So in order to solve your task correctly next things should be done:
Remove all pausing functionality, like calling sleep(). It is not needed at all. And doesn't change behavior of VideoWriter.
Then first way to solve the task is to change in your constructor cv::VideoWriter videoWriter(path, fourcc, 30, size); value 30 to 10. This already will solve your task, but you have to be sure that you grab 10 frames per second, not more not less. Then your video will be a correctly playing (correct speed) video with a frame rate of 10 frames per second. This is the simplest solution. Video doesn't need to be 30 FPS for correctly playing later, 10 FPS video will be correctly played later by any player.
Another solution, if you really want your resulting video to play 30 frames per second, not less, not more, then duplicate each frame of your grabbed video three times, thus you'll get 30 frames out of 10 frames of your grabbed video. By duplicating I just mean that you should call videWriter.write(...) three times (in a small loop) with same single frame, call this write without any pause (like sleep). Then again your resulting video will have exactly 30 frames per second.
I think you just miss understood how CV::VideoWriter works. You thought that write() renders resulting video in real time, meaning that if you feed it 10 frames but exactly within one second period, then it should render correct speed of video. But this writer renders video not in real time, meaning that it just supposes that 10 frames passed constitute just 1/3 of second of resulting videos, hence it expects 30 frames to be written for 1 resulting second.
If you have a camera that's not able to provide a constant frame rate of let's say 30 frames, you can also consider limiting the frame rate yourself to e.g. 25 and measure the time elapsed since writing the last frame. You can also change your framerate to arbitrary values, as long as the camera is able to provide it. An example of an implementation:
m_fps = 25;
std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
std::chrono::steady_clock::time_point now;
while (1) {
now = std::chrono::steady_clock::now();
long duration = std::chrono::duration_cast<std::chrono::nanoseconds>(now - start).count();
if ((double)duration > (1e9 / m_fps)) {
start = std::chrono::steady_clock::now();
m_duration += duration;
// Capture frame-by-frame
if(m_cap->grab()) { m_cap->retrieve(m_frame); }
// write frame
m_writer->write(m_frame);
cv::waitKey(1);
}
}
I figured out my problem and it was me, not OpenCV.
Because of the multithreading environment, I was writing the images (cv::Mat) to a queue. Since there are a lot of transformations YUV -> RGB -> BGR -> Crop etc. I was expecting the cv::Mat object I put in the queue was a deep copy. Instead of this, the cv::Mat was always the same object. Which means, even if there was already 20 cv::Mat entries in my queue, all of them was the same and all of them "changed" when ever there was a new image.
after scheduleUpdate, the update:(ccTime)dt will be called 60 times per second, what if at one time the update method's running time exceeds 1/60 second? the next call will be cancelled?
The framerate drops. Nothing will be cancelled.
At 60 fps there's exactly 1/60th of a second for cocos2d and your code to process everything that's needed to render a frame, including all OpenGL drawing operations. That's 0.016666666 seconds to do it all.
If one update cycle takes longer than that, the next frame will be rendered after 0.03333333 seconds instead, dropping the framerate to 30 fps if multiple frames continuously take longer to process. Provided that everything is done within that time, otherwise the next frame update will be deferred to 0.05 seconds or even 0.06666666 seconds.
You can only get 60, 30, 20 or 15 fps framerate with cocos2d since it uses CADisplayLink which synchronizes updates with the screen refresh rate. The framerate counter in cocos2d may show 40 fps or something because it averages over multiple frames.
I am currently working on an OpenGL application to display a few 3D spheres to the user, which they can rotate, move around, etc. That being said, there's not much in the way of complexity here, so the application runs at quite a high framerate (~500 FPS).
Obviously, this is overkill - even 120 would be more then enough, but my issue is that running the application at full-stat eats away my CPU, causing excess heat, power consumption, etc. What I want to do is be able to let the user set an FPS cap so that the CPU isn't being overly used when it doesn't need to be.
I'm working with freeglut and C++, and have already set up the animations/event handling to use timers (using the glutTimerFunc). The glutTimerFunc, however, only allows an integer amount of milliseconds to be set - so if I want 120 FPS, the closest I can get is (int)1000/120 = 8 ms resolution, which equates to 125 FPS (I know it's a neglegible amount, but I still just want to put in an FPS limit and get exactly that FPS if I know the system can render faster).
Furthermore, using glutTimerFunc to limit the FPS never works consistently. Let's say I cap my application to 100 FPS, it usually never goes higher then 90-95 FPS. Again, I've tried to work out the time difference between rendering/calculations, but then it always overshoots the limit by 5-10 FPS (timer resolution possibly).
I suppose the best comparison here would be a game (e.g. Half Life 2) - you set your FPS cap, and it always hits that exact amount. I know I could measure the time deltas before and after I render each frame and then loop until I need to draw the next one, but this doesn't solve my 100% CPU usage issue, nor does it solve the timing resolution issue.
Is there any way I can implement an effective, cross-platform, variable frame rate limiter/cap in my application? Or, in another way, is there any cross-platform (and open source) library that implements high resolution timers and sleep functions?
Edit: I would prefer to find a solution that doesn't rely on the end user enabling VSync, as I am going to let them specify the FPS cap.
Edit #2: To all who recommend SDL (which I did end up porting my application to SDL), is there any difference between using the glutTimerFunc function to trigger a draw, or using SDL_Delay to wait between draws? The documentation for each does mention the same caveats, but I wasn't sure if one was more or less efficient then the other.
Edit #3: Basically, I'm trying to figure out if there is a (simple way) to implement an accurate FPS limiter in my application (again, like Half Life 2). If this is not possible, I will most likely switch to SDL (makes more sense to me to use a delay function rather then use glutTimerFunc to call back the rendering function every x milliseconds).
I'd advise you to use SDL. I personnally use it to manage my timers. Moreover, it can limit your fps to your screen refresh rate (V-Sync) with SDL 1.3. That enables you to limit CPU usage while having the best screen performance (even if you had more frames, they wouldn't be able to be displayed since your screen doesn't refresh fast enough).
The function is
SDL_GL_SetSwapInterval(1);
If you want some code for timers using SDL, you can see that here :
my timer class
Good luck :)
I think a good way to achieve this, no matter what graphics library you use, is to have a single clock measurement in the gameloop to take every single tick (ms) into account. That way the average fps will be exactly the limit just like in Half-Life 2. Hopefully the following code snippet will explain what I am talking about:
//FPS limit
unsigned int FPS = 120;
//double holding clocktime on last measurement
double clock = 0;
while (cont) {
//double holding difference between clocktimes
double deltaticks;
//double holding the clocktime in this new frame
double newclock;
//do stuff, update stuff, render stuff...
//measure clocktime of this frame
//this function can be replaced by any function returning the time in ms
//for example clock() from <time.h>
newclock = SDL_GetTicks();
//calculate clockticks missing until the next loop should be
//done to achieve an avg framerate of FPS
// 1000 / 120 makes 8.333... ticks per frame
deltaticks = 1000 / FPS - (newclock - clock);
/* if there is an integral number of ticks missing then wait the
remaining time
SDL_Delay takes an integer of ms to delay the program like most delay
functions do and can be replaced by any delay function */
if (floor(deltaticks) > 0)
SDL_Delay(deltaticks);
//the clock measurement is now shifted forward in time by the amount
//SDL_Delay waited and the fractional part that was not considered yet
//aka deltaticks
the fractional part is considered in the next frame
if (deltaticks < -30) {
/*dont try to compensate more than 30ms(a few frames) behind the
framerate
//when the limit is higher than the possible avg fps deltaticks
would keep sinking without this 30ms limitation
this ensures the fps even if the real possible fps is
macroscopically inconsitent.*/
clock = newclock - 30;
} else {
clock = newclock + deltaticks;
}
/* deltaticks can be negative when a frame took longer than it should
have or the measured time the frame took was zero
the next frame then won't be delayed by so long to compensate for the
previous frame taking longer. */
//do some more stuff, swap buffers for example:
SDL_RendererPresent(renderer); //this is SDLs swap buffers function
}
I hope this example with SDL helps. It is important to measure the time only once per frame so every frame is taken into account.
I recommend to modularize this timing in a function which also makes your code clearer. This code snipped has no comments in the case they just annoyed you in the last one:
unsigned int FPS = 120;
void renderPresent(SDL_Renderer * renderer) {
static double clock = 0;
double deltaticks;
double newclock = SDL_GetTicks();
deltaticks = 1000.0 / FPS - (newclock - clock);
if (floor(deltaticks) > 0)
SDL_Delay(deltaticks);
if (deltaticks < -30) {
clock = newclock - 30;
} else {
clock = newclock + deltaticks;
}
SDL_RenderPresent(renderer);
}
Now you can call this function in your mainloop instead of your swapBuffer function (SDL_RenderPresent(renderer) in SDL). In SDL you'd have to make sure the SDL_RENDERER_PRESENTVSYNC flag is turned off. This function relies on the global variable FPS but you can think of other ways of storing it. I just put the whole thing in my library's namespace.
This method of capping the framerate delivers exactly the desired average framerate if there are no large differences in the looptime over multiple frames because of the 30ms limit to deltaticks. The deltaticks limit is required. When the FPS limit is higher than the actual framerate deltaticks will drop indefinitely. Also when the framerate then rises above the FPS limit again the code would try to compensate the lost time by rendering every frame immediately resulting in a huge framerate until deltaticks rises back to zero. You can modify the 30ms to fit your needs, it is just an estimate by me. I did a couple of benchmarks with Fraps. It works with every imaginable framerate really and delivers beautiful results from what I have tested.
I must admit I coded this just yesterday so it is not unlikely to have some kind of bug. I know this question was asked 5 years ago but the given answers did not statify me. Also feel free to edit this post as it is my very first one and probably flawed.
EDIT:
It has been brought to my attention that SDL_Delay is very very inaccurate on some systems. I heard a case where it delayed by far too much on android. This means my code might not be portable to all your desired systems.
The easiest way to solve it is to enable Vsync. That's what I do in most games to prevent my laptop from getting too hot.
As long as you make sure the speed of your rendering path is not connected to the other logic, this should be fine.
There is a function glutGet( GLUT_ELAPSED_TIME ) which returns the time since started in miliseconds, but that's likely still not fast enough.
A simple way is to make your own timer method, which uses the HighPerformanceQueryTimer on windows, and the getTimeOfDay for POSIX systems.
Or you can always use timer functions from SDL or SFML, which do basically the same as above.
You should not try to limit the rendering rate manually, but synchronize with the display vertical refresh. This is done by enabling V sync in the graphics driver settings. Apart from preventing (your) programs from rendering at to high rate, it also increases picture quality by avoiding tearing.
The swap interval extensions allow your application to fine tune the V sync behaviour. But in most cases just enabling V sync in the driver and letting the buffer swap block until sync suffices.
I would suggest using sub-ms precision system timers (QueryPerformanceCounter, gettimeofday) to get timing data. These can help you profile performance in optimized release builds also.
Some background information:
SDL_Delay is pretty much the same as Sleep/sleep/usleep/nanosleep but it is limited to milliseconds as parameter
Sleeping works by relying on the systems thread scheduler to continue your code.
Depending on your OS and hardware the scheduler may have a lower tick frequency than 1000hz, which results in longer timespans than you have specified when calling sleep, so you have no guarantee to get the desired sleep time.
You can try to change the scheduler's frequency. On Windows you can do it by calling timeBeginPeriod(). For linux systems checkout this answer.
Even if your OS supports a scheduler frequency of 1000hz your hardware may not, but most modern hardware does.
Even if your scheduler's frequency is at 1000hz sleep may take longer if the system is busy with higher priority processes, but this should not happen if your system isn't under super high load.
To sum up, you may sleep for microseconds on some tickless linux kernels, but if you are interested in a cross platform solution you should try to get the scheduler frequency up to 1000hz to ensure the sleeps are accurate in most of the cases.
To solve the rounding issue for 120FPS:
1000/120 = 8,333ms
(int)1000/120 = 8ms
Either you do a busy wait for 333 microseconds and sleep for 8ms afterwards. Which costs some CPU time but is super accurate.
Or you follow Neop approach by sleeping sometimes 8ms and sometimes 9ms seconds to average out at 8,333ms. Which is way more efficent but less accurate.
In OpenGL, everything works with the main loop (as far as I know). This is a problem if you want to draw an object for given time. For now, what I do is the following: I measure the approximate FPS of my application and set up a counter that I decrease at every iteration in the main loop. When the counter reaches 0, I stop drawing. So for example if I want to draw an object for 2s on the screen, and my FPS is 30, I set up the counter to 60 and draw the object until the counter reaches 0.
This works OK, but it's not great. For example, it makes it hard to write something like
drawObjectFor(object, time) since the object is sometimes "far" from the main loop. Is there a better way to do that?
Your FPS may (and usually will) vary, so counting frames is not a great idea.
Assume this example to see how this could bite you:
Your application runs vsynced to 60 FPS, and your frame time is 16.58ms. That means to have the object on screen for 2 seconds, you will have to draw your object for 120 frames. Easy, let's go!
Drawing the object adds 0.1 ms, giving a total frame time of 16.68ms. Oh sh*t. Now you have 30 FPS, and your object shows for 4 seconds...
Use proper time to decide whether or not to draw your object. When you've reached the conclusion that your object's time is up, just set a flag so you won't draw it (or, remove it from the scenegraph, if you have one, or delete it, or whatever).
This can be done in a variety of ways:
Under Windows
Use SetTimer, which will post a message to your window's message queue. When you receive the WM_TIMER notification, set the object's do_draw flag (or whatever you want to call it) to false.
If you already use WaitFor(Single|Multiple)Object somewhere once every frame, do a CreateWaitableTimer, and wait on this as well. You can use MsgWaitForMultipleObjectsEx to do the message pumping and waiting in one go, and check APCs at the same time.
Nothing prevents you from reading time manually once per frame. GetTickCount once, add 2000, and do an if(new_count <= old_count + 2000) draw_object(). While the accuracy of GetTickCount is abysmal (several dozens of milliseconds), this does not matter at all when waiting for 2 seconds.
Under Linux
Use a timerfd and an epoll to check readiness, or set the timerfd it to nonblocking and see whether reading from it returns EAGAIN
Use clock_gettime and do it manually (as with GetTickCount)
Use setitimer
You could have a "time to live" field in the object itself, which counts down until the object is to be destroyed (or at least made invisible).
Then drawObjectFor(object, time) would just set object.timeToLive = time. You would then also need to pass the frame time to the object, perhaps when rendering it, so it can decrease it's timeToLive accordingly.
Just looking on resources that break down how frames per second work. I know it has something to do with keeping track of Ticks and figure out how many ticks occured between each frame. But I never ran into any resources on why exactly you have to use the methods you use in order to get a smooth frame work. I am trying to get a thourough understanding of this. Can any explain or provide any good resources ? Thanks
There are basically two approaches.
In ActionScript (and many other engines), you request the player to call a certain function at a certain framerate. For Flash games, you'll set the framerate to be 30 FPS, and then you'll implement a function that listens for ENTER_FRAME events to do what you need to do. This means you get roughly 33 ms per frame (1000ms/30FPS=33.33ms/frame). If your code that responds to ENTER_FRAME takes more than 33 ms, you'll get some stuttering.
In home-rolled main loops (like you'd generally do in C++/SDL, for example), you run the main loop as fast as possible. This means the time between each frame will be variable. You still need to keep the "guts" of your frame code less than 33 ms to make sure you'll get at least 30 FPS, but your game will run faster than 30 FPS if not a lot's going on. To account for this, you need to program all your logic in terms of elapsed time since last frame, and abandon using frames themselves as a unit of time.
http://forums.xna.com/forums/t/42624.aspx
How do you separate game logic from display?
For a continously variable frame rate you can measure the time the last frame took and assume this frame will take the same length of time. This has the benefit of meaning that time runs more or less constantly. Your biggest issue with this approach is that it is entirely possible for a VSync'd game to change from 60 fps to 30 fps and back again on subsequent frames. From experience a good way to solve this is to average the last few frame times. This smooths the result out. In the 60 to 30 fps switch each frame will progress assuming 1/45 seconds and hence the 60fps frame run slow and the 30fps frame run fast and the perceived speed remains at 45fps.
Better still is to not use this sort of time step in your calculations. Set yourself a minimum fps ... say 10fps. You then calculate all your game logic at some multiple of these 1/10 second intervals. The render engine then knows where the object is and where it is heading towards and so can inter/extrapolate the object position until its next "decision" frame shows up. This has numerous advantages. It decouples your logic entirely from rendering. It allows you to spread the logic calculations over a number of frames. For example, at 60Hz, you can test at what point a logic object will interesect with the world if it maintains its path every 6th frame. This gives the bonus of allowing you to process some logic objects on different frames to spread calculation load across the time. Its biggest disadvantage is that if the frame rate drops below your target rate everything slows down.