I have a Millikey Response Box with a 1 000 Hz sampling rate and a light sensor with a 10 000 Hz sampling rate. I would like to measure end-to-end response time from the moment of a button press to a change on the screen triggered by the button press in my C++ program. I'm struggling to understand how to do it.
My idea was that, ideally, I would have the button press create a keypress event that holds a timestamp of when it was created at a 1 000 Hz sampling rate. My C++ program would handle this event at its own frequency by recording the high-frequency timestamp and triggering the brightness change on the screen. Next, the light sensor would pick up the change and generate its own keypress event at a sampling rate of 10 000 Hz. At the next cycle, my C++ program would pick up this event and get the actual end-to-end time by comparing the two high-resolution timestamps.
Is this idea sensible? If yes, how can I implement it?
So far I used GLFW to capture keyboard events and to get the timestamp in the key callback, but as far as I understand, that is not exactly the time when the key was pressed.
Trying to do all that with a computer and peripherals is likely going to lead you nowhere, because of the unknown and likely erratic latency in various parts of the chain.
So here is a trick I have used successfully to measure latency. I connected the light sensor to a sound generator. I then recorded the sound of the button being hit together with the sound from the sensor. Recorded at 96Khz, this will give you a very accurate and precise reading. You just have to measure the delay in an audio editor, or I guess you could have a stand alone program to analyse the audio too.
The answer for me was to use LabStreamingLayer. I use App-Input to capture keyboard events, LabRecorder to capture the stream of these events, and then Python importer to parse the resulting XDF file. All the above runs and captures events in the background while the keypress triggers the screen change in my C++ program which is detected by the light sensor.
I'm sure that the other answers and comments make good suggestions for when one would want to implement this on a low level themselves. My question was also not refined since my understanding was limited, so thank you for the contribution anyway!
Related
I am writing an application for Windows that runs a CUDA accelerated HDR algorithm. I've set up an external image signal processor device that presents as a UVC device, and delivers 60 frames per second to the Windows machine over USB 3.0.
Every "even" frame is a more underexposed frame, and every "odd" frame is a more overexposed frame, which allows my CUDA code perform a modified Mertens exposure fusion algorithm to generate a high quality, high dynamic range image.
Very abstract example of Mertens exposure fusion algorithm here
My only problem is that I don't know how to know when I'm missing frames, since the only camera API I have interfaced with on Windows (Media Foundation) doesn't make it obvious that a frame I grab with IMFSourceReader::ReadSample isn't the frame that was received after the last one I grabbed.
Is there any way that I can guarantee that I am not missing frames, or at least easily and reliably detect when I have, using a Windows available API like Media Foundation or DirectShow?
It wouldn't be such a big deal to miss a frame and then have to purposefully "skip" the next frame in order to grab the next oversampled or undersampled frame to pair with the last frame we grabbed, but I would need to know how many frames were actually missed since a frame was last grabbed.
Thanks!
There is IAMDroppedFrames::GetNumDropped method in DirectShow and chances are that it can be retrieved through Media Foundation as well (never tried - they are possibly obtainable with a method similar to this).
The GetNumDropped method retrieves the total number of frames that the filter has dropped since it started streaming.
However I would question its reliability. The reason is that with these both APIs, the attribute which is more or less reliable is a time stamp of a frame. Capture devices can flexibly reduce frame rate for a few reasons, including both external like low light conditions and internal like slow blocking processing downstream in the pipeline. This makes it hard to distinguish between odd and even frames, but time stamp remains accurate and you can apply frame rate math to convert to frame indices.
In your scenario I would however rather detect large gaps in frame times to identify possible gap and continuity loss, and from there run algorithm that compares exposure on next a few consecutive frames to get back to sync with under-/overexposition. Sounds like a more reliable way out.
After all this exposure problem is highly likely to be pretty much specific to the hardware you are using.
Normally MFSampleExtension_Discontinuity is here for this. When you use IMFSourceReader::ReadSample, check this.
I am playing back audio files in a program, and in the audio rendering callbacks, I apply a gain multiplier to the input signal and add it to the output buffer. Here's some pseudo code to illustrate my actions:
void audioCallback(AudioOutputBuffer* ao, AudioInput* ai, int startSample, int numSamples){
for (int i=startSample; i<numSamples+startSample; i++){
ao[i] = ai[i]*gain;
}
}
Basically I just multiply the data by some multiplier. In this case, gain is a float member that is being adjusted via a GUI callback. If I adjust this value while the audio is still playing, I can hear that the audio is getting softer or louder when I move the slider, but I hear lots of little pops and clicks.
Not really sure what the deal is. I know about interpolation, and I do that if the audio is pitch shifted, but I'm not sure if I need to do any extra interpolation or something if the gain is being adjusted in real time before the audio file is finished playing.
If I adjust the slider before the audio start playing, the gain is set properly and I get no clicks.
Am I missing something here? How else is gain implemented but a multiplier on the input signal?
Question: how does the multiplication operator know which operand is the audio signal and which one is the gain? Answer: it doesn't. They're both audio signals, and anything audible in either one will be audible in the output.
A flat, unchanging signal doesn't produce any audible sounds. As long as the gain remains constant, it won't introduce any sound of its own.
A signal that changes abruptly will be very audible, it sounds like a click, containing lots of high frequencies.
As you've determined on your own, one way to reduce the high frequency content and thus the audibility is to stretch out the change over a number of samples, using a constant slope. This would certainly suffice in an application where you have lots of time to make the gain change.
Another way would be to run a low-pass filter on the gain signal and use that as the input to the multiplication.
I fixed it by changing the gain in increments of the amount changed. For instance, if the gain multiplier was set to 1.0, then changed to 0.8, that's a difference of 0.2 gain. For each sample in the callback, add the difference / numSamples to the previous volume to create a slurring or gradual gain change.
Is there any way to calculate how much updates should be made to reach desired frame rate, NOT system specific? I found that for windows, but I would like to know if something like this exists in openGL itself. It should be some sort of timer.
Or how else can I prevent FPS to drop or raise dramatically? For this time I'm testing it on drawing big number of vertices in line, and using fraps I can see frame rate to go from 400 to 200 fps with evident slowing down of drawing it.
You have two different ways to solve this problem:
Suppose that you have a variable called maximum_fps, which contains for the maximum number of frames you want to display.
Then You measure the amount of time spent on the last frame (a timer will do)
Now suppose that you said that you wanted a maximum of 60FPS on your application. Then you want that the time measured be no lower than 1/60. If the time measured s lower, then you call sleep() to reach the amount of time left for a frame.
Or you can have a variable called tick, that contains the current "game time" of the application. With the same timer, you will incremented it at each main loop of your application. Then, on your drawing routines you calculate the positions based on the tick var, since it contains the current time of the application.
The big advantage of option 2 is that your application will be much easier to debug, since you can play around with the tick variable, go forward and back in time whenever you want. This is a big plus.
Rule #1. Do not make update() or loop() kind of functions rely on how often it gets called.
You can't really get your desired FPS. You could try to boost it by skipping some expensive operations or slow it down by calling sleep() kind of functions. However, even with those techniques, FPS will be almost always different from the exact FPS you want.
The common way to deal with this problem is using elapsed time from previous update. For example,
// Bad
void enemy::update()
{
position.x += 10; // this enemy moving speed is totally up to FPS and you can't control it.
}
// Good
void enemy::update(elapsedTime)
{
position.x += speedX * elapsedTime; // Now, you can control its speedX and it doesn't matter how often it gets called.
}
Is there any way to calculate how much updates should be made to reach desired frame rate, NOT system specific?
No.
There is no way to precisely calculate how many updates should be called to reach desired framerate.
However, you can measure how much time has passed since last frame, calculate current framerate according to it, compare it with desired framerate, then introduce a bit of Sleeping to reduce current framerate to the desired value. Not a precise solution, but it will work.
I found that for windows, but I would like to know if something like this exists in openGL itself. It should be some sort of timer.
OpenGL is concerned only about rendering stuff, and has nothing to do with timers. Also, using windows timers isn't a good idea. Use QueryPerformanceCounter, GetTickCount or SDL_GetTicks to measure how much time has passed, and sleep to reach desired framerate.
Or how else can I prevent FPS to drop or raise dramatically?
You prevent FPS from raising by sleeping.
As for preventing FPS from dropping...
It is insanely broad topic. Let's see. It goes something like this: use Vertex buffer objects or display lists, profile application, do not use insanely big textures, do not use too much alpha-blending, avoid "RAW" OpenGL (glVertex3f), do not render invisible objects (even if no polygons are being drawn, processing them takes time), consider learning about BSPs or OCTrees for rendering complex scenes, in parametric surfaces and curves, do not needlessly use too many primitives (if you'll render a circle using one million polygons, nobody will notice the difference), disable vsync. In short - reduce to absolute possible minimum number of rendering calls, number of rendered polygons, number of rendered pixels, number of texels read, read every available performance documentation from NVidia, and you should get a performance boost.
You're asking the wrong question. Your monitor will only ever display at 60 fps (50 fps in Europe, or possibly 75 fps if you're a pro-gamer).
Instead you should be seeking to lock your fps at 60 or 30. There are OpenGL extensions that allow you to do that. However the extensions are not cross platform (luckily they are not video card specific or it'd get really scary).
windows: wglSwapIntervalEXT
x11 (linux): glXSwapIntervalSGI
max os x: ?
These extensions are closely tied to your monitor's v-sync. Once enabled calls to swap the OpenGL back-buffer will block until the monitor is ready for it. This is like putting a sleep in your code to enforce 60 fps (or 30, or 15, or some other number if you're not using a monitor which displays at 60 Hz). The difference it the "sleep" is always perfectly timed instead of an educated guess based on how long the last frame took.
You absolutely do wan't to throttle your frame-rate it all depends on what you got
going on in that rendering loop and what your application does. Especially with it's
Physics/Network related. Or if your doing any type of graphics processing with an out side toolkit (Cairo, QPainter, Skia, AGG, ...) unless you want out of sync results or 100% cpu usage.
This code may do the job, roughly.
static int redisplay_interval;
void timer(int) {
glutPostRedisplay();
glutTimerFunc(redisplay_interval, timer, 0);
}
void setFPS(int fps)
{
redisplay_interval = 1000 / fps;
glutTimerFunc(redisplay_interval, timer, 0);
}
Here is a similar question, with my answer and worked example
I also like deft_code's answer, and will be looking into adding what he suggests to my solution.
The crucial part of my answer is:
If you're thinking about slowing down AND speeding up frames, you have to think carefully about whether you mean rendering or animation frames in each case. In this example, render throttling for simple animations is combined with animation acceleration, for any cases when frames might be dropped in a potentially slow animation.
The example is for animation code that renders at the same speed regardless of whether benchmarking mode, or fixed FPS mode, is active. An animation triggered before the change even keeps a constant speed after the change.
Just looking on resources that break down how frames per second work. I know it has something to do with keeping track of Ticks and figure out how many ticks occured between each frame. But I never ran into any resources on why exactly you have to use the methods you use in order to get a smooth frame work. I am trying to get a thourough understanding of this. Can any explain or provide any good resources ? Thanks
There are basically two approaches.
In ActionScript (and many other engines), you request the player to call a certain function at a certain framerate. For Flash games, you'll set the framerate to be 30 FPS, and then you'll implement a function that listens for ENTER_FRAME events to do what you need to do. This means you get roughly 33 ms per frame (1000ms/30FPS=33.33ms/frame). If your code that responds to ENTER_FRAME takes more than 33 ms, you'll get some stuttering.
In home-rolled main loops (like you'd generally do in C++/SDL, for example), you run the main loop as fast as possible. This means the time between each frame will be variable. You still need to keep the "guts" of your frame code less than 33 ms to make sure you'll get at least 30 FPS, but your game will run faster than 30 FPS if not a lot's going on. To account for this, you need to program all your logic in terms of elapsed time since last frame, and abandon using frames themselves as a unit of time.
http://forums.xna.com/forums/t/42624.aspx
How do you separate game logic from display?
For a continously variable frame rate you can measure the time the last frame took and assume this frame will take the same length of time. This has the benefit of meaning that time runs more or less constantly. Your biggest issue with this approach is that it is entirely possible for a VSync'd game to change from 60 fps to 30 fps and back again on subsequent frames. From experience a good way to solve this is to average the last few frame times. This smooths the result out. In the 60 to 30 fps switch each frame will progress assuming 1/45 seconds and hence the 60fps frame run slow and the 30fps frame run fast and the perceived speed remains at 45fps.
Better still is to not use this sort of time step in your calculations. Set yourself a minimum fps ... say 10fps. You then calculate all your game logic at some multiple of these 1/10 second intervals. The render engine then knows where the object is and where it is heading towards and so can inter/extrapolate the object position until its next "decision" frame shows up. This has numerous advantages. It decouples your logic entirely from rendering. It allows you to spread the logic calculations over a number of frames. For example, at 60Hz, you can test at what point a logic object will interesect with the world if it maintains its path every 6th frame. This gives the bonus of allowing you to process some logic objects on different frames to spread calculation load across the time. Its biggest disadvantage is that if the frame rate drops below your target rate everything slows down.
I made a program (in C++, using gl/glut) for study purposes where you can basically run around a screen (in first person), and it has several solids around the scene. I tried to run it on a different computer and the speed was completely different, so I searched on the subject and I'm currently doing something like this:
Idle function:
start = glutGet (GLUT_ELAPSED_TIME);
double dt = (start-end)*30/1000;
<all the movement*dt>
glutPostRedisplay ();
end = glutGet (GLUT_ELAPSED_TIME);
Display function:
<rendering for all objects>
glutSwapBuffers ();
My question is: is this the proper way to do it? The scene is being displayed after the idle function right?
I tried placing end = glutGet (GLUT_ELAPSED_TIME) before glutSwapBuffers () and didn't notice any change, but when I put it after glutSwapBuffers () it slows down alot and even stops sometimes.
EDIT: I just noticed that in the way I'm thinking, end-start should end up being the time that passed since all the drawing was done and before the movement update, as idle () would be called as soon as display () ends, so is it true that the only time that's not being accounted for here is the time the computer takes to do all of the movement? (Which should be barely nothing?)
Sorry if this is too confusing..
Thanks in advance.
I don't know what "Glut" is, but as a general rule of game development, I would never base movement speed off of how fast the computer can process the directives. That's what they did in the late 80's and that's why when you play an old game, things move at light speed.
I would set up a timer, and base all of my movements off of clear and specific timed events.
Set up a high-resolution timer (eg. QueryPerformanceCounter on Windows) and measure the time between every frame. This time, called delta-time (dt), should be used in all movement calculations, eg. every frame, set an object's position to:
obj.x += 100.0f * dt; // to move 100 units every second
Since the sum of dt should always be 1 over 1 second, the above code increments x by 100 every second, no matter what the framerate is. You should do this for all values which change over time. This way your game proceeds at the same rate on all machines (framerate independent), rather than depending on the rate the computer processes the logic (framerate dependent). This is also useful if the framerate starts to drop - the game doesn't suddenly start running in slow-motion, it keeps going at the same speed, just rendering less frequently.
I wouldn't use a timer. Things can go wrong, and events can stack up if the PC is too slow or too busy to run at the required rate. I'd let the loop run as fast as it's allowed, and each time calculate how much time has passed and put this into your movement/logic calculations.
Internally, you might actually implement small fixed-time sub-steps, because trying to make everything work right on variable time-steps is not as simple as x+=v*dt.
Try gamedev.net for stuff like this. lots of articles and a busy forum.
There is a perfect article about game loops that should give you all the information you need.
You have plenty of answers on how to do it the "right" way, but you're using GLUT, and GLUT sometimes sacrifices the "right" way for simplicity and maintaining platform independence. The GLUT way is to register a timer callback function with glutTimerFunc().
static void timerCallback (int value)
{
// Calculate the deltas
glutPostRedisplay(); // Have GLUT call your display function
glutTimerFunc(elapsedMilliseconds, timerCallback, value);
}
If you set elapsedMilliseconds to 40, this function will be called slightly less than 25 times a second. That slightly less will depend upon how long the computer takes to process your delta calculation code. If you keep that code simple, your animation will run the same speed on all systems, as long as each system can process the display function in less than 40 milliseconds. For more flexibility, you can adjust the frame rate at runtime with a command line option or by adding a control to your interface.
You start the timer loop by calling glutTimerFunc(elapsedMilliseconds, timerCallback, value); in your initialization process.
I'm a games programmer and have done this many times.
Most games run the AI in fixed time increments like 60hz for example. Also most are synced to the monitor refresh to avoid screen tearing so the max rate would be 60 even if the machine was really fast and could do 1000 fps. So if the machine was slow and was running at 20 fps then it would call the update ai function 3 times per render. Doing it this way solves rounding error problems with small values and also makes the AI deterministic across multiple machines since the AI update rate is decoupled from the machine speed ( necessary for online multiplayer games).
This is a very hard question.
The first thing you need to awnser yourself is, do you really want your application to really run at the same speed or just appear to run the same speed? 99% of the time you only want it to appear to run the same speed.
Now there are two problems: Speeding up you application or slowing it down.
Speeding up your application is really hard, since that requires things like dynamic LOD that adjusts to the current speed. This means LOD in everything, not only graphics.
Slowing your application down is fairly easy. You have two options sleeping or "busy waiting". It basically depends on your target frame rate for your simulation. If your simulation is way above something like 50 ms you can sleep. The problem is that when sleeping you are depended on the process scheduler and it works on average system at granularity of 10 ms.
In games busy waiting is not such a bad idea. What you do is you update your simulation and render your frame, then you use an time accumulator for the next frame. When rendering frames without simulation you then interpolate the state to get a smooth animation. A really great article on the subject can be found at http://gafferongames.com/game-physics/fix-your-timestep/.