Playback speed of video encoded with IMFSinkWriter changes based on width - c++

Question

I'm making a screen recorder (without audio) using Win32s Sink Writer to encode a series of bitmaps into an MP4 file.
For some reason, the video playback speed increases (seemingly) proportionally with the video width.
From this post, I've gathered that it's most likely because I'm calculating the buffer size incorrectly. The difference here is that their video playback issue was fixed once the calculation for the audio buffer size was correct, but since I don't encode any audio at all, I'm not sure what to take from it.
I've also tried to read about how the buffer works, but I'm really at a loss as to exactly how the buffer size is causing different playback speeds.
Here is a pastebin for the entirity of the code, I really can't track down the problem any more than the buffer size and/or the frame index/duration.
i.e.:
Depending on the width of the member variable m_width (measured in pixels), the playback speed changes. That is; the higher the width, the faster the video plays, and vice versa.
Here are two video examples:
3840x1080 and 640x1080, notice the system clock.
Imugr does not retain the original resolution of the files, but I double checked before uploading, and the program does indeed create files of the claimed resolutions.
rtStart and rtDuration are defined as such, and are both private members of the MP4File class.
LONGLONG rtStart = 0;
UINT64 rtDuration;
MFFrameRateToAverageTimePerFrame(m_FPS, 1, &rtDuration);
This is where rtStart is updated, and the individual bits of the bitmap is passed to the frame writer.
Moved the LPVOID object to private members to hopefully increase performance. Now there's no need for heap allocation every time a frame is appended.
HRESULT MP4File::AppendFrame(HBITMAP frame)
{
HRESULT hr = NULL;
if (m_isInitialFrame)
{
hr = InitializeMovieCreation();
if (FAILED(hr))
return hr;
m_isInitialFrame = false;
}
if (m_hHeap && m_lpBitsBuffer) // Makes sure buffer is initialized
{
BITMAPINFO bmpInfo;
bmpInfo.bmiHeader.biBitCount = 0;
bmpInfo.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
// Get individual bits from bitmap and loads it into the buffer used by `WriteFrame`
GetDIBits(m_hDC, frame, 0, 0, NULL, &bmpInfo, DIB_RGB_COLORS);
bmpInfo.bmiHeader.biCompression = BI_RGB;
GetDIBits(m_hDC, frame, 0, bmpInfo.bmiHeader.biHeight, m_lpBitsBuffer, &bmpInfo, DIB_RGB_COLORS);
hr = WriteFrame();
if (SUCCEEDED(hr))
{
rtStart += rtDuration;
}
}
return m_writeFrameResult = hr;
}
And lastly, the frame writer which actually loads the bits into the buffer, and then writes to the Sink Writer.
HRESULT MP4File::WriteFrame()
{
IMFSample *pSample = NULL;
IMFMediaBuffer *pBuffer = NULL;
const LONG cbWidth = 4 * m_width;
const DWORD cbBufferSize = cbWidth * m_height;
BYTE *pData = NULL;
// Create a new memory buffer.
HRESULT hr = MFCreateMemoryBuffer(cbBufferSize, &pBuffer);
// Lock the buffer and copy the video frame to the buffer.
if (SUCCEEDED(hr))
{
hr = pBuffer->Lock(&pData, NULL, NULL);
}
if (SUCCEEDED(hr))
{
hr = MFCopyImage(
pData, // Destination buffer.
cbWidth, // Destination stride.
(BYTE*)m_lpBitsBuffer, // First row in source image.
cbWidth, // Source stride.
cbWidth, // Image width in bytes.
m_height // Image height in pixels.
);
}
if (pBuffer)
{
pBuffer->Unlock();
}
// Set the data length of the buffer.
if (SUCCEEDED(hr))
{
hr = pBuffer->SetCurrentLength(cbBufferSize);
}
// Create a media sample and add the buffer to the sample.
if (SUCCEEDED(hr))
{
hr = MFCreateSample(&pSample);
}
if (SUCCEEDED(hr))
{
hr = pSample->AddBuffer(pBuffer);
}
// Set the time stamp and the duration.
if (SUCCEEDED(hr))
{
hr = pSample->SetSampleTime(rtStart);
}
if (SUCCEEDED(hr))
{
hr = pSample->SetSampleDuration(rtDuration);
}
// Send the sample to the Sink Writer and update the timestamp
if (SUCCEEDED(hr))
{
hr = m_pSinkWriter->WriteSample(m_streamIndex, pSample);
}
SafeRelease(&pSample);
SafeRelease(&pBuffer);
return hr;
}
A couple details about the encoding:
Framerate: 30FPS
Bitrate: 15 000 000
Output encoding format: H264 (MP4)

Answer 1

To me, this behavior makes sense.
See https://github.com/mofo7777/Stackoverflow/tree/master/ScreenCaptureEncode
My program uses DirectX9 instead of GetDIBits, but the behaviour is the same. Try this program with different screen resolutions, to confirm this behaviour.
And I confirm, with my program, the video playback speed increases proportionally with the video width (and also with the video height).
Why ?
More data to copy, more time to pass. And wrong sample time/sample duration.
Using 30 FPS, means one frame each 33.3333333 ms :
Do GetDIBits, MFCopyImage, WriteSample end exactly at 33.3333333 ms... no.
Do you write each frame exactly at 33.3333333 ms... no.
So just doing rtStart += rtDuration is wrong, because you don't capture and write screen exactly at this time. And GetDIBits/DirectX9 are not able to process at 30 FPS, trust me. And why Microsoft provided Windows Desktop Duplication (for windows 8/10 only) ?
The key is latency.
Do you know how long GetDIBits, MFCopyImage and WriteSample take ? You should know, to understand the problem. Usually, it takes more than 33.3333333 ms. But it is variable.
You must know it to adjust the correct FPS to the encoder. But you also will need to WriteSample at the right time.
If you use MF_MT_FRAME_RATE with 5-10 FPS instead of 30 FPS, You will see it is more realistic, but not optimal.
For example, use an IMFPresentationClock to handle the correct WriteSample time.