ID2D1Bitmap to Wic with Alpha but not PNG - c++

I 'm saving the bitmap of a Direct2D Device Context. My goal is to keep 32-bit (RGB with alpha) but not have a PNG, I 'd rather have a 32-bit bitmap.
I'm using this function:
HRESULT SaveBitmapToStream(
_In_ CComPtr<ID2D1Bitmap1> d2dBitmap,
_In_ CComPtr<IWICImagingFactory2> wicFactory2,
_In_ CComPtr<ID2D1DeviceContext> d2dContext,
_In_ REFGUID wicFormat,
_In_ IStream* stream
)
{
// Create and initialize WIC Bitmap Encoder.
CComPtr<IWICBitmapEncoder> wicBitmapEncoder;
auto hr =
wicFactory2->CreateEncoder(
wicFormat,
nullptr, // No preferred codec vendor.
&wicBitmapEncoder
);
hr =
wicBitmapEncoder->Initialize(
stream,
WICBitmapEncoderNoCache
);
// Create and initialize WIC Frame Encoder.
CComPtr<IWICBitmapFrameEncode> wicFrameEncode;
hr =
wicBitmapEncoder->CreateNewFrame(
&wicFrameEncode,
nullptr // No encoder options.
);
if (FAILED(hr))
return hr;
hr =
wicFrameEncode->Initialize(nullptr);
if (FAILED(hr))
return hr;
// Retrieve D2D Device.
CComPtr<ID2D1Device> d2dDevice;
if (!d2dContext)
return E_FAIL;
d2dContext->GetDevice(&d2dDevice);
// Create IWICImageEncoder.
CComPtr<IWICImageEncoder> imageEncoder;
hr =
wicFactory2->CreateImageEncoder(
d2dDevice,
&imageEncoder
);
if (FAILED(hr))
return hr;
hr =
imageEncoder->WriteFrame(
d2dBitmap, wicFrameEncode,
nullptr // Use default WICImageParameter options.
);
if (FAILED(hr))
return hr;
hr = wicFrameEncode->Commit();
if (FAILED(hr))
return hr;
hr = wicBitmapEncoder->Commit();
if (FAILED(hr))
return hr;
// Flush all memory buffers to the next-level storage object.
hr =
stream->Commit(STGC_DEFAULT);
return hr;
}
The problem is that, when I pass GUID_ContainerFormatBmp, the resulting bitmap does not have alpha. I must put GUID_ContainerFormatPng, but this will compress my image which I do not want, this is for video rendering and I don't want any compression.
Is there a way to get a capture of the Direct2D Context in a 32-bit format but not a compressed one?

The native BMP codec supports one property, EnableV5Header32bppBGRA of type VT_BOOL:
Specifies whether to allow encoding data in the
GUID_WICPixelFormat32bppBGRA pixel format. If this option is set to
VARIANT_TRUE, the BMP will be written out with a BITMAPV5HEADER
header. The default value is VARIANT_FALSE.
Note for 16-bit and 32-bit Windows BMP files, the BMP codec ignores
any alpha channel, as many legacy image files contain invalid data in
this extra channel. Starting with Windows 8, 32-bit Windows BMP files
written using the BITMAPV5HEADER with valid alpha channel content are
read as WICPixelFormat32bppBGRA

Related

Playback speed of video encoded with IMFSinkWriter changes based on width

I'm making a screen recorder (without audio) using Win32s Sink Writer to encode a series of bitmaps into an MP4 file.
For some reason, the video playback speed increases (seemingly) proportionally with the video width.
From this post, I've gathered that it's most likely because I'm calculating the buffer size incorrectly. The difference here is that their video playback issue was fixed once the calculation for the audio buffer size was correct, but since I don't encode any audio at all, I'm not sure what to take from it.
I've also tried to read about how the buffer works, but I'm really at a loss as to exactly how the buffer size is causing different playback speeds.
Here is a pastebin for the entirity of the code, I really can't track down the problem any more than the buffer size and/or the frame index/duration.
i.e.:
Depending on the width of the member variable m_width (measured in pixels), the playback speed changes. That is; the higher the width, the faster the video plays, and vice versa.
Here are two video examples:
3840x1080 and 640x1080, notice the system clock.
Imugr does not retain the original resolution of the files, but I double checked before uploading, and the program does indeed create files of the claimed resolutions.
rtStart and rtDuration are defined as such, and are both private members of the MP4File class.
LONGLONG rtStart = 0;
UINT64 rtDuration;
MFFrameRateToAverageTimePerFrame(m_FPS, 1, &rtDuration);
This is where rtStart is updated, and the individual bits of the bitmap is passed to the frame writer.
Moved the LPVOID object to private members to hopefully increase performance. Now there's no need for heap allocation every time a frame is appended.
HRESULT MP4File::AppendFrame(HBITMAP frame)
{
HRESULT hr = NULL;
if (m_isInitialFrame)
{
hr = InitializeMovieCreation();
if (FAILED(hr))
return hr;
m_isInitialFrame = false;
}
if (m_hHeap && m_lpBitsBuffer) // Makes sure buffer is initialized
{
BITMAPINFO bmpInfo;
bmpInfo.bmiHeader.biBitCount = 0;
bmpInfo.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
// Get individual bits from bitmap and loads it into the buffer used by `WriteFrame`
GetDIBits(m_hDC, frame, 0, 0, NULL, &bmpInfo, DIB_RGB_COLORS);
bmpInfo.bmiHeader.biCompression = BI_RGB;
GetDIBits(m_hDC, frame, 0, bmpInfo.bmiHeader.biHeight, m_lpBitsBuffer, &bmpInfo, DIB_RGB_COLORS);
hr = WriteFrame();
if (SUCCEEDED(hr))
{
rtStart += rtDuration;
}
}
return m_writeFrameResult = hr;
}
And lastly, the frame writer which actually loads the bits into the buffer, and then writes to the Sink Writer.
HRESULT MP4File::WriteFrame()
{
IMFSample *pSample = NULL;
IMFMediaBuffer *pBuffer = NULL;
const LONG cbWidth = 4 * m_width;
const DWORD cbBufferSize = cbWidth * m_height;
BYTE *pData = NULL;
// Create a new memory buffer.
HRESULT hr = MFCreateMemoryBuffer(cbBufferSize, &pBuffer);
// Lock the buffer and copy the video frame to the buffer.
if (SUCCEEDED(hr))
{
hr = pBuffer->Lock(&pData, NULL, NULL);
}
if (SUCCEEDED(hr))
{
hr = MFCopyImage(
pData, // Destination buffer.
cbWidth, // Destination stride.
(BYTE*)m_lpBitsBuffer, // First row in source image.
cbWidth, // Source stride.
cbWidth, // Image width in bytes.
m_height // Image height in pixels.
);
}
if (pBuffer)
{
pBuffer->Unlock();
}
// Set the data length of the buffer.
if (SUCCEEDED(hr))
{
hr = pBuffer->SetCurrentLength(cbBufferSize);
}
// Create a media sample and add the buffer to the sample.
if (SUCCEEDED(hr))
{
hr = MFCreateSample(&pSample);
}
if (SUCCEEDED(hr))
{
hr = pSample->AddBuffer(pBuffer);
}
// Set the time stamp and the duration.
if (SUCCEEDED(hr))
{
hr = pSample->SetSampleTime(rtStart);
}
if (SUCCEEDED(hr))
{
hr = pSample->SetSampleDuration(rtDuration);
}
// Send the sample to the Sink Writer and update the timestamp
if (SUCCEEDED(hr))
{
hr = m_pSinkWriter->WriteSample(m_streamIndex, pSample);
}
SafeRelease(&pSample);
SafeRelease(&pBuffer);
return hr;
}
A couple details about the encoding:
Framerate: 30FPS
Bitrate: 15 000 000
Output encoding format: H264 (MP4)
To me, this behavior makes sense.
See https://github.com/mofo7777/Stackoverflow/tree/master/ScreenCaptureEncode
My program uses DirectX9 instead of GetDIBits, but the behaviour is the same. Try this program with different screen resolutions, to confirm this behaviour.
And I confirm, with my program, the video playback speed increases proportionally with the video width (and also with the video height).
Why ?
More data to copy, more time to pass. And wrong sample time/sample duration.
Using 30 FPS, means one frame each 33.3333333 ms :
Do GetDIBits, MFCopyImage, WriteSample end exactly at 33.3333333 ms... no.
Do you write each frame exactly at 33.3333333 ms... no.
So just doing rtStart += rtDuration is wrong, because you don't capture and write screen exactly at this time. And GetDIBits/DirectX9 are not able to process at 30 FPS, trust me. And why Microsoft provided Windows Desktop Duplication (for windows 8/10 only) ?
The key is latency.
Do you know how long GetDIBits, MFCopyImage and WriteSample take ? You should know, to understand the problem. Usually, it takes more than 33.3333333 ms. But it is variable.
You must know it to adjust the correct FPS to the encoder. But you also will need to WriteSample at the right time.
If you use MF_MT_FRAME_RATE with 5-10 FPS instead of 30 FPS, You will see it is more realistic, but not optimal.
For example, use an IMFPresentationClock to handle the correct WriteSample time.

WICConvertBitmapSource + CopyPixels results in blue image

I'm trying to use WIC to load an image into an in-memory buffer for further processing then write it back to a file when done. Specifically:
Load the image into an IWICBitmapFrameDecode.
The loaded IWICBitmapFrameDecode reports that its pixel format is GUID_WICPixelFormat24bppBGR. I want to work in 32bpp RGBA, so I call WICConvertBitmapSource.
Call CopyPixels on the converted frame to get a memory buffer.
Write the memory buffer back into an IWICBitmapFrameEncode using WritePixels.
This results in a recognizable image, but the resulting image is mostly blueish, as if the red channel is being interpreted as blue.
If I call WriteSource to write the converted frame directly, instead of writing the memory buffer, it works. If I call CopyPixels from the original unconverted frame (and update my stride and pixel formats accordingly), it works. It's only the combination of WICConvertBitmapSource plus the use of a memory buffer (CopyPixels + WritePixels) that causes the problem, but I can't figure out what I'm doing wrong.
Here's my code.
int main() {
IWICImagingFactory *pFactory;
IWICBitmapDecoder *pDecoder = NULL;
CoInitializeEx(NULL, COINIT_MULTITHREADED);
CoCreateInstance(
CLSID_WICImagingFactory,
NULL,
CLSCTX_INPROC_SERVER,
IID_IWICImagingFactory,
(LPVOID*)&pFactory
);
// Load the image.
pFactory->CreateDecoderFromFilename(L"input.png", NULL, GENERIC_READ, WICDecodeMetadataCacheOnDemand, &pDecoder);
IWICBitmapFrameDecode *pFrame = NULL;
pDecoder->GetFrame(0, &pFrame);
// pFrame->GetPixelFormat shows that the image is 24bpp BGR.
// Convert to 32bpp RGBA for easier processing.
IWICBitmapSource *pConvertedFrame = NULL;
WICConvertBitmapSource(GUID_WICPixelFormat32bppRGBA, pFrame, &pConvertedFrame);
// Copy the 32bpp RGBA image to a buffer for further processing.
UINT width, height;
pConvertedFrame->GetSize(&width, &height);
const unsigned bytesPerPixel = 4;
const unsigned stride = width * bytesPerPixel;
const unsigned bitmapSize = width * height * bytesPerPixel;
BYTE *buffer = new BYTE[bitmapSize];
pConvertedFrame->CopyPixels(nullptr, stride, bitmapSize, buffer);
// Insert image buffer processing here. (Not currently implemented.)
// Create an encoder to turn the buffer back into an image file.
IWICBitmapEncoder *pEncoder = NULL;
pFactory->CreateEncoder(GUID_ContainerFormatPng, nullptr, &pEncoder);
IStream *pStream = NULL;
SHCreateStreamOnFileEx(L"output.png", STGM_WRITE | STGM_CREATE, FILE_ATTRIBUTE_NORMAL, true, NULL, &pStream);
pEncoder->Initialize(pStream, WICBitmapEncoderNoCache);
IWICBitmapFrameEncode *pFrameEncode = NULL;
pEncoder->CreateNewFrame(&pFrameEncode, NULL);
pFrameEncode->Initialize(NULL);
WICPixelFormatGUID pixelFormat = GUID_WICPixelFormat32bppRGBA;
pFrameEncode->SetPixelFormat(&pixelFormat);
pFrameEncode->SetSize(width, height);
pFrameEncode->WritePixels(height, stride, bitmapSize, buffer);
pFrameEncode->Commit();
pEncoder->Commit();
pStream->Commit(STGC_DEFAULT);
return 0;
}
The PNG encoder only supports GUID_WICPixelFormat32bppBGRA (BGR) for 32bpp as specified in PNG Native Codec official documentation. When you call it with GUID_WICPixelFormat32bppRGBA, it will not do channel switching. The pervert will just use your pixels as they were BGR, not RGB, and will not tell you there's a problem.
I don't know what you're trying to do, but in your example, you could just replace GUID_WICPixelFormat32bppRGBA by GUID_WICPixelFormat32bppBGRA in the call to WICConvertBitmapSource (and also replace the definition of the last pixelFormat variable to make sure your source code is correct, but it doesn't change anything).
PS: you can use Wic to save files, not need to create stream using another API, see my answer here: Capture screen using DirectX

GDI+ - converting bitmap pixels to JPG format

I have an array of a bitmap pixels. How can I convert them to JPG format and copy to another array? How to convert them back to bitmap from JPG pixels?
Checkout the Encoder CLSID function from here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms533843(v=vs.85).aspx
Modified original code from https://vctipsplusplus.wordpress.com/tag/image-conversion-gdi/:
int main()
{
// Initialize GDI+.
GdiplusStartupInput gdiplusStartupInput;
ULONG_PTR gdiplusToken;
GdiplusStartup(&gdiplusToken, &gdiplusStartupInput, NULL);
CLSID encoderClsid;
Status stat;
Image* image = new Image(L”Bird.bmp”);
// Get the CLSID of the JPEG encoder.
GetEncoderClsid(L”image/jpeg”, &encoderClsid);
stat = image->Save(L”Bird.png”, &encoderClsid, NULL);
if(stat == Ok)
printf(“Bird.png was saved successfully\n”);
else
printf(“Failure: stat = %d\n”, stat);
delete image;
GdiplusShutdown(gdiplusToken);
return 0;
}
Just change image/jpeg to whatever format you want to convert. The details are given on the MSDN link I mentioned above. Of course to work with the pixels, you'll need to convert the JPEG to BMP
You can create a memory stream using CreateStreamOnHGlobal or SHCreateMemStream, then use the GDI+ method Image.Save to save to the stream. Reverse the process to read back in.

WIC Direct2D CreateBitmapFromMemory: limitations on width and height?

CreateBitmapFromMemory executes successfully when _nWidth is equal to or less than 644.
If the value exceeds this value, the HRESULT value is -2003292276
Do limits exist on the width and height?
#include <d2d1.h>
#include <d2d1helper.h>
#include <wincodecsdk.h> // Use this for WIC Direct2D functions
void test()
{
IWICImagingFactory *m_pIWICFactory;
ID2D1Factory *m_pD2DFactory;
IWICBitmap *m_pEmbeddedBitmap;
ID2D1Bitmap *m_pD2DBitmap;
unsigned char *pImageBuffer = new unsigned char[1024*1024];
HRESULT hr = S_OK;
int _nHeight = 300;
int _nWidth = 644;
If nWidth exceeds 644, CreateBitmapFromMemory returns an Error.
//_nWidth = 648;
if (m_pIWICFactory == 0 )
{
hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE);
// Create WIC factory
hr = CoCreateInstance(
CLSID_WICImagingFactory,
NULL,
CLSCTX_INPROC_SERVER,
IID_PPV_ARGS(&m_pIWICFactory)
);
if (SUCCEEDED(hr))
{
// Create D2D factory
hr = D2D1CreateFactory( D2D1_FACTORY_TYPE_SINGLE_THREADED, &m_pD2DFactory );
}
}
hr = m_pIWICFactory->CreateBitmapFromMemory(
_nHeight, // height
_nWidth, // width
GUID_WICPixelFormat24bppRGB, // pixel format of the NEW bitmap
_nWidth*3, // calculated from width and bpp information
1024*1024, // height x width
pImageBuffer, // name of the .c array
&m_pEmbeddedBitmap // pointer to pointer to whatever an IWICBitmap is.
);
if (!SUCCEEDED(hr)) {
char *buffer = "Error in CreateBitmapFromMemory\n";
}
}
Error code is 0x88982F8C WINCODEC_ERR_INSUFFICIENTBUFFER and the reason is now obvious?
The first parameter is width, and the second is height. You have them in wrong order. All in all you provide incorrect arguments resulting in bad buffer.
Are you sure you passed in the correct pixelFormat for function CreateBitmapFromMemory? you hard code it to GUID_WICPixelFormat24bppRGB, I think this is the root cause, you should make sure this format same as the format with the source bitmap which you are copy the data from. try use the GetPixelFormat function to get the correct format instead of hard code.
There is an upper limit on the dimensions of images on the GPU.
Call GetMaximumBitmapSize on the render target.
http://msdn.microsoft.com/query/dev11.query?appId=Dev11IDEF1&l=EN-US&k=k(GetMaximumBitmapSize);k(DevLang-C%2B%2B);k(TargetOS-Windows)&rd=true
What you get back is the max pixels of either vertical or horiz.
For larger images you'd have to load them into a software render target such as a bitmap render target and then render what you want from that.

How can I render 3d graphics in a directshow source filter

I need to render a simple texture mapped model as the output of a directshow source filter. The 3d rendering doesnt need to come from Direct3D, but that would be nice. OpenGL or any other provider would be fine assuming I can fit it into the context of the DirectShow source filter.
visual studio 2008 c++
With direct3d I have found that you can call GetRenderTargetData from the d3d device to get you access to the raw image bytes that you can then copy into the source filters image buffer
Here is example code of how to get the d3d render
void CaptureRenderTarget(IDirect3DDevice9* pdev)
{
IDirect3DSurface9* pTargetSurface=NULL;
HRESULT hr=pdev->GetRenderTarget(0,&pTargetSurface);
if(SUCCEEDED(hr))
{
D3DSURFACE_DESC desc;
hr=pTargetSurface->GetDesc(&desc);
if(SUCCEEDED(hr))
{
IDirect3DTexture9* pTempTexture=NULL;
hr=pdev->CreateTexture(desc.Width,desc.Height,1,0,desc.Format,D3DPOOL_SYSTEMMEM,&pTempTexture,NULL);
if(SUCCEEDED(hr))
{
IDirect3DSurface9* pTempSurface=NULL;
hr=pTempTexture->GetSurfaceLevel(0,&pTempSurface);
if(SUCCEEDED(hr))
{
hr=pdev->GetRenderTargetData(pTargetSurface,pTempSurface);
if(SUCCEEDED(hr))
{
//D3DXSaveTextureToFile(L"Output.png",D3DXIFF_PNG,pTempTexture,NULL);
D3DLOCKED_RECT data;
hr=pTempTexture->LockRect(0, &data, NULL, 0);
if(SUCCEEDED(hr))
{
BYTE *d3dPixels = (BYTE*)data.pBits;
}
pTempTexture->UnlockRect(0);
}
pTempSurface->Release();
}
pTempTexture->Release();
}
}
pTargetSurface->Release();
}
}