WinRT C++ (Win10) Accessing bytes from SoftwareBitmap / BitmapBuffer - c++

Question

To process my previewFrames of my camera in OpenCV, I need access to the raw Pixel data / bytes. So, there is the new SoftwareBitmap, which should exactly provide this.
There is an example for c#, but in visual c++ I can't get the IMemoryBufferByteAccess (see remarks) Interface working.
Code with Exceptions:
// Capture the preview frame
return create_task(_mediaCapture->GetPreviewFrameAsync(videoFrame))
.then([this](VideoFrame^ currentFrame)
{
// Collect the resulting frame
auto previewFrame = currentFrame->SoftwareBitmap;
auto buffer = previewFrame->LockBuffer(Windows::Graphics::Imaging::BitmapBufferAccessMode::ReadWrite);
auto reference = buffer->CreateReference();
// Get a pointer to the pixel buffer
byte* pData = nullptr;
UINT capacity = 0;
// Obtain ByteAccess
ComPtr<IUnknown> inspectable = reinterpret_cast<IUnknown*>(buffer);
// Query the IBufferByteAccess interface.
Microsoft::WRL::ComPtr<IMemoryBufferByteAccess> bufferByteAccess;
ThrowIfFailed(inspectable.As(&bufferByteAccess)); // ERROR ---> Throws HRESULT = E_NOINTERFACE
// Retrieve the buffer data.
ThrowIfFailed(bufferByteAccess->GetBuffer(_Out_ &pData, _Out_ &capacity)); // ERROR ---> Throws HRESULT = E_NOINTERFACE, because bufferByteAccess is null
I tried this too:
HRESULT hr = ((IMemoryBufferByteAccess*)reference)->GetBuffer(&pData, &capacity);
HRESULT is ok, but I can't access pData -> Access Violation reading Memory.
Thanks for your help.

Answer 1

You should use reference instead of buffer in reinterpret_cast.
#include "pch.h"
#include <wrl\wrappers\corewrappers.h>
#include <wrl\client.h>
MIDL_INTERFACE("5b0d3235-4dba-4d44-865e-8f1d0e4fd04d")
IMemoryBufferByteAccess : IUnknown
{
virtual HRESULT STDMETHODCALLTYPE GetBuffer(
BYTE **value,
UINT32 *capacity
);
};
auto previewFrame = currentFrame->SoftwareBitmap;
auto buffer = previewFrame->LockBuffer(BitmapBufferAccessMode::ReadWrite);
auto reference = buffer->CreateReference();
ComPtr<IMemoryBufferByteAccess> bufferByteAccess;
HRESULT result = reinterpret_cast<IInspectable*>(reference)->QueryInterface(IID_PPV_ARGS(&bufferByteAccess));
if (result == S_OK)
{
WriteLine("Get interface successfully");
BYTE* data = nullptr;
UINT32 capacity = 0;
result = bufferByteAccess->GetBuffer(&data, &capacity);
if (result == S_OK)
{
WriteLine("get data access successfully, capacity: " + capacity);
}
}

Answer 2

Based on answer from #jeffrey-chen and example from #kennykerr, I've assembled a tiny bit cleaner solution:
#include <wrl/client.h>
// other includes, as required by your project
MIDL_INTERFACE("5b0d3235-4dba-4d44-865e-8f1d0e4fd04d")
IMemoryBufferByteAccess : ::IUnknown
{
virtual HRESULT __stdcall GetBuffer(BYTE **value, UINT32 *capacity) = 0;
};
// your code:
auto previewFrame = currentFrame->SoftwareBitmap;
auto buffer = previewFrame->LockBuffer(BitmapBufferAccessMode::ReadWrite);
auto bufferByteAccess= buffer->CreateReference().as<IMemoryBufferByteAccess>();
WriteLine("Get interface successfully"); // otherwise - exception is thrown
BYTE* data = nullptr;
UINT32 capacity = 0;
winrt::check_hresult(bufferByteAccess->GetBuffer(&data, &capacity));
WriteLine("get data access successfully, capacity: " + capacity);

Answer 3

I'm currently accessing the raw unsigned char* data from each frame I obtain on a MediaFrameReader::FrameArrived event without using WRL and COM...
Here it is how:
void MainPage::OnFrameArrived(MediaFrameReader ^reader, MediaFrameArrivedEventArgs ^args)
{
MediaFrameReference ^mfr = reader->TryAcquireLatestFrame();
VideoMediaFrame ^vmf = mfr->VideoMediaFrame;
VideoFrame ^vf = vmf->GetVideoFrame();
SoftwareBitmap ^sb = vf->SoftwareBitmap;
Buffer ^buff = ref new Buffer(sb->PixelHeight * sb->PixelWidth * 2);
sb->CopyToBuffer(buff);
DataReader ^dataReader = DataReader::FromBuffer(buffer);
Platform::Array<unsigned char, 1> ^arr = ref new Platform::Array<unsigned char, 1>(buffer->Length);
dataReader->ReadBytes(arr);
// here arr->Data is a pointer to the raw pixel data
}
NOTE: The MediaCapture object needs to be configured with MediaCaptureMemoryPreference::Cpu in order to have a valid SoftwareBitmap
Hope the above helps someone