Save bitmap to video (libavcodec ffmpeg) - c++

I'd like to convert a HBitmap to a video stream using libavcodec.
I get my HBitmap using:
HBITMAP hCaptureBitmap =CreateCompatibleBitmap(hDesktopDC, nScreenWidth, nScreenHeight);
SelectObject(hCaptureDC,hCaptureBitmap);
BitBlt(hCaptureDC,0,0,nScreenWidth,nScreenHeight,hDesktopDC,0,0,SRCCOPY);
And I'd like to convert it to YUV (which is required by the codec i'm using). For that I use:
SwsContext *fooContext = sws_getContext(c->width,c->height,PIX_FMT_BGR32, c->width,c->height,PIX_FMT_YUV420P,SWS_FAST_BILINEAR,NULL,NULL,NULL);
uint8_t *movie_dib_bits = reinterpret_cast<uint8_t *>(bm.bmBits) + bm.bmWidthBytes * (bm.bmHeight - 1);
int dibrowbytes = -bm.bmWidthBytes;
uint8_t* data_out[1];
int stride_out[1];
data_out[0] = movie_dib_bits;
stride_out[0] = dibrowbytes;
sws_scale(fooContext,data_out,stride_out,0,c->height,picture->data,picture->linesize);
But this is not working at all... Any idea why ? Or how could I do it differently ?
Thank you !

I am not familiar with the stuff you are using to get the bitmap, but assuming it is correct and you have a pointer to the BGR 32-bit/pixel data, try something like this:
uint8_t* inbuffer;
int in_width, in_height, out_width, out_height;
//here, make sure inbuffer points to the input BGR32 data,
//and the input and output dimensions are set correctly.
//calculate the bytes needed for the output image
int nbytes = avpicture_get_size(PIX_FMT_YUV420P, out_width, out_height);
//create buffer for the output image
uint8_t* outbuffer = (uint8_t*)av_malloc(nbytes);
//create ffmpeg frame structures. These do not allocate space for image data,
//just the pointers and other information about the image.
AVFrame* inpic = avcodec_alloc_frame();
AVFrame* outpic = avcodec_alloc_frame();
//this will set the pointers in the frame structures to the right points in
//the input and output buffers.
avpicture_fill((AVPicture*)inpic, inbuffer, PIX_FMT_BGR32, in_width, in_height);
avpicture_fill((AVPicture*)outpic, outbuffer, PIX_FMT_YUV420P, out_width, out_height);
//create the conversion context
SwsContext* fooContext = sws_getContext(in_width, in_height, PIX_FMT_BGR32, out_width, out_height, PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);
//perform the conversion
sws_scale(fooContext, inpic->data, inpic->linesize, 0, in_height, outpic->data, outpic->linesize);
//encode the frame here...
//free memory
av_free(outbuffer);
av_free(inpic);
av_free(outpic);
Of course, if you are going to be converting a sequence of frames, just make your allocations once at the beginning and deallocations once at the end.

Related

Failing to properly initialize AVFrame for sws_scale conversion

I'm decoding video using FFMpeg, and want to edit the decoded frames using OpenGL, but in order to do that I need to convert the data in AVFrame from YUV to RGB.
In order to do that I create a new AVFrame:
AVFrame *inputFrame = av_frame_alloc();
AVFrame *outputFrame = av_frame_alloc();
av_image_alloc(outputFrame->data, outputFrame->linesize, width, height, AV_PIX_FMT_RGB24, 1);
av_image_fill_arrays(outputFrame->data, outputFrame->linesize, NULL, AV_PIX_FMT_RGB24, width, height, 1);
Create a conversion context:
struct SwsContext *img_convert_ctx = sws_getContext(width, height, AV_PIX_FMT_YUV420P,
width, height, AV_PIX_FMT_RGB24,
0, NULL, NULL, NULL);
And then try to convert it to RGB:
sws_scale(img_convert_ctx, (const uint8_t *const *)&inputFrame->data, inputFrame->linesize, 0, inputFrame->height, outputFrame->data, outputFrame->linesize);
But this causes an "[swscaler # 0x123f15000] bad dst image pointers" error during run time. When I went over FFMpeg's source I found out that the reason is that outputFrame's data wasn't initialized, but I don't understand how it should be.
All existing answers or tutorials that I found (see example) seem to use deprecated APIs, and it's unclear how to use the new APIs. I'd appreciate any help.
Here's how I call sws_scale:
image buf2((buf.w + 15)/16*16, buf.h, 3);
sws_scale(sws_ctx, (const uint8_t * const *)frame->data, frame->linesize, 0, c->height, (uint8_t * const *)buf2.c, &buf2.ys);
There are two differences here:
You pass &inputFrame->data but it shall be inputFrame->data without the address-of operator.
You don't have to allocate a second frame structure. The sws_scale doesn't care about it. It just needs a chunk of memory of the proper size (and maybe alignment).
In my case the av_image_alloc / av_image_fill_arrays did not create the frame->data pointers.
Here is how I did it, not sure if everything is correct, but it works:
d->m_FrameCopy = av_frame_alloc();
uint8_t* buffer = NULL;
int numBytes;
// Determine required buffer size and allocate buffer
numBytes = avpicture_get_size(
AV_PIX_FMT_RGB24, d->m_Frame->width, d->m_Frame->height);
buffer = (uint8_t*)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill(
(AVPicture*)d->m_FrameCopy,
buffer,
AV_PIX_FMT_RGB24,
d->m_Frame->width,
d->m_Frame->height);
d->m_FrameCopy->format = AV_PIX_FMT_RGB24;
d->m_FrameCopy->width = d->m_Frame->width;
d->m_FrameCopy->height = d->m_Frame->height;
d->m_FrameCopy->channels = d->m_Frame->channels;
d->m_FrameCopy->channel_layout = d->m_Frame->channel_layout;
d->m_FrameCopy->nb_samples = d->m_Frame->nb_samples;

WICConvertBitmapSource + CopyPixels results in blue image

I'm trying to use WIC to load an image into an in-memory buffer for further processing then write it back to a file when done. Specifically:
Load the image into an IWICBitmapFrameDecode.
The loaded IWICBitmapFrameDecode reports that its pixel format is GUID_WICPixelFormat24bppBGR. I want to work in 32bpp RGBA, so I call WICConvertBitmapSource.
Call CopyPixels on the converted frame to get a memory buffer.
Write the memory buffer back into an IWICBitmapFrameEncode using WritePixels.
This results in a recognizable image, but the resulting image is mostly blueish, as if the red channel is being interpreted as blue.
If I call WriteSource to write the converted frame directly, instead of writing the memory buffer, it works. If I call CopyPixels from the original unconverted frame (and update my stride and pixel formats accordingly), it works. It's only the combination of WICConvertBitmapSource plus the use of a memory buffer (CopyPixels + WritePixels) that causes the problem, but I can't figure out what I'm doing wrong.
Here's my code.
int main() {
IWICImagingFactory *pFactory;
IWICBitmapDecoder *pDecoder = NULL;
CoInitializeEx(NULL, COINIT_MULTITHREADED);
CoCreateInstance(
CLSID_WICImagingFactory,
NULL,
CLSCTX_INPROC_SERVER,
IID_IWICImagingFactory,
(LPVOID*)&pFactory
);
// Load the image.
pFactory->CreateDecoderFromFilename(L"input.png", NULL, GENERIC_READ, WICDecodeMetadataCacheOnDemand, &pDecoder);
IWICBitmapFrameDecode *pFrame = NULL;
pDecoder->GetFrame(0, &pFrame);
// pFrame->GetPixelFormat shows that the image is 24bpp BGR.
// Convert to 32bpp RGBA for easier processing.
IWICBitmapSource *pConvertedFrame = NULL;
WICConvertBitmapSource(GUID_WICPixelFormat32bppRGBA, pFrame, &pConvertedFrame);
// Copy the 32bpp RGBA image to a buffer for further processing.
UINT width, height;
pConvertedFrame->GetSize(&width, &height);
const unsigned bytesPerPixel = 4;
const unsigned stride = width * bytesPerPixel;
const unsigned bitmapSize = width * height * bytesPerPixel;
BYTE *buffer = new BYTE[bitmapSize];
pConvertedFrame->CopyPixels(nullptr, stride, bitmapSize, buffer);
// Insert image buffer processing here. (Not currently implemented.)
// Create an encoder to turn the buffer back into an image file.
IWICBitmapEncoder *pEncoder = NULL;
pFactory->CreateEncoder(GUID_ContainerFormatPng, nullptr, &pEncoder);
IStream *pStream = NULL;
SHCreateStreamOnFileEx(L"output.png", STGM_WRITE | STGM_CREATE, FILE_ATTRIBUTE_NORMAL, true, NULL, &pStream);
pEncoder->Initialize(pStream, WICBitmapEncoderNoCache);
IWICBitmapFrameEncode *pFrameEncode = NULL;
pEncoder->CreateNewFrame(&pFrameEncode, NULL);
pFrameEncode->Initialize(NULL);
WICPixelFormatGUID pixelFormat = GUID_WICPixelFormat32bppRGBA;
pFrameEncode->SetPixelFormat(&pixelFormat);
pFrameEncode->SetSize(width, height);
pFrameEncode->WritePixels(height, stride, bitmapSize, buffer);
pFrameEncode->Commit();
pEncoder->Commit();
pStream->Commit(STGC_DEFAULT);
return 0;
}
The PNG encoder only supports GUID_WICPixelFormat32bppBGRA (BGR) for 32bpp as specified in PNG Native Codec official documentation. When you call it with GUID_WICPixelFormat32bppRGBA, it will not do channel switching. The pervert will just use your pixels as they were BGR, not RGB, and will not tell you there's a problem.
I don't know what you're trying to do, but in your example, you could just replace GUID_WICPixelFormat32bppRGBA by GUID_WICPixelFormat32bppBGRA in the call to WICConvertBitmapSource (and also replace the definition of the last pixelFormat variable to make sure your source code is correct, but it doesn't change anything).
PS: you can use Wic to save files, not need to create stream using another API, see my answer here: Capture screen using DirectX

Segmentation fault on ffmpeg sws_scale

I'm trying to convert an AVFrame from a JPEG (YUV pixel format) to an RGB24 format using ffmpeg's sws_scale function. I set up the SwsContext as follows:
struct SwsContext *sws_ctx = NULL;
int frameFinished;
AVPacket packet;
// initialize SWS context for software scaling
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
AV_PIX_FMT_RGB24,
SWS_BICUBIC,
NULL, NULL, NULL
);
And then, I perform the sws_scale, with the following command
sws_scale(sws_ctx,
(uint8_t const * const *)pFrame->data,
pFrame->linesize,
0,
pCodecCtx->height,
pFrameRGB->data,
pFrameRGB->linesize);
which gives me a segfault, though I'm not sure why. I've tried examining the values through prints and the heights and linesizes and everything all appear to have valid values.
For anyone who comes on this in the future, my issues was that I was not properly initalizing pFrame and pFrameRGB. The memory first has to be allocated for the frame struct using av_frame_alloc(), then the data buffers must be allocated with av_image_alloc().

What is AVHWAccel, and how can I use it?

I want to make use of hardware acceleration for decoding an h264 encoded MP4 file.
My computing environment:
Hardware: MacPro (2015 model)
Software: FFmpeg (installed by brew)
Here is the output of FFmpeg command:
$ffmpeg -hwaccels
Hardware acceleration methods:
vda
videotoolbox
According to this document, there are two options for my environment, that is, VDA and VideoToolBox. I tried VDA in C++:
Codec = avcodec_find_decoder_by_name("h264_vda");
It kind of worked, but the output of the pixel format is UYVY422 which I have trouble to deal with (any suggestion on how to render UYVY422 in C++? The ideal format is yuv420p)
So I want to try VideotoolBox, but there is no such simple thing like (it may work in the case of encoding though)
Codec = avcodec_find_decoder_by_name("h264_videotoolbox");
It seems I should use AVHWAccel, but what is AVHWAccel and how to use it?
Part of My C++ code:
for( unsigned int i = 0; i < pFormatCtx->nb_streams; i++ ){
if(pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO){
pCodecCtx = pFormatCtx->streams[i]->codec;
video_stream = pFormatCtx->streams[i];
if( pCodecCtx->codec_id == AV_CODEC_ID_H264 ){
//pCodec = avcodec_find_decoder(pCodecCtx->codec_id);
pCodec = avcodec_find_decoder_by_name("h264_vda");
break;
}
}
}
// open codec
if( pCodec ){
if((ret=avcodec_open2(pCodecCtx, pCodec, NULL)) < 0) {
....
It's nothing to do with the decoder for which pix format to choose.
Your video pix format is UYVY422, so you got this format after you decode the frame.
Like the answer #halfelf mentioned, you can perform a swscale after you decode a frame, to convert the pix format to your ideal format yuv420p, then render it.
Meanwhile, if you are sure it's the format UYVY422, SDL2 can handle the render directly for you.
In the example below, my format is yuv420p, and I use swscale to convert to UYVY422 to render to SDL2
// prepare swscale context, AV_PIX_FMT_UYVY422 is my destination pix format
SwsContext *swsCtx = sws_getContext(codecCtx->width, codecCtx->height, codecCtx->pix_fmt,
codecCtx->width, codecCtx->height, AV_PIX_FMT_UYVY422,
SWS_FAST_BILINEAR, NULL, NULL, NULL);
SDL_Init(SDL_INIT_EVERYTHING);
SDL_Window *window;
SDL_Renderer *render;
SDL_Texture *texture;
SDL_CreateWindowAndRenderer(codecCtx->width,
codecCtx->height, SDL_WINDOW_OPENGL, &window, &render);
texture = SDL_CreateTexture(render, SDL_PIXELFORMAT_UYVY, SDL_TEXTUREACCESS_STREAMING,
codecCtx->width, codecCtx->height);
// ......
// decode the frame
// ......
AVFrame *frameUYVY = av_frame_alloc();
av_image_alloc(frameUYVY->data, frameUYVY->linesize, codecCtx->width, codecCtx->height, AV_PIX_FMT_UYVY422, 32);
SDL_LockTexture(texture, NULL, (void **)frameUYVY->data, frameUYVY->linesize);
// convert the decoded frame to destination frameUYVY (yuv420p -> uyvy422)
sws_scale(swsCtx, frame->data, frame->linesize, 0, frame->height,
frameUYVY->data, frameUYVY->linesize);
SDL_UnlockTexture(texture);
// performa render
SDL_RenderClear(render);
SDL_RenderCopy(render, texture, NULL, NULL);
SDL_RenderPresent(render);
In your example, if your pix format is uyvy422, you can skip the swscale part, and perform the render directly after decode from ffmpeg.
Decoders won't choose which pixel format the output is, it is determined by the video itself. swscale lib is used to convert one pixel format to another.
auto sws_ctx = sws_getContext(src_width, src_height, AV_PIX_FMT_UYUV422, dst_width, dst_height, AV_PIX_FMT_YUV420P, 0,0,0,0);
av_image_alloc(new_data, new_linesize, dst_width, dst_height, AV_PIX_FMT_BGR24, FRAME_ALIGN);
sws_scale(sws_ctx, frame->data, frame->linesize, 0, src_height, new_data, new_linesize);
And there is no h264_videotoolbox decoder, only encoder. To list decoders/encoders available:
ffmpeg -encoders
ffmpeg -decoders
The decoder/encoder names is written in the source, for example, at the end of libavcodec/vda_h264_dec.c and libavcodec/videotoolboxenc.c.

Process AVFrame using opencv mat causing encoding error

I'm trying to decode a video file using ffmpeg, grab the AVFrame object, convert it to opencv mat object, do some processing then convert it back to AVFrame object and encode it back to a video file.
Well, the program can run, but it produces bad result.
I Keep getting errors like "top block unavailable for requested intra mode at 7 19", "error while decoding MB 7 19, bytestream 358", "concealing 294 DC, 294AC, 294 MV errors in P frame" etc.
And the result video got glithes all over it. like this,
I'm guessing it's because my AVFrame to Mat and Mat to AVFrame methods, and here they are
//unspecified function
temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);
void CoreProcessor::Mat2AVFrame(cv::Mat **input, AVFrame *output)
{
//create a AVPicture frame from the opencv Mat input image
avpicture_fill((AVPicture *)temp_rgb_frame,
(uint8_t *)(*input)->data,
AV_PIX_FMT_RGB24,
(*input)->cols,
(*input)->rows);
//convert the frame to the color space and pixel format specified in the sws context
sws_scale(
rgb_to_yuv_context,
temp_rgb_frame->data,
temp_rgb_frame->linesize,
0, height,
((AVPicture *)output)->data,
((AVPicture *)output)->linesize);
(*input)->release();
}
void CoreProcessor::AVFrame2Mat(AVFrame *pFrame, cv::Mat **mat)
{
sws_scale(
yuv_to_rgb_context,
((AVPicture*)pFrame)->data,
((AVPicture*)pFrame)->linesize,
0, height,
((AVPicture *)temp_rgb_frame)->data,
((AVPicture *)temp_rgb_frame)->linesize);
*mat = new cv::Mat(pFrame->height, pFrame->width, CV_8UC3, temp_rgb_frame->data[0]);
}
void CoreProcessor::process_frame(AVFrame *pFrame)
{
cv::Mat *mat = NULL;
AVFrame2Mat(pFrame, &mat);
Mat2AVFrame(&mat, pFrame);
}
Am I doing something wrong with the memory? Because if I remove the processing part, just decode and then encode the frame, the result is correct.
Well, it turns out I made a mistake at the initialization of temp_rgb_frame,if should be like this,
temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);