I'm trying to convert an AVFrame from a JPEG (YUV pixel format) to an RGB24 format using ffmpeg's sws_scale function. I set up the SwsContext as follows:
struct SwsContext *sws_ctx = NULL;
int frameFinished;
AVPacket packet;
// initialize SWS context for software scaling
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
AV_PIX_FMT_RGB24,
SWS_BICUBIC,
NULL, NULL, NULL
);
And then, I perform the sws_scale, with the following command
sws_scale(sws_ctx,
(uint8_t const * const *)pFrame->data,
pFrame->linesize,
0,
pCodecCtx->height,
pFrameRGB->data,
pFrameRGB->linesize);
which gives me a segfault, though I'm not sure why. I've tried examining the values through prints and the heights and linesizes and everything all appear to have valid values.
For anyone who comes on this in the future, my issues was that I was not properly initalizing pFrame and pFrameRGB. The memory first has to be allocated for the frame struct using av_frame_alloc(), then the data buffers must be allocated with av_image_alloc().
Related
I've made a program that takes a video file as input, edits it using opengl/glfw, then encodes that edited video. The program works just fine, I get the desired output. However the video quality is really low and I don't know how to adjust it. The editing seems fine, since the display on the glfw window is high resolution. I don'T think its about scaling since it just reads the pixels on the glfw window and passes it to the encoder, and the glfw window is high res.
Here is what the glfw window looks like when the program is running:
I'm encoding in YUV420P formatting, but the information I'm getting from the glfw window is in RGBA format. I'm getting the data using:
glReadPixels(0, 0,
gl_width, gl_height,
GL_RGBA, GL_UNSIGNED_BYTE,
(GLvoid*) state.glBuffer
);
I simply got the muxing.c example from ffmpeg's docs and edited it slightly so it looks something like this:
AVFrame* video_encoder::get_video_frame(OutputStream *ost)
{
AVCodecContext *c = ost->enc;
/* check if we want to generate more frames */
if (av_compare_ts(ost->next_pts, c->time_base,
(float) STREAM_DURATION / 1000, (AVRational){ 1, 1 }) > 0)
return NULL;
/* when we pass a frame to the encoder, it may keep a reference to it
* internally; make sure we do not overwrite it here */
if (av_frame_make_writable(ost->frame) < 0)
exit(1);
if (c->pix_fmt != AV_PIX_FMT_YUV420P) {
/* as we only generate a YUV420P picture, we must convert it
* to the codec pixel format if needed */
if (!ost->sws_ctx) {
ost->sws_ctx = sws_getContext(c->width, c->height,
AV_PIX_FMT_YUV420P,
c->width, c->height,
c->pix_fmt,
SCALE_FLAGS, NULL, NULL, NULL);
if (!ost->sws_ctx) {
fprintf(stderr,
"Could not initialize the conversion context\n");
exit(1);
}
}
#if __AUDIO_ONLY
image_for_audio_only(ost->tmp_frame, ost->next_pts, c->width, c->height);
#endif
sws_scale(ost->sws_ctx, (const uint8_t * const *) ost->tmp_frame->data,
ost->tmp_frame->linesize, 0, c->height, ost->frame->data,
ost->frame->linesize);
} else {
//This is where I set the information I got from the glfw window.
set_frame_yuv_from_rgb(ost->frame, ost->sws_ctx);
}
ost->frame->pts = ost->next_pts++;
return ost->frame;
}
void video_encoder::set_frame_yuv_from_rgb(AVFrame *frame, struct SwsContext *sws_context) {
const int in_linesize[1] = { 4 * width };
//uint8_t* dest[4] = { rgb_data, NULL, NULL, NULL };
sws_context = sws_getContext(
width, height, AV_PIX_FMT_RGBA,
width, height, AV_PIX_FMT_YUV420P,
SWS_BICUBIC, 0, 0, 0);
sws_scale(sws_context, (const uint8_t * const *)&rgb_data, in_linesize, 0,
height, frame->data, frame->linesize);
}
rgb_data is the buffer I got from the glfw window. It's simply an uint8_t*.
And at the end of all this, here is what the encoded output looks like when ran through mplayer:
It's much lower quality compare to the glfw window. How can I improve the quality of the video?
Here are encoding settings from youtube for a better quality:
https://support.google.com/youtube/answer/1722171
Make sure to have high bitrate and gop size. E.g. 5Mbps and 60 correspondingly.
I'm decoding video using FFMpeg, and want to edit the decoded frames using OpenGL, but in order to do that I need to convert the data in AVFrame from YUV to RGB.
In order to do that I create a new AVFrame:
AVFrame *inputFrame = av_frame_alloc();
AVFrame *outputFrame = av_frame_alloc();
av_image_alloc(outputFrame->data, outputFrame->linesize, width, height, AV_PIX_FMT_RGB24, 1);
av_image_fill_arrays(outputFrame->data, outputFrame->linesize, NULL, AV_PIX_FMT_RGB24, width, height, 1);
Create a conversion context:
struct SwsContext *img_convert_ctx = sws_getContext(width, height, AV_PIX_FMT_YUV420P,
width, height, AV_PIX_FMT_RGB24,
0, NULL, NULL, NULL);
And then try to convert it to RGB:
sws_scale(img_convert_ctx, (const uint8_t *const *)&inputFrame->data, inputFrame->linesize, 0, inputFrame->height, outputFrame->data, outputFrame->linesize);
But this causes an "[swscaler # 0x123f15000] bad dst image pointers" error during run time. When I went over FFMpeg's source I found out that the reason is that outputFrame's data wasn't initialized, but I don't understand how it should be.
All existing answers or tutorials that I found (see example) seem to use deprecated APIs, and it's unclear how to use the new APIs. I'd appreciate any help.
Here's how I call sws_scale:
image buf2((buf.w + 15)/16*16, buf.h, 3);
sws_scale(sws_ctx, (const uint8_t * const *)frame->data, frame->linesize, 0, c->height, (uint8_t * const *)buf2.c, &buf2.ys);
There are two differences here:
You pass &inputFrame->data but it shall be inputFrame->data without the address-of operator.
You don't have to allocate a second frame structure. The sws_scale doesn't care about it. It just needs a chunk of memory of the proper size (and maybe alignment).
In my case the av_image_alloc / av_image_fill_arrays did not create the frame->data pointers.
Here is how I did it, not sure if everything is correct, but it works:
d->m_FrameCopy = av_frame_alloc();
uint8_t* buffer = NULL;
int numBytes;
// Determine required buffer size and allocate buffer
numBytes = avpicture_get_size(
AV_PIX_FMT_RGB24, d->m_Frame->width, d->m_Frame->height);
buffer = (uint8_t*)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill(
(AVPicture*)d->m_FrameCopy,
buffer,
AV_PIX_FMT_RGB24,
d->m_Frame->width,
d->m_Frame->height);
d->m_FrameCopy->format = AV_PIX_FMT_RGB24;
d->m_FrameCopy->width = d->m_Frame->width;
d->m_FrameCopy->height = d->m_Frame->height;
d->m_FrameCopy->channels = d->m_Frame->channels;
d->m_FrameCopy->channel_layout = d->m_Frame->channel_layout;
d->m_FrameCopy->nb_samples = d->m_Frame->nb_samples;
I want to make use of hardware acceleration for decoding an h264 encoded MP4 file.
My computing environment:
Hardware: MacPro (2015 model)
Software: FFmpeg (installed by brew)
Here is the output of FFmpeg command:
$ffmpeg -hwaccels
Hardware acceleration methods:
vda
videotoolbox
According to this document, there are two options for my environment, that is, VDA and VideoToolBox. I tried VDA in C++:
Codec = avcodec_find_decoder_by_name("h264_vda");
It kind of worked, but the output of the pixel format is UYVY422 which I have trouble to deal with (any suggestion on how to render UYVY422 in C++? The ideal format is yuv420p)
So I want to try VideotoolBox, but there is no such simple thing like (it may work in the case of encoding though)
Codec = avcodec_find_decoder_by_name("h264_videotoolbox");
It seems I should use AVHWAccel, but what is AVHWAccel and how to use it?
Part of My C++ code:
for( unsigned int i = 0; i < pFormatCtx->nb_streams; i++ ){
if(pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO){
pCodecCtx = pFormatCtx->streams[i]->codec;
video_stream = pFormatCtx->streams[i];
if( pCodecCtx->codec_id == AV_CODEC_ID_H264 ){
//pCodec = avcodec_find_decoder(pCodecCtx->codec_id);
pCodec = avcodec_find_decoder_by_name("h264_vda");
break;
}
}
}
// open codec
if( pCodec ){
if((ret=avcodec_open2(pCodecCtx, pCodec, NULL)) < 0) {
....
It's nothing to do with the decoder for which pix format to choose.
Your video pix format is UYVY422, so you got this format after you decode the frame.
Like the answer #halfelf mentioned, you can perform a swscale after you decode a frame, to convert the pix format to your ideal format yuv420p, then render it.
Meanwhile, if you are sure it's the format UYVY422, SDL2 can handle the render directly for you.
In the example below, my format is yuv420p, and I use swscale to convert to UYVY422 to render to SDL2
// prepare swscale context, AV_PIX_FMT_UYVY422 is my destination pix format
SwsContext *swsCtx = sws_getContext(codecCtx->width, codecCtx->height, codecCtx->pix_fmt,
codecCtx->width, codecCtx->height, AV_PIX_FMT_UYVY422,
SWS_FAST_BILINEAR, NULL, NULL, NULL);
SDL_Init(SDL_INIT_EVERYTHING);
SDL_Window *window;
SDL_Renderer *render;
SDL_Texture *texture;
SDL_CreateWindowAndRenderer(codecCtx->width,
codecCtx->height, SDL_WINDOW_OPENGL, &window, &render);
texture = SDL_CreateTexture(render, SDL_PIXELFORMAT_UYVY, SDL_TEXTUREACCESS_STREAMING,
codecCtx->width, codecCtx->height);
// ......
// decode the frame
// ......
AVFrame *frameUYVY = av_frame_alloc();
av_image_alloc(frameUYVY->data, frameUYVY->linesize, codecCtx->width, codecCtx->height, AV_PIX_FMT_UYVY422, 32);
SDL_LockTexture(texture, NULL, (void **)frameUYVY->data, frameUYVY->linesize);
// convert the decoded frame to destination frameUYVY (yuv420p -> uyvy422)
sws_scale(swsCtx, frame->data, frame->linesize, 0, frame->height,
frameUYVY->data, frameUYVY->linesize);
SDL_UnlockTexture(texture);
// performa render
SDL_RenderClear(render);
SDL_RenderCopy(render, texture, NULL, NULL);
SDL_RenderPresent(render);
In your example, if your pix format is uyvy422, you can skip the swscale part, and perform the render directly after decode from ffmpeg.
Decoders won't choose which pixel format the output is, it is determined by the video itself. swscale lib is used to convert one pixel format to another.
auto sws_ctx = sws_getContext(src_width, src_height, AV_PIX_FMT_UYUV422, dst_width, dst_height, AV_PIX_FMT_YUV420P, 0,0,0,0);
av_image_alloc(new_data, new_linesize, dst_width, dst_height, AV_PIX_FMT_BGR24, FRAME_ALIGN);
sws_scale(sws_ctx, frame->data, frame->linesize, 0, src_height, new_data, new_linesize);
And there is no h264_videotoolbox decoder, only encoder. To list decoders/encoders available:
ffmpeg -encoders
ffmpeg -decoders
The decoder/encoder names is written in the source, for example, at the end of libavcodec/vda_h264_dec.c and libavcodec/videotoolboxenc.c.
I'm trying to decode a video file using ffmpeg, grab the AVFrame object, convert it to opencv mat object, do some processing then convert it back to AVFrame object and encode it back to a video file.
Well, the program can run, but it produces bad result.
I Keep getting errors like "top block unavailable for requested intra mode at 7 19", "error while decoding MB 7 19, bytestream 358", "concealing 294 DC, 294AC, 294 MV errors in P frame" etc.
And the result video got glithes all over it. like this,
I'm guessing it's because my AVFrame to Mat and Mat to AVFrame methods, and here they are
//unspecified function
temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);
void CoreProcessor::Mat2AVFrame(cv::Mat **input, AVFrame *output)
{
//create a AVPicture frame from the opencv Mat input image
avpicture_fill((AVPicture *)temp_rgb_frame,
(uint8_t *)(*input)->data,
AV_PIX_FMT_RGB24,
(*input)->cols,
(*input)->rows);
//convert the frame to the color space and pixel format specified in the sws context
sws_scale(
rgb_to_yuv_context,
temp_rgb_frame->data,
temp_rgb_frame->linesize,
0, height,
((AVPicture *)output)->data,
((AVPicture *)output)->linesize);
(*input)->release();
}
void CoreProcessor::AVFrame2Mat(AVFrame *pFrame, cv::Mat **mat)
{
sws_scale(
yuv_to_rgb_context,
((AVPicture*)pFrame)->data,
((AVPicture*)pFrame)->linesize,
0, height,
((AVPicture *)temp_rgb_frame)->data,
((AVPicture *)temp_rgb_frame)->linesize);
*mat = new cv::Mat(pFrame->height, pFrame->width, CV_8UC3, temp_rgb_frame->data[0]);
}
void CoreProcessor::process_frame(AVFrame *pFrame)
{
cv::Mat *mat = NULL;
AVFrame2Mat(pFrame, &mat);
Mat2AVFrame(&mat, pFrame);
}
Am I doing something wrong with the memory? Because if I remove the processing part, just decode and then encode the frame, the result is correct.
Well, it turns out I made a mistake at the initialization of temp_rgb_frame,if should be like this,
temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);
I'd like to convert a HBitmap to a video stream using libavcodec.
I get my HBitmap using:
HBITMAP hCaptureBitmap =CreateCompatibleBitmap(hDesktopDC, nScreenWidth, nScreenHeight);
SelectObject(hCaptureDC,hCaptureBitmap);
BitBlt(hCaptureDC,0,0,nScreenWidth,nScreenHeight,hDesktopDC,0,0,SRCCOPY);
And I'd like to convert it to YUV (which is required by the codec i'm using). For that I use:
SwsContext *fooContext = sws_getContext(c->width,c->height,PIX_FMT_BGR32, c->width,c->height,PIX_FMT_YUV420P,SWS_FAST_BILINEAR,NULL,NULL,NULL);
uint8_t *movie_dib_bits = reinterpret_cast<uint8_t *>(bm.bmBits) + bm.bmWidthBytes * (bm.bmHeight - 1);
int dibrowbytes = -bm.bmWidthBytes;
uint8_t* data_out[1];
int stride_out[1];
data_out[0] = movie_dib_bits;
stride_out[0] = dibrowbytes;
sws_scale(fooContext,data_out,stride_out,0,c->height,picture->data,picture->linesize);
But this is not working at all... Any idea why ? Or how could I do it differently ?
Thank you !
I am not familiar with the stuff you are using to get the bitmap, but assuming it is correct and you have a pointer to the BGR 32-bit/pixel data, try something like this:
uint8_t* inbuffer;
int in_width, in_height, out_width, out_height;
//here, make sure inbuffer points to the input BGR32 data,
//and the input and output dimensions are set correctly.
//calculate the bytes needed for the output image
int nbytes = avpicture_get_size(PIX_FMT_YUV420P, out_width, out_height);
//create buffer for the output image
uint8_t* outbuffer = (uint8_t*)av_malloc(nbytes);
//create ffmpeg frame structures. These do not allocate space for image data,
//just the pointers and other information about the image.
AVFrame* inpic = avcodec_alloc_frame();
AVFrame* outpic = avcodec_alloc_frame();
//this will set the pointers in the frame structures to the right points in
//the input and output buffers.
avpicture_fill((AVPicture*)inpic, inbuffer, PIX_FMT_BGR32, in_width, in_height);
avpicture_fill((AVPicture*)outpic, outbuffer, PIX_FMT_YUV420P, out_width, out_height);
//create the conversion context
SwsContext* fooContext = sws_getContext(in_width, in_height, PIX_FMT_BGR32, out_width, out_height, PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);
//perform the conversion
sws_scale(fooContext, inpic->data, inpic->linesize, 0, in_height, outpic->data, outpic->linesize);
//encode the frame here...
//free memory
av_free(outbuffer);
av_free(inpic);
av_free(outpic);
Of course, if you are going to be converting a sequence of frames, just make your allocations once at the beginning and deallocations once at the end.