Memory leak when using av_frame_get_buffer() - c++

I am making a simple video player with ffmpeg. I have noticed that there is a memory leak originating in libavutil. Because ffmpeg is a mature library I assume that I am allocating a new frame incorrectly. The documentation is also vague about freeing the buffer that is created when you call av_frame_get_buffer(). Below is the code I am using to decode the video and queue it up for display on the UI thread.
DWORD WINAPI DecoderThread(LPVOID lpParam)
{
AVFrame *frame = NULL;
AVPacket pkt;
SwsContext *swsCtx = NULL;
UINT8 *buffer = NULL;
INT iNumBytes = 0;
INT result = 0;
frame = av_frame_alloc();
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;
// Create scaling context
swsCtx = sws_getContext(codecCtx->width, codecCtx->height, codecCtx->pix_fmt, codecCtx->width, codecCtx->height, AV_PIX_FMT_BGR24, SWS_BICUBIC, NULL, NULL, NULL);
while (av_read_frame(fmtCtx, &pkt) >= 0) {
if (pkt.stream_index == videoStream) {
result = avcodec_send_packet(codecCtx, &pkt);
while (result >= 0) {
result = avcodec_receive_frame(codecCtx, frame);
if (result == AVERROR(EAGAIN) || result == AVERROR_EOF) {
break;
} else if (result < 0) {
// another error.
}
// Create a new frame to store the RGB24 data.
AVFrame *pFrameRGB = av_frame_alloc();
// Allocate space for the new RGB image.
//av_image_alloc(pFrameRGB->data, pFrameRGB->linesize, codecCtx->width, codecCtx->height, AV_PIX_FMT_BGR24, 1);
// Copy all of the properties from the YUV420P frame.
av_frame_copy_props(pFrameRGB, frame);
pFrameRGB->width = frame->width;
pFrameRGB->height = frame->height;
pFrameRGB->format = AV_PIX_FMT_BGR24;
av_frame_get_buffer(pFrameRGB, 0);
// Convert fram from YUV420P to BGR24 for display.
sws_scale(swsCtx, (const UINT8* const *) frame->data, frame->linesize, 0, codecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
// Queue thr BGR frame for drawing by the main thread.
AddItemToFrameQueue(pFrameRGB);
av_frame_unref(frame);
}
}
while (GetQueueSize() > 100) {
Sleep(10);
}
}
CloseFrameQueue();
av_frame_free(&frame);
avcodec_close(codecCtx);
avformat_close_input(&fmtCtx);
return 0;
}
Is there a better way to allocate a new frame for holding the post sws_scale() transformation?
There is a similar stackoverflow question that uses mostly depreciated function calls. I can't seem to find any answers that conform to the new version of ffmpeg in the documentation. Any help would be appreciated.

Following the suggestions made in the comments I added a av_packet_unref() call to my decoding loop, and it stopped the memory leak issues I was having.
sws_scale(swsCtx, (const UINT8* const *) frame->data, frame->linesize, 0, codecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
// Queue thr BGR frame for drawing by the main thread.
AddItemToFrameQueue(pFrameRGB);
av_frame_unref(frame);
}
av_packet_unref(&pkt);
}
while (GetQueueSize() > 100) {
Sleep(10);
}

Related

An Unhandled exception: YUV Packed to JPEG Conversion On Button Press VC++2010 Form Application

i have YUV packed images which i have first convert to planar and then jpeg format (on button press) for Displaying in a picture box in VC++ 2010, using command at the end of entire code (after jpeg conversion done)
pictureBox1->Image = Image::FromFile("d:\\encoded_pic_420.jpg");
i am able to store and format convert these received images from YUV packed to YUV planar First and then Second from "Planar to JPEG format" (below Code First & Second).
on first time it Convert and Display image in picture box sucessfully, but when i press button second time it generate an error (in this below code section) SECOND PART OF CODE (PLANAR TO JPEG CONVERSION
if (got_picture==1)
{
pkt.stream_index = video_st->index;
ret = av_write_frame(pFormatCtx, &pkt);
}
error mesage is at Code Line (above) ret = av_write_frame (pFormatCtx,&pkt);
message is :
"An unhandled exception of type accured System.AccessViolationException , Additional information: Attempted to read or write protected memory, this is often an indication that other memory is corrupted"
the Complete code which i am using for YUV packet to Planar first and then second Planar to Jpeg conversion is below one.
First: YUV PACKED to PLANAR CONVERSION
FILE *in_file = NULL; //fopen("myHexFile.yuv","rb"); input PACKED
FILE *out_file = NULL; //Output File Planar format
int in_width = 2448; //YUV's width
int in_height = 2050; //YUV's heigh
int out_width = 2448; //YUV's width
int out_height = 2050; //YUV's heigh
int in_linesize[4];
int out_linesize[4];
uint8_t *in_data[4], *out_data[4];
unsigned long int out_bufsize,in_bufsize;
in_file = fopen("myHexFile.yuv","rb"); //This is YUV422-UYVY Input packed image
if(in_file == NULL)
{
this->Print2TextBox1(L"Input File Opening error...!");
exit(1);
}
out_file = fopen("d:\\myHexFile_Planar.yuv", "wb"); //Source Input File
if(out_file == NULL)
{
this->Print2TextBox1(L"toutput File Opening error...!!");
exit(1);
}
else { this->Print2TextBox1(L"Output File Created...!!\n"); }
//-Loads the whole database of available codecs and formats-------
av_register_all();
this->Print2TextBox1(L"Codac database Loaded...\n");
//---Create scaling context------------------------sws_getContex
this->Print2TextBox1(L"Creating Scaling context..\n");
sws_ctx = sws_getContext( in_width, in_height, src_pix_fmt,
out_width,out_height,dst_pix_fmt,
SWS_BICUBIC, NULL, NULL, NULL);
if(!sws_ctx) { this->Print2TextBox1(L"Context Error..\n"); }
//--Allocate Source Image Buffer--------------------------
this->Print2TextBox1(L"Allocate Source Image Buffer...\n");
AVFrame *RawPic = av_frame_alloc();
if(!RawPic)
{
this->Print2TextBox1(L"Could not allocate Raw Image frame\n");
exit(1);
}
RawPic->format = src_pix_fmt;
RawPic->width = in_width;
RawPic->height = in_height;
int num_bytes1 = avpicture_get_size(src_pix_fmt,in_width,in_height);
uint8_t* RawPic_Buffer = (uint8_t*)av_malloc(num_bytes1*sizeof(int8_t));
ret =av_image_alloc(RawPic->data,in_linesize,in_width,in_height,src_pix_fmt, 1);
if(ret < 0)
{
this->Print2TextBox1(L"Could not allocate raw picture buffer\n");
exit(1);
}
in_bufsize = ret;
//------Reading Input Image and Store in RawPic->Data Pointer---
fread(RawPic->data[0],1,in_bufsize,in_file);
//----Allocate Desitnation Image Buffer-------------------
this->Print2TextBox1(L"Allocate Destination Image Buffer...\n");
AVFrame *ScalePic = av_frame_alloc();
if(!ScalePic)
{
this->Print2TextBox1(L"Could not allocate Scale Image frame\n");
exit(1);
}
ScalePic->format = dst_pix_fmt;//pCodecCtx->pix_fmt;
ScalePic->width = out_width;
ScalePic->height = out_height;
int num_bytes2 = avpicture_get_size(dst_pix_fmt,out_width,out_height);
uint8_t* ScalePic_Buffer = (uint8_t *)av_malloc(num_bytes2*sizeof(int8_t));
ret = av_image_alloc(ScalePic->data,out_linesize,out_width,out_height,dst_pix_fmt, 1); //16
if(ret < 0) { this->Print2TextBox1(L"Could not allocate Scale picture buffer\n"); exit(1);}
out_bufsize = ret;
//-Create scaling context-OR CONVERTED TO DESTINATION FORMAT-----sws_scale
this->Print2TextBox1(L"Creating Scaling context...sws_scale\n");
sws_scale(sws_ctx, RawPic->data, in_linesize, 0, ScalePic->height, ScalePic->data, out_linesize);
//-----Write Scale Image to outputfile-
this->Print2TextBox1(L"Write Scale Image to outputfile..\n");
fwrite(ScalePic->data[0],1,out_bufsize,out_file);
//---Release all memory and close file--
fclose(in_file);
fclose(out_file);
av_freep(&RawPic->data[0]);
av_freep(&ScalePic->data[0]);
av_frame_free(&ScalePic);
av_frame_free(&RawPic);
SECOND--CONVERT to PLANAR TO JPEG FORMAT------(in Continuation to Above Code)
const char* myJpeg_file = "d:\\encoded_pic_444.jpg"; //Output JPEG
in_file = fopen("d:\\myHexFile_Planar.yuv", "rb"); //Input Planar File
if(in_file == NULL)
{
this->Print2TextBox1(L"File Opening error...!!");
exit(1);
}
else this->Print2TextBox1(L"YUV File Open Sucessfully...!!\n\n");
av_register_all(); // Loads the whole database of available codecs and formats.
pFormatCtx = avformat_alloc_context();
fmt = NULL;
fmt = av_guess_format("mjpeg",NULL,NULL);
pFormatCtx->oformat = fmt;
if (avio_open(&pFormatCtx->pb,myJpeg_file, AVIO_FLAG_READ_WRITE) < 0)
{
this->Print2TextBox1(L"Couldn't open output file.");
}
video_st = avformat_new_stream(pFormatCtx, 0);
if (video_st==NULL)
{
this->Print2TextBox1(L"avformat_new_stream.");
}
pCodecCtx = video_st->codec;
pCodecCtx->codec_id = fmt->video_codec;
pCodecCtx->codec_type = AVMEDIA_TYPE_VIDEO;
pCodecCtx->pix_fmt = AV_PIX_FMT_YUVJ420P;
pCodecCtx->width = in_width;
pCodecCtx->height = in_height;
pCodecCtx->time_base.num = 1;
pCodecCtx->time_base.den = 1;//25;
this->Print2TextBox1(L"Conversion start\n");
//Output some information
av_dump_format(pFormatCtx, 0, myJpeg_file, 1);
// Determine if desired video encoder is installed
pCodec = avcodec_find_encoder(pCodecCtx->codec_id);
if (!pCodec)
{
this->Print2TextBox1(L"Codec not found.");
//return -1;
}
this->Print2TextBox1(L"Codec Identified done\n");
if (avcodec_open2(pCodecCtx, pCodec,NULL) < 0){
this->Print2TextBox1(L"Could not open codec.\n");
//return -1;
}
this->Print2TextBox1(L"Codec Open done\n");
//-----------------------------------------------
picture = av_frame_alloc();
size = avpicture_get_size(pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);
picture_buf = (uint8_t *)av_malloc(size);
if (!picture_buf)
{ this->Print2TextBox1(L"Size Allocation error\n");
//return -1;
}
avpicture_fill((AVPicture *)picture, picture_buf, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);
this->Print2TextBox1(L"Write Header..");
avformat_write_header(pFormatCtx,NULL);
y_size = pCodecCtx->width * pCodecCtx->height;
av_new_packet(&pkt,y_size*3);
//-------------------------------------------------------420 Format
//Read YUV
if (fread(picture_buf, 1, y_size*3/2, in_file) <=0)
{
this->Print2TextBox1(L"Could not read input file.");
//return -1;
}
//--------------------------------------------input image format UYVY
picture->data[0] = picture_buf; // Y
picture->data[1] = picture_buf+ y_size; // U
picture->data[2] = picture_buf+ y_size*5/4; // V
this->Print2TextBox1(L" Encode the image..\n");
ret = avcodec_encode_video2(pCodecCtx, &pkt,picture, &got_picture);
if(ret < 0)
{
this->Print2TextBox1(L"Encode Error.\n");
//return -1;
}
if (got_picture==1)
{
pkt.stream_index = video_st->index;
//#### PROBLEM IN THIS LINE BELOW WHEN RE-EXECUTE THE CODE ###
ret = av_write_frame(pFormatCtx, &pkt);
}
av_free_packet(&pkt);
//Write Trailer
av_write_trailer(pFormatCtx);
this->Print2TextBox1(L"Encode Successful.\n");
if (video_st)
{
avcodec_close(video_st->codec);
av_free(picture);
av_free(picture_buf);
}
avio_close(pFormatCtx->pb);
avformat_free_context(pFormatCtx);
fclose(in_file);
it seems that some of memory is not yet free or when i am trying to re-use this above code in second time in a loop,
plz suggest/guide me where i am doing wrong and not freeing up the memory..?
i am trying to Display Image (Current/updated) on every button press in VC++2010
You are allocating AVFrame
picture = av_frame_alloc();
, but casting it to deprecated AVPicture later:
avpicture_fill((AVPicture *)picture, picture_buf, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);
solved by renaming the "encoded_pic_420.jpg" file with some other name and display it on PictureBox
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363851(v=vs.85).aspx

Libav AVFrame to Opencv Mat to AVPacket conversion

I am new to libav and I am writing a video manipulation software which uses opencv as its heart. What I did is briefly as below:
1- read the video packet
2- decode the packet into AVFrame
3- convert
the AVFrame to CV Mat
4- manipulate the Mat
5- convert the CV Mat
into AVFrame
6- encode the AVFrame into AVPacket
7- write the packet
8- goto 1
I read dranger tutorial in http://dranger.com/ffmpeg/tutorial01.html and I also used decoding_encoding example. I can read the video, extract video frames and convert them to CV Mat. My problem starts from converting from cv Mat to AVFrame and encode it to AVPacket.
Would you please help me with this?
Here is my code :
int main(int argc, char **argv)
{
AVOutputFormat *ofmt = NULL;
AVFormatContext *ifmt_ctx = NULL, *ofmt_ctx = NULL;
AVPacket pkt;
AVCodecContext *pCodecCtx = NULL;
AVCodec *pCodec = NULL;
AVFrame *pFrame = NULL;
AVFrame *pFrameRGB = NULL;
int videoStream=-1;
int audioStream=-1;
int frameFinished;
int numBytes;
uint8_t *buffer = NULL;
struct SwsContext *sws_ctx = NULL;
FrameManipulation *mal_frame;
const char *in_filename, *out_filename;
int ret, i;
if (argc < 3) {
printf("usage: %s input output\n"
"API example program to remux a media file with libavformat and libavcodec.\n"
"The output format is guessed according to the file extension.\n"
"\n", argv[0]);
return 1;
}
in_filename = arg[1];
out_filename = arg[2];
av_register_all();
if ((ret = avformat_open_input(&ifmt_ctx, in_filename, 0, 0)) < 0) {
fprintf(stderr, "Could not open input file '%s'", in_filename);
goto end;
}
if ((ret = avformat_find_stream_info(ifmt_ctx, 0)) < 0) {
fprintf(stderr, "Failed to retrieve input stream information");
goto end;
}
av_dump_format(ifmt_ctx, 0, in_filename, 0);
avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, out_filename);
if (!ofmt_ctx) {
fprintf(stderr, "Could not create output context\n");
ret = AVERROR_UNKNOWN;
goto end;
}
ofmt = ofmt_ctx->oformat;
for (i = 0; i < ifmt_ctx->nb_streams; i++) {
AVStream *in_stream = ifmt_ctx->streams[i];
AVStream *out_stream = avformat_new_stream(ofmt_ctx, in_stream->codec->codec);
if(ifmt_ctx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO &&
videoStream < 0) {
videoStream=i;
}
if(ifmt_ctx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO &&
audioStream < 0) {
audioStream=i;
}
if (!out_stream) {
fprintf(stderr, "Failed allocating output stream\n");
ret = AVERROR_UNKNOWN;
goto end;
}
ret = avcodec_copy_context(out_stream->codec, in_stream->codec);
if (ret < 0) {
fprintf(stderr, "Failed to copy context from input to output stream codec context\n");
goto end;
}
out_stream->codec->codec_tag = 0;
if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)
out_stream->codec->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
}
pCodec=avcodec_find_decoder(ifmt_ctx->streams[videoStream]->codec->codec_id);
pCodecCtx = avcodec_alloc_context3(pCodec);
if(avcodec_copy_context(pCodecCtx, ifmt_ctx->streams[videoStream]->codec) != 0) {
fprintf(stderr, "Couldn't copy codec context");
return -1; // Error copying codec context
}
// Open codec
if(avcodec_open2(pCodecCtx, pCodec, NULL)<0)
return -1; // Could not open codec
// Allocate video frame
pFrame=av_frame_alloc();
// Allocate an AVFrame structure
pFrameRGB=av_frame_alloc();
// Determine required buffer size and allocate buffer
numBytes=avpicture_get_size(AV_PIX_FMT_RGB24, ifmt_ctx->streams[videoStream]->codec->width,
ifmt_ctx->streams[videoStream]->codec->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
// Assign appropriate parts of buffer to image planes in pFrameRGB
// Note that pFrameRGB is an AVFrame, but AVFrame is a superset
// of AVPicture
avpicture_fill((AVPicture *)pFrameRGB, buffer, AV_PIX_FMT_BGR24,
ifmt_ctx->streams[videoStream]->codec->width, ifmt_ctx->streams[videoStream]->codec->height);
av_dump_format(ofmt_ctx, 0, out_filename, 1);
if (!(ofmt->flags & AVFMT_NOFILE)) {
ret = avio_open(&ofmt_ctx->pb, out_filename, AVIO_FLAG_WRITE);
if (ret < 0) {
fprintf(stderr, "Could not open output file '%s'", out_filename);
goto end;
}
}
ret = avformat_write_header(ofmt_ctx, NULL);
if (ret < 0) {
fprintf(stderr, "Error occurred when opening output file\n");
goto end;
}
// Assign appropriate parts of buffer to image planes in pFrameRGB
// Note that pFrameRGB is an AVFrame, but AVFrame is a superset
// of AVPicture
avpicture_fill((AVPicture *)pFrameRGB, buffer, AV_PIX_FMT_BGR24,
ifmt_ctx->streams[videoStream]->codec->width,
ifmt_ctx->streams[videoStream]->codec->height);
// initialize SWS context for software scaling
sws_ctx = sws_getContext(
ifmt_ctx->streams[videoStream]->codec->width,
ifmt_ctx->streams[videoStream]->codec->height,
ifmt_ctx->streams[videoStream]->codec->pix_fmt,
ifmt_ctx->streams[videoStream]->codec->width,
ifmt_ctx->streams[videoStream]->codec->height,
AV_PIX_FMT_BGR24,
SWS_BICUBIC,
NULL,
NULL,
NULL
);
// Loop through packets
while (1) {
AVStream *in_stream, *out_stream;
ret = av_read_frame(ifmt_ctx, &pkt);
if(pkt.stream_index==videoStream)
// Decode video frame
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &pkt);
if(frameFinished) {
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameRGB->data, pFrameRGB->linesize);
cv::Mat img= mal_frame->process(
pFrameRGB,pFrame->width,pFrame->height);
/* My problem is Here ------------*/
avpicture_fill((AVPicture*)pFrameRGB,
img.data,
PIX_FMT_BGR24,
outStream->codec->width,
outStream->codec->height);
pFrameRGB->width = ifmt_ctx->streams[videoStream]->codec->width;
pFrameRGB->height = ifmt_ctx->streams[videoStream]->codec->height;
avcodec_encode_video2(ifmt_ctx->streams[videoStream]->codec ,
&pkt , pFrameRGB , &gotPacket);
/*
I get this error
[swscaler # 0x14b58a0] bad src image pointers
[swscaler # 0x14b58a0] bad src image pointers
*/
/* My Problem Ends here ---------- */
}
if (ret < 0)
break;
in_stream = ifmt_ctx->streams[pkt.stream_index];
out_stream = ofmt_ctx->streams[pkt.stream_index];
//log_packet(ifmt_ctx, &pkt, "in");
/* copy packet */
pkt.pts = av_rescale_q_rnd(pkt.pts, in_stream->time_base, out_stream->time_base,
AV_ROUND_NEAR_INF);
pkt.dts = av_rescale_q_rnd(pkt.dts, in_stream->time_base, out_stream->time_base, AV_ROUND_NEAR_INF);
pkt.duration = av_rescale_q(pkt.duration, in_stream->time_base, out_stream->time_base);
pkt.pos = -1;
log_packet(ofmt_ctx, &pkt, "out");
ret = av_interleaved_write_frame(ofmt_ctx, &pkt);
if (ret < 0) {
fprintf(stderr, "Error muxing packet\n");
break;
}
av_free_packet(&pkt);
}
av_write_trailer(ofmt_ctx);
end:
avformat_close_input(&ifmt_ctx);
/* close output */
if (ofmt_ctx && !(ofmt->flags & AVFMT_NOFILE))
avio_closep(&ofmt_ctx->pb);
avformat_free_context(ofmt_ctx);
if (ret < 0 && ret != AVERROR_EOF) {
return 1;
}
return 0;
}
When I run this code, I get unknown fatal error in this part:
/* My problem is Here ------------*/
avpicture_fill((AVPicture*)pFrameRGB,
img.data,
PIX_FMT_BGR24,
outStream->codec->width,
outStream->codec->height);
pFrameRGB->width = ifmt_ctx->streams[videoStream]->codec->width;
pFrameRGB->height = ifmt_ctx->streams[videoStream]->codec->height;
avcodec_encode_video2(ifmt_ctx->streams[videoStream]->codec ,
&pkt , pFrameRGB , &gotPacket);
/*
I get this error
[swscaler # 0x14b58a0] bad src image pointers
[swscaler # 0x14b58a0] bad src image pointers
*/
/* My Problem Ends here ---------- */
Here is where I want to convert back cv Mat to AVFrame and encode it to AVPacket. I appreciate your help.
After reading some examples, reading source code and some helps people offered, I managed to make the code run. I used transcoding and encoding examples and mixed them up. Here is my code
here are the highlights:
1- libswscale should be used to convert AVFrame with required packet format to be fed into openCV Mat. To do so, we define
struct SwsContext *sws_ctx = NULL;
sws_ctx = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
AV_PIX_FMT_BGR24,
SWS_BICUBIC,
NULL,
NULL,
NULL
);
To convert back the opencv Mat to AVFrame, one should again use swscale and translate opencv BGR frame format to YUV. So, I do this:
struct SwsContext *sws_ctx_bgr_yuv = NULL;
sws_ctx_bgr_yuv = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
AV_PIX_FMT_BGR24,
pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt //AV_PIX_FMT_YUV420p
,0,0,NULL,NULL);
And, here is frame reading/decoding/encoding loop:
while (1) {
if ((ret = av_read_frame(ifmt_ctx, &packet)) < 0)
break;
stream_index = packet.stream_index;
type = ifmt_ctx->streams[packet.stream_index]->codec->codec_type;
av_log(NULL, AV_LOG_DEBUG, "Demuxer gave frame of stream_index %u\n",
stream_index);
if (filter_ctx[stream_index].filter_graph) {
av_log(NULL, AV_LOG_DEBUG, "Going to reencode&filter the frame\n");
frame = av_frame_alloc();
if (!frame) {
ret = AVERROR(ENOMEM);
break;
}
av_packet_rescale_ts(&packet,
ifmt_ctx->streams[stream_index]->time_base,
ifmt_ctx->streams[stream_index]->codec->time_base);
dec_func = (type == AVMEDIA_TYPE_VIDEO) ? avcodec_decode_video2 :
avcodec_decode_audio4;
ret = dec_func(ifmt_ctx->streams[stream_index]->codec, frame,
&got_frame, &packet);
if (ret < 0) {
av_frame_free(&frame);
av_log(NULL, AV_LOG_ERROR, "Decoding failed\n");
break;
}
if (got_frame) {
if(stream_index==video_index){
sws_scale(sws_ctx, (uint8_t const * const *)frame->data,
frame->linesize, 0, pCodecCtx->height,
pFrameRGB->data, pFrameRGB->linesize);
/*------------------------------------------------------------------------
/* Frame converts to opencv Mat
/*------------------------------------------------------------------------*/
cv::Mat img(frame->height,frame->width,CV_8UC3,pFrameRGB->data[0]);
img=manipulate_image(img); //this is opencv Mat, do whatever you want, but don't change its dimensions and format
//manipulate_function can be considered as as simple as blurring
const int stride[] = {img.step[0] };
/* opencv Mat converts back to AVFrame */
sws_scale(sws_ctx_bgr_yuv, &img.data, stride, 0, img.rows, frame->data, frame->linesize);
}
frame->pts = av_frame_get_best_effort_timestamp(frame);
/* AVFrame re-encodes to AVPacket and will be sent to encoder */
ret = filter_encode_write_frame(frame, stream_index);
av_frame_free(&frame);
if (ret < 0)
goto end;
} else {
av_frame_free(&frame);
}
} else {
/* remux this frame without reencoding */
av_packet_rescale_ts(&packet,
ifmt_ctx->streams[stream_index]->time_base,
ofmt_ctx->streams[stream_index]->time_base);
ret = av_interleaved_write_frame(ofmt_ctx, &packet);
if (ret < 0)
goto end;
}
av_free_packet(&packet);
}
Here is another way to convert between cv::Mat and AVframe using c++ based on some code I found and debugged. Please note its specifically for 8 bit 3 channel images but that can be changed by changing AV_PIX_FMT_BGR24 in both functions.
Hope this helps.
AVFrame cvmat_to_avframe(cv::Mat* frame)
{
AVFrame dst;
cv::Size frameSize = frame->size();
AVCodec *encoder = avcodec_find_encoder(AV_CODEC_ID_RAWVIDEO);
AVFormatContext* outContainer = avformat_alloc_context();
AVStream *outStream = avformat_new_stream(outContainer, encoder);
avcodec_get_context_defaults3(outStream->codec, encoder);
outStream->codec->pix_fmt = AV_PIX_FMT_BGR24;
outStream->codec->width = frame->cols;
outStream->codec->height = frame->rows;
avpicture_fill((AVPicture*)&dst, frame->data, AV_PIX_FMT_BGR24, outStream->codec->width, outStream->codec->height);
dst.width = frameSize.width;
dst.height = frameSize.height;
return dst;
}
cv::Mat avframe_to_cvmat(AVFrame *frame)
{
AVFrame dst;
cv::Mat m;
memset(&dst, 0, sizeof(dst));
int w = frame->width, h = frame->height;
m = cv::Mat(h, w, CV_8UC3);
dst.data[0] = (uint8_t *)m.data;
avpicture_fill( (AVPicture *)&dst, dst.data[0], AV_PIX_FMT_BGR24, w, h);
struct SwsContext *convert_ctx=NULL;
enum AVPixelFormat src_pixfmt = AV_PIX_FMT_BGR24;
enum AVPixelFormat dst_pixfmt = AV_PIX_FMT_BGR24;
convert_ctx = sws_getContext(w, h, src_pixfmt, w, h, dst_pixfmt,
SWS_FAST_BILINEAR, NULL, NULL, NULL);
sws_scale(convert_ctx, frame->data, frame->linesize, 0, h,
dst.data, dst.linesize);
sws_freeContext(convert_ctx);
return m;
}

FFMPEG scaling error-invalid dimension

I'm working on a Project in FFMPEG, and right now i'm stuck on a Problem.
What i am trying to do is, convert a png pic in to a mpeg Video file. I've already managed to take informatin from the Picture, but somehow i cannot convert the Picture in YUV Format. It returns "0x0->0x0 is invalid scaling dimensions".
Here'S my code:
AVFrame *pFrame;
AVFrame *pFrameYUV;
pFrame = av_frame_alloc();
pFrameYUV = av_frame_alloc();
int numBytes;//Groesse des Bildes
uint8_t *buffer= NULL;
numBytes=avpicture_get_size(AV_PIX_FMT_RGB24, pCodecCtx->width,pCodecCtx->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
/*avpicture_fill((AVPicture *)pFrameYUV, buffer, AV_PIX_FMT_YUV420P,
pCodecCtx->width, pCodecCtx->height);*/
av_image_fill_arrays(pFrameYUV->data,pFrameYUV->linesize,buffer,AV_PIX_FMT_YUV420P,pCodecCtx->width,pCodecCtx->height,32);
struct SwsContext *sws_ctx = NULL;
AVPacket packet;
// initialize SWS context for software scaling
sws_ctx=sws_getCachedContext(NULL,pFrame->width,pFrame->height,AV_PIX_FMT_RGB24,pFrameYUV->width,pFrameYUV->height,AV_PIX_FMT_YUV420P,0,0,0,0);
pFrameYUV->height= pFrame->height;
pFrameYUV->width= pFrame->width;
while (av_read_frame(pFormatCtx,&packet)>=0)
{
if(packet.stream_index == videoStream)
{
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished)
{
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameYUV->data, pFrameYUV->linesize);
printf("%d",pFrameYUV->height);
}
}
av_free_packet(&packet);
}
EDIT:
After the converting, i tried to encode the Frame in o a packet, but he size of the packet is 0.
Code
AVPacket pkt;
av_init_packet(&pkt);
pkt.stream_index= st->index;
pkt.data= buffer;
pkt.size=numBytes;
int got_pkt;
test=avcodec_encode_video2(st->codec,&pkt,pFrameYUV,&got_pkt);
printf("%d",got_pkt);
These values pFrameYUV->height, pFrame->height, pFrameYUV->width, pFrame->width didn't set when calling sws_getCachedContext.
Do you mean the dimension not changing? If so, set them before sws_getCachedContext.
pFrameYUV->height = pFrame->height = pCodecCtx->height;
pFrameYUV->width = pFrame->width = pCodecCtx->width;
sws_ctx=sws_getCachedContext(NULL,pFrame->width,pFrame->height,AV_PIX_FMT_RGB24,pFrameYUV->width,pFrameYUV->height,AV_PIX_FMT_YUV420P,0,0,0,0);

c++ videoplayer ffmpeg: get pixel data?

I want to get the pixel data of a frame. I found this (in original version as old code) and changed some things.
I have this code:
AVFormatContext *pFormatCtx;
pFormatCtx = avformat_alloc_context();
// Open file
if (int err = avformat_open_input(&pFormatCtx, file, NULL, 0) != 0)
{
exit(2);
}
// Get infromation about streams
if (avformat_find_stream_info(pFormatCtx, NULL) < 0)
{
exit(2);
}
// # video stream
int videoStreamIndex = -1;
AVCodecContext *pVideoCodecCtx;
AVCodec *pVideoCodec;
int res = 0;
int width = 0;
int height = 0;
for (unsigned int i = 0; i < pFormatCtx->nb_streams; i++)
{
if (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
{
videoStreamIndex = i;
pVideoCodecCtx = pFormatCtx->streams[i]->codec;
// Find decoder
pVideoCodec = avcodec_find_decoder(pVideoCodecCtx->codec_id);
if (pVideoCodec)
{
// Open decoder
res = !(avcodec_open2(pVideoCodecCtx, pVideoCodec, NULL) < 0);
width = pVideoCodecCtx->coded_width;
height = pVideoCodecCtx->coded_height;
}
break;
}
}
// Frame width
width = pFormatCtx->streams[videoStreamIndex]->codec->width;
// Frame height
height = pFormatCtx->streams[videoStreamIndex]->codec->height;
AVPacket packet;
int got_picture_ptr;
AVPacket *avpkt;
AVFrame * pOutFrame;
pOutFrame = av_frame_alloc();
AVFrame * rgbOutFrame = av_frame_alloc();
if (!pOutFrame) {
fprintf(stderr, "Could not allocate video frame\n");
exit(1);
}
while (av_read_frame(pFormatCtx, &packet) >= 0)
{
if (packet.stream_index == videoStreamIndex)
{
// Decode packeg to frame.
int videoFrameBytes = avcodec_decode_video2(pVideoCodecCtx, pOutFrame,
&got_picture_ptr, &packet);
// Create context
SwsContext* pImgConvertCtx = sws_getContext(pVideoCodecCtx->width,
pVideoCodecCtx->height,
pVideoCodecCtx->pix_fmt,
pVideoCodecCtx->width, pVideoCodecCtx->height,
AV_PIX_FMT_YUV420P,
SWS_BICUBIC, NULL, NULL, NULL);
// Convert frame
sws_scale(pImgConvertCtx, pOutFrame->data, pOutFrame->linesize,
width, height, rgbOutFrame->data, rgbOutFrame->linesize);
}
}
I know, that the code from SwsContext and sws_scale is wrong but I wonder, where can I find the pixel data of my frame... (and in which format it is stored).
Can someone help me here?
Pixel data is stored in data field.
According to the documentation:
uint8_t* AVFrame::data[AV_NUM_DATA_POINTERS]
pointer to the picture/channel planes.
Look here for more information.
Generally speaking, your code is a bit misleading and rather buggy. I can point out some drawbacks:
1) You don't need to create new SwsContext on every incoming video packet. Just create it once before while cycle.
2) Next, you have an rgbOutFrame, but SwsContext is created for scaling into the YUV420 pixel format. It looks strange.
3) Besides, avcodec_decode_video2 is invoked, but you never check neither return value nor got_picture_ptr flag. Such practice is really error-prone.
And so on...
Hope it'll help you to improve your program and get necessary results.

libavcodec video decoding not working

I am trying to decode video encoded with H264. I am sending AVPacket's data and its size to decoder code. there I am trying to decode the frame and display it on a GUI. problem is when I am decoding the frame it is returning same number of frame byte as the size of packet means it is not decompressing the data. Can any one tell what will be the problem. My encoding program is working fine.
here is code for encoding
static struct SwsContext *img_convert_ctx;
pkt.data = NULL;
pkt.size = 0;
avpicture_fill((AVPicture *)srcFrame, frame,AV_PIX_FMT_BGR24, 640, 480);
if(img_convert_ctx == NULL) {
int w = 640;
int h = 480;
img_convert_ctx = sws_getContext(w, h,
AV_PIX_FMT_BGR24, c->width, c->height, PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL);
if(img_convert_ctx == NULL) {
fprintf(stderr, "Cannot initialize the conversion context!\n");
}
}
sws_scale(img_convert_ctx, srcFrame->data, srcFrame->linesize, 0,480,picture->data, picture->linesize);
fflush(stdout);
picture->pts=counter;
ret = avcodec_encode_video2(c, &pkt, picture, &got_output);
if (ret < 0) {
fprintf(stderr, "Error encoding frame\n");
}
if (got_output) {
vdec.decode_frame(pkt.data ,pkt.size);
av_free_packet(&pkt);
}
decoder code...
int len ,got_frame;
avpkt.size = data_length;
avpkt.data = frame_buffer;
if(!frame_buffer){
return "frame buffer empty\n";
}
len = avcodec_decode_video2(avctx ,frame ,&got_frame ,&avpkt);
if( len < 0){
return "error while decoding\n";
}
if( got_frame ){
static struct SwsContext *img_convert_ctx;
if(img_convert_ctx == NULL) {
img_convert_ctx = sws_getContext(w, h,
PIX_FMT_YUV420P, avctx->width,
avctx->height, PIX_FMT_BGR24,
SWS_BICUBIC, NULL, NULL, NULL);
if(img_convert_ctx == NULL) {
return "Cannot initialize the conversion context!\n";
}
}
j=sws_scale(img_convert_ctx,
frame->data , frame->linesize ,
0, h ,picture->data,
picture->linesize );
if(j==0){
exit(1);
}
I am initializing all other code like AVCodecContext and Codec into other method.
Please help me to find the solution.
The avcodec_decode_video2 function should return the number of bytes processed, not the number of bytes of result picture. You just have to check the value of got_frame to find out when decoded a complete frame.