I have an application that encodes a video stream coming from a camera in H264 and stores it on the disk. In another part of the application I'm loading this movie and showing it to the user. All of this works except the fact that the unique frame index number that I add when encoding the frames aren't the same as the frame numbers that I get when I decode the file.
this is a snippet of my code when I encode a frame. I use "display_picture_number" to store this unique frame number, is this correct? why is the number not the same?
void MovieCodec::createFrame( const MyImage& image, AVFrame* frame )
{
frame->format = streamPixelFormat;
frame->width = image.width();
frame->height = image.height();
frame->pict_type = AV_PICTURE_TYPE_P;
frame->display_picture_number = image.uniqueImageNumber();
int ret = av_image_alloc( frame->data, frame->linesize, frame->width, frame->height, AV_PIX_FMT_BGR24, 1);
if (ret < 0)
{
return;
}
struct SwsContext* sws_ctx = sws_getContext((int)image.width(), (int)image.height(), AV_PIX_FMT_BGR24,
(int)image.width(), (int)image.height(), streamPixelFormat,
0, NULL, NULL, NULL);
const uint8_t* rgbData[1] = { (uint8_t* )image.getData() };
int rgbLineSize[1] = { 3 * (int)image.width() };
sws_scale(sws_ctx, rgbData, rgbLineSize, 0, image.height(), frame->data, frame->linesize);
}
Check AVFrame structure documentation.
You cannot set display_picture_number as this is will not be processed by ffmpeg during encoding
You could use AVFrame 's metadata instead to store some "notes"
Related
I am making a simple video player with ffmpeg. I have noticed that there is a memory leak originating in libavutil. Because ffmpeg is a mature library I assume that I am allocating a new frame incorrectly. The documentation is also vague about freeing the buffer that is created when you call av_frame_get_buffer(). Below is the code I am using to decode the video and queue it up for display on the UI thread.
DWORD WINAPI DecoderThread(LPVOID lpParam)
{
AVFrame *frame = NULL;
AVPacket pkt;
SwsContext *swsCtx = NULL;
UINT8 *buffer = NULL;
INT iNumBytes = 0;
INT result = 0;
frame = av_frame_alloc();
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;
// Create scaling context
swsCtx = sws_getContext(codecCtx->width, codecCtx->height, codecCtx->pix_fmt, codecCtx->width, codecCtx->height, AV_PIX_FMT_BGR24, SWS_BICUBIC, NULL, NULL, NULL);
while (av_read_frame(fmtCtx, &pkt) >= 0) {
if (pkt.stream_index == videoStream) {
result = avcodec_send_packet(codecCtx, &pkt);
while (result >= 0) {
result = avcodec_receive_frame(codecCtx, frame);
if (result == AVERROR(EAGAIN) || result == AVERROR_EOF) {
break;
} else if (result < 0) {
// another error.
}
// Create a new frame to store the RGB24 data.
AVFrame *pFrameRGB = av_frame_alloc();
// Allocate space for the new RGB image.
//av_image_alloc(pFrameRGB->data, pFrameRGB->linesize, codecCtx->width, codecCtx->height, AV_PIX_FMT_BGR24, 1);
// Copy all of the properties from the YUV420P frame.
av_frame_copy_props(pFrameRGB, frame);
pFrameRGB->width = frame->width;
pFrameRGB->height = frame->height;
pFrameRGB->format = AV_PIX_FMT_BGR24;
av_frame_get_buffer(pFrameRGB, 0);
// Convert fram from YUV420P to BGR24 for display.
sws_scale(swsCtx, (const UINT8* const *) frame->data, frame->linesize, 0, codecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
// Queue thr BGR frame for drawing by the main thread.
AddItemToFrameQueue(pFrameRGB);
av_frame_unref(frame);
}
}
while (GetQueueSize() > 100) {
Sleep(10);
}
}
CloseFrameQueue();
av_frame_free(&frame);
avcodec_close(codecCtx);
avformat_close_input(&fmtCtx);
return 0;
}
Is there a better way to allocate a new frame for holding the post sws_scale() transformation?
There is a similar stackoverflow question that uses mostly depreciated function calls. I can't seem to find any answers that conform to the new version of ffmpeg in the documentation. Any help would be appreciated.
Following the suggestions made in the comments I added a av_packet_unref() call to my decoding loop, and it stopped the memory leak issues I was having.
sws_scale(swsCtx, (const UINT8* const *) frame->data, frame->linesize, 0, codecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
// Queue thr BGR frame for drawing by the main thread.
AddItemToFrameQueue(pFrameRGB);
av_frame_unref(frame);
}
av_packet_unref(&pkt);
}
while (GetQueueSize() > 100) {
Sleep(10);
}
i have YUV packed images which i have first convert to planar and then jpeg format (on button press) for Displaying in a picture box in VC++ 2010, using command at the end of entire code (after jpeg conversion done)
pictureBox1->Image = Image::FromFile("d:\\encoded_pic_420.jpg");
i am able to store and format convert these received images from YUV packed to YUV planar First and then Second from "Planar to JPEG format" (below Code First & Second).
on first time it Convert and Display image in picture box sucessfully, but when i press button second time it generate an error (in this below code section) SECOND PART OF CODE (PLANAR TO JPEG CONVERSION
if (got_picture==1)
{
pkt.stream_index = video_st->index;
ret = av_write_frame(pFormatCtx, &pkt);
}
error mesage is at Code Line (above) ret = av_write_frame (pFormatCtx,&pkt);
message is :
"An unhandled exception of type accured System.AccessViolationException , Additional information: Attempted to read or write protected memory, this is often an indication that other memory is corrupted"
the Complete code which i am using for YUV packet to Planar first and then second Planar to Jpeg conversion is below one.
First: YUV PACKED to PLANAR CONVERSION
FILE *in_file = NULL; //fopen("myHexFile.yuv","rb"); input PACKED
FILE *out_file = NULL; //Output File Planar format
int in_width = 2448; //YUV's width
int in_height = 2050; //YUV's heigh
int out_width = 2448; //YUV's width
int out_height = 2050; //YUV's heigh
int in_linesize[4];
int out_linesize[4];
uint8_t *in_data[4], *out_data[4];
unsigned long int out_bufsize,in_bufsize;
in_file = fopen("myHexFile.yuv","rb"); //This is YUV422-UYVY Input packed image
if(in_file == NULL)
{
this->Print2TextBox1(L"Input File Opening error...!");
exit(1);
}
out_file = fopen("d:\\myHexFile_Planar.yuv", "wb"); //Source Input File
if(out_file == NULL)
{
this->Print2TextBox1(L"toutput File Opening error...!!");
exit(1);
}
else { this->Print2TextBox1(L"Output File Created...!!\n"); }
//-Loads the whole database of available codecs and formats-------
av_register_all();
this->Print2TextBox1(L"Codac database Loaded...\n");
//---Create scaling context------------------------sws_getContex
this->Print2TextBox1(L"Creating Scaling context..\n");
sws_ctx = sws_getContext( in_width, in_height, src_pix_fmt,
out_width,out_height,dst_pix_fmt,
SWS_BICUBIC, NULL, NULL, NULL);
if(!sws_ctx) { this->Print2TextBox1(L"Context Error..\n"); }
//--Allocate Source Image Buffer--------------------------
this->Print2TextBox1(L"Allocate Source Image Buffer...\n");
AVFrame *RawPic = av_frame_alloc();
if(!RawPic)
{
this->Print2TextBox1(L"Could not allocate Raw Image frame\n");
exit(1);
}
RawPic->format = src_pix_fmt;
RawPic->width = in_width;
RawPic->height = in_height;
int num_bytes1 = avpicture_get_size(src_pix_fmt,in_width,in_height);
uint8_t* RawPic_Buffer = (uint8_t*)av_malloc(num_bytes1*sizeof(int8_t));
ret =av_image_alloc(RawPic->data,in_linesize,in_width,in_height,src_pix_fmt, 1);
if(ret < 0)
{
this->Print2TextBox1(L"Could not allocate raw picture buffer\n");
exit(1);
}
in_bufsize = ret;
//------Reading Input Image and Store in RawPic->Data Pointer---
fread(RawPic->data[0],1,in_bufsize,in_file);
//----Allocate Desitnation Image Buffer-------------------
this->Print2TextBox1(L"Allocate Destination Image Buffer...\n");
AVFrame *ScalePic = av_frame_alloc();
if(!ScalePic)
{
this->Print2TextBox1(L"Could not allocate Scale Image frame\n");
exit(1);
}
ScalePic->format = dst_pix_fmt;//pCodecCtx->pix_fmt;
ScalePic->width = out_width;
ScalePic->height = out_height;
int num_bytes2 = avpicture_get_size(dst_pix_fmt,out_width,out_height);
uint8_t* ScalePic_Buffer = (uint8_t *)av_malloc(num_bytes2*sizeof(int8_t));
ret = av_image_alloc(ScalePic->data,out_linesize,out_width,out_height,dst_pix_fmt, 1); //16
if(ret < 0) { this->Print2TextBox1(L"Could not allocate Scale picture buffer\n"); exit(1);}
out_bufsize = ret;
//-Create scaling context-OR CONVERTED TO DESTINATION FORMAT-----sws_scale
this->Print2TextBox1(L"Creating Scaling context...sws_scale\n");
sws_scale(sws_ctx, RawPic->data, in_linesize, 0, ScalePic->height, ScalePic->data, out_linesize);
//-----Write Scale Image to outputfile-
this->Print2TextBox1(L"Write Scale Image to outputfile..\n");
fwrite(ScalePic->data[0],1,out_bufsize,out_file);
//---Release all memory and close file--
fclose(in_file);
fclose(out_file);
av_freep(&RawPic->data[0]);
av_freep(&ScalePic->data[0]);
av_frame_free(&ScalePic);
av_frame_free(&RawPic);
SECOND--CONVERT to PLANAR TO JPEG FORMAT------(in Continuation to Above Code)
const char* myJpeg_file = "d:\\encoded_pic_444.jpg"; //Output JPEG
in_file = fopen("d:\\myHexFile_Planar.yuv", "rb"); //Input Planar File
if(in_file == NULL)
{
this->Print2TextBox1(L"File Opening error...!!");
exit(1);
}
else this->Print2TextBox1(L"YUV File Open Sucessfully...!!\n\n");
av_register_all(); // Loads the whole database of available codecs and formats.
pFormatCtx = avformat_alloc_context();
fmt = NULL;
fmt = av_guess_format("mjpeg",NULL,NULL);
pFormatCtx->oformat = fmt;
if (avio_open(&pFormatCtx->pb,myJpeg_file, AVIO_FLAG_READ_WRITE) < 0)
{
this->Print2TextBox1(L"Couldn't open output file.");
}
video_st = avformat_new_stream(pFormatCtx, 0);
if (video_st==NULL)
{
this->Print2TextBox1(L"avformat_new_stream.");
}
pCodecCtx = video_st->codec;
pCodecCtx->codec_id = fmt->video_codec;
pCodecCtx->codec_type = AVMEDIA_TYPE_VIDEO;
pCodecCtx->pix_fmt = AV_PIX_FMT_YUVJ420P;
pCodecCtx->width = in_width;
pCodecCtx->height = in_height;
pCodecCtx->time_base.num = 1;
pCodecCtx->time_base.den = 1;//25;
this->Print2TextBox1(L"Conversion start\n");
//Output some information
av_dump_format(pFormatCtx, 0, myJpeg_file, 1);
// Determine if desired video encoder is installed
pCodec = avcodec_find_encoder(pCodecCtx->codec_id);
if (!pCodec)
{
this->Print2TextBox1(L"Codec not found.");
//return -1;
}
this->Print2TextBox1(L"Codec Identified done\n");
if (avcodec_open2(pCodecCtx, pCodec,NULL) < 0){
this->Print2TextBox1(L"Could not open codec.\n");
//return -1;
}
this->Print2TextBox1(L"Codec Open done\n");
//-----------------------------------------------
picture = av_frame_alloc();
size = avpicture_get_size(pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);
picture_buf = (uint8_t *)av_malloc(size);
if (!picture_buf)
{ this->Print2TextBox1(L"Size Allocation error\n");
//return -1;
}
avpicture_fill((AVPicture *)picture, picture_buf, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);
this->Print2TextBox1(L"Write Header..");
avformat_write_header(pFormatCtx,NULL);
y_size = pCodecCtx->width * pCodecCtx->height;
av_new_packet(&pkt,y_size*3);
//-------------------------------------------------------420 Format
//Read YUV
if (fread(picture_buf, 1, y_size*3/2, in_file) <=0)
{
this->Print2TextBox1(L"Could not read input file.");
//return -1;
}
//--------------------------------------------input image format UYVY
picture->data[0] = picture_buf; // Y
picture->data[1] = picture_buf+ y_size; // U
picture->data[2] = picture_buf+ y_size*5/4; // V
this->Print2TextBox1(L" Encode the image..\n");
ret = avcodec_encode_video2(pCodecCtx, &pkt,picture, &got_picture);
if(ret < 0)
{
this->Print2TextBox1(L"Encode Error.\n");
//return -1;
}
if (got_picture==1)
{
pkt.stream_index = video_st->index;
//#### PROBLEM IN THIS LINE BELOW WHEN RE-EXECUTE THE CODE ###
ret = av_write_frame(pFormatCtx, &pkt);
}
av_free_packet(&pkt);
//Write Trailer
av_write_trailer(pFormatCtx);
this->Print2TextBox1(L"Encode Successful.\n");
if (video_st)
{
avcodec_close(video_st->codec);
av_free(picture);
av_free(picture_buf);
}
avio_close(pFormatCtx->pb);
avformat_free_context(pFormatCtx);
fclose(in_file);
it seems that some of memory is not yet free or when i am trying to re-use this above code in second time in a loop,
plz suggest/guide me where i am doing wrong and not freeing up the memory..?
i am trying to Display Image (Current/updated) on every button press in VC++2010
You are allocating AVFrame
picture = av_frame_alloc();
, but casting it to deprecated AVPicture later:
avpicture_fill((AVPicture *)picture, picture_buf, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height);
solved by renaming the "encoded_pic_420.jpg" file with some other name and display it on PictureBox
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363851(v=vs.85).aspx
I have an animation/sprite developed in C++ on SDL2 libs (based on this answer). The bitmaps are saved to a certain path. They are of dimensions 640x480 and format is given by the SDL constant SDL_PIXELFORMAT_ARGB8888.
I have a second program written in C on top of FFmpeg libs, which reads one image from the above path (just one for the time being, will read the whole series when it works for just one).
This does the following (in gist - skipping validation & comments for conciseness)
AVCodec *codec;
AVCodecContext *c = NULL;
int i, ret, x, y, got_output;
FILE *f;
AVFrame *frame;
AVPacket pkt;
uint8_t endcode[] = { 0, 0, 1, 0xb7 };
codec = avcodec_find_encoder(codec_id);
c = avcodec_alloc_context3(codec);
c->bit_rate = 400000;
/* resolution must be a multiple of two */
c->width = 640;
c->height = 480;
c->time_base = (AVRational ) { 1, 25 };
c->gop_size = 5;
c->max_b_frames = 1;
c->pix_fmt = AV_PIX_FMT_YUV420P;
av_opt_set(c->priv_data, "preset", "slow", 0);
avcodec_open2(c, codec, NULL);
fopen(filename, "wb");
frame = av_frame_alloc();
av_image_alloc(frame->data, frame->linesize, c->width, c->height, c->pix_fmt, 32);
for (i = 0; i < 25; ++i) {
readSingleFile("/tmp/alok1/ss099.bmp", &frame->data);//Read the saved BMP into frame->data
frame->pts = i;
frame->width = 640;
frame->height = 480;
frame->format = -1;
av_init_packet(&pkt);
pkt.data = NULL; // packet data will be allocated by the encoder
pkt.size = 0;
ret = avcodec_encode_video2(c, &pkt, frame, &got_output);
if (got_output) {
printf("Write frame %3d (size=%5d)\n", i, pkt.size);
fwrite(pkt.data, 1, pkt.size, f);
}
av_packet_unref(&pkt);
}
for (got_output = 1; got_output; i++) {
fflush(stdout);
ret = avcodec_encode_video2(c, &pkt, NULL, &got_output);
if (ret < 0) {
fprintf(stderr, "Error encoding frame\n");
exit(1);
}
if (got_output) {
printf("[DELAYED]Write frame %3d (size=%5d)\n", i, pkt.size);
fwrite(pkt.data, 1, pkt.size, f);
av_packet_unref(&pkt);
}
}
fwrite(endcode, 1, sizeof(endcode), f);
//cleanup
As a result of the above code(which compiles without trouble), I can get a video which plays for 1 second - this part is working as expected. Problem is that the image seen is a green full screen like below.
The image that is being read using the readSingleImage(...) function is rendered by image viewer(linux, gwenview and okular) as follows:
Any pointers as to what could be going wrong?
To sum up the comments:
Encoder expects raw image data in format specified upon opening; it will not try to convert anything
Colorspace/format conversion has to be done manually; use swscale
if you are using Windows API to load image: it uses BGR, not RGB
BMP files usually but not always store image bottom-up as opposed to top-down used by FFmpeg; if it is bottom-up then it has to be flipped (there might be a way to do that without much or any performance hit if combined with colorspace conversion).
Also keep an eye on linesizes. Each line in an image can occupy more bytes than its width. This applies both to images allocated by ffmpeg and to images loaded from BMP - one has to be careful to always provide valid linesizes to each API call.
In addition to the above, below are "must-reads" for the final solution:
1. Taking a screenshot with SDL
2. RGB to YUV conversion
3. FFmpeg-related source code was written with this as base
I'm working on a Project in FFMPEG, and right now i'm stuck on a Problem.
What i am trying to do is, convert a png pic in to a mpeg Video file. I've already managed to take informatin from the Picture, but somehow i cannot convert the Picture in YUV Format. It returns "0x0->0x0 is invalid scaling dimensions".
Here'S my code:
AVFrame *pFrame;
AVFrame *pFrameYUV;
pFrame = av_frame_alloc();
pFrameYUV = av_frame_alloc();
int numBytes;//Groesse des Bildes
uint8_t *buffer= NULL;
numBytes=avpicture_get_size(AV_PIX_FMT_RGB24, pCodecCtx->width,pCodecCtx->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
/*avpicture_fill((AVPicture *)pFrameYUV, buffer, AV_PIX_FMT_YUV420P,
pCodecCtx->width, pCodecCtx->height);*/
av_image_fill_arrays(pFrameYUV->data,pFrameYUV->linesize,buffer,AV_PIX_FMT_YUV420P,pCodecCtx->width,pCodecCtx->height,32);
struct SwsContext *sws_ctx = NULL;
AVPacket packet;
// initialize SWS context for software scaling
sws_ctx=sws_getCachedContext(NULL,pFrame->width,pFrame->height,AV_PIX_FMT_RGB24,pFrameYUV->width,pFrameYUV->height,AV_PIX_FMT_YUV420P,0,0,0,0);
pFrameYUV->height= pFrame->height;
pFrameYUV->width= pFrame->width;
while (av_read_frame(pFormatCtx,&packet)>=0)
{
if(packet.stream_index == videoStream)
{
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished)
{
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameYUV->data, pFrameYUV->linesize);
printf("%d",pFrameYUV->height);
}
}
av_free_packet(&packet);
}
EDIT:
After the converting, i tried to encode the Frame in o a packet, but he size of the packet is 0.
Code
AVPacket pkt;
av_init_packet(&pkt);
pkt.stream_index= st->index;
pkt.data= buffer;
pkt.size=numBytes;
int got_pkt;
test=avcodec_encode_video2(st->codec,&pkt,pFrameYUV,&got_pkt);
printf("%d",got_pkt);
These values pFrameYUV->height, pFrame->height, pFrameYUV->width, pFrame->width didn't set when calling sws_getCachedContext.
Do you mean the dimension not changing? If so, set them before sws_getCachedContext.
pFrameYUV->height = pFrame->height = pCodecCtx->height;
pFrameYUV->width = pFrame->width = pCodecCtx->width;
sws_ctx=sws_getCachedContext(NULL,pFrame->width,pFrame->height,AV_PIX_FMT_RGB24,pFrameYUV->width,pFrameYUV->height,AV_PIX_FMT_YUV420P,0,0,0,0);
I'm using ffmpeg to decode a png picture and use the AVFrame as a opengl texture.
But the strangest thing is that I can get the png converted to opengl texture nicely on a iphone simulator, but I got a blank texture on a real iphone.
on both simulator and iphone, I got a null pointer for AVFrame's data
avcodec_decode_video2(codecContext/* AVCodecContext* */,frame /* AVFrame */,&finished,&tempPacket);
Then I covert the color space to AV_PIX_FMT_RGBA
void convertToRGBColor()
{
int numBytes = avpicture_get_size(
AV_PIX_FMT_RGBA,
codecContext->width,
codecContext->height);
uint8_t *buffer = (uint8_t *)av_malloc(numBytes);
avpicture_fill(rgbFrame/* AVFrame* */, buffer, AV_PIX_FMT_RGBA, codecContext->width, codecContext->height);
struct SwsContext *img_convert_ctx = NULL;
img_convert_ctx = sws_getCachedContext(
img_convert_ctx,
codecContext->width,
codecContext->height,
codecContext->pix_fmt,
codecContext->width,
codecContext->height,
COLOR_SPACE,
SWS_BILINEAR,
NULL,
NULL,
NULL);
if( !img_convert_ctx )
{
fprintf(stderr, "Cannot initialize sws conversion context\n");
}
sws_scale(img_convert_ctx,
frame->data,
frame->linesize,
0,
codecContext->height,
rgbFrame->data,
rgbFrame->linesize);
sws_freeContext(img_convert_ctx);
}
On a simulator, rgbFrame's data[0] will be a valid pointer, but on a iphone, it's null.
So, does anyone had the same problem before?