I'm using ffmpeg to decode a png picture and use the AVFrame as a opengl texture.
But the strangest thing is that I can get the png converted to opengl texture nicely on a iphone simulator, but I got a blank texture on a real iphone.
on both simulator and iphone, I got a null pointer for AVFrame's data
avcodec_decode_video2(codecContext/* AVCodecContext* */,frame /* AVFrame */,&finished,&tempPacket);
Then I covert the color space to AV_PIX_FMT_RGBA
void convertToRGBColor()
{
int numBytes = avpicture_get_size(
AV_PIX_FMT_RGBA,
codecContext->width,
codecContext->height);
uint8_t *buffer = (uint8_t *)av_malloc(numBytes);
avpicture_fill(rgbFrame/* AVFrame* */, buffer, AV_PIX_FMT_RGBA, codecContext->width, codecContext->height);
struct SwsContext *img_convert_ctx = NULL;
img_convert_ctx = sws_getCachedContext(
img_convert_ctx,
codecContext->width,
codecContext->height,
codecContext->pix_fmt,
codecContext->width,
codecContext->height,
COLOR_SPACE,
SWS_BILINEAR,
NULL,
NULL,
NULL);
if( !img_convert_ctx )
{
fprintf(stderr, "Cannot initialize sws conversion context\n");
}
sws_scale(img_convert_ctx,
frame->data,
frame->linesize,
0,
codecContext->height,
rgbFrame->data,
rgbFrame->linesize);
sws_freeContext(img_convert_ctx);
}
On a simulator, rgbFrame's data[0] will be a valid pointer, but on a iphone, it's null.
So, does anyone had the same problem before?
Related
I have an application that encodes a video stream coming from a camera in H264 and stores it on the disk. In another part of the application I'm loading this movie and showing it to the user. All of this works except the fact that the unique frame index number that I add when encoding the frames aren't the same as the frame numbers that I get when I decode the file.
this is a snippet of my code when I encode a frame. I use "display_picture_number" to store this unique frame number, is this correct? why is the number not the same?
void MovieCodec::createFrame( const MyImage& image, AVFrame* frame )
{
frame->format = streamPixelFormat;
frame->width = image.width();
frame->height = image.height();
frame->pict_type = AV_PICTURE_TYPE_P;
frame->display_picture_number = image.uniqueImageNumber();
int ret = av_image_alloc( frame->data, frame->linesize, frame->width, frame->height, AV_PIX_FMT_BGR24, 1);
if (ret < 0)
{
return;
}
struct SwsContext* sws_ctx = sws_getContext((int)image.width(), (int)image.height(), AV_PIX_FMT_BGR24,
(int)image.width(), (int)image.height(), streamPixelFormat,
0, NULL, NULL, NULL);
const uint8_t* rgbData[1] = { (uint8_t* )image.getData() };
int rgbLineSize[1] = { 3 * (int)image.width() };
sws_scale(sws_ctx, rgbData, rgbLineSize, 0, image.height(), frame->data, frame->linesize);
}
Check AVFrame structure documentation.
You cannot set display_picture_number as this is will not be processed by ffmpeg during encoding
You could use AVFrame 's metadata instead to store some "notes"
I am making a simple video player with ffmpeg. I have noticed that there is a memory leak originating in libavutil. Because ffmpeg is a mature library I assume that I am allocating a new frame incorrectly. The documentation is also vague about freeing the buffer that is created when you call av_frame_get_buffer(). Below is the code I am using to decode the video and queue it up for display on the UI thread.
DWORD WINAPI DecoderThread(LPVOID lpParam)
{
AVFrame *frame = NULL;
AVPacket pkt;
SwsContext *swsCtx = NULL;
UINT8 *buffer = NULL;
INT iNumBytes = 0;
INT result = 0;
frame = av_frame_alloc();
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;
// Create scaling context
swsCtx = sws_getContext(codecCtx->width, codecCtx->height, codecCtx->pix_fmt, codecCtx->width, codecCtx->height, AV_PIX_FMT_BGR24, SWS_BICUBIC, NULL, NULL, NULL);
while (av_read_frame(fmtCtx, &pkt) >= 0) {
if (pkt.stream_index == videoStream) {
result = avcodec_send_packet(codecCtx, &pkt);
while (result >= 0) {
result = avcodec_receive_frame(codecCtx, frame);
if (result == AVERROR(EAGAIN) || result == AVERROR_EOF) {
break;
} else if (result < 0) {
// another error.
}
// Create a new frame to store the RGB24 data.
AVFrame *pFrameRGB = av_frame_alloc();
// Allocate space for the new RGB image.
//av_image_alloc(pFrameRGB->data, pFrameRGB->linesize, codecCtx->width, codecCtx->height, AV_PIX_FMT_BGR24, 1);
// Copy all of the properties from the YUV420P frame.
av_frame_copy_props(pFrameRGB, frame);
pFrameRGB->width = frame->width;
pFrameRGB->height = frame->height;
pFrameRGB->format = AV_PIX_FMT_BGR24;
av_frame_get_buffer(pFrameRGB, 0);
// Convert fram from YUV420P to BGR24 for display.
sws_scale(swsCtx, (const UINT8* const *) frame->data, frame->linesize, 0, codecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
// Queue thr BGR frame for drawing by the main thread.
AddItemToFrameQueue(pFrameRGB);
av_frame_unref(frame);
}
}
while (GetQueueSize() > 100) {
Sleep(10);
}
}
CloseFrameQueue();
av_frame_free(&frame);
avcodec_close(codecCtx);
avformat_close_input(&fmtCtx);
return 0;
}
Is there a better way to allocate a new frame for holding the post sws_scale() transformation?
There is a similar stackoverflow question that uses mostly depreciated function calls. I can't seem to find any answers that conform to the new version of ffmpeg in the documentation. Any help would be appreciated.
Following the suggestions made in the comments I added a av_packet_unref() call to my decoding loop, and it stopped the memory leak issues I was having.
sws_scale(swsCtx, (const UINT8* const *) frame->data, frame->linesize, 0, codecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
// Queue thr BGR frame for drawing by the main thread.
AddItemToFrameQueue(pFrameRGB);
av_frame_unref(frame);
}
av_packet_unref(&pkt);
}
while (GetQueueSize() > 100) {
Sleep(10);
}
I have an animation/sprite developed in C++ on SDL2 libs (based on this answer). The bitmaps are saved to a certain path. They are of dimensions 640x480 and format is given by the SDL constant SDL_PIXELFORMAT_ARGB8888.
I have a second program written in C on top of FFmpeg libs, which reads one image from the above path (just one for the time being, will read the whole series when it works for just one).
This does the following (in gist - skipping validation & comments for conciseness)
AVCodec *codec;
AVCodecContext *c = NULL;
int i, ret, x, y, got_output;
FILE *f;
AVFrame *frame;
AVPacket pkt;
uint8_t endcode[] = { 0, 0, 1, 0xb7 };
codec = avcodec_find_encoder(codec_id);
c = avcodec_alloc_context3(codec);
c->bit_rate = 400000;
/* resolution must be a multiple of two */
c->width = 640;
c->height = 480;
c->time_base = (AVRational ) { 1, 25 };
c->gop_size = 5;
c->max_b_frames = 1;
c->pix_fmt = AV_PIX_FMT_YUV420P;
av_opt_set(c->priv_data, "preset", "slow", 0);
avcodec_open2(c, codec, NULL);
fopen(filename, "wb");
frame = av_frame_alloc();
av_image_alloc(frame->data, frame->linesize, c->width, c->height, c->pix_fmt, 32);
for (i = 0; i < 25; ++i) {
readSingleFile("/tmp/alok1/ss099.bmp", &frame->data);//Read the saved BMP into frame->data
frame->pts = i;
frame->width = 640;
frame->height = 480;
frame->format = -1;
av_init_packet(&pkt);
pkt.data = NULL; // packet data will be allocated by the encoder
pkt.size = 0;
ret = avcodec_encode_video2(c, &pkt, frame, &got_output);
if (got_output) {
printf("Write frame %3d (size=%5d)\n", i, pkt.size);
fwrite(pkt.data, 1, pkt.size, f);
}
av_packet_unref(&pkt);
}
for (got_output = 1; got_output; i++) {
fflush(stdout);
ret = avcodec_encode_video2(c, &pkt, NULL, &got_output);
if (ret < 0) {
fprintf(stderr, "Error encoding frame\n");
exit(1);
}
if (got_output) {
printf("[DELAYED]Write frame %3d (size=%5d)\n", i, pkt.size);
fwrite(pkt.data, 1, pkt.size, f);
av_packet_unref(&pkt);
}
}
fwrite(endcode, 1, sizeof(endcode), f);
//cleanup
As a result of the above code(which compiles without trouble), I can get a video which plays for 1 second - this part is working as expected. Problem is that the image seen is a green full screen like below.
The image that is being read using the readSingleImage(...) function is rendered by image viewer(linux, gwenview and okular) as follows:
Any pointers as to what could be going wrong?
To sum up the comments:
Encoder expects raw image data in format specified upon opening; it will not try to convert anything
Colorspace/format conversion has to be done manually; use swscale
if you are using Windows API to load image: it uses BGR, not RGB
BMP files usually but not always store image bottom-up as opposed to top-down used by FFmpeg; if it is bottom-up then it has to be flipped (there might be a way to do that without much or any performance hit if combined with colorspace conversion).
Also keep an eye on linesizes. Each line in an image can occupy more bytes than its width. This applies both to images allocated by ffmpeg and to images loaded from BMP - one has to be careful to always provide valid linesizes to each API call.
In addition to the above, below are "must-reads" for the final solution:
1. Taking a screenshot with SDL
2. RGB to YUV conversion
3. FFmpeg-related source code was written with this as base
I'm working on a Project in FFMPEG, and right now i'm stuck on a Problem.
What i am trying to do is, convert a png pic in to a mpeg Video file. I've already managed to take informatin from the Picture, but somehow i cannot convert the Picture in YUV Format. It returns "0x0->0x0 is invalid scaling dimensions".
Here'S my code:
AVFrame *pFrame;
AVFrame *pFrameYUV;
pFrame = av_frame_alloc();
pFrameYUV = av_frame_alloc();
int numBytes;//Groesse des Bildes
uint8_t *buffer= NULL;
numBytes=avpicture_get_size(AV_PIX_FMT_RGB24, pCodecCtx->width,pCodecCtx->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
/*avpicture_fill((AVPicture *)pFrameYUV, buffer, AV_PIX_FMT_YUV420P,
pCodecCtx->width, pCodecCtx->height);*/
av_image_fill_arrays(pFrameYUV->data,pFrameYUV->linesize,buffer,AV_PIX_FMT_YUV420P,pCodecCtx->width,pCodecCtx->height,32);
struct SwsContext *sws_ctx = NULL;
AVPacket packet;
// initialize SWS context for software scaling
sws_ctx=sws_getCachedContext(NULL,pFrame->width,pFrame->height,AV_PIX_FMT_RGB24,pFrameYUV->width,pFrameYUV->height,AV_PIX_FMT_YUV420P,0,0,0,0);
pFrameYUV->height= pFrame->height;
pFrameYUV->width= pFrame->width;
while (av_read_frame(pFormatCtx,&packet)>=0)
{
if(packet.stream_index == videoStream)
{
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished)
{
sws_scale(sws_ctx, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameYUV->data, pFrameYUV->linesize);
printf("%d",pFrameYUV->height);
}
}
av_free_packet(&packet);
}
EDIT:
After the converting, i tried to encode the Frame in o a packet, but he size of the packet is 0.
Code
AVPacket pkt;
av_init_packet(&pkt);
pkt.stream_index= st->index;
pkt.data= buffer;
pkt.size=numBytes;
int got_pkt;
test=avcodec_encode_video2(st->codec,&pkt,pFrameYUV,&got_pkt);
printf("%d",got_pkt);
These values pFrameYUV->height, pFrame->height, pFrameYUV->width, pFrame->width didn't set when calling sws_getCachedContext.
Do you mean the dimension not changing? If so, set them before sws_getCachedContext.
pFrameYUV->height = pFrame->height = pCodecCtx->height;
pFrameYUV->width = pFrame->width = pCodecCtx->width;
sws_ctx=sws_getCachedContext(NULL,pFrame->width,pFrame->height,AV_PIX_FMT_RGB24,pFrameYUV->width,pFrameYUV->height,AV_PIX_FMT_YUV420P,0,0,0,0);
I am trying to decode video encoded with H264. I am sending AVPacket's data and its size to decoder code. there I am trying to decode the frame and display it on a GUI. problem is when I am decoding the frame it is returning same number of frame byte as the size of packet means it is not decompressing the data. Can any one tell what will be the problem. My encoding program is working fine.
here is code for encoding
static struct SwsContext *img_convert_ctx;
pkt.data = NULL;
pkt.size = 0;
avpicture_fill((AVPicture *)srcFrame, frame,AV_PIX_FMT_BGR24, 640, 480);
if(img_convert_ctx == NULL) {
int w = 640;
int h = 480;
img_convert_ctx = sws_getContext(w, h,
AV_PIX_FMT_BGR24, c->width, c->height, PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL);
if(img_convert_ctx == NULL) {
fprintf(stderr, "Cannot initialize the conversion context!\n");
}
}
sws_scale(img_convert_ctx, srcFrame->data, srcFrame->linesize, 0,480,picture->data, picture->linesize);
fflush(stdout);
picture->pts=counter;
ret = avcodec_encode_video2(c, &pkt, picture, &got_output);
if (ret < 0) {
fprintf(stderr, "Error encoding frame\n");
}
if (got_output) {
vdec.decode_frame(pkt.data ,pkt.size);
av_free_packet(&pkt);
}
decoder code...
int len ,got_frame;
avpkt.size = data_length;
avpkt.data = frame_buffer;
if(!frame_buffer){
return "frame buffer empty\n";
}
len = avcodec_decode_video2(avctx ,frame ,&got_frame ,&avpkt);
if( len < 0){
return "error while decoding\n";
}
if( got_frame ){
static struct SwsContext *img_convert_ctx;
if(img_convert_ctx == NULL) {
img_convert_ctx = sws_getContext(w, h,
PIX_FMT_YUV420P, avctx->width,
avctx->height, PIX_FMT_BGR24,
SWS_BICUBIC, NULL, NULL, NULL);
if(img_convert_ctx == NULL) {
return "Cannot initialize the conversion context!\n";
}
}
j=sws_scale(img_convert_ctx,
frame->data , frame->linesize ,
0, h ,picture->data,
picture->linesize );
if(j==0){
exit(1);
}
I am initializing all other code like AVCodecContext and Codec into other method.
Please help me to find the solution.
The avcodec_decode_video2 function should return the number of bytes processed, not the number of bytes of result picture. You just have to check the value of got_frame to find out when decoded a complete frame.