FFmpeg AVERROR(EAGAIN) error when call avcodec receive for h264 - c++

I'm working with ffmpeg 4.1 and I'm showing live streams of multiple cameras, h264 and h265.
My program collects packets of the same frame and then calls decodeVideo function. Actually it sends all packets of a frame at once.
Program works well if there is no missing packets. When I remove packet in random I-frames, both h264 and h265 streams work as expected (jumps some seconds but continues streaming).
When I remove packet in random P-frame from h265 streams, avcodec_send_packet function gives AVERROR_INVALIDDATA and streams continue.
However when I remove packet in random P-frame from h264 streams, avcodec_send_packet function gives 0. Then avcodec_receive_frame function gives AVERROR(EAGAIN) continuously and streams freeze.
void decodeVideo(array<uint8_t>^ data, int length, AvFrame^ finishedFrame)
{
AVPacket* videoPacket = new AVPacket();
av_init_packet(videoPacket);
pin_ptr<unsigned char> dataPtr = &data[0];
videoPacket->data = dataPtr;
videoPacket->size = length;
int retVal = avcodec_send_packet((AVCodecContext*)context, videoPacket);
if(retVal < 0)
{
if (retVal == AVERROR_EOF)
Utility::Log->ErrorFormat("avcodec_send_packet() return value is AVERROR_EOF.");
else if( retVal == AVERROR_INVALIDDATA)
Utility::Log->ErrorFormat("avcodec_send_packet() INVALID DATA!");
else
Utility::Log->ErrorFormat("avcodec_send_packet() return value is negative:{0}",retVal);
}
else
{
int receive_frame = avcodec_receive_frame((AVCodecContext*)context, (AVFrame*)finishedFrame);
if (receive_frame == AVERROR(EAGAIN))
Utility::Log->ErrorFormat("avcodec_receive_frame() returns AVERROR(EAGAIN)");
else if(receive_frame == AVERROR_EOF)
Utility::Log->ErrorFormat("avcodec_receive_frame() returns AVERROR(AVERROR_EOF)");
else
Utility::Log->ErrorFormat("avcodec_receive_frame() return value is negative:{0}",receive_frame);
}
av_packet_unref(videoPacket);
delete videoPacket;
}
EDIT
When I add avcodec_flush_buffers like shown, my problem is temporarily solved. However it freeze again after a while.
if(receive_frame == AVERROR(EAGAIN))
{
Utility::Log->ErrorFormat("avcodec_receive_frame() returns AVERROR(EAGAIN)");
avcodec_flush_buffers((AVCodecContext*)context);
}
Tested with ffmpeg version 4.1.1 same results.
Find an ffmpeg version like 2.5 decode function is different but there is no problem when i remove packets. However I'm working with h265 streams too.
EDIT2
AVCodecID id = AVCodecID::AV_CODEC_ID_H264;
AVCodec* dec = avcodec_find_decoder(id);
AVCodecContext* decContext = avcodec_alloc_context3(dec);
After these lines, my code included the following lines. When i delete them, there is no problem now.
if(dec->capabilities & AV_CODEC_CAP_TRUNCATED)
decContext->flags |= AV_CODEC_FLAG_TRUNCATED;
decContext->flags2 |= AV_CODEC_FLAG2_CHUNKS;

Related

Decoding with OGG/Vorbis gives no sound

I'd like to play an Ogg/Vorbis audio/video file, but right now I can't get to read audio from a file.
My algorithm to read audio is:
Initialize required structures:
vorbis_info info;
vorbis_comment comment;
vorbis_dsp_state dsp;
vorbis_block block;
vorbis_info_init(&info);
vorbis_comment_init(&comment);
Read headers:
Call vorbis_synthesis_headerin(&info, &comment, packet); until it returns OV_ENOTVORBIS
vorbis_synthesis_init(&dsp, &info);
vorbis_block_init(&dsp, &block);
Pass the first non-header packet to function below
Parse packets, do it until audioReady == READY
putPacket(ogg_packet *packet) {
int ret;
ret = vorbis_synthesis(&block, packet);
if( ret == 0 ) {
ret = vorbis_synthesis_blockin(&dsp, &block);
audioReady = (ret == 0) ? READY : NOT_READY;
} else {
audioReady = NOT_READY;
}
}
Read audio data:
float** rawData = nullptr;
readSamples = vorbis_synthesis_pcmout(&dsp, &rawData);
if( readSamples == 0 ) {
audioReady = NOT_READY;
return;
}
int16_t* newData = new int16_t[readSamples * getChannels()];
int16_t* dst = newData;
for(unsigned int i=0; i<readSamples; ++i) {
for(unsigned char ch=0; ch<getChannels(); ++ch) {
*(dst++) = math::clamp<int16_t>(rawData[ch][i]*32767 + 0.5f, -32767, 32767);
}
}
audioData.push_back({readSamples * getChannels() , newData});
vorbis_synthesis_read(&dsp, static_cast<int>(readSamples));
audioReady = NOT_READY;
This is where it gets wrong: after examining the newData contents it is revealed that it contains a very silent sound. I doubt if it is the right data which means somewhere along my algorithm I did something wrong.
I tried to find some examples of similar programs, but all I got are sources with very spaghetti-like code, which seems to do the same algorithm like mine, yet they do their job. (There is one off such library: https://github.com/icculus/theoraplay )
Is there any reason why I'm getting (almost) silence in my application?
PS: If you are wondering if I might getting OGG packets wrong, then I assure you this part of my code is working right, as I'm also reading video data from the same file, using the same code and it shows the video right.
I've found it: during reading packets I assumed that one Ogg Page = one Ogg packet. I's wrong: for audio one page can contain many packets. To read it properly one has to make a code like:
do{
putPacket(&packet);
}while( ogg_stream_packetout(&state, &packet) == 1 );
I did this mistake because for video packets (which I did first) a page contains only one packet.

How to Skip frames while decoding H264 stream?

I'm using FFMPEG to decode H264 (or H265) RTSP Stream.
My system have 2 software: Server and Client
Server: Read frames from RTSP stream --> Forward frames to Client
Client: Receive frames from Server --> Decode --> Render
I have implemented and it worked ok, but there is a case make my system work not good. That is when internet from Server - Client is slow, frames can not transfer real-time to Client.
In present, I deal with this issue by Skip some frames (not send to Client) when the Queue is reached limit of count. The following is my summary code
//At Server Software (include 2 threads A and B)
//Thread A: Read AVPacket and forward to Client
while(true)
{
AVPacket packet;
av_init_packet(&packet);
packet.size = 0;
packet.data = NULL;
int ret = AVERROR(EAGAIN);
while (AVERROR(EAGAIN) == ret)
ret = av_read_frame(pFormatCtx, &packet);
if(packet.size > 0)
{
if(mySendQueue.count < 120) //limit 120 packet in queue
mySendQueue.Enqueue(packet); ////Thread B will read from this queue, to send packets to Client via TCP socket
else
;//SkipThisFrame ***: No send
}
}
//Thread B: Send To Client via TCP Socket
While(true)
{
AVPacket packet;
if(mySendQueue.Dequeue(packet))
{
SendPacketToClient(packet);
}
}
//At Server Software : Receive AVPacket from Server --> Decode --> Render
While(true)
{
AVPacket packet;
AVFrame frame;
ReadPacketFromServer(packet);
if (av_decode_asyn(pCodecCtx, &frame, &frameFinished, &packet) == RS_OK)
{
if (frameFinished)
{
RenderFrame(frame);
}
}
}
UINT32 __clrcall av_decode_asyn(AVCodecContext *pCodecCtx, AVFrame *frame, int *frameFinished, AVPacket *packet)
{
int ret = -1;
*frameFinished = 0;
if (packet)
{
ret = avcodec_send_packet(pCodecCtx, packet);
// In particular, we don't expect AVERROR(EAGAIN), because we read all
// decoded frames with avcodec_receive_frame() until done.
if (ret < 0 && ret != AVERROR_EOF)
return RS_NOT_OK;
}
ret = avcodec_receive_frame(pCodecCtx, frame);
if (ret < 0 && ret != AVERROR(EAGAIN))
{
return RS_NOT_OK;
}
if (ret >= 0)
*frameFinished = 1;
return RS_OK;
}
My question is focus in line of code SkipThisFrame ***, this algorithm skip frame continuously, so it maybe make the decoder on Client occur unexpectedly error or Crash?
And when skip frame like that, make Client Render frames is not normally?
And someone call show me the proper algorithm to skip frames in my case?
Thank you very much!
I have a brief read on doc of AVPacket, it says:
For video, it should typically contain one compressed frame.
Theoretically you cannot skip frames for a compressed video stream, as most frames do not contain complete information about that frame's image, but only contain changes compared with some previous frames. So if you skip a frame, it is probable that many trailing decoded frames won't contain correct result (until next key frame flushes whole image).
"My question is focus in line of code SkipThisFrame ***, this algorithm
skip frame continuously, so it maybe make the decoder on Client occur
unexpectedly error or Crash?"
One thing I notice is wrong...
Your While(true) statements also need a break; to stop, otherwise they will run forever, blocking other functions and causing the system to crash. Think about it, you say "While the loop is true do X-Y-Z instructions" but you never say when to stop (eg: break out of this While loop to do next instructions). Computer is stuck doing first While loop only and also repeating that to infinity...
Try setting up like this:
//At Server Software (include 2 threads A and B)
//Thread A: Read AVPacket and forward to Client
while(true)
{
AVPacket packet;
av_init_packet(&packet);
packet.size = 0;
packet.data = NULL;
int ret = AVERROR(EAGAIN);
while (AVERROR(EAGAIN) == ret) { ret = av_read_frame(pFormatCtx, &packet); }
if(packet.size > 0)
{
if(mySendQueue.count < 120) //limit 120 packet in queue
{
mySendQueue.Enqueue(packet); ////Thread B will read from this queue, to send packets to Client via TCP socket
}
//else { } //no need for ELSE if doing nothing... //SkipThisFrame ***: No send
}
break; //stop this part and move to "Thead B"
}
//Thread B: Send To Client via TCP Socket
While(true)
{
AVPacket packet;
if( mySendQueue.Dequeue(packet) )
{ SendPacketToClient(packet); break; }
}
//At Server Software : Receive AVPacket from Server --> Decode --> Render
While(true)
{
AVPacket packet; AVFrame frame;
ReadPacketFromServer(packet);
if (av_decode_asyn(pCodecCtx, &frame, &frameFinished, &packet) == RS_OK)
{
if (frameFinished) { RenderFrame(frame); break; }
}
}
UINT32 __clrcall av_decode_asyn(AVCodecContext *pCodecCtx, AVFrame *frame, int *frameFinished, AVPacket *packet)
{
int ret = -1;
*frameFinished = 0;
if (packet)
{
ret = avcodec_send_packet(pCodecCtx, packet);
// In particular, we don't expect AVERROR(EAGAIN), because we read all
// decoded frames with avcodec_receive_frame() until done.
if (ret < 0 && ret != AVERROR_EOF)
return RS_NOT_OK;
}
ret = avcodec_receive_frame(pCodecCtx, frame);
if (ret < 0 && ret != AVERROR(EAGAIN))
{
return RS_NOT_OK;
}
if (ret >= 0)
*frameFinished = 1;
return RS_OK;
}
Hope it helps. Let me know of results / errors.

Replacing av_read_frame() to reduce delay

I am implementing a (very) low latency video streaming C++ application using ffmpeg. The client receives a video which is encoded with x264’s zerolatency preset, so there is no need for buffering. As described here, if you use av_read_frame() to read packets of the encoded video stream, you will always have at least one frame delay because of internal buffering done in ffmpeg. So when I call av_read_frame() after frame n+1 has been sent to the client, the function will return frame n.
Getting rid of this buffering by setting the AVFormatContext flags AVFMT_FLAG_NOPARSE | AVFMT_FLAG_NOFILLIN as suggested in the source disables packet parsing and therefore breaks decoding, as noted in the source.
Therefore, I am writing my own packet receiver and parser. First, here are the relevant steps of the working solution (including one frame delay) using av_read_frame():
AVFormatContext *fctx;
AVCodecContext *cctx;
AVPacket *pkt;
AVFrame *frm;
//Initialization of AV structures
//…
//Main Loop
while(true){
//Receive packet
av_read_frame(fctx, pkt);
//Decode:
avcodec_send_packet(cctx, pkt);
avcodec_receive_frame(cctx, frm);
//Display frame
//…
}
And below is my solution, which mimics the behavior of av_read_frame(), as far as I could reproduce it. I was able to track the source code of av_read_frame() down to ff_read_packet(),but I cannot find the source of AVInputformat.read_packet().
int tcpsocket;
AVCodecContext *cctx;
AVPacket *pkt;
AVFrame *frm;
uint8_t recvbuf[(int)10e5];
memset(recvbuf,0,10e5);
int pos = 0;
AVCodecParserContext * parser = av_parser_init(AV_CODEC_ID_H264);
parser->flags |= PARSER_FLAG_COMPLETE_FRAMES;
parser->flags |= PARSER_FLAG_USE_CODEC_TS;
//Initialization of AV structures and the tcpsocket
//…
//Main Loop
while(true){
//Receive packet
int length = read(tcpsocket, recvbuf, 10e5);
if (length >= 0) {
//Creating temporary packet
AVPacket * tempPacket = new AVPacket;
av_init_packet(tempPacket);
av_new_packet(tempPacket, length);
memcpy(tempPacket->data, recvbuf, length);
tempPacket->pos = pos;
pos += length;
memset(recvbuf,0,length);
//Parsing temporary packet into pkt
av_init_packet(pkt);
av_parser_parse2(parser, cctx,
&(pkt->data), &(pkt->size),
tempPacket->data, tempPacket->size,
tempPacket->pts, tempPacket->dts, tempPacket->pos
);
pkt->pts = parser->pts;
pkt->dts = parser->dts;
pkt->pos = parser->pos;
//Set keyframe flag
if (parser->key_frame == 1 ||
(parser->key_frame == -1 &&
parser->pict_type == AV_PICTURE_TYPE_I))
pkt->flags |= AV_PKT_FLAG_KEY;
if (parser->key_frame == -1 && parser->pict_type == AV_PICTURE_TYPE_NONE && (pkt->flags & AV_PKT_FLAG_KEY))
pkt->flags |= AV_PKT_FLAG_KEY;
pkt->duration = 96000; //Same result as in av_read_frame()
//Decode:
avcodec_send_packet(cctx, pkt);
avcodec_receive_frame(cctx, frm);
//Display frame
//…
}
}
I checked the fields of the resulting packet (pkt) just before avcodec_send_packet() in both solutions. They are as far as I can tell identical. The only difference might be the actual content of pkt->data. My solution decodes I-Frames fine, but the references in P-Frames seem to be broken, causing heavy artifacts and error messages such as “invalid level prefix”, “error while decoding MB xx”, and similar.
I would be very grateful for any hints.
Edit 1: I have developed a workaround for the time being: in the video server, after sending the packet containing the encoded data of a frame, I send one dummy packet which only contains the delimiters marking beginning and end of the packet. This way, I push the actual video data frames through av_read_frame(). I discard the dummy packets immediately after av_frame_read().
Edit 2: Solved here by rom1v, as written in his comment to this question.
av_parser_parse2() does not neccessarily consume your tempPacket in one go. You have to call it in another loop and check its return value, like in the API docs.

C++ ffmpeg video missing frames and won't play in Quicktime

I wrote some C++ code that uses ffmpeg to encode a video. I'm having two strange issues:
The final video is always missing 1 frame. That is, if I have it encode 10 frames the final video only has 9 (at least that's what ffprobe -show_frames -pretty $VIDEO | grep -F '[FRAME]' | wc -l tells me.
The final video plays fine in some players (mpv and vlc) but not in Quicktime. Quicktime just shows a completely black screen.
My code is roughly this (modified a bit to remove types that are unique to our code base):
First, I open the video file, write the headers and initialize things:
template <class PtrT>
using UniquePtrWithDeleteFunction = std::unique_ptr<PtrT, std::function<void (PtrT*)>>;
std::unique_ptr<FfmpegEncodingFrameSink> FfmpegEncodingFrameSink::Create(
const std::string& dest_url) {
AVFormatContext* tmp_format_ctxt;
auto alloc_format_res = avformat_alloc_output_context2(&tmp_format_ctxt, nullptr, "mp4", dest_url.c_str());
if (alloc_format_res < 0) {
throw FfmpegException("Error opening output file.");
}
auto format_ctxt = UniquePtrWithDeleteFunction<AVFormatContext>(
tmp_format_ctxt, CloseAvFormatContext);
AVStream* out_stream_video = avformat_new_stream(format_ctxt.get(), nullptr);
if (out_stream_video == nullptr) {
throw FfmpegException("Could not create outputstream");
}
auto codec_context = GetCodecContext(options);
out_stream_video->time_base = codec_context->time_base;
auto ret = avcodec_parameters_from_context(out_stream_video->codecpar, codec_context.get());
if (ret < 0) {
throw FfmpegException("Failed to copy encoder parameters to outputstream");
}
if (!(format_ctxt->oformat->flags & AVFMT_NOFILE)) {
ret = avio_open(&format_ctxt->pb, dest_url.c_str(), AVIO_FLAG_WRITE);
if (ret < 0) {
throw VideoDecodeException("Could not open output file: " + dest_url);
}
}
ret = avformat_init_output(format_ctxt.get(), nullptr);
if (ret < 0) {
throw FfmpegException("Unable to initialize the codec.");
}
ret = avformat_write_header(format_ctxt.get(), nullptr);
if (ret < 0) {
throw FfmpegException("Error occurred writing format header");
}
return std::unique_ptr<FfmpegEncodingFrameSink>(
new FfmpegEncodingFrameSink(std::move(format_ctxt), std::move(codec_context)));
}
Then, every time I get a new frame to encode I pass it to this function (the frames are being decoded via ffmpeg from another mp4 file which Quicktime plays just fine):
// If frame == nullptr then we're done and we're just flushing the encoder
// otherwise encode an actual frame
void FfmpegEncodingFrameSink::EncodeAndWriteFrame(
const AVFrame* frame) {
auto ret = avcodec_send_frame(codec_ctxt_.get(), frame);
if (ret < 0) {
throw FfmpegException("Error encoding the frame.");
}
AVPacket enc_packet;
enc_packet.data = nullptr;
enc_packet.size = 0;
av_init_packet(&enc_packet);
do {
ret = avcodec_receive_packet(codec_ctxt_.get(), &enc_packet);
if (ret == AVERROR(EAGAIN)) {
CHECK(frame != nullptr);
break;
} else if (ret == AVERROR_EOF) {
CHECK(frame == nullptr);
break;
} else if (ret < 0) {
throw FfmpegException("Error putting the encoded frame into the packet.");
}
assert(ret == 0);
enc_packet.stream_index = 0;
LOG(INFO) << "Writing packet to stream.";
av_interleaved_write_frame(format_ctxt_.get(), &enc_packet);
av_packet_unref(&enc_packet);
} while (ret == 0);
}
Finally, in my destructor I close everything up like so:
FfmpegEncodingFrameSink::~FfmpegEncodingFrameSink() {
// Pass a nullptr to EncodeAndWriteFrame so it flushes the encoder
EncodeAndWriteFrame(nullptr);
// write mp4 trailer
av_write_trailer(format_ctxt_.get());
}
If I run this passing n frames to EncodeAndWriteFrame line LOG(INFO) << "Writing packet to stream."; gets run n times indicating the n packets were written to the stream. But ffprobe always shows only n - 1 frames int he video. And the final video doesn't play on quicktime.
What am I doing wrong??
Sorry for the delay but as i just had the same problem and noticed that this question deserves an answer, here how i solved this.
Up in front, the Problem only occured for me when using mov, mp4, 3gp as format. It worked frame accurate when using e.g. avi format. When i wrote uncompressed video frames to the container, i saw that the avi and mov had the same count of frames stored but the mov obviously had some problem in it's header.
Counting the number of frames in the mov using header metadata showed one frame is missing:
ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 c:\temp\myinput.mov
While ignoring the index showed the correct number of frames:
-ignore_editlist 1
The solution for me was, set the timebase to the AVStream->CodeContext of the video stream.
The code above attempts to do this in this line:
out_stream_video->time_base = codec_context->time_base;
But the problem is that the posted code above does not expose the function GetCodecContext so we do not know if the time_base is correctly set for "codec_context". So it is my believe that the author's problem was that his function GetCodecContext did not set the time_base correctly.

How to read YUV8 data from avi file?

I have avi file that contains uncompressed gray video data. I need to extract frames from it. The size of file is 22 Gb.
How do i do that?
I have already tried ffmpeg, but it gives me "could not find codec parameters for video stream" message - because there is no codec at work, just frames.
Since Opencv just uses ffmpeg to read video, that rules out opencv as well.
The only path that seems to be left is to try and dig into the raw data, but i do not know how.
Edit: this is the code i use to read from the file with opencv. The failure occurs inside the second if. Running ffmpeg binary on the file also fails with the message above (could not find codec aprameters etc)
/* register all formats and codecs */
av_register_all();
/* open input file, and allocate format context */
if (avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) < 0) {
fprintf(stderr, "Could not open source file %s\n", src_filename);
ret = 1;
goto end;
}
fmt_ctx->seek2any = true;
/* retrieve stream information */
int res = avformat_find_stream_info(fmt_ctx, NULL);
if (res < 0) {
fprintf(stderr, "Could not find stream information\n");
ret = 1;
goto end;
}
Edit:
Here is sample code i have tried to make the extraction: pastebin. The result i get is an unchanging buffer after every call to AVIStreamRead.
If you do not need cross platform functionality Video for Windows (VFW) API is a good alternative (http://msdn.microsoft.com/en-us/library/windows/desktop/dd756808(v=vs.85).aspx), i will not put an entire code block, since there's quite much to do, but you should be able to figure it out from the reference link. Basically, you do a AVIFileOpen, then get the video stream via AVIFileGetStream with streamtypeVIDEO, or alternatively do it at once with AVIStreamOpenFromFile and then read samples from the stream with AVIStreamRead. If you get to a point where you fail I can try to help, but it should be pretty straightforward.
Also, not sure why ffmpeg is failing, I have been doing raw AVI reading with ffmpeg without any codecs involved, can you post what call to ffpeg actually fails?
EDIT:
For the issue that you are seeing when the read data size is 0. The AVI file has N slots for frames in each second where N is the fps of the video. In real life the samples won't come exactly at that speed (e.g. IP surveillance cameras) so the actual data sample indexes can be non continuous like 1,5,11,... and VFW would insert empty samples between them (that is from where you read a sample with a zero size). What you have to do is call AVIStreamRead with NULL as buffer and 0 as size until the bRead is not 0 or you run past last sample. When you get an actual size, then you can again call AVIStreamRead on that sample index with the buffer pointer and size. I usually do compressed video so i don't use the suggested size, but at least according to your code snipplet I would do something like this:
...
bRead = 0;
do
{
aviOpRes = AVIStreamRead(ppavi,smpS,1,NULL,0,&bRead,&smpN);
} while (bRead == 0 && ++smpS < si.dwLength + si.dwStart);
if(smpS >= si.dwLength + si.dwStart)
break;
PUCHAR tempBuffer = new UCHAR[bRead];
aviOpRes = AVIStreamRead(ppavi,smpS,1,tempBuffer,bRead,&bRead,&smpN);
/* do whatever you need */
delete tempBuffer;
...
EDIT 2:
Since this may come in handy to someone or yourself to make a choice between VFW and FFMPEG I also updated your FFMPEG example so that it parsed the same file (sorry for the code quality since it lacks error checking but i guess you can see the logical flow):
/* register all formats and codecs */
av_register_all();
AVFormatContext* fmt_ctx = NULL;
/* open input file, and allocate format context */
const char *src_filename = "E:\\Output.avi";
if (avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) < 0) {
fprintf(stderr, "Could not open source file %s\n", src_filename);
abort();
}
/* retrieve stream information */
int res = avformat_find_stream_info(fmt_ctx, NULL);
if (res < 0) {
fprintf(stderr, "Could not find stream information\n");
abort();
}
int video_stream_index = 0; /* video stream is usualy 0 but still better to lookup in case it's not present */
for(; video_stream_index < fmt_ctx->nb_streams; ++video_stream_index)
{
if(fmt_ctx->streams[video_stream_index]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
break;
}
if(video_stream_index == fmt_ctx->nb_streams)
abort();
AVPacket *packet = new AVPacket;
while(av_read_frame(fmt_ctx, packet) == 0)
{
if (packet->stream_index == video_stream_index)
printf("Sample nr %d\n", packet->pts);
av_free_packet(packet);
}
Basically you open the context and read packets from it. You will get both audio and video packets so you should check if the packet belongs to the stream of interest. FFMPEG will save you the trouble with empty frames and give only those samples that have data in them.