Losing quality when encoding with ffmpeg - c++

I am using the c libraries of ffmpeg to read frames from a video and create an output file that is supposed to be identical to the input.
However, somewhere during this process some quality gets lost and the result is "less sharp". My guess is that the problem is the encoding and that the frames are too compressed (also because the size of the file decreases quite significantly). Is there some parameter in the encoder that allows me to control the quality of the result? I found that AVCodecContext has a compression_level member, but changing it that does not seem to have any effect.
I post here part of my code in case it could help. I would say that something must be changed in the init function of OutputVideoBuilder when I set the codec. The AVCodecContext that is passed to the method is the same of InputVideoHandler.
Here are the two main classes that I created to wrap the ffmpeg functionalities:
// This class opens the video files and sets the decoder
class InputVideoHandler {
public:
InputVideoHandler(char* name);
~InputVideoHandler();
AVCodecContext* getCodecContext();
bool readFrame(AVFrame* frame, int* success);
private:
InputVideoHandler();
void init(char* name);
AVFormatContext* formatCtx;
AVCodec* codec;
AVCodecContext* codecCtx;
AVPacket packet;
int streamIndex;
};
void InputVideoHandler::init(char* name) {
streamIndex = -1;
int numStreams;
if (avformat_open_input(&formatCtx, name, NULL, NULL) != 0)
throw std::exception("Invalid input file name.");
if (avformat_find_stream_info(formatCtx, NULL)<0)
throw std::exception("Could not find stream information.");
numStreams = formatCtx->nb_streams;
if (numStreams < 0)
throw std::exception("No streams in input video file.");
for (int i = 0; i < numStreams; i++) {
if (formatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
streamIndex = i;
break;
}
}
if (streamIndex < 0)
throw std::exception("No video stream in input video file.");
// find decoder using id
codec = avcodec_find_decoder(formatCtx->streams[streamIndex]->codec->codec_id);
if (codec == nullptr)
throw std::exception("Could not find suitable decoder for input file.");
// copy context from input stream
codecCtx = avcodec_alloc_context3(codec);
if (avcodec_copy_context(codecCtx, formatCtx->streams[streamIndex]->codec) != 0)
throw std::exception("Could not copy codec context from input stream.");
if (avcodec_open2(codecCtx, codec, NULL) < 0)
throw std::exception("Could not open decoder.");
}
// frame must be initialized with av_frame_alloc() before!
// Returns true if there are other frames, false if not.
// success == 1 if frame is valid, 0 if not.
bool InputVideoHandler::readFrame(AVFrame* frame, int* success) {
*success = 0;
if (av_read_frame(formatCtx, &packet) < 0)
return false;
if (packet.stream_index == streamIndex) {
avcodec_decode_video2(codecCtx, frame, success, &packet);
}
av_free_packet(&packet);
return true;
}
// This class opens the output and write frames to it
class OutputVideoBuilder{
public:
OutputVideoBuilder(char* name, AVCodecContext* inputCtx);
~OutputVideoBuilder();
void writeFrame(AVFrame* frame);
void writeVideo();
private:
OutputVideoBuilder();
void init(char* name, AVCodecContext* inputCtx);
void logMsg(AVPacket* packet, AVRational* tb);
AVFormatContext* formatCtx;
AVCodec* codec;
AVCodecContext* codecCtx;
AVStream* stream;
};
void OutputVideoBuilder::init(char* name, AVCodecContext* inputCtx) {
if (avformat_alloc_output_context2(&formatCtx, NULL, NULL, name) < 0)
throw std::exception("Could not determine file extension from provided name.");
codec = avcodec_find_encoder(inputCtx->codec_id);
if (codec == nullptr) {
throw std::exception("Could not find suitable encoder.");
}
codecCtx = avcodec_alloc_context3(codec);
if (avcodec_copy_context(codecCtx, inputCtx) < 0)
throw std::exception("Could not copy output codec context from input");
codecCtx->time_base = inputCtx->time_base;
codecCtx->compression_level = 0;
if (avcodec_open2(codecCtx, codec, NULL) < 0)
throw std::exception("Could not open encoder.");
stream = avformat_new_stream(formatCtx, codec);
if (stream == nullptr) {
throw std::exception("Could not allocate stream.");
}
stream->id = formatCtx->nb_streams - 1;
stream->codec = codecCtx;
stream->time_base = codecCtx->time_base;
av_dump_format(formatCtx, 0, name, 1);
if (!(formatCtx->oformat->flags & AVFMT_NOFILE)) {
if (avio_open(&formatCtx->pb, name, AVIO_FLAG_WRITE) < 0) {
throw std::exception("Could not open output file.");
}
}
if (avformat_write_header(formatCtx, NULL) < 0) {
throw std::exception("Error occurred when opening output file.");
}
}
void OutputVideoBuilder::writeFrame(AVFrame* frame) {
AVPacket packet = { 0 };
int success;
av_init_packet(&packet);
if (avcodec_encode_video2(codecCtx, &packet, frame, &success))
throw std::exception("Error encoding frames");
if (success) {
av_packet_rescale_ts(&packet, codecCtx->time_base, stream->time_base);
packet.stream_index = stream->index;
logMsg(&packet,&stream->time_base);
av_interleaved_write_frame(formatCtx, &packet);
}
av_free_packet(&packet);
}
This is the part of the main function that reads and write frames:
while (inputHandler->readFrame(frame,&gotFrame)) {
if (gotFrame) {
try {
outputBuilder->writeFrame(frame);
}
catch (std::exception e) {
std::cout << e.what() << std::endl;
return -1;
}
}
}

Your qmin/qmax answer is partially correct, but it misses the point, in that the quality indeed goes up, but the compression ratio (in terms of quality per bit) will suffer significantly as you restrict the qmin/qmax range - i.e. you will spend many more bits to accomplish the same quality than should really be necessary if you used the encoder optimally.
To increase quality without hurting the compression ratio, you need to actually increase the quality target. How you do this differs a little depending on the codec, but you typically increase the target CRF value or target bitrate. For commandline options, see e.g. the H264 docs. There's identical docs for HEVC/VP9 also. To use these options in the C API, use av_opt_set() with the same option names/values.

In case this could be useful to someone else, I add the answer that damjeux suggested, which worked for me. AVCodecContex has two members qmin and qmax which control the QP (quantization parameter) of the encoder. By default in my case qmin is 2 and qmax is 31. By setting qmax to a lower value the quality of the output improves.

Related

ffmpeg api alternate transcoding and remuxing for same file

Context
Hello !
I'm currently working on the development of a small library allowing to cut an h.264 video on any frame, but without re-encoding (transcoding) the whole video. The idea is to re-encode only the GOP on which we want to cut, and to rewrite (remux) directly the others GOP.
The avcut project (https://github.com/anyc/avcut) allows to do that, but requires a systematic decoding of each package, and seems to not work with the recent versions of ffmpeg from the tests I could do and from the recent feedbacks in the github issues.
As a beginner, I started from the code examples provided in the ffmpeg documentation, in particular: transcoding.c and remuxing.c.
Problem encountered
The problem I'm having is that I can't get both transcoding and remuxing to work properly at the same time. In particular, depending on the method I use to initialize the AVCodecParameters of the output video stream, transcoding works, or remuxing works:
avcodec_parameters_copy works well for remuxing
avcodec_parameters_from_context works well for transcoding
In case I choose avcodec_parameters_from_context, the transcoded GOP are correctly read by my video player (parole), but the remuxed packets are not read, and ffprobe does not show/detect them.
In case I choose avcodec_parameters_from_context, the remuxing GOP are correctly read by my video player, but the transcoding key_frame are bugged (I have the impression that the b-frame and p-frame are ok), and ffprobe -i return an error about the NAL of the key-frames:
[h264 # 0x55ec8a079300] sps_id 32 out of range
[h264 # 0x55ec8a079300] Invalid NAL unit size (1677727148 > 735).
[h264 # 0x55ec8a079300] missing picture in access unit with size 744
I suspect that the problem is related to the extradata of the packets. Through some experiments on the different attributes of the output AVCodecParameters, it seems that it is the extradata and extradata_size attributes that are responsible for the functioning of one method or the other.
Version
ffmpeg development branch retrieved on 2022-05-17 from https://github.com/FFmpeg/FFmpeg.
Compiled with --enable-libx264 --enable-gpl --enable-decoder=png --enable-encoder=png
Code
My code is written in c++ and is based on two classes: a class defining the parameters and methods on the input file (InputContexts) and a class defining them for the output file (OutputContexts). The code of these two classes is defined in the following files:
contexts.h
contexts.cpp
The code normally involved in the problem is the following:
stream initialization
int OutputContexts::init(const char* out_filename, InputContexts* input_contexts){
int ret;
int stream_index = 0;
avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, out_filename);
if (!ofmt_ctx) {
fprintf(stderr, "Could not create output context\n");
ret = AVERROR_UNKNOWN;
return ret;
}
av_dump_format(ofmt_ctx, 0, out_filename, 1);
encoders.resize(input_contexts->ifmt_ctx->nb_streams, nullptr);
codecs.resize(input_contexts->ifmt_ctx->nb_streams, nullptr);
// stream mapping
for (int i = 0; i < input_contexts->ifmt_ctx->nb_streams; i++) {
AVStream *out_stream;
AVStream *in_stream = input_contexts->ifmt_ctx->streams[i];
AVCodecContext* decoder_ctx = input_contexts->decoders[i];
// add new stream to output context
out_stream = avformat_new_stream(ofmt_ctx, NULL);
if (!out_stream) {
fprintf(stderr, "Failed allocating output stream\n");
ret = AVERROR_UNKNOWN;
return ret;
}
// from avcut blog
av_dict_copy(&out_stream->metadata, in_stream->metadata, 0);
out_stream->time_base = in_stream->time_base;
// encoder
if (decoder_ctx->codec_type == AVMEDIA_TYPE_VIDEO){
ret = prepare_encoder_video(i, input_contexts);
if (ret < 0){
fprintf(stderr, "Error while preparing encoder for stream #%u\n", i);
return ret;
}
// from avcut
out_stream->sample_aspect_ratio = in_stream->sample_aspect_ratio;
// works well for remuxing
ret = avcodec_parameters_copy(out_stream->codecpar, in_stream->codecpar);
if (ret < 0) {
fprintf(stderr, "Failed to copy codec parameters\n");
return ret;
}
// works well for transcoding
// ret = avcodec_parameters_from_context(out_stream->codecpar, encoders[i]);
// if (ret < 0) {
// av_log(NULL, AV_LOG_ERROR, "Failed to copy encoder parameters to output stream #%u\n", i);
// return ret;
// }
} else if (decoder_ctx->codec_type == AVMEDIA_TYPE_AUDIO) {
...
} else {
...
}
// TODO useful ???
// set current stream position to 0
// out_stream->codecpar->codec_tag = 0;
}
// opening output file in write mode with the ouput context
if (!(ofmt_ctx->oformat->flags & AVFMT_NOFILE)) {
ret = avio_open(&ofmt_ctx->pb, out_filename, AVIO_FLAG_WRITE);
if (ret < 0) {
fprintf(stderr, "Could not open output file '%s'", out_filename);
return ret;
}
}
// write headers from output context in output file
ret = avformat_write_header(ofmt_ctx, NULL);
if (ret < 0) {
fprintf(stderr, "Error occurred when opening output file\n");
return ret;
}
return ret;
}
AVCodecContext initialization for encoder
int OutputContexts::prepare_encoder_video(int stream_index, InputContexts* input_contexts){
int ret;
const AVCodec* encoder;
AVCodecContext* decoder_ctx = input_contexts->decoders[stream_index];
AVCodecContext* encoder_ctx;
if (video_index >= 0){
fprintf(stderr, "Impossible to mark stream #%u as video, stream #%u is already registered as video stream.\n",
stream_index, video_index);
return -1; //TODO change this value for correct error code
}
video_index = stream_index;
if(decoder_ctx->codec_id == AV_CODEC_ID_H264){
encoder = avcodec_find_encoder_by_name("libx264");
if (!encoder) {
av_log(NULL, AV_LOG_FATAL, "Encoder libx264 not found\n");
return AVERROR_INVALIDDATA;
}
fmt::print("Encoder libx264 will be used for stream {}.\n", stream_index);
} else {
std::string s = fmt::format("No video encoder found for the given codec_id: {}\n", avcodec_get_name(decoder_ctx->codec_id));
av_log(NULL, AV_LOG_FATAL, s.c_str());
return AVERROR_INVALIDDATA;
}
encoder_ctx = avcodec_alloc_context3(encoder);
if (!encoder_ctx) {
av_log(NULL, AV_LOG_FATAL, "Failed to allocate the encoder context\n");
return AVERROR(ENOMEM);
}
// from avcut
encoder_ctx->time_base = decoder_ctx->time_base;
encoder_ctx->ticks_per_frame = decoder_ctx->ticks_per_frame;
encoder_ctx->delay = decoder_ctx->delay;
encoder_ctx->width = decoder_ctx->width;
encoder_ctx->height = decoder_ctx->height;
encoder_ctx->pix_fmt = decoder_ctx->pix_fmt;
encoder_ctx->sample_aspect_ratio = decoder_ctx->sample_aspect_ratio;
encoder_ctx->color_primaries = decoder_ctx->color_primaries;
encoder_ctx->color_trc = decoder_ctx->color_trc;
encoder_ctx->colorspace = decoder_ctx->colorspace;
encoder_ctx->color_range = decoder_ctx->color_range;
encoder_ctx->chroma_sample_location = decoder_ctx->chroma_sample_location;
encoder_ctx->profile = decoder_ctx->profile;
encoder_ctx->level = decoder_ctx->level;
encoder_ctx->thread_count = 1; // spawning more threads causes avcodec_close to free threads multiple times
encoder_ctx->codec_tag = 0;
// correct values ???
encoder_ctx->qmin = 16;
encoder_ctx->qmax = 26;
encoder_ctx->max_qdiff = 4;
// end from avcut
// according to avcut, should not be set
// if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER){
// encoder_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
// }
ret = avcodec_open2(encoder_ctx, encoder, NULL);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR, "Cannot open video encoder for stream #%u\n", stream_index);
return ret;
}
codecs[stream_index] = encoder;
encoders[stream_index] = encoder_ctx;
return ret;
}
Example
To illustrate my problem, I provide here a test code using the two classes that alternates between transcoding and remuxing at each key-frame encountered in the file using my classes.
trans_remux.cpp
To compile the code:
g++ -o trans_remux trans_remux.cpp contexts.cpp -D__STDC_CONSTANT_MACROS `pkg-config --libs libavfilter` -lfmt -g
Currently the code is using avcodec_parameters_copy (contexts.cpp:333), so it works well for remuxing. If you want to test the version with avcodec_parameters_from_context, pls comment from line 333 to 337 in contexts.cpp and uncomment from line 340 to 344 and recompile.

Sound playback using FFmpeg and libsoundio in c++

I am trying to make a video player desktop application in c++ using primarily FFmpeg and Qt6. As of for now, I can decode and play video frames correctly at the right speed, that is not a problem. I am now trying to get to playback audio, which is much harder than I expected it to be. I am using libsoundio for my audio library but the documentation is really poor and there are not many examples/tutorials on it. I am also a beginner when it comes to audio programming, although I understand the basics. First off, if anyone can recommend an audio library for this type of job let me know, but I would like to use open source libraries. Anyways, here is how I decode my audio data with FFmpeg. I'm not sure if I am doing it correctly as I could barely find documentation on that as well...
I have a struct that contains all the information which is initiated through a function:
struct VideoReader
{
bool valid;
int width, height;
int video_stream_index;
int audio_stream_index;
AVRational time_base;
AVFormatContext* av_format_ctx;
AVCodecContext* av_vi_codec_ctx;
AVCodecContext* av_au_codec_ctx;
AVPacket* packet;
AVFrame* frame;
SwsContext* sws_ctx;
SwrContext* swr_ctx;
};
The function that initiates it is quite long and is not necessary to share but it populates all those values except for the sws_ctx and the swr_ctx.
Here is how I decode packets, this function is simplified, I left the video decoding out of it, ill take care of syncing once I can properly playback audio:
bool video_reader_read_au_frame(VideoReader *video_reader, unsigned char **frame_buffer)
{
// Unpack video_reader
auto& av_format_ctx = video_reader->av_format_ctx;
auto& av_codec_ctx = video_reader->av_au_codec_ctx;
auto& av_packet = video_reader->packet;
auto& av_frame = video_reader->frame;
auto& swr_ctx = video_reader->swr_ctx;
int& audio_stream_index = video_reader->audio_stream_index;
// Decode the video frame data
int response;
while (av_read_frame(av_format_ctx, av_packet) >= 0)
{
last_frame = false;
if (av_packet->stream_index != audio_stream_index)
{
av_packet_unref(av_packet);
continue;
}
response = avcodec_send_packet(av_codec_ctx, av_packet);
if (response < 0)
{
Logger::error("Could not decode packet.");
return false;
}
response = avcodec_receive_frame(av_codec_ctx, av_frame);
if (response == AVERROR(EAGAIN) || response == AVERROR_EOF)
{
av_packet_unref(av_packet);
continue;
}
else if (response < 0)
{
Logger::error("Could not decode packet.");
return false;
}
av_packet_unref(av_packet);
break;
}
// Initialize SwrContext
if (!swr_ctx) {
swr_ctx = swr_alloc_set_opts(nullptr,
av_codec_ctx->channel_layout, AV_SAMPLE_FMT_FLT,
av_codec_ctx->sample_rate, av_codec_ctx->channel_layout,
av_codec_ctx->sample_fmt, av_codec_ctx->sample_rate,
0, nullptr);
if (!swr_ctx)
{
Logger::error("Could not create SwrContext.");
return false;
}
if (swr_init(swr_ctx) < 0)
{
Logger::error("Could not initialize SwrContext.");
return false;
}
}
const int MAX_BUFFER_SIZE = av_samples_get_buffer_size(nullptr, av_frame->channels, av_frame->nb_samples, AV_SAMPLE_FMT_FLT, 1);
*frame_buffer = (unsigned char*)av_malloc(MAX_BUFFER_SIZE);
swr_convert(swr_ctx, frame_buffer, av_frame->nb_samples,
(const unsigned char**)av_frame->data, av_frame->nb_samples);
av_frame_unref(av_frame);
return true;
}
Here is how I would normally call this function:
VideoReader vr{};
if(!video_reader_open(&vr, "C:/Path/to/file.mp4"))
{
Logger::error("Could not initialize VideoReader.");
return 1;
}
unsigned char* buffer;
if(!video_reader_read_au_frame(&vr, &buffer))
{
Logger::error("Could not read audio data.");
return 1;
}
play_audio(&buffer); <-- Find a way to play audio once buffer has data in it
video_reader_close(&vr);
return 0;
Obviously I will loop over video_reader_read_au_frame(&vr, &buffer) to playback the whole video.
I believe my code puts the samples from the decoded frame in buffer, but I am really not sure.. I am unsure as well if I need to convert to AV_SAMPLE_FMT_FLT audio format or something else or just leave it as it is. For libsoundio, I kind of understand this example: http://libsound.io/ but I'm not sure I fully understand how this library works, especially the callback function. I know I have to pass buffer in outstream->userdata as a void pointer, but I don't know how to use it in the callback function. Any help or guidance would be greatly appreciated. Note that later on in this project I might want to send this data over a network to play the video on another computer in sync.

Why does adding audio stream to ffmpeg's libavcodec output container cause a crash?

As it stands, my project correctly uses libavcodec to decode a video, where each frame is manipulated (it doesn't matter how) and output to a new video. I've cobbled this together from examples found online, and it works. The result is a perfect .mp4 of the manipulated frames, minus the audio.
My problem is, when I try to add an audio stream to the output container, I get a crash in mux.c that I can't explain. It's in static int compute_muxer_pkt_fields(AVFormatContext *s, AVStream *st, AVPacket *pkt). Where st->internal->priv_pts->val = pkt->dts; is attempted, priv_pts is nullptr.
I don't recall the version number, but this is from a November 4, 2020 ffmpeg build from git.
My MediaContentMgr is much bigger than what I have here. I'm stripping out everything to do with the frame manipulation, so if I'm missing anything, please let me know and I'll edit.
The code that, when added, triggers the nullptr exception, is called out inline
The .h:
#ifndef _API_EXAMPLE_H
#define _API_EXAMPLE_H
#include <glad/glad.h>
#include <GLFW/glfw3.h>
#include "glm/glm.hpp"
extern "C" {
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/avutil.h>
#include <libavutil/opt.h>
#include <libswscale/swscale.h>
}
#include "shader_s.h"
class MediaContainerMgr {
public:
MediaContainerMgr(const std::string& infile, const std::string& vert, const std::string& frag,
const glm::vec3* extents);
~MediaContainerMgr();
void render();
bool recording() { return m_recording; }
// Major thanks to "shi-yan" who helped make this possible:
// https://github.com/shi-yan/videosamples/blob/master/libavmp4encoding/main.cpp
bool init_video_output(const std::string& video_file_name, unsigned int width, unsigned int height);
bool output_video_frame(uint8_t* buf);
bool finalize_output();
private:
AVFormatContext* m_format_context;
AVCodec* m_video_codec;
AVCodec* m_audio_codec;
AVCodecParameters* m_video_codec_parameters;
AVCodecParameters* m_audio_codec_parameters;
AVCodecContext* m_codec_context;
AVFrame* m_frame;
AVPacket* m_packet;
uint32_t m_video_stream_index;
uint32_t m_audio_stream_index;
void init_rendering(const glm::vec3* extents);
int decode_packet();
// For writing the output video:
void free_output_assets();
bool m_recording;
AVOutputFormat* m_output_format;
AVFormatContext* m_output_format_context;
AVCodec* m_output_video_codec;
AVCodecContext* m_output_video_codec_context;
AVFrame* m_output_video_frame;
SwsContext* m_output_scale_context;
AVStream* m_output_video_stream;
AVCodec* m_output_audio_codec;
AVStream* m_output_audio_stream;
AVCodecContext* m_output_audio_codec_context;
};
#endif
And, the hellish .cpp:
#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <inttypes.h>
#include "media_container_manager.h"
MediaContainerMgr::MediaContainerMgr(const std::string& infile, const std::string& vert, const std::string& frag,
const glm::vec3* extents) :
m_video_stream_index(-1),
m_audio_stream_index(-1),
m_recording(false),
m_output_format(nullptr),
m_output_format_context(nullptr),
m_output_video_codec(nullptr),
m_output_video_codec_context(nullptr),
m_output_video_frame(nullptr),
m_output_scale_context(nullptr),
m_output_video_stream(nullptr)
{
// AVFormatContext holds header info from the format specified in the container:
m_format_context = avformat_alloc_context();
if (!m_format_context) {
throw "ERROR could not allocate memory for Format Context";
}
// open the file and read its header. Codecs are not opened here.
if (avformat_open_input(&m_format_context, infile.c_str(), NULL, NULL) != 0) {
throw "ERROR could not open input file for reading";
}
printf("format %s, duration %lldus, bit_rate %lld\n", m_format_context->iformat->name, m_format_context->duration, m_format_context->bit_rate);
//read avPackets (?) from the avFormat (?) to get stream info. This populates format_context->streams.
if (avformat_find_stream_info(m_format_context, NULL) < 0) {
throw "ERROR could not get stream info";
}
for (unsigned int i = 0; i < m_format_context->nb_streams; i++) {
AVCodecParameters* local_codec_parameters = NULL;
local_codec_parameters = m_format_context->streams[i]->codecpar;
printf("AVStream->time base before open coded %d/%d\n", m_format_context->streams[i]->time_base.num, m_format_context->streams[i]->time_base.den);
printf("AVStream->r_frame_rate before open coded %d/%d\n", m_format_context->streams[i]->r_frame_rate.num, m_format_context->streams[i]->r_frame_rate.den);
printf("AVStream->start_time %" PRId64 "\n", m_format_context->streams[i]->start_time);
printf("AVStream->duration %" PRId64 "\n", m_format_context->streams[i]->duration);
printf("duration(s): %lf\n", (float)m_format_context->streams[i]->duration / m_format_context->streams[i]->time_base.den * m_format_context->streams[i]->time_base.num);
AVCodec* local_codec = NULL;
local_codec = avcodec_find_decoder(local_codec_parameters->codec_id);
if (local_codec == NULL) {
throw "ERROR unsupported codec!";
}
if (local_codec_parameters->codec_type == AVMEDIA_TYPE_VIDEO) {
if (m_video_stream_index == -1) {
m_video_stream_index = i;
m_video_codec = local_codec;
m_video_codec_parameters = local_codec_parameters;
}
m_height = local_codec_parameters->height;
m_width = local_codec_parameters->width;
printf("Video Codec: resolution %dx%d\n", m_width, m_height);
}
else if (local_codec_parameters->codec_type == AVMEDIA_TYPE_AUDIO) {
if (m_audio_stream_index == -1) {
m_audio_stream_index = i;
m_audio_codec = local_codec;
m_audio_codec_parameters = local_codec_parameters;
}
printf("Audio Codec: %d channels, sample rate %d\n", local_codec_parameters->channels, local_codec_parameters->sample_rate);
}
printf("\tCodec %s ID %d bit_rate %lld\n", local_codec->name, local_codec->id, local_codec_parameters->bit_rate);
}
m_codec_context = avcodec_alloc_context3(m_video_codec);
if (!m_codec_context) {
throw "ERROR failed to allocate memory for AVCodecContext";
}
if (avcodec_parameters_to_context(m_codec_context, m_video_codec_parameters) < 0) {
throw "ERROR failed to copy codec params to codec context";
}
if (avcodec_open2(m_codec_context, m_video_codec, NULL) < 0) {
throw "ERROR avcodec_open2 failed to open codec";
}
m_frame = av_frame_alloc();
if (!m_frame) {
throw "ERROR failed to allocate AVFrame memory";
}
m_packet = av_packet_alloc();
if (!m_packet) {
throw "ERROR failed to allocate AVPacket memory";
}
}
MediaContainerMgr::~MediaContainerMgr() {
avformat_close_input(&m_format_context);
av_packet_free(&m_packet);
av_frame_free(&m_frame);
avcodec_free_context(&m_codec_context);
glDeleteVertexArrays(1, &m_VAO);
glDeleteBuffers(1, &m_VBO);
}
bool MediaContainerMgr::advance_frame() {
while (true) {
if (av_read_frame(m_format_context, m_packet) < 0) {
// Do we actually need to unref the packet if it failed?
av_packet_unref(m_packet);
continue;
//return false;
}
else {
if (m_packet->stream_index == m_video_stream_index) {
//printf("AVPacket->pts %" PRId64 "\n", m_packet->pts);
int response = decode_packet();
av_packet_unref(m_packet);
if (response != 0) {
continue;
//return false;
}
return true;
}
else {
printf("m_packet->stream_index: %d\n", m_packet->stream_index);
printf(" m_packet->pts: %lld\n", m_packet->pts);
printf(" mpacket->size: %d\n", m_packet->size);
if (m_recording) {
int err = 0;
//err = avcodec_send_packet(m_output_video_codec_context, m_packet);
printf(" encoding error: %d\n", err);
}
}
}
// We're done with the packet (it's been unpacked to a frame), so deallocate & reset to defaults:
/*
if (m_frame == NULL)
return false;
if (m_frame->data[0] == NULL || m_frame->data[1] == NULL || m_frame->data[2] == NULL) {
printf("WARNING: null frame data");
continue;
}
*/
}
}
int MediaContainerMgr::decode_packet() {
// Supply raw packet data as input to a decoder
// https://ffmpeg.org/doxygen/trunk/group__lavc__decoding.html#ga58bc4bf1e0ac59e27362597e467efff3
int response = avcodec_send_packet(m_codec_context, m_packet);
if (response < 0) {
char buf[256];
av_strerror(response, buf, 256);
printf("Error while receiving a frame from the decoder: %s\n", buf);
return response;
}
// Return decoded output data (into a frame) from a decoder
// https://ffmpeg.org/doxygen/trunk/group__lavc__decoding.html#ga11e6542c4e66d3028668788a1a74217c
response = avcodec_receive_frame(m_codec_context, m_frame);
if (response == AVERROR(EAGAIN) || response == AVERROR_EOF) {
return response;
} else if (response < 0) {
char buf[256];
av_strerror(response, buf, 256);
printf("Error while receiving a frame from the decoder: %s\n", buf);
return response;
} else {
printf(
"Frame %d (type=%c, size=%d bytes) pts %lld key_frame %d [DTS %d]\n",
m_codec_context->frame_number,
av_get_picture_type_char(m_frame->pict_type),
m_frame->pkt_size,
m_frame->pts,
m_frame->key_frame,
m_frame->coded_picture_number
);
}
return 0;
}
bool MediaContainerMgr::init_video_output(const std::string& video_file_name, unsigned int width, unsigned int height) {
if (m_recording)
return true;
m_recording = true;
advance_to(0L); // I've deleted the implmentation. Just seeks to beginning of vid. Works fine.
if (!(m_output_format = av_guess_format(nullptr, video_file_name.c_str(), nullptr))) {
printf("Cannot guess output format.\n");
return false;
}
int err = avformat_alloc_output_context2(&m_output_format_context, m_output_format, nullptr, video_file_name.c_str());
if (err < 0) {
printf("Failed to allocate output context.\n");
return false;
}
//TODO(P0): Break out the video and audio inits into their own methods.
m_output_video_codec = avcodec_find_encoder(m_output_format->video_codec);
if (!m_output_video_codec) {
printf("Failed to create video codec.\n");
return false;
}
m_output_video_stream = avformat_new_stream(m_output_format_context, m_output_video_codec);
if (!m_output_video_stream) {
printf("Failed to find video format.\n");
return false;
}
m_output_video_codec_context = avcodec_alloc_context3(m_output_video_codec);
if (!m_output_video_codec_context) {
printf("Failed to create video codec context.\n");
return(false);
}
m_output_video_stream->codecpar->codec_id = m_output_format->video_codec;
m_output_video_stream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
m_output_video_stream->codecpar->width = width;
m_output_video_stream->codecpar->height = height;
m_output_video_stream->codecpar->format = AV_PIX_FMT_YUV420P;
// Use the same bit rate as the input stream.
m_output_video_stream->codecpar->bit_rate = m_format_context->streams[m_video_stream_index]->codecpar->bit_rate;
m_output_video_stream->avg_frame_rate = m_format_context->streams[m_video_stream_index]->avg_frame_rate;
avcodec_parameters_to_context(m_output_video_codec_context, m_output_video_stream->codecpar);
m_output_video_codec_context->time_base = m_format_context->streams[m_video_stream_index]->time_base;
//TODO(P1): Set these to match the input stream?
m_output_video_codec_context->max_b_frames = 2;
m_output_video_codec_context->gop_size = 12;
m_output_video_codec_context->framerate = m_format_context->streams[m_video_stream_index]->r_frame_rate;
//m_output_codec_context->refcounted_frames = 0;
if (m_output_video_stream->codecpar->codec_id == AV_CODEC_ID_H264) {
av_opt_set(m_output_video_codec_context, "preset", "ultrafast", 0);
} else if (m_output_video_stream->codecpar->codec_id == AV_CODEC_ID_H265) {
av_opt_set(m_output_video_codec_context, "preset", "ultrafast", 0);
} else {
av_opt_set_int(m_output_video_codec_context, "lossless", 1, 0);
}
avcodec_parameters_from_context(m_output_video_stream->codecpar, m_output_video_codec_context);
m_output_audio_codec = avcodec_find_encoder(m_output_format->audio_codec);
if (!m_output_audio_codec) {
printf("Failed to create audio codec.\n");
return false;
}
I've commented out all of the audio stream init beyond this next line, because this is where
the trouble begins. Creating this output stream causes the null reference I mentioned. If I
uncomment everything below here, I still get the null deref. If I comment out this line, the
deref exception vanishes. (IOW, I commented out more and more code until I found that this
was the trigger that caused the problem.)
I assume that there's something I'm doing wrong in the rest of the commented out code, that,
when fixed, will fix the nullptr and give me a working audio stream.
m_output_audio_stream = avformat_new_stream(m_output_format_context, m_output_audio_codec);
if (!m_output_audio_stream) {
printf("Failed to find audio format.\n");
return false;
}
/*
m_output_audio_codec_context = avcodec_alloc_context3(m_output_audio_codec);
if (!m_output_audio_codec_context) {
printf("Failed to create audio codec context.\n");
return(false);
}
m_output_audio_stream->codecpar->codec_id = m_output_format->audio_codec;
m_output_audio_stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
m_output_audio_stream->codecpar->format = m_format_context->streams[m_audio_stream_index]->codecpar->format;
m_output_audio_stream->codecpar->bit_rate = m_format_context->streams[m_audio_stream_index]->codecpar->bit_rate;
m_output_audio_stream->avg_frame_rate = m_format_context->streams[m_audio_stream_index]->avg_frame_rate;
avcodec_parameters_to_context(m_output_audio_codec_context, m_output_audio_stream->codecpar);
m_output_audio_codec_context->time_base = m_format_context->streams[m_audio_stream_index]->time_base;
*/
//TODO(P2): Free assets that have been allocated.
err = avcodec_open2(m_output_video_codec_context, m_output_video_codec, nullptr);
if (err < 0) {
printf("Failed to open codec.\n");
return false;
}
if (!(m_output_format->flags & AVFMT_NOFILE)) {
err = avio_open(&m_output_format_context->pb, video_file_name.c_str(), AVIO_FLAG_WRITE);
if (err < 0) {
printf("Failed to open output file.");
return false;
}
}
err = avformat_write_header(m_output_format_context, NULL);
if (err < 0) {
printf("Failed to write header.\n");
return false;
}
av_dump_format(m_output_format_context, 0, video_file_name.c_str(), 1);
return true;
}
//TODO(P2): make this a member. (Thanks to https://emvlo.wordpress.com/2016/03/10/sws_scale/)
void PrepareFlipFrameJ420(AVFrame* pFrame) {
for (int i = 0; i < 4; i++) {
if (i)
pFrame->data[i] += pFrame->linesize[i] * ((pFrame->height >> 1) - 1);
else
pFrame->data[i] += pFrame->linesize[i] * (pFrame->height - 1);
pFrame->linesize[i] = -pFrame->linesize[i];
}
}
This is where we take an altered frame and write it to the output container. This works fine
as long as we haven't set up an audio stream in the output container.
bool MediaContainerMgr::output_video_frame(uint8_t* buf) {
int err;
if (!m_output_video_frame) {
m_output_video_frame = av_frame_alloc();
m_output_video_frame->format = AV_PIX_FMT_YUV420P;
m_output_video_frame->width = m_output_video_codec_context->width;
m_output_video_frame->height = m_output_video_codec_context->height;
err = av_frame_get_buffer(m_output_video_frame, 32);
if (err < 0) {
printf("Failed to allocate output frame.\n");
return false;
}
}
if (!m_output_scale_context) {
m_output_scale_context = sws_getContext(m_output_video_codec_context->width, m_output_video_codec_context->height,
AV_PIX_FMT_RGB24,
m_output_video_codec_context->width, m_output_video_codec_context->height,
AV_PIX_FMT_YUV420P, SWS_BICUBIC, nullptr, nullptr, nullptr);
}
int inLinesize[1] = { 3 * m_output_video_codec_context->width };
sws_scale(m_output_scale_context, (const uint8_t* const*)&buf, inLinesize, 0, m_output_video_codec_context->height,
m_output_video_frame->data, m_output_video_frame->linesize);
PrepareFlipFrameJ420(m_output_video_frame);
//TODO(P0): Switch m_frame to be m_input_video_frame so I don't end up using the presentation timestamp from
// an audio frame if I threadify the frame reading.
m_output_video_frame->pts = m_frame->pts;
printf("Output PTS: %d, time_base: %d/%d\n", m_output_video_frame->pts,
m_output_video_codec_context->time_base.num, m_output_video_codec_context->time_base.den);
err = avcodec_send_frame(m_output_video_codec_context, m_output_video_frame);
if (err < 0) {
printf(" ERROR sending new video frame output: ");
switch (err) {
case AVERROR(EAGAIN):
printf("AVERROR(EAGAIN): %d\n", err);
break;
case AVERROR_EOF:
printf("AVERROR_EOF: %d\n", err);
break;
case AVERROR(EINVAL):
printf("AVERROR(EINVAL): %d\n", err);
break;
case AVERROR(ENOMEM):
printf("AVERROR(ENOMEM): %d\n", err);
break;
}
return false;
}
AVPacket pkt;
av_init_packet(&pkt);
pkt.data = nullptr;
pkt.size = 0;
pkt.flags |= AV_PKT_FLAG_KEY;
int ret = 0;
if ((ret = avcodec_receive_packet(m_output_video_codec_context, &pkt)) == 0) {
static int counter = 0;
printf("pkt.key: 0x%08x, pkt.size: %d, counter:\n", pkt.flags & AV_PKT_FLAG_KEY, pkt.size, counter++);
uint8_t* size = ((uint8_t*)pkt.data);
printf("sizes: %d %d %d %d %d %d %d %d %d\n", size[0], size[1], size[2], size[2], size[3], size[4], size[5], size[6], size[7]);
av_interleaved_write_frame(m_output_format_context, &pkt);
}
printf("push: %d\n", ret);
av_packet_unref(&pkt);
return true;
}
bool MediaContainerMgr::finalize_output() {
if (!m_recording)
return true;
AVPacket pkt;
av_init_packet(&pkt);
pkt.data = nullptr;
pkt.size = 0;
for (;;) {
avcodec_send_frame(m_output_video_codec_context, nullptr);
if (avcodec_receive_packet(m_output_video_codec_context, &pkt) == 0) {
av_interleaved_write_frame(m_output_format_context, &pkt);
printf("final push:\n");
} else {
break;
}
}
av_packet_unref(&pkt);
av_write_trailer(m_output_format_context);
if (!(m_output_format->flags & AVFMT_NOFILE)) {
int err = avio_close(m_output_format_context->pb);
if (err < 0) {
printf("Failed to close file. err: %d\n", err);
return false;
}
}
return true;
}
EDIT
The call stack on the crash (which I should have included in the original question):
avformat-58.dll!compute_muxer_pkt_fields(AVFormatContext * s, AVStream * st, AVPacket * pkt) Line 630 C
avformat-58.dll!write_packet_common(AVFormatContext * s, AVStream * st, AVPacket * pkt, int interleaved) Line 1122 C
avformat-58.dll!write_packets_common(AVFormatContext * s, AVPacket * pkt, int interleaved) Line 1186 C
avformat-58.dll!av_interleaved_write_frame(AVFormatContext * s, AVPacket * pkt) Line 1241 C
CamBot.exe!MediaContainerMgr::output_video_frame(unsigned char * buf) Line 553 C++
CamBot.exe!main() Line 240 C++
If I move the call to avformat_write_header so it's immediately before the audio stream initialization, I still get a crash, but in a different place. The crash happens on line 6459 of movenc.c, where we have:
/* Non-seekable output is ok if using fragmentation. If ism_lookahead
* is enabled, we don't support non-seekable output at all. */
if (!(s->pb->seekable & AVIO_SEEKABLE_NORMAL) && // CRASH IS HERE
(!(mov->flags & FF_MOV_FLAG_FRAGMENT) || mov->ism_lookahead)) {
av_log(s, AV_LOG_ERROR, "muxer does not support non seekable output\n");
return AVERROR(EINVAL);
}
The exception is a nullptr exception, where s->pb is NULL. The call stack is:
avformat-58.dll!mov_init(AVFormatContext * s) Line 6459 C
avformat-58.dll!init_muxer(AVFormatContext * s, AVDictionary * * options) Line 407 C
[Inline Frame] avformat-58.dll!avformat_init_output(AVFormatContext *) Line 489 C
avformat-58.dll!avformat_write_header(AVFormatContext * s, AVDictionary * * options) Line 512 C
CamBot.exe!MediaContainerMgr::init_video_output(const std::string & video_file_name, unsigned int width, unsigned int height) Line 424 C++
CamBot.exe!main() Line 183 C++
Please note that you should always try to provide a self-contained minimal working example to make it easier for others to help. With the actual code, the matching FFmpeg version, and an input video that triggers the segmentation fault (to be sure), the issue would be a matter of analyzing the control flow to identify why st->internal->priv_pts was not allocated. Without the full scenario, I have to report to making assumptions that may or may not correspond to your actual code.
Based on your description, I attempted to reproduce the issue by cloning https://github.com/FFmpeg/FFmpeg.git and creating a new branch from commit b52e0d95 (November 4, 2020) to approximate your FFmpeg version.
I recreated your scenario using the provided code snippets by
including the avformat_new_stream() call for the audio stream
keeping the remaining audio initialization commented out
including the original avformat_write_header() call site (unchanged order)
With that scenario, the video write with MP4 video/audio input fails in avformat_write_header():
[mp4 # 0x2b39f40] sample rate not set 0
The call stack of the error location:
#0 0x00007ffff75253d7 in raise () from /lib64/libc.so.6
#1 0x00007ffff7526ac8 in abort () from /lib64/libc.so.6
#2 0x000000000094feca in init_muxer (s=0x2b39f40, options=0x0) at libavformat/mux.c:309
#3 0x00000000009508f4 in avformat_init_output (s=0x2b39f40, options=0x0) at libavformat/mux.c:490
#4 0x0000000000950a10 in avformat_write_header (s=0x2b39f40, options=0x0) at libavformat/mux.c:514
[...]
In init_muxer(), the sample rate in the stream parameters is checked unconditionally:
case AVMEDIA_TYPE_AUDIO:
if (par->sample_rate <= 0) {
av_log(s, AV_LOG_ERROR, "sample rate not set %d\n", par->sample_rate); abort();
ret = AVERROR(EINVAL);
goto fail;
}
That condition has been in effect since 2014-06-18 at the very least (didn't go back any further) and still exists. With a version from November 2020, the check must be active and the parameter must be set accordingly.
If I uncomment the remaining audio initialization, the situation remains unchanged (as expected). So, satisfy the condition, I added the missing parameter as follows:
m_output_audio_stream->codecpar->sample_rate =
m_format_context->streams[m_audio_stream_index]->codecpar->sample_rate;
With that, the check succeeds, avformat_write_header() succeeds, and the actual video write succeeds.
As you indicated in your question, the segmentation fault is caused by st->internal->priv_pts being NULL at this location:
#0 0x00000000009516db in compute_muxer_pkt_fields (s=0x2b39f40, st=0x2b3a580, pkt=0x7fffffffe2d0) at libavformat/mux.c:632
#1 0x0000000000953128 in write_packet_common (s=0x2b39f40, st=0x2b3a580, pkt=0x7fffffffe2d0, interleaved=1) at libavformat/mux.c:1125
#2 0x0000000000953473 in write_packets_common (s=0x2b39f40, pkt=0x7fffffffe2d0, interleaved=1) at libavformat/mux.c:1188
#3 0x0000000000953634 in av_interleaved_write_frame (s=0x2b39f40, pkt=0x7fffffffe2d0) at libavformat/mux.c:1243
[...]
In the FFmpeg code base, the allocation of priv_pts is handled by init_pts() for all streams referenced by the context. init_pts() has two call sites:
libavformat/mux.c:496:
if (s->oformat->init && ret) {
if ((ret = init_pts(s)) < 0)
return ret;
return AVSTREAM_INIT_IN_INIT_OUTPUT;
}
libavformat/mux.c:530:
if (!s->internal->streams_initialized) {
if ((ret = init_pts(s)) < 0)
goto fail;
}
In both cases, the calls are triggered by avformat_write_header() (indirectly via avformat_init_output() for the first, directly for the second). According to control flow analysis, there's no success case that would leave priv_pts unallocated.
Considering a high probability that our versions of FFmpeg are compatible in terms of behavior, I have to assume that 1) the sample rate must be provided for audio streams and 1) priv_pts is always allocated by avformat_write_header() in the absence of errors. Therefore, two possible root causes come to mind:
Your stream is not an audio stream (unlikely; the type is based on the codec, which in turn is based on the output file extension - assuming mp4)
You do not call avformat_write_header() (unlikely) or do not handle the error in the caller of your C++ member function (the return value of avformat_write_header() is checked but I do not have code corresponding to the caller of the C++ member function; your actual code might differ significantly from the code provided, so it's possible and the only plausible conclusion that can be drawn from available data)
The solution: Ensure that processing does not continue if avformat_write_header() fails. By adding the audio stream, avformat_write_header() starts to fail unless you set the stream sample rate. If the error is ignored, av_interleaved_write_frame() triggers a segmentation fault by accessing the unallocated st->internal->priv_pts.
As mentioned initially, scenario is incomplete. If you do call avformat_write_header() and stop processing in case of an error (meaning you do not call av_interleaved_write_frame()), more information is needed. As it stands now, that is unlikely. For further analysis, the executable output (stdout, stderr) is required to see your traces and FFmpeg log messages. If that does not reveal new information, a self-contained minimal working example and the video input are needed to get all the full picture.

How to decode MJPEG with FFmpeg

I am trying to decode a MJPEG stream with libav. The stream comes from V4L2 driver. When working with high resolutions the following code works fine. However, when using low resolutions (For instance 320x190) it produces strange artefacts
void V4L2::compressedCapturingThread(){
const AVCodec *codec=avcodec_find_decoder(m_currVidMode.codec.toAVCodecID());
if(!codec)
return; //Something went horribly wrong
//Allocate the context
AVCodecContext* codecCtx=avcodec_alloc_context3(codec);
if(!codecCtx){
return; //Error
}
codecCtx->width=m_currVidMode.res.width;
codecCtx->height=m_currVidMode.res.height;
//Open the context
if (avcodec_open2(codecCtx, codec, nullptr) < 0) {
avcodec_free_context(&codecCtx);
return;
}
//Allocate space for the packet
AVPacket *pkt=av_packet_alloc();
if(!pkt){
avcodec_free_context(&codecCtx);
return;
}
AVFrame* decodedFrame=av_frame_alloc();
if(!decodedFrame){
avcodec_free_context(&codecCtx);
avcodec_free_context(&codecCtx);
return;
}
Graphics::Uploader uplo;
v4l2_buffer buf;
int ret;
Utils::ImageBuffer decodedImgBuf(
Utils::ImageAttributes(
m_currVidMode.res,
m_currVidMode.pixFmt
), (u_int8_t*)nullptr
);
//Main loop
while(!m_threadExit){
reqBuffer(&buf);
if(!buf.bytesused){
continue;
}
//Create the packet with the given data
u_int8_t* bufData=(u_int8_t*)av_malloc(buf.bytesused);
memcpy(bufData, m_buffers[buf.index].buffer, buf.bytesused); //copy the data
av_packet_from_data (pkt, bufData, buf.bytesused);
freeBuffer(&buf); //V4L2 buffer no longer needed
//Try to decode the packet
ret=avcodec_send_packet(codecCtx, pkt);
av_packet_unref(pkt);
if(ret<0){
continue;
}
ret = avcodec_receive_frame(codecCtx, decodedFrame);
if(ret<0){
continue;
}
memcpy(decodedImgBuf.data, decodedFrame->data, sizeof(decodedImgBuf.data)); //Copy plane pointers
if(decodedImgBuf.att.pixFmt == Utils::PixelFormats::NONE){
decodedImgBuf.att.pixFmt=Utils::PixelFormat(codecCtx->pix_fmt);
//Change deprecated formats
if(decodedImgBuf.att.pixFmt == Utils::PixelFormats::YUVJ420P)
decodedImgBuf.att.pixFmt = Utils::PixelFormats::YUV420P;
else if(decodedImgBuf.att.pixFmt == Utils::PixelFormats::YUVJ422P)
decodedImgBuf.att.pixFmt = Utils::PixelFormats::YUV422P;
else if(decodedImgBuf.att.pixFmt == Utils::PixelFormats::YUVJ440P)
decodedImgBuf.att.pixFmt = Utils::PixelFormats::YUV440P;
else if(decodedImgBuf.att.pixFmt == Utils::PixelFormats::YUVJ444P)
decodedImgBuf.att.pixFmt = Utils::PixelFormats::YUV444P;
}
std::unique_ptr<const Graphics::Frame> frame;
{
Graphics::UniqueContext ctx(Graphics::Context::getAvalibleCtx());
frame=uplo.getFrame(decodedImgBuf);
}
Stream::AsyncSource<Graphics::Frame>::push(std::move(frame));
}
//Free everything
avcodec_free_context(&codecCtx);
av_packet_free(&pkt);
av_frame_free(&decodedFrame);
}
If I try to read the AVFrame's contents to disk just after avcodec_receive_frame() I get the mentioned "strange results" (I see it by uploading it to rawpixels.net ), so the problem is not after this line. If I save the pkt 's data to disk as JPEG just before freeBuffer() the image can be seen properly. I'll attach some pictures
V4L2 configured at 1280x720
V4L2 configured at 320x190
The complete code can be found at:
https://github.com/oierlauzi/zuazo
https://github.com/oierlauzi/zuazo/blob/master/src/Zuazo/Sources/V4L2.cpp
Edit 1:
I forgot to mention, codecCtx->pix_fmt has the value of AL_PIX_FMT_YUVJ422P (given by libav)

C++ ffmpeg video missing frames and won't play in Quicktime

I wrote some C++ code that uses ffmpeg to encode a video. I'm having two strange issues:
The final video is always missing 1 frame. That is, if I have it encode 10 frames the final video only has 9 (at least that's what ffprobe -show_frames -pretty $VIDEO | grep -F '[FRAME]' | wc -l tells me.
The final video plays fine in some players (mpv and vlc) but not in Quicktime. Quicktime just shows a completely black screen.
My code is roughly this (modified a bit to remove types that are unique to our code base):
First, I open the video file, write the headers and initialize things:
template <class PtrT>
using UniquePtrWithDeleteFunction = std::unique_ptr<PtrT, std::function<void (PtrT*)>>;
std::unique_ptr<FfmpegEncodingFrameSink> FfmpegEncodingFrameSink::Create(
const std::string& dest_url) {
AVFormatContext* tmp_format_ctxt;
auto alloc_format_res = avformat_alloc_output_context2(&tmp_format_ctxt, nullptr, "mp4", dest_url.c_str());
if (alloc_format_res < 0) {
throw FfmpegException("Error opening output file.");
}
auto format_ctxt = UniquePtrWithDeleteFunction<AVFormatContext>(
tmp_format_ctxt, CloseAvFormatContext);
AVStream* out_stream_video = avformat_new_stream(format_ctxt.get(), nullptr);
if (out_stream_video == nullptr) {
throw FfmpegException("Could not create outputstream");
}
auto codec_context = GetCodecContext(options);
out_stream_video->time_base = codec_context->time_base;
auto ret = avcodec_parameters_from_context(out_stream_video->codecpar, codec_context.get());
if (ret < 0) {
throw FfmpegException("Failed to copy encoder parameters to outputstream");
}
if (!(format_ctxt->oformat->flags & AVFMT_NOFILE)) {
ret = avio_open(&format_ctxt->pb, dest_url.c_str(), AVIO_FLAG_WRITE);
if (ret < 0) {
throw VideoDecodeException("Could not open output file: " + dest_url);
}
}
ret = avformat_init_output(format_ctxt.get(), nullptr);
if (ret < 0) {
throw FfmpegException("Unable to initialize the codec.");
}
ret = avformat_write_header(format_ctxt.get(), nullptr);
if (ret < 0) {
throw FfmpegException("Error occurred writing format header");
}
return std::unique_ptr<FfmpegEncodingFrameSink>(
new FfmpegEncodingFrameSink(std::move(format_ctxt), std::move(codec_context)));
}
Then, every time I get a new frame to encode I pass it to this function (the frames are being decoded via ffmpeg from another mp4 file which Quicktime plays just fine):
// If frame == nullptr then we're done and we're just flushing the encoder
// otherwise encode an actual frame
void FfmpegEncodingFrameSink::EncodeAndWriteFrame(
const AVFrame* frame) {
auto ret = avcodec_send_frame(codec_ctxt_.get(), frame);
if (ret < 0) {
throw FfmpegException("Error encoding the frame.");
}
AVPacket enc_packet;
enc_packet.data = nullptr;
enc_packet.size = 0;
av_init_packet(&enc_packet);
do {
ret = avcodec_receive_packet(codec_ctxt_.get(), &enc_packet);
if (ret == AVERROR(EAGAIN)) {
CHECK(frame != nullptr);
break;
} else if (ret == AVERROR_EOF) {
CHECK(frame == nullptr);
break;
} else if (ret < 0) {
throw FfmpegException("Error putting the encoded frame into the packet.");
}
assert(ret == 0);
enc_packet.stream_index = 0;
LOG(INFO) << "Writing packet to stream.";
av_interleaved_write_frame(format_ctxt_.get(), &enc_packet);
av_packet_unref(&enc_packet);
} while (ret == 0);
}
Finally, in my destructor I close everything up like so:
FfmpegEncodingFrameSink::~FfmpegEncodingFrameSink() {
// Pass a nullptr to EncodeAndWriteFrame so it flushes the encoder
EncodeAndWriteFrame(nullptr);
// write mp4 trailer
av_write_trailer(format_ctxt_.get());
}
If I run this passing n frames to EncodeAndWriteFrame line LOG(INFO) << "Writing packet to stream."; gets run n times indicating the n packets were written to the stream. But ffprobe always shows only n - 1 frames int he video. And the final video doesn't play on quicktime.
What am I doing wrong??
Sorry for the delay but as i just had the same problem and noticed that this question deserves an answer, here how i solved this.
Up in front, the Problem only occured for me when using mov, mp4, 3gp as format. It worked frame accurate when using e.g. avi format. When i wrote uncompressed video frames to the container, i saw that the avi and mov had the same count of frames stored but the mov obviously had some problem in it's header.
Counting the number of frames in the mov using header metadata showed one frame is missing:
ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 c:\temp\myinput.mov
While ignoring the index showed the correct number of frames:
-ignore_editlist 1
The solution for me was, set the timebase to the AVStream->CodeContext of the video stream.
The code above attempts to do this in this line:
out_stream_video->time_base = codec_context->time_base;
But the problem is that the posted code above does not expose the function GetCodecContext so we do not know if the time_base is correctly set for "codec_context". So it is my believe that the author's problem was that his function GetCodecContext did not set the time_base correctly.