I have a small project which decodes h264 I frame (SPS+PPS in extradata).
It uses "dxva2" hw acceleration and works fine until I reinitialize the codec. This can be occured when an rtsp stream change the format. Fe.: h264 to h264H. and the extradata must be changed.
The extradata modifying code is:
...
ret = decode_write(decoder_ctx, packet); //GOOD
ret = decode_write(decoder_ctx, NULL);
if (avcodec_close(decoder_ctx) < 0)
return -2;
if (hw_decoder_init(decoder_ctx, type) < 0)
return -1;
if ((ret = avcodec_open2(decoder_ctx, decoder, NULL)) < 0) {
fprintf(stderr, "Failed to open codec for stream.\n");
return -1;
}
ret = decode_write(decoder_ctx, packet); //ACCESS VIOLATION
ret = decode_write(decoder_ctx, NULL);
The next avcodec_send_packet goes access violation exception.
When I switch off the HW acceleration and fallback to CPU this change is works fine.
So the only problem when I use dxva2 and reinitialize the codec.
The extradata change code is not in this small example, because the error reason is the reinitialize.
Related
Context
Hello !
I'm currently working on the development of a small library allowing to cut an h.264 video on any frame, but without re-encoding (transcoding) the whole video. The idea is to re-encode only the GOP on which we want to cut, and to rewrite (remux) directly the others GOP.
The avcut project (https://github.com/anyc/avcut) allows to do that, but requires a systematic decoding of each package, and seems to not work with the recent versions of ffmpeg from the tests I could do and from the recent feedbacks in the github issues.
As a beginner, I started from the code examples provided in the ffmpeg documentation, in particular: transcoding.c and remuxing.c.
Problem encountered
The problem I'm having is that I can't get both transcoding and remuxing to work properly at the same time. In particular, depending on the method I use to initialize the AVCodecParameters of the output video stream, transcoding works, or remuxing works:
avcodec_parameters_copy works well for remuxing
avcodec_parameters_from_context works well for transcoding
In case I choose avcodec_parameters_from_context, the transcoded GOP are correctly read by my video player (parole), but the remuxed packets are not read, and ffprobe does not show/detect them.
In case I choose avcodec_parameters_from_context, the remuxing GOP are correctly read by my video player, but the transcoding key_frame are bugged (I have the impression that the b-frame and p-frame are ok), and ffprobe -i return an error about the NAL of the key-frames:
[h264 # 0x55ec8a079300] sps_id 32 out of range
[h264 # 0x55ec8a079300] Invalid NAL unit size (1677727148 > 735).
[h264 # 0x55ec8a079300] missing picture in access unit with size 744
I suspect that the problem is related to the extradata of the packets. Through some experiments on the different attributes of the output AVCodecParameters, it seems that it is the extradata and extradata_size attributes that are responsible for the functioning of one method or the other.
Version
ffmpeg development branch retrieved on 2022-05-17 from https://github.com/FFmpeg/FFmpeg.
Compiled with --enable-libx264 --enable-gpl --enable-decoder=png --enable-encoder=png
Code
My code is written in c++ and is based on two classes: a class defining the parameters and methods on the input file (InputContexts) and a class defining them for the output file (OutputContexts). The code of these two classes is defined in the following files:
contexts.h
contexts.cpp
The code normally involved in the problem is the following:
stream initialization
int OutputContexts::init(const char* out_filename, InputContexts* input_contexts){
int ret;
int stream_index = 0;
avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, out_filename);
if (!ofmt_ctx) {
fprintf(stderr, "Could not create output context\n");
ret = AVERROR_UNKNOWN;
return ret;
}
av_dump_format(ofmt_ctx, 0, out_filename, 1);
encoders.resize(input_contexts->ifmt_ctx->nb_streams, nullptr);
codecs.resize(input_contexts->ifmt_ctx->nb_streams, nullptr);
// stream mapping
for (int i = 0; i < input_contexts->ifmt_ctx->nb_streams; i++) {
AVStream *out_stream;
AVStream *in_stream = input_contexts->ifmt_ctx->streams[i];
AVCodecContext* decoder_ctx = input_contexts->decoders[i];
// add new stream to output context
out_stream = avformat_new_stream(ofmt_ctx, NULL);
if (!out_stream) {
fprintf(stderr, "Failed allocating output stream\n");
ret = AVERROR_UNKNOWN;
return ret;
}
// from avcut blog
av_dict_copy(&out_stream->metadata, in_stream->metadata, 0);
out_stream->time_base = in_stream->time_base;
// encoder
if (decoder_ctx->codec_type == AVMEDIA_TYPE_VIDEO){
ret = prepare_encoder_video(i, input_contexts);
if (ret < 0){
fprintf(stderr, "Error while preparing encoder for stream #%u\n", i);
return ret;
}
// from avcut
out_stream->sample_aspect_ratio = in_stream->sample_aspect_ratio;
// works well for remuxing
ret = avcodec_parameters_copy(out_stream->codecpar, in_stream->codecpar);
if (ret < 0) {
fprintf(stderr, "Failed to copy codec parameters\n");
return ret;
}
// works well for transcoding
// ret = avcodec_parameters_from_context(out_stream->codecpar, encoders[i]);
// if (ret < 0) {
// av_log(NULL, AV_LOG_ERROR, "Failed to copy encoder parameters to output stream #%u\n", i);
// return ret;
// }
} else if (decoder_ctx->codec_type == AVMEDIA_TYPE_AUDIO) {
...
} else {
...
}
// TODO useful ???
// set current stream position to 0
// out_stream->codecpar->codec_tag = 0;
}
// opening output file in write mode with the ouput context
if (!(ofmt_ctx->oformat->flags & AVFMT_NOFILE)) {
ret = avio_open(&ofmt_ctx->pb, out_filename, AVIO_FLAG_WRITE);
if (ret < 0) {
fprintf(stderr, "Could not open output file '%s'", out_filename);
return ret;
}
}
// write headers from output context in output file
ret = avformat_write_header(ofmt_ctx, NULL);
if (ret < 0) {
fprintf(stderr, "Error occurred when opening output file\n");
return ret;
}
return ret;
}
AVCodecContext initialization for encoder
int OutputContexts::prepare_encoder_video(int stream_index, InputContexts* input_contexts){
int ret;
const AVCodec* encoder;
AVCodecContext* decoder_ctx = input_contexts->decoders[stream_index];
AVCodecContext* encoder_ctx;
if (video_index >= 0){
fprintf(stderr, "Impossible to mark stream #%u as video, stream #%u is already registered as video stream.\n",
stream_index, video_index);
return -1; //TODO change this value for correct error code
}
video_index = stream_index;
if(decoder_ctx->codec_id == AV_CODEC_ID_H264){
encoder = avcodec_find_encoder_by_name("libx264");
if (!encoder) {
av_log(NULL, AV_LOG_FATAL, "Encoder libx264 not found\n");
return AVERROR_INVALIDDATA;
}
fmt::print("Encoder libx264 will be used for stream {}.\n", stream_index);
} else {
std::string s = fmt::format("No video encoder found for the given codec_id: {}\n", avcodec_get_name(decoder_ctx->codec_id));
av_log(NULL, AV_LOG_FATAL, s.c_str());
return AVERROR_INVALIDDATA;
}
encoder_ctx = avcodec_alloc_context3(encoder);
if (!encoder_ctx) {
av_log(NULL, AV_LOG_FATAL, "Failed to allocate the encoder context\n");
return AVERROR(ENOMEM);
}
// from avcut
encoder_ctx->time_base = decoder_ctx->time_base;
encoder_ctx->ticks_per_frame = decoder_ctx->ticks_per_frame;
encoder_ctx->delay = decoder_ctx->delay;
encoder_ctx->width = decoder_ctx->width;
encoder_ctx->height = decoder_ctx->height;
encoder_ctx->pix_fmt = decoder_ctx->pix_fmt;
encoder_ctx->sample_aspect_ratio = decoder_ctx->sample_aspect_ratio;
encoder_ctx->color_primaries = decoder_ctx->color_primaries;
encoder_ctx->color_trc = decoder_ctx->color_trc;
encoder_ctx->colorspace = decoder_ctx->colorspace;
encoder_ctx->color_range = decoder_ctx->color_range;
encoder_ctx->chroma_sample_location = decoder_ctx->chroma_sample_location;
encoder_ctx->profile = decoder_ctx->profile;
encoder_ctx->level = decoder_ctx->level;
encoder_ctx->thread_count = 1; // spawning more threads causes avcodec_close to free threads multiple times
encoder_ctx->codec_tag = 0;
// correct values ???
encoder_ctx->qmin = 16;
encoder_ctx->qmax = 26;
encoder_ctx->max_qdiff = 4;
// end from avcut
// according to avcut, should not be set
// if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER){
// encoder_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
// }
ret = avcodec_open2(encoder_ctx, encoder, NULL);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR, "Cannot open video encoder for stream #%u\n", stream_index);
return ret;
}
codecs[stream_index] = encoder;
encoders[stream_index] = encoder_ctx;
return ret;
}
Example
To illustrate my problem, I provide here a test code using the two classes that alternates between transcoding and remuxing at each key-frame encountered in the file using my classes.
trans_remux.cpp
To compile the code:
g++ -o trans_remux trans_remux.cpp contexts.cpp -D__STDC_CONSTANT_MACROS `pkg-config --libs libavfilter` -lfmt -g
Currently the code is using avcodec_parameters_copy (contexts.cpp:333), so it works well for remuxing. If you want to test the version with avcodec_parameters_from_context, pls comment from line 333 to 337 in contexts.cpp and uncomment from line 340 to 344 and recompile.
I want to update parameters like fps, bitrate, gop of video encoder which were already passed to AVCodecContext structure previously.I want to get it's reflection at same time whenever I update any parameters.
One thing can be done, is that need to close codec using av codec close and again open it.
But I think that is not good way.
Here is my ffmpeg's source code for video encoding:
int got_output = 0, ret = 0;
//av_init_packet(&pkt);
pkt.data = NULL; // packet data will be allocated by the encoder
pkt.size = 0;
ret = avcodec_encode_video2(c, &pkt, frame, &got_output);
if (ret < 0)
{
cerr << "Error sending a frame for encoding\n";
exit(1);
}
Is there any FFMPEG's API that can be used to reload encoding parameters?
No, FFmpeg does not have an API for a running process. It is something you would need to develop yourself.
I'm working with ffmpeg 4.1 and I'm showing live streams of multiple cameras, h264 and h265.
My program collects packets of the same frame and then calls decodeVideo function. Actually it sends all packets of a frame at once.
Program works well if there is no missing packets. When I remove packet in random I-frames, both h264 and h265 streams work as expected (jumps some seconds but continues streaming).
When I remove packet in random P-frame from h265 streams, avcodec_send_packet function gives AVERROR_INVALIDDATA and streams continue.
However when I remove packet in random P-frame from h264 streams, avcodec_send_packet function gives 0. Then avcodec_receive_frame function gives AVERROR(EAGAIN) continuously and streams freeze.
void decodeVideo(array<uint8_t>^ data, int length, AvFrame^ finishedFrame)
{
AVPacket* videoPacket = new AVPacket();
av_init_packet(videoPacket);
pin_ptr<unsigned char> dataPtr = &data[0];
videoPacket->data = dataPtr;
videoPacket->size = length;
int retVal = avcodec_send_packet((AVCodecContext*)context, videoPacket);
if(retVal < 0)
{
if (retVal == AVERROR_EOF)
Utility::Log->ErrorFormat("avcodec_send_packet() return value is AVERROR_EOF.");
else if( retVal == AVERROR_INVALIDDATA)
Utility::Log->ErrorFormat("avcodec_send_packet() INVALID DATA!");
else
Utility::Log->ErrorFormat("avcodec_send_packet() return value is negative:{0}",retVal);
}
else
{
int receive_frame = avcodec_receive_frame((AVCodecContext*)context, (AVFrame*)finishedFrame);
if (receive_frame == AVERROR(EAGAIN))
Utility::Log->ErrorFormat("avcodec_receive_frame() returns AVERROR(EAGAIN)");
else if(receive_frame == AVERROR_EOF)
Utility::Log->ErrorFormat("avcodec_receive_frame() returns AVERROR(AVERROR_EOF)");
else
Utility::Log->ErrorFormat("avcodec_receive_frame() return value is negative:{0}",receive_frame);
}
av_packet_unref(videoPacket);
delete videoPacket;
}
EDIT
When I add avcodec_flush_buffers like shown, my problem is temporarily solved. However it freeze again after a while.
if(receive_frame == AVERROR(EAGAIN))
{
Utility::Log->ErrorFormat("avcodec_receive_frame() returns AVERROR(EAGAIN)");
avcodec_flush_buffers((AVCodecContext*)context);
}
Tested with ffmpeg version 4.1.1 same results.
Find an ffmpeg version like 2.5 decode function is different but there is no problem when i remove packets. However I'm working with h265 streams too.
EDIT2
AVCodecID id = AVCodecID::AV_CODEC_ID_H264;
AVCodec* dec = avcodec_find_decoder(id);
AVCodecContext* decContext = avcodec_alloc_context3(dec);
After these lines, my code included the following lines. When i delete them, there is no problem now.
if(dec->capabilities & AV_CODEC_CAP_TRUNCATED)
decContext->flags |= AV_CODEC_FLAG_TRUNCATED;
decContext->flags2 |= AV_CODEC_FLAG2_CHUNKS;
I am trying to stream H264 based data using Live555 over RTSP.
I am capturing data using V4L2, and then encodes it using FFMPEG and then passing data to Live555's DeviceSource file, in that I using H264VideoStreamFramer class,
Below is my codec settings to configure AVCodecContext of encoder,
codec = avcodec_find_encoder_by_name(CODEC_NAME);
if (!codec) {
cerr << "Codec " << codec_name << " not found\n";
exit(1);
}
c = avcodec_alloc_context3(codec);
if (!c) {
cerr << "Could not allocate video codec context\n";
exit(1);
}
pkt = av_packet_alloc();
if (!pkt)
exit(1);
/* put sample parameters */
c->bit_rate = 400000;
/* resolution must be a multiple of two */
c->width = PIC_HEIGHT;
c->height = PIC_WIDTH;
/* frames per second */
c->time_base = (AVRational){1, FPS};
c->framerate = (AVRational){FPS, 1};
c->gop_size = 10;
c->max_b_frames = 1;
c->pix_fmt = AV_PIX_FMT_YUV420P;
c->rtp_payload_size = 30000;
if (codec->id == AV_CODEC_ID_H264)
av_opt_set(c->priv_data, "preset", "fast", 0);
av_opt_set_int(c->priv_data, "slice-max-size", 30000, 0);
/* open it */
ret = avcodec_open2(c, codec, NULL);
if (ret < 0) {
cerr << "Could not open codec\n";
exit(1);
}
And I am getting encoded data using avcodec_receive_packet() function. which will return AVPacket.
And I am passing AVPacket's data into DeviceSource file below is code snippet of my Live555 code:
void DeviceSource::deliverFrame() {
if (!isCurrentlyAwaitingData()) return; // we're not ready for the data yet
u_int8_t* newFrameDataStart = (u_int8_t*) pkt->data;
unsigned newFrameSize = pkt->size; //%%% TO BE WRITTEN %%%
// Deliver the data here:
if (newFrameSize > fMaxSize) { // Condition becomes true many times
fFrameSize = fMaxSize;
fNumTruncatedBytes = newFrameSize - fMaxSize;
} else {
fFrameSize = newFrameSize;
}
gettimeofday(&fPresentationTime, NULL); // If you have a more accurate time - e.g., from an encoder - then use that instead.
// If the device is *not* a 'live source' (e.g., it comes instead from a file or buffer), then set "fDurationInMicroseconds" here.
memmove(fTo, newFrameDataStart, fFrameSize);
}
But here, sometimes my packet's size is getting more than fMaxSize value and as per LIVE555 logic it will truncate frame data, so that sometimes I am getting bad frames on my VLC,
From Live555 forum, I get to know that encoder should not send packet whose size is more than fMaxSize value, so my question is:
How to restrict encoder to limit size of packet?
Thanks in Advance,
Harshil
You can increase the maximum allowed sample size by changing "maxSize" in the OutPacketBuffer class in MediaSink.cpp. This worked for me. There are cases we may require high-quality video to be streamed, I don't think we will always be able to restrict the encoder to not to produce samples of size more than a particular value which would result in video quality issues. In fact, the samples are fragmented by the UDP sink live555 to match the default MTU (1500), so increasing the max sample size limit has no side effects.
I wrote some C++ code that uses ffmpeg to encode a video. I'm having two strange issues:
The final video is always missing 1 frame. That is, if I have it encode 10 frames the final video only has 9 (at least that's what ffprobe -show_frames -pretty $VIDEO | grep -F '[FRAME]' | wc -l tells me.
The final video plays fine in some players (mpv and vlc) but not in Quicktime. Quicktime just shows a completely black screen.
My code is roughly this (modified a bit to remove types that are unique to our code base):
First, I open the video file, write the headers and initialize things:
template <class PtrT>
using UniquePtrWithDeleteFunction = std::unique_ptr<PtrT, std::function<void (PtrT*)>>;
std::unique_ptr<FfmpegEncodingFrameSink> FfmpegEncodingFrameSink::Create(
const std::string& dest_url) {
AVFormatContext* tmp_format_ctxt;
auto alloc_format_res = avformat_alloc_output_context2(&tmp_format_ctxt, nullptr, "mp4", dest_url.c_str());
if (alloc_format_res < 0) {
throw FfmpegException("Error opening output file.");
}
auto format_ctxt = UniquePtrWithDeleteFunction<AVFormatContext>(
tmp_format_ctxt, CloseAvFormatContext);
AVStream* out_stream_video = avformat_new_stream(format_ctxt.get(), nullptr);
if (out_stream_video == nullptr) {
throw FfmpegException("Could not create outputstream");
}
auto codec_context = GetCodecContext(options);
out_stream_video->time_base = codec_context->time_base;
auto ret = avcodec_parameters_from_context(out_stream_video->codecpar, codec_context.get());
if (ret < 0) {
throw FfmpegException("Failed to copy encoder parameters to outputstream");
}
if (!(format_ctxt->oformat->flags & AVFMT_NOFILE)) {
ret = avio_open(&format_ctxt->pb, dest_url.c_str(), AVIO_FLAG_WRITE);
if (ret < 0) {
throw VideoDecodeException("Could not open output file: " + dest_url);
}
}
ret = avformat_init_output(format_ctxt.get(), nullptr);
if (ret < 0) {
throw FfmpegException("Unable to initialize the codec.");
}
ret = avformat_write_header(format_ctxt.get(), nullptr);
if (ret < 0) {
throw FfmpegException("Error occurred writing format header");
}
return std::unique_ptr<FfmpegEncodingFrameSink>(
new FfmpegEncodingFrameSink(std::move(format_ctxt), std::move(codec_context)));
}
Then, every time I get a new frame to encode I pass it to this function (the frames are being decoded via ffmpeg from another mp4 file which Quicktime plays just fine):
// If frame == nullptr then we're done and we're just flushing the encoder
// otherwise encode an actual frame
void FfmpegEncodingFrameSink::EncodeAndWriteFrame(
const AVFrame* frame) {
auto ret = avcodec_send_frame(codec_ctxt_.get(), frame);
if (ret < 0) {
throw FfmpegException("Error encoding the frame.");
}
AVPacket enc_packet;
enc_packet.data = nullptr;
enc_packet.size = 0;
av_init_packet(&enc_packet);
do {
ret = avcodec_receive_packet(codec_ctxt_.get(), &enc_packet);
if (ret == AVERROR(EAGAIN)) {
CHECK(frame != nullptr);
break;
} else if (ret == AVERROR_EOF) {
CHECK(frame == nullptr);
break;
} else if (ret < 0) {
throw FfmpegException("Error putting the encoded frame into the packet.");
}
assert(ret == 0);
enc_packet.stream_index = 0;
LOG(INFO) << "Writing packet to stream.";
av_interleaved_write_frame(format_ctxt_.get(), &enc_packet);
av_packet_unref(&enc_packet);
} while (ret == 0);
}
Finally, in my destructor I close everything up like so:
FfmpegEncodingFrameSink::~FfmpegEncodingFrameSink() {
// Pass a nullptr to EncodeAndWriteFrame so it flushes the encoder
EncodeAndWriteFrame(nullptr);
// write mp4 trailer
av_write_trailer(format_ctxt_.get());
}
If I run this passing n frames to EncodeAndWriteFrame line LOG(INFO) << "Writing packet to stream."; gets run n times indicating the n packets were written to the stream. But ffprobe always shows only n - 1 frames int he video. And the final video doesn't play on quicktime.
What am I doing wrong??
Sorry for the delay but as i just had the same problem and noticed that this question deserves an answer, here how i solved this.
Up in front, the Problem only occured for me when using mov, mp4, 3gp as format. It worked frame accurate when using e.g. avi format. When i wrote uncompressed video frames to the container, i saw that the avi and mov had the same count of frames stored but the mov obviously had some problem in it's header.
Counting the number of frames in the mov using header metadata showed one frame is missing:
ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 c:\temp\myinput.mov
While ignoring the index showed the correct number of frames:
-ignore_editlist 1
The solution for me was, set the timebase to the AVStream->CodeContext of the video stream.
The code above attempts to do this in this line:
out_stream_video->time_base = codec_context->time_base;
But the problem is that the posted code above does not expose the function GetCodecContext so we do not know if the time_base is correctly set for "codec_context". So it is my believe that the author's problem was that his function GetCodecContext did not set the time_base correctly.