Insights to encode/decode video with ffmpeg h264 (lossless)
So I got something working on the encoding part, encode an avi in 264 however VLC wont play it, however Totem will.
Decoding the same file proves troublesome. (I want the exact same data/frame going in as going out), I get these ;
saving frame 5
Video decoding
[h264 # 0x1d19880] decode_slice_header error
frame :6
saving frame 6
Video decoding
[h264 # 0x1d19880] error while decoding MB 15 7, bytestream -27
[h264 # 0x1d19880] concealing 194 DC, 194 AC, 194 MV errors in I frame
frame :7
saving frame 7
Video decoding
[h264 # 0x1d19880] decode_slice_header error
and ultimatly this
[H264 Decoder # 0x7f1320766040] frame :11
Broken frame packetizing
[h264 # 0x1d19880] SPS changed in the middle of the frame
[h264 # 0x1d19880] decode_slice_header error
[h264 # 0x1d19880] no frame!
Error while decoding frame 11
GAME OVER.
Now I suspect that I have to go back to 1. the encoding part, there is problary a good reason VLC wont play it!
I encode like this.
void encode(char *Y,char *U,char *V){
av_init_packet(&pkt);
pkt.data = NULL; // packet data will be allocated by the encoder
pkt.size = 0;
fflush(stdout);
frame->data[0] = (uint8_t*)Y;
frame->data[1] = (uint8_t*)U;
frame->data[2] = (uint8_t*)V;
frame->pts = ++i;
ret = avcodec_encode_video2(c, &pkt, frame, &got_output);
if (ret < 0) {
fprintf(stderr, "Error encoding frame\n");
exit (EXIT_FAILURE);
}
if (got_output) {
printf("Write frame %3d (size=%5d)\n", i, pkt.size);
fwrite(pkt.data, 1, pkt.size, f);
av_free_packet(&pkt);
}
}
And the codec is setup like this:
AVCodecID dasd = AV_CODEC_ID_H264;
codec = avcodec_find_encoder(dasd);
c = avcodec_alloc_context3(codec);
c->bit_rate = 400000;
c->width = 320;
c->height = 240;
c->time_base= (AVRational){1,25};
c->gop_size = 10;
c->max_b_frames=1;
c->pix_fmt = AV_PIX_FMT_YUV420P;
av_opt_set(c->priv_data, "preset", "slow", 0);
avcodec_open2(c, codec, NULL);
Since I am going for lossless i am not dealing with delayed frames(is this a correct assumption?)
I may not actually be encoding lossless, it seems like I may have to go with something like
AVDictionary *param;
av_dict_set(¶m, "qp", "0", 0);
And then open...
So I guess me questions is these :
What are the correct codec params for lossless encoding (and advice if h264 is a terrible idea in this regard).
Do I need to handle delayed frames when going for lossless?
Why is VLC mad at me?
Thanks.
To achieve lossless: av_dict_set(¶m, "crf", "0", 0);
Delayed frames (B-frames) has nothing to do with lossless. If you need low-delay, then don't use B-frames.
Some thing is seriously wrong in your encoding. The error "MV errors in I frame" is odd one here, there shouldn't be any MVs in I-frame. It seems the header parsing it-self gone wrong. Please share the bit-stream & more details for VLC failure
You're writing raw annexb frames into a file without any container wrapping. Use a container like mp4 or matroska and VLC should be happy.
Related
I need to record frames in real time. To test this situation, I make pts non-linear (since frames may be lost), thus:
// AVFrame
video_frame->pts = prev_pts + 2;
I use libavformat to write to a file. Parameters AVCodecContext and AVStream:
#define STREAM_FRAME_RATE 25
#define CODEC_PIX_FMT AV_PIX_FMT_YUV420P
#define FRAME_WIDTH 1440
#define FRAME_HEIGHT 900
// AVCodecContext
cc->codec_id = video_codec->id;
cc->bit_rate = 400000;
cc->width = FRAME_WIDTH;
cc->height = FRAME_HEIGHT;
cc->gop_size = 12;
cc->pix_fmt = CODEC_PIX_FMT;
// AVStream
video_stream->time_base = AVRational{ 1, STREAM_FRAME_RATE };
cc->time_base = video_stream->time_base;
cc->framerate = AVRational{ STREAM_FRAME_RATE , 1 };
Write to file:
static int write_frame(AVFormatContext *fmt_ctx, const AVRational *time_base, AVStream *st, AVPacket *pkt)
{
/* rescale output packet timestamp values from codec to stream timebase */
//av_packet_rescale_ts(pkt, *time_base, st->time_base);
pkt->pts = av_rescale_q(pkt->pts, *time_base, st->time_base);
pkt->dts = av_rescale_q(pkt->dts, *time_base, st->time_base);
pkt->stream_index = st->index;
/* Write the compressed frame to the media file. */
//log_packet(fmt_ctx, pkt);
//return av_write_frame(fmt_ctx, pkt);
return av_interleaved_write_frame(fmt_ctx, pkt);
}
If you use the avi container, then the information on the number of frames per second is indicated correctly in the file: 25 fps
If you use the mp4 container, then the file information about the number of frames per second is indicated incorrectly: 12.5 fps
Tell me, please, what other settings need to be added?
MP4s do not store framerate, AVIs do.
In MP4s, only timing info for packets is stored. Since your pts expr is video_frame->pts = prev_pts + 2 and stream time base is 1/25, frames are spaced 80ms apart and hence ffmpeg probes the frame rate to be 12.5 fps (correctly).
AVIs do not have per-frame timing. Instead, they write the user-supplied frame rate. Should a packet timing be greater than the pervious frame pts by 1/fps, the muxer will write skip frame(s) which are empty packets, to maintain the frame rate.
I am working on a screen streaming program, till now, I can capture and encode my screen into a video perfectly.
However, whenever I try to stream it over LAN, and play using Mplayer, it is able to run in simple desktop (text document, etc...) but when I try to play video the frame stated to corrupted until I quit the video (this video cause the most corrupted frame https://www.youtube.com/watch?v=1La4QzGeaaQ&t=230s)
I am not really sure why when streaming, I got corrupted frame with the youtube video but when save it to a file (change URL from udp://192.168.1.xxx:YYYY to file name C:\Test.ts) there were no corrupted frame at all.
My Mplayer log:
`V: 0.0 1563/1563 51% 43% 0.0% 0 0
[h264 # 0000000001d04940]Invalid NAL unit 0, skipping.
[h264 # 0000000001d04940]error while decoding MB 23 51, bytestream -64
[h264 # 0000000001d04940]concealing 2066 DC, 2066 AC, 2066 MV errors in I frame
V: 0.0 1564/1564 51% 43% 0.0% 0 0
[h264 # 0000000001d04940]concealing 7598 DC, 7598 AC, 7598 MV errors in P frame
V: 0.0 1652/1652 50% 43% 0.0% 0 0
[h264 # 0000000001d04940]Invalid NAL unit 0, skipping.
[h264 # 0000000001d04940]error while decoding MB 26 49, bytestream -55
[h264 # 0000000001d04940]concealing 2303 DC, 2303 AC, 2303 MV errors in I frame
V: 0.0 1653/1653 50% 43% 0.0% 0 0
[h264 # 0000000001d04940]concealing 7727 DC, 7727 AC, 7727 MV errors in P frame
V: 0.0 1741/1741 49% 43% 0.0% 0 0
[h264 # 0000000001d04940]Invalid NAL unit 0, skipping.
[h264 # 0000000001d04940]error while decoding MB 65 62, bytestream -57
[h264 # 0000000001d04940]concealing 704 DC, 704 AC, 704 MV errors in I frame
V: 0.0 1742/1742 49% 43% 0.0% 0 0`
Stream initialize code
static void Stream_ini(const char *Url, AVFormatContext *&ofmt_ctx, AVCodec *codec, AVCodecContext *c, AVStream *&out_stream)
{
int ret;
avformat_alloc_output_context2(&ofmt_ctx, NULL, "mpegts", Url);
out_stream = avformat_new_stream(ofmt_ctx, codec)
out_stream->codec = c;
av_dump_format(ofmt_ctx, 0, Url, 1);
if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)
out_stream->codec->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
if (!(ofmt_ctx->flags & AVFMT_NOFILE))
{
ret = avio_open(&ofmt_ctx->pb, Url, AVIO_FLAG_WRITE);
if (ret < 0)
{
printf("Could not open output URL '%s'", Url);
return;
}
}
ret = avformat_write_header(ofmt_ctx, NULL);
if (ret < 0)
{
printf("Error occurred when opening output URL\n");
return;
}
}
this code will get the captured screen and send it to encoder:
ScreenCap.load_screen_data(inFrame,timmer) // this will capture screen and return the time and AVFrame for encoding
sws_scale(ctx, inFrame->data, inFrame->linesize, 0,c->height, frame->data, frame->linesize);
frame->pts = (timmer - first_frame_time) / fps;
EncodeToPkT(c, frame, pkt, ofmt_ctx,out_stream);
av_frame_free(&inFrame);
the AVFrame will then be send to encoder using avcodec_send_frame() to get Packet data and av_interleaved_write_frame() to stream it over LAN.
All error check are removed for simplicity
Also, this is my AVCodecContex setting for the encoder:
c->bit_rate = 15000000;
c->width = 1920;
c->height = 1080;
c->time_base = AVRational{ 1, 90 };
c->framerate = AVRational{ 90, 1 };
c->gop_size = 90;
c->max_b_frames = 0;
c->pix_fmt = AV_PIX_FMT_RGB0;
I also notice it only happen when I increase encoder bitrate (15 MBIT) but when lower it down to (10 MBIT) it happen less. And when lower it down to 2 MBIT the corrupted frame not happen anymore however the quality is really bad.
I test the streaming in LAN using: PC -> cable ->laptop, PC->wireless->laptop, PC->Virtual PC, PC only (enter PC IP address in both program and Mplayer), all got the same result.
I also test my DXGICap.load_screen_data function by make it output raw image and there were no corrupted image at all
Does anybody have an idea why.
Thank Nam.
It's not just a case of using UDP, ie. you'll receive packets out-of-order and the decoding will then fail/complain? Try using TCP and see what happens.
I have 16 bit PCM data with stereo setup which I grab from a microphone.
Once I get the data I encode it using the following encoder settings
AVCodec* audio_codec = avcodec_find_encoder(AV_CODEC_ID_MP2);
AVCodecContext* audio_codec_ctx = avcodec_alloc_context3(audio_codec);
audio_codec_ctx->bit_rate = 64000;
audio_codec_ctx->channels = 2;
audio_codec_ctx->channel_layout = AV_CH_LAYOUT_STEREO;
audio_codec_ctx->sample_rate = 44100;
audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_S16;
When I pass the audio data into the encoder I see that it takes 4608 bytes of data each time and encodes it correctly to MP2 data. PCM data grabbed by the microphone is 88320 bytes and the encoder takes 4608 bytes each time and compresses it.
If I take each 4608 byte section that has been encoded and pass it through a decoder with the same settings as above but with a decoder.
AVCodecID audio_codec_id = AV_CODEC_ID_MP2;
AVCodec * audio_decodec = avcodec_find_decoder(audio_codec_id);
audio_decodecContext = avcodec_alloc_context3(audio_decodec);
audio_decodecContext->bit_rate = 64000;
audio_decodecContext->channels = 2;
audio_decodecContext->channel_layout = AV_CH_LAYOUT_STEREO;
audio_decodecContext->sample_rate = 44100;
audio_decodecContext->sample_fmt = AV_SAMPLE_FMT_S16;
The decoding works and is successful but when I look at the data size it is exactly half 2034 of what was encoded. I dont understand why that would be. I would of imagined I would get 4608 considering the encoder and decoder are the same.
Can anyone shed some light into why this would be happening. Anything I should be setting?
The requested decoder sample format should be set using audio_decodecContext->request_sample_fmt. sample_fmt is set by the decoder itself, and may be different, in which case you should use libswresample to convert between sample formats.
Marked question as outdated as using the deprecated avcodec_decode_video2
I'm currently experiencing artifacts when decoding video using ffmpegs api. On what I would assume to be intermediate frames, artifacts build slowly only from active movement in the frame. These artifacts build for 50-100 frames until I assume a keyframe resets them. Frames are then decoded correctly and the artifacts proceed to build again.
One thing that is bothering me is I have a few video samples that are 30fps(h264) that work correctly, but all of my 60fps videos(h264) experience the problem.
I don't currently have enough reputation to post an image, so hopefully this link will work.
http://i.imgur.com/PPXXkJc.jpg
int numBytes;
int frameFinished;
AVFrame* decodedRawFrame;
AVFrame* rgbFrame;
//Enum class for decoding results, used to break decode loop when a frame is gathered
DecodeResult retResult = DecodeResult::Fail;
decodedRawFrame = av_frame_alloc();
rgbFrame = av_frame_alloc();
if (!decodedRawFrame) {
fprintf(stderr, "Could not allocate video frame\n");
return DecodeResult::Fail;
}
numBytes = avpicture_get_size(PIX_FMT_RGBA, mCodecCtx->width,mCodecCtx->height);
uint8_t* buffer = (uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
avpicture_fill((AVPicture *) rgbFrame, buffer, PIX_FMT_RGBA, mCodecCtx->width, mCodecCtx->height);
AVPacket packet;
while(av_read_frame(mFormatCtx, &packet) >= 0 && retResult != DecodeResult::Success)
{
// Is this a packet from the video stream?
if (packet.stream_index == mVideoStreamIndex)
{
// Decode video frame
int decodeValue = avcodec_decode_video2(mCodecCtx, decodedRawFrame, &frameFinished, &packet);
// Did we get a video frame?
if (frameFinished)// && rgbFrame->pict_type != AV_PICTURE_TYPE_NONE )
{
// Convert the image from its native format to RGB
int SwsFlags = SWS_BILINEAR;
// Accurate round clears up a problem where the start
// of videos have green bars on them
SwsFlags |= SWS_ACCURATE_RND;
struct SwsContext *ctx = sws_getCachedContext(NULL, mCodecCtx->width, mCodecCtx->height, mCodecCtx->pix_fmt, mCodecCtx->width, mCodecCtx->height,
PIX_FMT_RGBA, SwsFlags, NULL, NULL, NULL);
sws_scale(ctx, decodedRawFrame->data, decodedRawFrame->linesize, 0, mCodecCtx->height, rgbFrame->data, rgbFrame->linesize);
//if(count%5 == 0 && count < 105)
// DebugSavePPMImage(rgbFrame, mCodecCtx->width, mCodecCtx->height, count);
++count;
// Viewable frame is a struct to hold buffer and frame together in a queue
ViewableFrame frame;
frame.buffer = buffer;
frame.frame = rgbFrame;
mFrameQueue.push(frame);
retResult = DecodeResult::Success;
sws_freeContext(ctx);
}
}
// Free the packet that was allocated by av_read_frame
av_free_packet(&packet);
}
// Check for end of file leftover frames
if(retResult != DecodeResult::Success)
{
int result = av_read_frame(mFormatCtx, &packet);
if(result < 0)
isEoF = true;
av_free_packet(&packet);
}
// Free the YUV frame
av_frame_free(&decodedRawFrame);
I'm attempting to build a queue of the decoded frames that I then use and free as needed. Is my seperation of the frames causing the intermediate frames to be decoded incorrectly? I also break the decoding loop once I've successfully gathered a frame(Decode::Success, most examples I've seen tend to loop through the whole video.
All codec contect, video stream information, and format contexts are setup up exactly as shown in the main function of https://github.com/chelyaev/ffmpeg-tutorial/blob/master/tutorial01.c
Any suggestions would be greatly appreciated.
For reference if someone finds themselves in a similar position. Apparently with some of the older versions of FFMPEG there's an issue when using sws_scale to convert an image and not changing the actual dimensions of the final frame. If instead you create a flag for the SwsContext using:
int SwsFlags = SWS_BILINEAR; //Whatever you want
SwsFlags |= SWS_ACCURATE_RND; // Under the hood forces ffmpeg to use the same logic as if scaled
SWS_ACCURATE_RND has a performance penalty but for regular video it's probably not that noticeable. This will remove the splash of green, or green bars along the edges of textures if present.
I wanted to thank Multimedia Mike, and George Y, they were also right in that the way I was decoding the frame wasn't preserving the packets correctly and that was what caused the video artifacts building from previous frames.
I'm trying to encode images into an H264 MP4 video. The issues I'm having is that some of the images are skipped or at the end of the video simply missing. I need the video to play every single image I encode since it is an animation.
Any help setting the encoder properly would be greatly appreciated!
Encoder settings:
AVCodecContext *c;
...
c->codec_id = AV_CODEC_ID_H264;
c->bit_rate = mOutputWidth*mOutputHeight*4;//400000;
/* Resolution must be a multiple of two. */
c->width = mOutputWidth;
c->height = mOutputHeight;
/* timebase: This is the fundamental unit of time (in seconds) in terms
* of which frame timestamps are represented. For fixed-fps content,
* timebase should be 1/framerate and timestamp increments should be
* identical to 1. */
c->time_base.den = mFps;
c->time_base.num = 1;
c->gop_size = 12; /* emit one intra frame every twelve frames at most */
c->pix_fmt = AV_PIX_FMT_YUV420P;
...
av_dict_set(&pOptions, "preset", "medium", 0);
av_dict_set(&pOptions, "tune", "animation", 0);
/* open the codec */
ret = avcodec_open2(c, codec, &pOptions);
if (ret < 0) {
LOGE("Could not open video codec: %s", av_err2str(ret));
return -1;
}
Update 07/24/13:
I was able to achieve a better video by setting the gop_size=FPS and writing the last video frame repeatedly FPS+1 times seemed to resolve all issues. To me it seems odd to do that but might be something standard in the video encoding world? Any tips feedback about this?
From what I understand, you have a set of images and you want to make a video out of them. If this is the case and you don't care about the size of the video, you can try to disable inter prediction. Maybe the encoder finds that some of the images are not required and skips them.
Inter frame prediction can be disabled by setting gop_size to 0.