FFmpeg, decode mp3 packet after sending by network - c++

Good day, everyone.
Working with FFmpeg.
And have some issues with decoding, I cannot find in docs and forums.
The code is not simple, so I will try to explain firstly in words, may be someone really skilled in ffmpeg will understand the issue by words. But if the code really helps, i will try to post it.
So, firstly in general, what i want to do. I want to capture voice, encode to mp3, get packet, send to network, on the other side: accept the packet, decode and play. Why not to use ffmpeg streaming? Well, because this packet will be modified a little bit, and may be encoded/crypted, so ffmpeg has no functions to do this, and i should do it manually.
Well, what i managed to do now. Now i can encode and decode via file. This code works fine, without any issues. So generally i can encode and decode mp3, and play, so i assume that my code for encoding/decoding works fine.
So, than i just change code, making no saving to file, and send packets via network instead.
This is the method, standard, well, i use to send packet to decoder on the accepting side.
result = avcodec_send_packet( codecContext, networkPacket );
if( result < 0 ) {
if( result != AVERROR(EAGAIN) ) {
qDebug() << "Some decoding error occured: " << result;
return;
}
}
networkPacket is the AVPacket restored from network.
AVPacket* networkPacket = NULL;
networkPacket = av_packet_alloc();
...
And that is the way i restore it.
void FFmpegPlay::processNetworkPacket( MediaPacket* mediaPacket ) {
qDebug() << "FFmpegPlay::processNetworkPacket start";
int result;
AVPacket* networkPacket = NULL;
networkPacket = av_packet_alloc();
networkPacket->size = mediaPacket->data.size();
networkPacket->data = (uint8_t*) malloc( mediaPacket->data.size() + AV_INPUT_BUFFER_PADDING_SIZE );
memcpy( networkPacket->data, mediaPacket->data.data(), mediaPacket->data.size() );
networkPacket->pts = mediaPacket->pts;
networkPacket->dts = mediaPacket->dts;
networkPacket->flags = mediaPacket->flags;
networkPacket->duration = mediaPacket->duration;
networkPacket->pos = mediaPacket->pos;
...
And there i get -22, EINVAL, invalid argument.
Docs tell me:
AVERROR(EINVAL): codec not opened, it is an encoder, or requires flush
Well, my codec really opened, this is decoder, and this is first call, so i think that flush is not required. So i assume, that issue is in packet and codec setup. I also tried different flags, and always get this error. Codec just doesn't want to accept packet.
So, now i have explained the situation.
And question is: Is there any special options or flags for ffmpeg mp3 decoder to implement what is explained above? Which one of them i should change?
Upd.
After some testing, i decided to make more clear test and check if i can decode immediately after encoding, without network, and it looks like i can do that.
So it looks like in network case the decoder should be initialized somehow special, or have some options.
Well, i'm dealing initialization by copying AVCodecParameters from original and sending them by network. Maybe i should change them some special way?
I'm stuck on this one. And have no idea how to deal with it. So any help is appreciated.

Related

Get composition time C++

I made my own rtmp server using libav and ffmpeg. I receive as input either an flv file or an rtmp streaming "containing" an flv file.
Since I manipulate the flv file and the relative composition time of each frame, I would like to know if there is a way to get this composition time.
I thought that given my AVPacket, I could analyze the raw buffer in order to extract the right information since I know that the flv header is 11 bytes and then in the next 16 bytes I should find the composition time.
But it doesn't work.
This is a rough example of code:
AVPacket pkt;
AVFormatContext *ifmt_ctx
while(true)
{
AVStream *in_stream, *out_stream;
ret = av_read_frame(ifmt_ctx, &pkt);
//get the composite time
}
AVPacket needs to be able to represent the data found in all media formats. Some formats (like mp4 and flv) have a decode_time and a composition_time, other (like transport streams) have a decode_time and a presentation_time. To make it easier for the programmer, AVPacket chose one method to store the information and converts when needed. Luckily its an an easy to convert back:
auto cts = pkt.pts - pkt.dts

FFmpeg - RTCP BYE packets

I’m working on some C++ project which depends on Wi-Fi RAK5206 electronic board. I’m using ffmpeg library to obtain video and audio stream and I have issue where I can start and stop stream for four times, but when I want to start for the fifth time I get error. Error description is Invalid data found when processing input and it happens when I call avformat_open_input function and I need to restart the electronic board, reconnect to Wi-Fi etc.
I figured out with Wireshark application that VLC is working and it is sending some BYE packets when TEARDOWN is called. I wonder if error depends to them, because from my application I’m not sending. How I can make setup to force ffmpeg to send BYE packets?
I found some declarations in rtpenc.h file which options to set and tried when I want to connect, but obviously without success.
The code that I used for setting options and opening input:
AVDictionary* stream_opts = 0;
av_dict_set(&stream_opts, "rtpflags", "send_bye", 0);
avformat_open_input(&format_ctx, url.c_str(), NULL, &stream_opts);
Make sure you are calling this av_write_trailer function, from your application.
if not please debug and check it.
/* Write the trailer, if any. The trailer must be written before you
* close the CodecContexts open when you wrote the header; otherwise
* av_write_trailer() may try to use memory that was freed on
* av_codec_close(). */
av_write_trailer(oc);
function Call flow code snippet from ffmpeg source:
av_write_trailer ->
....
ret = s->oformat->write_trailer(s);
} else {
s->oformat->write_trailer(s);
}
...
.write_trailer = rtp_write_trailer ->
...
if (s1->pb && (s->flags & FF_RTP_FLAG_SEND_BYE))
rtcp_send_sr(s1, ff_ntp_time(), 1)
Resolved issue with adding flag 16 (binary: 10000) to AVFormatContext object's flag.
formatCtx->flags = formatCtx->flags | 16;
According to rtpenc.h:
#define FF_RTP_FLAG_SEND_BYE 16

ffmpeg H264 Encode Frame at a time for network streaming

I'm working on a remote desktop application, I would like to send an encoded H264 packet over TCP by using ffmpeg for the encoding. However I couldn't find useful info for the particular case of encoding just one frame (already on YUV444) and get the packet.
I have several issues, the first was that:
avcodec_encode_video2
Was not blocking, I found that most of the time you get the "delayed" frames at the end, however, since this is a real time streaming the solution was:
av_opt_set(mCodecContext->priv_data, "tune", "zerolatency", 0);
Now I got the frame, but several issues, it takes a while and even worse I got a gray with trash pixels video as result. My configuration for the Codec Context:
m_pCodecCtx->bit_rate=8000000;
m_pCodecCtx->codec_id=AV_CODEC_ID_H264;
m_pCodecCtx->codec_type = AVMEDIA_TYPE_VIDEO;
m_pCodecCtx->width=1920;
m_pCodecCtx->height=1080;
m_pCodecCtx->pix_fmt=AV_PIX_FMT_YUV444P;
m_pCodecCtx->time_base.num = 1;
m_pCodecCtx->time_base.den = 25;
m_pCodecCtx->gop_size = 1;
m_pCodecCtx->keyint_min = 1;
m_pCodecCtx->i_quant_factor = float(0.71);
m_pCodecCtx->b_frame_strategy = 20;
m_pCodecCtx->qcompress = (float)0.6;
m_pCodecCtx->qmax = 51;
m_pCodecCtx->qmin = 20;
m_pCodecCtx->max_qdiff = 4;
m_pCodecCtx->refs = 4;
m_pCodecCtx->max_b_frames = 1;
m_pCodecCtx->thread_count = 1;
I would like to know how this could be done, how do I set the "I Frames"? and, that would be the optimal for a "one at a time" encoding? Also I'm not concerned right now with the quality, just need to be fast enough (under 16 ms).
For the encoding part:
nres = avcodec_encode_video2(m_pCodecCtx,&packet,m_pFrame,&framefinished);
if(nres<0){
qDebug() << "error encoding: " << nres << endl;
}
if(framefinished){
m_pFrame->pts++;
ofstream vidout("video.h264",ios::app);
if(vidout.good()){
vidout.write((const char*)&packet.data[0],packet.size);
}
vidout.close();
av_packet_unref(&packet);
}
I'm not using a container, just a raw file, ffplay reproduce raw files if the packets are right, and that's my principal issue. I'm planning to send the packet over tcp and decode on the client. Any help would be greatly appreciated.
You could take a look at the source code of webrtc.
It use openh264 and ffmpeg to accomplish your work.
I was study in it for a while. But I can't the the latest source code currently.
I found this :
source code.
Hope it helps.
Turns out I got it working since the beginning, I made very simple but important mistake, I was writing as text a binary file, so...
Thanks for the feedback and your help

ffmpeg audio frame from directshow sampleCB imediasample

i use isamplegrabber sampleCB callback to get audio sample, i can get buffer and buffer length from imediasample and i use avcodec_fill_audio_frame(frame,ost->enc->channels,ost->enc->sample_fmt,(uint8_t *)buffer,length,0) to make an avframe , but this frame does not make any audio in my mux file! i think the length is very smaller than frame_size.
can every one help me please? or give me some example if it is possible.
thank you
this is my samplecb code :
HRESULT AudioSampleGrabberCallBack::SampleCB(double Time, IMediaSample*pSample){
BYTE *pBuffer;
pSample->GetPointer(&pBuffer);
long BufferLen = pSample->GetActualDataLength();
muxer->PutAudioFrame(pBuffer,BufferLen);
}
and this is samplegrabber pin media type :
AM_MEDIA_TYPE pmt2;
ZeroMemory(&pmt2, sizeof(AM_MEDIA_TYPE));
pmt2.majortype = MEDIATYPE_Audio;
pmt2.subtype = FOURCCMap(0x1602);
pmt2.formattype = FORMAT_WaveFormatEx;
hr = pSampleGrabber_audio->SetMediaType(&pmt2);
after that i using ffmpeg muxing example to process frames and i think i need only to change the signal generating part of code :
AVFrame *Muxing::get_audio_frame(OutputStream *ost,BYTE* buffer,long length)
{
AVFrame *frame = ost->tmp_frame;
int j, i, v;
uint16_t *q = (uint16_t*)frame->data[0];
int buffer_size = av_samples_get_buffer_size(NULL, ost->enc->channels,
ost->enc->frame_size,
ost->enc->sample_fmt, 0);
// uint8_t *sample = (uint8_t *) av_malloc(buffer_size);
av_samples_alloc(&frame->data[0], frame->linesize, ost->enc->channels, ost->enc->frame_size, ost->enc->sample_fmt, 1);
avcodec_fill_audio_frame(frame, ost->enc->channels, ost->enc->sample_fmt,frame->data[0], buffer_size, 1);
frame->pts = ost->next_pts;
ost->next_pts += frame->nb_samples;
return frame;
}
The code snippets suggest you are getting AAC data using Sample Grabber and you are trying to write that into file using FFmpeg's libavformat. This can work out.
You initialize your sample grabber to get audio data in WAVE_FORMAT_AAC_LATM format. This format is not so wide spread and you are interested in reviewing your filter graph to make sure the upstream connection on the Sample Grabber is such that you expect. There is a chance that somehow there is a weird chain of filter that pretend to produce AAC-LATM and the reality is that the data is invalid (or not even reaching grabber callback). So you need to review the filter graph (see Loading a Graph From an External Process and Understanding Your DirectShow Filter Graph), then step through your callback with debugger to make sure you get the data and it makes sense.
Next thing, you are expected to initialize AVFormatContext, AVStream to indicate that you will be writing data in AAC LATM format. Provided code does not show you are doing it right. The sample you are referring to is using default codecs.
Related reading: Support LATM AAC in MP4 container
Then, you need to make sure that both incoming data and your FFmpeg output setup are in agreement about whether the data has or does not have ADTS headers, the provided code does not shed any light on this.
Furthermore, I am afraid you might be preparing your audio data incorrectly. The sample in question generates raw audio data and applies encoder to produce compressed content using avcodec_encode_audio2. Then a packed with compressed audio is being sent to writing using av_interleaved_write_frame. The way you attached your code snippets to the question makes me thing you are doing it wrong. For starters, you still don't show relevant code which makes me think you have troubles identifying what code is relevant exactly. Then you are dealing with your AAC data as if it was raw PCM audio in get_audio_frame code snippet whereas you are interested in reviewing FFmpeg sample code with the thought in mind that you already have compressed AAC data and sample gets to thins point after return from avcodec_encode_audio2 call. This is where you are supposed to merge your code and the sample.

How to use FFMPEG to play H.264 stream from NAL units that are stored as video AVPackets

I am writing client-server system that uses FFMPEG library to parse H.264 stream into NAL units on the server side, then uses channel coding to send them over network to client side, where my application must be able to play video.
The question is how to play received AVPackets (NAL units) in my application as video stream.
I have found this tutorial helpful and used it as base for both server and client side.
Some sample code or resource related to playing video not from file, but from data inside program using FFMPEG library would be very helpful.
I am sure that received information will be sufficient to play video, because I tried to save received data as .h264 or .mp4 file and it can be played by VLC player.
Of what I understand from your question, you have the AVPackets and want to play a video. In reality this is two problems; 1. decoding your packets, and 2. playing the video.
For decoding your packets, with FFmpeg, you should take a look at the documentation for AVPacket, AVCodecContext and avcodec_decode_video2 to get some ideas; the general idea is that you want to do something (just wrote this in the browser, take with a grain of salt) along the lines of:
//the context, set this appropriately based on your video. See the above links for the documentation
AVCodecContext *decoder_context;
std::vector<AVPacket> packets; //assume this has your packets
...
AVFrame *decoded_frame = av_frame_alloc();
int ret = -1;
int got_frame = 0;
for(AVPacket packet : packets)
{
avcodec_get_frame_defaults(frame);
ret = avcodec_decode_video2(decoder_context, decoded_frame, &got_frame, &packet);
if (ret <= 0) {
//had an error decoding the current packet or couldn't decode the packet
break;
}
if(got_frame)
{
//send to whatever video player queue you're using/do whatever with the frame
...
}
got_frame = 0;
av_free_packet(&packet);
}
It's a pretty rough sketch, but that's the general idea for your problem of decoding the AVPackets. As for your problem of playing the video, you have many options, which will likely depend more on your clients. What you're asking is a pretty large problem, I'd advise familiarizing yourself with the FFmpeg documentation and the provided examples at the FFmpeg site. Hope that makes sense