I'm currently having issues trying to encapsulate raw H264 nal packets into a mp4 container. Instead of writing them to disk however, I want to have the result stored in memory. I followed this approach Raw H264 frames in mpegts container using libavcodec but haven't been successful so far.
First, is this the right way to write to memory? I have a small struct in my header
struct IOOutput {
uint8_t* outBuffer;
int bytesSet;
};
where I initialize the buffer and bytesset. I then initialize my AVIOContext variable
AVIOContext* pIOCtx = avio_alloc_context(pBuffer, iBufSize, 1, outptr, NULL, write_packet, NULL);
where outptr is a void pointer to IOOutput output, and write_packet looks like the following
int write_packet (void *opaque, uint8_t *buf, int buf_size) {
IOOutput* out = reinterpret_cast<IOOutput*>(opaque);
memcpy(out->outBuffer+out->bytesSet, buf, buf_size);
out->bytesSet+=buf_size;
return buf_size;
}
I then set
fc->pb = pIOCtx;
fc->flags = AVFMT_FLAG_CUSTOM_IO;
on my AVFormatContext *fc variable.
Then, whenever I encode the nal packets I have from a frame, I write them to the AVFormatContext via av_interleaved_write_frame and then get the mp4 contents via
void getBufferContent(char* buffer) {
memcpy(buffer, output.outBuffer, output.bytesSet);
output.bytesSet=0;
}
and thus reset the variable bytesSet, so during the next writing operation bytes will be inserted at the start of the buffer. Is there a better way to do this? Is this actually a valid way to do it? Does FFMPEG do any reading operation if I only do call av_interleaved_write_frame and avformat_write_header in order to add packets?
Thank you very much in advance!
EDIT
Here is the code regarding the muxing process - in my encode Function I have something like
int frame_size = x264_encoder_encode(obj->mEncoder, &obj->nals, &obj->i_nals, obj->pic_in, obj->pic_out);
int total_size=0;
for(int i=0; i<obj->i_nals;i++)
{
if ( !obj->fc ) {
obj->create( obj->nals[i].p_payload, obj->nals[i].i_payload );
}
if ( obj->fc ) {
obj->write_frame( obj->nals[i].p_payload, obj->nals[i].i_payload);
}
}
// Here I get the output values
int currentBufferSize = obj->output.bytesSet;
char* mem = new char[currentBufferSize];
obj->getBufferContent(mem);
And the create and write functions look like this
int create(void *p, int len) {
AVOutputFormat *of = av_guess_format( "mp4", 0, 0 );
fc = avformat_alloc_context();
// Add video stream
AVStream *pst = av_new_stream( fc, 0 );
vi = pst->index;
void* outptr = (void*) &output;
// Create Buffer
pIOCtx = avio_alloc_context(pBuffer, iBufSize, 1, outptr, NULL, write_packet, NULL);
fc->oformat = of;
fc->pb = pIOCtx;
fc->flags = AVFMT_FLAG_CUSTOM_IO;
pcc = pst->codec;
AVCodec c= {0};
c.type= AVMEDIA_TYPE_VIDEO;
avcodec_get_context_defaults3( pcc, &c );
pcc->codec_type = AVMEDIA_TYPE_VIDEO;
pcc->codec_id = codec_id;
pcc->bit_rate = br;
pcc->width = w;
pcc->height = h;
pcc->time_base.num = 1;
pcc->time_base.den = fps;
}
void write_frame( const void* p, int len ) {
AVStream *pst = fc->streams[ vi ];
// Init packet
AVPacket pkt;
av_init_packet( &pkt );
pkt.flags |= ( 0 >= getVopType( p, len ) ) ? AV_PKT_FLAG_KEY : 0;
pkt.stream_index = pst->index;
pkt.data = (uint8_t*)p;
pkt.size = len;
pkt.dts = AV_NOPTS_VALUE;
pkt.pts = AV_NOPTS_VALUE;
av_interleaved_write_frame( fc, &pkt );
}
See the AVFormatContext.pb documentation. You set it correctly, but you shouldn't touch AVFormatContext.flags. Also, make sure you set it before calling avformat_write_header().
When you say "it doesn't work", what exactly doesn't work? Is the callback not invoked? Is the data in it not of the expected type/format? Something else? If all you want to do is write raw nal packets, then you could just take encoded data directly from the encoder (in the AVPacket), that's the raw nal data. If you use libx264's api directly, it even gives you each nal individually so you don't need to parse it.
Related
I am currently trying to learn audio programming. My goal is to open a wav file, extract everything and play the samples with RtAudio.
I made a WaveLoader class which let's me extract the samples and meta data. I used this guide to do that and I checked that everything is correct with 010 editor. Here is a snapshot of 010 editor showing the structure and data.
And this is how i store the raw samples inside WaveLoader class:
data = new short[wave_data.payloadSize]; // - Allocates memory size of chunk size
if (!fread(data, 1, wave_data.payloadSize, sound_file))
{
throw ("Could not read wav data");
}
If i print out each sample I get : 1, -3, 4, -5 ... which seems ok.
The problem is that I am not sure how I can play them. This is what I've done:
/*
* Using PortAudio to play samples
*/
bool Player::Play()
{
ShowDevices();
rt.showWarnings(true);
RtAudio::StreamParameters oParameters; //, iParameters;
oParameters.deviceId = rt.getDefaultOutputDevice();
oParameters.firstChannel = 0;
oParameters.nChannels = mAudio.channels;
//iParameters.deviceId = rt.getDefaultInputDevice();
//iParameters.nChannels = 2;
unsigned int sampleRate = mAudio.sampleRate;
// Use a buffer of 512, we need to feed callback with 512 bytes everytime!
unsigned int nBufferFrames = 512;
RtAudio::StreamOptions options;
options.flags = RTAUDIO_SCHEDULE_REALTIME;
options.flags = RTAUDIO_NONINTERLEAVED;
//¶meters, NULL, RTAUDIO_FLOAT64,sampleRate, &bufferFrames, &mCallback, (void *)&rawData
try {
rt.openStream(&oParameters, NULL, RTAUDIO_SINT16, sampleRate, &nBufferFrames, &mCallback, (void*) &mAudio);
rt.startStream();
}
catch (RtAudioError& e) {
std::cout << e.getMessage() << std::endl;
return false;
}
return true;
}
/*
* RtAudio Callback
*
*/
int mCallback(void * outputBuffer, void * inputBuffer, unsigned int nBufferFrames, double streamTime, RtAudioStreamStatus status, void * userData)
{
unsigned int i = 0;
short *out = static_cast<short*>(outputBuffer);
auto *data = static_cast<Player::AUDIO_DATA*>(userData);
// if i is more than our data size, we are done!
if (i > data->dataSize) return 1;
// First time callback is called data->ptr is 0, this means that the offset is 0
// Second time data->ptr is 1, this means offset = nBufferFrames (512) * 1 = 512
unsigned int offset = nBufferFrames * data->ptr++;
printf("Offset: %i\n", offset);
// First time callback is called offset is 0, we are starting from 0 and looping nBufferFrames (512) times, this gives us 512 bytes
// Second time, the offset is 1, we are starting from 512 bytes and looping to 512 + 512 = 1024
for (i = offset; i < offset + nBufferFrames; ++i)
{
short sample = data->rawData[i]; // Get raw sample from our struct
*out++ = sample; // Pass to output buffer for playback
printf("Current sample value: %i\n", sample); // this is showing 1, -3, 4, -5 check 010 editor
}
printf("Current time: %f\n", streamTime);
return 0;
}
Inside callback function, when I print out sample values I get exactly like 010 editor? Why isnt rtaudio playing them. What is wrong here? Do I need to normalize sample values to between -1 and 1?
Edit:
The wav file I am trying to play:
Chunksize: 16
Format: 1
Channel: 1
SampleRate: 48000
ByteRate: 96000
BlockAlign: 2
BitPerSample: 16
Size of raw samples total: 2217044 bytes
For some reason it works when I pass input parameters to the openStream()
RtAudio::StreamParameters oParameters, iParameters;
oParameters.deviceId = rt.getDefaultOutputDevice();
oParameters.firstChannel = 0;
//oParameters.nChannels = mAudio.channels;
oParameters.nChannels = mAudio.channels;
iParameters.deviceId = rt.getDefaultInputDevice();
iParameters.nChannels = 1;
unsigned int sampleRate = mAudio.sampleRate;
// Use a buffer of 512, we need to feed callback with 512 bytes everytime!
unsigned int nBufferFrames = 512;
RtAudio::StreamOptions options;
options.flags = RTAUDIO_SCHEDULE_REALTIME;
options.flags = RTAUDIO_NONINTERLEAVED;
//¶meters, NULL, RTAUDIO_FLOAT64,sampleRate, &bufferFrames, &mCallback, (void *)&rawData
try {
rt.openStream(&oParameters, &iParameters, RTAUDIO_SINT16, sampleRate, &nBufferFrames, &mCallback, (void*) &mAudio);
rt.startStream();
}
catch (RtAudioError& e) {
std::cout << e.getMessage() << std::endl;
return false;
}
return true;
It was so random when I was trying to playback my mic. I left input parameters and my wav file was suddenly playing. Is this is a bug?
(I'm not fluent in english i'll try to do my best)
I try to code (C++) a simple mkv player. I'm very new in this subject, so I discover all I need little by little. For the beginning, I use VP8 codec for video and Vorbis for audio.
The video side seem ok for now, but I'm in trouble with audio.
I can't figure out the loop logic to decode the audio frames I get from mkvparser with the libvorbis.
I looked up to this sample and this brief explanation but can't manage to make it work in my case. And I didn't find other simple examples.
Here is a chunk of my code:
const mkvparser::Block* const pBlock = m_pMkvContext->pBlockEntry->GetBlock();
const mkvparser::Track* const pTrack = m_pMkvContext->pTracks->GetTrackByNumber( (unsigned long)pBlock->GetTrackNumber() );
if ( pTrack != NULL )
{
const long long trackType = pTrack->GetType();
const int frameCount = pBlock->GetFrameCount();
if ( frameCount > 0 )
{
const mkvparser::Block::Frame& oFrame = pBlock->GetFrame( 0 );
unsigned char* pData = (unsigned char*)malloc( (size_t)oFrame.len );
oFrame.Read( &m_pMkvContext->oReader, pData );
if ( trackType == mkvparser::Track::kVideo )
{
// i'm ok here
}
else if ( trackType == mkvparser::Track::kAudio )
{
// what to do here with my audio frame data ?
}
free( pData );
}
}
And maybe the way I get frames is good for video and not for audio...
Do you guys know some good resources to share about it? Or some advices?
Thanks for help !
[EDIT] : I forgot to add one of my try:
bool MoviePlayer::DecodeAudioData( unsigned char* pData, uint32 iSize )
{
int ret;
char* pBuffer = NULL;
pBuffer = ogg_sync_buffer( &m_pOVContext->oOggSyncState, iSize );
memcpy( pBuffer, pData, iSize );
ogg_sync_wrote( &m_pOVContext->oOggSyncState, iSize );
ret = ogg_sync_pageout( &m_pOVContext->oOggSyncState, &m_pOVContext->oOggPage );
ret = ogg_stream_init( &m_pOVContext->oOggStreamState, ogg_page_serialno(&m_pOVContext->oOggPage) );
ret = ogg_stream_pagein( &m_pOVContext->oOggStreamState, &m_pOVContext->oOggPage );
int iPacketsCount = ogg_page_packets( &m_pOVContext->oOggPage );
for ( int i = 0; i < iPacketsCount; ++i )
{
ret = ogg_stream_packetout(&m_pOVContext->oOggStreamState, &m_pOVContext->oOggPacket);
// do something with the packet...
}
return true;
}
It crashes at ogg_sync_pageout, as my ogg_page was not correctly initialized.
But, not coming from a proper .ogg file as in the examples i found, i don't know how to correctly initialize the vorbis structures.
https://matroska.org/technical/specs/codecid/index.html
in A_VORBIS section
The private data contains the first three Vorbis packet in order....
and the codec private is here
https://matroska.org/technical/specs/index.html
"CodecPrivate 3 [63][A2]"
I'm working with a FFmpeg project right now, I have to store the data of AVPacket which from av_read_frame, and fill data to new AVPacket for following decoding.
Here is my problem: when I try to new & free an AVPacket, memory leaks always happen.
I am just doing a simple testing:
for(;;) {
AVPacket pkt;
av_new_packet(&pkt, 1000);
av_init_packet(&pkt);
av_free_packet(&pkt);
}
What am I doing wrong?
av_new_packet creates a packet and allocates data
av_init_packet
sets all packet members to default, and sets data pointer to NULL,
the leak is here
av_free_packet clears all visible members, but
your data is already leaking
If you want FFmpeg to allocate the data for you, do not call av_init_packet. If you want to handle the data yourself, allocate the packet object on the stack and set its data yourself (and free it yourself):
AVPacket pkt;
av_init_packet(&pkt);
pkt.data = dataBuffer;
pkt.size = dataBufferSize;
// use your packet
// free your dataBuffer
I just read the FFmpeg 2.2 AVPacket.c source code.
int av_new_packet(AVPacket *pkt, int size) {
AVBufferRef *buf = NULL;
int ret = packet_alloc(&buf, size);
if (ret < 0)
return ret;
av_init_packet(pkt);
pkt->buf = buf;
pkt->data = buf->data;
pkt->size = size;
#if FF_API_DESTRUCT_PACKET
FF_DISABLE_DEPRECATION_WARNINGS
pkt->destruct = dummy_destruct_packet;
FF_ENABLE_DEPRECATION_WARNINGS
#endif
return 0;
}
void av_init_packet(AVPacket *pkt) {
pkt->pts = AV_NOPTS_VALUE;
pkt->dts = AV_NOPTS_VALUE;
pkt->pos = -1;
pkt->duration = 0;
pkt->convergence_duration = 0;
pkt->flags = 0;
pkt->stream_index = 0;
#if FF_API_DESTRUCT_PACKET
FF_DISABLE_DEPRECATION_WARNINGS
pkt->destruct = NULL;
FF_ENABLE_DEPRECATION_WARNINGS
#endif
pkt->buf = NULL;
pkt->side_data = NULL;
pkt->side_data_elems = 0;
}
I don't really know about the defines FF_API_DESTRUCT_PACKET, FF_DISABLE_DEPRECATION_WARNINGS, FF_ENABLE_DEPRECATION_WARNINGS
Some reason makes destruct of av_free_packet leak
According the source code, av_init_packet is called in av_new_packet and av_new_packet already allocates the AVBuffer, so if you want to set the data to new AVPacket.
Just memory copy to data of AVPacket, and call av_free_packet when you are done.
I have encoded some frames using x264, using x264_encoder_encode and after that I have created AVPackets using a function like this:
bool PacketizeNals( uint8_t* a_pNalBuffer, int a_nNalBufferSize, AVPacket* a_pPacket )
{
if ( !a_pPacket )
return false;
a_pPacket->data = a_pNalBuffer;
a_pPacket->size = a_nNalBufferSize;
a_pPacket->stream_index = 0;
a_pPacket->flags = AV_PKT_FLAG_KEY;
a_pPacket->pts = int64_t(0x8000000000000000);
a_pPacket->dts = int64_t(0x8000000000000000);
}
I call this function like this:
x264_nal_t* nals;
int num_nals = encode_frame(pic, &nals);
for (int i = 0; i < num_nals; i++)
{
AVPacket* pPacket = ( AVPacket* )av_malloc( sizeof( AVPacket ) );
av_init_packet( pPacket );
if ( PacketizeNals( nals[i].p_payload, nals[i].i_payload, pPacket ) )
{
packets.push_back( pPacket );
}
}
Now what I want to do is to decode these AVPackets using avcodec_decode_video2. I think the problem is that I haven't initialized properly the decoder because to encode I used "ultrafast" profile and "zerolatency" tune ( x264 ) and to decode I don't know how to specify to ffmpeg these options.
In some examples I have read people initialize the decoder using the file where the video is stored, but in this case I have directly the AVPackets.
What I'm doing to try to decode is:
avcodec_init();
avcodec_register_all();
AVCodec* pCodec;
pCodec=avcodec_find_decoder(CODEC_ID_H264);
AVCodecContext* pCodecContext;
pCodecContext=avcodec_alloc_context();
avcodec_open(pCodecContext,pCodec);
pCodecContext->width = 320;
pCodecContext->height = 200;
pCodecContext->extradata = NULL;
unsigned int nNumPackets = packets.size();
int frameFinished = 0;
for ( auto it = packets.begin(); it != packets.end(); it++ )
{
AVFrame* pFrame;
pFrame = avcodec_alloc_frame();
AVPacket* pPacket = *it;
int iReturn = avcodec_decode_video2( pCodecContext, pFrame, &frameFinished, pPacket );
}
But in iReturn always is -1.
Can anyone help me? Sorry if my knowledge in this area es low, I'm new.
Thanks.
I have written a simple client/server application that streams raw RGB video using lib x264 for encoding and ffmpeg for decoding.
You can find the code here: https://github.com/filippobrizzi/raw_rgb_straming
It shows how to setup x264 and ffmpeg to encode/decode.
Right now you initialize the decoder like
pCodecContext->extradata = NULL;
this is not correct. You need to allocate a memory for this field and copy data from the encoder's AVCodecContext::extradata into the allocated buffer. AVCodecContext::extradata_size specifies size of this extradata buffer in bytes
Make sure that you are building correct packets. See how this is done in the ffmpeg: http://ffmpeg.org/doxygen/trunk/libx264_8c_source.html (static int encode_nals(AVCodecContext *ctx, AVPacket *pkt, x264_nal_t *nals, int nnal) and static int X264_frame(AVCodecContext *ctx, AVPacket *pkt, const AVFrame *frame, int *got_packet))
I have found this function which uses libjpeg to write to a file:
int write_jpeg_file( char *filename )
{
struct jpeg_compress_struct cinfo;
struct jpeg_error_mgr jerr;
/* this is a pointer to one row of image data */
JSAMPROW row_pointer[1];
FILE *outfile = fopen( filename, "wb" );
if ( !outfile )
{
printf("Error opening output jpeg file %s\n!", filename );
return -1;
}
cinfo.err = jpeg_std_error( &jerr );
jpeg_create_compress(&cinfo);
jpeg_stdio_dest(&cinfo, outfile);
/* Setting the parameters of the output file here */
cinfo.image_width = width;
cinfo.image_height = height;
cinfo.input_components = bytes_per_pixel;
cinfo.in_color_space = color_space;
/* default compression parameters, we shouldn't be worried about these */
jpeg_set_defaults( &cinfo );
/* Now do the compression .. */
jpeg_start_compress( &cinfo, TRUE );
/* like reading a file, this time write one row at a time */
while( cinfo.next_scanline < cinfo.image_height )
{
row_pointer[0] = &raw_image[ cinfo.next_scanline * cinfo.image_width * cinfo.input_components];
jpeg_write_scanlines( &cinfo, row_pointer, 1 );
}
/* similar to read file, clean up after we're done compressing */
jpeg_finish_compress( &cinfo );
jpeg_destroy_compress( &cinfo );
fclose( outfile );
/* success code is 1! */
return 1;
}
I would actually need to write the jpeg compressed image just to memory buffer, without saving it to a file, to save time. Could somebody give me an example how to do it?
I have been searching the web for a while but the documentation is very rare if any and examples are also difficult to come by.
You can define your own destination manager quite easily. The jpeg_compress_struct contains a pointer to a jpeg_destination_mgr, which contains a pointer to a buffer, a count of space left in the buffer, and 3 pointers to functions:
init_destination (j_compress_ptr cinfo)
empty_output_buffer (j_compress_ptr cinfo)
term_destination (j_compress_ptr cinfo)
You need to fill in the function pointers before you make the first call into the jpeg library, and let those functions handle the buffer. If you create a buffer that is larger than the largest possible output that you expect, this becomes trivial; init_destination just fills in the buffer pointer and count, and empty_output_buffer and term_destination do nothing.
Here's some sample code:
std::vector<JOCTET> my_buffer;
#define BLOCK_SIZE 16384
void my_init_destination(j_compress_ptr cinfo)
{
my_buffer.resize(BLOCK_SIZE);
cinfo->dest->next_output_byte = &my_buffer[0];
cinfo->dest->free_in_buffer = my_buffer.size();
}
boolean my_empty_output_buffer(j_compress_ptr cinfo)
{
size_t oldsize = my_buffer.size();
my_buffer.resize(oldsize + BLOCK_SIZE);
cinfo->dest->next_output_byte = &my_buffer[oldsize];
cinfo->dest->free_in_buffer = my_buffer.size() - oldsize;
return true;
}
void my_term_destination(j_compress_ptr cinfo)
{
my_buffer.resize(my_buffer.size() - cinfo->dest->free_in_buffer);
}
cinfo->dest->init_destination = &my_init_destination;
cinfo->dest->empty_output_buffer = &my_empty_output_buffer;
cinfo->dest->term_destination = &my_term_destination;
There is a predefined function jpeg_mem_src defined in jdatasrc.c. The simplest usage example:
unsigned char *mem = NULL;
unsigned long mem_size = 0;
struct jpeg_compress_struct cinfo;
struct jpeg_error_mgr jerr;
cinfo.err = jpeg_std_error(&jerr);
jpeg_create_compress(&cinfo);
jpeg_mem_dest(&cinfo, &mem, &mem_size);
// do compression
// use mem buffer
Do not forget to deallocate your buffer.
I have tried Mark's solution and on my platform it always gives SEGMENTATION FALUT error when it executes
cinfo->dest->term_destination = &my_term_destination;
And I turned to the jpeglib source codes (jdatadst.c) and found this:
jpeg_mem_dest (j_compress_ptr cinfo, unsigned char ** outbuffer, unsigned long * outsize)
just below the method jpeg_stdio_dest(), and I've tried it by simply fill in the address of the buffer(char*) and the address of the buffer size(int). The destination manager automatically allocates memory for the buffer and the program need to free the memory after use.
It successfully runs on my platform, Beaglebone Black with the pre-installed Angstrom Linux. My libjpeg version is 8d.
All you need to do is pass a FILE-like object to jpeg_stdio_dest().
unsigned char ***image_ptr
unsigned char* ptr;
unsigned char** image_buf;
for(int i=0;i<h;i++){
image_buf[i] = new unsigned char[w*o];
}
ptr = image_buf[0];
while (info.output_scanline < info.image_height) {
jpeg_read_scanlines(&info,&ptr,1);
ptr = image_buf[c];
c++;
}
*image_ptr = image_buf;
This is all you need to read.
JSAMPROW row_pointer;
while (info.next_scanline < info.image_height) {
row_pointer = &image_buf[info.next_scanline][0];
(void) jpeg_write_scanlines(&info, &row_pointer, 1);
}
And this is all you need to write.