Currently I am receiving video stream(H264 encoded buffer) and audio stream(PCMU encoded buffer) from remote end from which I can decode and render these as audio and video. Now I want to provide some APIs like -
string fileName = "dir/dir2/..../rec.mp4";
startRecord()
stopRecord()
User can start recording from any time and stop recording and the video & audio stream will be written as combined mp4 file. I can use ffmpeg by which I can merge a .h264 and .wav file as .mp4 file. But I want to do it programmatically directly from streams(not .h264 or .wav file) using any library or write my own. Is it possible?
See this answer for details. However, mp4 doesn't support G.711 PCM mu-law encoded data, either avi or mov can be used or trans-code the data from pcm to aac will work.
Related
I can record audio using MediaFoundation, but it only gives me a PCM Wave buffer. I want to grab the full buffer, encode it to MP3, and then use the new buffer for networking stuff.
What is the right way to encode the audio after receiving the samples? I have gotten lost reading through MediaSession, MediaSinks, SinkWriter, Transcode API, Transform API, Source Resolver, etc.
I see there is an MP3 encoder object, but I cant find the documentation on how to use it. I also found an MP3 MediaSink but im not sure how the MediaSink fits in with the SourceReader / SinkWriter schema or how to create/use the IMFByteStream it requires.
Is the MediaFoundation the right WinAPI for the task?
I was able to successfully encode an MP4 file which contains H.264 encoded video only (using IMFSinkWriter interface). Now I want to add an audio stream to it.
Whenever I try to create a sink writer for an audio using:
MFCreateSinkWriterFromURL(filePath, null, null, &pSinkWriter)
it deletes the previous file and writes only the audio (well, according to this link it is expected).
So my question is: how to add an audio stream to an existing file which contains only video stream?
Or, If I have both raw data from audio and video how do I encode both of them into a single media file (I suppose I have to do something called multiplexing. If so, can someone provide me helpful references)?
Sink Writer API creates a media file from scratch when you do IMFSinkWriter::BeginWriting to final completion when you do IMFSinkWriter::Finalize. You don't add new streams to finalized file (well, you can do it, but it works differently - see last paragraph below).
To create a media file with both video and audio you need to add two streams before you begin. Two calls IMFSinkWriter::AddStream, then two IMFSinkWriter::SetInputMediaType, then you start writing IMFSinkWriter::BeginWriting and you feed both video and audio data IMFSinkWriter::WriteSample providing respective stream index.
To adding a new stream to already existing file you need to create a completely new file. One of the options you have is to read already compressed data from existing file you have and write it to the new file using IMFSinkWriter::WriteSample method without re-compression. At the same time second stream can be written doing the compression. This way you can create a video and audio MP4 file by taking video from existing file and adding/encoding an additional audio track.
I'm developing app which sends mpeg2ts stream using FFMPEG API.(avio_open, avformat_new_stream etc..)
The problem is that the app already has AAC-LC audio so audio frame does not need to be encoded because my app just bypass data received from socket buffer.
To open and send mpegts using FFMPEG, I must have AVFormattContext data which is created from FFMPEG API for encoder as far as I know.
Can I create AVFormatContext manually with encoded AAC-LC data? or I should decode and encode the data? The information I know is samplerate, codec, bitrate..
Any help will be greatly appreciated. Thanks in advance.
Yes, you can use the encoded data as-is if your container supports it. There are two steps involved here - encoding and muxing. Encoding compress the data, muxing mixes it together in the output file, so the packets are properly interleaved. Muxing example in FFMpeg distribution helped me with this.
You might also take a look at the following class: https://sourceforge.net/p/karlyriceditor/code/HEAD/tree/src/ffmpegvideoencoder.cpp - this file is from one of my projects, and contains video encoder. Starting from the line 402 you'll see the setup for non-converted audio - it is kind of a hackish way, but it worked. Unfortunately I still end up reencoding audio because for my formats it was not possible to achieve frame-perfect synchronization which I needed
I'm doing some integration work with video (H.264) and audio (AAC) from an IP camera.
I've made a bit of progress and I can store the video & audio streams individually with the ability to play it back using VLC player. The H.264 is being stored in Annex B format and the audio is using an adts formatted file.
I'm now trying to mux the streams into an MP4 file without doing any decoding or encoding but so far haven't managed to find the answer.
I can do this manually with ffmpeg:
ffmpeg -i recording.h264 -i recording.aac -vcodec copy -acodec copy -absf aac_adtstoasc recording.mp4
How do I do this with the ffmpeg library from C++?
Check out the muxing sample; the key is to keep track of your audio/video timestamps and write the next one in time using av_interleaved_write_frame.
I'm looking to write already compressed (h264) image data into an MPEG-4 video file. Since this code needs to be optimized to run on an embedded platform the code should be as simple as possible.
Best would be to just give some header information (like height width format fourcc etc.) and a filename and the compressed data and have that transformed into a data chunck and writen to that file.
So what i need either of these:
MPEG-4 header information (what goes where exactly)
Is there a main header or are there just headers for each data chunck
What header information is needed for a single video stream (rectangular)
What header information is needed for adding audio
A simple MPEG-4 file writer that does not have to do the compression itself and also allows to add audio frames. (c/c++)
.MP4 file format is described in MPEG-4 Part 14 specification. It is not just main header and subheaders, it has certain hierarchy and so called boxes in there. Some of your choice to write data into .MP4 file:
FFmpeg (libavcodec, libavformat) - related Q and related code link
In Windows via DirectShow API - GDCL MP4 Multiplexer or numerous commerical similar offerings
In Windows via Media Foundation API - MPEG-4 File Sink