I want to read a per-frame timecode out of a video file using libav (FFMPEG). I've started by digging into FFProbe. Using this as a starting point for my code, I can get to the AVStream that has the timecode in it. From there, I can use the dictionary to look at the stream's metadata.
int show_stream(WriterContext *w, AVFormatContext *fmt_ctx, int stream_idx, InputStream *ist, int in_program)
{
AVStream *stream = ist->st;
...
auto tcr = av_dict_get(stream->metadata, "timecode", NULL, 0);
std::cerr << "Timecode: " << tcr->value << ", Total Frames: << stream->nb_frames << "\n";
The time code is the correct one that was embedded into the video. The nb_frames is correctly the total number of video frames that I have. What I can't get is the per-frame timecode. I don't want to compute it if I don't have to, I want to know exactly what was stamped on each frame. Is this possible?
// Pseudocode for what I want
for(const auto& f : allOfMyFrames)
{
std::cerr << "Frame number " << f.number << ", Timecode: " << f.timecode << "\n";
}
Timecode, such as that in a MOV/MP4, is just a single packet with the starting timecode expressed as a rate-adjusted frame count. There is no per-frame timecode.
Related
I'm trying use pylon library for work with industrial camera (Basler ACE acA1600-20uc).
I started with examples program for grabbing pictures from camera.
Problem is, that example program is "Build Succeeded" (xcode) but nothing happens. Command line is still clean, no logs, no exit code. Nothing. I expect some informations because several cout << in code below.
Where's the problem? Give me some hint. Thank you.
// Grab.cpp
/*
Note: Before getting started, Basler recommends reading the Programmer's Guide topic
in the pylon C++ API documentation that gets installed with pylon.
If you are upgrading to a higher major version of pylon, Basler also
strongly recommends reading the Migration topic in the pylon C++ API documentation.
This sample illustrates how to grab and process images using the CInstantCamera class.
The images are grabbed and processed asynchronously, i.e.,
while the application is processing a buffer, the acquisition of the next buffer is done
in parallel.
The CInstantCamera class uses a pool of buffers to retrieve image data
from the camera device. Once a buffer is filled and ready,
the buffer can be retrieved from the camera object for processing. The buffer
and additional image data are collected in a grab result. The grab result is
held by a smart pointer after retrieval. The buffer is automatically reused
when explicitly released or when the smart pointer object is destroyed.
*/
// Include files to use the PYLON API.
#include <pylon/PylonIncludes.h>
#ifdef PYLON_WIN_BUILD
# include <pylon/PylonGUI.h>
#endif
// Namespace for using pylon objects.
using namespace Pylon;
// Namespace for using cout.
using namespace std;
// Number of images to be grabbed.
static const uint32_t c_countOfImagesToGrab = 100;
int main(int argc, char* argv[])
{
// The exit code of the sample application.
int exitCode = 0;
// Before using any pylon methods, the pylon runtime must be initialized.
PylonInitialize();
try
{
// Create an instant camera object with the camera device found first.
CInstantCamera camera( CTlFactory::GetInstance().CreateFirstDevice());
// Print the model name of the camera.
cout << "Using device " << camera.GetDeviceInfo().GetModelName() << endl;
// The parameter MaxNumBuffer can be used to control the count of buffers
// allocated for grabbing. The default value of this parameter is 10.
camera.MaxNumBuffer = 5;
// Start the grabbing of c_countOfImagesToGrab images.
// The camera device is parameterized with a default configuration which
// sets up free-running continuous acquisition.
camera.StartGrabbing( c_countOfImagesToGrab);
// This smart pointer will receive the grab result data.
CGrabResultPtr ptrGrabResult;
// Camera.StopGrabbing() is called automatically by the RetrieveResult() method
// when c_countOfImagesToGrab images have been retrieved.
while ( camera.IsGrabbing())
{
// Wait for an image and then retrieve it. A timeout of 5000 ms is used.
camera.RetrieveResult( 5000, ptrGrabResult, TimeoutHandling_ThrowException);
// Image grabbed successfully?
if (ptrGrabResult->GrabSucceeded())
{
// Access the image data.
cout << "SizeX: " << ptrGrabResult->GetWidth() << endl;
cout << "SizeY: " << ptrGrabResult->GetHeight() << endl;
const uint8_t *pImageBuffer = (uint8_t *) ptrGrabResult->GetBuffer();
cout << "Gray value of first pixel: " << (uint32_t) pImageBuffer[0] << endl << endl;
#ifdef PYLON_WIN_BUILD
// Display the grabbed image.
Pylon::DisplayImage(1, ptrGrabResult);
#endif
}
else
{
cout << "Error: " << ptrGrabResult->GetErrorCode() << " " << ptrGrabResult->GetErrorDescription() << endl;
}
}
}
catch (const GenericException &e)
{
// Error handling.
cerr << "An exception occurred." << endl
<< e.GetDescription() << endl;
exitCode = 1;
}
// Comment the following two lines to disable waiting on exit.
cerr << endl << "Press Enter to exit." << endl;
while( cin.get() != '\n');
// Releases all pylon resources.
PylonTerminate();
return exitCode;
}
check if you are compiling the app target, not a framework or other .. take a look at the right of RUN/STOp button
Oh yes,
you are right thank you! Because it is a lot of examples in one project i've builded something other again and again. :)
Does anyone know how to get OpenCV to grab all the frames from a video file?
I've been trying to just grab frames from video files (.wmv files, in particular), but on most videos, I end up with "nan" as my framerate and it only gets a single frame from my video and doesn't think there are any more. However, on at least one video, it succeeds and gets the right framerate. I tried to manually set the frame rate, but this did not work.
One thing to note is that it seems like the videos where it doesn't work are very short (~5 seconds long). However, I have not thoroughly tested this theory as I don't have very many videos (only ~10 videos).
The minimum code necessary to reproduce this is below:
int main(int argc, char** argv)
{
VideoCapture capture;
char* video = argv[1];
int flag = arg_parse(argc, argv);
capture.open(video);
//capture.set(CV_CAP_PROP_FPS, 25); // Trying to set frame rate.
std::cout << "frame rate: " << capture.get(CV_CAP_PROP_FPS) << std::endl;
if(!capture.isOpened()) {
fprintf(stderr, "Could not initialize capturing..\n");
return -1;
}
while(true) {
Mat frame;
int i, j, c;
// get a new frame
capture >> frame;
std::cout << "GOT FRAME!" << std::endl;
if(frame.empty()) {
std::cout << "breaking..." << std::endl;
break;
}
}
return 0;
}
Thanks so much!
I have two functions:
a internet-socket function which gets mp3-data and writes it to file ,
a function which decodes mp3-files.
However, I would rather decode the data, which is currently written to disk, be decoded in-memory by the decode function.
My decode function looks like this, and it is all initialized via
avformat_open_input(AVCodecContext, filename, NULL, NULL)
How can I read in the AVCodecContext without a filename, and instead using only the in-memory buffer?
I thought I would post some code to illustrate how to achieve this, I have tried to comment but am pressed for time, however it should all be relatively straightforward stuff. Return values are based on interpolation of the associated message into a hex version of 1337 speak converted to decimal values, and I have tried to keep it as light as possible in tone:)
#include <iostream>
extern "C"
{
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/avutil.h>
};
std::string tooManyChannels = "The audio stream (and its frames) has/have too many channels to properly fit in\n to frame->data. Therefore, to access the audio data, you need to use\nframe->extended_data to access the audio data."
"It is a planar store, so\neach channel is in a different element.\n"
" E.G.: frame->extended_data[0] has the data for channel 1\n"
" frame->extended_data[1] has the data for channel 2\n"
"And so on.\n";
std::string nonPlanar = "Either the audio data is not planar, or there is not enough room in\n"
"frame->data to store all the channel data. Either use\n"
"frame->data\n or \nframe->extended_data to access the audio data\n"
"both should just point to the same data in this instance.\n";
std::string information1 = "If the frame is planar, each channel is in a separate element:\n"
"frame->data[0]/frame->extended_data[0] contains data for channel 1\n"
"frame->data[1]/frame->extended_data[1] contains data for channel 2\n";
std::string information2 = "If the frame is in packed format( and therefore not planar),\n"
"then all the data is contained within:\n"
"frame->data[0]/frame->extended_data[0] \n"
"Similar to the manner in which some image formats have RGB(A) pixel data packed together,\n"
"rather than containing separate R G B (and A) data.\n";
void printAudioFrameInfo(const AVCodecContext* codecContext, const AVFrame* frame)
{
/*
This url: http://ffmpeg.org/doxygen/trunk/samplefmt_8h.html#af9a51ca15301871723577c730b5865c5
contains information on the type you will need to utilise to access the audio data.
*/
// format the tabs etc. in this string to suit your font, they line up for mine but may not for yours:)
std::cout << "Audio frame info:\n"
<< "\tSample count:\t\t" << frame->nb_samples << '\n'
<< "\tChannel count:\t\t" << codecContext->channels << '\n'
<< "\tFormat:\t\t\t" << av_get_sample_fmt_name(codecContext->sample_fmt) << '\n'
<< "\tBytes per sample:\t" << av_get_bytes_per_sample(codecContext->sample_fmt) << '\n'
<< "\tPlanar storage format?:\t" << av_sample_fmt_is_planar(codecContext->sample_fmt) << '\n';
std::cout << "frame->linesize[0] tells you the size (in bytes) of each plane\n";
if (codecContext->channels > AV_NUM_DATA_POINTERS && av_sample_fmt_is_planar(codecContext->sample_fmt))
{
std::cout << tooManyChannels;
}
else
{
stc::cout << nonPlanar;
}
std::cout << information1 << information2;
}
int main()
{
// You can change the filename for any other filename/supported format
std::string filename = "../my file.ogg";
// Initialize FFmpeg
av_register_all();
AVFrame* frame = avcodec_alloc_frame();
if (!frame)
{
std::cout << "Error allocating the frame. Let's try again shall we?\n";
return 666; // fail at start: 66 = number of the beast
}
// you can change the file name to whatever yo need:)
AVFormatContext* formatContext = NULL;
if (avformat_open_input(&formatContext, filename, NULL, NULL) != 0)
{
av_free(frame);
std::cout << "Error opening file " << filename<< "\n";
return 800; // cant open file. 800 = Boo!
}
if (avformat_find_stream_info(formatContext, NULL) < 0)
{
av_free(frame);
avformat_close_input(&formatContext);
std::cout << "Error finding the stream information.\nCheck your paths/connections and the details you supplied!\n";
return 57005; // stream info error. 0xDEAD in hex is 57005 in decimal
}
// Find the audio stream
AVCodec* cdc = nullptr;
int streamIndex = av_find_best_stream(formatContext, AVMEDIA_TYPE_AUDIO, -1, -1, &cdc, 0);
if (streamIndex < 0)
{
av_free(frame);
avformat_close_input(&formatContext);
std::cout << "Could not find any audio stream in the file. Come on! I need data!\n";
return 165; // no(0) (a)udio s(5)tream: 0A5 in hex = 165 in decimal
}
AVStream* audioStream = formatContext->streams[streamIndex];
AVCodecContext* codecContext = audioStream->codec;
codecContext->codec = cdc;
if (avcodec_open2(codecContext, codecContext->codec, NULL) != 0)
{
av_free(frame);
avformat_close_input(&formatContext);
std::cout << "Couldn't open the context with the decoder. I can decode but I need to have something to decode.\nAs I couldn't find anything I have surmised the decoded output is 0!\n (Well can't have you thinking I am doing nothing can we?\n";
return 1057; // cant find/open context 1057 = lost
}
std::cout << "This stream has " << codecContext->channels << " channels with a sample rate of " << codecContext->sample_rate << "Hz\n";
std::cout << "The data presented in format: " << av_get_sample_fmt_name(codecContext->sample_fmt) << std::endl;
AVPacket readingPacket;
av_init_packet(&readingPacket);
// Read the packets in a loop
while (av_read_frame(formatContext, &readingPacket) == 0)
{
if (readingPacket.stream_index == audioStream->index)
{
AVPacket decodingPacket = readingPacket;
// Audio packets can have multiple audio frames in a single packet
while (decodingPacket.size > 0)
{
// Try to decode the packet into a frame(s)
// Some frames rely on multiple packets, so we have to make sure the frame is finished
// before utilising it
int gotFrame = 0;
int result = avcodec_decode_audio4(codecContext, frame, &gotFrame, &decodingPacket);
if (result >= 0 && gotFrame)
{
decodingPacket.size -= result;
decodingPacket.data += result;
// et voila! a decoded audio frame!
printAudioFrameInfo(codecContext, frame);
}
else
{
decodingPacket.size = 0;
decodingPacket.data = nullptr;
}
}
}
// You MUST call av_free_packet() after each call to av_read_frame()
// or you will leak so much memory on a large file you will need a memory-plumber!
av_free_packet(&readingPacket);
}
// Some codecs will cause frames to be buffered in the decoding process.
// If the CODEC_CAP_DELAY flag is set, there can be buffered frames that need to be flushed
// therefore flush them now....
if (codecContext->codec->capabilities & CODEC_CAP_DELAY)
{
av_init_packet(&readingPacket);
// Decode all the remaining frames in the buffer
int gotFrame = 0;
while (avcodec_decode_audio4(codecContext, frame, &gotFrame, &readingPacket) >= 0 && gotFrame)
{
// Again: a fully decoded audio frame!
printAudioFrameInfo(codecContext, frame);
}
}
// Clean up! (unless you have a quantum memory machine with infinite RAM....)
av_free(frame);
avcodec_close(codecContext);
avformat_close_input(&formatContext);
return 0; // success!!!!!!!!
}
Hope this helps. Let me know if you need more info, and I will try and help out:)
There is also some very good tutorial information available at dranger.com which you may find useful.
Preallocate the format context and set its pb field as suggested in the note of avformat_open_input() documentation.
.
I have a Tcl procedure which magnifies an image.
proc ShowWindow {wtitle zfactor imgdata} {
puts stdout "Now in the Tcl procedure ShowWindow!";
image create photo idata -data $imgdata; # create a photo image from the input data
image create photo the_window; # final window
the_window copy idata -zoom $zfactor $zfactor; # copy the original and magnify
wm title . $wtitle; # window title
wm resizable . false false; # no resizing in both x and y directions
catch {destroy .dwindow}; # since this procedure will be called multiple times
# we need to suppress the 'window name "dwindow" already exists in parent' message
label .dwindow -image the_window; # create a label to display the image
pack .dwindow; # display the image
}
I would like to call this Tcl procedure from C++.
I think that "imgdata" is a ByteArray. Is this correct?
The pertinent code fragment is shown below:
// ...Tcl/Tk initialization omitted...
unsigned char *img; // PPM image data
int num_bytes; // PPM image file size
// ...routines to initialize img and num_bytes omitted...
Tcl_Obj *tcl_raw;
// transfer the PPM image data into Tcl_Obj
if (!(tcl_raw = Tcl_NewByteArrayObj(img, num_bytes))) {
cerr << "Tcl_NewByteArrayObj() failed!" << endl;
cerr << "Exiting..." << endl;
return 1;
} else {
cerr << "Tcl_NewByteArrayObj() succeeded!" << endl;
}
Tcl_IncrRefCount(tcl_raw); // is this really necessary?
// set the Tcl variable "imgdata" to the Tcl_Obj
if (Tcl_SetVar2Ex(tcl_interpreter, "imgdata", "", tcl_raw, TCL_LEAVE_ERR_MSG) == NULL) {
cerr << "Tcl_SetVar2Ex() failed!" << endl;
cerr << "Exiting..." << endl;
return 1;
} else {
cerr << "Tcl_SetVar2Ex() succeeded!" << endl;
}
// execute the Tcl procedure
if (Tcl_Eval(tcl_interpreter, "ShowWindow TheImage 8 $imgdata") != TCL_OK) {
cerr << "Tcl_Eval() failed!" << endl;
cerr << "Tcl_GetStringResult() = " << Tcl_GetStringResult(tcl_interpreter) << endl;
cerr << "Exiting..." << endl;
return 1;
} else {
cerr << "Tcl_Eval() succeeded!" << endl;
}
The program fails at Tcl_Eval(). The program output is:
...
Tcl_NewByteArrayObj() succeeded!
Tcl_SetVar2Ex() succeeded!
Tcl_Eval() failed!
Tcl_GetStringResult() = can't read "imgdata": variable is array
Exiting...
What is the recommended way of doing this?
You can treat the Tcl ByteArrayObj type as a buffer by using Tcl_ByteArraySetLength or Tcl_GetByteArrayFromObj which both give you access to the data part of the Tcl object.
Tcl_Obj *dataObj = Tcl_NewObj();
char *dataPtr = Tcl_SetByteArrayLength(dataObj, 1024);
Now you can use dataPtr to set the bytes in the object. The Tcl_SetByteArrayLength function will make this object become a ByteArray type.
However, you might also want to look at imgscale which I use for stretching Tk images and this uses various interpolation modes eg:
image create photo thumbnail
::imgscale::bilinear $myphoto 64 64 thumbnail 1.0
to downscale some photo into a thumbnail image.
Next iteration of my question:
Thank you for your inputs, it has helped me to understand a little bit more about the Frame and inputSamples utility.
I’ve done modifications to my source code with the new knowledge you’ve given me. But I still have problems, so I might not have understood fully what you meant.
Here is my OpenFile function, sorry for the name but I’ll refactor later; when it’ll work =)
//-----------------------------------------------------------------------------
/*
This Function Open a File containing the Audio, Binary, Data.
*///___________________________________________________________________________
const short* OpenFile(const char* fileName, long& fileSize, WavFormat* wav)
{
// ouvre le fichier
ifstream file;
file.open((char*)fileName, ios::binary|ios::in);
if (file.good())
{
// Read the WAV's Header
wav = CheckWavHeader(file, wav);
cout << "chunkID: " << wav->chunkID <<'\n';
cout << "chunkSize: " << wav->chunkSize <<'\n';
cout << "format: " << wav->format <<'\n';
cout << "subChunk1ID: " << wav->subChunk1ID <<'\n';
cout << "subChunk1Size: " << wav->subChunk1Size <<'\n';
cout << "audioFormat: " << wav->audioFormat <<'\n'; // audioFormat == 1, alors PCM 16bits
cout << "numChannels: " << wav->numChannels <<'\n';
cout << "sampleRate: " << wav->sampleRate <<'\n';
cout << "byteRate: " << wav->byteRate <<'\n';
cout << "blockAlign: " << wav->blockAlign <<'\n';
cout << "bitsPerSample: " << wav->bitsPerSample <<'\n';
cout << "subChunk2ID: " << wav->subChunk2ID <<'\n';
cout << "subChunk2Size: " << wav->subChunk2Size <<'\n';
// Get the file’s size
file.seekg(0L, ios::end);
fileSize = ((long)file.tellg() - DATA_POS);
file.seekg(DATA_POS, ios::beg); // back to the data.
// Read the Data into the Buffer
uint nbSamples = fileSize / sizeof(short);
short* inputArray = new short[nbSamples];
file.read((char*)inputArray, fileSize);
// Close the file and return the Data
file.close();
return (const short*)inputArray;
}
else
{
exit(-1);
}
}
I’m opening the file, checking its size, create a short buffer and read the wav’s data into the short buffer and finally I return it.
In the main, for now I commented the G711 decoder.
When I run the application, the faacEncOpen gives me 2048 for inputSamples (it’s logic since I have 2 channels in the Wav’s file for a FRAME_LEN of 1024).
So if I understood correctly, 1 Frame == 2048 samples for my application. So for each Frame I call the faacEncEncode, I give the tmpInputBuffer that is a buffer of the same size as inputSamples at the inputBuffer[i * inputSamples] index.
//-----------------------------------------------------------------------------
/*
The Main entry Point of the Application
*///_____________________________________________________________________________
int main()
{
// Get the File's Data
WavFormat* wav = new WavFormat;
long fileSize;
const short* fileInput = OpenFile("audioTest.wav", fileSize, wav);
// G711 mu-Law Decoder
//MuLawDecoder* decoder = new MuLawDecoder();
//short* inputBuffer = decoder->MuLawDecode_shortArray((byte*)fileInput, (int)nbChunk);
short* inputBuffer = (short*)fileInput;
// Info for FAAC
ulong sampleRate = wav->sampleRate;
uint numChannels = wav->numChannels;
ulong inputSamples;
ulong maxOutputBytes;
// Ouvre l'Encodeur et assigne la Configuration.
faacEncHandle hEncoder = faacEncOpen(sampleRate, numChannels, &inputSamples, &maxOutputBytes);
faacEncConfigurationPtr faacConfig = faacEncGetCurrentConfiguration(hEncoder);
faacConfig->inputFormat = FAAC_INPUT_16BIT;
faacConfig->bitRate = 64000;
int result = faacEncSetConfiguration(hEncoder, faacConfig);
/*Input Buffer and Output Buffer*/
byte* outputBuffer = new byte[maxOutputBytes];
int nbBytesWritten = 0;
Sink* sink = new Sink();
uint nbFrame = fileSize / inputSamples;
int32_t* tmpInputBuffer = new int32_t[inputSamples];
for (uint i = 0; i < nbFrame; i++)
{
strncpy((char*)tmpInputBuffer, (const char*)&inputBuffer[i * inputSamples], inputSamples);
nbBytesWritten = faacEncEncode(hEncoder, tmpInputBuffer, inputSamples, outputBuffer, maxOutputBytes);
cout << 100.0 * (float)i / nbFrame << "%\t nbBytesWritten = " << nbBytesWritten << "\n";
if (nbBytesWritten > 0)
{
sink->AddAACStream(outputBuffer, nbBytesWritten);
}
}
sink->WriteToFile("output.aac");
// Close AAC Encoder
faacEncClose(hEncoder);
// Supprimer tous les pointeurs
delete sink;
//delete decoder;
delete[] fileInput;
//delete[] inputBuffer;
delete[] outputBuffer;
delete[] tmpInputBuffer;
system("pause");
return 0;
}
When the output Data is Dumped into an .acc file (as RAW AAC), I use the application mp4muxer.exe to create an .mp4 file to listen to the final converted sound. But the sound is not good at all...
I'm wondering if there is something I'm not seeing or do not unserstand that I should.
Thank you in advance for your useful inputs.
Each call to faacEncEncode encodes inputSamples samples, not just one. Your main loop should read that many samples from the WAV file into the input buffer, then call faacEncEncode once for that buffer, and finally write the output buffer to the AAC file.
It's possible that I've misunderstood what you're doing (if so, it would be useful to know: (1) What's the OpenFile function you're calling, and does it (despite its name) actually read the file as well as opening it? (2) How is inputBuffer set up?) but:
faacEncEncode expects to be given a whole frame's worth of samples. A frame is the number of samples you got passed back in inputSamples when you called faacEncOpen. (You can give it less than a whole frame if you've reached the end of the input, of course.)
So you're getting 460 and 539 bytes for each of two frames -- not for 16 bits in each case. And it looks as if your input-data pointers are actually offset by only one sample each time, so you're handing it badly overlapping frames. (And the wrong number of them; nbChunk is not the number of frames you have.)