C++ - applying filter in ffmpeg - c++

I'm trying to deinterlace a frame using ffmpeg (latest release). Related with this question, I can get the filter I want using this sentence:
AVFilter *filter = avfilter_get_by_name("yadif");
After that, I open the filter context as:
AVFilterContext *filter_ctx;
avfilter_open(&filter_ctx, filter, NULL);
My first question is about this function. Visual Studio warns me about avfilter_open is deprecated. Which is the alternative?
After that, I do:
avfilter_init_str(filter_ctx, "yadif=1:-1");
And always fails. I've tried "1:-1" instead "yadif=1:-1", but always fails too, what parameter I should use?
EDIT: A value of "1" or "2", for example, it works. Debuging it, I found that with one of this values, the function uses mode=1 or mode=2. The explanation of those values is in this link.
Then, I have a AVFrame *frame that is the frame I want to deinterlace. When the last sentence work, I'll have the filter and his context init. How do I apply this filter to my frame?
Thanks for your help.

I undertsnad your question is over a year now but recently I had to work with interlaced DVB-TS streams so I might be able to help anybody else coming across this subject.
These snippets are from a finished player I've written
Initialise the filter graph:
void VideoManager::init_filter_graph(AVFrame *frame) {
if (filter_initialised) return;
int result;
AVFilter *buffer_src = avfilter_get_by_name("buffer");
AVFilter *buffer_sink = avfilter_get_by_name("buffersink");
AVFilterInOut *inputs = avfilter_inout_alloc();
AVFilterInOut *outputs = avfilter_inout_alloc();
AVCodecContext *ctx = ffmpeg.vid_stream.context;
char args[512];
int frame_fix = 0; // fix bad width on some streams
if (frame->width < 704) frame_fix = 2;
else if (frame->width > 704) frame_fix = -16;
snprintf(args, sizeof(args),
"video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
frame->width + frame_fix,
frame->height,
frame->format,// ctx->pix_fmt,
ctx->time_base.num,
ctx->time_base.den,
ctx->sample_aspect_ratio.num,
ctx->sample_aspect_ratio.den);
const char *description = "yadif=1:-1:0";
LOGD("Filter: %s - Settings: %s", description, args);
filter_graph = avfilter_graph_alloc();
result = avfilter_graph_create_filter(&filter_src_ctx, buffer_src, "in", args, NULL, filter_graph);
if (result < 0) {
LOGI("Filter graph - Unable to create buffer source");
return;
}
AVBufferSinkParams *params = av_buffersink_params_alloc();
enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE };
params->pixel_fmts = pix_fmts;
result = avfilter_graph_create_filter(&filter_sink_ctx, buffer_sink, "out", NULL, params, filter_graph);
av_free(params);
if (result < 0) {
LOGI("Filter graph - Unable to create buffer sink");
return;
}
inputs->name = av_strdup("out");
inputs->filter_ctx = filter_sink_ctx;
inputs->pad_idx = 0;
inputs->next = NULL;
outputs->name = av_strdup("in");
outputs->filter_ctx = filter_src_ctx;
outputs->pad_idx = 0;
outputs->next = NULL;
result = avfilter_graph_parse_ptr(filter_graph, description, &inputs, &outputs, NULL);
if (result < 0) LOGI("avfilter_graph_parse_ptr ERROR");
result = avfilter_graph_config(filter_graph, NULL);
if (result < 0) LOGI("avfilter_graph_config ERROR");
filter_initialised = true;
}
When processing the video packets from the stream, check if it is an interlaced frame and send the frame off to the filter. The filter will then return the de-interlaced frames back to you.
void FFMPEG::process_video_packet(AVPacket *pkt) {
int got;
AVFrame *frame = vid_stream.frame;
avcodec_decode_video2(vid_stream.context, frame, &got, pkt);
if (got) {
if (!frame->interlaced_frame) { // not interlaced
Video.add_av_frame(frame, 0);
} else {
if (!Video.filter_initialised) {
Video.init_filter_graph(frame);
}
av_buffersrc_add_frame_flags(Video.filter_src_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF);
int c = 0;
while (true) {
AVFrame *filter_frame = ffmpeg.vid_stream.filter_frame;
int result = av_buffersink_get_frame(Video.filter_sink_ctx, filter_frame);
if (result == AVERROR(EAGAIN) || result == AVERROR(AVERROR_EOF)) break;
if (result < 0) return;
Video.add_av_frame(filter_frame, c++);
av_frame_unref(filter_frame);
}
}
}
}
Hope this helps anyone because finding information about ffmpeg is tough going.

Related

FFMPEG encoder:Application provided invalid, non monotonically increasing dts to muxer in stream

I am encoding video with h264 codec and mp4 container. The source is a sequence of images I load from memory. The encoding seems to be working good. I am able to verify the exact amount of frames encoded, but what bugs me is the following warning/error I am getting at the end of the session from the muxer in terminal:
Application provided invalid, non monotonically increasing dts to
muxer in stream 0:
I tried different techniques to calculate pts/dts and it is the same. I did notice though, that avcodec_receive_packet() sometimes returns packets out of order,so may be that's the reason for the warning? Below is my function where I encode the image data.
bool LibAVEncoder::EncodeFrame(uint8_t* data)
{
int err;
if (!mVideoFrame)//init frame once
{
mVideoFrame = av_frame_alloc();
mVideoFrame->format = mVideoStream->codecpar->format;
mVideoFrame->width = mCodecContext->width;
mVideoFrame->height = mCodecContext->height;
if ((err = av_frame_get_buffer(mVideoFrame, 0)) < 0)
{
printf("Failed to allocate picture: %i\n", err);
return false;
}
}
err = av_frame_make_writable(mVideoFrame);
if (!mSwsCtx)
{
mSwsCtx = sws_getContext(mCodecContext->width, mCodecContext->height, AV_PIX_FMT_RGBA, mCodecContext->width,
mCodecContext->height, AV_PIX_FMT_YUV420P, SWS_BICUBIC, 0, 0, 0);
}
const int inLinesize[1] = { 4 * mCodecContext->width };
// RGBA to YUV
sws_scale(mSwsCtx, (const uint8_t* const*)&data, inLinesize, 0, mCodecContext->height, mVideoFrame->data, mVideoFrame->linesize);
mVideoFrame->pts = mFrameCount++;
//Encoding
err = avcodec_send_frame(mCodecContext, mVideoFrame);
if (err < 0)
{
printf("Failed to send frame: %i\n", err);
return false;
}
while (err >= 0)
{
err = avcodec_receive_packet(mCodecContext, mPacket);
if (err == AVERROR(EAGAIN))
{
return true;
}
else if (err == AVERROR_EOF)
{
return true;
}
else if (err < 0)
{
fprintf(stderr, "Error during encoding,shutting down\n");
exit(1);
}
// 1. way of calc pts/dts
mPacket->pts = av_rescale_q_rnd(mPacket->pts, mCodecContext->time_base, mVideoStream->time_base, AVRounding(AV_ROUND_NEAR_INF | AV_ROUND_PASS_MINMAX));
mPacket->dts = av_rescale_q_rnd(mPacket->dts, mCodecContext->time_base, mVideoStream->time_base, AVRounding(AV_ROUND_NEAR_INF | AV_ROUND_PASS_MINMAX));
mPacket->duration = av_rescale_q(mPacket->duration, mCodecContext->time_base, mVideoStream->time_base);
/*
//2. another way of calc pts/dts
const int64_t duration = av_rescale_q(1, mCodecContext->time_base, mVideoStream->time_base);
mPacket->duration = duration;
mPacket->pts = totalDuration;
mPacket->dts = totalDuration;
totalDuration += duration;
*/
mPacket->stream_index = mVideoStream->index;
av_interleaved_write_frame(mOFormatContext, mPacket);
av_packet_unref(mPacket);
}
return true;
}
I would appreciate any pointer to what I am doing wrong here.

Resampling audio using libswresample, leaves small amount of noise after resampling

I'm trying to resample audio from 44Khz to 48Khz and I'm getting s small light noise after resampling. As if someone is gently ticking the mic. This happens both ways. From 48Khz to 44Khz and vice versa.
I've read that this can happen because swrContext still has some data left and that I shoudl flush the context before resampling next frame. And although this helps a little (less noticeable noise), it's still present.
I've tried using FFmpeg resample filter instead, but the output is just loud incoherent noise. I'm pretty sure that libswresample should not output any noise on resampling which means that I just don't know how to use it well and I'm missing some options.
This is the code for resampler.
int ResampleFrame(VideoState * videoState, AVFrame *decoded_audio_frame, enum AVSampleFormat out_sample_fmt, uint8_t * out_buf)
{
int in_sample_rate = videoState->audio->ptrAudioCodecCtx_->sample_rate;
int out_sample_rate = SAMPLE_RATE;
// get an instance of the AudioResamplingState struct, create if NULL
AudioResamplingState* arState = getAudioResampling(videoState->audio->ptrAudioCodecCtx_->channel_layout);
if (!arState->swr_ctx)
{
printf("swr_alloc error.\n");
return -1;
}
// get input audio channels
arState->in_channel_layout = (videoState->audio->ptrAudioCodecCtx_->channels ==
av_get_channel_layout_nb_channels(videoState->audio->ptrAudioCodecCtx_->channel_layout)) ?
videoState->audio->ptrAudioCodecCtx_->channel_layout :
av_get_default_channel_layout(videoState->audio->ptrAudioCodecCtx_->channels);
// check input audio channels correctly retrieved
if (arState->in_channel_layout <= 0)
{
printf("in_channel_layout error.\n");
return -1;
}
arState->out_channel_layout = AV_CH_LAYOUT_STEREO;
// retrieve number of audio samples (per channel)
arState->in_nb_samples = decoded_audio_frame->nb_samples;
if (arState->in_nb_samples <= 0)
{
printf("in_nb_samples error.\n");
return -1;
}
// Set SwrContext parameters for resampling
av_opt_set_int(arState->swr_ctx, "in_channel_layout", arState->in_channel_layout, 0);
av_opt_set_int(arState->swr_ctx, "in_sample_rate", in_sample_rate, 0);
av_opt_set_sample_fmt(arState->swr_ctx, "in_sample_fmt", videoState->audio->ptrAudioCodecCtx_->sample_fmt, 0);
// Set SwrContext parameters for resampling
av_opt_set_int(arState->swr_ctx, "out_channel_layout", arState->out_channel_layout, 0);
av_opt_set_int(arState->swr_ctx, "out_sample_rate", out_sample_rate, 0);
av_opt_set_sample_fmt(arState->swr_ctx, "out_sample_fmt", out_sample_fmt, 0);
// initialize SWR context after user parameters have been set
int ret = swr_init(arState->swr_ctx);
if (ret < 0)
{
printf("Failed to initialize the resampling context.\n");
return -1;
}
// retrieve output samples number taking into account the progressive delay
int64_t delay = swr_get_delay(arState->swr_ctx, videoState->audio->ptrAudioCodecCtx_->sample_rate) + arState->in_nb_samples;
arState->out_nb_samples = av_rescale_rnd(delay, out_sample_rate, in_sample_rate, AV_ROUND_UP );
// check output samples number was correctly rescaled
if (arState->out_nb_samples <= 0)
{
printf("av_rescale_rnd error\n");
return -1;
}
// get number of output audio channels
arState->out_nb_channels = av_get_channel_layout_nb_channels(arState->out_channel_layout);
// allocate data pointers array for arState->resampled_data and fill data
// pointers and linesize accordingly
// check memory allocation for the resampled data was successful
ret = av_samples_alloc_array_and_samples(&arState->resampled_data, &arState->out_linesize, arState->out_nb_channels, arState->out_nb_samples, out_sample_fmt, 0);
if (ret < 0)
{
printf("av_samples_alloc_array_and_samples() error: Could not allocate destination samples.\n");
return -1;
}
if (arState->swr_ctx)
{
// do the actual audio data resampling
// check audio conversion was successful
int ret_num_samples = swr_convert(arState->swr_ctx,arState->resampled_data,arState->out_nb_samples,(const uint8_t**)decoded_audio_frame->data, decoded_audio_frame->nb_samples);
//int ret_num_samples = swr_convert_frame(arState->swr_ctx,arState->resampled_data,arState->out_nb_samples,(const uint8_t**)decoded_audio_frame->data, decoded_audio_frame->nb_samples);
if (ret_num_samples < 0)
{
printf("swr_convert_error.\n");
return -1;
}
// get the required buffer size for the given audio parameters
// check audio buffer size
arState->resampled_data_size = av_samples_get_buffer_size(&arState->out_linesize, arState->out_nb_channels,ret_num_samples,out_sample_fmt,1);
if (arState->resampled_data_size < 0)
{
printf("av_samples_get_buffer_size error.\n");
return -1;
}
} else {
printf("swr_ctx null error.\n");
return -1;
}
// copy the resampled data to the output buffer
memcpy(out_buf, arState->resampled_data[0], arState->resampled_data_size);
// flush the swr context
int delayed = swr_convert(arState->swr_ctx,arState->resampled_data,arState->out_nb_samples,NULL,0);
if (arState->resampled_data)
{
av_freep(&arState->resampled_data[0]);
}
av_freep(&arState->resampled_data);
arState->resampled_data = NULL;
int ret_data_size = arState->resampled_data_size;
return ret_data_size;
}
I also tries using the filter as shown here but my output is just noise.
This is my filter code
int ResampleFrame(AVFrame *frame, uint8_t *out_buf)
{
/* Push the decoded frame into the filtergraph */
qint32 ret;
ret = av_buffersrc_add_frame_flags(buffersrc_ctx1, frame, AV_BUFFERSRC_FLAG_KEEP_REF);
if (ret < 0)
{
printf("ResampleFrame: Error adding frame to buffer\n");
// Delete input frame and return null
av_frame_unref(frame);
return 0;
}
//printf("resampling\n");
AVFrame *resampled_frame = av_frame_alloc();
/* Pull filtered frames from the filtergraph */
ret = av_buffersink_get_frame(buffersink_ctx1, resampled_frame);
/* Set the timestamp on the resampled frame */
resampled_frame->best_effort_timestamp = resampled_frame->pts;
if (ret < 0)
{
av_frame_unref(frame);
av_frame_unref(resampled_frame);
return 0;
}
int buffer_size = av_samples_get_buffer_size(NULL, 2,resampled_frame->nb_samples,AV_SAMPLE_FMT_S16,1);
memcpy(out_buf,resampled_frame->data,buffer_size);
//av_frame_unref(frame);
av_frame_unref(resampled_frame);
return buffer_size;
}
QString filter_description1 = "aresample=48000,aformat=sample_fmts=s16:channel_layouts=stereo,asetnsamples=n=1024:p=0";
int InitAudioFilter(AVStream *inputStream)
{
char args[512];
int ret;
const AVFilter *buffersrc = avfilter_get_by_name("abuffer");
const AVFilter *buffersink = avfilter_get_by_name("abuffersink");
AVFilterInOut *outputs = avfilter_inout_alloc();
AVFilterInOut *inputs = avfilter_inout_alloc();
filter_graph = avfilter_graph_alloc();
const enum AVSampleFormat out_sample_fmts[] = {AV_SAMPLE_FMT_S16, AV_SAMPLE_FMT_NONE};
const int64_t out_channel_layouts[] = {AV_CH_LAYOUT_STEREO, -1};
const int out_sample_rates[] = {48000, -1};
snprintf(args, sizeof(args), "time_base=%d/%d:sample_rate=%d:sample_fmt=%s:channel_layout=0x%" PRIx64,
inputStream->codec->time_base.num, inputStream->codec->time_base.den,
inputStream->codec->sample_rate,
av_get_sample_fmt_name(inputStream->codec->sample_fmt),
inputStream->codec->channel_layout);
ret = avfilter_graph_create_filter(&buffersrc_ctx1, buffersrc, "in", args, NULL, filter_graph);
if (ret < 0)
{
printf("InitAudioFilter: Unable to create buffersrc\n");
return -1;
}
ret = avfilter_graph_create_filter(&buffersink_ctx1, buffersink, "out", NULL, NULL, filter_graph);
if (ret < 0)
{
printf("InitAudioFilter: Unable to create buffersink\n");
return ret;
}
// set opt SAMPLE FORMATS
ret = av_opt_set_int_list(buffersink_ctx1, "sample_fmts", out_sample_fmts, -1, AV_OPT_SEARCH_CHILDREN);
if (ret < 0)
{
printf("InitAudioFilter: Cannot set output sample format\n");
return ret;
}
// set opt CHANNEL LAYOUTS
ret = av_opt_set_int_list(buffersink_ctx1, "channel_layouts", out_channel_layouts, -1, AV_OPT_SEARCH_CHILDREN);
if (ret < 0) {
printf("InitAudioFilter: Cannot set output channel layout\n");
return ret;
}
// set opt OUT SAMPLE RATES
ret = av_opt_set_int_list(buffersink_ctx1, "sample_rates", out_sample_rates, -1, AV_OPT_SEARCH_CHILDREN);
if (ret < 0)
{
printf("InitAudioFilter: Cannot set output sample rate\n");
return ret;
}
/* Endpoints for the filter graph. */
outputs -> name = av_strdup("in");
outputs -> filter_ctx = buffersrc_ctx1;
outputs -> pad_idx = 0;
outputs -> next = NULL;
/* Endpoints for the filter graph. */
inputs -> name = av_strdup("out");
inputs -> filter_ctx = buffersink_ctx1;
inputs -> pad_idx = 0;
inputs -> next = NULL;
if ((ret = avfilter_graph_parse_ptr(filter_graph, filter_description1.toStdString().c_str(), &inputs, &outputs, NULL)) < 0)
{
printf("InitAudioFilter: Could not add the filter to graph\n");
}
if ((ret = avfilter_graph_config(filter_graph, NULL)) < 0)
{
printf("InitAudioFilter: Could not configure the graph\n");
}
/* Print summary of the sink buffer
* Note: args buffer is reused to store channel layout string */
AVFilterLink *outlink = buffersink_ctx1->inputs[0];
av_get_channel_layout_string(args, sizeof(args), -1, outlink->channel_layout);
QString str = args;
printf("Output: srate:%dHz fmt:%s chlayout: %s\n", (int) outlink->sample_rate,
av_get_sample_fmt_name((AVSampleFormat) outlink->format),
str.toStdString().c_str());
filterGraphInitialized_ = true;
}
And since I don't have much experience with filters or audio for that matter, I'm also probably missing something here. But Can't figure out what.
Thanks

av_read_frame of ffmpeg in windows always returning random packet.stream_index

I'm using ffmpeg to decode an rtsp stream, the av_read_frame is always returning random value on packet.stream_index.
The stream have only 2 index, 1 video and 1 audio respectively on streams[0] and streams[1].
But when i call av_read_frame, i've random value on packet.stream_index and no data in the packet.
If someone can help me, i'm beginner in FFMPEG.
I add my simplified code to make you an idea of what i do :
AVFormatContext* input_ctx = NULL;
AVStream* video_st = NULL;
AVCodecContext* decoder_ctx = NULL;
AVCodec* decoder;
AVPacket pkt = { 0 };
AVFrame* frame = NULL;
DWORD dwHeight = 576;
DWORD dwWidth = 704;
int ret, i;
input_ctx = avformat_alloc_context();
/* open the input file */
ret = avformat_open_input(&input_ctx, "rtsp://**************************/Streaming/Channels/102?profile=Profile_1", NULL, NULL);
avformat_find_stream_info(input_ctx, NULL);
av_dump_format(input_ctx, 0, "rtsp://**************************/Streaming/Channels/102?profile=Profile_1", 0);
av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, 0, -1, &decoder, 0);
/* find the first H.264 video stream */
for (i = 0; i < input_ctx->nb_streams-1; i++) {
AVStream *st = input_ctx->streams[i];
video_st = st;
}
decoder_ctx = input_ctx->streams[0]->codec;
decoder = avcodec_find_decoder(decoder_ctx->codec_id);
decoder_ctx = avcodec_alloc_context3(decoder);
ret = avcodec_open2(decoder_ctx, NULL, NULL);
frame = av_frame_alloc();
av_read_play(input_ctx);
/* actual decoding */
while (ret >= 0) {
ret = av_read_frame(input_ctx, &pkt);
if (pkt.stream_index == video_st->index) {
if ((avcodec_send_packet != NULL)&&(avcodec_receive_frame!=NULL)) {
ret = avcodec_send_packet(decoder_ctx, pkt);
Thx in advance.

Getting left and right frames with BlackMagic SDK and convert them into opencv mat

I am currently trying to use BlackMagic SDK with a stero camera using a DeckLink 4K pro card on Linux.
I already used it with a monocular one and successfully get the image and convert it in an opencv::mat but no matter what I try with this one I have nothing but empty matrix corresponding to left and right frames. I suspect there is an issue with the mode and/or flags I use but I cannot figure it out... When I try to see the frames with mediaExpress, I have satisfying results so I guess it is not a hardware problem.
I tried to do something that look like to the different sample given by the SDK and I convert the final frames in opencv mat to see the result. When !isStereo, eveything goes well and I get a fusion of both right and left frames but when it's true, I have empty matrix for both left and right frames.
Here is the corresponding part of my code (the things commented are things I tried: definitively, I am almost sure it's something linked with the flags I am using in g_deckLinkInput->EnableVideoInput(...) ):
...
void VideoInputFromBlackMagic::runInput(){
this->running=true;
int m_deckLinkIndex;
int idx;
//Check result
HRESULT result;
IDeckLink* deckLink = NULL;
IDeckLinkInput* g_deckLinkInput = NULL;
IDeckLinkAttributes* deckLinkAttributes = NULL;
IDeckLinkIterator* deckLinkIterator = CreateDeckLinkIteratorInstance();
IDeckLinkInputCallback* callBack;
IDeckLinkDisplayModeIterator* displayModeIterator = NULL;
IDeckLinkDisplayMode* displayMode = NULL;
char* displayModeName = NULL;
BMDDisplayModeSupport displayModeSupported;
bool formatDetectionSupported;
if (!deckLinkIterator)
{
fprintf(stderr, "This application requires the DeckLink drivers installed.\n");
return;
}
//Get the DeckLink Inputs
result = deckLinkIterator->Next(&deckLink);
result = deckLink->QueryInterface(IID_IDeckLinkInput, (void**)&g_deckLinkInput);
if(result != S_OK){
fprintf(stdout, "Cannot get the Input : DeckLink Error\n");
return;
}
//Get the DeckLink attributes (that may not correctly work: format detection does not properly work)
result = deckLink->QueryInterface(IID_IDeckLinkAttributes, (void**)&deckLinkAttributes);
if (!(result == S_OK)){
fprintf(stdout, "Cannot get the DeckLink attributes : DeckLink Error\n");
return;
}
//Format detection
result = deckLinkAttributes->GetFlag(BMDDeckLinkSupportsInputFormatDetection, &formatDetectionSupported);
if (result != S_OK || !formatDetectionSupported){
fprintf(stdout,"Cannot get the format input: DeckLink Error\n");
return;
}
//Index for the different inputs
idx = 0;
//Get all the displayModes
result = g_deckLinkInput->GetDisplayModeIterator(&displayModeIterator);
if (result != S_OK){
fprintf(stdout,"Cannot set an iterator on the different display modes: DeckLink Error\n");
}
//Set idx
while ((result = displayModeIterator->Next(&displayMode)) == S_OK)
{
if (idx == 0)
break;
--idx;
displayMode->Release();
}
if (result != S_OK || displayMode == NULL){
fprintf(stdout,"Cannot get the main display mode: DeckLink Error\n");
return;
}
//Get Mode name: useless
result = displayMode->GetName((const char**)&displayModeName);
// Check display mode is supported with given options
if(this->isStereo){
//result = g_deckLinkInput->DoesSupportVideoMode(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeSupports3D | bmdDisplayModeColorspaceRec709, &displayModeSupported, NULL);
//result = g_deckLinkInput->DoesSupportVideoMode(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeColorspaceRec709 & bmdDisplayModeSupports3D, &displayModeSupported, NULL);
result = g_deckLinkInput->DoesSupportVideoMode(bmdModeHD1080p30, bmdFormat8BitYUV, bmdVideoInputDualStream3D, &displayModeSupported, NULL);
//result = g_deckLinkInput->DoesSupportVideoMode(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeColorspaceRec709 | bmdVideoInputDualStream3D, &displayModeSupported, NULL);
} else {
result = g_deckLinkInput->DoesSupportVideoMode(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeColorspaceRec709, &displayModeSupported, NULL);
}
if (result != S_OK){
fprintf(stdout,"Video Mode not supported : aborted\n");
return;
}
if (displayModeSupported == bmdDisplayModeNotSupported)
{
fprintf(stdout, "The display mode %s is not supported with the selected pixel format\n", displayModeName);
return;
}
//Set the callback on this ( will defined callback on VideoInputFrameArrived and others functions when images arrives or when other events happens
g_deckLinkInput->SetCallback(this);
//Enable the video input with the selected format
if(this->isStereo){
//result = g_deckLinkInput->EnableVideoInput(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeSupports3D | bmdDisplayModeColorspaceRec709);
//result = g_deckLinkInput->EnableVideoInput(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeColorspaceRec709 & bmdDisplayModeSupports3D);
result = g_deckLinkInput->EnableVideoInput(bmdModeHD1080p30, bmdFormat8BitYUV, bmdVideoInputDualStream3D);
//result = g_deckLinkInput->EnableVideoInput(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeColorspaceRec709 | bmdVideoInputDualStream3D);
//result = g_deckLinkInput->EnableVideoInput(bmdModeHD1080p30, bmdFormat8BitYUV, bmd3DPreviewFormatRightOnly);
} else {
result = g_deckLinkInput->EnableVideoInput(bmdModeHD1080p30, bmdFormat8BitYUV, bmdDisplayModeColorspaceRec709);
}
if (result != S_OK)
{
fprintf(stderr, "Failed to enable video input. Maybe another application is using the card.\n");
return;
}
//Disable the audio
result = g_deckLinkInput->DisableAudioInput();
//Start the stream
result = g_deckLinkInput->StartStreams();
if (result != S_OK){
fprintf(stdout,"Error while starting the streaming : aborted\n");
}
while(this->running){
}
}
...
HRESULT VideoInputFromBlackMagic::VideoInputFrameArrived(IDeckLinkVideoInputFrame* videoFrame, IDeckLinkAudioInputPacket* audioFrame){
if (!videoFrame){
fprintf(stdout,"Update: No video frame\n");
return S_FALSE;
}
void* data;
void* dataRight;
IDeckLinkVideoFrame3DExtensions* threeDExtensions = NULL;
IDeckLinkVideoFrame* rightEyeFrame = NULL;
if (FAILED(videoFrame->GetBytes(&data))){
fprintf(stdout,"Fail obtaining the data from videoFrame\n");
return S_FALSE;
}
if( this->isStereo ){
if ((videoFrame->QueryInterface(IID_IDeckLinkVideoFrame3DExtensions, (void **) &threeDExtensions) != S_OK) || (threeDExtensions->GetFrameForRightEye(&rightEyeFrame) != S_OK)){
fprintf(stdout,"Fail obtaining right eye frame\n");
return S_FALSE;
}
if (FAILED(rightEyeFrame->GetBytes(&dataRight))){
fprintf(stdout,"Fail obtaining the data from videoFrame\n");
return S_FALSE;
}
}
cv::Mat loadedImage;
cv::Mat mat = cv::Mat(videoFrame->GetHeight(), videoFrame->GetWidth(), CV_8UC2, data, videoFrame->GetRowBytes());
cv::cvtColor(mat, loadedImage, CV_YUV2BGR_UYVY);
//Right eye
cv::Mat loadedImageRight;
cv::Mat matRight;
if(this->isStereo){
matRight = cv::Mat(rightEyeFrame->GetHeight(), rightEyeFrame->GetWidth(), CV_8UC2, dataRight, rightEyeFrame->GetRowBytes());
if (rightEyeFrame){
rightEyeFrame->Release();
}
cv::cvtColor(matRight, loadedImageRight, CV_YUV2BGR_UYVY);
}
if (!loadedImage.data){
fprintf(stdout,"No frame loaded from the video : mainImage will not be updated\n");
} else {
this->currentImage = loadedImage;
//this->currentImage = loadedImageRight;
if(this->isStereo){
//this->currentImageRight = loadedImageRight;
this->currentImageRight = loadedImage;
}
this->initialized = true;
if(this->debugMode){
if(this->isStereo){
cv::imshow("DEBUG : right eye", this->currentImageRight);
cv::imshow("DEBUG : left eye", this->currentImage);
} else {
cv::imshow("DEBUG", this->currentImage);
}
}
if(this->debugMode){
int kp = cv::waitKey(1);
if(kp == 1048603){
//Remove debugMode and remove the window
this->setDebugMode(false);
}
}
}
return S_OK;
}
...
Any idea?

c++ videoplayer ffmpeg: get pixel data?

I want to get the pixel data of a frame. I found this (in original version as old code) and changed some things.
I have this code:
AVFormatContext *pFormatCtx;
pFormatCtx = avformat_alloc_context();
// Open file
if (int err = avformat_open_input(&pFormatCtx, file, NULL, 0) != 0)
{
exit(2);
}
// Get infromation about streams
if (avformat_find_stream_info(pFormatCtx, NULL) < 0)
{
exit(2);
}
// # video stream
int videoStreamIndex = -1;
AVCodecContext *pVideoCodecCtx;
AVCodec *pVideoCodec;
int res = 0;
int width = 0;
int height = 0;
for (unsigned int i = 0; i < pFormatCtx->nb_streams; i++)
{
if (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
{
videoStreamIndex = i;
pVideoCodecCtx = pFormatCtx->streams[i]->codec;
// Find decoder
pVideoCodec = avcodec_find_decoder(pVideoCodecCtx->codec_id);
if (pVideoCodec)
{
// Open decoder
res = !(avcodec_open2(pVideoCodecCtx, pVideoCodec, NULL) < 0);
width = pVideoCodecCtx->coded_width;
height = pVideoCodecCtx->coded_height;
}
break;
}
}
// Frame width
width = pFormatCtx->streams[videoStreamIndex]->codec->width;
// Frame height
height = pFormatCtx->streams[videoStreamIndex]->codec->height;
AVPacket packet;
int got_picture_ptr;
AVPacket *avpkt;
AVFrame * pOutFrame;
pOutFrame = av_frame_alloc();
AVFrame * rgbOutFrame = av_frame_alloc();
if (!pOutFrame) {
fprintf(stderr, "Could not allocate video frame\n");
exit(1);
}
while (av_read_frame(pFormatCtx, &packet) >= 0)
{
if (packet.stream_index == videoStreamIndex)
{
// Decode packeg to frame.
int videoFrameBytes = avcodec_decode_video2(pVideoCodecCtx, pOutFrame,
&got_picture_ptr, &packet);
// Create context
SwsContext* pImgConvertCtx = sws_getContext(pVideoCodecCtx->width,
pVideoCodecCtx->height,
pVideoCodecCtx->pix_fmt,
pVideoCodecCtx->width, pVideoCodecCtx->height,
AV_PIX_FMT_YUV420P,
SWS_BICUBIC, NULL, NULL, NULL);
// Convert frame
sws_scale(pImgConvertCtx, pOutFrame->data, pOutFrame->linesize,
width, height, rgbOutFrame->data, rgbOutFrame->linesize);
}
}
I know, that the code from SwsContext and sws_scale is wrong but I wonder, where can I find the pixel data of my frame... (and in which format it is stored).
Can someone help me here?
Pixel data is stored in data field.
According to the documentation:
uint8_t* AVFrame::data[AV_NUM_DATA_POINTERS]
pointer to the picture/channel planes.
Look here for more information.
Generally speaking, your code is a bit misleading and rather buggy. I can point out some drawbacks:
1) You don't need to create new SwsContext on every incoming video packet. Just create it once before while cycle.
2) Next, you have an rgbOutFrame, but SwsContext is created for scaling into the YUV420 pixel format. It looks strange.
3) Besides, avcodec_decode_video2 is invoked, but you never check neither return value nor got_picture_ptr flag. Such practice is really error-prone.
And so on...
Hope it'll help you to improve your program and get necessary results.