I am trying to use OpenCV to apply treatments on an mj2 file encoded as grayscale uint16 per pixel. Unfortunately, a non disclosure agreement covers this file (which I did not generate myself) and I can not post a sample mj2 file.
The description of my .mj2 file as provided by ffmpeg is:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'data/DEVISSAGE_181.mj2':
Metadata:
major_brand : mjp2
minor_version : 0
compatible_brands: mjp2
creation_time : 2015-10-09 08:07:43
Duration: 00:01:03.45, start: 0.000000, bitrate: 14933 kb/s
Stream #0:0: Video: jpeg2000 (mjp2 / 0x32706A6D), gray16le, 1152x288, lossless, 14933 kb/s, SAR 1:4 DAR 1:1, 5.50 fps, 5.50 tbr, 55 tbn, 55 tbc (default)
Metadata:
creation_time : 2015-10-09 08:07:43
handler_name : Video
encoder : Motion JPEG2000
I take it that gray16le confirms the uint16 encoding somehow.
Here is my C++ code:
#include<iostream>
#include "opencv2/opencv.hpp"
int main(int, char**) {
cv::VideoCapture cap("data/DEVISSAGE_181.mj2"); // open the video file
cap.set(CV_CAP_PROP_FORMAT, CV_16UC1);
cv::Mat frame;
cap.read(frame); // get a new frame from file
std::cout << "frame.rows: " << frame.rows << ", frame.cols: " << frame.cols << ", frame.channels(): " << frame.channels() << std::endl;
return 0;
}
The result of running this code is:
frame.rows: 288, frame.cols: 1152, frame.channels(): 3, frame.depth(): 0
Which indicates a 3 channels, CV_8U pixel encoding. Why does the cap.set instruction appear to be ignored ? What should I do to get the correct encoding ?
Related
i have some ffmpeg code in c++ that generates a RTMP stream from H264 NALU and audio samples encoded in AAC. I'am using NGINX to take the RTMP stream and forwards to clients and it is working fine. My issue is that when i use NGINX to convert the RTMP stream to HLS, there is no HLS chunks and playlist generated. I use ffmpeg to copy the RTMP stream and generate a new stream to NGINX, the HLS conversion works.
Here is what i get when i do the stream copy using FFMPEG :
Input #0, flv, from 'rtmp://127.0.0.1/live/beam_0'
Metadata:
Server : NGINX RTMP (github.com/sergey-dryabzhinsky/nginx-rtmp-module)
displayWidth : 1920
displayHeight : 1080
fps : 30
profile :
level :
Duration: 00:00:00.00, start: 5.019000, bitrate: N/A
Stream #0:0: Audio: aac, 44100 Hz, mono, fltp, 128 kb/s
Stream #0:1: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1920x1080 (1920x1088), 8000 kb/s, 30 fps, 30.30 tbr, 1k tbn, 60 tbc
Output #0, flv, to 'rtmp://localhost/live/copy_stream':
Metadata:
Server : NGINX RTMP (github.com/sergey-dryabzhinsky/nginx-rtmp-module)
displayWidth : 1920
displayHeight : 1080
fps : 30
profile :
level :
encoder : Lavf57.83.100
Stream #0:0: Video: h264 (High), 1 reference frame ([7][0][0][0] / 0x0007), yuv420p(progressive, left), 1920x1080 (0x0), q=2-31, 8000 kb/s, 30 fps, 30.30 tbr, 1k tbn, 1k tbc
Stream #0:1: Audio: aac ([10][0][0][0] / 0x000A), 44100 Hz, mono, fltp, 128 kb/s
There are not much differences between the two streams, so i don't really get what is going wrong and what i should change in my C++ code. One very weird issue i see is that my audio stream is 48kHz when i publish it, but here it is recognized as 44100Hz :
Output #0, flv, to 'rtmp://127.0.0.1/live/beam_0':
Stream #0:0, 0, 1/1000: Video: h264 (libx264), 1 reference frame, yuv420p, 1920x1080, 0/1, q=-1--1, 8000 kb/s, 30 fps, 1k tbn, 1k tbc
Stream #0:1, 0, 1/1000: Audio: aac, 48000 Hz, 1 channels, fltp, 128 kb/s
UPDATE 1 :
The output context is created using the following code :
pOutputFormatContext->oformat = av_guess_format("flv", url.toStdString().c_str(), nullptr);
memcpy(pOutputFormatContext->filename, url.toStdString().c_str(), url.length());
avio_open(&pOutputFormatContext->pb, url.toStdString().c_str(), AVIO_FLAG_WRITE));
pOutputFormatContext->oformat->video_codec = AV_CODEC_ID_H264 ;
pOutputFormatContext->oformat->audio_codec = AV_CODEC_ID_AAC ;
The audio stream is created with :
AVDictionary *opts = nullptr;
//pAudioCodec = avcodec_find_encoder(AV_CODEC_ID_VORBIS);
pAudioCodec = avcodec_find_encoder(AV_CODEC_ID_AAC);
pAudioCodecContext = avcodec_alloc_context3(pAudioCodec);
pAudioCodecContext->thread_count = 1;
pAudioFrame = av_frame_alloc();
av_dict_set(&opts, "strict", "experimental", 0);
pAudioCodecContext->bit_rate = 128000;
pAudioCodecContext->sample_fmt = AV_SAMPLE_FMT_FLTP;
pAudioCodecContext->sample_rate = static_cast<int>(sample_rate) ;
pAudioCodecContext->channels = nb_channels ;
pAudioCodecContext->time_base.num = 1;
pAudioCodecContext->time_base.den = 1000 ;
//pAudioCodecContext->time_base.den = static_cast<int>(sample_rate) ;
pAudioCodecContext->codec_type = AVMEDIA_TYPE_AUDIO;
avcodec_open2(pAudioCodecContext, pAudioCodec, &opts);
pAudioFrame->nb_samples = pAudioCodecContext->frame_size;
pAudioFrame->format = pAudioCodecContext->sample_fmt;
pAudioFrame->channel_layout = pAudioCodecContext->channel_layout;
mAudioSamplesBufferSize = av_samples_get_buffer_size(nullptr, pAudioCodecContext->channels, pAudioCodecContext->frame_size, pAudioCodecContext->sample_fmt, 0);
avcodec_fill_audio_frame(pAudioFrame, pAudioCodecContext->channels, pAudioCodecContext->sample_fmt, (const uint8_t*) pAudioSamples, mAudioSamplesBufferSize, 0);
if( pOutputFormatContext->oformat->flags & AVFMT_GLOBALHEADER )
pAudioCodecContext->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
pAudioStream = avformat_new_stream(pOutputFormatContext, 0);
pAudioStream->codec = pAudioCodecContext ;
pAudioStream->id = pOutputFormatContext->nb_streams-1;;
pAudioStream->time_base.den = pAudioStream->pts.den = pAudioCodecContext->time_base.den;
pAudioStream->time_base.num = pAudioStream->pts.num = pAudioCodecContext->time_base.num;
mAudioPacketTs = 0 ;
The video stream is created with :
pVideoCodec = avcodec_find_encoder(AV_CODEC_ID_H264);
pVideoCodecContext = avcodec_alloc_context3(pVideoCodec);
pVideoCodecContext->codec_type = AVMEDIA_TYPE_VIDEO ;
pVideoCodecContext->thread_count = 1 ;
pVideoCodecContext->width = width;
pVideoCodecContext->height = height;
pVideoCodecContext->bit_rate = 8000000 ;
pVideoCodecContext->time_base.den = 1000 ;
pVideoCodecContext->time_base.num = 1 ;
pVideoCodecContext->gop_size = 10;
pVideoCodecContext->pix_fmt = AV_PIX_FMT_YUV420P;
pVideoCodecContext->flags = 0x0007 ;
pVideoCodecContext->extradata_size = sizeof(extra_data_buffer);
pVideoCodecContext->extradata = (uint8_t *) av_malloc ( sizeof(extra_data_buffer) );
memcpy ( pVideoCodecContext->extradata, extra_data_buffer, sizeof(extra_data_buffer));
avcodec_open2(pVideoCodecContext,pVideoCodec,0);
pVideoFrame = av_frame_alloc();
AVDictionary *opts = nullptr;
av_dict_set(&opts, "strict", "experimental", 0);
memcpy(pOutputFormatContext->filename, this->mStreamUrl.toStdString().c_str(), this->mStreamUrl.length());
pOutputFormatContext->oformat->video_codec = AV_CODEC_ID_H264 ;
if( pOutputFormatContext->oformat->flags & AVFMT_GLOBALHEADER )
pVideoCodecContext->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
pVideoStream = avformat_new_stream(pOutputFormatContext, pVideoCodec);
//This following section is because AVFormat complains about parameters being passed throught the context and not CodecPar
pVideoStream->codec = pVideoCodecContext ;
pVideoStream->id = pOutputFormatContext->nb_streams-1;
pVideoStream->time_base.den = pVideoStream->pts.den = pVideoCodecContext->time_base.den;
pVideoStream->time_base.num = pVideoStream->pts.num = pVideoCodecContext->time_base.num;
pVideoStream->avg_frame_rate.num = fps ;
pVideoStream->avg_frame_rate.den = 1 ;
pVideoStream->codec->gop_size = 10 ;
mVideoPacketTs = 0 ;
Then each video packet and audio packet is pushed with correct scaled pts/dts. I have corrected the 48kHz issue. It was because i was configuring the stream through the codec context and the through the codec parameters (because of waarning at runtime).
This RTMP stream still does not work for HLS conversion by NGINX, but if i just use FFMPEG to take the RTMP stream from NGINX and re-publish it with copy codec then it works.
I have videofile. How can I get fps for this video with ffmpeg in c++?
Type full code, please.
This is a simple program I wrote to dump video information to console:
#include <libavformat/avformat.h>
int main(int argc, const char *argv[])
{
if (argc < 2)
{
printf("No video file.\n");
return -1;
}
av_register_all();
AVFormatContext *pFormatCtx = NULL;
//open video file
if (avformat_open_input(&pFormatCtx, argv[1], NULL, NULL) != 0)
return -1;
//get stream info
if (avformat_find_stream_info(pFormatCtx, NULL) < 0)
return -1;
av_dump_format(pFormatCtx, 0, argv[1], 0);
}
Compile and run it, output looks like:
s#ubuntu-vm:~/Desktop/video-info-dump$ ./vdump a.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'a.mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
creation_time : 2014-04-23 06:18:02
encoder : FormatFactory : www.pcfreetime.com
Duration: 00:07:20.60, start: 0.000000, bitrate: 1354 kb/s
Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 1228 kb/s, 24 fps, 24 tbr, 24k tbn, 24 tbc (default)
Metadata:
creation_time : 2014-04-23 06:18:02
handler_name : video
Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 123 kb/s (default)
Metadata:
creation_time : 2014-04-23 06:18:25
handler_name : sound
Recommend a very good tutorial for ffmpeg and SDL.
#zhm answer was very close, I've made a small update to get the frame rate only. On my end, I need the bit_rate and this is an int64_t value in the AVFormatContext *.
For the FPS, you need to go through the list of streams, probably check whether it's audio or video, and then access the r_frame_rate, which is an AVRational value. The parameter is a nominator and denominator, you can simple divide one by the other to get a double and they even offer a function (av_q2d()) to do it.
int main(int argc, char * argv[])
{
if (argc < 2)
{
printf("No video file.\n");
return -1;
}
av_register_all();
AVFormatContext *pFormatCtx = NULL;
//open video file
if (avformat_open_input(&pFormatCtx, argv[1], NULL, NULL) != 0)
return -1;
//get stream info
if (avformat_find_stream_info(pFormatCtx, NULL) < 0)
return -1;
// dump the whole thing like ffprobe does
//av_dump_format(pFormatCtx, 0, argv[1], 0);
// get the frame rate of each stream
for(int idx(0); idx < pFormatCtx->nb_streams; ++idx)
{
AVStream *s(pFormatCtx->streams[idx]);
std::cout << idx << ". " << s->r_frame_rate.nom
<< " / " << s->r_frame_rate.den
<< " = " << av_q2d(s->r_frame_rate)
<< "\n";
}
// get the video bit rate
std::cout << "bit rate " << pFormatCtx->bit_rate << "\n";
return 0;
}
For more information, you may want to take a look at the avformat.h header where the AVFormatContext and AVStream structures are defined.
You could execute ffmpeg.exe like this ffmpeg -i filename and it would output the framerate if its not variable.
Example:
Input #0, matroska,webm, from 'somerandom.mkv':
Duration: 01:16:10.90, start: 0.000000, bitrate: N/A
Stream #0.0: Video: h264 (High), yuv420p, 720x344 [PAR 1:1 DAR 90:43], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
Stream #0.1: Audio: aac, 48000 Hz, stereo, s16 (default)
This video has a fps of 25.
To execute a program you can use the answer in https://stackoverflow.com/a/17703834/58553
Source: https://askubuntu.com/questions/110264/how-to-find-frames-per-second-of-any-video-file
Good day.
For brevity, the code omits error handling and memory management.
I want to capture h264 video stream and pack it to mp4 container without changes. Since i don't control the source of stream, i can not make assumptions about stream structure. In this way i must probe input stream.
AVProbeData probeData;
probeData.buf_size = s->BodySize();
probeData.buf = s->GetBody();
probeData.filename = "";
AVInputFormat* inFormat = av_probe_input_format(&probeData, 1);
This code correctly defines h264 stream.
Next, i create input format context,
unsigned char* avio_input_buffer = reinterpret_cast<unsigned char*> (av_malloc(AVIO_BUFFER_SIZE));
AVIOContext* avio_input_ctx = avio_alloc_context(avio_input_buffer, AVIO_BUFFER_SIZE,
0, this, &read_packet, NULL, NULL);
AVFormatContext* ifmt_ctx = avformat_alloc_context();
ifmt_ctx->pb = avio_input_ctx;
int ret = avformat_open_input(&ifmt_ctx, NULL, inFormat, NULL);
set image size,
ifmt_ctx->streams[0]->codec->width = ifmt_ctx->streams[0]->codec->coded_width = width;
ifmt_ctx->streams[0]->codec->height = ifmt_ctx->streams[0]->codec->coded_height = height;
create output format context,
unsigned char* avio_output_buffer = reinterpret_cast<unsigned char*>(av_malloc(AVIO_BUFFER_SIZE));
AVIOContext* avio_output_ctx = avio_alloc_context(avio_output_buffer, AVIO_BUFFER_SIZE,
1, this, NULL, &write_packet, NULL);
AVFormatContext* ofmt_ctx = nullptr;
avformat_alloc_output_context2(&ofmt_ctx, NULL, "mp4", NULL);
ofmt_ctx->pb = avio_output_ctx;
AVDictionary* dict = nullptr;
av_dict_set(&dict, "movflags", "faststart", 0);
av_dict_set(&dict, "movflags", "frag_keyframe+empty_moov", 0);
AVStream* outVideoStream = avformat_new_stream(ofmt_ctx, nullptr);
avcodec_copy_context(outVideoStream->codec, ifmt_ctx->streams[0]->codec);
ret = avformat_write_header(ofmt_ctx, &dict);
Initialization is done. Further there is a shifting packets from h264 stream to mp4 container. I dont calculate pts and dts, because source packet has AV_NOPTS_VALUE in them.
AVPacket pkt;
while (...)
{
ret = av_read_frame(ifmt_ctx, &pkt);
ret = av_interleaved_write_frame(ofmt_ctx, &pkt);
av_free_packet(&pkt);
}
Further i write trailer and free allocated memory. That is all. Code works and i got playable mp4 file.
Now the problem: the stream characteristics of the resulting file is not completely consisent with the characteristics of the source stream. In particular, fps and bitrate is higher than it should be.
As sample, below is output ffplay.exe for source stream
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'd:/movies/source.mp4':0/0
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
creation_time : 2014-04-14T13:03:54.000000Z
Duration: 00:00:58.08, start: 0.000000, bitrate: 12130 kb/s
Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661),yuv420p, 1920x1080, 12129 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler
Switch subtitle stream from #-1 to #-1 vq= 1428KB sq= 0B f=0/0
Seek to 49% ( 0:00:28) of total duration ( 0:00:58) B f=0/0
30.32 M-V: -0.030 fd= 87 aq= 0KB vq= 1360KB sq= 0B f=0/0
and for resulting stream (contains part of source stream)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'd:/movies/target.mp4':f=0/0
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1iso6mp41
encoder : Lavf57.56.101
Duration: 00:00:11.64, start: 0.000000, bitrate: 18686 kb/s
Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080, 18683 kb/s, 38.57 fps, 40 tbr, 90k tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler
Switch subtitle stream from #-1 to #-1 vq= 2309KB sq= 0B f=0/0
5.70 M-V: 0.040 fd= 127 aq= 0KB vq= 2562KB sq= 0B f=0/0
So there is a question, what i missed when copying stream? I will be grateful for any help.
Best regards
I dont calculate pts and dts This is your problem. frame rate and bit rate are both ratios where time is the denominator. But not writing pts/dts you end up with a video shorter than you want. h.264 does not timestamp every frame. that is the containers job. You must make up time stamps from the known frame rate, or another value.
I have a stereo camera that gives an image in the format YUYV with a a resolution of 320 x 480, where in each pixel (16 bits) has encoded 2 pixels of 8 bits. I'm using OpenCV in order to get the image but when I try to get the real resolution of the image I wont get good results. I guess I missing how to properly split the 16 bits in two.
Using this and this I'm able to reconstruct an image but still is not the real one.
Mat frame;
unsigned int width= cap.get(CV_CAP_PROP_FRAME_WIDTH);
unsigned int height= cap.get(CV_CAP_PROP_FRAME_HEIGHT);
m_pDepthImgBuf = (unsigned char*)calloc(width*height*2, sizeof(unsigned char));
...
cap >> frame; // get a new frame from camera
imshow("YUVY 320x480", frame);
memcpy( (void*)m_pDepthImgBuf, (void*)frame.data, width*height*2 * sizeof(unsigned char) );
cv::Mat depth(height,width*2,CV_8UC1,(void*)m_pDepthImgBuf);
camera properties:
SDL information:
Video driver: x11
Device information:
Device path: /dev/video2
Stream settings:
Frame format: YUYV (MJPG is not supported by device)
Frame size: 320x480
Frame rate: 30 fps
v4l2-ctl -d /dev/video1 --list-formats
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'YUYV'
Name : YUV 4:2:2 (YUYV)
In the following image you can see in green the initial 320x480 image and in gray scale the depth that I'm trying to extract.
The expected result should be:
I have multiple recorded video samples, when I run these through my program it returns the FPS among other things. It is accurate enough for all of my video samples (see table below) but when I run a video sample taken through my smartphone it is returning the FPS at 90000, this happens with every video that captured through my smartphone so it is not just a problem with a single video file.
File Actual FPS OpenCV FPS ffmpeg FPS
action-60fps 60 59 60
action-24fps 24 24 24
phone_panning 29 90000 29
What is causing this problem?
EDIT: Managed to forget to add my code...
VideoCapture capture(argv[1]);
Mat frame;
if(capture.isOpened()) {
int fps = capture.get(CV_CAP_PROP_FPS);
int width = capture.get(CV_CAP_PROP_FRAME_WIDTH);
int height = capture.get(CV_CAP_PROP_FRAME_HEIGHT);
cout << "FPS: " << fps << ", width: " << width << ", height: " << height << endl;
VideoWriter writer("output.mpg",
CV_FOURCC('P','I','M','1'), fps, cvSize(width, height), 0); // 0 means gray, 1 means color
if(writer.isOpened()) {
while(true) {
capture >> frame;
if(!frame.empty()) {
imshow("Video", frame);
}
Mat frame_gray = frame.clone();
cvtColor(frame, frame_gray, CV_RGB2GRAY);
writer << frame_gray;
int key = waitKey(25);
if((char)key == 'q') {
break;
}
}
}
}
I had the same problem with opencv in calculating the FPS and number of frames in a video. (It was returning 90,000 for FPS and 5,758,245 for frame count for a 64-second video!!)
According to this answer:
OpenCV captures only a fraction of the frames from a video file
it's an opencv issue and they are working on it.
Another reason could be a problem with file header, mine was caused by converting video format. I solved it by using original video format mp4, instead of avi.