Create video using ffmpeg - c++

I have 100 images(PNG) and I want to create a video using these images. I am using the ffmpeg library for this. Using command line I can create video easily. But how do I do it through coding?
Any help will be appreciated.
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#ifdef HAVE_AV_CONFIG_H
#undef HAVE_AV_CONFIG_H
#endif
extern "C"
{
#include "libavutil/imgutils.h"
#include "libavutil/opt.h"
#include "libavcodec/avcodec.h"
#include "libavutil/mathematics.h"
#include "libavutil/samplefmt.h"
}
#define INBUF_SIZE 4096
#define AUDIO_INBUF_SIZE 20480
#define AUDIO_REFILL_THRESH 4096
static void video_encode_example(const char *filename, int codec_id)
{
AVCodec *codec;
AVCodecContext *c= NULL;
int i, out_size, size, x, y, outbuf_size;
FILE *f;
AVFrame *picture;
uint8_t *outbuf;
int nrOfFramesPerSecond =25;
int nrOfSeconds =1;
printf("Video encoding\n");
// find the mpeg1 video encoder
codec = avcodec_find_encoder((CodecID) codec_id);
if (!codec) {
fprintf(stderr, "codec not found\n");
exit(1);
}
c = avcodec_alloc_context3(codec);
picture= avcodec_alloc_frame();
// put sample parameters
c->bit_rate = 400000;
// resolution must be a multiple of two
c->width = 352;
c->height = 288;
// frames per second
c->time_base= (AVRational){1,25};
c->gop_size = 10; //emit one intra frame every ten frames
c->max_b_frames=1;
c->pix_fmt = PIX_FMT_YUV420P;
if(codec_id == CODEC_ID_H264)
av_opt_set(c->priv_data, "preset", "slow", 0);
// open it
if (avcodec_open2(c, codec, NULL) < 0) {
fprintf(stderr, "could not open codec\n");
exit(1);
}
f = fopen(filename, "wb");
if (!f) {
fprintf(stderr, "could not open %s\n", filename);
exit(1);
}
// alloc image and output buffer
outbuf_size = 100000;
outbuf = (uint8_t*) malloc(outbuf_size);
// the image can be allocated by any means and av_image_alloc() is
// * just the most convenient way if av_malloc() is to be used
av_image_alloc(picture->data, picture->linesize,
c->width, c->height, c->pix_fmt, 1);
// encode 1 second of video
int nrOfFramesTotal = nrOfFramesPerSecond * nrOfSeconds;
// encode 1 second of video
for(i=0;i < nrOfFramesTotal; i++) {
fflush(stdout);
// prepare a dummy image
for(y=0;y<c->height;y++) {
for(x=0;x<c->width;x++) {
picture->data[0][y * picture->linesize[0] + x] = x + y + i * 3;
}
}
// Cb and Cr
for(y=0;y<c->height/2;y++) {
for(x=0;x<c->width/2;x++) {
picture->data[1][y * picture->linesize[1] + x] = 128 + y + i * 2;
picture->data[2][y * picture->linesize[2] + x] = 64 + x + i * 5;
}
}
// encode the image
out_size = avcodec_encode_video(c, outbuf, outbuf_size, picture);
printf("encoding frame %3d (size=%5d)\n", i, out_size);
fwrite(outbuf, 1, out_size, f);
}
// get the delayed frames
for(; out_size; i++) {
fflush(stdout);
out_size = avcodec_encode_video(c, outbuf, outbuf_size, NULL);
printf("write frame %3d (size=%5d)\n", i, out_size);
fwrite(outbuf, 1, out_size, f);
}
// add sequence end code to have a real mpeg file
outbuf[0] = 0x00;
outbuf[1] = 0x00;
outbuf[2] = 0x01;
outbuf[3] = 0xb7;
fwrite(outbuf, 1, 4, f);
fclose(f);
free(outbuf);
avcodec_close(c);
// av_free(c);
// av_free(picture->data[0]);
// av_free(picture);
printf("\n");
}
int main(int argc, char **argv)
{
const char *filename;
avcodec_register_all();
if (argc <= 1) {
video_encode_example("/home/radix/Desktop/OpenCV/FFMPEG_Output/op89.png", AV_CODEC_ID_H264);
} else {
filename = argv[1];
}
return 0;
}
On searching everytime i m getting code similar to this.But i don't understood hot to use it for creating video from images.

The reason this comes up again and again is because you're using encoding_example.c as your reference. Please don't do that. The most fundamental mistake in this example is that it doesn't teach you the difference between codecs and containers. In fact, it ignored containers altogether.
What is a codec?
A codec is a method of compressing a media type. H264, for example, will compress raw video. Imagine a 1080p video frame, which is typically in YUV format with 4:2:0 chroma subsampling. Raw, this is 1080*1920*3/2 bytes per frame, i.e. ~3MB/f. For 60fps, this is 180MB/sec, or 1.44 gigabit/sec (gbps). That's a lot of data, so we compress it. At that resolution, you can get pretty quality at a few megabit/sec (mbps) for modern codecs, like H264, HEVC or VP9. For audio, codecs like AAC or Opus are popular.
What is a container?
A container takes video or audio (or subtitle) packets (compressed or uncompressed) and interleaves them for combined storage in a single output file. So rather than getting one file for video and one for audio, you get one file that interleaves packets for both. This allows effective seeking and indexing, it typically also allows adding metadata storage ("author", "title") and so on. Examples of popular containers are MOV, MP4 (which is really just mov), AVI, Ogg, Matroska or WebM (which is really just matroska).
(You can store video-only data in a file if you want. For H264, this is called "annexb" raw H264. This is actually what you were doing above. So why didn't it work? Well, you're ignoring "header" packets like the SPS and PPS. These are in avctx->extradata and need to be written before the first video packet. Using a container would take care of that for you, but you didn't, so it didn't work.)
How do you use a container in FFmpeg? See e.g. this post, particularly the sections calling functions like avformat_write_*() (basically anything that sounds like output). I'm happy to answer more specific questions, but I think the above post should clear out most confusion for you.

Related

libav producing MP4 file with extremely high frame rate

I am trying to write a program to generate frames to be encoded via ffmpeg/libav into an mp4 file with a single h264 stream. I found these two examples and am sort of trying to merge them together to make what I want: [video transcoder] [raw MPEG1 encoder]
I have been able to get video output (green circle changing size), but no matter how I set the PTS values of the frames or what time_base I specify in the AVCodecContext or AVStream, I'm getting frame rates of about 7000-15000 instead of 60, resulting in a video file that lasts 70ms instead of 1000 frames / 60 fps = 166 seconds. Every time I change some of my code, the frame rate changes a little bit, almost as if it's reading from uninitialized memory. Other references to an issue like this on StackOverflow seem to be related to incorrectly set PTS values; however, I've tried printing out all the PTS, DTS, and time base values I can find and they all seem normal. Here's my proof-of-concept code (with the error catching stuff around the libav calls removed for clarity):
#include <iostream>
#include <opencv2/opencv.hpp>
#include <math.h>
extern "C" {
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/opt.h>
#include <libavutil/timestamp.h>
}
using namespace cv;
int main(int argc, char *argv[]) {
const char *filename = "testvideo.mp4";
AVFormatContext *avfc;
avformat_alloc_output_context2(&avfc, NULL, NULL, filename);
AVStream *stream = avformat_new_stream(avfc, NULL);
AVCodec *h264 = avcodec_find_encoder(AV_CODEC_ID_H264);
AVCodecContext *avcc = avcodec_alloc_context3(h264);
av_opt_set(avcc->priv_data, "preset", "fast", 0);
av_opt_set(avcc->priv_data, "crf", "20", 0);
avcc->thread_count = 1;
avcc->width = 1920;
avcc->height = 1080;
avcc->pix_fmt = AV_PIX_FMT_YUV420P;
avcc->time_base = av_make_q(1, 60);
stream->time_base = avcc->time_base;
if(avfc->oformat->flags & AVFMT_GLOBALHEADER)
avcc->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
avcodec_open2(avcc, h264, NULL);
avcodec_parameters_from_context(stream->codecpar, avcc);
avio_open(&avfc->pb, filename, AVIO_FLAG_WRITE);
avformat_write_header(avfc, NULL);
Mat frame, nothing = Mat::zeros(1080, 1920, CV_8UC1);
AVFrame *avf = av_frame_alloc();
AVPacket *avp = av_packet_alloc();
int ret;
avf->format = AV_PIX_FMT_YUV420P;
avf->width = 1920;
avf->height = 1080;
avf->linesize[0] = 1920;
avf->linesize[1] = 1920;
avf->linesize[2] = 1920;
for(int x=0; x<1000; x++) {
frame = Mat::zeros(1080, 1920, CV_8UC1);
circle(frame, Point(1920/2, 1080/2), 250*(sin(2*M_PI*x/1000*3)+1.01), Scalar(255), 10);
avf->data[0] = frame.data;
avf->data[1] = nothing.data;
avf->data[2] = nothing.data;
avf->pts = x;
ret = 0;
do {
if(ret == AVERROR(EAGAIN)) {
av_packet_unref(avp);
ret = avcodec_receive_packet(avcc, avp);
if(ret) break; // deal with error
av_write_frame(avfc, avp);
} //else if(ret) deal with error
ret = avcodec_send_frame(avcc, avf);
} while(ret);
}
// flush the rest of the packets
avcodec_send_frame(avcc, NULL);
do {
av_packet_unref(avp);
ret = avcodec_receive_packet(avcc, avp);
if(!ret)
av_write_frame(avfc, avp);
} while(!ret);
av_frame_free(&avf);
av_packet_free(&avp);
av_write_trailer(avfc);
avformat_close_input(&avfc);
avformat_free_context(avfc);
avcodec_free_context(&avcc);
return 0;
}
This is the output of ffprobe run on the output video file
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'testvideo.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.76.100
Duration: 00:00:00.07, start: 0.000000, bitrate: 115192 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 115389 kb/s, 15375.38 fps, 15360 tbr, 15360 tbn, 120 tbc (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
What might be causing my frame rate to be so high? Thanks in advance for any help.
You are getting high frame rate because you have failed to set packet duration.
Set the time_base to higher resolution (like 1/60000) as described here:
avcc->time_base = av_make_q(1, 60000);
Set avp->duration as described here:
AVRational avg_frame_rate = av_make_q(60, 1); //60 fps
avp->duration = avcc->time_base.den / avcc->time_base.num / avg_frame_rate.num * avg_frame_rate.den; //avp->duration = 1000 (60000/60)
And set the pts accordingly.
Complete code:
#include <iostream>
#include <opencv2/opencv.hpp>
#include <math.h>
extern "C" {
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/opt.h>
#include <libavutil/timestamp.h>
}
using namespace cv;
int main(int argc, char* argv[]) {
const char* filename = "testvideo.mp4";
AVFormatContext* avfc;
avformat_alloc_output_context2(&avfc, NULL, NULL, filename);
AVStream* stream = avformat_new_stream(avfc, NULL);
AVCodec* h264 = avcodec_find_encoder(AV_CODEC_ID_H264);
AVCodecContext* avcc = avcodec_alloc_context3(h264);
av_opt_set(avcc->priv_data, "preset", "fast", 0);
av_opt_set(avcc->priv_data, "crf", "20", 0);
avcc->thread_count = 1;
avcc->width = 1920;
avcc->height = 1080;
avcc->pix_fmt = AV_PIX_FMT_YUV420P;
//Sey the time_base to higher resolution like 1/60000
avcc->time_base = av_make_q(1, 60000); //avcc->time_base = av_make_q(1, 60);
stream->time_base = avcc->time_base;
if (avfc->oformat->flags & AVFMT_GLOBALHEADER)
avcc->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
avcodec_open2(avcc, h264, NULL);
avcodec_parameters_from_context(stream->codecpar, avcc);
avio_open(&avfc->pb, filename, AVIO_FLAG_WRITE);
avformat_write_header(avfc, NULL);
Mat frame, nothing = Mat::zeros(1080, 1920, CV_8UC1);
AVFrame* avf = av_frame_alloc();
AVPacket* avp = av_packet_alloc();
int ret;
avf->format = AV_PIX_FMT_YUV420P;
avf->width = 1920;
avf->height = 1080;
avf->linesize[0] = 1920;
avf->linesize[1] = 1920;
avf->linesize[2] = 1920;
for (int x = 0; x < 1000; x++) {
frame = Mat::zeros(1080, 1920, CV_8UC1);
circle(frame, Point(1920 / 2, 1080 / 2), (int)(250.0 * (sin(2 * M_PI * x / 1000 * 3) + 1.01)), Scalar(255), 10);
AVRational avg_frame_rate = av_make_q(60, 1); //60 fps
int64_t avp_duration = avcc->time_base.den / avcc->time_base.num / avg_frame_rate.num * avg_frame_rate.den;
avf->data[0] = frame.data;
avf->data[1] = nothing.data;
avf->data[2] = nothing.data;
avf->pts = (int64_t)x * avp_duration; // avp->duration = 1000
ret = 0;
do {
if (ret == AVERROR(EAGAIN)) {
av_packet_unref(avp);
ret = avcodec_receive_packet(avcc, avp);
if (ret) break; // deal with error
////////////////////////////////////////////////////////////////
//avp->duration was zero.
avp->duration = avp_duration; //avp->duration = 1000 (60000/60)
//avp->pts = (int64_t)x * avp->duration;
////////////////////////////////////////////////////////////////
av_write_frame(avfc, avp);
} //else if(ret) deal with error
ret = avcodec_send_frame(avcc, avf);
} while (ret);
}
// flush the rest of the packets
avcodec_send_frame(avcc, NULL);
do {
av_packet_unref(avp);
ret = avcodec_receive_packet(avcc, avp);
if (!ret)
av_write_frame(avfc, avp);
} while (!ret);
av_frame_free(&avf);
av_packet_free(&avp);
av_write_trailer(avfc);
avformat_close_input(&avfc);
avformat_free_context(avfc);
avcodec_free_context(&avcc);
return 0;
}
Result of FFprobe:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'testvideo.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.76.100
Duration: 00:00:16.65, start: 0.000000, bitrate: 456 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 450 kb/s, 60.06 fps, 60 tbr, 60k tbn, 120k tbc (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Notes:
I don't know why the fps is 60.06 and not 60.
There is a warning message MB rate (734400000) > level limit (16711680) that I didn't fix.
Though the answer I accepted fixes the problem I was having, here is some more information I've figured out that may be useful:
The time_base field has some restrictions on its value (for example 1/10000 works, but 1/9999 doesn't) based on the container format, and this seems to have been the root problem I was having. When the time base was set to 1/60, the call to avformat_write_header() changed it to 1/15360. Because I had hardcoded the PTS increment to 1, this resulted in the 15360 FPS video. The strange denominator of 15360 seems to result from the given denominator being multiplied by 2 repeatedly until it reaches some minimum value. I have no idea how this algorithm works actually works. This SO question led me on to this.
By setting the time base to 1/60000 and making the PTS increment by 1000 each frame, the fast video problem was fixed. Setting the packet duration doesn't seem necessary, but is probably a good idea.
The main lesson here is to use whatever time_base libav gives you instead of assuming the value you set it to stays unchanged. #Rotem's updated code does this, and would therefore "work" with a time base of 1/60, since the PTS and packet duration will actually be based off the 1/15360 value time_base changes to.

ffmpeg writes invalid fps to the mp4 container (and in avi it is true). What is the reason?

I need to record frames in real time. To test this situation, I make pts non-linear (since frames may be lost), thus:
// AVFrame
video_frame->pts = prev_pts + 2;
I use libavformat to write to a file. Parameters AVCodecContext and AVStream:
#define STREAM_FRAME_RATE 25
#define CODEC_PIX_FMT AV_PIX_FMT_YUV420P
#define FRAME_WIDTH 1440
#define FRAME_HEIGHT 900
// AVCodecContext
cc->codec_id = video_codec->id;
cc->bit_rate = 400000;
cc->width = FRAME_WIDTH;
cc->height = FRAME_HEIGHT;
cc->gop_size = 12;
cc->pix_fmt = CODEC_PIX_FMT;
// AVStream
video_stream->time_base = AVRational{ 1, STREAM_FRAME_RATE };
cc->time_base = video_stream->time_base;
cc->framerate = AVRational{ STREAM_FRAME_RATE , 1 };
Write to file:
static int write_frame(AVFormatContext *fmt_ctx, const AVRational *time_base, AVStream *st, AVPacket *pkt)
{
/* rescale output packet timestamp values from codec to stream timebase */
//av_packet_rescale_ts(pkt, *time_base, st->time_base);
pkt->pts = av_rescale_q(pkt->pts, *time_base, st->time_base);
pkt->dts = av_rescale_q(pkt->dts, *time_base, st->time_base);
pkt->stream_index = st->index;
/* Write the compressed frame to the media file. */
//log_packet(fmt_ctx, pkt);
//return av_write_frame(fmt_ctx, pkt);
return av_interleaved_write_frame(fmt_ctx, pkt);
}
If you use the avi container, then the information on the number of frames per second is indicated correctly in the file: 25 fps
If you use the mp4 container, then the file information about the number of frames per second is indicated incorrectly: 12.5 fps
Tell me, please, what other settings need to be added?
MP4s do not store framerate, AVIs do.
In MP4s, only timing info for packets is stored. Since your pts expr is video_frame->pts = prev_pts + 2 and stream time base is 1/25, frames are spaced 80ms apart and hence ffmpeg probes the frame rate to be 12.5 fps (correctly).
AVIs do not have per-frame timing. Instead, they write the user-supplied frame rate. Should a packet timing be greater than the pervious frame pts by 1/fps, the muxer will write skip frame(s) which are empty packets, to maintain the frame rate.

How can I Convert raw file to an audio file (.wav) by FFmpeg in c++

I want to covert mp4 format to wav format with different sample rate in my c++ application.
First of all I have extracted audio from mp4 file by ffmpeg in c++, then i have converted that to a raw file, but I down not know how can I convert raw file to a wav file with different sample rate.
How can I solve this?
#include "ffmpeg.h"
int decode_packet(int *got_frame, int cached)
{
int ret = 0;
int decoded = pkt.size;
*got_frame = 0;
if (pkt.stream_index == video_stream_idx) {
/* decode video frame */
ret = avcodec_decode_video2(video_dec_ctx, frame, got_frame, &pkt);
if (ret < 0) {
// fprintf(stderr, "Error decoding video frame (%s)\n", av_err2str(ret));
return ret;
}
if (*got_frame) {
if (frame->width != width || frame->height != height ||
frame->format != pix_fmt) {
/* To handle this change, one could call av_image_alloc again and
* decode the following frames into another rawvideo file. */
// fprintf(stderr, "Error: Width, height and pixel format have to be "
// "constant in a rawvideo file, but the width, height or "
// "pixel format of the input video changed:\n"
// "old: width = %d, height = %d, format = %s\n"
// "new: width = %d, height = %d, format = %s\n",
// width, height, av_get_pix_fmt_name(pix_fmt),
// frame->width, frame->height,
// av_get_pix_fmt_name(frame->format));
return -1;
}
printf("video_frame%s n:%d coded_n:%d\n",
cached ? "(cached)" : "",
video_frame_count++, frame->coded_picture_number);
/* copy decoded frame to destination buffer:
* this is required since rawvideo expects non aligned data */
av_image_copy(video_dst_data, video_dst_linesize,
(const uint8_t **)(frame->data), frame->linesize,
pix_fmt, width, height);
/* write to rawvideo file */
fwrite(video_dst_data[0], 1, video_dst_bufsize, video_dst_file);
}
} else if (pkt.stream_index == audio_stream_idx) {
/* decode audio frame */
ret = avcodec_decode_audio4(audio_dec_ctx, frame, got_frame, &pkt);
if (ret < 0) {
// fprintf(stderr, "Error decoding audio frame (%s)\n", av_err2str(ret));
return ret;
}
/* Some audio decoders decode only part of the packet, and have to be
* called again with the remainder of the packet data.
* Sample: fate-suite/lossless-audio/luckynight-partial.shn
* Also, some decoders might over-read the packet. */
decoded = FFMIN(ret, pkt.size);
if (*got_frame) {
size_t unpadded_linesize = frame->nb_samples * av_get_bytes_per_sample((AVSampleFormat)frame->format);
// printf("audio_frame%s n:%d nb_samples:%d pts:%s\n",
// cached ? "(cached)" : "",
// audio_frame_count++, frame->nb_samples,
// av_ts2timestr(frame->pts, &audio_dec_ctx->time_base));
/* Write the raw audio data samples of the first plane. This works
* fine for packed formats (e.g. AV_SAMPLE_FMT_S16). However,
* most audio decoders output planar audio, which uses a separate
* plane of audio samples for each channel (e.g. AV_SAMPLE_FMT_S16P).
* In other words, this code will write only the first audio channel
* in these cases.
* You should use libswresample or libavfilter to convert the frame
* to packed data. */
// fwrite(frame->extended_data[0], 1, unpadded_linesize, audio_dst_file);
//encode function
encode(cOut, frame, &pktout, audio_dst_file);
// av_init_packet(&pktout);
// pktout.data = NULL; // packet data will be allocated by the encoder
// pktout.size = 0;
// /* encode the samples */
// ret = avcodec_encode_audio2(cOut, &pktout, frame, &got_outputOut);
// if (ret < 0) {
// fprintf(stderr, "Error encoding audio frame\n");
// exit(1);
// }
// if (got_outputOut) {
// fwrite(pktout.data, 1, pktout.size, audio_dst_file);
// av_free_packet(&pktout);
// }
}
}
/* If we use frame reference counting, we own the data and need
* to de-reference it when we don't use it anymore */
if (*got_frame && refcount)
av_frame_unref(frame);
return decoded;
}
First you should use Libswresample to resample audio data.
Then you can save audio raw data with wav format.

Using libVLC to read RTSP video stream + convert data to OpenCV Mat

I'm attempting to read an RTSP stream from a Sony FCB-EV7520 camera through an IP interface using the libVLC library and convert the data to the format used by OpenCV, namely the Mat type. I've been trying to find a good example of this for a couple of days, but the only results I've found so far are this and this. I followed the code in the first example, adapting it to the RTSP use case, but I have yet to retrieve any data in the Mat. However, from the terminal output I seem to achieve a connection to the camera. Do you see any obvious flaws in the code? Are there any other (easier) ways of achieving my goal? Any other libraries that could be used? Any help would be appreciated! The code im running is:
#include <stdio.h>
#include <vlc/vlc.h>
#include <stdint.h>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>
using namespace std;
using namespace cv;
struct VideoDataStruct {
int param;
};
int done = 0;
libvlc_media_player_t *mp;
unsigned int videoBufferSize = 0;
uint8_t *videoBuffer = 0;
void cbVideoPrerender(void *p_video_data, uint8_t **pp_pixel_buffer, int size) {
// Locking
// Locking a mutex here once I get the code to work.
if (size > videoBufferSize || !videoBuffer){
printf("Reallocate raw video buffer\n");
free(videoBuffer);
videoBuffer = (uint8_t *) malloc(size);
videoBufferSize = size;
}
*pp_pixel_buffer = videoBuffer;
}
void cbVideoPostrender(void *p_video_data, uint8_t *p_pixel_buffer, int width, int height, int pixel_pitch, int size, int64_t pts) {
// Unlocking
// Unlocking the mutex here once I get the code to work.
}
static void handleEvent(const libvlc_event_t* pEvt, void* pUserData) {
libvlc_time_t time;
switch(pEvt->type){
case libvlc_MediaPlayerTimeChanged:
time = libvlc_media_player_get_time(mp);
printf("MediaPlayerTimeChanged %lld ms\n", (long long)time);
break;
case libvlc_MediaPlayerEndReached:
printf ("MediaPlayerEndReached\n");
done = 1;
break;
default:
printf("%s\n", libvlc_event_type_name(pEvt->type));
}
}
int main(int argc, char* argv[]) {
// VLC pointers:
libvlc_instance_t *inst;
libvlc_media_t *m;
void *pUserData = 0;
VideoDataStruct dataStruct;
// VLC options:
char smem_options[1000];
// RV24:
sprintf(smem_options
, "#transcode{vcodec=h264}:smem{"
"video-prerender-callback=%lld,"
"video-postrender-callback=%lld,"
"video-data=%lld,"
"no-time-sync},"
, (long long int)(intptr_t)(void*)&cbVideoPrerender
, (long long int)(intptr_t)(void*)&cbVideoPostrender
, (long long int)(intptr_t)(void*)&dataStruct
);
const char * const vlc_args[] = {
"-I", "dummy", // Don't use any interface
"--ignore-config", // Don't use VLC's config
"--extraintf=logger", // Log anything
"--verbose=2", // Be verbose
"--sout", smem_options // Stream to memory
};
// We launch VLC
inst = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args);
/* Create a new item */
m = libvlc_media_new_location(inst, "rtsp://*****:*******#IP/videoinput_1/h264_1/media.stm");
/* Create a media player playing environement */
mp = libvlc_media_player_new_from_media (m);
libvlc_event_manager_t* eventManager = libvlc_media_player_event_manager(mp);
libvlc_event_attach(eventManager, libvlc_MediaPlayerTimeChanged, handleEvent, pUserData);
libvlc_event_attach(eventManager, libvlc_MediaPlayerEndReached, handleEvent, pUserData);
libvlc_event_attach(eventManager, libvlc_MediaPlayerPositionChanged, handleEvent, pUserData);
libvlc_video_set_format(mp, "h264", 1920, 1080, 1920* 3 );
/* play the media_player */
libvlc_media_player_play (mp);
while(1){
if(videoBuffer){ // Check for invalid input
Mat img = Mat(Size(1920, 1080), CV_8UC3, videoBuffer); // CV_8UC3 = 8 bits, 3 chanels
namedWindow("Display window", WINDOW_AUTOSIZE); // Create a window for display.
imshow("Display window", img); // Show our image inside it.
}
}
libvlc_release (inst);
}
I am running the code on Ubuntu 16.04 using OpenCV 3.2 and libVLC 2.2.5.1. If you need any additional information, just ask.
PS: I know that the stream is working as I can open it through the streaming interface of the VLC player. I also know that libVLC can decode the stream as I have successfully opened recorded mp4s of the stream.
Not a complete answer, but too long for a comment:
In cbVideoPrerender, videoBuffer is always allocated as needed by vlc.
However, in main, you have the loop while(1){ if (videoBuffer) { ... } } where videoBuffer is always true from the first time cbVideoPrerender has been called. This means that from then on, the loop is infinite and non-blocking, so there is no synchronization between the input of the video and processing and if you're just trying to get the first image, you will be too early.
Your first link suggests using cbVideoPostrender as a synchronisation point so you know the frame can be read and thus converted to the needed format. There it is done in the function itself, but you could have some mechanism with a condition variable (queue, event,...) to process the frame in another thread and pass the image to openCV.
By the way: using a variable by setting in one thread and reading it in another one without thread mechanisms (mutex, atomic) is usably a bad idea.

Opencv how to get image stream from Mat quickly?

I can use imwrite() to write the image(like "face.jpg") into disk,
then use fstream to read this jpg into a array.this array is what I want.
but,how to get this quickly? from memmory not disk.
I thought the image data in Mat.data,length is Mat.cols*Mat.rows.I was not sure it is or not right.so,I used fstream write it into disk,then opened it with image viewer,nothing.there must something wrong.
Mat frame;
VideoCapture cap(0);
if (!cap.isOpened())
{
return -1;
}
cap.set(CV_CAP_PROP_FRAME_WIDTH, 160);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
cap >> frame;
if(frame.empty()){
return -2;
}
//I just want the pointer and length of image information,the following is just for testing
//whether that the same as I thought,if it's right ,frame.data and len is what I want,but it not work.
FILE *fp = fopen("face.jpg", "wb");
if (NULL==fp)
{
return -1;
}
int len = frame.cols*frame.rows; //or 3*frame.cols*frame.rows
fwrite(frame.data, len, sizeof(char), fp);
fclose(fp);
namedWindow("face", 1);
imshow("face", frame);
waitKey(1000);
I'm new in opencv,and I just want get the image data.thanks for help!
Have you check the dimensions before you write it to disk? It'll be helpful for the others to see your code here. In the case of Mat, unless your data is grayscale, the size will be more than cols * rows. You should verify if the format is RGB, RGBA, or YUV, etc. In the case of JPEG, it'll be most likely RGBX; so you should really check that your stream size is either 3 * cols * rows or 4 * cols * rows.
I did this just with imencode(),thanks for #ZdaR.
vector<uchar> buff;
vector<int>param = vector<int>(2);
param[0] = CV_IMWRITE_JPEG_QUALITY;
param[1] = 95;
imencode(".jpg", frame, buff, param);
int len = buff.size();
FILE *fout;
fout = fopen("555.jpg", "wb");
if(NULL==fout){
return -3;
}
fwrite(&buff[0], 1, len*sizeof(uchar), fout);
fclose(fout);