I'm working with IP camera, and I have got Jpeg frames and audio data (PCM) from camera.
Now, I want to create video file (both audio and video) under .avi or .mp4 format from above data.
I searched and I knew that ffmpeg library can do it. But I don't know how to using ffmpeg to do this.
Can you suggest me some sample code or the function of ffmpeg to do it?
If your objective is to write a c++ app to do this for you please disregard this answer, I'll just leave it here for future reference. If not, here's how you can do it in bash:
First, make sure your images are in a nice format, easy to handle by ffmpeg. You can copy the images to a different directory:
mkdir tmp
x=1; for i in *jpg; do counter=$(printf %03d $x); cp "$i" tmp/img"$counter".jpg; x=$(($x+1)); done
Copy your audio data to the tmp directory and encode the video. Let's say your camera took a picture every ten seconds:
cd tmp
ffmpeg -i audio.wav -f image2 -i img%03d.jpg -vcodec msmpeg4v2 -r 0.1 -intra out.avi
Where -r 0.1 indicates a framerate of 0.1 which is one frame every 10 seconds.
The possible issues here are:
Your audio/video might go slightly out of sync unless you calculate your desired framerate carefully in advance. You should be able to get the length of the audio (or video) using ffmpeg and some grep magic. Even so the sync might be an issue with longer clips.
if you have more than 999 images the %03d format will not be enough, make sure to change the 3 to the desired length of the index
The video will inherit its length from the longer of the streams, you can restrict it using the -t switch:
-t duration - Restrict the transcoded/captured video sequence to the duration specified in seconds. "hh:mm:ss[.xxx]" syntax is also supported.
Related
I am using cImg to generate images in a CImgList.
when I save that stack to video by calling the save_video method on the stack, it ignores the fps and the output seems to be always 25fps. I opened the file in different players (VLC, windows movie,...) and its always 25fps.
cImg is using ffmpeg to create the video. I'm not specifying any codec so i assume the default mpeg2 is used (based on what VLC tells me).
I also have no specfic settings for cImg.
I have a fixed amount of images of 500 and it always produces around 20 seconds which is 25fps.
What do I need to do to output it to for example 60fps?
I fixed it but not sure if it's a bug in cImg or just me not knowing how to use it properly but from what I found is that cImg doesn't pass the --framerate parameter to ffmpeg, only the -r parameter.
I updated the code to include the framerate parameter and it does seem to work. This is the updated cImg code in the save_ffmpeg_external function:
cimg_snprintf(command,command._width,
"\"%s\" -framerate %u -v -8 -y -i \"%s_%%6d.ppm\" -pix_fmt yuv420p -vcodec %s -b %uk -r %u \"%s\"",
cimg::ffmpeg_path(),
fps,
CImg<charT>::string(filename_tmp)._system_strescape().data(),
_codec,bitrate,fps,
CImg<charT>::string(filename)._system_strescape().data());
Hi stackoverflow community,
I have a tricky problem and I need your help to understand what is going on here.
My program captures frames from a video grabber card (Blackmagic) which just works fine so far, at the same time I display the captured images with opencv (cv::imshow) which works good as well (But pretty cpu wasting).
The captured images are supposed to be stored on the disk as well, for this I put the captured Frames (cv::Mat) on a stack, to finally write them async with opencv:
cv::VideoWriter videoWriter(path, cv::CAP_FFMPEG, fourcc, fps, *size);
videoWriter.set(cv::VIDEOWRITER_PROP_QUALITY, 100);
int id = metaDataWriter.insertNow(path);
while (this->isRunning) {
while (!this->stackFrames.empty()) {
cv:Mat m = this->stackFrames.pop();
videoWriter << m;
}
}
videoWriter.release();
This code is running in an additional thread and will be stopped from outside.
The code is working so far, but it is sometimes pretty slow, which means my stack size increases and my system runs out of ram and get killed by the OS.
Currently it is running on my developing system:
Ubuntu 18.04.05
OpenCV 4.4.0 compiled with Cuda
Intel i7 10. generation 32GB RAM, GPU Nvidia p620, M.2 SSD
Depending on the codec (fourcc) this produces a high CPU load. So far I used mainly "MJPG", "x264". Sometimes even MJPG turns one core of the CPU to 100% load, and my stack raises until the programs run out of run. After a restart, sometimes, this problem is fixed, and it seems the load is distributed over all cores.
Regarding to the Intel Doc. for my CPU, it has integrated hardware encoding/decoding for several codecs. But I guess opencv is not using them. Opencv even uses its own ffmpeg and not the one of my system. Here is my build command of opencv:
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_TBB=ON \
-D WITH_CUDA=ON \
-D BUILD_opencv_cudacodec=OFF \
-D ENABLE_FAST_MATH=1 \
-D CUDA_FAST_MATH=1 \
-D WITH_CUBLAS=1 \
-D WITH_V4L=ON \
-D WITH_QT=OFF \
-D WITH_OPENGL=ON \
-D WITH_GSTREAMER=ON \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D OPENCV_ENABLE_NONFREE=ON \
-D WITH_FFMPEG=1 \
-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
-D WITH_CUDNN=ON \
-D OPENCV_DNN_CUDA=ON \
-D CUDA_ARCH_BIN=6.1 ..
I just started development with linux and C++, before I was working with Java/Maven, so the use of cmake is still a work in progress, pls go easy on me.
Basically my question is, how can I make the video encoding/writing faster, use the hardware acceleration at best?
Or if you think there is something else fishy, pls let me know.
BR Michael
-------- old - look up answer on bottom --------
Thank #Micka for the many advises, I found the right thing on the way.
Using cudacodec::VideoWriter is not that easy, after compiling I was not able to use it because of this error, and even if I can make it run, the deployment PC does not have a nvidia GPU.
Since I am going to use PCs with AMD CPUs as well, I can't use the cv::CAP_INTEL_MFX for the api-reference parameter of the cv::VideoWriter.
But there is also the cv::CAP_OPENCV_MJPEG, which works fine for the MJPG codec (not all video container are supported, I use .avi, sadly .mkv was not working with this configuration). If the user does not use MJPG as a codec I use cv::CAP_ANY, and opencv decides what is to use.
So,
cv::VideoWriter videoWriter(path, cv::CAP_OPENCV_MJPEG, fourcc, fps, *size);
works pretty well, even on my old system.
Unfortunately I never changed the api-reference parameter before, only from ffmpeg to gstreamer, I read in the doc of opencv only the last line "cv::CAP_FFMPEG or cv::CAP_GSTREAMER." and I did not see that there is an "e.g." before...
Thank you #Micka to make me read again.
P.S. for my performance problem with cv::imshow I changed from
cv::namedWindow(WINDOW_NAME, cv::WINDOW_NORMAL);
to
cv::namedWindow(WINDOW_NAME, cv::WINDOW_OPENGL);
Which obviously uses OpenGL, and does a better job. Also changing from cv::Mat to cv::UMat can speed up the performance, see here
-------------- EDIT better solution ----------------
Since I still had problems with the OpenCV VideoWriter for some systems, I was looking for another solution. Now I write the frames with FFMPEG.
For FFMPEG I can use the GPU or CPU depending on the codec I use.
If FFMPEG is installed via snapd (Ubuntu 18.04) it comes with cuda enabled by default:
sudo snap install ffmpeg --devmode
(--devmode is optional, but I had problems writing files on specific location, this was the only way for me to fix it)
And here is my code:
//this string is automatically created in my program, depending on user input and the parameters of the input frames
string ffmpegCommand = "ffmpeg -y -f rawvideo -vcodec rawvideo -framerate 50 -pix_fmt bgr24 -s 1920x1080 -i - -c:v h264_nvenc -crf 14 -maxrate:v 10M -r 50 myVideoFile.mkv";
FILE *pipeout = popen(ffmpegCommand.data(), "w");
int id = metaDataWriter.insertNow(path);
//loop will be stopped from another thread
while (this->isRunning) {
//this->frames is a stack with cv::Mat elements in the right order
//it is filled by another thread
while (!this->frames.empty()) {
cv::Mat mat = frames.front();
frames.pop();
fwrite(mat.data, 1, s, pipeout);
}
}
fflush(pipeout);
pclose(pipeout);
So a file (pipeout) is used to write the mat.data to ffmpeg, ffmpeg itself is doing the encoding and file writing. To the parameters:
-y = Overwrite output files without asking
-f = format, in this case used for input rawvideo
-vcodec = codec for input which is rawvideo as well, because the used cv::Mat.data has no compression/codec
-framerate = the input framerate I receive from my grabber card/OpenCv
-pix_fmt = the format of my raw data, in this case bgr24, so 8 bit each channel, because I use a regular OpenCV bgr cv::Mat
-s = size of each frame, in my case 1920x1080
-i = input, in this case we read from the stdinput you can see it here "-", so the file (pipeout) is captured by ffmpeg
-c:v = output codec, so this is to encode the video, here h264_nvenc is used, which is a GPU codec
-r = frame output rate, also 50 in this case myVideoFile.mkv = this is just the name of the file which is produced by ffmpeg, you can change this file and path
Additional parameters for higher quality: -crf 14 -maxrate:v 10M
This works very good for me and uses my hardware acceleration of the GPU or with another codec in charge the CPU.
I hope this helps other developers as well.
Working hard for 4 days now to fix the google cloud speech to text api to work, but still see no light at the end of the tunnel. Searched on the net a lot, read the documentations a lot but see no result.
Our site is bbsradio.com, we are trying to auto extract transcript from our mp3 files using google speech-to-text api. Code is written on PHP and almost exact copy of this: https://github.com/GoogleCloudPlatform/php-docs-samples/blob/master/speech/src/transcribe_async.php
I see process is completed and its reached out here "$operation->pollUntilComplete();" but its not showing it was successful at "if ($operation->operationSucceeded()) {" and its not returning any error either at $operation->getError().
I am converting the mp3 to raw file like this: ffmpeg -y -loglevel panic -i /public_html/sites/default/files/show-archives/audio-clips-9-23-2020/911freefall2020-05-24.mp3 -f s16le -acodec pcm_s16le -vn -ac 1 -ar 16000 -map_metadata -1 /home/mp3_to_raw/911freefall2020-05-24.raw
While tried with FLAC format as well, not worked. I tested converted FLAC file using windows media player, I can listen conversation clearly. I checked the files its Hz 16000, channel = 1 and its 16 bit. I see file is uploaded in cloud storage. Checked this:
https://cloud.google.com/speech-to-text/docs/troubleshooting and
https://cloud.google.com/speech-to-text/docs/best-practices
There are lot of discussion and documentation, seems nothing is helpful at this moment. If some one can really help me out to find out the issue, it will be really really really great!
TLDR; convert from MP3 to a 1-channel FLAC file with the same sample rate as your MP3 file.
Long explanation:
Since you're using MP3 files as your process input, probably you MP3 compression artifacts might be hurting you when you resample to to 16KHz (you cannot hear this, but the algoritm will).
To confirm this theory:
Execute ffprobe -hide_banner filename.mp3 it will output something like this:
Metadata:
...
Duration: 00:02:12.21, start: 0.025057, bitrate: 320 kb/s
Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 320 kb/s
Metadata:
encoder : LAME3.99r
In this case, the sample rate is OK for Google-Spech-Api. Just transcode the file without changing the sample rate (remove the -ar 16000 from your ffmpeg command)
You might get into trouble if the original MP3 bitrate is low. 320kb/s seems safe (unless the recording has a lot of noise).
Take into account that voice recoded under 64kb/s (ISDN line quality) can be understood only by humans if there is some noise.
At last I found the solution and reason of the issue. Actually getting empty results is a bug of the php api code. What you need to do:
Replace this:
$operation->pollUntilComplete();
by this:
while(!$operation->isDone()){
$operation->pollUntilComplete();
}
Read this: enter link description here
I use the library of ffmpeg to decode stream from [TTQ HD Camera] and encode it to a rtmp stream.
but I receive a lot of warnings like the picture below.
i try to set qmin and qmax , it seems a little better. but still not totally resolve the problem.
encoder_context->qmin = 10;
encoder_context->qmax = 51;
who knows this is why ?
[dshow # 04bfc640] real-time buffer [TTQ HD Camera] [video input] too full or near too full (101% of size: 3041280 [rtbufsize parameter])! frame dropped!
Have you tried increasing the -rtbufsize parameter to something larger than 3041280? If you have the RAM for it, try something like 2000M. It should be defined before the -i of the camera.
So something like:
ffmpeg -f dshow -video_size 1920x1080 -rtbufsize 2147.48M -framerate 30 -pixel_format bgr0 -i video=...
Note that the resolution and frame rate are just examples and you would have to fill in your values that you have used in the ffmpeg command already.
I want to receive images from an IP camera over HTTP via GET request. I have written a program that creates TCP socket connection with the camera and sends the following GET request to the camera:
GET /mjpeg?res=full HTTP/1.1\r\nHost: 143.205.116.14\r\n\r\n
After that, I receive images with the following function in a while loop:
while((tmpres = recv(sock,(void *) buf, SIZE, 0)) > 0 && check<10)
.....
where SIZE represents the size of the buffer. I, infact, don't know what size to define here. I am receiving a color image of size 2940x1920. So I define SIZE=2940x1920x3. After receiving the MJPEG image, I decode it with ffmpeg. But I observe that ffmpeg just partially/uncorrectly decodes the image and I just see a half (or even less) of the image. I assume it could be a size problem.
Any help in this regard would be highly appreciated.
Regards,
Khan
Why reinvent the wheel (for the umpteenth time)? Use a ready-made HTTP client library, such as libcurl.
For that matter, perhaps you can even just write your entire solution as a shell script using the curl command line program:
#!/bin/sh
curl -O "http://143.205.116.14/mjpeg?res=full" || echo "Error."
curl -o myfile.jpg "http://143.205.116.14/mjpeg?res=full" || echo "Error."
# ffmpeg ...
Save bytes received as binary file and analyze. May be an incomplete image (image can be encoded as progressive JPEG - is interlaced in fact - that means if you truncate the file you'll see horizontal lines.) or can be a ffmpeg decoding issue. Or something different. What is check < 10 condition ?