I have created an FFMPEG pipeline to stream video frames to an RTSP server. I initiate my pipeline as follows:
FILE* openPipeLine = _popen("ffmpeg -f rawvideo -r 10 -video_size 1280x720 -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 25 -pix_fmt yuv420p -f rtsp rtsp://localHost:8554/mystream", "wb");
I then have a loop where I process the frames and end up with a Mat variable that contains the 720x1280 processed image. I then use the fwrite function to write the frame to the server as shown below.
string filename = "file.mp4";
VideoCapture capture(filename);
Mat frame;
for( ; ; )
{
capture >> frame;
if(frame.empty())
break;
fwrite(frame.data, 1, 1280 * 720 * 3, openPipeLine);
}
Everything runs perfectly but the fwrite function takes approximately 0.1 seconds to run. I need it to be far more efficient. Is there a way to store the frames and batch write a group of frames instead of calling fwrite on each iteration? perhaps I can change the number 1 in the second input to fwrite?
Related
I am trying to simply open a video with openCV, process frames and write the processed frames into a new video file.
My problem is that even if I don't process frames at all (just opening a video, reading frames with VideoCapture and writing them with VideoWriter to a new file), the output file appears more "green" than the input.
The code to do that can be found in any openCV tutorial, nothing special.
I use openCV c++ 4.4.0 on Windows 10.
I use openCV with ffmpeg through opencv_videoio_ffmpeg440_64.dll
The input video is mp4.
I write the output as a .avi with huffyuv codec :
m_video_writer.reset(new cv::VideoWriter(m_save_video_path.toStdString(), cv::VideoWriter::fourcc('H', 'F', 'Y', 'U'), // lossless compression
m_model->getFps(), cv::Size(m_frame_size.width(), m_frame_size.height())));
I tried many other codecs and the problem remains.
The difference in pixels is small, not constant in value but always varying in the same way : blue channel is lower, red and green are higher.
Strange fact : when I open both input or output video with opencv, the matrix are actually exactly the same. So I guess the problem is in the reading ??
Here are the properties of each video file, as exported with Windows Media Playre (MPC-HC).
VS
What should I investigate ?
Thx !!
Full code here (copying the first 100 frames of my video):
VideoCapture original("C:/Users/axelle/Videos/original.MP4");
int frame_height = original.get(CAP_PROP_FRAME_HEIGHT);
int frame_width = original.get(CAP_PROP_FRAME_WIDTH);
int fps = original.get(CAP_PROP_FPS);
VideoWriter output("C:/Users/axelle/Videos/output.avi", VideoWriter::fourcc('H', 'F', 'Y', 'U'),
fps, cv::Size(frame_width, frame_height));
int count = 0;
while (count < 100)
{
count++;
Mat frame;
original >> frame;
if (frame.empty())
{
break;
}
//imshow("test", frame);
//waitKey(0);
output.write(frame);
}
original.release();
output.release();
Note: the difference in colors can be seen in the imshow already.
There is a bug in OpenCV VideoCapture when reading video frames using FFmpeg backend.
The bug results a "color shift" when H.264 video stream is marked as BT.709 color standard.
The subject is too important to leave it unanswered...
The important part of the post, is reproducing the problem, and proving the problem is real.
The solution I found is selecting GStreamer backend instead of FFmpeg backend.
The suggested solution has downsides (like the need to build OpenCV with GStreamer support).
Note:
The problem is reproducible using OpenCV 4.53 under Windows 10.
The problem is also reproducible under Ubuntu 18.04 (using OpenCV in Python).
The issue applies both "full range" and "limited range" of BT.709 color standard.
Building synthetic video pattern for reproducing the problem:
We can use FFmpeg command line tool create a synthetic video to be used as input.
The following command generates an MP4 video file with H.264 codec, and BT.709 color standard:
ffmpeg -y -f lavfi -src_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -i testsrc=size=192x108:rate=1:duration=5 -vcodec libx264 -crf 17 -pix_fmt yuv444p -dst_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 bt709_full_range.mp4
The above command uses yuv444p pixel format (instead of yuv420p) for getting more pure colors.
The arguments -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 use Bitstream Filter for marking the H.264 stream as "full range" BT.709.
Using MediaInfo tool, we can view the following color characteristics:
colour_range: Full
colour_primaries: BT.709
transfer_characteristics: BT.709
matrix_coefficients: BT.709
Capturing the video using OpenCV:
The following C++ code grabs the first frame, and save it to 1.png image file:
#include "opencv2/opencv.hpp"
void main()
{
cv::VideoCapture cap("bt709_full_range.mp4");
cv::Mat frame;
cap >> frame;
cv::imwrite("1.png", frame);
cap.release();
}
We may also use the following Python code:
import cv2
cap = cv2.VideoCapture('bt709_full_range.mp4')
_, frame = cap.read()
cv2.imwrite('1.png', frame)
cap.release()
Converting bt709_full_range.mp4 into images sequence using FFmpeg:
ffmpeg -i bt709_full_range.mp4 -pix_fmt rgb24 %03d.png
The file name of the first "extracted" frame is 001.png.
Comparing the results:
The left side is 1.png (result of OpenCV)
The right side is 001.png (result of FFmpeg command line tool)
As you can see, the colors are different.
The value of the red color pixels of OpenCV are RGB = [232, 0, 3].
The value of the red color pixels of FFmpeg are RGB = [254, 0, 0].
The original RGB value is probably [255, 0, 0] (value is 254 due to colors conversion).
As you can see, the OpenCV colors are wrong!
Solution - selecting GStreamer backend instead of FFmpeg backend:
The default OpenCV release excludes GStreamer support (at least in Windows).
You may use the following instruction for building OpenCV with GStreamer.
Here is a C++ code sample that uses GStreamer backend for grabbing the first frame:
void main()
{
cv::VideoCapture cap("filesrc location=bt709_full_range.mp4 ! decodebin ! videoconvert ! appsink", cv::CAP_GSTREAMER);
cv::Mat frame;
cap >> frame;
cv::imwrite("1g.png", frame);
cap.release();
}
Result:
The left side is 1g.png (result of OpenCV using GStreamer)
The right side is 001.png (result of FFmpeg command line tool)
The value of the red color pixels of OpenCV using GStreamer are RGB = [254, 0, 1]. (blue is 1 and not zero due to colors conversion).
Conclusions:
Using GStreamer backend (instead of FFmpeg) backend seems to solve the "color shifting" problem.
OpenCV users need to be aware of the color shifting problem.
Let's hope that OpenCV developers (or FFmpeg plugin developers) fix the problem.
I'm trying read a .bmp grayscale image from a file with a given width and height, convert it to std::vector<uint8_t>, run some sort of filter function on that vector, and then create a new image from that std::vector. I'm stuck in the last part.
How do I create a .bmp file from a given std::vector<uint8_t>, height and width?
P.S. I'm trying to do this without using external libraries.
This is the code I have thus far:
class Image {
int weight;
int width;
std::vector<uint8_t> image;
Image(int weight,int width) : weight(weight),width(width);
void read_image(char* pic);
void save_image(const std::vector<uint8_t>& new_image);
std::vector<uint8_t> filter_image(int ww,int wh,double filter);
};
void Image::read_image(char *pic) {
std::ifstream file(pic,std::ios_base::in | std::ios_base::binary);
if(!file.is_open()) return;
while(file.peek()!=EOF){
this->image.push_back(static_cast<uint8_t>(file.get()));
}
}
void Image::save_image(const std::vector<uint8_t> &new_image) {
//what to do here?
}
A .bmp file does not only store raw pixel data. It begins with a header describing the image stored inside the file: width, height, pixel size, color type, etc... The read_image() function you wrote reads the whole file, including the header, and running any image processing algorithm on your vector will ruin your data and produce garbage.
If you are learning image processing, it would be far easier to use raw image files. A raw image file contains only pixel data, without any metadata. When working with a raw image file, it is your responsibility to know the width and height of the image, as well as the pixel encoding.
Converting an image file to a raw image file, and vice versa, involves the use of an external tool. ffmpeg is such a tool. ffmpeg is a linux tool, but it should be easy to find ffmpeg packaged for any operating system.
For converting from a file in almost any format to a raw image file (ffmpeg deduces the size of the image from the input file). The order of each parameter is important:
ffmpeg -i your_file.jpeg -f rawvideo -c rawvideo -pix_fmt gray output.raw
When converting back to your input format, you have to explicitly tell ffmpeg the size of your picture. Again the order of each parameter is important:
ffmpeg -f rawvideo -c rawvideo -pix_fmt gray -s 1280x720 -i input.raw your_processed_file.jpeg
Adapt the width and height to the real size of your image, or ffmpeg will resize the image. you can also play with the pixel type: gray specifies an 8 bits per pixel grayscale format, but you can use rgb24 to keep color information (use ffmpeg -pix_fmts to see a list of all available formats).
If you are lucky enough to have ffplay availabel in your ffmpeg package, you can view the raw file directly on screen:
ffplay -f rawvideo -c rawvideo -pix_fmt gray -s 1280x720 input.raw
Additionally some image processing software are able to open a raw image file: gimp, photoshop, ...
I followed this to encode a sequences images to h.264 video.
Here is outputting part of my code:
int srcstride = outwidth*4;
sws_scale(convertCtx, src_data, &srcstride, 0, outheight, pic_in.img.plane, pic_in.img.i_stride);
x264_nal_t* nals;
int i_nals;
int frame_size = x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
if (frame_size) {
fwrite(nals[0].p_payload, frame_size, 1, fp);
}
This is in a loop to process frames and write them into a file.
Now, I'm trying to stream these encoded frames through RTMP. As I know, the container for the RTMP is FLV. So I used command line as a trial:
ffmpeg -i test.h264 -vcodec copy -f flv rtmp://localhost:1935/hls/test
This one works well as streaming a h.264 encoded video file.
But how can I implement it as C++ code and stream the frames at the same time when they are generated, just like what I did to stream my Facetime camera.
ffmpeg -f avfoundation -pix_fmt uyvy422 -video_size 1280x720 -framerate 30 -i "1:0" -pix_fmt yuv420p -vcodec libx264 -preset veryfast -acodec libvo_aacenc -f flv -framerate 30 rtmp://localhost:1935/hls/test
This may be a common and practical topic. But I'm stuck here for days, really need some relevant exprience. Thank you!
I had the commands for exporting a video stream to an mpeg file working correctly with the following code:
ffmpeg -r 24 -pix_fmt rgba -s 1280x720 -f rawvideo -y -i -vf vflip -vcodec mpeg1video -qscale 4 -bufsize 500KB -maxrate 5000KB OUTPUT_FILE
Now, I wanted to add the commands so that audio can be used as well since there's no option for that right now.
I've edited the previous command to the next one:
ffmpeg -r 24 -pix_fmt rgba -s 1280x720 -f rawvideo -y -i -f s16le -ac 1 -ar 44100 -i - -acodec pcm_s16le -ac 1 -b:a 320k -ar 44100 -vf vflip -vcodec mpeg1video -qscale 4 -bufsize 500KB -maxrate 5000KB OUTPUT_FILE
So as you can see I added a new input with the settings for the audio I'm going to be inputting (I'm going to test this with the values of a sine wave).
I'm writing the data to the file like this:
// Write a frame to the ffmpeg stream
fwrite(frame, sizeof(unsigned char*) * frameWidth * frameHeight, 1, ffmpeg);
// Write multiple sound samples per written frame
for (int t = 0; t < 44100/24; ++t)
fwrite(&audio, sizeof(short int), 1, ffmpeg);
The first line is the one that only writes the video (where the frame object is a render to texture from the video I'm inputting)
After that I'm trying to add the audio. I'm using a for-loop so I can use multiple samples per video frame (because otherwise you would only have 24 audio samples per second)
This does render with a couple of issues:
The rendered video shows green flashes
The video slides across the screen. For example, if it slides 200 pixels to the right those pixels get rendered at the other side. Also a bit of the frame that should be at the bottom is rendered at the top (so the frame also slides down but this is a constant, it doesn't move over time)
I can't figure out where my mistake is. I've tried multiple codecs and tried different orders for the commands but it stays the same or gets worse.
Thanks in advance
How to convert set of YUV frames to a Video and later convert the video to YUV frames without any loss using C ?(I dont want to convert it to RGB in between)
if you have a raw YUV file, you need to tell ffmpeg which pixel format/subsampling that is used. YUV have no header, so you also need to specify the width and height of the data.
The following ffmpeg commands encodes a 1080p YUV 4:2:0 to H.264 using the x264 encoder and place the result in a mp4-container. This operation is however not losless.
ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r 25 -i input.yuv \
-c:v libx264 output.mp4
To get the YUV-frames back again, do
ffmpeg -i output.mp4 frames.yuv
If you are looking for a lossless encoder, try HuffYUV or FFV1