OpenCv read / write video color difference - c++

Question

I am trying to simply open a video with openCV, process frames and write the processed frames into a new video file.
My problem is that even if I don't process frames at all (just opening a video, reading frames with VideoCapture and writing them with VideoWriter to a new file), the output file appears more "green" than the input.
The code to do that can be found in any openCV tutorial, nothing special.
I use openCV c++ 4.4.0 on Windows 10.
I use openCV with ffmpeg through opencv_videoio_ffmpeg440_64.dll
The input video is mp4.
I write the output as a .avi with huffyuv codec :
m_video_writer.reset(new cv::VideoWriter(m_save_video_path.toStdString(), cv::VideoWriter::fourcc('H', 'F', 'Y', 'U'), // lossless compression
m_model->getFps(), cv::Size(m_frame_size.width(), m_frame_size.height())));
I tried many other codecs and the problem remains.
The difference in pixels is small, not constant in value but always varying in the same way : blue channel is lower, red and green are higher.
Strange fact : when I open both input or output video with opencv, the matrix are actually exactly the same. So I guess the problem is in the reading ??
Here are the properties of each video file, as exported with Windows Media Playre (MPC-HC).
VS
What should I investigate ?
Thx !!
Full code here (copying the first 100 frames of my video):
VideoCapture original("C:/Users/axelle/Videos/original.MP4");
int frame_height = original.get(CAP_PROP_FRAME_HEIGHT);
int frame_width = original.get(CAP_PROP_FRAME_WIDTH);
int fps = original.get(CAP_PROP_FPS);
VideoWriter output("C:/Users/axelle/Videos/output.avi", VideoWriter::fourcc('H', 'F', 'Y', 'U'),
fps, cv::Size(frame_width, frame_height));
int count = 0;
while (count < 100)
{
count++;
Mat frame;
original >> frame;
if (frame.empty())
{
break;
}
//imshow("test", frame);
//waitKey(0);
output.write(frame);
}
original.release();
output.release();
Note: the difference in colors can be seen in the imshow already.

Answer 1

There is a bug in OpenCV VideoCapture when reading video frames using FFmpeg backend.
The bug results a "color shift" when H.264 video stream is marked as BT.709 color standard.
The subject is too important to leave it unanswered...
The important part of the post, is reproducing the problem, and proving the problem is real.
The solution I found is selecting GStreamer backend instead of FFmpeg backend.
The suggested solution has downsides (like the need to build OpenCV with GStreamer support).
Note:
The problem is reproducible using OpenCV 4.53 under Windows 10.
The problem is also reproducible under Ubuntu 18.04 (using OpenCV in Python).
The issue applies both "full range" and "limited range" of BT.709 color standard.
Building synthetic video pattern for reproducing the problem:
We can use FFmpeg command line tool create a synthetic video to be used as input.
The following command generates an MP4 video file with H.264 codec, and BT.709 color standard:
ffmpeg -y -f lavfi -src_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -i testsrc=size=192x108:rate=1:duration=5 -vcodec libx264 -crf 17 -pix_fmt yuv444p -dst_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 bt709_full_range.mp4
The above command uses yuv444p pixel format (instead of yuv420p) for getting more pure colors.
The arguments -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 use Bitstream Filter for marking the H.264 stream as "full range" BT.709.
Using MediaInfo tool, we can view the following color characteristics:
colour_range: Full
colour_primaries: BT.709
transfer_characteristics: BT.709
matrix_coefficients: BT.709
Capturing the video using OpenCV:
The following C++ code grabs the first frame, and save it to 1.png image file:
#include "opencv2/opencv.hpp"
void main()
{
cv::VideoCapture cap("bt709_full_range.mp4");
cv::Mat frame;
cap >> frame;
cv::imwrite("1.png", frame);
cap.release();
}
We may also use the following Python code:
import cv2
cap = cv2.VideoCapture('bt709_full_range.mp4')
_, frame = cap.read()
cv2.imwrite('1.png', frame)
cap.release()
Converting bt709_full_range.mp4 into images sequence using FFmpeg:
ffmpeg -i bt709_full_range.mp4 -pix_fmt rgb24 %03d.png
The file name of the first "extracted" frame is 001.png.
Comparing the results:
The left side is 1.png (result of OpenCV)
The right side is 001.png (result of FFmpeg command line tool)
As you can see, the colors are different.
The value of the red color pixels of OpenCV are RGB = [232, 0, 3].
The value of the red color pixels of FFmpeg are RGB = [254, 0, 0].
The original RGB value is probably [255, 0, 0] (value is 254 due to colors conversion).
As you can see, the OpenCV colors are wrong!
Solution - selecting GStreamer backend instead of FFmpeg backend:
The default OpenCV release excludes GStreamer support (at least in Windows).
You may use the following instruction for building OpenCV with GStreamer.
Here is a C++ code sample that uses GStreamer backend for grabbing the first frame:
void main()
{
cv::VideoCapture cap("filesrc location=bt709_full_range.mp4 ! decodebin ! videoconvert ! appsink", cv::CAP_GSTREAMER);
cv::Mat frame;
cap >> frame;
cv::imwrite("1g.png", frame);
cap.release();
}
Result:
The left side is 1g.png (result of OpenCV using GStreamer)
The right side is 001.png (result of FFmpeg command line tool)
The value of the red color pixels of OpenCV using GStreamer are RGB = [254, 0, 1]. (blue is 1 and not zero due to colors conversion).
Conclusions:
Using GStreamer backend (instead of FFmpeg) backend seems to solve the "color shifting" problem.
OpenCV users need to be aware of the color shifting problem.
Let's hope that OpenCV developers (or FFmpeg plugin developers) fix the problem.