I want to read and show a video using opencv. I've recorded with Direct-show, the Video has UYVY (4:2:2) codec, since opencv can't read that format, I want to convert the codec to an RGB color model, I readed about ffmpeg and I want to know if it's possible to get this done with it ? if not if you a suggestion I'll be thankful.
As I explained to you before, OpenCV can read some formats of YUV, including UYVY (thanks to FFmpeg/GStreamer). So I believe the cv::Mat you get from the camera is already converted to the BGR color space which is what OpenCV uses by default.
I modified my previous program to store the first frame of the video as PNG:
cv::Mat frame;
if (!cap.read(frame))
{
return -1;
}
cv::imwrite("mat.png", frame);
for(;;)
{
// ...
And the image is perfect. Executing the command file on mat.png reveals:
mat.png: PNG image data, 1920 x 1080, 8-bit/color RGB, non-interlaced
A more accurate test would be to dump the entire frame.data() to the disk and open it with an image editor. If you do that keep in mind that the R and B channels will be switched.
Related
I am trying to simply open a video with openCV, process frames and write the processed frames into a new video file.
My problem is that even if I don't process frames at all (just opening a video, reading frames with VideoCapture and writing them with VideoWriter to a new file), the output file appears more "green" than the input.
The code to do that can be found in any openCV tutorial, nothing special.
I use openCV c++ 4.4.0 on Windows 10.
I use openCV with ffmpeg through opencv_videoio_ffmpeg440_64.dll
The input video is mp4.
I write the output as a .avi with huffyuv codec :
m_video_writer.reset(new cv::VideoWriter(m_save_video_path.toStdString(), cv::VideoWriter::fourcc('H', 'F', 'Y', 'U'), // lossless compression
m_model->getFps(), cv::Size(m_frame_size.width(), m_frame_size.height())));
I tried many other codecs and the problem remains.
The difference in pixels is small, not constant in value but always varying in the same way : blue channel is lower, red and green are higher.
Strange fact : when I open both input or output video with opencv, the matrix are actually exactly the same. So I guess the problem is in the reading ??
Here are the properties of each video file, as exported with Windows Media Playre (MPC-HC).
VS
What should I investigate ?
Thx !!
Full code here (copying the first 100 frames of my video):
VideoCapture original("C:/Users/axelle/Videos/original.MP4");
int frame_height = original.get(CAP_PROP_FRAME_HEIGHT);
int frame_width = original.get(CAP_PROP_FRAME_WIDTH);
int fps = original.get(CAP_PROP_FPS);
VideoWriter output("C:/Users/axelle/Videos/output.avi", VideoWriter::fourcc('H', 'F', 'Y', 'U'),
fps, cv::Size(frame_width, frame_height));
int count = 0;
while (count < 100)
{
count++;
Mat frame;
original >> frame;
if (frame.empty())
{
break;
}
//imshow("test", frame);
//waitKey(0);
output.write(frame);
}
original.release();
output.release();
Note: the difference in colors can be seen in the imshow already.
There is a bug in OpenCV VideoCapture when reading video frames using FFmpeg backend.
The bug results a "color shift" when H.264 video stream is marked as BT.709 color standard.
The subject is too important to leave it unanswered...
The important part of the post, is reproducing the problem, and proving the problem is real.
The solution I found is selecting GStreamer backend instead of FFmpeg backend.
The suggested solution has downsides (like the need to build OpenCV with GStreamer support).
Note:
The problem is reproducible using OpenCV 4.53 under Windows 10.
The problem is also reproducible under Ubuntu 18.04 (using OpenCV in Python).
The issue applies both "full range" and "limited range" of BT.709 color standard.
Building synthetic video pattern for reproducing the problem:
We can use FFmpeg command line tool create a synthetic video to be used as input.
The following command generates an MP4 video file with H.264 codec, and BT.709 color standard:
ffmpeg -y -f lavfi -src_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -i testsrc=size=192x108:rate=1:duration=5 -vcodec libx264 -crf 17 -pix_fmt yuv444p -dst_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 bt709_full_range.mp4
The above command uses yuv444p pixel format (instead of yuv420p) for getting more pure colors.
The arguments -bsf:v h264_metadata=video_full_range_flag=1:colour_primaries=1:transfer_characteristics=1:matrix_coefficients=1 use Bitstream Filter for marking the H.264 stream as "full range" BT.709.
Using MediaInfo tool, we can view the following color characteristics:
colour_range: Full
colour_primaries: BT.709
transfer_characteristics: BT.709
matrix_coefficients: BT.709
Capturing the video using OpenCV:
The following C++ code grabs the first frame, and save it to 1.png image file:
#include "opencv2/opencv.hpp"
void main()
{
cv::VideoCapture cap("bt709_full_range.mp4");
cv::Mat frame;
cap >> frame;
cv::imwrite("1.png", frame);
cap.release();
}
We may also use the following Python code:
import cv2
cap = cv2.VideoCapture('bt709_full_range.mp4')
_, frame = cap.read()
cv2.imwrite('1.png', frame)
cap.release()
Converting bt709_full_range.mp4 into images sequence using FFmpeg:
ffmpeg -i bt709_full_range.mp4 -pix_fmt rgb24 %03d.png
The file name of the first "extracted" frame is 001.png.
Comparing the results:
The left side is 1.png (result of OpenCV)
The right side is 001.png (result of FFmpeg command line tool)
As you can see, the colors are different.
The value of the red color pixels of OpenCV are RGB = [232, 0, 3].
The value of the red color pixels of FFmpeg are RGB = [254, 0, 0].
The original RGB value is probably [255, 0, 0] (value is 254 due to colors conversion).
As you can see, the OpenCV colors are wrong!
Solution - selecting GStreamer backend instead of FFmpeg backend:
The default OpenCV release excludes GStreamer support (at least in Windows).
You may use the following instruction for building OpenCV with GStreamer.
Here is a C++ code sample that uses GStreamer backend for grabbing the first frame:
void main()
{
cv::VideoCapture cap("filesrc location=bt709_full_range.mp4 ! decodebin ! videoconvert ! appsink", cv::CAP_GSTREAMER);
cv::Mat frame;
cap >> frame;
cv::imwrite("1g.png", frame);
cap.release();
}
Result:
The left side is 1g.png (result of OpenCV using GStreamer)
The right side is 001.png (result of FFmpeg command line tool)
The value of the red color pixels of OpenCV using GStreamer are RGB = [254, 0, 1]. (blue is 1 and not zero due to colors conversion).
Conclusions:
Using GStreamer backend (instead of FFmpeg) backend seems to solve the "color shifting" problem.
OpenCV users need to be aware of the color shifting problem.
Let's hope that OpenCV developers (or FFmpeg plugin developers) fix the problem.
I am trying to process video frames from a Qt application. The input from the QML Camera is of format YUYV and I could not set it to something else like YUV420 by default. I need to capture the frame and create a Mat object of YUV420 or grayscale format.
I have tried cvtColor with the following codes and they all crashed during runtime -
COLOR_YUV2GRAY_YVY
COLOR_YUV2RGB_YUYV
COLOR_YUV2GRAY_YUYV
Any idea how I can do this?
So after playing around with the color conversion codes a bit this is the solution I found to be working for me.
cv::Mat img = cv::Mat(m_videoHeight, m_videoWidth, CV_8UC2, input->bits());
cv::Mat gray;
cv::cvtColor(img, gray, cv::COLOR_YUV2GRAY_YVYU);
This converts the given YUYV Mat object - img to a grayscale Mat object - gray. input->bits() refers to the first bit of the video frame buffer.
I'm trying to capture the raw data of the logitech pro 9000 (eg. the so called Bayer pattern). This can be achieved by using the so called bayer application, that can be found floating over the internet. It should return a 8 bit bayer pattern, but the results are quite obviously not such a pattern.
However; The image that is being streamed seems to be quite off. As can be seen in the image below, I get 2 images of the scene in a 3 channel image (meaning 6 channels in total). Each image is 1/4th of the total capture area, so it would seem that there is some kind of YUV data being streamed.
I was unable to convert this data into anything meaningful using the conversions provided by openCV. Any ideas what kind of data is being sent and (more importantly) how to convert this into RGB?
EDIT
As requested; the codesnippet that is used to generate the image.
system("Bayer.exe 1 8"); //Sets the camera to raw mode
// set up camera
VideoCapture capture(0);
if(!capture.isOpened()){
waitKey();
exit(0);
}
Mat capturedFrame;
while(true){
capture>>capturedFrame;
imshow("Raw",capturedFrame);
waitKey(25);
}
How did you get frames from stream using openCV? Can you share some code snippets? There are too many video formats in openCV for getting correct color channel and compressed data.
I think you should be able to obtain correct image frames as mentioned here :
http://forum.openrobotino.org/archive/index.php/t-295.html?s=c33acb1fb91f5916080f8dfd687598ec
This is most likely to happen if the out put data format ( width, height, bit depth, no of channels...) of camera and the data format your program expect is different.
However i could capture of logitec pro cam, simply by using
Mat img;
VideoCapture cap(0);
cap >> img;
I have an unsigned char* buffer containing data of a jpeg image. I would like to display that image using c++ and opencv. If i do:
Mat img(Size(640, 480), CV_8UC3, data);
namedWindow("image", 1);
imShow("image", img);
I get a noisy mess of pixels.
I suppose it's because the data is jpeg (with a header). Because this works:
Mat imgbuf(Size(640, 480), CV_8UC3, data);
Mat img = imdecode(imgbuf, CV_LOAD_IMAGE_COLOR);
BUT I cannot use the imdecode function as it is from highgui.h which is based upon GTK 2, and in my project I use GTK 3.
So, how can I display the buffer data? Is there a way to decode the jpeg image other than imdecode in opencv, if that's the problem. I don't really want to have to rebuild opencv with Qt...
Any other suggestions?
(Using Linux)
I have seen many responses to this question around on the net saying that you should call libjpeg directly and bypass OpenCV's imread() routine.
This is NOT necessary! You can use imdecode() to decode a raw image buffer from memory. The way to do it is NOT intuitive, and isn't documented enough to help people trying to do this for the first time.
If you have a pointer/size for your raw file data (fread() directly from the .jpg, .png, .tif, files, etc...
int nSize = ... // Size of buffer
uchar* pcBuffer = ... // Raw buffer data
// Create a Size(1, nSize) Mat object of 8-bit, single-byte elements
Mat rawData( 1, nSize, CV_8UC1, (void*)pcBuffer );
Mat decodedImage = imdecode( rawData /*, flags */ );
if ( decodedImage.data == NULL )
{
// Error reading raw image data
}
That's IT!
Hope this helps someone in the future.
I have decompressed the JPEG image using libjpeg using the standard procedure described in the libjpeg API documentation under 'Decompression details'.
After having decompressed the data you can use it to construct the cv::Mat. Mind you, the decompressed image is in RGB format, whereas openCV uses a BGR format so a cvtColor() operation with format CV_RGB2BGR is needed.
I'm trying to process each frame in a pair of video files in OpenCV and then write the resulting frames to an output avi file. Everything works, except that the output video file looks strange: instead of one solid image, the image is repeated three times and horizontally compressed so all three copies fit into the window. I suspect there is something going wrong with the number of channels the writer is expecting, but I'm giving it 8-bit single channel images to write. Below are the setting with which I'm initializing my videowriter:
//Initialize the video writer
CvVideoWriter *writer = cvCreateVideoWriter("out.avi",CV_FOURCC('D','I','V','X'), 30, frame_sizeL, 0);
Has anyone encountered this strange output from the openCV videowriter before? I've been checking the resulting frames with cvSaveImage just to see if somehow my processing step is creating the "tripled" image, but it's not. It's only when I write to the output avi with cvWriteFrame that the image gets "tripled" and compressed.
Edit: So I've discovered that this only happens when I attempt to write single channel images using write frame. If I write 3 channel 8-bit RGB images, the output video turns out fine. Why is it doing this? I am correctly passing "0" for the color argument when initializing the CvVideoWriter, so it should be expecting single channel images.
In the C++ version you have to tell cv::VideoWriter that you are sending a single channel image by setting the last paramter "false", are you sure you are doing this?
edit: alternatively you can convert a greyscale image to color using cvtColor() and CV_GRAY2RGB