Objective and problem
I'm trying to process a video file on the fly using OpenCV 3.4.1 by grabbing each frame, converting to grayscale, then doing Canny edge detection on it. In order to display the images (on the fly as well), I created a Mat class with 3 additional headers that is three times as wide as the original frame. The 3 extra headers represent the images I would like to display in the composite, and are positioned to the 1st, 2nd and 3rd horizontal segment of the composite.
After image processing however, the display of the composite image is not as expected: the first segment (where the original frame should be) is completely black, while the other segments (of processed images) are displayed fine. If, on the other hand, I display the ROIs one by one in separate windows, all the images look fine.
These are the things I tried to overcome this issue:
use .copyTo to actually copy the data into the appropriate image segments. The result was the same.
I put the Canny image to the compOrigPart ROI, and it did display in the first segment, so it is not a problem with the definition of the ROIs.
Define the composite as three channel image
In the loop convert it to grayscale
put processed images into it
convert back to BGR
put the original in.
This time around the whole composite was black, nothing showed.
As per gameon67's suggestion, I tried to create a namedWindow as well, but that doesn't help either.
Code:
int main() {
cv::VideoCapture vid("./Vid.avi");
if (!vid.isOpened()) return -1;
int frameWidth = vid.get(cv::CAP_PROP_FRAME_WIDTH);
int frameHeight = vid.get(cv::CAP_PROP_FRAME_HEIGHT);
int frameFormat = vid.get(cv::CAP_PROP_FORMAT);
cv::Scalar fontColor(250, 250, 250);
cv::Point textPos(20, 20);
cv::Mat frame;
cv::Mat compositeFrame(frameHeight, frameWidth*3, frameFormat);
cv::Mat compOrigPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(0, frameWidth));
cv::Mat compBwPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth, frameWidth*2));
cv::Mat compEdgePart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth*2, frameWidth*3));
while (vid.read(frame)) {
if (frame.empty()) break;
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
compOrigPart = frame;
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
cv::imshow("Original", compOrigPart);
cv::imshow("BW", compBwPart);
cv::imshow("Canny", compEdgePart);
cv::waitKey(33);
}
}
Questions
Why can't I display the entirety of the composite image in a single window, while displaying them separately is OK?
What is the difference between these displays? The data is obviously there, as evidenced by the separate windows.
Why only the original frame is misbehaving?
Your compBwPart and compEdgePart are grayscale images so the Mat type is CV8UC1 - single channel and therefore your compositeFrame is in grayscale too. If you want to combine these two images with a color image you have to convert it to BGR first and then fill the compOrigPart.
while (vid.read(frame)) {
if (frame.empty()) break;
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
cv::cvtColor(compositeFrame, compositeFrame, cv::COLOR_GRAY2BGR);
frame.copyTo(compositeFrame(cv::Rect(0, 0, frameWidth, frameHeight)));
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor); //the rest of your code
This is a combination of several issues.
The first problem is that you set the type of compositeFrame to the value returned by vid.get(cv::CAP_PROP_FORMAT). Unfortunately that property doesn't seem entirely reliable -- I've just had it return 0 (meaning CV_8UC1) after opening a color video, and then getting 3 channel (CV_8UC3) frames. Since you want to have the compositeFrame the same type as the input frame, this won't work.
To work around it, instead of using those properties, I'd lazy initialize compositeFrame and the 3 ROIs after receiving the first frame (based on it's dimensions and type).
The next set of problems lies in those two statements:
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
In this case assumption is made that frame is BGR (since you're trying to convert), meaning compositeFrame and its ROIs are also BGR. Unfortunately, in both cases you're writing a grayscale image into the ROI. This will cause a reallocation, and the target Mat will cease to be a ROI.
To correct this, use temporary Mats for the grayscale data, and use cvtColor to turn it back to BGR to write into the ROIs.
Similar problem lies in the following statement:
compOrigPart = frame;
That's a shallow copy, meaning it will just make compOrigPart another reference to frame (and therefore it will cease to be a ROI of compositeFrame).
What you need is a deep copy, using copyTo (note that the data types still need to match, but that was fixed earlier).
Finally, even though you try to be flexible regarding the type of the input video (judging by the vid.get(cv::CAP_PROP_FORMAT)), the rest of the code really assumes that the input is 3 channel, and will break if it isn't.
At the least, there should be some assertion to cover this expectation.
Putting this all together:
#include <opencv2/opencv.hpp>
int main()
{
cv::VideoCapture vid("./Vid.avi");
if (!vid.isOpened()) return -1;
cv::Scalar fontColor(250, 250, 250);
cv::Point textPos(20, 20);
cv::Mat frame, frame_gray, edges_gray;
cv::Mat compositeFrame;
cv::Mat compOrigPart, compBwPart, compEdgePart; // ROIs
while (vid.read(frame)) {
if (frame.empty()) break;
if (compositeFrame.empty()) {
// The rest of code assumes video to be BGR (i.e. 3 channel)
CV_Assert(frame.type() == CV_8UC3);
// Lazy initialize once we have the first frame
compositeFrame = cv::Mat(frame.rows, frame.cols * 3, frame.type());
compOrigPart = compositeFrame(cv::Range::all(), cv::Range(0, frame.cols));
compBwPart = compositeFrame(cv::Range::all(), cv::Range(frame.cols, frame.cols * 2));
compEdgePart = compositeFrame(cv::Range::all(), cv::Range(frame.cols * 2, frame.cols * 3));
}
cv::cvtColor(frame, frame_gray, cv::COLOR_BGR2GRAY);
cv::Canny(frame_gray, edges_gray, 100, 150);
// Deep copy data to the ROI
frame.copyTo(compOrigPart);
// The ROI is BGR, so we need to convert back
cv::cvtColor(frame_gray, compBwPart, cv::COLOR_GRAY2BGR);
cv::cvtColor(edges_gray, compEdgePart, cv::COLOR_GRAY2BGR);
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
cv::imshow("Original", compOrigPart);
cv::imshow("BW", compBwPart);
cv::imshow("Canny", compEdgePart);
cv::waitKey(33);
}
}
Screenshot of the composite window (using some random test video off the web):
While reading the image with IMREAD_COLOR, 'dft' function throws the error:
DFT function works just fine when reading an image with IMREAD_GRAYSCALE. But I want to read the image with IMREAD_COLOR.
main function
const char* filename = "face.jpg";
Mat I = imread(filename, IMREAD_COLOR);
if(I.empty()) return 0;
Mat padded;
I.convertTo(padded, CV_32F);
Mat fft;
Mat planes[2];
dft(padded, fft, DFT_SCALE|DFT_COMPLEX_OUTPUT);
Mat fftBlur = fft.clone();
fftBlur *= 0.5;
split(fftBlur, planes);
Mat ph, mag;
mag.zeros(planes[0].rows, planes[0].cols, CV_32F);
ph.zeros(planes[0].rows, planes[0].cols, CV_32F);
cartToPolar(planes[0], planes[1], mag, ph);
merge(planes, 2, fftBlur);
//inverse
Mat invfft;
dft(fftBlur, invfft, DFT_INVERSE|DFT_REAL_OUTPUT);
Mat result;
invfft.convertTo(result, CV_8U);
Mat image;
cvtColor(result, image, COLOR_GRAY2RGB);
imshow("Output", result);
imshow("Image", image);
waitKey();
The message you receive is an assertion it tells you DFT function only takes single precision floating point image with one or two channels (CV_32FC1, CV_32FC2, the letter C at the end of the flag mean channel) or double precision floating point images with one or two channels (CV_64FC1, CV_64FC2).
The two channel case is actually the representation of complex image in OpenCV data storage.
If you want you can split you image to std::vector<cv::Mat> where each element does represent one channel, using cv::split apply the DFT on each channels do the processing you want on it and recreate an multichannel image thanks to cv::merge.
From Learning OpenCV (about dft function):
The input array must be of floating-point type and may be single- or double-channel. In the single-channel case, the entries are assumed to be real numbers, and the output will be packed in a special space-saving format called complex conjugate symmetrical.
The same question is mentioned here in terms of matlab image processing.
You can check out cv::split function if you want to separate channels of your initial image.
I've created a filter extending QAbstractVideoFilter and
QVideoFilterRunnable and I've overrided the
QVideoFrame run(QVideoFrame* input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)`
method
The problem is that QVideoFrame format is Format_YUV420P and has no handle. I need to convert it into a CV_8UC1 in order to use OpenCV algorithms.
Which is the best way to accomplish this?
First you need to create a cv::Mat which has an API for initializing using data pointer as:
cv::Mat img = cv::Mat(rows, cols, CV_8UC3, input.data/*Change this to point the first element of array containing the YUV color info*/)
Now since the img is initialized with YUV color data, you may use various cvtColor modes to convert the YUV mat to other formats, for converting it to gray-scale you may try:
cv::Mat gray;
cv::cvtColor(img, gray, cv::COLOR_YUV2GRAY_I420);
I'm new in opencv and I had this problem...
Given the following Mat type (globally declarated)
Mat src_gray;
Mat dst;
I have dst being a zero grayscale Mat with this initialization
dst=Mat::zeros(src_gray.size(), CV_BGR2GRAY);
It seems I can't edit the pixels on the dst image (when I use imwrite, it gives me a black image as if I hadn't done anything).
This is the code I currently have:
for(int i=0;i<=dst.cols;i++)
for(int j=0;j<=dst.rows;j++)
{
dst.at<uchar>(j,i)=255;
}
imwrite( "img_res.png", dst );
The result Image has the dimensions it's supposed to have, but it is a black pixeled picture, shouldn't it be a white pixeled Image?
I don't know if it is relevant if I mention that I have 3 global Mats
Mat image;
Mat src_gray;
Mat dst;
Which are initialized this way:
image = imread( argv[1], 1 );
cvtColor( image, src_gray, CV_BGR2GRAY );
Then, I release them as:
image.release();
dst.release();
src_gray.release();
The other problem I get is that when I release the Mats (during execution), I get the "Segmentation fault (core dumped)" error. (I code from Linux Ubuntu distri)
Try:
dst=Mat::zeros(src_gray.size(), CV_8UC1);
When you use CV_BGR2GRAY, you are creating a Mat with 3 color channels, then, it's not possible to assign a number when you have an array of numbers (B,G,R).
With CV_8UC1, you create a Mat with 1 color channel of uchar then it should works with:
dst.at<uchar>(j,i)=255;
Here is my code. It's pretty simple.
Mat image = imread("filename.png");
imshow("image", image);
waitKey();
//Image looks great.
Mat image_gray;
image.convertTo(image_gray, CV_RGB2GRAY);
imshow("image", image_gray);
waitKey();
But when I call the image.convertTo(image_gray, CV_RGB2GRAY); line, I get the following error message:
OpenCV Error: Assertion failed (func != 0) in unknown function, file ..\..\..\sr
c\opencv\modules\core\src\convert.cpp, line 1020
Using OpenCV 2.4.3
The method convertTo does not do color conversion.
If you want to convert from BGR to GRAY you can use the function cvtColor:
Mat image_gray;
cvtColor(image, image_gray, CV_BGR2GRAY);
The function cv::Mat::convertTo is not for color conversion. It is for type conversion. The destination image should have same size and number of channels as the source image.
To convert from RGB to Gray, use the function cv::cvtColor.
cv::cvtColor(image,image_gray,CV_RGB2GRAY);
If you need to acquire video (e.g. from a webcam) in Grayscale, you can also set the saturation of the video feed to zero. (Ex. in Python syntax)
capture = cv.CaptureFromCAM(camera_index)
...
cv.SetCaptureProperty(capture, cv.CV_CAP_PROP_SATURATION,0)
image.convertTo(image_gray, CV_RGB2GRAY);
This's wrong.Correct one is,
Mat gray_image;
cvtColor(image, gray_image, CV_BGR2GRAY);
Try this.