If got a very strange problem here. Im developing with Visual Studio 10 and OpenCV
In the following code segment I'm creating a 1 channel Mat and write into two different Mats.
First window "test1" shows a black picture. That's correct.
The "test2" window still shows a black picture. Still correct.
Then the last window "test3", shows the same picture as stored in bwHSVred after the inRange command.
Why would the bwHSVblue change during this inRange operation?
Does anybody know why? It doesn't make any sense to me a all.
frame = imread(pathtopicture);
cvtColor(frame, calHSV, COLOR_BGR2HSV);
inRange(calibHSV, Scalar(255, 255, 255), Scalar(0, 0, 0), bwAll);
bwHSVred = bwAll;
bwHSVblue = bwAll;
imshow("test1",bwHSVblue);
//load red
//set the x_MIN,x_MAX values to Hmin=0,Smin=119,Vmin=108,Hmax=218,Smax=234,Vmax=168
setHSVval(redCube);
updateTrackbars();
currentColor = RED;
imshow("test2",bwHSVblue);
inRange(calibHSV, Scalar(H_MIN, S_MIN, V_MIN), Scalar(H_MAX, S_MAX, V_MAX), bwHSVred);
imshow("test3",bwHSVblue);
Definition of the Mat objects in the .h-file
private:
Mat calHSV;
Mat bwAll;
Mat bwHSVred;
Mat bwHSVblue;
The problem is in your C/C++ usage of pointers.
All three matrices are the same instance:
bwHSVred = bwAll;
bwHSVblue = bwAll;
You are copying the pointer, meaning that they now all point to the same matrix.
If you want to make copies of a matrix, you should use clone or copyTo as explained in the docs:
Mat F = A.clone();
Mat G;
A.copyTo(G);
Mat a=b; does a shallow copy only ( both point to the same data )
that's why bwHSVred == bwHSVblue == bwAll
Related
Objective and problem
I'm trying to process a video file on the fly using OpenCV 3.4.1 by grabbing each frame, converting to grayscale, then doing Canny edge detection on it. In order to display the images (on the fly as well), I created a Mat class with 3 additional headers that is three times as wide as the original frame. The 3 extra headers represent the images I would like to display in the composite, and are positioned to the 1st, 2nd and 3rd horizontal segment of the composite.
After image processing however, the display of the composite image is not as expected: the first segment (where the original frame should be) is completely black, while the other segments (of processed images) are displayed fine. If, on the other hand, I display the ROIs one by one in separate windows, all the images look fine.
These are the things I tried to overcome this issue:
use .copyTo to actually copy the data into the appropriate image segments. The result was the same.
I put the Canny image to the compOrigPart ROI, and it did display in the first segment, so it is not a problem with the definition of the ROIs.
Define the composite as three channel image
In the loop convert it to grayscale
put processed images into it
convert back to BGR
put the original in.
This time around the whole composite was black, nothing showed.
As per gameon67's suggestion, I tried to create a namedWindow as well, but that doesn't help either.
Code:
int main() {
cv::VideoCapture vid("./Vid.avi");
if (!vid.isOpened()) return -1;
int frameWidth = vid.get(cv::CAP_PROP_FRAME_WIDTH);
int frameHeight = vid.get(cv::CAP_PROP_FRAME_HEIGHT);
int frameFormat = vid.get(cv::CAP_PROP_FORMAT);
cv::Scalar fontColor(250, 250, 250);
cv::Point textPos(20, 20);
cv::Mat frame;
cv::Mat compositeFrame(frameHeight, frameWidth*3, frameFormat);
cv::Mat compOrigPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(0, frameWidth));
cv::Mat compBwPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth, frameWidth*2));
cv::Mat compEdgePart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth*2, frameWidth*3));
while (vid.read(frame)) {
if (frame.empty()) break;
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
compOrigPart = frame;
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
cv::imshow("Original", compOrigPart);
cv::imshow("BW", compBwPart);
cv::imshow("Canny", compEdgePart);
cv::waitKey(33);
}
}
Questions
Why can't I display the entirety of the composite image in a single window, while displaying them separately is OK?
What is the difference between these displays? The data is obviously there, as evidenced by the separate windows.
Why only the original frame is misbehaving?
Your compBwPart and compEdgePart are grayscale images so the Mat type is CV8UC1 - single channel and therefore your compositeFrame is in grayscale too. If you want to combine these two images with a color image you have to convert it to BGR first and then fill the compOrigPart.
while (vid.read(frame)) {
if (frame.empty()) break;
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
cv::cvtColor(compositeFrame, compositeFrame, cv::COLOR_GRAY2BGR);
frame.copyTo(compositeFrame(cv::Rect(0, 0, frameWidth, frameHeight)));
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor); //the rest of your code
This is a combination of several issues.
The first problem is that you set the type of compositeFrame to the value returned by vid.get(cv::CAP_PROP_FORMAT). Unfortunately that property doesn't seem entirely reliable -- I've just had it return 0 (meaning CV_8UC1) after opening a color video, and then getting 3 channel (CV_8UC3) frames. Since you want to have the compositeFrame the same type as the input frame, this won't work.
To work around it, instead of using those properties, I'd lazy initialize compositeFrame and the 3 ROIs after receiving the first frame (based on it's dimensions and type).
The next set of problems lies in those two statements:
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
In this case assumption is made that frame is BGR (since you're trying to convert), meaning compositeFrame and its ROIs are also BGR. Unfortunately, in both cases you're writing a grayscale image into the ROI. This will cause a reallocation, and the target Mat will cease to be a ROI.
To correct this, use temporary Mats for the grayscale data, and use cvtColor to turn it back to BGR to write into the ROIs.
Similar problem lies in the following statement:
compOrigPart = frame;
That's a shallow copy, meaning it will just make compOrigPart another reference to frame (and therefore it will cease to be a ROI of compositeFrame).
What you need is a deep copy, using copyTo (note that the data types still need to match, but that was fixed earlier).
Finally, even though you try to be flexible regarding the type of the input video (judging by the vid.get(cv::CAP_PROP_FORMAT)), the rest of the code really assumes that the input is 3 channel, and will break if it isn't.
At the least, there should be some assertion to cover this expectation.
Putting this all together:
#include <opencv2/opencv.hpp>
int main()
{
cv::VideoCapture vid("./Vid.avi");
if (!vid.isOpened()) return -1;
cv::Scalar fontColor(250, 250, 250);
cv::Point textPos(20, 20);
cv::Mat frame, frame_gray, edges_gray;
cv::Mat compositeFrame;
cv::Mat compOrigPart, compBwPart, compEdgePart; // ROIs
while (vid.read(frame)) {
if (frame.empty()) break;
if (compositeFrame.empty()) {
// The rest of code assumes video to be BGR (i.e. 3 channel)
CV_Assert(frame.type() == CV_8UC3);
// Lazy initialize once we have the first frame
compositeFrame = cv::Mat(frame.rows, frame.cols * 3, frame.type());
compOrigPart = compositeFrame(cv::Range::all(), cv::Range(0, frame.cols));
compBwPart = compositeFrame(cv::Range::all(), cv::Range(frame.cols, frame.cols * 2));
compEdgePart = compositeFrame(cv::Range::all(), cv::Range(frame.cols * 2, frame.cols * 3));
}
cv::cvtColor(frame, frame_gray, cv::COLOR_BGR2GRAY);
cv::Canny(frame_gray, edges_gray, 100, 150);
// Deep copy data to the ROI
frame.copyTo(compOrigPart);
// The ROI is BGR, so we need to convert back
cv::cvtColor(frame_gray, compBwPart, cv::COLOR_GRAY2BGR);
cv::cvtColor(edges_gray, compEdgePart, cv::COLOR_GRAY2BGR);
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
cv::imshow("Original", compOrigPart);
cv::imshow("BW", compBwPart);
cv::imshow("Canny", compEdgePart);
cv::waitKey(33);
}
}
Screenshot of the composite window (using some random test video off the web):
I create a Bird-View-Image with the warpPerspective()-function like this:
warpPerspective(frame, result, H, result.size(), CV_WARP_INVERSE_MAP, BORDER_TRANSPARENT);
The result looks very good and also the border is transparent:
Bird-View-Image
Now I want to put this image on top of another image "out". I try doing this with the function warpAffine like this:
warpAffine(result, out, M, out.size(), CV_INTER_LINEAR, BORDER_TRANSPARENT);
I also converted "out" to a four channel image with alpha channel according to a question which was already asked on stackoverflow:
Convert Image
This is the code: cvtColor(out, out, CV_BGR2BGRA);
I expected to see the chessboard but not the gray background. But in fact, my result looks like this:
Result Image
What am I doing wrong? Do I forget something to do? Is there another way to solve my problem? Any help is appreciated :)
Thanks!
Best regards
DamBedEi
I hope there is a better way, but here it is something you could do:
Do warpaffine normally (without the transparency thing)
Find the contour that encloses the image warped
Use this contour for creating a mask (white values inside the image warped, blacks in the borders)
Use this mask for copy the image warped into the other image
Sample code:
// load images
cv::Mat image2 = cv::imread("lena.png");
cv::Mat image = cv::imread("IKnowOpencv.jpg");
cv::resize(image, image, image2.size());
// perform warp perspective
std::vector<cv::Point2f> prev;
prev.push_back(cv::Point2f(-30,-60));
prev.push_back(cv::Point2f(image.cols+50,-50));
prev.push_back(cv::Point2f(image.cols+100,image.rows+50));
prev.push_back(cv::Point2f(-50,image.rows+50 ));
std::vector<cv::Point2f> post;
post.push_back(cv::Point2f(0,0));
post.push_back(cv::Point2f(image.cols-1,0));
post.push_back(cv::Point2f(image.cols-1,image.rows-1));
post.push_back(cv::Point2f(0,image.rows-1));
cv::Mat homography = cv::findHomography(prev, post);
cv::Mat imageWarped;
cv::warpPerspective(image, imageWarped, homography, image.size());
// find external contour and create mask
std::vector<std::vector<cv::Point> > contours;
cv::Mat imageWarpedCloned = imageWarped.clone(); // clone the image because findContours will modify it
cv::cvtColor(imageWarpedCloned, imageWarpedCloned, CV_BGR2GRAY); //only if the image is BGR
cv::findContours (imageWarpedCloned, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE);
// create mask
cv::Mat mask = cv::Mat::zeros(image.size(), CV_8U);
cv::drawContours(mask, contours, 0, cv::Scalar(255), -1);
// copy warped image into image2 using the mask
cv::erode(mask, mask, cv::Mat()); // for avoid artefacts
imageWarped.copyTo(image2, mask); // copy the image using the mask
//show images
cv::imshow("imageWarpedCloned", imageWarpedCloned);
cv::imshow("warped", imageWarped);
cv::imshow("image2", image2);
cv::waitKey();
One of the easiest ways to approach this (not necessarily the most efficient) is to warp the image twice, but set the OpenCV constant boundary value to different values each time (i.e. zero the first time and 255 the second time). These constant values should be chosen towards the minimum and maximum values in the image.
Then it is easy to find a binary mask where the two warp values are close to equal.
More importantly, you can also create a transparency effect through simple algebra like the following:
new_image = np.float32((warp_const_255 - warp_const_0) *
preferred_bkg_img) / 255.0 + np.float32(warp_const_0)
The main reason I prefer this method is that openCV seems to interpolate smoothly down (or up) to the constant value at the image edges. A fully binary mask will pick up these dark or light fringe areas as artifacts. The above method acts more like true transparency and blends properly with the preferred background.
Here's a small test program that warps with transparent "border", then copies the warped image to a solid background.
int main()
{
cv::Mat input = cv::imread("../inputData/Lenna.png");
cv::Mat transparentInput, transparentWarped;
cv::cvtColor(input, transparentInput, CV_BGR2BGRA);
//transparentInput = input.clone();
// create sample transformation mat
cv::Mat M = cv::Mat::eye(2,3, CV_64FC1);
// as a sample, just scale down and translate a little:
M.at<double>(0,0) = 0.3;
M.at<double>(0,2) = 100;
M.at<double>(1,1) = 0.3;
M.at<double>(1,2) = 100;
// warp to same size with transparent border:
cv::warpAffine(transparentInput, transparentWarped, M, transparentInput.size(), CV_INTER_LINEAR, cv::BORDER_TRANSPARENT);
// NOW: merge image with background, here I use the original image as background:
cv::Mat background = input;
// create output buffer with same size as input
cv::Mat outputImage = input.clone();
for(int j=0; j<transparentWarped.rows; ++j)
for(int i=0; i<transparentWarped.cols; ++i)
{
cv::Scalar pixWarped = transparentWarped.at<cv::Vec4b>(j,i);
cv::Scalar pixBackground = background.at<cv::Vec3b>(j,i);
float transparency = pixWarped[3] / 255.0f; // pixel value: 0 (0.0f) = fully transparent, 255 (1.0f) = fully solid
outputImage.at<cv::Vec3b>(j,i)[0] = transparency * pixWarped[0] + (1.0f-transparency)*pixBackground[0];
outputImage.at<cv::Vec3b>(j,i)[1] = transparency * pixWarped[1] + (1.0f-transparency)*pixBackground[1];
outputImage.at<cv::Vec3b>(j,i)[2] = transparency * pixWarped[2] + (1.0f-transparency)*pixBackground[2];
}
cv::imshow("warped", outputImage);
cv::imshow("input", input);
cv::imwrite("../outputData/TransparentWarped.png", outputImage);
cv::waitKey(0);
return 0;
}
I use this as input:
and get this output:
which looks like ALPHA channel isn't set to ZERO by warpAffine but to something like 205...
But in general this is the way I would do it (unoptimized)
The implementation of this functionality seems pretty straightforward in Python, as shown here: http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html
Yet, when I tried to do exactly the same in C++, I get bad arguments error (for the grabcut function). How to put the mask image in the right format?
I am a newbie at this, so I'd be very thankful if someone could help me understand better. Thank you!
Here's what I have so far:
Mat image;
image= imread(file);
Mat mask;
mask.setTo( GC_BGD );
mask = imread("messi5.png");
Mat image2 = image.clone();
// define bounding rectangle
cv::Rect rectangle(startX, startY, width, height);
cv::Mat result; // segmentation result (4 possible values)
cv::Mat bgModel,fgModel; // the models (internally used)
//// GrabCut segmentation that works, but with a rectangle, not with the mask I need
//cv::grabCut(image, // input image
// result, // segmentation result
// rectangle,// rectangle containing foreground
// bgModel,fgModel, // models
// 1, // number of iterations
// cv::GC_INIT_WITH_RECT); // use rectangle
grabCut( image, mask, rectangle, bgModel, fgModel, 1, GC_INIT_WITH_MASK);
cv::compare(mask,cv::GC_PR_FGD,mask,cv::CMP_EQ);
cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
image.copyTo(foreground,mask); // bg pixels not copied
namedWindow( "Display window", WINDOW_AUTOSIZE );
imshow( "Display window", foreground );
waitKey(0);
return 0;
}
It looks like you have misunderstood the guide, repeated here from the linked guide in the question:
# newmask is the mask image I manually labelled
newmask = cv2.imread('newmask.png',0)
# whereever it is marked white (sure foreground), change mask=1
# whereever it is marked black (sure background), change mask=0
mask[newmask == 0] = 0
mask[newmask == 255] = 1
mask, bgdModel, fgdModel = cv2.grabCut(img,mask,None,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_MASK)
mask = np.where((mask==2)|(mask==0),0,1).astype('uint8')
img = img*mask[:,:,np.newaxis]
plt.imshow(img),plt.colorbar(),plt.show()
this is not what you have done i'm afraid. For a start you seem to have set the mask to the rgb image:
mask = imread("messi5.png");
whereas is should be set to the mask image:
mask = imread("newmask.png",CV_LOAD_IMAGE_GRAYSCALE);
EDIT from comments:
from a pure red mask painted over the image (an actual mask would be better).
maskTmp = imread("messi5.png");
std::vector<cv::Mat> channels(3)
split( messi5, channels);
cv::Mat maskRed = channels[2];
now threshold on the red channel to get your binary mask.
I have two images, the first one smaller than the other. I need to copy the second image on the first image. To do so, I need to set the ROI on the first one, copy the second image onto the first one and then reset the ROI.
However I am using the C++ interface so I have no idea how to do this. In C I could have used cvSetImageROI but this doesn't work on the C++ interface.
So basically whats the C++ alternative to cvSetImageROI?
//output is a pointer to the mat whom I want the second image (colourMiniBinMask) copied upon
Rect ROI (478, 359, 160, 120);
Mat imageROI (*output, ROI);
colourMiniBinMask.copyTo (imageROI);
imshow ("Gravity", *output);
I think you have something wrong. If the first one is smaller than the other one and you want to copy the second image in the first one, you don't need an ROI. You can just resize the second image in copy it into the first one.
However if you want to copy the first one in the second one, I think this code should work:
cv::Rect roi = cv::Rect((img2.cols - img1.cols)/2,(img2.rows - img1.rows)/2,img1.cols,img1.rows);
cv::Mat roiImg;
roiImg = img2(roi);
img1.copyTo(roiImg);
This is the code I used. I think the comments explain it.
/* ROI by creating mask for the parallelogram */
Mat mask = cvCreateMat(480, 640, CV_8UC1);
// Create black image with the same size as the original
for(int i=0; i<mask.cols; i++)
for(int j=0; j<mask.rows; j++)
mask.at<uchar>(Point(i,j)) = 0;
// Create Polygon from vertices
vector<Point> approxedRectangle;
approxPolyDP(rectangleVertices, approxedRectangle, 1.0, true);
// Fill polygon white
fillConvexPoly(mask, &approxedRectangle[0], approxedRectangle.size(), 255, 8, 0);
// Create new image for result storage
Mat imageDest = cvCreateMat(480, 640, CV_8UC3);
// Cut out ROI and store it in imageDest
image->copyTo(imageDest, mask);
I also wrote about this and put some pictures here.
I am using OpenCVwith Eclipse.
I need to detect the human skin, so I convert the image to HSV and the I use inRange function to obtain a Mat with the image with the skin in white.
The problem is that now,I need to detect in which components are the white color to modify this pixels in the original frame ( i am changing the skin color with the video camera), but I cant access to the Mat returned in InRange
cvtColor(frame,frame,CV_BGR2HSV);
Mat n;
inRange(frame, Scalar(0, 10, 60), Scalar(20, 150, 255), n);
for(int i=0;i<frame.rows;i++)
{
for(int j=0;j<frame.cols;j++)
{
n.at(&i);
//n(i,j);
}
}
That is the problematic code. When I get to the internal loop, the build fails giving a lot of error refering to the template.
Anyone knows how can I access to this matrix? Is there another way to achieve my objective? Maybe I am complicating the problem.
Thanks for your time.
nothing to do with inRange or such, it's just your Mat access code, that is broken.
Vec3b & hsvPixel = n.at<Vec3b>(i,j);
// hsvPixel[0] = h;
// hsvPixel[1] = s;
// hsvPixel[2] = v;