opencv - Stitching more than 2 images - c++

I am now trying to align more than two images together in C++ with opencv. The problem is when I stitch more than 2, the previous image cannot be loaded.
For example, imageContainer now contains three images.
First Image:
Second Image:
Third Image:
First iteration of the loop: (Combining the first and second image)
Second iteration of the loop: (Combining the result from first iteration and third image)
You can see that after the second iteration, the result image does not contain the object. (Left side of the last image is all black),
In main.cpp
cv::Mat result = *imageContainer.begin();
for(vector<cv::Mat>::iterator itr = imageContainer.begin(); itr != imageContainer.end(); itr++){
if(itr == imageContainer.begin())
continue;
result = applySURF(result, *itr);
}
In SURF.cpp
cv::Mat applySURF(cv::Mat object, cv::Mat image){
/* More codes here but it won't affect solving the problem */
cv::Mat result;
cv::warpPerspective(image, result, transformationMat, cv::Size(object.cols + image.cols, image.rows));
cv::Mat half(result, cv::Rect(0, 0, image.cols, image.rows));
object.copyTo(half);
imshow("Object", object);
imshow("Result", result);
cvWaitKey(0);
return result;
}
I guess the problem is related to Region Of Interest (ROI). How can I solve it?
Many Thanks.

Try the following code:)
I tested some cases and got a conclusion that if the size of target image is not same as the source image, it will reallocate a new Mat to be pasted. In your case the size of ROI is not same as object, it allocates a new Mat half and it is not related to result
anymore. So your copyTo function copies the object into the new Mat half instead of the ROI of result.
cv::Mat applySURF(cv::Mat object, cv::Mat image){
/* More codes here but it won't affect solving the problem */
cv::Mat result;
cv::warpPerspective(image, result, transformationMat, cv::Size(object.cols + image.cols, image.rows));
cv::Mat half(result, cv::Rect(0, 0, object.cols, object.rows));
object.copyTo(half);
cv::imshow("Object", object);
cv::imshow("Result", result);
cv::WaitKey(0);
return result;
}

Related

cv::imshow in opencv is only displaying parts of a composite image, but displaying the parts separately works. Why?

Objective and problem
I'm trying to process a video file on the fly using OpenCV 3.4.1 by grabbing each frame, converting to grayscale, then doing Canny edge detection on it. In order to display the images (on the fly as well), I created a Mat class with 3 additional headers that is three times as wide as the original frame. The 3 extra headers represent the images I would like to display in the composite, and are positioned to the 1st, 2nd and 3rd horizontal segment of the composite.
After image processing however, the display of the composite image is not as expected: the first segment (where the original frame should be) is completely black, while the other segments (of processed images) are displayed fine. If, on the other hand, I display the ROIs one by one in separate windows, all the images look fine.
These are the things I tried to overcome this issue:
use .copyTo to actually copy the data into the appropriate image segments. The result was the same.
I put the Canny image to the compOrigPart ROI, and it did display in the first segment, so it is not a problem with the definition of the ROIs.
Define the composite as three channel image
In the loop convert it to grayscale
put processed images into it
convert back to BGR
put the original in.
This time around the whole composite was black, nothing showed.
As per gameon67's suggestion, I tried to create a namedWindow as well, but that doesn't help either.
Code:
int main() {
cv::VideoCapture vid("./Vid.avi");
if (!vid.isOpened()) return -1;
int frameWidth = vid.get(cv::CAP_PROP_FRAME_WIDTH);
int frameHeight = vid.get(cv::CAP_PROP_FRAME_HEIGHT);
int frameFormat = vid.get(cv::CAP_PROP_FORMAT);
cv::Scalar fontColor(250, 250, 250);
cv::Point textPos(20, 20);
cv::Mat frame;
cv::Mat compositeFrame(frameHeight, frameWidth*3, frameFormat);
cv::Mat compOrigPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(0, frameWidth));
cv::Mat compBwPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth, frameWidth*2));
cv::Mat compEdgePart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth*2, frameWidth*3));
while (vid.read(frame)) {
if (frame.empty()) break;
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
compOrigPart = frame;
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
cv::imshow("Original", compOrigPart);
cv::imshow("BW", compBwPart);
cv::imshow("Canny", compEdgePart);
cv::waitKey(33);
}
}
Questions
Why can't I display the entirety of the composite image in a single window, while displaying them separately is OK?
What is the difference between these displays? The data is obviously there, as evidenced by the separate windows.
Why only the original frame is misbehaving?
Your compBwPart and compEdgePart are grayscale images so the Mat type is CV8UC1 - single channel and therefore your compositeFrame is in grayscale too. If you want to combine these two images with a color image you have to convert it to BGR first and then fill the compOrigPart.
while (vid.read(frame)) {
if (frame.empty()) break;
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
cv::cvtColor(compositeFrame, compositeFrame, cv::COLOR_GRAY2BGR);
frame.copyTo(compositeFrame(cv::Rect(0, 0, frameWidth, frameHeight)));
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor); //the rest of your code
This is a combination of several issues.
The first problem is that you set the type of compositeFrame to the value returned by vid.get(cv::CAP_PROP_FORMAT). Unfortunately that property doesn't seem entirely reliable -- I've just had it return 0 (meaning CV_8UC1) after opening a color video, and then getting 3 channel (CV_8UC3) frames. Since you want to have the compositeFrame the same type as the input frame, this won't work.
To work around it, instead of using those properties, I'd lazy initialize compositeFrame and the 3 ROIs after receiving the first frame (based on it's dimensions and type).
The next set of problems lies in those two statements:
cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);
In this case assumption is made that frame is BGR (since you're trying to convert), meaning compositeFrame and its ROIs are also BGR. Unfortunately, in both cases you're writing a grayscale image into the ROI. This will cause a reallocation, and the target Mat will cease to be a ROI.
To correct this, use temporary Mats for the grayscale data, and use cvtColor to turn it back to BGR to write into the ROIs.
Similar problem lies in the following statement:
compOrigPart = frame;
That's a shallow copy, meaning it will just make compOrigPart another reference to frame (and therefore it will cease to be a ROI of compositeFrame).
What you need is a deep copy, using copyTo (note that the data types still need to match, but that was fixed earlier).
Finally, even though you try to be flexible regarding the type of the input video (judging by the vid.get(cv::CAP_PROP_FORMAT)), the rest of the code really assumes that the input is 3 channel, and will break if it isn't.
At the least, there should be some assertion to cover this expectation.
Putting this all together:
#include <opencv2/opencv.hpp>
int main()
{
cv::VideoCapture vid("./Vid.avi");
if (!vid.isOpened()) return -1;
cv::Scalar fontColor(250, 250, 250);
cv::Point textPos(20, 20);
cv::Mat frame, frame_gray, edges_gray;
cv::Mat compositeFrame;
cv::Mat compOrigPart, compBwPart, compEdgePart; // ROIs
while (vid.read(frame)) {
if (frame.empty()) break;
if (compositeFrame.empty()) {
// The rest of code assumes video to be BGR (i.e. 3 channel)
CV_Assert(frame.type() == CV_8UC3);
// Lazy initialize once we have the first frame
compositeFrame = cv::Mat(frame.rows, frame.cols * 3, frame.type());
compOrigPart = compositeFrame(cv::Range::all(), cv::Range(0, frame.cols));
compBwPart = compositeFrame(cv::Range::all(), cv::Range(frame.cols, frame.cols * 2));
compEdgePart = compositeFrame(cv::Range::all(), cv::Range(frame.cols * 2, frame.cols * 3));
}
cv::cvtColor(frame, frame_gray, cv::COLOR_BGR2GRAY);
cv::Canny(frame_gray, edges_gray, 100, 150);
// Deep copy data to the ROI
frame.copyTo(compOrigPart);
// The ROI is BGR, so we need to convert back
cv::cvtColor(frame_gray, compBwPart, cv::COLOR_GRAY2BGR);
cv::cvtColor(edges_gray, compEdgePart, cv::COLOR_GRAY2BGR);
cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
cv::imshow("Original", compOrigPart);
cv::imshow("BW", compBwPart);
cv::imshow("Canny", compEdgePart);
cv::waitKey(33);
}
}
Screenshot of the composite window (using some random test video off the web):

Getting masked area to be transparent?

So far i have managed to use masks and get the second image from the first. But what i want is the black area in second image to be transparent (i.e the output i an trying to get is the third image) Here is the code so far. Please advice me on this.
EDIT: Third one is from photoshop
//imwrite parameters
compression_params.push_back(CV_IMWRITE_JPEG_QUALITY);
compression_params.push_back(100);
//reading image to be masked
image = imread(main_img, -1);
//CV_LOAD_IMAGE_COLOR
namedWindow("output", WINDOW_NORMAL);
//imshow("output", image);
//Creating mask image with same size as original image
Mat mask(image.rows, image.cols, CV_8UC1, Scalar(0));
// Create Polygon from vertices
ROI_Vertices.push_back(Point2f(float(3112),float(58)));
ROI_Vertices.push_back(Point2f(float(3515),float(58)));
ROI_Vertices.push_back(Point2f(float(3515),float(1332)));
ROI_Vertices.push_back(Point2f(float(3112),float(958)));
approxPolyDP(ROI_Vertices, ROI_Poly, 1, true);
// Fill polygon white
fillConvexPoly(mask, &ROI_Poly[0] , ROI_Poly.size(), 255, 8, 0);
//imshow("output", mask);
// Create new image for result storage
imageDest = cvCreateMat(image.rows, image.cols, CV_8UC4);
// Cut out ROI and store it in imageDest
image.copyTo(imageDest, mask);
imwrite("masked.jpeg", imageDest, compression_params);
imshow("output", imageDest);
cvWaitKey(0);
This can be done by first setting its alpha value to 0 of the regions that you want to make them fully transparent (255 for others), and then save it to PNG.
To set the alpha value of pixel-(x,y), it can be done:
image.at<cv::Vec4b>(y, x)[3] = 0;
PS: you need to convert it to 4-channel format first if the image is not currently. For example:
cv::cvtColor(image, image, CV_BGR2BGRA);
Updated: It will be easier if you have already computed the mask for the ROI region, where you can simply merge it with the original image (assume having 3 channels) to get the final result. Like:
cv::Mat mask; // 0 for transparent regions, 255 otherwise (serve as the alpha channel)
std::vector<cv::Mat> channels;
cv::split(image, channels);
channels.push_back(mask);
cv::Mat result;
cv::merge(channels, result);

Merge two Mat images into one

I have a problem. I have an image. Then I have to split the image into two equal parts. I made this like that (the code is compiled, everything is good):
Mat image_temp1 = image(Rect(0, 0, image.cols, image.rows/2)).clone();
Mat image_temp2 = image(Rect(0, image.rows/2, image.cols, image.rows/2)).clone();
Then I have to change each part independently and finally to merge into one. I have no idea how to make this correctly. How I should merge this 2 parts of image into one image?
Example: http://i.stack.imgur.com/CLDK7.jpg
There is several way to do this, but the best way I found is to use cv::hconcat(mat1, mat2, dst) for horizontal merge orcv::vconcat(mat1, mat2, dst) for vertical.
Don't forget to take care of empty matrix merge case !
Seems that cv::Mat::push_back is exactly what are you looking for:
C++: void Mat::push_back(const Mat& m) : Adds elements to the bottom of the matrix.
Parameters:
m – Added line(s).
The methods add one or more elements to the bottom of the matrix. When elem is
Mat , its type and the number of columns must be the same as in the
container matrix.
Optionally, you could create new cv::Mat of proper size and place image parts directly into it:
Mat image_temp1 = image(Rect(0, 0, image.cols, image.rows/2)).clone();
Mat image_temp2 = image(Rect(0, image.rows/2, image.cols, image.rows/2)).clone();
...
cv::Mat result(image.rows, image.cols);
image_temp1.copyTo(result(Rect(0, 0, image.cols, image.rows/2)));
image_temp2.copyTo(result(Rect(0, image.rows/2, image.cols, image.rows/2));
How about this:
Mat newImage = image.clone();
Mat image_temp1 = newImage(Rect(0, 0, image.cols, image.rows/2));
Mat image_temp2 = newImage(Rect(0, image.rows/2, image.cols, image.rows/2));
By not using clone() to create the temp images, you're implicitly modifying newImage when you modify the temp images without the need to merge them again. After changing image_temp1 and image_temp2, newImage will be exactly the same as if you had split, modified, and then merged the subimages.

How to get cv::calcOpticalFlowSF to work?

I am useing the 2.4.4 version of OpenCV. - i know its a beta
but there is an example about cv::calcOpticalFlowSF the method in the example folder called: simpleflow_demo.cpp. But when i copy this demo and use it with my input images, it starts processing and after some seconds it came back a crash report.
The documentation about the method is a little bit strange, saying the output files are a x- and yflow instead of the cv::Mat& flow which the method actually wants.
Any ideas how to fix the problem to get the function working?
Try this simple demo that worked for me, then modify for your needs (display help from here):
Mat frame1 = imread("/home/radford/Desktop/1.png");
Mat frame2 = imread("/home/radford/Desktop/2.png");
namedWindow("flow");
Mat flow;
calcOpticalFlowSF(frame1, frame2, flow, 3, 2, 4);
Mat xy[2];
split(flow, xy);
//calculate angle and magnitude
Mat magnitude, angle;
cartToPolar(xy[0], xy[1], magnitude, angle, true);
//translate magnitude to range [0;1]
double mag_max;
minMaxLoc(magnitude, 0, &mag_max);
magnitude.convertTo(magnitude, -1, 1.0/mag_max);
//build hsv image
Mat _hsv[3], hsv;
_hsv[0] = angle;
_hsv[1] = Mat::ones(angle.size(), CV_32F);
_hsv[2] = magnitude;
merge(_hsv, 3, hsv);
//convert to BGR and show
Mat bgr;//CV_32FC3 matrix
cvtColor(hsv, bgr, COLOR_HSV2BGR);
imshow("flow", bgr);
waitKey(0);
In the example opencv/samples/cpp/simpleflow_demo.cpp there is a code block
if (frame1.type() != 16 || frame2.type() != 16) {
printf(APP_NAME "Images should be of equal type CV_8UC3\n");
exit(1);
}
So, grey images should be converted to CV_8UC3. For example using cvtColor(grey, grey3, CV_GRAY2RGB);

Flipping an image to get mirror effect

I am working on a video processing project which needs some flipping of frame. I tried using cvFlip but doesnt seem to flip along y axis (x axis working...) and results in segmentation fault. Is there any other option??
cv::Mat dst=src; //src= source image from cam
cv::flip(dst, dst, 1); //segmentation fault shown
imshow("flipped",dst);
cv::Mat src=imload("bla.png");
cv::Mat dst; // dst must be a different Mat
cv::flip(src, dst, 1); // because you can't flip in-place (leads to segfault)
Use cv::flip and pass 1 as flipcode.
Looking at your edit with the sample code, you cannot flip in place. You need a separate destination cv::Mat:
cv::Mat dst;
cv::flip(src, dst, 1);
imshow("flipped",dst);
The key is to create the dst exactly like the src:
cv::Mat dst = cv::Mat(src.rows, src.cols, CV_8UC3);
cv::flip(src, dst, 1);
imshow("flipped", dst);