OpenCV CascadeClassifier detectMultiScale resulting Rect outside input Mat bounds - c++

I'm having a problem with detectMultiScale returning rectangles outside the bounds of the input Mat.
So what I'm doing is an optimization technique where the first frame of a video feed is passed to detectMultiScale in it's entirety.
If an object was detected, I create a temp Mat, which i clone the previous frames detected object's rect from the current full frame.
Then i pass this temp Mat to detectMultiScale, so only the area around the rectangle where the previous frame detected an object.
The problem i'm having is that the results from detectMultiScale when passing the temp Mat give rectangles that are outside the bounds of the input temp Mat.
Mostly I would just like to know exactly what is going on here. I have two ideas of what could be happening, but I can't figure out for sure.
Either the clone operation when cloning a rect from a full frame to the temp Mat is somewhere inside the Mat object setting the cloned area at the rows and columns of the full frame. So for example, i have a full frame of 100x100, i'm trying to clone a 10x10 rectangle from it at position 80x80. The resulting Mat will then be size 10x10, but maybe inside the Mat somewhere it is saying the Mat starts at 80x80?
CascadeClassifier is keeping state somewhere of the full frame i had passed to it previously?
I don't know what is happening here for sure, but was hoping someone could shed some light.
Here's a little code example of what i'm trying to do, with comments explaining the results i'm seeing:
std::vector<cv::Rect> DetectObjects(cv::Mat fullFrame, bool useFullFrame, cv::Rect detectionRect)
{
// fullFrame is 100x100
// detectionRect is 10x10 at position 80x80 eg. cv::Rect(80,80,10,10)
// useFullFrame is False
std::vector<cv::Rect> results;
if(useFullFrame)
{
object_cascade.detectMultiScale(fullFrame,
results,
m_ScaleFactor,
m_Neighbors,
0 | cv::CASCADE_SCALE_IMAGE | cv::CASCADE_DO_ROUGH_SEARCH | cv::CASCADE_DO_CANNY_PRUNING,
m_MinSize,
m_MaxSize);
}
else
{
// useFullFrame is false, so we run this block
cv::Mat tmpMat = fullFrame(detectionRect).clone();
// tmpMat is size 10,10
object_cascade.detectMultiScale(tmpMat,
results,
m_ScaleFactor,
m_Neighbors,
0 | cv::CASCADE_SCALE_IMAGE | cv::CASCADE_DO_ROUGH_SEARCH | cv::CASCADE_DO_CANNY_PRUNING,
m_MinSize,
m_MaxSize);
}
if(results.size() > 0)
{
// this is the weird part. When looking at the first element of
// results, (result[0]), it is at position 80,80, size 10,10
// so it is cv::Rect(80,80,10,10)
// even though the second detectMultiScale was ran with a Mat of 10x10
// do stuff
}
}
This is pretty darn close to what i have in code, except for the actual example values i mention above in the comments, i used values that were easy rather than full frame values like 1920x1080 and actual results, something like 367x711 for example.
So why am i getting results from detectMultiScale that are outside the bounds of the input Mat?
EDIT:
I had written this program originally for an embedded linux distribution, where this problem does not arise (i've always gotten expected results). This problem is happening on a windows release and build of opencv, so i'm currently going through the opencv code to see if there's anything that stands out related to this.

I believe this is a simple logic error. This:
if(fullFrame)
should be this:
if(useFullFrame)

Related

Why does any OpenCv (C++) substract operation result in a full black image?

I want to start with that I am certainly no beginner using OpenCV. But I have been mostly using it in C#, and now I am completely lost on a simple issue on the C++ variant that I cannot seem to solve...
Issue: Whenever I use any operation that will subtract one image from another, the output image is 100% black. With 1 in 50 images showing partly the expected result, but also partly being full black (I will link example images in the end). To clarify, with full black I mean every pixel value 0. I am expecting quite some visable noise after this operation, but none is present.
Here is my simple piece of debugging code:
Mat moveImage, stillImage, subst;
while(capture) {
moveImage = CameraClass::cameraVector[0]->imageList.front().clone();
stillImage = CameraClass::cameraVector[0]->imageList.back().clone();
cv::resize(moveImage, moveImage, Size(moveImage.cols * 0.5, moveImage.rows * 0.5), INTER_LINEAR);
cv::resize(stillImage, stillImage, Size(stillImage.cols * 0.5, stillImage.rows * 0.5), INTER_LINEAR);
cv::addWeighted(moveImage, 1, stillImage, -1, 0, subst, CV_8UC1);
stringstream ss, ss2, ss3;
ss << path + "/camtest/image" + to_string(ui32FrameCount) + ".png";
ss2 << path + "/camtest2/image" + to_string(ui32FrameCount) + ".png";
ss3 << path + "/camtest3/image" + to_string(ui32FrameCount) + ".png";
string filepath1 = ss.str();
string filepath2 = ss2.str();
string filepath3 = ss3.str();
cv::imwrite(filepath1, moveImage);
cv::imwrite(filepath2, stillImage);
cv::imwrite(filepath3, subst);
}
My results here are 2 perfectly nice images being saved from moveImage and stillImage, while subst is just completely black.
The addWeighted option is already an attempt to solve it from my side. I have also tried:
cv::subtract(moveImage, stillImage, subst);
And also the C++ operation:
subst = moveImage - stillImage;
All resulting in the same result, black images. I tried adding the images in all these different ways, and the output result is completely fine. So it must be something with the subtract operation and perhaps values dropping below 0? But a CV_8UC1 MatType should truncate below 0 values automatically right?
Here are some example images and results:
Partly black, partly noise (Had to amplify the noise or else it got lost in upload compression)
100% black image result
moveImage frame 1
stillImage frame 1
Additional information that could be important:
My code is Threaded/Asynchronous. This exmaple piece of code is in one thread and isolated though. Since moveImage and stillImage are fine I do not see why other threads or code could badly influence my subtract operation. Subst only exists in this piece of code and is not used or accessable anywhere else.
I am not releasing my Mat resources at the moment. I have tried doing that in the debugging process but it made no difference in the results.
Right now all 3 Mat variables are declared once before the loop of this code and then used over and over. I don't know if that could be an issue, but I also already tried declaring them again in each loop.
Every loop of this code stillImage and moveImage are updated with a new frame from a camera. Maybe the subtract operation somehow keeps a reference somewhere instead of a copy?
The size of moveImage and stillImage is exactly the same and does not change throughout the runtime.
It can easily happen that cv::Mat share their data memory within lists or arrays so that if on of the images is updated, all the images still point to the same memory. For example if capturing updates the same cv::Mat element in every iteration and this cv::Mat is pushed back to a list or vector without deep-copying the pixels (with .clone() or .copyTo() ).
std::vector<cv::Mat> buffer;
cv::Mat img;
while(true)
{
bool success = capture.read(img);
//buffer.push_back(img); // in this case Mat objects will share their memory!
buffer.push_back(img.clone()); // this one is ok
}
Another, less common case is to initialize a cv::Mat with known image size and to create a container with a specific size and a default-object. In that case, all the objects will share their data memory.
cv::Mat img = cv::Mat::zeros(cv::Size(knownWidth, knownHeight), CV_8UC3);
// WARNING: this initialized element creation will lead to shared memory between different Mat objects in the buffer!
std::vector<cv::Mat> buffer(BUFFER_SIZE, img);
The reason in both cases is that cv::Mat is basically some reference counting smart header which can be copied without the need to copy underlying pixel memory space, so an explicit deep-copying of data is necessary if memory shall not be shared.
For debugging of such cases:
Make sure that within your function/loop, both images aren't identical. You can do that with cv::absdiff and thresholding the result by > 0. If there are any white pixels, the image content isnt identical.

How to align 2 images based on their content with OpenCV

I am totally new to OpenCV and I have started to dive into it. But I'd need a little bit of help.
So I want to combine these 2 images:
I would like the 2 images to match along their edges (ignoring the very right part of the image for now)
Can anyone please point me into the right direction? I have tried using the findTransformECC function. Here's my implementation:
cv::Mat im1 = [imageArray[1] CVMat3];
cv::Mat im2 = [imageArray[0] CVMat3];
// Convert images to gray scale;
cv::Mat im1_gray, im2_gray;
cvtColor(im1, im1_gray, CV_BGR2GRAY);
cvtColor(im2, im2_gray, CV_BGR2GRAY);
// Define the motion model
const int warp_mode = cv::MOTION_AFFINE;
// Set a 2x3 or 3x3 warp matrix depending on the motion model.
cv::Mat warp_matrix;
// Initialize the matrix to identity
if ( warp_mode == cv::MOTION_HOMOGRAPHY )
warp_matrix = cv::Mat::eye(3, 3, CV_32F);
else
warp_matrix = cv::Mat::eye(2, 3, CV_32F);
// Specify the number of iterations.
int number_of_iterations = 50;
// Specify the threshold of the increment
// in the correlation coefficient between two iterations
double termination_eps = 1e-10;
// Define termination criteria
cv::TermCriteria criteria (cv::TermCriteria::COUNT+cv::TermCriteria::EPS, number_of_iterations, termination_eps);
// Run the ECC algorithm. The results are stored in warp_matrix.
findTransformECC(
im1_gray,
im2_gray,
warp_matrix,
warp_mode,
criteria
);
// Storage for warped image.
cv::Mat im2_aligned;
if (warp_mode != cv::MOTION_HOMOGRAPHY)
// Use warpAffine for Translation, Euclidean and Affine
warpAffine(im2, im2_aligned, warp_matrix, im1.size(), cv::INTER_LINEAR + cv::WARP_INVERSE_MAP);
else
// Use warpPerspective for Homography
warpPerspective (im2, im2_aligned, warp_matrix, im1.size(),cv::INTER_LINEAR + cv::WARP_INVERSE_MAP);
UIImage* result = [UIImage imageWithCVMat:im2_aligned];
return result;
I have tried playing around with the termination_eps and number_of_iterations and increased/decreased those values, but they didn't really make a big difference.
So here's the result:
What can I do to improve my result?
EDIT: I have marked the problematic edges with red circles. The goal is to warp the bottom image and make it match with the lines from the image above:
I did a little bit of research and I'm afraid the findTransformECC function won't give me the result I'd like to have :-(
Something important to add:
I actually have an array of those image "stripes", 8 in this case, they all look similar to the images shown here and they all need to be processed to match the line. I have tried experimenting with the stitch function of OpenCV, but the results were horrible.
EDIT:
Here are the 3 source images:
The result should be something like this:
I transformed every image along the lines that should match. Lines that are too far away from each other can be ignored (the shadow and the piece of road on the right portion of the image)
By your images, it seems that they overlap. Since you said the stitch function didn't get you the desired results, implement your own stitching. I'm trying to do something close to that too. Here is a tutorial on how to implement it in c++: https://ramsrigoutham.com/2012/11/22/panorama-image-stitching-in-opencv/
You can use Hough algorithm with high threshold on two images and then compare the vertical lines on both of them - most of them should be shifted a bit, but keep the angle.
This is what I've got from running this algorithm on one of the pictures:
Filtering out horizontal lines should be easy(as they are represented as Vec4i), and then you can align the remaining lines together.
Here is the example of using it in OpenCV's documentation.
UPDATE: another thought. Aligning the lines together can be done with the concept similar to how cross-correlation function works. Doesn't matter if picture 1 has 10 lines, and picture 2 has 100 lines, position of shift with most lines aligned(which is, mostly, the maximum for CCF) should be pretty close to the answer, though this might require some tweaking - for example giving weight to every line based on its length, angle, etc. Computer vision never has a direct way, huh :)
UPDATE 2: I actually wonder if taking bottom pixels line of top image as an array 1 and top pixels line of bottom image as array 2 and running general CCF over them, then using its maximum as shift could work too... But I think it would be a known method if it worked good.

How do I crop an open CV matrix with an rectangle that overlaps the boundary of the source image

There is an assert in the implementation of cropping a matrix that prevents the cropRect from overlapping the edges of the source image.
// Asserts that cropRect fits inside the image's bounds.
cv::Mat croppedImage = image(cropRect);
I want to lift this restriction and be able to do this using black pixels that lie outside the image. Is this possible?
The answer is: technically it is possible but you really really don't want to do it. There no "black pixels" that lie around your image. Your 'image' allocated just enough memory for himself, and that's it. So if you try to access pixels outside of allocated memory you will get runtime error. If you want to have some black pixels you will have to do that yourself in the way that #ffriend described. image(cropRect) is not allocating anything, it just creating new pointer to memory that already exist.
In case you are still curious about how this crop can be done, OpenCV is doing the following:
// check that cropRect is inside the image
if ((cropRect & Rect(0,0,image.cols,image.rows)) != cropRect)
return -1; // some kind of error notification
// use one of Mat constructors (x, y, width and height are taken from cropRect)
Mat croppedImage(Size(width,height), image.type(), image.ptr(y)+x, image.step);
You can skip the test and go to initialization, but as I said this is a good recipe for disaster.

Extracting Background Image Using GrabCut

I've an image (.jpg image), and I want to extract the background from the original image. I've googled a lot but have only found tutorials of extracting foreground image.
I've taken the code from another stackoverflow question. The code is working fine for me, and I've successfully extracted the foreground (as per my requirements). Now I want to completely remove this foreground from the original image. I want it to be something like this:-
Background = Original Image - Foreground
The empty space can be filled with black or white color. How can I achieve this?
I've tried using this technique:-
Mat background = image2 - foreground;
but it gives a complete black image.
Code:-
#include <opencv2/opencv.hpp>
#include <iostream>
using namespace cv;
using namespace std;
int main( )
{
// Open another image
Mat image;
image= cv::imread("images/abc.jpg");
Mat image2 = image.clone();
// define bounding rectangle
cv::Rect rectangle(40,90,image.cols-80,image.rows-170);
cv::Mat result; // segmentation result (4 possible values)
cv::Mat bgModel,fgModel; // the models (internally used)
// GrabCut segmentation
cv::grabCut(image, // input image
result, // segmentation result
rectangle,// rectangle containing foreground
bgModel,fgModel, // models
1, // number of iterations
cv::GC_INIT_WITH_RECT); // use rectangle
cout << "oks pa dito" <<endl;
// Get the pixels marked as likely foreground
cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
// Generate output image
cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
//cv::Mat background(image.size(),CV_8UC3,cv::Scalar(255,255,255));
image.copyTo(foreground,result); // bg pixels not copied
// draw rectangle on original image
cv::rectangle(image, rectangle, cv::Scalar(255,255,255),1);
imwrite("img_1.jpg",image);
imwrite("Foreground.jpg",foreground);
Mat background = image2 - foreground;
imwrite("Background.jpg",background);
return 0;
}
Note: I'm an opencv beginner and don't have much knowledge of it right now. I shall be very thankful to you if you can either post the complete code (as required by me) or just post the lines of code and tell me where these lines of code be placed. Thanks.
P.S. This is my second question at StackOverflow.com. apologies ... if not following any convention.
Instead of copying all the pixels that are foreground, it copies all pixels which are not foreground. You can do this by using ~, which negates the mask:
image.copyTo(background,~result);
What if you //Get the pixels marked as likely background:
// Get the pixels marked as likely background
cv::compare(result,cv::GC_PR_BGD,result,cv::CMP_EQ);
Edit: The above code is missing GC_BGD pixels. Despite a more efficient answer was given, let's finish what we started:
// Get the pixels marked as background
cv::compare(result,cv::GC_BGD,result_a,cv::CMP_EQ);
// Get the pixels marked as likely background
cv::compare(result,cv::GC_PR_BGD,result_b,cv::CMP_EQ);
// Final results
result=result_a+result_b;
Just a small suggestion,#William's
answer can be written more concisely as:
result = result & 1;
in order to get the binary mask.
Maybe another example helps, in which I assumed that the middle portion of the image is definitely foreground.
So try this link.
Example

weird behaviour saving image in opencv

After doing some opencv operation, I initialize a new image that I'd like to use. Saving this empty image gives a weird result
The lines I use to save this image are:
Mat dst2 (Size (320, 240), CV_8UC3);
imwrite("bla.jpg", dst2);
I should get a black image, but this is what I get. Moving these two lines to the start of the program everything wordks fine
Anyone had this problem before?
I just noticed that these white lines contain portions from other images I'm processing in the same program
Regards
Because you did not initialize the image with any values, you just defined the size and type, you will get random pixels (or not so random, it is probably showing pieces of pixels in memory).
It is the same concept of using/accessing an uninitialized variable.
To paint the image black you can use Mat::setTo, docs here:
http://docs.opencv.org/modules/core/doc/basic_structures.html#mat-setto