Extracting Background Image Using GrabCut - c++

I've an image (.jpg image), and I want to extract the background from the original image. I've googled a lot but have only found tutorials of extracting foreground image.
I've taken the code from another stackoverflow question. The code is working fine for me, and I've successfully extracted the foreground (as per my requirements). Now I want to completely remove this foreground from the original image. I want it to be something like this:-
Background = Original Image - Foreground
The empty space can be filled with black or white color. How can I achieve this?
I've tried using this technique:-
Mat background = image2 - foreground;
but it gives a complete black image.
Code:-
#include <opencv2/opencv.hpp>
#include <iostream>
using namespace cv;
using namespace std;
int main( )
{
// Open another image
Mat image;
image= cv::imread("images/abc.jpg");
Mat image2 = image.clone();
// define bounding rectangle
cv::Rect rectangle(40,90,image.cols-80,image.rows-170);
cv::Mat result; // segmentation result (4 possible values)
cv::Mat bgModel,fgModel; // the models (internally used)
// GrabCut segmentation
cv::grabCut(image, // input image
result, // segmentation result
rectangle,// rectangle containing foreground
bgModel,fgModel, // models
1, // number of iterations
cv::GC_INIT_WITH_RECT); // use rectangle
cout << "oks pa dito" <<endl;
// Get the pixels marked as likely foreground
cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
// Generate output image
cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
//cv::Mat background(image.size(),CV_8UC3,cv::Scalar(255,255,255));
image.copyTo(foreground,result); // bg pixels not copied
// draw rectangle on original image
cv::rectangle(image, rectangle, cv::Scalar(255,255,255),1);
imwrite("img_1.jpg",image);
imwrite("Foreground.jpg",foreground);
Mat background = image2 - foreground;
imwrite("Background.jpg",background);
return 0;
}
Note: I'm an opencv beginner and don't have much knowledge of it right now. I shall be very thankful to you if you can either post the complete code (as required by me) or just post the lines of code and tell me where these lines of code be placed. Thanks.
P.S. This is my second question at StackOverflow.com. apologies ... if not following any convention.

Instead of copying all the pixels that are foreground, it copies all pixels which are not foreground. You can do this by using ~, which negates the mask:
image.copyTo(background,~result);

What if you //Get the pixels marked as likely background:
// Get the pixels marked as likely background
cv::compare(result,cv::GC_PR_BGD,result,cv::CMP_EQ);
Edit: The above code is missing GC_BGD pixels. Despite a more efficient answer was given, let's finish what we started:
// Get the pixels marked as background
cv::compare(result,cv::GC_BGD,result_a,cv::CMP_EQ);
// Get the pixels marked as likely background
cv::compare(result,cv::GC_PR_BGD,result_b,cv::CMP_EQ);
// Final results
result=result_a+result_b;

Just a small suggestion,#William's
answer can be written more concisely as:
result = result & 1;
in order to get the binary mask.

Maybe another example helps, in which I assumed that the middle portion of the image is definitely foreground.
So try this link.
Example

Related

OpenCV function to merge Mat's except for black/transparent pixels

I have a Mat containing text with a black background and a Mat containing a background. I want to merge the 2 Mat's so the text sits ontop of the background. Both Mats are BGR (CV_8UC3) and the same size (cols and rows).
Whats the OpenCV way of doing this?
I'm aware of a couple of functions but |= and += are changing the text colour and addWeighted() only works if I want one image to be slightly transparent. I just want to do a complete merge of the two (except not the black pixels in the text image). I'm looking for an OpenCV function rather than pixel editting myself, well a LUT would be good if thats an option.
Try this:
sourceText = yourTextImage;
sourceBackground = yourBackgroundImage;
cv::Mat textTransparent;
cv::inRange(sourceText, cv::Scalar(0,0,0), cv::Scalar(0,0,0), textTransparent);
cv::imshow("transparent pixel", textTransparent); // this should show all the transparent pixel as being white. If that's not the case, adjust the range.
std::cout << "is the transparency mask ok? Press any key with openCV window focus." << std::endl;
cv::Mat result = sourceBackground.clone();
sourceText.copyTo(sourceBackground, 255-textTransparent); // copy pixels, using the created mask.
instead you could copy the transparent pixels from the sourceBackground to the sourceText which is a bit faster because you don't need the 255-textTransparent...
Could not test the code, please tell me if there are bugs.

Merging rgb and depth images from a kinect

I'm creating a vision algorithm that is implemented in a Simulink S-function( which is c++ code). I accomplished every thing wanted except the alignment of the color and depth image.
My question is how can i make the 2 images correspond to each other. in other words how can i make a 3d image with opencv.
I know my question might be a little vague so i will include my code which will explain the question
#include "opencv2/opencv.hpp"
using namespace cv;
int main(int argc, char** argv)
{
// reading in the color and depth image
Mat color = imread("whitepaint_col.PNG", CV_LOAD_IMAGE_UNCHANGED);
Mat depth = imread("whitepaint_dep.PNG", CV_LOAD_IMAGE_UNCHANGED);
// show bouth the color and depth image
namedWindow("color", CV_WINDOW_AUTOSIZE);
imshow("color", color);
namedWindow("depth", CV_WINDOW_AUTOSIZE);
imshow("depth", depth);
// thershold the color image for the color white
Mat onlywhite;
inRange(color, Scalar(200, 200, 200), Scalar(255, 255, 255), onlywhite);
//display the mask
namedWindow("onlywhite", CV_WINDOW_AUTOSIZE);
imshow("onlywhite", onlywhite);
// apply the mask to the depth image
Mat nocalibration;
depth.copyTo(nocalibration, onlywhite);
//show the result
namedWindow("nocalibration", CV_WINDOW_AUTOSIZE);
imshow("nocalibration", nocalibration);
waitKey(0);
destroyAllWindows;
return 0;
}
output of the program:
As can be seen in the output of my program when i apply the onlywhite mask to the depth image the quad copter body does not consist out of 1 color. The reason for this is that there is a miss match between the 2 images.
I know that i need calibration parameters of my camera and i got these from the last person who worked with this setup. Did the calibration in Matlab and this resulted in the following.
Matlab calibration esults:
I have spent allot of time reading the following opencv page about Camera Calibration and 3D Reconstruction ( cannot include the link because of stack exchange lvl)
But i cannot for the life of me figure out how i could accomplish my goal of adding the correct depth value to each colored pixel.
I tried using reprojectImageTo3D() but i cannot figure out the Q matrix.
i also tried allot of other functions from that page but i cannot seem to get my inputs correct.
As far as I know, Matlab has very good support for Kinect (especially for v1). You may use a function named alignColorToDepth, as follows:
[alignedFlippedImage,flippedDepthImage] = alignColorToDepth(depthImage,colorImage,depthDevice)
The returned values are alignedFlippedImage (the RGB registrated image) and flippedDepthImage (the registrated depth image). These two images are aligned and ready for you to process them.
You can find more at this MathWorks documentation page.
Hope it's what you need :)
As far as I can tell, you are missing the transformation between camera coordinate frames. The Kinect (v1 and v2) uses two separate camera systems to capture the depth and RGB data, and so there is a translation and rotation between them. You may be able to assume no rotation, but you will have to account for the translation to fix the misalignment you are seeing.
Try starting with this thread.

How to increase brightness in specify part of image using Histogram Equalization

Here I did coding for increasing brightness using Histogram Equalization but it changes the overall brightness of images. And I need brightness in specify location.
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include <iostream>
using namespace cv;
using namespace std;
int main( int argc, const char** argv {
Mat img = imread("MyPic.JPG", CV_LOAD_IMAGE_COLOR);
if (img.empty())
{
cout << "Image cannot be loaded..!!" << endl;
return -1;
}
cvtColor(img, img, CV_BGR2GRAY);
Mat img_hist_equalized;
equalizeHist(img, img_hist_equalized); //equalize the histogram
namedWindow("Original Image", CV_WINDOW_AUTOSIZE);
namedWindow("Histogram Equalized", CV_WINDOW_AUTOSIZE);
imshow("Original Image", img);
imshow("Histogram Equalized", img_hist_equalized);
waitKey(0); //wait for key press
destroyAllWindows(); //destroy all open windows
return 0;}
Input image:
Output I gets:
But I expected output
The above code is based on the Histogram Equalization.Is there any other approach means specify here.
The answer on your question is in the method, used for generation of last image with expected output. Which commands of graphical editor have you used to generate this result? Just repeat it with OpenCV.
You can try something like this:
increase contrast,
reduce number of colors to 4,
replace all three "near black" colors by real black, and replace one
"near white" color by real white.
You will have something like this (top image after second step, bottom image after third step):
Is it good enought for your task? Do you need better quality? Any way, try to process your image by any graphical editor before coding. And only when you understand, which operations you need to apply to your image, try to implement it using OpenCV.
To understand better the problem, let's see the histogram of your initial image:
We can see, that difference between "expected black" and "expected white" colors are very small. More over, inside "white circle" we can see pixels with "expected black colors". So it is not enough to change histogram for fixing of this mix of pixels. We need to analyze surroundings of each pixel (in fact, I done it in first step - when increased contrast of the image). We can discuss methods for it, but first of all we need more information and efforts from you. Editing of the pallet without analyzing of surroundings will give you strange results like this:
So, you need to open any graphical editor and find way to convert your input to expected output.

How to use BackgroundSubtractorMOG2 for images

I am pretty new to OpenCV and I am stuck at the moment. I am dealing with images, not a video. Since I will have same background in my project, I thought it would be easier to work, if I could remove my background. But first, I have to ask one thing. Can I use BackgroundSubtractorMOG2 for images? Because it is under video analysis/motion analysis title.
I read the documentation on opencv.org and looked through countless examples/tutorials but I am still having difficulty understanding how MOG2 works.
Quick question: What is history that in parameters?
So, I have written a simple code. I get a foreground mask. So, what is the next step? How can I remove the background and left with my object only? Shouldn't I load my background first, then the actual image, so that MOG2 could do the background subtraction?
I am using OpenCV 2.4.11.
Code:
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/video/background_segm.hpp>
using namespace cv;
using namespace std;
//global variables
int history = 1;
float varThreshold = 16;
bool bShadowDetection = true;
Mat src; //source image
Mat fgMaskMOG2; //fg mask generated by MOG2 method
Ptr<BackgroundSubtractor> pMOG2; //MOG2 Background subtractor
int main(int argc, char* argv[])
{
//create GUI windows
namedWindow("Source");
namedWindow("FG Mask MOG 2");
src = imread("bluePaper1.png", 1);
//create Background Subtractor objects
pMOG2 = new BackgroundSubtractorMOG2(history, varThreshold, bShadowDetection); //MOG2 approach
pMOG2->setInt("nmixtures", 3);
pMOG2->setDouble("fTau", 0.5);
pMOG2->operator()(src, fgMaskMOG2);
imshow("Source", src);
imshow("FG Mask MOG 2", fgMaskMOG2);
waitKey(0);
return 0;
}
Source image:
fgMask that I get from MOG2:
Mixture of Gaussian method learns background according to history of frames in a fixed camera and so you can not use it for only one image. The history parameter shows how many frames would have effect on construction of the background.
Shadow detection is not a process which depends on BGS method and should be implemented alongside.
for example in MOG2 documentation we have:
The shadow is detected if the pixel is a darker version of the background. Tau is a threshold defining how much darker the shadow can be. Tau= 0.5 means that if a pixel is more than twice darker then it is not shadow
In case of your example the foreground could easily be obtained by a simple frame difference and you can easily remove shadows by the mentioned solution.
you can have the foreground by the following steps:
Subtract given image from known background and threshold the result to obtain the foreground mask
Apply AND operation on foreground mask and the given image to get your object with possible shadows.
Remove pixels which is darker (amount of it should be adjust) than their corresponding pixel in background.
Do some post processing like morphological and connected-component-labeling to have a better result.

What happens in GrabCut Algorithm

I want to know the what actually happnens in the following code.
cv::Rect rectangle(x,y,width,height);
cv::Mat result;
cv::Mat bgModel,fgModel;
cv::grabCut(image,
result,
rectangle,
bgModel,fgModel,
1,
cv::GC_INIT_WITH_RECT);
cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
// Generate output image
cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
image.copyTo(foreground,result);
According to my knowledge when we define the rectangle outside the rectangle will be consider as the known background and the inside as the unknown foreground.
Then bgmodel and fgmodel are Gausian mixed models which keeps the foreground and background pixels separately.
The parameter we are passing as 1 means, we are asking to divide the pixels to separate pixels process to run only once.
what i can't understand is
cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
What actually happens in the above method.
If anyone could explain, that would be a great help.
thanx.
I've found this book which tells that this code compares values of pixels in result to GC_PR_FGD value. Those that are equal it doesn't touch, all the other pixels it deletes. Here's a citation from there:
The input/output segmentation image can have one of the four values:
cv::GC_BGD, for pixels certainly belonging to the background (for example, pixels outside the rectangle in our example)
cv::GC_FGD, for pixels certainly belonging to the foreground (none in our example)
cv::GC_PR_BGD, for pixels probably belonging to the background
cv::GC_PR_FGD for pixels probably belonging to the foreground (that is the initial value for the pixels inside the rectangle in our
example).
We get a binary image of the segmentation by extracting the pixels
having a value equal to cv::GC_PR_FGD:
// Get the pixels marked as likely foreground
cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
// Generate output image
cv::Mat foreground(image.size(),CV_8UC3,
cv::Scalar(255,255,255));
image.copyTo(foreground, // bg pixels are not copied
result);