OpenCV keep background transparent during warpAffine - c++

I create a Bird-View-Image with the warpPerspective()-function like this:
warpPerspective(frame, result, H, result.size(), CV_WARP_INVERSE_MAP, BORDER_TRANSPARENT);
The result looks very good and also the border is transparent:
Bird-View-Image
Now I want to put this image on top of another image "out". I try doing this with the function warpAffine like this:
warpAffine(result, out, M, out.size(), CV_INTER_LINEAR, BORDER_TRANSPARENT);
I also converted "out" to a four channel image with alpha channel according to a question which was already asked on stackoverflow:
Convert Image
This is the code: cvtColor(out, out, CV_BGR2BGRA);
I expected to see the chessboard but not the gray background. But in fact, my result looks like this:
Result Image
What am I doing wrong? Do I forget something to do? Is there another way to solve my problem? Any help is appreciated :)
Thanks!
Best regards
DamBedEi

I hope there is a better way, but here it is something you could do:
Do warpaffine normally (without the transparency thing)
Find the contour that encloses the image warped
Use this contour for creating a mask (white values inside the image warped, blacks in the borders)
Use this mask for copy the image warped into the other image
Sample code:
// load images
cv::Mat image2 = cv::imread("lena.png");
cv::Mat image = cv::imread("IKnowOpencv.jpg");
cv::resize(image, image, image2.size());
// perform warp perspective
std::vector<cv::Point2f> prev;
prev.push_back(cv::Point2f(-30,-60));
prev.push_back(cv::Point2f(image.cols+50,-50));
prev.push_back(cv::Point2f(image.cols+100,image.rows+50));
prev.push_back(cv::Point2f(-50,image.rows+50 ));
std::vector<cv::Point2f> post;
post.push_back(cv::Point2f(0,0));
post.push_back(cv::Point2f(image.cols-1,0));
post.push_back(cv::Point2f(image.cols-1,image.rows-1));
post.push_back(cv::Point2f(0,image.rows-1));
cv::Mat homography = cv::findHomography(prev, post);
cv::Mat imageWarped;
cv::warpPerspective(image, imageWarped, homography, image.size());
// find external contour and create mask
std::vector<std::vector<cv::Point> > contours;
cv::Mat imageWarpedCloned = imageWarped.clone(); // clone the image because findContours will modify it
cv::cvtColor(imageWarpedCloned, imageWarpedCloned, CV_BGR2GRAY); //only if the image is BGR
cv::findContours (imageWarpedCloned, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE);
// create mask
cv::Mat mask = cv::Mat::zeros(image.size(), CV_8U);
cv::drawContours(mask, contours, 0, cv::Scalar(255), -1);
// copy warped image into image2 using the mask
cv::erode(mask, mask, cv::Mat()); // for avoid artefacts
imageWarped.copyTo(image2, mask); // copy the image using the mask
//show images
cv::imshow("imageWarpedCloned", imageWarpedCloned);
cv::imshow("warped", imageWarped);
cv::imshow("image2", image2);
cv::waitKey();

One of the easiest ways to approach this (not necessarily the most efficient) is to warp the image twice, but set the OpenCV constant boundary value to different values each time (i.e. zero the first time and 255 the second time). These constant values should be chosen towards the minimum and maximum values in the image.
Then it is easy to find a binary mask where the two warp values are close to equal.
More importantly, you can also create a transparency effect through simple algebra like the following:
new_image = np.float32((warp_const_255 - warp_const_0) *
preferred_bkg_img) / 255.0 + np.float32(warp_const_0)
The main reason I prefer this method is that openCV seems to interpolate smoothly down (or up) to the constant value at the image edges. A fully binary mask will pick up these dark or light fringe areas as artifacts. The above method acts more like true transparency and blends properly with the preferred background.

Here's a small test program that warps with transparent "border", then copies the warped image to a solid background.
int main()
{
cv::Mat input = cv::imread("../inputData/Lenna.png");
cv::Mat transparentInput, transparentWarped;
cv::cvtColor(input, transparentInput, CV_BGR2BGRA);
//transparentInput = input.clone();
// create sample transformation mat
cv::Mat M = cv::Mat::eye(2,3, CV_64FC1);
// as a sample, just scale down and translate a little:
M.at<double>(0,0) = 0.3;
M.at<double>(0,2) = 100;
M.at<double>(1,1) = 0.3;
M.at<double>(1,2) = 100;
// warp to same size with transparent border:
cv::warpAffine(transparentInput, transparentWarped, M, transparentInput.size(), CV_INTER_LINEAR, cv::BORDER_TRANSPARENT);
// NOW: merge image with background, here I use the original image as background:
cv::Mat background = input;
// create output buffer with same size as input
cv::Mat outputImage = input.clone();
for(int j=0; j<transparentWarped.rows; ++j)
for(int i=0; i<transparentWarped.cols; ++i)
{
cv::Scalar pixWarped = transparentWarped.at<cv::Vec4b>(j,i);
cv::Scalar pixBackground = background.at<cv::Vec3b>(j,i);
float transparency = pixWarped[3] / 255.0f; // pixel value: 0 (0.0f) = fully transparent, 255 (1.0f) = fully solid
outputImage.at<cv::Vec3b>(j,i)[0] = transparency * pixWarped[0] + (1.0f-transparency)*pixBackground[0];
outputImage.at<cv::Vec3b>(j,i)[1] = transparency * pixWarped[1] + (1.0f-transparency)*pixBackground[1];
outputImage.at<cv::Vec3b>(j,i)[2] = transparency * pixWarped[2] + (1.0f-transparency)*pixBackground[2];
}
cv::imshow("warped", outputImage);
cv::imshow("input", input);
cv::imwrite("../outputData/TransparentWarped.png", outputImage);
cv::waitKey(0);
return 0;
}
I use this as input:
and get this output:
which looks like ALPHA channel isn't set to ZERO by warpAffine but to something like 205...
But in general this is the way I would do it (unoptimized)

Related

how to change the black pixels in the below image to red pixels using opencv libraries with c++

I want to change the black pixels in the image to red pixels, such that the ball should look white and red. I want to use OpenCV libraries and code it in C++. I have tried converting the image to RGB.
Common approach is to threshold the image, so in your case you would say that each pixel with an intensity less than some threshold will be considered as being black and then recolored to red. One way to find a good threshold (that divides the image's pixel into two classes ("more black" and "more white") is OTSU thresholding:
int main()
{
cv::Mat input = cv::imread("../inputData/ball_thresholding.jpg");
cv::Mat gray;
cv::cvtColor(input,gray,CV_BGR2GRAY);
cv::Mat mask;
// compute inverse thresholding (dark areas become "active" pixel in the mask) with OTSU thresholding:
double grayThres = cv::threshold(gray, mask, 0, 255, CV_THRESH_BINARY_INV | CV_THRESH_OTSU);
// color all masked pixel red:
input.setTo(cv::Scalar(0,0,255), mask);
// compute median filter to remove the whitish black parts and darker white parts
cv::imshow("input", input);
cv::waitKey(0);
return 0;
}
Giving this mask:
and this result:
For this image, the threshold that was computed by OTSU is 127, which means that each grayscale pixel intensity of 127 or less (or less than 127, I'm not sure) will be recolored to red.
If you want to keep the shading effect withing the black/red region, you can remove input.setTo(cv::Scalar(0,0,255), mask); lind and replace it by:
// keep the shading:
for(int j=0; j<input.rows; ++j)
for(int i=0; i<input.cols; ++i)
{
if(mask.at<unsigned char>(j,i))
{
input.at<cv::Vec3b>(j,i)[2] = 255;
}
}
which will result int:
cv::Mat imBW = imread('bwImg.jpg',CV_LOAD_IMAGE_GRAYSCALE);
cv::Mat RGB_img = cv::Mat(imBW.rows, imBW.cols, CV_8UC3);
cv::Mat R_channel = 255-imBW;
cv::Mat B_channel = cv::Mat::zeros(imBW.rows, imBW.cols, CV_8UC1);
cv::Mat G_channel = cv::Mat::zeros(imBW.rows, imBW.cols, CV_8UC1);
vector<cv::Mat> channels;
channels.push_back(B_channel);
channels.push_back(G_channel);
channels.push_back(R_channel);
cv::merge(channels, RGB_img);

How to use Multi-band Blender in opencv

I want to blend two images using multiband blending but I am not clear to the input parameter of this function:
void detail::Blender::prepare(const std::vector<Point>& corners, const std::vector<Size>& sizes)
In my case ,I just input two warped images with black gap, and with masks all white.(forgive me can not add pictures...)
And I set the two corners (0.0,0.0),because the warped images has been registered.
but my result is not good enough.with obvious seam in the result
can someone tell me why?How can I solve this problem?
I'm not sure what do you mean when you say "my result is not good enough". It's better to watch that result, but I'll try to guess. My main part of code, which makes panorama, looks like this:
void makePanorama(Rect bounding_box, vector<Mat> images, vector<Mat> homographies, vector<vector<Point>> corners) {
detail::MultiBandBlender blender;
blender.prepare(bounding_box);
Mat mask, bigImage, curImage;
for (int i = 0; i < (int)images.size(); ++i) {
warpPerspective(images[i], curImage, homographies[i],
bounding_box.size(), INTER_LINEAR, ORDER_TRANSPARENT);
mask = makeMask(curImage.size(), corners[i], homographies[i]);
blender.feed(curImage.clone(), mask, Point(0, 0));
}
blender.blend(bigImage, mask);
bigImage.convertTo(bigImage, (bigImage.type() / 8) * 8);
imshow("Result", bigImage);
waitKey();
}
So, prepare blender and then loop: warp image, make the mask after warped image and feed blender. At the end, turn this blender on and that's all. I met two problems, which influence on my result badly. May be you have one of them or both.
The first is type. My images had CV_16SC3, and after blending you need to convert blended image type into unsigned one. Like this
bigImage.convertTo(bigImage, (bigImage.type() / 8) * 8);
If you not, the result image would be gray.
The second is borders. In the beginning, my function makeMask was calculating non-black area of warped images. As a result, the one could see borders of the warped images on the blended image. The solution is to make mask smaller than non-black warped image area. So, my function makeMask is looks like this:
Mat makeMask(Size sz, vector<Point2f> imageCorners, Mat homorgaphy) {
Scalar white(255, 255, 255);
Mat mask = Mat::zeros(sz, CV_8U);
Point2f innerPoint;
vector<Point2f> transformedCorners(4);
perspectiveTransform(imageCorners, transformedCorners, homorgaphy);
// Calculate inner point
for (auto& point : transformedCorners)
innerPoint += point;
innerPoint.x /= 4;
innerPoint.y /= 4;
// Make indent for each corner
vector<Point> corners;
for (int ind = 0; ind < 4; ++ind) {
Point2f direction = innerPoint - transformedCorners[ind];
double normOfDirection = norm(direction);
corners[ind].x += settings.indent * direction.x / normOfDirection;
corners[ind].y += settings.indent * direction.y / normOfDirection;
}
// Draw borders
Point prevPoint = corners[3];
for (auto& point : corners) {
line(mask, prevPoint, point, white);
prevPoint = point;
}
// Fill with white
floodFill(mask, innerPoint, white);
return mask;
}
I took this pieces of code from my real code, so I could possibly forget to specify something. But I hope, the idea of how to work with MultiBandBlender is clear.

Getting masked area to be transparent?

So far i have managed to use masks and get the second image from the first. But what i want is the black area in second image to be transparent (i.e the output i an trying to get is the third image) Here is the code so far. Please advice me on this.
EDIT: Third one is from photoshop
//imwrite parameters
compression_params.push_back(CV_IMWRITE_JPEG_QUALITY);
compression_params.push_back(100);
//reading image to be masked
image = imread(main_img, -1);
//CV_LOAD_IMAGE_COLOR
namedWindow("output", WINDOW_NORMAL);
//imshow("output", image);
//Creating mask image with same size as original image
Mat mask(image.rows, image.cols, CV_8UC1, Scalar(0));
// Create Polygon from vertices
ROI_Vertices.push_back(Point2f(float(3112),float(58)));
ROI_Vertices.push_back(Point2f(float(3515),float(58)));
ROI_Vertices.push_back(Point2f(float(3515),float(1332)));
ROI_Vertices.push_back(Point2f(float(3112),float(958)));
approxPolyDP(ROI_Vertices, ROI_Poly, 1, true);
// Fill polygon white
fillConvexPoly(mask, &ROI_Poly[0] , ROI_Poly.size(), 255, 8, 0);
//imshow("output", mask);
// Create new image for result storage
imageDest = cvCreateMat(image.rows, image.cols, CV_8UC4);
// Cut out ROI and store it in imageDest
image.copyTo(imageDest, mask);
imwrite("masked.jpeg", imageDest, compression_params);
imshow("output", imageDest);
cvWaitKey(0);
This can be done by first setting its alpha value to 0 of the regions that you want to make them fully transparent (255 for others), and then save it to PNG.
To set the alpha value of pixel-(x,y), it can be done:
image.at<cv::Vec4b>(y, x)[3] = 0;
PS: you need to convert it to 4-channel format first if the image is not currently. For example:
cv::cvtColor(image, image, CV_BGR2BGRA);
Updated: It will be easier if you have already computed the mask for the ROI region, where you can simply merge it with the original image (assume having 3 channels) to get the final result. Like:
cv::Mat mask; // 0 for transparent regions, 255 otherwise (serve as the alpha channel)
std::vector<cv::Mat> channels;
cv::split(image, channels);
channels.push_back(mask);
cv::Mat result;
cv::merge(channels, result);

GrabCut reading mask from PNG file in OpenCV (C++)

The implementation of this functionality seems pretty straightforward in Python, as shown here: http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html
Yet, when I tried to do exactly the same in C++, I get bad arguments error (for the grabcut function). How to put the mask image in the right format?
I am a newbie at this, so I'd be very thankful if someone could help me understand better. Thank you!
Here's what I have so far:
Mat image;
image= imread(file);
Mat mask;
mask.setTo( GC_BGD );
mask = imread("messi5.png");
Mat image2 = image.clone();
// define bounding rectangle
cv::Rect rectangle(startX, startY, width, height);
cv::Mat result; // segmentation result (4 possible values)
cv::Mat bgModel,fgModel; // the models (internally used)
//// GrabCut segmentation that works, but with a rectangle, not with the mask I need
//cv::grabCut(image, // input image
// result, // segmentation result
// rectangle,// rectangle containing foreground
// bgModel,fgModel, // models
// 1, // number of iterations
// cv::GC_INIT_WITH_RECT); // use rectangle
grabCut( image, mask, rectangle, bgModel, fgModel, 1, GC_INIT_WITH_MASK);
cv::compare(mask,cv::GC_PR_FGD,mask,cv::CMP_EQ);
cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
image.copyTo(foreground,mask); // bg pixels not copied
namedWindow( "Display window", WINDOW_AUTOSIZE );
imshow( "Display window", foreground );
waitKey(0);
return 0;
}
It looks like you have misunderstood the guide, repeated here from the linked guide in the question:
# newmask is the mask image I manually labelled
newmask = cv2.imread('newmask.png',0)
# whereever it is marked white (sure foreground), change mask=1
# whereever it is marked black (sure background), change mask=0
mask[newmask == 0] = 0
mask[newmask == 255] = 1
mask, bgdModel, fgdModel = cv2.grabCut(img,mask,None,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_MASK)
mask = np.where((mask==2)|(mask==0),0,1).astype('uint8')
img = img*mask[:,:,np.newaxis]
plt.imshow(img),plt.colorbar(),plt.show()
this is not what you have done i'm afraid. For a start you seem to have set the mask to the rgb image:
mask = imread("messi5.png");
whereas is should be set to the mask image:
mask = imread("newmask.png",CV_LOAD_IMAGE_GRAYSCALE);
EDIT from comments:
from a pure red mask painted over the image (an actual mask would be better).
maskTmp = imread("messi5.png");
std::vector<cv::Mat> channels(3)
split( messi5, channels);
cv::Mat maskRed = channels[2];
now threshold on the red channel to get your binary mask.

copying ipl image pixel by pixel

The problem is solved....I used cvGet2D,below is the sample code
CvScalar s;
s=cvGet2D(src_Image,pixel[i].x,pixel[i].y);
cvSet2D(dst_Image,pixel[i].x,pixel[i].y,s);
Where src_Iamge and dst_Image is the source and destination image correspondingly and pixel[i] is the selected pixel i wanted to draw in the dst image. I have include the real out image below.
have an source Ipl image, I want to copy some of the part of the image to a new destination image pixel by pixel. can any body tell me how can do it? I use c,c++ in opencv. For example if the below image is source image,
The real output image
EDIT:
I can see the comments suggesting cvGet2d. I think, if you just want to show "points", it is best to show them with a small neighbourhood so they can be seen where they are. For that you can draw white filled circles with origins at (x,y), on a mask, then you do the copyTo.
using namespace cv;
Mat m(input_iplimage);
Mat mask=Mat::zeros(m.size(), CV_8UC1);
p1 = Point(x,y);
r = 3;
circle(mask,p1,r, 1); // draws the circle around your point.
floodFill(mask, p1, 1); // fills the circle.
//p2, p3, ...
Mat output = Mat::zeros(m.size(),m.type()); // output starts with a black background.
m.copyTo(output, mask); // copies the selected parts of m to output
OLD post:
Create a mask and copy those pixels:
#include<opencv2/opencv.hpp>
using namespace cv;
Mat m(input_iplimage);
Mat mask=Mat::zeros(m.size(), CV_8UC1); // set mask 1 for every pixel you wanna copy.
Rect roi=Rect(x,y,width,height); // create a rectangle
mask(roi) = 1; // set it to 0.
roi = Rect(x2,y2,w2,h2);
mask(roi)=1; // set the second rectangular area for copying...
Mat output = 100*Mat::ones(m.size(),m.type()); // output with a gray background.
m.copyTo(output, mask); // copy selected areas of m to output
Alternatively you can copy Rect-by-Rect:
Mat m(input_iplimage);
Mat output = 100*Mat::ones(m.size(),m.type()); // output with a gray background.
Rect roi=Rect(x,y,width,height);
Mat m_temp, out_temp;
m_temp=m(roi);
out_temp = output(roi);
m_temp.copyTo(out_temp);
roi=Rect(x2,y2,w2,h2);
Mat m_temp, out_temp;
m_temp=m(roi);
out_temp = output(roi);
m_temp.copyTo(out_temp);
The answer to your question only requires to have look at the OpenCV documentation or just to search in your favourite search engine.
Here you've an answer for Ipl images and for newer Mat data.
For having an output as I see in your images, I'd do it setting ROI's, it's more efficient.