I am interested in perspective transformation to bird's eye view. So far I have tried getPerspectiveTransform and findHomography and then passing it onto warpPerspective. The results are quite close but a skew in TL and BR is present. Also the contourArea are not translated equally post transformation.
The contour is a square with multiple shapes inside.
Any suggestion on how to go ahead.
Code block of what I have done so far.
std::vector<Point2f> quad_pts;
std::vector<Point2f> squre_pts;
cv::approxPolyDP( Mat(validContours[largest_contour_index]), contours_poly[0], epsilon, true );
if (approx_poly.size() > 4) return false;
for (int i=0; i< 4; i++)
quad_pts.push_back(contours_poly[0][i]);
if (! orderRectPoints(quad_pts))
return false;
float widthTop = (float)distanceBetweenPoints(quad_pts[1], quad_pts[0]); // sqrt( pow(quad_pts[1].x - quad_pts[0].x, 2) + pow(quad_pts[1].y - quad_pts[0].y, 2));
float widthBottom = (float)distanceBetweenPoints(quad_pts[2], quad_pts[3]); // sqrt( pow(quad_pts[2].x - quad_pts[3].x, 2) + pow(quad_pts[2].y - quad_pts[3].y, 2));
float maxWidth = max(widthTop, widthBottom);
float heightLeft = (float)distanceBetweenPoints(quad_pts[1], quad_pts[2]); // sqrt( pow(quad_pts[1].x - quad_pts[2].x, 2) + pow(quad_pts[1].y - quad_pts[2].y, 2));
float heightRight = (float)distanceBetweenPoints(quad_pts[0], quad_pts[3]); // sqrt( pow(quad_pts[0].x - quad_pts[3].x, 2) + pow(quad_pts[0].y - quad_pts[3].y, 2));
float maxHeight = max(heightLeft, heightRight);
int mDist = (int)max(maxWidth, maxHeight);
// transform TO points
const int offset = 50;
squre_pts.push_back(Point2f(offset, offset));
squre_pts.push_back(Point2f(mDist-1, offset));
squre_pts.push_back(Point2f(mDist-1, mDist-1));
squre_pts.push_back(Point2f(offset, mDist-1));
maxWidth += offset; maxHeight += offset;
Size matSize ((int)maxWidth, (int)maxHeight);
Mat transmtx = getPerspectiveTransform(quad_pts, squre_pts);
// Mat homo = findHomography(quad_pts, squre_pts);
warpPerspective(mRgba, mRgba, transmtx, matSize);
return true;
Link to transformed image
Image pre-transformation
corner on pre-transformed image
Corners from CornerSubPix
Your original pre-transformation image is not so good, the squares have different sizes there and it looks wavy. The results you get are quite good given the quality of your input.
You could try to calibrate your camera (https://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html) to compensate lens distortion, and your results may improve.
EDIT: Just to summarize the comments below, approxPolyDp may not locate the corners properly if the square has rounded corners or it is blurred. You may need to improve the corner location by other means such as a sharper original image, different preprocessing (median filter or threshold, as you suggest in the comments), or other algorithms for finer corner location (such as using the cornersubpix function or detecting the sides with Hough Transform and then calculating the intersections of them)
Related
My idea was to implement a simple square detection using openCV & C++ (objc++). I've already extracted the biggest areas of the image like you can see below (the colored ones) but now I'd like to extract the corner points (like TopLeft, TopRight, BottomLeft, BottomRight) of all the areas to afterwards check if the distance between the 4 corners is similar on each of them or the angle between the lines is nearly 45°.
See the images I was talking about:
However - I got to this point where I've tried to extract the areas corner points to get something like this afterwards:
This was my first idea how to get the 4 corner points (See the steps below):
1.calculate the contours center by
for (int i=0; i<contourPoints.size(); i++) {
avgx += contourPoints[i].x;
avgy += contourPoints[i].y;
}
avgx/=contourPoints.size(); // centerx
avgy/=contourPoints.size(); // centery
2.loop trough all contour points to get the points with the highest distance to the center --> Probably the corners if the contour is a square/rectangle
for (int i=0; i<contourPoints.size(); i++) {
dx = abs(avgx - contourPoints[i].x);
dy = abs(avgy - contourPoints[i].y);
dist = sqrt( dx*dx + dy*dy );
distvector.push_back(dist);
}
// sort distvector > distvector and get 4 corners with highest distance to the center -> hopefully the corners.
This procedure was my idea but I'm pretty sure there must be a better way to detect squares and extract it's corner points using just the given contour coordinates.
So any help how to improve my code to get a way better & more efficient detection would be very appreciated. Thanks a million in advance, Tempi.
I´m trying to use a chessboard pattern, to get the information of the cylinder map and rectifie the "distortion" so that image shows the cap surface unrolled. I made a first test with a one shot calibration and cv::fisheye::undistortImage to get a un-distortion (attached two images).
*//runCalibrationFishEye
void runCalibrationFishEye(cv::Mat& image, cv::Matx33d&, cv::Vec4d&);
cv::Mat removeFisheyeLensDist(cv::Mat&, cv::Matx33d&, cv::Vec4d&);*
It is to remark that i am not interested in calibrate the image, to get metric values. I just want to use the chessboard information to unroll the image on the cylinder surface.
The final aim is to use the rectified images of 4 cameras and to stitch the rectified images to one unrolled image.
Do i need to make a full calibration of the camera? Or is there another way to get a remap of the cylinder surface?
I will try to implement this interesting unwarp method: https://dsp.stackexchange.com/questions/2406/how-to-flatten-the-image-of-a-label-on-a-food-jar/2409#2409
cap with chessboard
Rectification
I have found a similar approach, of another problem but with a similar Mathematics. And it was solved without a calibration pattern. Link here. Its a approximation, but the result is quite good enough.
the user Hammer gave an answer that helped me to get a solution. I have changed the way he do the mapping, using OpenCV remap. The formula to recalculate the coordinates is just as he gave it, using different values, and making a preprocessing to adjust the image (Rotation, zoom, and other adjustments).Unrolled image. I am now improving the distortion of the edges, so that it is not so pronounced at the edges. But the main question is solved.
cv::Point2f convert_pt(cv::Point2f point, int w, int h)
{
cv::Point2f pc(point.x - w / 2, point.y - h / 2);
float f = w;
float r = w;
float omega = w / 2;
float z0 = f - sqrt(r*r - omega*omega);
//Formula para remapear el cylindro
float zc = (2 * z0 + sqrt(4 * z0*z0 - 4 * (pc.x*pc.x / (f*f) + 1)*(z0*z0 - r*r))) / (2 * (pc.x*pc.x / (f*f) + 1));
cv::Point2f final_point(pc.x*zc / f, pc.y*zc / f);
final_point.x += w / 2;
final_point.y += h / 2;
return final_point;
}
I get image from a camera (calibrated and without lens distortions) and I need to detect a rectangular object. Markers are a good example. For markers I check corner count, min size, board contrast and convexity. I had an idea on how to improve this in cases where there is large amount of false rectangles.
Here is an example image:
Normally all of these are valid, because without knowing anything about camera we cannot determine if perspective allows these kinds of shapes. I know the size (or at least the ratio) of the rectangle in real-life. So I had an idea that I should be able to disregard many of these shapes just by reprojecting them and checking for error.
Like if I use solvePnPRansac it would not be able to converge if the shape is not possible. If it doesn't converge I just disregard it. Sadly, none of the OpenCV solve functions allow checking me for error or convergence. I actually need some ratio or quality, because it is possible that some of the rectangles overlap. For example my object finder identifies these rectangles:
One of the three is actually correct, or at least "the best". But I need some way to know which one it is. I cannot use things like line lengths because of the camera perspective. So I just thought I could solve and see which has the smallest error.
There are no lens distortions in the image, but even if there were solvePnP usually allows passing D to it as well.
Is this even possible or am I missing something?
I guess I could try hacking around solvePnPRansac just to return convergence, but maybe there is a simpler way?
I figured I can do something like what is done during calibration with a grid. I can calculate the reprojection error. So first I solve to get the transformation matrix. Then I transform the points in 3D using the transformation matrix and afterwards use projectPoints to project them back in 2D. Then I check distance between original 2D points and the projected 2D points. This can then be used for quality. Objects that are not possible often have 100 pixels or more reprojection error in my images, but possible objects have less than 20px. So I just did a 25 pixel cutoff and it seems to work fine.
Note that more transformations are possible than I though. In my original image maybe two are not possible with my current camera, but it still did reject a lot of fakes.
If nobody else has some ideas I will accept this as answer.
Here is some code for the method I use:
//This is the object in 3D
double width = 50.0; //Object is 50mm wide
double height = 30.0; //Object is 30mm tall
cv::Mat object_points(4,3,CV_64FC1);
object_points.at<double>(0,0)=0;
object_points.at<double>(0,1)=0;
object_points.at<double>(0,2)=0;
object_points.at<double>(1,0)=width;
object_points.at<double>(1,1)=0;
object_points.at<double>(1,2)=0;
object_points.at<double>(2,0)=width;
object_points.at<double>(2,1)=height;
object_points.at<double>(2,2)=0;
object_points.at<double>(3,0)=0;
object_points.at<double>(3,1)=height;
object_points.at<double>(3,2)=0;
//Check all rectangles for error
cv::Mat image_points(4,2,CV_64FC1);
for (size_t i = 0; i < rectangles_to_test.size(); i++) {
// Get rectangle points
for (size_t c = 0; c < 4; ++c) {
image_points.at<double>(c,0) = (rectangles_to_test[i].points[c].x);
image_points.at<double>(c,1) = (rectangles_to_test[i].points[c].y);
}
// Calculate transformation matrix
cv::Mat rvec, tvec;
cv::solvePnP(object_points, image_points, M1, D1, rvec, tvec);
cv::Mat rotation;
Matrix4<double> transform;
transform.init_identity();
cv::Rodrigues(rvec, rotation);
for(size_t row = 0; row < 3; ++row) {
for(size_t col = 0; col < 3; ++col) {
transform.set(row, col, rotation.at<double>(row, col));
}
transform.set(row, 3, tvec.at<double>(row, 0));
}
// Calculate projection
std::vector<cv::Point3f> p3(4);
std::vector<cv::Point2f> p2;
Vector4<double> p = transform * Vector4<double>(0, 0, 0, 1);
p3[0] = cv::Point3f((float)p.x, (float)p.y, (float)p.z);
p = transform * Vector4<double>(width, 0, 0, 1);
p3[1] = cv::Point3f((float)p.x, (float)p.y, (float)p.z);
p = transform * Vector4<double>(width, height, 0, 1);
p3[2] = cv::Point3f((float)p.x, (float)p.y, (float)p.z);
p = transform * Vector4<double>(0, height, 0, 1);
p3[3] = cv::Point3f((float)p.x, (float)p.y, (float)p.z);
cv::projectPoints(p3, cv::Mat::zeros(1, 3, CV_64FC1), cv::Mat::zeros(1, 3, CV_64FC1), M1, D1, p2);
// Calculate reprojection error
rectangles_to_test[i].reprojection_error = 0.0;
for (size_t c = 0; c < 4; ++c) {
double dx = p2[c].x - rectangles_to_test[i].points[c].x;
double dy = p2[c].y - rectangles_to_test[i].points[c].y;
rectangles_to_test[i].reprojection_error += std::sqrt(dx*dx + dy*dy);
}
if (rectangles_to_test[i].reprojection_error > reprojection_error_threshold) {
//rectangle is no good
}
}
So, here is the code for my 2D point class to rotate:
float nx = (x * cos(angle)) - (y * sin(angle));
float ny = (y * cos(angle)) + (x * sin(angle));
x = nx;
y = ny;
x and y are local variables in the point class.
And here is the code for my sprite class's rotation:
//Make clip
SDL_Rect clip;
clip.w = width;
clip.h = height;
clip.x = (width * _frameX) + (sep * (_frameX) + osX);
clip.y = (height * _frameY) + (sep * (_frameY) + osY);
//Make a rotated image
col bgColor = image->format->colorkey;
//Surfaces
img *toEdit = newImage(clip.w, clip.h);
img *toDraw = 0;
//Copy the source into the workspace
drawRect(0, 0, toEdit->w, toEdit->h, toEdit, bgColor);
drawImage(0, 0, image, toEdit, &clip);
//Edit the image
toDraw = SPG_Transform(toEdit, bgColor, angle, xScale, yScale, SPG_NONE);
SDL_SetColorKey(toDraw, SDL_SRCCOLORKEY, bgColor);
//Find new origin and offset by pivot
2DVec *pivot = new xyVec(pvX, pvY);
pivot->rotate(angle);
//Draw and remove the finished image
drawImage(_x - pivot->x - (toDraw->w / 2), _y - pivot->y - (toDraw->h / 2), toDraw, _destination);
//Delete stuff
deleteImage(toEdit);
delete pivot;
deleteImage(toDraw);
The code uses the center of the sprite as the origin. It works fine if I leave the pivot at (0,0), but if I move it somewhere else, the character's shoulder for instance, it starts making the sprite dance around as it spins like a spirograph, instead of the pivot staying on the character's shoulder.
The image rotation function is from SPriG, a library for drawing primitives and transformed images in SDL. Since the pivot is coming from the center of the image, I figure the new size of the clipped surface produced by rotating shouldn't matter.
[EDIT]
I've messed with the code a bit. By slowing it down, I found that for some reason, the vector is rotating 60 times faster than the image, even though I'm not multiplying anything by 60. So, I tried to just divide the input by 60, only now, it's coming out all jerky and not rotating to anything between multiples of 60.
The vector rotation code I found on this very site, and people have repeatedly confirmed that it works, so why does it only rotate in increments of 60?
I haven't touched the source of SPriG in a long time, but I can give you some info.
If SPriG has problems with rotating off of center, it would probably be faster and easier for you to migrate to SDL_gpu (and I suggest SDL 2.0). That way you get a similar API but the performance is much better (it uses the graphics card).
I can guess that the vector does not rotate 60 times faster than the image, but rather more like 57 times faster! This is because you are rotating the vector with sin() and cos(), which accept values in radians. The image is being rotated by an angle in degrees. The conversion factor for radians to degrees is 180/pi, which is about 57. SPriG can use either degrees or radians, but uses degrees by default. Use SPG_EnableRadians(1) to switch that behavior. Alternatively, you can stick to degree measure in your angle variable by multiplying the argument to sin() and cos() by pi/180.
In a project of mine (VC++2010, MFC), I want to draw a circle using the CDC::Ellipse. I set two points: the first one is the center of the circle, the second one is a point I want it to be on the circumference.
I pass to the CDC::Ellipse( int x1, int y1, int x2, int y2 ) the coordinates of the upper-left corner and lower-right one.
Briefly: with Pitagora Theorem I calculate the distance between the two points ( radius ), then I subtract this value from the coordinates of the center to obtain the upper-left corner and add to obtain the lower-right one.
When I draw the cirlce and the points, and I zoom in, I see that the second one isn't on the circumference as expected, it is slightly inside unless you set it at 0°, 45°, 90° and so on with respect to the absolute sistem of coordinates.
Then I tried to draw the same circle using CDC::Polyline, I gave to this method the points obtained rotating another point around the center, at the distance equal to the radius. In this case the point is on the circumference every where I set it.
The overlap of these two circles has shown that they perfectly overlap at 0°, 45°, 90° and so on, but the gap is maximum at 22.5°, 67.5° and so on.
Has anyone ever noticed a similar behavior?
Thanks to everybody that can help me!
Code snippet:
this is how I calculate the radius given 2 points:
centerPX = vvFPoint( 1380, 845 );
secondPointPX = vvFPoint( 654,654 );
double radiusPX = (sqrt( (secondPointPX.x - centerPX.x) * (secondPointPX.x - centerPX.x) + (secondPointPX.y - centerPX.y) * (secondPointPX.y - centerPX.y) ));
( vvFPoint is a custom type derived from CPoint )
this is how I draw the "circle" with the CDC::Ellipse:
int up = (int)(((double)(m_p1.y-(double)originY - m_radius) / zoom) + 0.5) + offY;
int left = (int)(((double)(m_p1.x-(double)originX - m_radius) / zoom) + 0.5) + offX;
int down = (int)(((double)(m_p1.y-(double)originY + m_radius) / zoom) + 0.5) + offY;
int right = (int)(((double)(m_p1.x-(double)originX + m_radius) / zoom) + 0.5) + offX;
pDC->Ellipse( left, up, right, down);
(m_p1 is the center of the circle, originX/Y is the origin of the image, m_radius is the radius of the circle, zoom is the scale factor, offX/Y is an offset in the client area of my SW)
this is how I draw the circle "manually" (and quite trivial method) using a custom polyline class:
1) create the array of points:
point.x = centerPX.x + radiusPX;
point.y = centerPX.y;
for ( i=0; i < 3600; i++ )
{
pt1.RotateDeg ( centerPX, (double)0.1 );
poly->AddPoint( pt1 );
}
(RotateDeg is a custom method to rotate a point using first argument as a pivot and second argument as angle value in degrees, AddPoint is a custom method to create the array of points, poly is my custom polyline object).
2) draw it:
When I call the Draw( CDC* pDC ) I use the previous array to draw the polyline:
pDC->MoveTo(p);
I hope this can help you to reproduce my weird observations!
code snippet 2:
void vvPoint<Tipo>::RotateDeg(const vvPoint<Tipo> ¢er, double angle)
{
vvPoint<Tipo> ptB;
angle *= -(M_PI / 180);
*this -= center;
ptB.x = ((this->x * cos(angle)) - (this->y * sin(angle)));
ptB.y = ((this->x * sin(angle)) + (this->y * cos(angle)));
*this = ptB + center;
}
But to let you better understand my observations I would like to add a few images so you can see where my whole question started from... The problem is: I can't add images since I need to have 10 reputation. I uploaded a .zip file on dropbox and if you want I can send you the URL of this file. Let me know if this is the correct (and safe..) way to bypass this problem.
Thanks!
This might be a possible explanation. As MSDN says about CDC::Ellipse (with my emphasis):
The center of the ellipse is the center of the bounding rectangle
specified by x1, y1, x2, and y2, or lpRect. The ellipse is drawn with
the current pen, and its interior is filled with the current brush.
The figure drawn by this function extends up to, but does not include,
the right and bottom coordinates. This means that the height of the
figure is y2 – y1 and the width of the figure is x2 – x1.
The way you described how you calculate the bounding rectangle is not entirely clear (some source code would have helped) but, given the second paragraph quoted above, you possibly need to add 1 to your x2 and y2 values, to make sure you have a circle with the desired radius.
It's also worth noting that there may be slight rounding differences between your two drawing methods where you have an odd-sized bounding box (i.e. so the centre point falls logically on a half-pixel).
UPDATE
Using your code snippets (thanks), and assuming no zoom and zero offsets etc., I get a radius of 750.704 pixels and the following parameters for the ellipse:
pDC->Ellipse(629, 94, 2131, 1596);
According to MSDN, this means that the ellipse will be drawn in a figure of the following dimensions:
width = (2131 - 629) = 1502
height = (1596 - 94) = 1502
So as far as I can see, this should produce a circle rather than an ellipse.
The next thing to do is to find out how you're drawing the polygon - for that we need to see the implementation of RotateDeg - can you post that code? I'm suspecting some simple rounding error here, that maybe gets magnified when you zoom.
UPDATE 2
Just looking at this code:
for ( i=0; i < 3600; i++ )
{
pt1.RotateDeg ( centerPX, (double)0.1 );
poly->AddPoint( pt1 );
}
You are rotating your polygon points incrementally by 0.1 degrees each time. This will possibly accumulate some errors, so it may be worth doing it something like this instead:
for ( i=0; i < 3600; i++ )
{
vvFPoint ptNew = pt1;
ptNew.RotateDeg ( centerPX, (double)i * 0.1 );
poly->AddPoint( ptNew );
}
Maybe this will mean you have to change your RotateDeg function to take care of the correct quadrants.
One other point, you mentioned that you see the problem when you zoom into the image. If this means you are using you zoom variable, it is worth checking in this line ...:
pDC->Ellipse( left, up, right, down);
... that the parameters still form a square shape, so (right - left) == (down - up).
UPDATE 3
I just ran your RotateDeg function, in its current form, to see how the error accumulates (by feeding in the previous result to the next iteration). At each step, I calculated the distance between the new point and the centre and compared this with the required radius.
The chart below shows the result, where you can see an error of 4 pixels by the time the points have been calculated.
I think that this at least explains part of the difference (i.e. your polygon drawing is flawed) and - depending on zoom - you may introduce asymmetry into the ellipse parameters, which you can debug by comparing the width to the height as I described above.