I have a little problem with some projection and geometry. I have an image where I detect a square. After the square detection, I crop the square from image. In the ROI I detect the point P(x,y) (see the image below).
My problem is that I know the coordinate of point P in the ROI, the coordinates of A,B,C,D, and rotation of ROI (RotatedRect::angle) but I want to get the coordinate of P in original image. Any advice could help.
For ROI crop I have this code
vector< RotatedRect > rect(squares.size());
for (int i=0;i<squares.size();i++)
{
rect[i] = minAreaRect(Mat(squares[i]));
Mat M,rotated,cropped;
float angle = rect[i].angle;
Size rect_size = rect[i].size;
if (rect[i].angle<-45)
{
angle += 90;
swap(rect_size.width,rect_size.height);
}
M = getRotationMatrix2D(rect[i].center,angle,1.0);
warpAffine(cameraFeed,rotated,M,cameraFeed.size(),INTER_CUBIC);
getRectSubPix(rotated,rect_size,rect[i].center,cropped);
cropped.copyTo(SatelliteClass[i].m_matROIcropped);
SatelliteClass[i].m_vecRect = rect[i];
}
It's basically a question of vector addition. Take the inverse of M, apply it to P ( so you're rotating P back to the original frame ) and then add P to the left corner of the rectangle.
There might be a way to do this within the API you're using instead of reinventing the wheel.
Related
I am working on a project and I am stuck with one thing. I have two sets of points extracted from contours of the same object but rotated in 2D and I need to find the best rotation transformation (or just angle of rotation) between those points.
What I did is that I rescaled one of the contours so that I have two contours of the same size and also I made those two contours have the same center of mass. Then what I do is that I choose 3 random points from the first set of points and 3 points of the second set of points (some kind of RANSAC in which I draw N times random points) and I need to find the rotation transformation around their center of mass of one set to the other one. I tried to use Kabsch algorithm but I'm not sure if I'm implementing it correctly because it is not working properly. Here is my code:
Here is my code of Kabsch:
// P,Q - sets of points of contours
cv::Mat Pt;
transpose(P, Pt);
cv::Mat H = Pt * Q;
cv::Mat Ht;
transpose(H,Ht);
cv::Mat invH = H.inv();
cv::Mat HtH = Ht * H;
cv::Mat sqrtH(HtH.size(), CV_32F);
for (int i=0;i<2;i++)
for (int j = 0; j < 2; j++)
{
sqrtH.at<float>(j, i) = sqrt(HtH.at<float>(j,i));
}
// Final transform
cv::Mat R = sqrtH * invH;
I would like to at least get an angle of rotation between two sets of points. When I use my code I get strange results and transformation that are messing my sets of points up.
I have been struggling trying to implement the outlining algorithm described here and here.
The general idea of the paper is determining the Hausdorff distance of binary images and using it to find the template image from a test image.
For template matching, it is recommended to construct image pyramids along with sliding windows which you'll use to slide over your test image for detection. I was able to do both of these as well.
I am stuck on how to move forward from here on. Do I slide my template over the test image from different pyramid layers? Or is it the test image over the template? And with regards to the sliding window, is/are they meant to be a ROI of the test or template image?
In a nutshell, I have pieces to the puzzle but no idea of which direction to take to solve the puzzle
int distance(vector<Point>const& image, vector<Point>const& tempImage)
{
int maxDistance = 0;
for(Point imagePoint: image)
{
int minDistance = numeric_limits<int>::max();
for(Point tempPoint: tempImage)
{
Point diff = imagePoint - tempPoint;
int length = (diff.x * diff.x) + (diff.y * diff.y);
if(length < minDistance) minDistance = length;
if(length == 0) break;
}
maxDistance += minDistance;
}
return maxDistance;
}
double hausdorffDistance(vector<Point>const& image, vector<Point>const& tempImage)
{
double maxDistImage = distance(image, tempImage);
double maxDistTemp = distance(tempImage, image);
return sqrt(max(maxDistImage, maxDistTemp));
}
vector<Mat> buildPyramids(Mat& frame)
{
vector<Mat> pyramids;
int count = 6;
Mat prevFrame = frame, nextFrame;
while(count > 0)
{
resize(prevFrame, nextFrame, Size(), .85, .85);
prevFrame = nextFrame;
pyramids.push_back(nextFrame);
--count;
}
return pyramids;
}
vector<Rect> slidingWindows(Mat& image, int stepSize, int width, int height)
{
vector<Rect> windows;
for(size_t row = 0; row < image.rows; row += stepSize)
{
if((row + height) > image.rows) break;
for(size_t col = 0; col < image.cols; col += stepSize)
{
if((col + width) > image.cols) break;
windows.push_back(Rect(col, row, width, height));
}
}
return windows;
}
Edit I: More analysis on my solution can be found here
This is a bi-directional task.
Forward Direction
1. Translation
For each contour, calculate its moment. Then for each point in that contour, translate it about the moment i.e. contour.point[i] = contour.point[i] - contour.moment[i]. This moves all of the contour points to the origin.
PS: You need to keep track of each contour's produced moment because it will be used in the next section
2. Rotation
With the newly translated points, calculate their rotated rect. This will give you the angle of rotation. Depending on this angle, you would want to calculate the new angle which you want to rotate this contour by; this answer would be helpful.
After attaining the new angle, calculate the rotation matrix. Remember that your center here will be the origin i.e. (0, 0). I did not take scaling into account (that's where the pyramids come into play) when calculating the rotation matrix hence I passed 1.
PS: You need to keep track of each contour's produced matrix because it will be used in the next section
Using this matrix, you can go ahead and rotate each point in the contour by it as shown here*.
Once all of this is done, you can go ahead and calculate the Hausdorff distance and find contours which pass your set threshold.
Back Direction
Everything done in the first section, has to be undone in order for us to draw the valid contours onto our camera feed.
1. Rotation
Recall that each detected contour produced a rotation matrix. You want to undo the rotation of the valid contours. Just perform the same rotation but using the inverse matrix.
For each valid contour and corresponding matrix
inverse_matrix = matrix[i].inv(cv2.DECOMP_SVD)
Use * to rotate the points but with inverse_matrix as parameter
PS: When calculating the inverse, if the produced matrix was not a square one, it would fail. cv2.DECOMP_SVD will produce an inverse matrix even if the original matrix was a non-square.
2. Translation
With the valid contours' points rotated back, you just have to undo the previously performed translation. Instead of subtracting, just add the moment to each point.
You can now go ahead and draw these contours to your camera feed.
Scaling
This is were image pyramids come into play.
All you have to do is resize your template image by a fixed size/ratio upto your desired number of times (called layers). The tutorial found here does a good job of explaining how to do this in OpenCV.
It goes without saying that the values you choose to resize your image by and number of layers will and do play a huge role in how robust your program will be.
Put it all together
Template Image Operations
Create a pyramid consisting of n layers
For each layer in n
Find contours
Translate the contour points
Rotate the contour points
This operation should only be performed once and only store the results of the rotated points.
Camera Feed Operations
Assumptions
Let the rotated contours of the template image at each level be stored in templ_contours. So if I say templ_contours[0], this is going to give me the rotated contours at pyramid level 0.
Let the image's translated, rotated contours and moments be stored in transCont, rotCont and moment respectively.
image_contours = Find Contours
for each contour detected in image
moment = calculate moment
for each point in image_contours
transCont.thisPoint = forward_translate(image_contours.thisPoint)
rotCont.thisPoint = forward_rotate(transCont.thisPoint)
for each contour_layer in templ_contours
for each contour in rotCont
calculate Hausdorff Distance
valid_contours = contours_passing_distance_threshold
for each point in valid_contours
valid_point = backward_rotate(valid_point)
for each point in valid_contours
valid_point = backward_translate(valid_point)
drawContours(valid_contours, image)
I succesfully calibrate my camera using opencv. The camera lens i am using.
https://www.baslerweb.com/en/products/vision-components/lenses/basler-lens-c125-0418-5m-f1-8-f4mm/
The internal and external camera parameter is given below.
cv::Mat cameraMatrix(3, 3, cv::DataType<double>::type);
cameraMatrix.at<double>(0) = 1782.80;//fx //432.2 in mm
cameraMatrix.at<double>(1) = 0;
cameraMatrix.at<double>(2) = 3.0587694283633488e+002;//cx
cameraMatrix.at<double>(3) = 0;
cameraMatrix.at<double>(4) = 1782.80;//fy
cameraMatrix.at<double>(5) = 3.0535864258476721e+002;//cy
cameraMatrix.at<double>(6) = 0;
cameraMatrix.at<double>(7) = 0;
cameraMatrix.at<double>(8) = 1;
cv::Mat disCoeffs(1, 5, cv::DataType<double>::type);
disCoeffs.at<double>(0) = -8.1752937039996709e-001;//k1
disCoeffs.at<double>(1) = -2.5660653367749450e+001;//k2
disCoeffs.at<double>(2) = -1.5556922931812768e-002;//p1
disCoeffs.at<double>(3) = -4.4021541217208054e-002;//p2
disCoeffs.at<double>(4) = 1.5042036073609015e+002;//k3
I know this formula is used to calculate the distance of the object. But i am very confuse how to proper use it.
Resolution of my camera is 640x480.
focal length = 1782.80 (px) do not know how to correctly convert to mm
i know focal length is distance from sensor to image plane. So what actually this value represent? Pixel is just a unit represent dot on screen.
Object i am using is circle.
radius = 22. (width and height 44*44)
circle center point: 300,300 (x,y)
sensor height do not know how to get?
Where do i use principle points?
How i get distance from camera to object? How do get real world coordinate of the circle?
I know its too much to ask. I try one month. Did not find any proper solution.
i use function solvePnP to get the camera translation and rotation matrix. But i have problem how to calculate object point?
Your cx and cy seems to be wrong because they should be half the resolution: 640/2 & 480/2.
fx and fy are in pixel unit you get from calibration process. To convert them to mm use that formula:
pixels width = (image width in pixels) * (focal length in mm) / (CCD width in mm)
pixels height = (image height in pixels) * (focal length in mm) / (CCD height in mm)
When you calibrate your camera, you use those formulas to make sure you've the right values. For me cx and cy are wrong because they represent the center of the image (they shouldn't be equal unless your image is square which is not the case). For fx and fy I can't tell because I don't know the CCD of your camera. They can be equal if the CCD is square.
Don't change those parameters manually but let the your calibration software compute them.
Now you've those parameters, how you compute the distance?
The formula you presented is not useful in a sense that if you can measure the real height, you usually can measure the distance (at least in your case).. so why using a camera!?
So to compute the distance in real world, you need two more things: The extrinsic parameters (Your cameraMatrix matrix is the intrinsic parameters) and at least four points (the more points the better) in real world coordinates.
Once you have those things, you can use solvePnP function to find the pose of an object. The pose represents the translation and rotation with respect to the camera frame.
http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#solvepnp
This is a piece of code can help to do that:
//Four points in real world with `x,y and z` coordinates
vector<Point3f> vec3d;
vec3d.push_back(Point3f(0, 0, 0));
vec3d.push_back(Point3f(0, 211, 0));
vec3d.push_back(Point3f(295, 211, 0));
vec3d.push_back(Point3f(295, 0, 0));
The z=0 because your real points are in a plane.
//The same four points but in your image plan, therefore there is no z and they're in pixel unit
vector<Point2f> vec2d;
vec2d.push_back(Point2f(532, 412)); //(y,x)
vec2d.push_back(Point2f(583, 594));
vec2d.push_back(Point2f(927, 535));
vec2d.push_back(Point2f(817, 364));
//The pose of the object: rvec is your rotation vector, tvec is your translation vector
cv::Mat rvec, tvec;
solvePnP(vec3d, vec2d, cameraMatrix, distCoeffs, rvec, tvec);
Finally, you can compute the real distance from the tvec as euclidean distance: d=std::sqrt(tx*tx+ty*ty+tz*tz).
Your questions:
sensor height do not know how to get?
Look for your camera specification in the internet or in the manual book and you'll find it.
Where do i use principle points?
They're your intrinsic parameters. You're not gonna use them separately.
How i get distance from camera to object? How do get real world coordinate of the circle?
I explained that above. You need four points and with a circle you have only one which not enough to compute the pose.
But i have problem how to calculate object point?
objectPoints in solvePnP are your real world coordinates. For example, a chessboard has corners in which we know the exact position in mm of each one with respect to a world frame that you choose in the chessboard. It can be in the left top corner or something like that and z=0 because the chessboard is printed in a paper just like your circle!
EDIT:
You can find more specifications in the manual page 13 here. It is said 7.4 x 7.4µm:
f (mm)=f(pixel) x pixel_size(mm) => f (mm) = 1782.80x7.2e-6 = 12.83616 (mm)
Which is not 4mm!! then you need to do the calibration again, something is wrong!
3D points:
vector vec3d;
vec3d is where you gonna store your 3D coordinates point. I gave you an example for the first point which the origin:
vec3d.push_back(Point3f(0, 0, 0)); //y,x,z
EDIT3
If you take a pattern like this
Then choose for example the circle in top left or right corner and it will have a coordinate of (0,0,0), that the origin. After that the circle next to it is your second point and it will have (x,0,0) x is the distance in (mm) between the two circles.. You do the same for four points in your pattern. You can choose any pattern you want as long as you can detect it in your image and retrieve their coordinates in pixel.
If you still don't understand, I advise you take a course in projective geometry and camera models.. so as you can understand what every parameter means.
If I could detect a circle by using canny edge detector, How can I have access to all values which are inside the circle?
void Canny(InputArray image, OutputArray edges, double threshold1, double threshold2, int apertureSize=3, bool L2gradient=false )
output of this function will give me the edge values which is detected by the edge detector, but what I want is all values inside the circle.
thanks in advance
------after editing.......
Mat mask = Mat::zeros(canny_edge.rows, canny_edge.cols, CV_8UC1);
Mat crop(main.rows, main.cols, CV_32FC1);
main.copyTo( crop, mask );
for(unsigned int y=0; y< height; y++)
for(unsigned int x=0; x< width; x++)
if(mask.at<unsigned char>(y,x) > 0)
{
}
For a circle, as asked in the original question:
first you want to detect the circle (e.g. by using hough circle detection methods). If you've done that you have some kind of circle center and a radius. Have a look at http://docs.opencv.org/doc/tutorials/imgproc/imgtrans/hough_circle/hough_circle.html
After that you have to test whether a pixel is inside the circle. So one idea (and with openCV quite fast) is to draw a filled circle on a mask image and test for each pixel in the original image, whether the mask pixel at the same image coordinates is set (then the pixel is inside the object). This works for any other drawable object, draw it (filled) on the mask and test mask values.
Assuming you have a circle center and a radius, and the size of your original image is image_height x image_width, try this:
cv::Mat mask = cv::Mat::zeros(image_height,image_width, CV_8U);
cv::circle(mask, center, radius, cv::Scalar(255), -1);
for(unsigned int y=0; y<image_height; ++y)
for(unsigned int x=0; x<image_width; ++x)
if(mask.at<unsigned char>(y,x) > 0)
{
//pixel (x,y) in original image is within that circle so do whatever you want.
}
though it will be more efficient if you limit the mask region (circle center +/- radius in both dimensions) instead of looping over the whole image ;)
For circles you should use the Hough Circle Transform. From it you will get the centres and radii of circles in your image. A given pixel is inside a particular circle if its distance from the center is less than the radius of the circle.
For a general shape, use findCountours to get the outline of the shape, then you can usepointPolygonTest to determine wheether points are inside that shape. There is a tutorialon it.
I work on Windows7 x64 with opencv and Visual Studio 2010 on c++ language.
I created a project in which I show to my camera a rectangular area (call squared_surface). This area is recognized by tracing a rectangle with findSquare () and drawSquares () of opencv file squares.cpp.
On this rectangle I create a ROI and there I copy an image (let's call copied_image)
My problem is that when I rotate squared_surface (in front of camera), copied_image does not follow it.
I think I need to use the functions getPerpective () and warpPerspective (), but I do not know how. Can anyone help me?
Here's the code:
int main(){
vector<vector<Point> > squares;
cv::VideoCapture cap(0);
for (;;) {
cv::Mat image;
cap >> image;
findSquares(image, squares);
for (size_t i = 0; i < squares.size(); i++) {
Rect rectangle = boundingRect(Mat(squares[i]));
if((rectangle.width<=630)&& (rectangle.width >= 420) && (rectangle.height<= 490) &&(rectangle.height >= 250 )) {
cv::Size dsize = Size(rectangle.width, rectangle.height);
Mat img1 = imread("scacchiera.jpg");
cv::resize(img1,img1,dsize,0,0, INTER_LINEAR);
Rect roi (rectangle.x, rectangle.y,rectangle.width, rectangle.height);
Mat imageRoi(image, roi);
img1.copyTo(imageRoi);
}
}
drawSquares(image, squares);
imshow("camera",image);
if(waitKey(30) >= 0) break;
}
return 0;
}
Thanks!
EDIT.
I was thinking of rotating Copied_image, so it follows Squared_surface, but I need to calculate the angle of rotation of the rectangle identified by the camera (drawn in red in the above images). Is there a way to calculate this angle?
Or how can I do so that Copied_image follows Squared_surface when I rotate squared_surface?
Help me, please!
I think I found the bug. Rect rectangle = boundingRect(Mat(squares[i])); This is where the problem is. You are creating the variable rectangle as a bounding rectangle of the coordinates in squares[i]. So your code always tries to find out the bounding rectangle and not the actual rectangle.
Instead of using a bounding rectangle, try using a rotated rectangle. Here is how to use it: http://www710.univ-lyon1.fr/~eguillou/documentation/opencv2/classcv_1_1_rotated_rect.html
The rotated rectangle RotatedRect (const Point2f &_center, const Size2f &_size, float _angle) requires the center point location, floating point angle and the size. Since you have all the coordinates I think you can use basic math and trigonometry to calculate the center and the angle on how your rectangle should be rotated/orientated.
Let me know if this helps.