3D Reconstruction Of Planar Markers usin OpenCV - c++

I am trying to perform 3D Reconstruction(Structure From Motion) from Multiple Images of Planar Markers. I very new to MVG and openCV.
As far I have understood I have to do the following steps:
Identify corresponding 2D corner points in the one images.
Calculate the Camera Pose of the first image us cv::solvePNP(assuming the
origin to be center of the marker).
Repeat 1 and 2 for the second image.
Estimate the relative motion of the camera by Rot_relative = R2 - R1,
Trans_relative = T2-T1.
Now assume the first camera to be the origin construct the 3x4 Projection
Matrix for both views, P1 =[I|0]*CameraMatrix(known by Calibration) and P2 =
[Rot_relative |Trans_relative ].
Use the created projection matrices and 2D corner points to triangulate the
3D coordinate using cv::triangulatePoints(P1,P2,point1,point2,OutMat)
The 3D coordinate can be found by dividing the each rows of OutMat by the 4th
row.
I was hoping to keep my "First View" as my origin and iterate
through n views repeating steps from 1-7(I suppose its called Global SFM).
I was hoping to get (n-1)3D points of the corners with "The first View as origin" which we could optimize using Bundle Adjustment.
But the result I get is very disappointing the 3D points calculated are displaced by a huge factor.
These are questions:
1.Is there something wrong with the steps I followed?
2.Should I use cv::findHomography() and cv::decomposeHomographyMat() to find the
relative motion of the camera?
3.Should point1 and point2 in cv::triangulatePoints(P1,P2,point1,point2,OutMat)
be normalized and undistorted? If yes, how should the "Outmat" be interpreted?
Please anyone who has insights towards the topic, can you point out my mistake?
P.S. I have come to above understanding after reading "MultiView Geometry in Computer Vision"
Please find the code snippet below:
cv::Mat Reconstruction::Triangulate(std::vector<cv::Point2f>
ImagePointsFirstView, std::vector<cv::Point2f>ImagePointsSecondView)
{
cv::Mat rVectFirstView, tVecFristView;
cv::Mat rVectSecondView, tVecSecondView;
cv::Mat RotMatFirstView = cv::Mat(3, 3, CV_64F);
cv::Mat RotMatSecondView = cv::Mat(3, 3, CV_64F);
cv::solvePnP(RealWorldPoints, ImagePointsFirstView, cameraMatrix, distortionMatrix, rVectFirstView, tVecFristView);
cv::solvePnP(RealWorldPoints, ImagePointsSecondView, cameraMatrix, distortionMatrix, rVectSecondView, tVecSecondView);
cv::Rodrigues(rVectFirstView, RotMatFirstView);
cv::Rodrigues(rVectSecondView, RotMatSecondView);
cv::Mat RelativeRot = RotMatFirstView-RotMatSecondView ;
cv::Mat RelativeTrans = tVecFristView-tVecSecondView ;
cv::Mat RelativePose;
cv::hconcat(RelativeRot, RelativeTrans, RelativePose);
cv::Mat ProjectionMatrix_0 = cameraMatrix*cv::Mat::eye(3, 4, CV_64F);
cv::Mat ProjectionMatrix_1 = cameraMatrix* RelativePose;
cv::Mat X;
cv::undistortPoints(ImagePointsFirstView, ImagePointsFirstView, cameraMatrix, distortionMatrix, cameraMatrix);
cv::undistortPoints(ImagePointsSecondView, ImagePointsSecondView, cameraMatrix, distortionMatrix, cameraMatrix);
cv::triangulatePoints(ProjectionMatrix_0, ProjectionMatrix_1, ImagePointsFirstView, ImagePointsSecondView, X);
X.row(0) = X.row(0) / X.row(3);
X.row(1) = X.row(1) / X.row(3);
X.row(2) = X.row(2) / X.row(3);
return X;
}

Related

opencv stereo calibration result

As if i have two web camera,mark as cam1,cam2.And i want to calibrate them to get the transformation between them.
I used cv::stereoCalibrate() to calibrate.
After i got the transformation from cam1 to cam2,mark as R,T.I want to check the accuracy of the calibration result.
So i used cam1 and cam2 to take a picture of a chessboard,mark as pic1,pic2.I got the cam1's extrinsic parameters by cv::solvePnP().And i drew the cam1's world coordinate system by cv::projectPoints() in pic1.
Then,i think the cam2's rotation matrix=cam1's rotation matrix * R.And the cam2's translation matrix=cam1's translation matrix + T.
I calculated the cam2's extrinsic parameters by the above thought.And also drew the cam2's world coordinate system by cv::projectPoints() in pic2.
But the pic2's origin was not in right position.
Here is part of the code i used.
void check_res(const vector<string> &imgs_nm,const Mat &R,const Mat &T,const Mat &cam_c,const Mat &cam_h,const Mat &dist_c,const Mat &dist_h)
{
int imgs_cnt=imgs_nm.size()/2;
vector<Point3f> obj_pts;
for(int i=0;i<boardDimensions.height;i++)
for(int j=0;j<boardDimensions.width;j++)
obj_pts.push_back(Point3f(i*CHESS_LEN,j*CHESS_LEN,0.f));
for(int i=0;i<imgs_cnt;i++)
{
vector<Point2f> c_cners,h_cners;
Mat imgc_gray,imgh_gray;
Mat imgc=imread(imgs_nm[i*2],1);
Mat imgc_rz=imgc.clone();
bool c_found,h_found;
c_found=HasChessBoard(imgc_rz,imgc_gray,c_cners);
if(c_found)
cv::cornerSubPix(imgc_gray, c_cners, cv::Size(11, 11), cv::Size(-1, -1),cv::TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 30, 0.1));
Mat imgh=imread(imgs_nm[i*2+1],1);
h_found=HasChessBoard(imgh,imgh_gray,h_cners);
if(h_found)
cv::cornerSubPix(imgh_gray, h_cners, cv::Size(11, 11), cv::Size(-1, -1),cv::TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 30, 0.1));
Mat rvec_c,rvec_h,tvec_c,tvec_h;
cv::solvePnP(obj_pts,c_cners,cam_c,dist_c,rvec_c,tvec_c);
cv::solvePnP(obj_pts,h_cners,cam_h,dist_h,rvec_h,tvec_h);
Mat rrvec_c,rrvec_h;
cv::Rodrigues(rvec_c,rrvec_c);
cv::Rodrigues(rvec_h,rrvec_h);
Mat r1=rrvec_c*R;
Mat t1=tvec_c+T;
Mat img1=imgh.clone();
draw_chess(imgh,rrvec_h,tvec_h,cam_h,dist_h);
imshow("pic1",imgh);
draw_chess(img1,r1,t1,cam_h,dist_h);
imshow("pic2",img1);
char resc=waitKey(0);
if(resc=='q')
exit(1);
}
}
And below is the result i tested by using the sample in opencv.
I don't think it was low calibration accuracy,because i use the opencv's sample and the cv::stereoCalibrate() return rms less than 1 pixel.
Any advice is appreciated.
Thank you!
The formulas are:
pose for the camera 1 (in homogeneous matrix):
homogeneous transformation from camera 1 to camera 2:
pose for camera 2:
For checking the accuracy of your stereo calibration, I would consider a different approach:
Use the function stereoRectify to get the rectification transformation for the camera. Use the translation and rotation matrices you got from stereoCalibrate.
Perform initUndistortRectifyMap once for each camera. Use
Use the results you got to remap the images from both cameras.
If your calibration went well, the output images should be rectified and undistorted.

Image points (pixels) to real world coordinates (meters)

I have a fish-eye camera in the ceiling and I want to locate some points on the floor. I have put the origin of my reference system (real world) just below the camera and I want to know the position of every object in centimeters. This picture shows this:
Reference system - Real world
Firstly, I have done the camera calibration and I have obtained the next result with an RMS of 1.11:
Undistorted image after calibration
As a result of the calibration I obtained intrinsic parameters (camera matrix), so I used cv::solvePnP to get rotation and translation vectors. For apply this I marked some points in the undistorted image (in pixels) and I measured them in real world according to my reference system.
For example, the origin is in the center of a 1024x768 image, so:
Point 0: ImagePoint(512, 384) [pixels] --> ObjectPoint(0,0) [centimeters]
The next code shows this:
std::vector<cv::Point2f> imagePointsPix;
std::vector<cv::Point3f> objectPointsCm;
imagePointsPix.push_back(cv::Point2f(512.,384.));
imagePointsPix.push_back(cv::Point2f(404.,512.));
imagePointsPix.push_back(cv::Point2f(666.,211.));
imagePointsPix.push_back(cv::Point2f(519.,66.));
objectPointsCm.push_back(cv::Point3f(0., 0., 0.));
objectPointsCm.push_back(cv::Point3f(-80.,-132.,0.));
objectPointsCm.push_back(cv::Point3f(120.,188.,0.));
objectPointsCm.push_back(cv::Point3f(-40.,268.,0.));
cv::Mat rvec(1,3,cv::DataType<double>::type);
cv::Mat tvec(1,3,cv::DataType<double>::type);
cv::Mat rotationMatrix(3,3,cv::DataType<double>::type);
cv::solvePnP(objectPointsCm, imagePointsPix, cameraMatrix, distCoeffs, rvec, tvec, 0, SOLVEPNP_ITERATIVE);
cv::Rodrigues(rvec,rotationMatrix);
Now I have the camera matrix, the rotation matrix and the traslation vector, so by using this as reference I am able to compute any point if I have its position in pixels. This is the code:
cv::Mat uvPoint = cv::Mat::ones(3,1,cv::DataType<double>::type); //u,v,1
uvPoint.at<double>(0,0) = 512.; //img point for which we want its real coordinates
uvPoint.at<double>(1,0) = 384.;
cv::Mat tempMat, tempMat2;
double s;
tempMat = rotationMatrix.inv() * cameraMatrix.inv() * uvPoint;
tempMat2 = rotationMatrix.inv() * tvec;
s = 0 + tempMat2.at<double>(2,0); //before 0 it was 285, which represents the height Zconst
s /= tempMat.at<double>(2,0);
std::cout << "P = " << rotationMatrix.inv() * (s * cameraMatrix.inv() * uvPoint - tvec) << std::endl;
I get this results for the same points I used for obtaining my parameters:
Point 0 --> (0.213, 3.391) (it should be (0,0)) ERROR: 3.69 cm
Point 1 --> (-68.28, -112.82) (it should be (-80, -132)) ERROR: 17.49 cm
Point 2 --> (84.48, 137.61) (it should be (120, 188)) ERROR: 49.62 cm
The rest of points also show an error too big... I have used more points but the results do not improve. I don't know where I went wrong, could anyone help me?
Thanks in advance.
It looks like you may be effectively undistorting your image twice from solvePNP's perspective. This is due to passing in the distortion coefficients along with point correspondences that are already derived from an undistorted image.
Try passing the actual camera matrix from your calibration to solvePNP instead of an identity matrix, but still pass NULL for the distortion coefficients to avoid the double-undistortion.
Finally I have found out that the error was caused by the distortion coefficients, i.e. my calibration. I set the cameraMatrix to the Identity matrix (eye(3)) and the distCoefficients to NULL so that solvePNP assumed I have a perfect camera. Using this approach I obtained an error much lower. I will have to make a better calibration.

Can I get the point position using remap in OpenCV

I have taken a photo A using an RGB camera. And I know the position of a point g in photo A. The camera needs to do a camera calibration. Now I want to know the position of point g after calibration. I am using the code as following, but I want to get the point position, not image. How can I do that? Can you give me some advice?
initUndistortRectifyMap(
cameraMatrix,
distCoeffs,
Mat(),
Mat(),
Size(640, 480),
CV_32FC1,
map1, map2);
remap(A, B, map1, map2, cv::INTER_LINEAR);
Point2f g = Point2f(...,...);//i want to get the new position of the point not image B
Just get coordinates using maps:
x,y - coordinates after (not before),as pasbi correctly noticed in comments, mapping.
(map1(y,x),map2(y,x)) - coordinates before mapping
In other words:
map1.at<float>(y,x) contains source x coordinates for each destination point
p(x,y).
map2.at<float>(y,x) contains source y coordinates for each destination point
p(x,y).
See documentation on remap function.
best method i found was to recreate a camera matrix, with inverted parameters. work to a certain extent, with like basic images modifications
undistortPoints() is your need。
// src_pts are points in raw(distort) img, rectify_pt_vec are in rectifyImageL
// RL, PL are from stereoRectify()
cv::undistortPoints(src_pts, rectify_pt_vec, cameraMatrixL, distCoeffL, RL, PL);
how to get point in srcimg from dstimg just like pasbi commented below.

Matching small grayscale images

I want to test whether two images match. Partial matches also interest me.
The problem is that the images suffer from strong noise. Another problem is that the images might be rotated with an unknown angle. The objects shown in the images will roughly always have the same scale!
The images show area scans from a top-shot perspective. "Lines" are mostly walls and other objects are mostly trees and different kinds of plants.
Another problem was, that the left image was very blurry and the right one's lines were very thin.
To compensate for this difference I used dilation. The resulting images are the ones I uploaded.
Although It can easily be seen that these images match almost perfectly I cannot convince my algorithm of this fact.
My first idea was a feature based matching, but the matches are horrible. It only worked for a rotation angle of -90°, 0° and 90°. Although most descriptors are rotation invariant (in past projects they really were), the rotation invariance seems to fail for this example.
My second idea was to split the images into several smaller segments and to use template matching. So I segmented the images and, again, for the human eye they are pretty easy to match. The goal of this step was to segment the different walls and trees/plants.
The upper row are parts of the left, and the lower are parts of the right image. After the segmentation the segments were dilated again.
As already mentioned: Template matching failed, as did contour based template matching and contour matching.
I think the dilation of the images was very important, because it was nearly impossible for the human eye to match the segments without dilation before the segmentation. Another dilation after the segmentation made this even less difficult.
Your first job should be to fix the orientation. I am not sure what is the best algorithm to do that but here is an approach I would use: fix one of the images and start rotating the other. For each rotation compute a histogram for the color intense on each of the rows/columns. Compute some distance between the resulting vectors(e.g. use cross product). Choose the rotation that results in smallest cross product. It may be good idea to combine this approach with hill climbing.
Once you have the images aligned in approximately the same direction, I believe matching should be easier. As the two images are supposed to be at the same scale, compute something analogous to the geometrical center for both images: compute weighted sum of all pixels - a completely white pixel would have a weight of 1, and a completely black - weight 0, the sum should be a vector of size 2(x and y coordinate). After that divide those values by the dimensions of the image and call this "geometrical center of the image". Overlay the two images in a way that the two centers coincide and then once more compute cross product for the difference between the images. I would say this should be their difference.
You can also try following methods to find rotation and similarity.
Use image moments to get the rotation as shown here.
Once you rotate the image, use cross-correlation to evaluate the similarity.
EDIT
I tried this with OpenCV and C++ for the two sample images. I'm posting the code and results below as it seems to work well at least for the given samples.
Here's the function to calculate the orientation vector using image moments:
Mat orientVec(Mat& im)
{
Moments m = moments(im);
double cov[4] = {m.mu20/m.m00, m.mu11/m.m00, m.mu11/m.m00, m.mu02/m.m00};
Mat covMat(2, 2, CV_64F, cov);
Mat evals, evecs;
eigen(covMat, evals, evecs);
return evecs.row(0);
}
Rotate and match sample images:
Mat im1 = imread(INPUT_FOLDER_PATH + string("WojUi.png"), 0);
Mat im2 = imread(INPUT_FOLDER_PATH + string("XbrsV.png"), 0);
// get the orientation vector
Mat v1 = orientVec(im1);
Mat v2 = orientVec(im2);
double angle = acos(v1.dot(v2))*180/CV_PI;
// rotate im2. try rotating with -angle and +angle. here using -angle
Mat rot = getRotationMatrix2D(Point(im2.cols/2, im2.rows/2), -angle, 1.0);
Mat im2Rot;
warpAffine(im2, im2Rot, rot, Size(im2.rows, im2.cols));
// add a border to rotated image
int borderSize = im1.rows > im2.cols ? im1.rows/2 + 1 : im1.cols/2 + 1;
Mat im2RotBorder;
copyMakeBorder(im2Rot, im2RotBorder, borderSize, borderSize, borderSize, borderSize,
BORDER_CONSTANT, Scalar(0, 0, 0));
// normalized cross-correlation
Mat& image = im2RotBorder;
Mat& templ = im1;
Mat nxcor;
matchTemplate(image, templ, nxcor, CV_TM_CCOEFF_NORMED);
// take the max
double max;
Point maxPt;
minMaxLoc(nxcor, NULL, &max, NULL, &maxPt);
// draw the match
Mat rgb;
cvtColor(image, rgb, CV_GRAY2BGR);
rectangle(rgb, maxPt, Point(maxPt.x+templ.cols-1, maxPt.y+templ.rows-1), Scalar(0, 255, 255), 2);
cout << "max: " << max << endl;
With -angle rotation in code, I get max = 0.758. Below is the rotated image in this case with the matching region.
Otherwise max = 0.293

opencCV Calculation of distortion co-efficients(Uncalibrated) to Undistort

I'm new to openCV using version 2.4.9
I am trying to generate a 3D projection of points from a sequence of images without any knowledge of the camera parameters nor have camera used with me to calibrate. The camera used had a fish eye lens.
I used goodFeaturesToTrack() for detecting feature points followed by LK implementation in openCV to track the feature points in the sequence of images. Using these points I was successfully able to estimate the Fundamental Matrix from findFundamentalMat() and implemented stereoRectifyUncalibrated() to generate rectification homography matrices H1 and H2.Then I have computed Rotation matrix R from H as
R = cameraMatrix^{-1}*H*cameraMatrix
Now I need to undistort my images after rectification. Either by initUndistortRectifyMap() and remap() or directly by undistort(), but both the functions also require "distortion co-efficients" to compute corrected image.
I tried to find various methods to estimate those parameters, neither the documentation of the camera model is made available by the company, nor I could find any other method apart from calibrating camera using chessboards or circles grid.
How do I do it??
Am I doing it right?
Is there any other better method?
Can someone kindly help?
Thanks in Advance.
//Code
//Fundamental Matrix
Mat fundamental_matrix = findFundamentalMat(points[0], points[1], FM_RANSAC, 3, 0.99);
cout<<"F:\n" <<fundamental_matrix<<endl;
//Rectification Homographies
Mat H1, H2,F;
F = fundamental_matrix;
stereoRectifyUncalibrated(points[0],points[1], F, image.size(), H1, H2, 3);
cout<<"H1:\n" <<H1<<endl;
cout<<"H2:\n" <<H2<<endl;
//calculating Rotation matrix from homographic maps
Mat fInv= fundamental_matrix.inv();
R = (fInv)*H1*fundamental_matrix;
// Mat distCoeffs = Mat::zeros(8, 1, CV_64F);
initUndistortRectifyMap(fundamental_matrix, distCoeffs, R, fundamental_matrix, image.size() ,CV_32FC1, map1, map2);
//How to compute distCoeffs without a camera nor prior knowledge.Thank You