My camera has different resolutions
1280*480
640*240
320*120
I have used the algorithm of Opencv3 to calibrate the camera with the resolution of 1280*480 and I have got the camera matrix (fx fy cx cy)and the distortion matrix (k1 k2 p1 p2 k3)for this resolution.
But now I want to use these camera matrix and distortion matrix to calibrate the camera with the resolution of 320*120. I don't know how to apply these two matrix of the resolution 1280*480 to the resolution of 320*120.
PS I haven't calibrated the camera with the resolution of 320*120 directly because the image is too small and the algorithm of Opencv can't find the chessboard.
I want to know how the camera matrix (fx fy cx cy)and the distortion matrix (k1 k2 p1 p2 k3) will change if i change the resolution 1280*480 to 320*120.
The algorithm of opencv is the following one:
http://docs.opencv.org/3.0-beta/doc/tutorials/calib3d/camera_calibration/camera_calibration.html
You don't need to change the distortion matrix. As for the camera matrix (the one containing fx, fy, cx, cy), you just need to divide them by 4 in your case. The general formula is:
fx' = (dimx' / dimx) * fx
fy' = (dimy' / dimy) * fy
fx' is the value for your new resolution, fx is the value you already have for your original resolution, dimx' is the new resolution along the x axis, dimx is the original one. The same applies to fy.
cx and cy are calculated analogically because all these values are expressed in pixel coordinates.
As per the OpenCV docs regarding the camera matrix:
if an image from the camera is scaled by a factor, all of these
parameters should be scaled (multiplied/divided, respectively) by the
same factor.
Related
I have two chessboard poses obtained with solvePnp:
Mat rotationVector1, translationVector1;
solvePnP(chess1WorldPoints, chess1ImagePoints, intrinsicMatrix, distortCoefficients, rotationVector1, translationVector1);
Mat rotationVector2, translationVector2;
solvePnP(chess2WorldPoints, chess2ImagePoints, intrinsicMatrix, distortCoefficients, rotationVector2, translationVector2);
How can I check if the planes of the poses are parallel, or find the angle between these planes?
More info
I tried obtaining Euler angles and computing the difference between each alpha, beta and gamma but that only tells me relative rotation for each axis I think:
Vec3d eulerAnglesPose1;
Mat rotationMatrix1;
Rodrigues(rotationVector1, rotationMatrix1);
getEulerAngles(rotationMatrix1, eulerAngles1);
Vec3d eulerAnglesPose2;
Mat rotationMatrix2;
Rodrigues(rotationVector2, rotationMatrix2);
getEulerAngles(rotationMatrix2, eulerAngles2);
I used the getEulerAngles implementation from Camera Rotation SolvePnp :
void getEulerAngles(Mat &rotCamerMatrix, Vec3d &eulerAngles)
{
Mat cameraMatrix, rotMatrix, transVect, rotMatrixX, rotMatrixY, rotMatrixZ;
double* _r = rotCamerMatrix.ptr<double>();
double projMatrix[12] =
{
_r[0],_r[1],_r[2],0,
_r[3],_r[4],_r[5],0,
_r[6],_r[7],_r[8],0
};
decomposeProjectionMatrix(Mat(3, 4, CV_64FC1, projMatrix), cameraMatrix, rotMatrix, transVect, rotMatrixX, rotMatrixY, rotMatrixZ, eulerAngles);
}
Edit
In my case a rotation-translation pair (R, T) gives the correspondence between a coordinate system where the camera is at (0,0,0) (the camera coordinate system) to a coordinate system where (0,0,0) is something I defined in the first two parameters of solvePnp (the world coordinate system). So I have two world coordinate systems relative to the same camera coordinate system.
If I could switch from coord. system 2 to coord. system 1 I could use the Z=0 planes for each one to find the normals and solve my problem.
I think that for example switching from coord. system 2 to camera system should be done like in this post:
Rinv = R' (just the transpose as it's a rotation matrix)
Tinv = -Rinv * T (T is 3x1 column vector)
Then if Pw = [X Y Z] is a point in world coord. system 2 I can get its camera system coords.with:
Pc = [ Rinv Tinv] * [X Y Z 1] transposed.
Pc looks like [a b c d]
Following the same logic again I can get the coordinates of Pc relative to coord. system 1:
Pw1 = [ R1 T1] * Pc
Should I normalize Pc or just normalize Pw1 at the end?
I found how to translate points between coordinate systems in this OpenCV Demo.
The explanation from "Demo 3: Homography from the camera displacement" (the section spanning from title right until the first lines of code) shows how to translate between coordinate systems using matrix multiplication. I just had to apply it to my situation (I had CMO1 and CMO2 and needed to find O1MO2).
This way I can get two planes in the same coord. system, get their normals and find the angle between them.
Also it helped to realize that the extrinsic matrix [R T] translates a 3D point from the world coord. system to the camera coord. system (where camera is at (0,0,0)), not the other way around.
I'm currently stuck on achieving an equirectangular rotation on a 360° image with OpenCV because of my mathematical understanding (nearly zero) of projections and rotations matrixes.
The result of a such rotation would be exactly what you can see here: https://www.youtube.com/watch?v=l1N0lEKIeLA
I found some code here: https://github.com/FoxelSA/libgnomonic/wiki/Equirectangular-rotation_v0.1 but I didn't succeed to apply it to opencv
If someone has any idea how to apply it for an OpenCV Mat and Pitch, Yaw, Roll angles it would be highly appreciated!
Thanks!
Instead of talking about yaw, pitch and roll, I'll talk here about Euler angles x, y and z.
To perform a rotation of your equirectangular mapping, you can follow this procedure:
Consider coordinates (i2, j2) in your result image. We'll try to find which color to put here. These coordinates correspond to a point on the sphere with latitude lat2 = 180 * i2 / image.height and longitude lon2 = 360 * j2 / image.width. Compute the corresponding 3D vector v2.
Compute the rotation matrix R with angles x, y and z (look at the formulas here). Take the transpose of this matrix to get the inverse rotation from the new image to the old one. We'll name this inverse rotation matrix Rt.
Compute v1 = Rt * v2. Then compute the latitude lat1 and longitude lon1 of v1.
Find the color in the original image at coordinates i1 = image.height * lat1 / 180 and j1 = image.width * lon1 / 360. This might not be integer coordinates. You might have to interpolate between several pixels to get your value. This is the color of the pixel at position (i2, j2) in your new image.
You'll need to look at how to convert between 3D vectors on a sphere and their latitude and longitude angles but this shouldn't be too hard to find. The algorithm described here should be rather straightforward to implement.
Let me know if I made any mistake as I haven't tested it myself.
My stereo camera has different resolutions 1280x480, 640x240 and 320x120. (The camera stream a synchronized pair of images 640X480 pasted horizontally that's why is 1280x480).
I have used the algorithm of Opencv3 Stereo Calibration in the following link to calibrate the stereo camera with the resolution of 1280*480.
stereoRectify( M1, D1, M2, D2, img_size, R, T, R1, R2, P1, P2, Q, CALIB_ZERO_DISPARITY, -1, img_size, &roi1, &roi2 );
Mat map11, map12, map21, map22;
initUndistortRectifyMap(M1, D1, R1, P1, img_size, CV_16SC2, map11, map12);
initUndistortRectifyMap(M2, D2, R2, P2, img_size, CV_16SC2, map21, map22);
Mat img1r, img2r;
remap(img1, img1r, map11, map12, INTER_LINEAR);
remap(img2, img2r, map21, map22, INTER_LINEAR);
The stereoRectify computes the rotation matrix R, translation matrix T between left and right camera. And also, it calculates two matrices of rotation R1, R2 and two matrices of projection P1 P2. I used the output of stereoRectify as the input of initUndistortRectifyMap and then remap to apply the projection.
Here is a stackoverflow answer to explain how to do it.
Now I have got the two matrices of rectification map of left map11, map12 and right camera map21, map22.
But now I want to use these camera matrices M1 and M2, distortion matrices D1 and D2, and extrinsic matrices R, T, R1, R2, P1 and P2 to rectify the camera images in a lower resolution (320x120) each. PS I haven't calibrated the camera with the resolution of 320*120 directly because the image is too small and the algorithm of Opencv can't find the chessboard corners to perform the calibration.
I know that "The distortion coefficients do not depend on the scene viewed. Thus, they also belong to the intrinsic camera parameters. And they remain the same regardless of the captured image resolution. If, for example, a camera has been calibrated on images of 320 x 240 resolution, absolutely the same distortion coefficients can be used for 640 x 480 images from the same camera while f_x, f_y, c_x, and c_y need to be scaled appropriately."
according the documentation of opencv. ( I tested and it is working)
I want to know: how should I modify the matrices of R, T, R1, R2, P1, P2 to do the remap in a lower resolution from 1280x480 to 320x120?.
Actually
I change the matrices of P1 and P2 which are the combination of camera intrinsic matrix and camera translation matrix.
I just divide them by 4 for using 320x120. The general formula is:
fx' = (dimx' / dimx) * fx
fy' = (dimy' / dimy) * fy
fx' is the value for your new resolution, fx is the value you already have for your original resolution, dimx' is the new resolution along the x axis, dimx is the original one. The same applies to fy.
cx and cy are calculated analogically because all these values are expressed in pixel coordinates.
After much playing around not only P1 and P2 should be scaled, but also M1 and M2 (in the OP's code).
Remember, that scaling is only two first diagonal entries of left scaling matrix, and preserve the third one to be 1.
I am wrestling with the Fish-Eye Camera Model used in OpenCV 3.0.0.dev. I have read the documentation in this link several times, especially the "Detailed Description" part and formulas modeling fish-eye distortion. By now I have two concerns:
Based on the projection models listed here and their conceptual explanations in "Accuracy of Fish-Eye Lens Model" By Hughes, I can't figure out which projection model has been used in the OpenCV implementation.
Since the description is so concise, I need to know the main reference papers used by OpenCV developers for implementing fish-eye namespace, so that I could be on the ball and get through more details.
P.S. I checked OpenCV 3.0.0-dev documentation and did not find anything useful.
Short answer:
OpenCV 3.0.0 Fisheye camera model doesn't use the Brown model nor any of the models that the OP refers from Panotools, it uses a Generic Camera Model by Juho Kannala and Sami S. Brandt.
Detailed answer:
As Cfr points in his answer, this comment from Ilya Krylov (who implemented the fisheye model in OpenCV) says they ported the Camera Calibration Toolbox for Matlab of Jean-Yves Bouguet:
Jean-Yves Bouguet website (link), in turn, mentions the paper A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses, and says :
The "undocumented" fisheye model contained in the calibration toolbox follows the equidistance projection model described by equation (3) in this very nice paper. The distortion model follows equation (6), to the exception that k1=1 (otherwise indistinguishable from f).
Which in my opinion is a misleading statement or a plain misconception as equation(3) and equation(6) correspond to different models: equation (6) is the actual model introduced in this paper which the authors refer as the Generic Camera Model (hence the name of the paper).
To be more precise, equation (6) was meant to be used as the camera model and equation (8) and (9) as the "distortion" or deviation from this model.
But the odyssey is not over. OpenCV's implementation (according to its documentation) first computes the pinhole projection to find the field angle (the angle between the 3D point, the center of projection and the optical axis). This means you can't use their fisheye model to project rays at 90º (or you would divide by 0) or close to 90º (numerical stability problems, like overflow could happen if z is small enough).
Moreover I'm not sure whether it will work for rays for more than 90º.
All this makes me really wonder the "usefulness" of their fisheye camera model for fisheye or wide angle lenses.
In case you're skeptic about that you can take a look at OpenCV's source code, concretely at sources\modules\calib3d\src\fisheye.cpp (I added some comments)
void cv::fisheye::projectPoints(InputArray objectPoints, OutputArray imagePoints, InputArray _rvec,
InputArray _tvec, InputArray _K, InputArray _D, double alpha, OutputArray jacobian)
{
...
Rodrigues(om, R, dRdom);
Affine3d aff(om, T);
...
Vec3d Xi = objectPoints.depth() == CV_32F ? (Vec3d)Xf[i] : Xd[i];
Vec3d Y = aff*Xi; /* To transform to camera reference frame*/
Vec2d x(Y[0]/Y[2], Y[1]/Y[2]); /* <- The root of all evil (division by z) */
double r2 = x.dot(x);
double r = std::sqrt(r2);
// Angle of the incoming ray:
double theta = atan(r);
double theta2 = theta*theta, theta3 = theta2*theta, theta4 = theta2*theta2, theta5 = theta4*theta,
theta6 = theta3*theta3, theta7 = theta6*theta, theta8 = theta4*theta4, theta9 = theta8*theta;
double theta_d = theta + k[0]*theta3 + k[1]*theta5 + k[2]*theta7 + k[3]*theta9;
double inv_r = r > 1e-8 ? 1.0/r : 1;
double cdist = r > 1e-8 ? theta_d * inv_r : 1;
Vec2d xd1 = x * cdist;
Vec2d xd3(xd1[0] + alpha*xd1[1], xd1[1]);
Vec2d final_point(xd3[0] * f[0] + c[0], xd3[1] * f[1] + c[1]);
...
}
Update: This pull request fixes the problem with rays at angles ≥ 90º. As of April 2018 it hasn't been merged into master yet but is being considered for OpenCV 4.x Calibration Module (check calibration module discussion too)
After hours of reading, I found that the formula θ=atan(r) in OpenCV's fish-eye documentation, is the normalized inverse of r=f*tanθ pertaining to pinhole projection and hence, none of the fish-eye projection models mentioned in the above links are used in OpenCV.
In addition, regarding the distortion model, what I guess is that the Division Model of Fitzgibbon in his 2001 paper "Simultaneous linear estimation of multiple view geometry and lens distortion" is used. According to Hughes in his 2008 paper "Review of Geometric Distortion Compensation in Fish-Eye Cameras", among the other alternatives are "Odd Polynomial Model" and "Polynomial Fish-Eye Transform". In his paper, in page 2, he has written:
"(1) (which refers to Odd Polynomial Model) and (3) (which refers to Division Model, which I guess is the one used by OpenCV) can be used to describe distortion in standard,non-fisheye lenses. However, it is generally considered that these polynomial models are insufficient to describe the level of distortion introduced by fish-eye lenses. Shah and Aggarwal have shown in [9] (Intrinsic Parameter Calibration Procedure For A High-Distortion Fish-Eye Lens Camera With Distortion Model And Accuracy Estimation) that even when using a 7th order version of (1) to model fish-eye radial distortion, considerable distortion remains, to the extent that they had to use a model with greater degrees of freedom. Therefore, a polynomial that uses both odd and even coefficients (instead of simply one or the other) can be used to model the radial distortion introduced by a fisheye lens"
After all, I conclude that the fish-eye model in OpenCV has very limited applicability and could be much more strengthened in terms of distortion models and projection models. I'd like to re-emphasize that I still need to know what papers were used by the OpenCV developers to implement the fisheye namespace.
I'd deeply appreciate anyone's comments on this.
according to Bouguet's calib_tool code "project_points_fisheye.m"
%Definitions:
%Let P be a point in 3D of coordinates X in the world reference frame (stored in the matrix X)
%The coordinate vector of P in the camera reference frame is: Xc = R*X + T
%where R is the rotation matrix corresponding to the rotation vector om: R = rodrigues(om);
%call x, y and z the 3 coordinates of Xc: x = Xc(1); y = Xc(2); z = Xc(3);
%The pinehole projection coordinates of P is [a;b] where a=x/z and b=y/z.
%call r^2 = a^2 + b^2,
%call theta = atan(r),
%Fisheye distortion -> theta_d = theta * (1 + k(1)*theta^2 + k(2)*theta^4 + k(3)*theta^6 + k(4)*theta^8)
%
%The distorted point coordinates are: xd = [xx;yy] where:
%
%xx = (theta_d / r) * x
%yy = (theta_d / r) * y
%
%Finally, convertion into pixel coordinates: The final pixel coordinates vector xp=[xxp;yyp] where:
%
%xxp = f(1)*(xx + alpha*yy) + c(1)
%yyp = f(2)*yy + c(2)
It is Brown-Conrady model, mentioned in Camera Calibration Toolbox for Matlab references. Discussed in this paper: D. C. Brown "Close-Range Camera Calibration".
Also it seems current OpenCV model ignores tangential distortion (P(r)).
See this comment.
I am doing camera calibration from tsai algo. I got intrensic and extrinsic matrix, but how can I reconstruct the 3D coordinates from that inormation?
1) I can use Gaussian Elimination for find X,Y,Z,W and then points will be X/W , Y/W , Z/W as homogeneous system.
2) I can use the
OpenCV documentation approach:
as I know u, v, R , t , I can compute X,Y,Z.
However both methods end up in different results that are not correct.
What am I'm doing wrong?
If you got extrinsic parameters then you got everything. That means that you can have Homography from the extrinsics (also called CameraPose). Pose is a 3x4 matrix, homography is a 3x3 matrix, H defined as
H = K*[r1, r2, t], //eqn 8.1, Hartley and Zisserman
with K being the camera intrinsic matrix, r1 and r2 being the first two columns of the rotation matrix, R; t is the translation vector.
Then normalize dividing everything by t3.
What happens to column r3, don't we use it? No, because it is redundant as it is the cross-product of the 2 first columns of pose.
Now that you have homography, project the points. Your 2d points are x,y. Add them a z=1, so they are now 3d. Project them as follows:
p = [x y 1];
projection = H * p; //project
projnorm = projection / p(z); //normalize
Hope this helps.
As nicely stated in the comments above, projecting 2D image coordinates into 3D "camera space" inherently requires making up the z coordinates, as this information is totally lost in the image. One solution is to assign a dummy value (z = 1) to each of the 2D image space points before projection as answered by Jav_Rock.
p = [x y 1];
projection = H * p; //project
projnorm = projection / p(z); //normalize
One interesting alternative to this dummy solution is to train a model to predict the depth of each point prior to reprojection into 3D camera-space. I tried this method and had a high degree of success using a Pytorch CNN trained on 3D bounding boxes from the KITTI dataset. Would be happy to provide code but it'd be a bit lengthy for posting here.