I'm currently stuck on achieving an equirectangular rotation on a 360° image with OpenCV because of my mathematical understanding (nearly zero) of projections and rotations matrixes.
The result of a such rotation would be exactly what you can see here: https://www.youtube.com/watch?v=l1N0lEKIeLA
I found some code here: https://github.com/FoxelSA/libgnomonic/wiki/Equirectangular-rotation_v0.1 but I didn't succeed to apply it to opencv
If someone has any idea how to apply it for an OpenCV Mat and Pitch, Yaw, Roll angles it would be highly appreciated!
Thanks!
Instead of talking about yaw, pitch and roll, I'll talk here about Euler angles x, y and z.
To perform a rotation of your equirectangular mapping, you can follow this procedure:
Consider coordinates (i2, j2) in your result image. We'll try to find which color to put here. These coordinates correspond to a point on the sphere with latitude lat2 = 180 * i2 / image.height and longitude lon2 = 360 * j2 / image.width. Compute the corresponding 3D vector v2.
Compute the rotation matrix R with angles x, y and z (look at the formulas here). Take the transpose of this matrix to get the inverse rotation from the new image to the old one. We'll name this inverse rotation matrix Rt.
Compute v1 = Rt * v2. Then compute the latitude lat1 and longitude lon1 of v1.
Find the color in the original image at coordinates i1 = image.height * lat1 / 180 and j1 = image.width * lon1 / 360. This might not be integer coordinates. You might have to interpolate between several pixels to get your value. This is the color of the pixel at position (i2, j2) in your new image.
You'll need to look at how to convert between 3D vectors on a sphere and their latitude and longitude angles but this shouldn't be too hard to find. The algorithm described here should be rather straightforward to implement.
Let me know if I made any mistake as I haven't tested it myself.
Related
I'm trying to use quaternions to do rotation animation.
My algorithm creates Quaternions, and slerps every frame.
Here is my code to construct a quaternion by the axis and the rotation angle.
template <typename U>
Quaternion(Vector3<U> vec, const float& angle)
{
vec.normalize();
float cosa = cos(angle/2);
float sina = sin(angle/2);
w = cosa;
x = sina * vec.x;
y = sina * vec.y;
z = sina * vec.z;
}
Then I found that when I tried to rotate 4π radians, the animation does not work because the quaternion I created is equivalent to 0 degrees.
I wonder if quaternions can represent rotations over 360 degrees? Or is my animation algorithm in need of improvement?
I wonder if quaternions can represent rotations over 360 degrees?
No, it can not.
Quaternions between the range [360;720] will treated as rotations at the other direction: [-360;0].
And quaternions between the range [720*k; 720*(k+1)] will be treated as rotations [0;720].
If you use slerp for this kind of animation, quaternions are not good for them.
Quaternions can only slerp between angles which are smaller than 360.
If you still want to do this, use a different representation, like axis-angle.
Rotating be 360 degrees is the same as rotating by 0 degrees. To rotate by an angle alpha bigger than 360 simply rotate by alpha-360 or more general by alpha % 360.
(360 used as synonym for 2pi, you need to take care about degree vs radians of course. And not sure if thats a typo, but 360 degree is 2pi not 4pi)
PS: Actually I think there is nothing wrong with your code, and maybe you dont have to change anything. It's just your expectations that were wrong: You should get the same for a rotation by 4pi as for a rotation by 0.
Think of quaternions as instant rotations - rotating by 4π radians instantly is the same as doing nothing.
This is not what you want when you animate rotation of 4π radians over 20 seconds. You can solve it by creating an Euler Vector (a 3D vector whose direction represents the axis of rotation, same as in quaternion, while its length represents the speed/angle of the rotation), see https://en.wikipedia.org/wiki/Axis%E2%80%93angle_representation. Later, multiply it by time passed and convert it into quaternion or 3D matrix depending on what your graphics wants.
The following link suggests that we can convert dual fisheye coordinates to equirectangular coordinates using the following equations:
// 2D fisheye to 3D vector
phi = r * aperture / 2
theta = atan2(y, x)
// 3D vector to longitude/latitude
longitude = atan2(Py, Px)
latitude = atan2(Pz, (Px^2 + Py^2)^(0.5))
// 3D vector to 2D equirectangular
x = longitude / PI
y = 2 * latitude / PI
I applied to above equations to write my source code like this:
const float FOV = 220.0f * PI / 180.0f;
float r = sqrt(u*u + v*v);
float theta = atan2(v, u);
float phi = r * FOV * 0.5f;
float px = u;
float py = r * sin(phi);
float pz = v;
float longitude = atan2(py, px); // theta
float latitude = atan2(pz, sqrt(px*px + py*py)); // phi
x = longitude / PI;
y = 2.0f * latitude / PI;
Unfortunately my math is not good enough to understand this and not sure if I write the above code correctly, where I tried to guess the values for px, py and pz.
Assume my camera FOV is 220 degrees, and the camera resolution is 2880x1440, I would expect the point (358, 224) for rear camera in the overlapped area and the point (2563, 197) for front camera in the overlapped area would both map to a coordinate close to (2205, 1009). However the actual mapping points are (515.966370,1834.647949) and (1644.442017,1853.060669) respectively, which are both very far away from (2205,1009). Please kindly suggest how to fix the above code. Many thanks!
You are building the equirectangular image, so I would suggest you to use the inverse mapping.
Start with pixel locations in the target image you are painting. Convert the 2D location to longitude/latitude. Then convert that to a 3D point on the surface of the unit sphere. Then convert from the 3D point to a location in the 2D fisheye source image. In Paul Bourke page, you would start with the bottom equation, then the rightmost one, then the topmost one.
Use landmark points like 90° long 0° lat, to verify the results make sense at each step.
The final result should be a location in the source fisheye image in the [-1..+1] range. Remap to pixel or to UV as needed. Since the source is split in two eye images you will also need a mapping from target (equirect) longitudes to the correct source sub-image.
Recently I used RGBD-Slam to obtain kinect camera trajectory. Each frame will have this:
timestamp tx ty tz qx qy qz qw
tx ty tz give the position of the optical center of the color camera with respect to the world origin as defined by the motion capture system.
qx qy qz qw give the orientation of the optical center of the color camera in form of a unit quaternion with respect to the world origin as defined by the motion capture system.
I need to convert it to 4x4 homogeneous transformation matrix. Any hints to solve this?. Let's say the example as follows:
1421307756.191874 0.004530 0.050319 0.003332 0.003734 0.023578 -0.054199 0.998245
so,
timestamp: 1421307756.191874 ; tx: 0.004530
ty: 0.050319 ; tz: 0.003332
qx: 0.003734 ; qy: 0.023578
qz: -0.054199 ; qw: 0.998245
and,
fx = 525.0; fy = 525.0; // default focal length
cx = 319.5; cy = 239.5; // default optical center
Kind assistance would be greatly appreciated.
Thanks.
extracting angle and axis of the quaternion shouldn't be hard right ?
then you, can find the general expression of a rotation matrix with a given axis
then just combining the obtained matrix with the translation matrix in the right order should do the trick
it's my first post plz be indulgent if I'm saying shit :-P
I have a chessboard in two images with some angle of rotation. Lets find the rotation angle of second image with reference of first image.
For that I found the Rotation Matrix (3x3) and translation matrix (3x1) of those objects.
How can I find the Rotation Angle and Rotation Axis of object using those matrices?
For every type of conversion between rotation representations you have this website euclidean space.
You will find theory and code samples of:
Rotation matrix to quaternion: link
Quaternion to axis angle: link
Rotations in general and all representations: link
And in relation to your question you have Axis Angle. If you have the rotation matrix R (3x3), you can obtain the angle and axis this way (see Matrix to Axis Angle):
angle = acos(( R00 + R11 + R22 - 1)/2);
Axis x,y,x:
x = (R21 - R12)/sqrt((R21 - R12)^2+(R02 - R20)^2+(R10 - R01)^2);
y = (R02 - R20)/sqrt((R21 - R12)^2+(R02 - R20)^2+(R10 - R01)^2);
z = (R10 - R01)/sqrt((R21 - R12)^2+(R02 - R20)^2+(R10 - R01)^2);
Already working wih openCV I would rcommend using the Rodrigues method:
cv::Rodrigues(src, dst, jacobian), that computes the rotation vector if you have a rotation matrix for an argument and vice versa.
I am doing camera calibration from tsai algo. I got intrensic and extrinsic matrix, but how can I reconstruct the 3D coordinates from that inormation?
1) I can use Gaussian Elimination for find X,Y,Z,W and then points will be X/W , Y/W , Z/W as homogeneous system.
2) I can use the
OpenCV documentation approach:
as I know u, v, R , t , I can compute X,Y,Z.
However both methods end up in different results that are not correct.
What am I'm doing wrong?
If you got extrinsic parameters then you got everything. That means that you can have Homography from the extrinsics (also called CameraPose). Pose is a 3x4 matrix, homography is a 3x3 matrix, H defined as
H = K*[r1, r2, t], //eqn 8.1, Hartley and Zisserman
with K being the camera intrinsic matrix, r1 and r2 being the first two columns of the rotation matrix, R; t is the translation vector.
Then normalize dividing everything by t3.
What happens to column r3, don't we use it? No, because it is redundant as it is the cross-product of the 2 first columns of pose.
Now that you have homography, project the points. Your 2d points are x,y. Add them a z=1, so they are now 3d. Project them as follows:
p = [x y 1];
projection = H * p; //project
projnorm = projection / p(z); //normalize
Hope this helps.
As nicely stated in the comments above, projecting 2D image coordinates into 3D "camera space" inherently requires making up the z coordinates, as this information is totally lost in the image. One solution is to assign a dummy value (z = 1) to each of the 2D image space points before projection as answered by Jav_Rock.
p = [x y 1];
projection = H * p; //project
projnorm = projection / p(z); //normalize
One interesting alternative to this dummy solution is to train a model to predict the depth of each point prior to reprojection into 3D camera-space. I tried this method and had a high degree of success using a Pytorch CNN trained on 3D bounding boxes from the KITTI dataset. Would be happy to provide code but it'd be a bit lengthy for posting here.