I'm making multicamera-stereo calibration program.
My idea is to rectify each pair of cameras separately.
For example: for given 3 cameras I compute undistortion and rectification maps (using stereoRectify() and initUndistortRectifyMap()) separately for {camera[1] and camera[2]}, {camera[2] and camera[3]} and {camera[1] and camera[3]}.
Using remap(), I can transform any original image (from, lets say, camera[1]) to one of two different rectified images: rectified[1][2] and rectified[1][3].
Now, also using remap(), for any point from that original image, I can compute its new coordinates separately in rectified[1][2] and rectified[1][3] images.
It works well, but now I need to compute these coordinates in opposite direction: for any point from any of rectified images I need to find its original coordinates in its original image.
How can I do this?
Related
I have to do an assignment in which I should compare the function solvePnP() used with SOLVEPNP_EPNP and solvePnPRansac() used with SOLVEPNP_ITERATIVE. The goal is to calculate a warped image from an input image.
To do this, I get an RGB input image, the same image as 16bit depth information image, the camera intrinsics and a list of feature match points between the given image and the wanted resulting warped image (which is the same scene from a different perspective.
This is how I went about this task so far:
Calculate a list of 3D object points form the depth image and the the intrinsics which correspond to the list of feature matches.
use solvePnP() and solvePnPRansac() with the respective algorithms where the calculated 3D object points and the feature match points of the resulting image are the inputs. As a result I get a rotation and a translation vector for both methods.
As sanity check I calculate the average reprojection error using projectPoints() for all feature match points and comparing the resulting projected points to the feature match points of the resulting image.
Finally I calculate 3D object points for each pixel of the input image and again project them using the rotation and translation vector from before. Each projected point will get the color from the corresponding pixel in the input image resulting in the final warped image.
These are my inputs:
Using steps described above I get the following output with the Ransac Method:
This looks pretty much like the reference solution I have, so this should be mostly correct.
However with the solvePnP() method using SOLVEPNP_EPNP the resulting rotation and translation vectors look like this, which doesn't make sense at all:
================ solvePnP using SOVLEPNP_EPNP results: ===============
Rotation: [-4.3160208e+08; -4.3160208e+08; -4.3160208e+08]
Translation: [-4.3160208e+08; -4.3160208e+08; -4.3160208e+08]
The assignment sheet states, that the list of feature matches contain some miss - matches, so basically outliers. As far as I know, Ransac handles outliers better, however can this be the reason for this weird results for the other method? I was expecting some anomalies, but this is completely wrong and the resulting image is completely black since no points are inside the image area.
Maybe someone can point me into the right direction.
OK, I could solve the issue. I used float for all all calculation (3D object points, matches, ...) beforehand and tried to change everything to double - it did the trick.
The warped perspective is still off and I get a rather high re-projection error, however this should be due to the nature of the algorithm itself, which doesn't handle outliers very well.
The weird thing about this is, that in the OpenCV documentation on solvePnP() it states that vector<Point3f> and vector<Point2f>can be passed as arguments for object points and image points respectively.
I'm currently working on an assignment, consisting of camera calibration, stereo-calibration and finally stereo matching. For this I am allowed to use the available OpenCV samples and tutorials and adapt them to our needs.
While the first two parts weren't much of a problem, a Question about the stereo matching part:
We should create a colored point cloud in .ply format from the disparity map of two provided images.
I'm using this code as a template:
https://github.com/opencv/opencv/blob/master/samples/cpp/stereo_match.cpp
I get the intrinsic and extrinsic files from the first two parts of the assignment.
My question is, how to get the corresponding color for each 3D-point from the original images and the disparity map?
I'm guessing that each coordinate of the disparity map corresponds to a pixel both input-images share. But how to get those pixelvalues?
EDIT: I know that the value of each element of the disparity map represents the disparity the corresponding pixel between the left and right image. But how do I get the corresponding pixels from the coordinates of the disparity map?
Example:
my disparity value at coordinates (x, y) is 128. 128 represents the depth. But how do I know which pixel in the original left or right image this corresponds to?
Additional Questions
I'm having further questions about StereoSGBM and which parameters make sense.
Here are my (downscaled for upload) input-images:
left:
right
Which give me these rectified images:
left
right
From this I get this disparity image:
For the disparity image: this is the best result I could achieve using blocksize=3 and numDisparities=512. However I'm not at all sure if those parameters make any sense. Are these values sensible?
My question is, how to get the corresponding color for each 3D-point from the original images and the disparity map?
So a disparity map is nothing but distance between matching pixels in the epipolar plane in the left and right images. This means, you just need the pixel intensity to compute the disparity which in turn implies, you could do this computation on either just the grey-scale left-right image or any of the channels of the left-right images.
I am pretty sure the disparity image you are computing operates on grey-scale images obtained from the original rgb images. If you want to compute a color disparity image, you just need to extract the individual color channels of the left-right images, and compute the corresponding disparity map channel. The outcome will then be a 3 channel disparity map.
Additional Questions I'm having further questions about StereoSGBM and which parameters make sense. Here are my (downscaled for upload) input-images:
There is never a good answer to this for the most general case. You need a parameter tuner for this. See https://github.com/guimeira/stereo-tuner as an example. You should be able to write your own in open cv pretty easily if you want.
Ok the solution to this problem is to use the projectpoint() function from OpenCV.
Basically calculate 3D-Points from the disparity image and project them onto the 2D image and use the color you hit.
I am new to Stereo vision. I know what disparity and disparity map is. I couldn't understand the concept of V-disparity. Can someone explain how it can be generated and how can it be used for image stabilization?
Here is a sample image and its V disparity
Each row of the V-disparity image is a histogram of the various values of disparity that appeared on that row in the disparity map.
When done right, the disparities of the points on the ground plane will appear as a strong line in the V-disparity map.
When you are considering how to use it for stabilization, I suspect you should first consider what kinds of transforms you are trying to stabilize against. E.g., are you trying to stabilize it against translations of the stereo rig that are up/down or left/right in the image plane? Or are you more concerned about rotations of the stereo rig around the optical axis?
Each of these kinds of transforms will have different effects on that strong line. If you can model what these effects will be for the different kinds of transforms, then you can achieve the stabilization.
I am using open cv and C++. I have 2 face images which contain marker points on them. I have already found the coordinates of the marker points. Now I need to align those 2 face images based on those coordinates. The 2 images may not be necessarily of the same height, that is why I can't figure out how to start aligning them, what should be done etc.
In your case, you cannot apply the homography based alignment procedure. Why not? Because it does not fit in this use case. It was designed to align flat surfaces. Faces (3D objects) with markers at different places and depths are clearly no planar surface.
Instead, you can:
try to match the markers between images then interpolate the displacement field of the other pixels. Classical ways of doing it will include moving least squares interpolation or RBF's;
otherwise, a more "Face Processing" way of doing it would be to use the decomposition of faces images between a texture and a face model (like AAM does) and work using the decomposition of your faces in this setup.
Define "align".
Or rather, notice that there does not exist a unique warp of the face-side image that matches the overlapping parts of the frontal one - meaning that there are infinite such warps.
So you need to better specify what your goal is, and what extra information you have, in addition to the images and a few matched points on them. For example, is your camera setup calibrated? I.e do you know the focal lengths of the cameras and their relative position and poses?
Are you trying to build a texture map (e.g. a projective one) so you can plaster a "merged" face image on top of a 3d model that you already have? Then you may want to look into cylindrical or spherical maps, and build a cylindrical or spherical projection of your images from their calibrated poses.
Or are you trying to reconstruct the whole 3d shape of the head based on those 2 views? Obviously you can do this only over the small strip where the two images overlap, and they quality of the images you posted seems a little too poor for that.
Or...?
The stereo_match.cpp example converts L and R images into disparity and point cloud. I want to adapt this example for compute the disparity and point cloud from 2 consecutive frames of a single calibrated camera. Is it possible? If this example isn't good for my scope, what are the steps for obtain what I want?
Disparity map, on stereo systems, is used to obtain depth information - distance to objects in scene. For that, you need the distance between cameras, to be able to convert disparity info to real dimensions.
On the other hand, if you have consecutive frames from a static camera, I suppose you want the differences between them. You can obtain it with an optical flow algorithm. Dense optical flow is calculated for each pixel in image, in the same way as disparity, and it outputs the movement direction and magnitude. Most common OF are sparse - they track only a set of "strong", or well-defined points.
It may make sense to obtain disparity algorithms if you have a static scene, but you move the camera, simulating the two cameras in a stereo rig.
Yes if the camera (or scene) is moving
I suppose we cannot calculate an accurate disparity map from a single camera. In computing the disparity map we basically assume that the vertical pixel coordinate in both the images in a stereo rig is same, only the horizontal pixel coordinate changes, but in monocular image sequence, this may not hold true as the camera is moving between two consecutive frames.