Perspective projection backwards - computer-vision

Question

[1 0 0 0; 0 1 0 0; 0 0 1/f 0][x y z 1]' = [x y z/f]' -> (fx/z f*y/z) = (u,v)
This converts 3D points (x,y,z) to pixels (u,v). How can I go from pixels to 3D points? Sorry, I'm not very smart.

Answer 1

Unfortunately, you lose depth information when you project a point. So you can recover the original 3d point only up to scale. Let's re-write your transformation like this:
calib_mat=[f 0 0 ;
0 f 0 ;
0 0 1]
I removed the last column since it doesn't have any impact. Then we have
calib_mat*[x y z]'==[fx fy z]=1/z * [fx/z fy/z 1]= 1/z * [u v 1].
Now, assume you know [u v 1] and you want to recover the 3d point. But now, the depth information is lost, so what you know is
calib_mat*[x y z]'= (1/unknown_depth) * [ u v 1]
Therefore,
[x y z]'=(1/unknown_depth)*inverse(calib_mat)*[u v 1]
So you have obtained what you wanted, but up to scale. To recover the depth of the point, you either need multiple (at least two) views of the point in question (for triangluation, for example). Or, if you are not in a computer vision context but in a rendering context, you can save the depth in some sort of z-buffer when you are projecting the point.

Answer 2

When you project three dimensional space onto a two dimensional image, you lose information about the depth and it is difficult to obtain the lost depth information about depth from only one frame. However, depth information can be regained if you have another image of the same scene taken from a different angle. Your brain does something similar to understand depth by using the "images" from your two eyes to give you an understanding of the depth of the world around you.
The underlying principles of stereo reconstruction is best explained this way: hold any object close to your eyes, then close one eye. Then open that eye and close the other. Then do the same thing again, but move the object farther away from your eyes. You will notice that the object will move a lot more when you switch which eye is opened when the object is close to your eyes than it does when it is farther. In the context of two images, the amount (in pixels) a single feature on one image moves between two images of the same scene is called "disparity." To calculate relative depth of the scene, simply take (1.0/disparity). To obtain the absolute depth of the scene (e.g. in meters or some unit of measurement), the focal length and baseline (distance between the two camera locations) is needed (and equations for doing so are discussed later).
Now that you know how the depth of each pixel is calculated, all that is left is to match features so that you can calculate disparity. If you were to iteratively find each pixel in your first image in the second image, it would become quickly unwieldy. However, the "search problem" is simplified by the fact that there exists an "epipolar line" between any two images that significantly decreases the possible locations for a feature in image1 to appear in image2. The easiest way to visualize this is to think of two cameras placed such that the only difference between the first and second camera is that the second camera has been moved horizontally from the first (so the cameras are at the same height, and both are the same depth away from the scene). Intuitively, say there is a ball in image1 at a certain pixel (x1, y1). Given that the cameras have taken a picture of the same ball at the same height as each other, it is intuitive that, while the pixel location of the feature in image1 of the ball may not be in the same location in image2, that at least the same feature in image2 will have the same y as it did in image1 since they were both taken at the same height. In that case, the epipolar line would be completely horizontal. With knowledge of this epipolar line, one no longer needs to search all of image2 for the location of a feature found image1 -- instead only the epipolar line through the location of the feature in image1 needs to be searched in image2. While the cameras do not need to be placed next to each other with no difference between their positions except horizontal translation, it makes computation much simpler and more intuitive, as otherwise the epipolar line would be sloped. So, in order to match feature1 from image1 to feature2 in image2, one simply must use a feature comparison technique (normalized cross correlation is often used) to determine what the most likely location of feature2 in image2 is. Given the matched location of a feature in both images, the disparity can be calculated by taking the magnitude between the two pixels.
After features are matched, the disparity of the pixel is calculated through some equations shown on page 7 of these lecture notes, where b is the baseline between the cameras, and l is the focal length in the unit of measurement you wish to use (e.g. inches, meters, etc.). If you are only looking for relative three dimensional location of the pixels in the image, and don't care about the location of the pixels (i.e. a point on the left of an image will still be in the left in the reconstruction, and a point farther back in the image will be farther back in the reconstruction), arbitrary non-zero values can be chosen for focal length and baseline. Those notes also explain some more intuition as to why this works if you are still curious.
Feel free to ask any questions, and there's no reason to be down on yourself -- either way you are seeking out knowledge and that is commendable.