I have 2 images of the same scene but with slightly difference. It is due to the fact that I have taken the picture with 2 different equipment, and one is not perfectly right.
That create a little parallax problem somewhere on the photo. It is really slight when you compare the 2 photos, but just enough disturbing for what I want to do with those images.
My question is : what would be the best algorithm to minimize the created error?
What I was thinking was :
pick X points in the first image, pick their correspondance points in the other image, then create a function that project the first points to the other and then apply this function to all the points in the scene.
Something like that :
(Red point are picked points, and other colors are the deformation of each points in the scene (probably not correct since I did it fastly).
Would it be a good way to proceed? If yes, what would be a good interpolation algorithm using the points.
If no, is there any other algorithm that I could use in order the minimize the error?
Thanks a lot!
Yes, using 4 pairs of points you can calculate matrix of perspective transformation and later apply this transformation to the whole image.
OpenCV library contains getPerspectiveTransform function for this case (also findHomography). You can also apply warpPerspective to correct image.
Example
Related
I'm using a 3D scanner in order to scan rectangular objects and measure them (width and length). But, due to the position respect the sensor or also to the vertexs of the rectangle, blur appears at some sides. This causes the measure to have not enought accuracy.
What kind of preprocessing (OpenCV with C++) do you suggest me in order to find the correct contour? Do you think there is a better solution that using preprocessing? Note that the intensity of a pixel is a translation of it height respect the zero plane.
Here you have an example: a rubber on three different places. As you can see, blur appears at one side depending on this placing. The real size of the rubber is (at the image) 179x182 px.
Thank you!
EDIT: Forget to say that the blur affects different sides depending on the rubber's position respect the horizontal axis (middle row).
I can't see that you can ever measure it accurately if the image is blurred. I don't think this is a C++ question, it's a mechanical one.
You need to have a model/concept about the reason for the blur. Without it, there is no computation that comes closer to the truth. With such a model you may be able to adjust your computations. If for example your idea is that the blur is caused by the fact that the object is not orthogonal to the sensor and there is data from another side of the object you might want to cut off the object at the maximum value (closest to the sensor). For this you could use a threshold close to the maximum value. Please note that this is just an example.
You will need to de-blurr your image before doing measurement, it's all depends on the way the blurring happened, somehow you will need to estimate the way blurring happened. The simplest model is called shift-invariant mode, take a look at this link
MATLAB link and this link
I realize there are many cans of worms related to what I'm asking, but I have to start somewhere. Basically, what I'm asking is:
Given two photos of a scene, taken with unknown cameras, to what extent can I determine the (relative) warping between the photos?
Below are two images of the 1904 World's Fair. They were taken at different levels on the wireless telegraph tower, so the cameras are more or less vertically in line. My goal is to create a model of the area (in Blender, if it matters) from these and other photos. I'm not looking for a fully automated solution, e.g., I have no problem with manually picking points and features.
Over the past month, I've taught myself what I can about projective transformations and epipolar geometry. For some pairs of photos, I can do pretty well by finding the fundamental matrix F from point correspondences. But the two below are causing me problems. I suspect that there's some sort of warping - maybe just an aspect ratio change, maybe more than that.
My process is as follows:
I find correspondences between the two photos (the red jagged lines seen below).
I run the point pairs through Matlab (actually Octave) to find the epipoles. Currently, I'm using Peter Kovesi's
Peter's Functions for Computer Vision.
In Blender, I set up two cameras with the images overlaid. I orient the first camera based on the vanishing points. I also determine the focal lengths from the vanishing points. I orient the second camera relative to the first using the epipoles and one of the point pairs (below, the point at the top of the bandstand).
For each point pair, I project a ray from each camera through its sample point, and mark the closest covergence of the pair (in light yellow below). I realize that this leaves out information from the fundamental matrix - see below.
As you can see, the points don't converge very well. The ones from the left spread out the further you go horizontally from the bandstand point. I'm guessing that this shows differences in the camera intrinsics. Unfortunately, I can't find a way to find the intrinsics from an F derived from point correspondences.
In the end, I don't think I care about the individual intrinsics per se. What I really need is a way to apply the intrinsics to "correct" the images so that I can use them as overlays to manually refine the model.
Is this possible? Do I need other information? Obviously, I have little hope of finding anything about the camera intrinsics. There is some obvious structural info though, such as which features are orthogonal. I saw a hint somewhere that the vanishing points can be used to further refine or upgrade the transformations, but I couldn't find anything specific.
Update 1
I may have found a solution, but I'd like someone with some knowledge of the subject to weigh in before I post it as an answer. It turns out that Peter's Functions for Computer Vision has a function for doing a RANSAC estimate of the homography from the sample points. Using m2 = H*m1, I should be able to plot the mapping of m1 -> m2 over top of the actual m2 points on the second image.
The only problem is, I'm not sure I believe what I'm seeing. Even on an image pair that lines up pretty well using the epipoles from F, the mapping from the homography looks pretty bad.
I'll try to capture an understandable image, but is there anything wrong with my reasoning?
A couple answers and suggestions (in no particular order):
A homography will only correctly map between point correspondences when either (a) the camera undergoes a pure rotation (no translation) or (b) the corresponding points are all co-planar.
The fundamental matrix only relates uncalibrated cameras. The process of recovering a camera's calibration parameters (intrinsics) from unknown scenes, known as "auto-calibration" is a rather difficult problem. You'd need these parameters (focal length, principal point) to correctly reconstruct the scene.
If you have (many) more images of this scene, you could try using a system such as Visual SFM: http://ccwu.me/vsfm/ It will attempt to automatically solve the Structure From Motion problem, including point matching, auto-calibration and sparse 3D reconstruction.
I'm trying to track little white dots on edges table. In most of case it works. I'm using cornerHarris function like it's used in this tutorial : http://docs.opencv.org/doc/tutorials/features2d/trackingmotion/harris_detector/harris_detector.html .
Sometimes, I have got a problem : reflection of the light on edges creates point of interest which I have to not consider.
For example :
I'm searching to the two nearest points of the top corners, as you can see on the right edges, i have find dots(red and green dots) and on the left edges, light noise is a problem (cyan and blue dots).
Does someone knows a method to keep only dots white on my picture ? Thankyou and sorry for my english
On the purely image processing part, I would recommend using some kind of shape feature analysis(like comparing the histogram in say 8x8 around your currently examined point of interest to precomputed ones of the features you want .
This would mean that you first look for points with Harris corners, then compare the features to dismiss unwanted ones ( euclidian distance in the 8x8 = 64D ?). This of course assumes the existence of a strong feature (read "taking time to find a good one")
It also assumes you already know what your feature points look like beforehand.
Alternative more on the computer vision side : use geometry of your corner points repartition to your advantage : you probably want a distorted rectangle, so make sure you find one ! Surely you can compute a function that gives the validity of the last feature point assuming 3 others ? (Distance of intersection of the 2 lines generated by the other 3 points ...)
The typical and coolest approach would then apply RANSAC to it : try random (but not all !) combinations of your points and check which one fits best using that function, and consider those as good.
If you intend on tracking over time or over several images, you will have to tune it a bit, as ransac can occasionally fail (statistics of random combinations ...), and you would then use points from your previously successful run to guestimate the position.
Last idea for the moment : use some color-aware derivation technique : do you compute Harris corner of the rgb image or of a flattened version to gray ? Some gradients use color information as extra tip to discern edges, and I'm not sure the corners you're finding use any of those. Then again it might mean reimplementing Harris corners algorithm (try it, it's fun, and not that hard if you have a good algebra library to do the heavy work)
I recommend the geometric test of fitting as it uses wisely model-info of your system rather than assumptions on how reflections look like.
Really funny introduction to RANSAC : danielwedge.com/ransac/
Edit : Trusty photoshop knows what I mean : I highlighted the invalid shapes
Valid grid, photoshop says so
Invalid grid, logical, right ?
I don't really know how to explain it in a better way, so please look at the following images :
This is what I create for the moment
This is what I whish to create instead
I am currently using C++ with Qt 4.8.
Do you know a way that would allow me to reach my goal ? Using a library or a transformation matrix ? Or something else ?
I am a total newbie to image manipulation, so every advice is precious for me.
Thanks
EDIT :
I draw each colored pixel from Lat/Long measures, if it can help.
Use what is called a morphological operator. In this case, you would require the 'open' operator. OpenCV provides a pretty good implementation (and documentation of these) which can be found here.
Draw circles instead of points is all I can think of. Creating a triangle mesh is tricky with the concave elements of the distribution.
EDIT: Just looked at the full size version of the image and wondered if the data set is stored radially? You could scan adjacent radial lines and try to match up the changes in value along each line to form a set of quads. There will be a large number of edge conditions to consider though.
EDIT2: Alternatively, form a uniformly distributed set of quads and interpolate the vertex colours.
you can start by increasing the size of the points,
you could create a triangle mesh by using a sweepline algorithm:
sort the points by lat
keep a subset sorted by long
when you add a point compare to the 4 adjacent points and add triangles to the "to draw" set (remove points too far away from the current lat as needed)
with opengl you can use an index buffer to hold which point should be drawn
I'm trying to remove foreground from two images, here's a sample pair of images:
As you can see, the Budweiser bottle is removed from the scene before the second shot is taken.
These photos were captured from a pinhole camera (iPhone), and, the tricky part is I'm hand-holding the camera, so it cannot be guaranteed that the images are perfectly aligned pixel by pixel, so a simple minus-threshold method will not work.
Then, I've decided to perform image registration using findHomography and warpPerspective from OpenCV, here's the result image:
This image is warped with the matrix I've got from findHomography, it kind of improved the alignment quality, but still not that aligned so I can use a simple way to remove the foreground.
So, finally, I decided to implement a "fuzzy-minus" algorithm: for every pixel in image1, I'll look through a 7x7 neighbour in image2 (a 7 by 7 kernel?), using the minimal difference in grayscale as the result of minus, and threshold the result into binary image, here's what I've got:
And the result is still not good. Notice the white wholes in the bottle, this is produced due to similar grayscale value of foreground and background. So I'm not sure what to do now.
I can think of two ways to solve the problem, the first is to get a better aligned pair of images, and simply minus the pairs; the second is to use a more robust way to extract the foreground.
Can anyone give me some advice on how to deal with this kind of problem? I believe there should be some state-of-art algorithms or processing pipelines, but after googling around, I get nothing.
I'm using OpenCV with C++, it would be fantastic if you can tell me how to do it with these tools in hand.
Big big thanks in advance!
The problem is not in your algorithm. You are having problem because the two scenes were not taken from exactly the same angle, as shown in the animation below. This slight difference highlight the edges in the subtraction.
You need a static camera in order to apply this approach.
I suggest using mathematical morphology on the mask that you got to get rid of the artifacts.
Try applying both opening and closing to get rid of the black and the white small regions.
Mathematical Morphology
Mathematical Morphology in opencv
The difference between the two picture is pretty huge, so you will need to use a large structure element, but I don't think you will be able to get rid of the shadow.
For the two large strips in the background, you may try to use a horizontally shaped structure element as well.
Edit
Is it possible to produce a grayscale image instead of a binary image? if yes, you may try to experiment with the hat method for the shadow, but I am not sure about this point.
This is what I got using two different structure elements for closing THEN opening
Mat mask = imread("mask.jpg",CV_LOAD_IMAGE_GRAYSCALE);
morphologyEx(mask,mask,MORPH_CLOSE,getStructuringElement(CV_SHAPE_ELLIPSE,Size(50,10)));
morphologyEx(mask,mask,MORPH_OPEN,getStructuringElement(CV_SHAPE_ELLIPSE,Size(10,50)));
imshow("open",mask);
imwrite("maskopenclose.jpg",mask);
I would suggest optical flow for alignment and OpenCV's background subtraction algorithm:
http://docs.opencv.org/trunk/doc/tutorials/video/background_subtraction/background_subtraction.html
I suggest that instead of using findHomography try using some of openCV's stereo correspondence functions: http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html
there is a sample code here: https://github.com/Itseez/opencv/blob/master/samples/cpp/stereo_calib.cpp