I have been working on video stabilization for quite a few weeks now. The algorithm I'm following basically involves 3 steps :-
1. FAST feature detection and Matching
2. Calculating affine transformation (scale + rotation + translation x + translation y ) from matched keypoints
3. Smooth motion parameters using cubic spline or b-spline.
I have been able to calculate affine transform. But I am stuck at smoothing motion parameters. I have been unable to evaluate spline function to smooth the three parameters.
Here is a graph for smoothed data points
Any suggestion or help as to how can I code to get a desired result as shown in the graph?
Here is the code that calculate the points on the curve
B-spline Curves
But now the code will use all control points as transform parameters to formulate.
I think i will run in post-processing (not real time).
Did you run B spline smoothing in real time?
Related
The issue that I'm having is working out how 3D Perlin Noise is used as a density function with the Dual Contouring Algorithm. I've found it difficult to find any Dual Contouring implementations or blogs/write ups that use or discuss the specifics of how they use 3D Perlin Noise with Dual Contouring to generate terrain.
I've managed to implement Dual Contouring with an implicit sphere so know its working correctly and have a basic idea of how density functions works (at least in a spheres case), have a basic knowledge of Perlin Noise (implemented the 3D version tested via texture output and basic cubic voxel terrain generation) and understand how to compute the gradient (analytical derivative in this case) for use with the QEF calculation.
I've experimented with a number of ways to try get relatively decent looking terrain with no success:
1) Generate the density values for the grid via Perlin noise (i.e. value < 1 is inside, > 1 is outside and 0 is on the contour) and then generate the intersection values (where the contour intersects the cube) for the cubes completely separately again using Perlin Noise, using the co-ordinate input as the index of the cube we're on with a minor offset. (Basically just generated a really cubic surface)
2) Use Perlin Noise in a similar way to the sphere function, generate the density grid via Perlin Noise and then for each intersection interpolate along the edge between the two nearby cubes density values until the Perlin Noise hits 0 (i.e. the exact point that the density function intersects the edge) then use the co-ordinate that triggered the 0 output from the Perlin Noise. (Again generates a cubic surface, think it might be because the interpolation usually triggers 0 when the co-ords hit an integer value i.e. on the Perlin latices edge.)
3) This is the one I've had most success with, similar to 2) except i keep the input values to the Perlin Noise within a 0.1 to 0.9 range i.e. kept within a single lattice. It feeds out ok values for some cubes but calculating the QEF (the averaged point for the cube) sends a fair amount of points well outside the range of a cube despite all the intersection points being well within the cube. I think its something to do with the way I use the Perlin Gradients, essentially I interpolate to find the co-ordinate that generates the 0 for the Perlin Noise return value, I then use the gradient(analytically derivative) from this co-ordinate and the culmination of all these sends the co-ordinate miles away.
Feel free to ask for more information to be posted up if its required I'll do my best, this is my first time posting so sorry if the post is sub-par.
tl;dr How is 3D Perlin Noise used as a density function to generate terrain in a Dual Contouring Algorithm? specifically in generating intersection points and normals for edges for the QEF calculation.
Edit: The new current method I'm trying (and failing) to use to integrate Perlin Noise using the perlin noise generator to fill up the initial density data, after this point for any edge intersection I do some simple linear maths to find the ratio on the line where the 0 crosses (so if a density value on one side was 0.5 and on the other side it was -0.5 the value given out would be that the 0 is 50% of the way between them so 0.5f). Once I have this value i simply add it on to the relevant co-ordinate to get the intersection point on that edge. After this point I use the intersection as input to calculate the Perlin Noises 3D Gradient (essentially just running the noise function with it as an input and calculating the analytically derivative using the method found here: 3D Perlin noise analytical derivative).
Once this is done the gradient and intersection values are fed into the QEF function, which uses SVD decomposition instead of the QR decomposition described as being more accurate within the original Dual Contouring Paper (found SVD to be simpler looking and it also sounded like it'd be less performance heavy from the paper).
A few screenshots to hopefully help a little by showing what it currently looks like. The first has the diagonal code commented out and the second has the Diagonal in place, the number immediately after all the prints of "vertex out of bounds" is the number of vertices outside of their cube.
http://imgur.com/di1DxRQ
The second with the diagonal division occurring:
http://imgur.com/X4uCab3
I'm working on a Project for my University where we want a Quadrcopter to stabilize himself with his camera. Unfortunately the Fundamental matrix reacts very sensible to little changes within the featurpoints, i'll give you examples later on.
I think my matching already works pretty good thanks to ocv.
I'm using SURF Features and match them with the knn-Method:
SurfFeatureDetector surf_detect;
surf_detect = SurfFeatureDetector(400);
//detect keypoints
surf_detect.detect(fr_one.img, fr_one.kp);
surf_detect.detect(fr_two.img, fr_two.kp);
//extract keypoints
SurfDescriptorExtractor surf_extract;
surf_extract.compute(fr_one.img, fr_one.kp, fr_one.descriptors);
surf_extract.compute(fr_two.img, fr_two.kp, fr_two.descriptors);
//match keypoints
vector<vector<DMatch> > matches1,matches2;
vector<DMatch> symMatches,goodMatches;
FlannBasedMatcher flann_match;
flann_match.knnMatch(fr_one.descriptors, fr_two.descriptors, matches1,2);
flann_match.knnMatch(fr_two.descriptors, fr_one.descriptors, matches2,2);
//test matches in both ways
symmetryTest(matches1,matches2,symMatches);
std::vector<cv::Point2f> points1, points2;
for (std::vector<cv::DMatch>::const_iterator it= symMatches.begin();
it!= symMatches.end(); ++it)
{
//left keypoints
float x= fr_one.kp[it->queryIdx].pt.x;
float y= fr_one.kp[it->queryIdx].pt.y;
points1.push_back(cv::Point2f(x,y));
//right keypoints
x = fr_two.kp[it->trainIdx].pt.x;
y = fr_two.kp[it->trainIdx].pt.y;
points2.push_back(cv::Point2f(x,y));
}
//kill outliers with ransac
vector<uchar> inliers(points1.size(),0);
findFundamentalMat(Mat(points1),Mat(points2),
inliers,CV_FM_RANSAC,3.f,0.99f);
std::vector<uchar>::const_iterator
itIn= inliers.begin();
std::vector<cv::DMatch>::const_iterator
itM= symMatches.begin();
for ( ;itIn!= inliers.end(); ++itIn, ++itM)
{
if (*itIn)
{
goodMatches.push_back(*itM);
}
}
Now i want to compute the Fundamental Matrix with these matches. I'm using the 8POINT method for this example - i already tried it with LMEDS and RANSAC - there it only get's worse because there are more matches which change.
vector<int> pointIndexes1;
vector<int> pointIndexes2;
for (vector<DMatch>::const_iterator it= goodMatches.begin();
it!= goodMatches.end(); ++it) {
pointIndexes1.push_back(it->queryIdx);
pointIndexes2.push_back(it->trainIdx);
}
vector<Point2f> selPoints1, selPoints2;
KeyPoint::convert(fr_one.kp,selPoints1,pointIndexes1);
KeyPoint::convert(fr_two.kp,selPoints2,pointIndexes2);
Mat F = findFundamentalMat(Mat(selPoints1),Mat(selPoints2),CV_FM_8POINT);
When i call these calculations within a loop on the same pair of images the result of F varies very much - theres no way to extract movement from such calculations.
I generated an example where i filtered out some matches so that you can see the effect i mentioned for yourselves.
http://abload.de/img/div_c_01ascel.png
http://abload.de/img/div_c_02zpflj.png
Is there something wrong with my code or do i have to think about other reasons like image-quality and so on ?
Thanks in advance for the Help !
derfreak
To summarize what others have already stated and elaborate in more detail,
As currently implemented in OpenCV, the 8-point algorithm has no outlier rejection. It is a least-squares algorithm and cannot be used with RANSAC or LMEDS because these flags override the 8-point flag. It is recommended that the input points are normalized to improve the condition number of the matrix in the linear equation, as stated in "In Defence of the 8-point Algorithm". However, the OpenCV implementation automatically normalizes the input points, so there is no need to normalize them manually.
The 5-point and 7-point algorithms both have outlier rejection, using RANSAC or LMEDS. If you are using RANSAC, you may need to tune the threshold to get good results. The OpenCV documentation shows that the default threshold for RANSAC is 1.0, which in my opinion is a bit large. I might recommend using something around 0.1 pixels. On the other hand, if you are using LMEDS you won't need to worry about the threshold, because LMEDS minimizes the median error instead of counting inliers. LMEDS and RANSAC both have similar accuracy if the correct threshold is used and both have comparable computation time.
The 5-point algorithm is more robust than the 7-point algorithm because it only has 5 degrees of freedom (3 rotation and 2 for the unit-vector translation) instead of 7 (the additional 2 parameters are for the camera principle points). This minimal parameterization allows the rotation and translation to be easily extracted from the matrix using SVD and avoids the planar structure degeneracy problem.
However, in order to get accurate results with the 5-point algorithm, the focal length must be known. The paper suggests that the focal length should be known within 10%, otherwise the 5-point algorithm is no better than the other uncalibrated algorithms. If you haven't performed camera calibration before, check out the OpenCV camera calibration tutorial. Also, if you are using ROS, there is a nice camera calibration package.
When using the OpenCV findEssentialMat function I recommend first passing the pixel points to undistortPoints. This not only reverses the effect of lens distortion, but also transforms the coordinates to normalized image coordinates. Normalized image coordinates (not to be confused with the normalization done in the 8-point algorithm) are camera agnostic coordinates that do not depend on any of the camera intrinsic parameters. They represent the angle of the bearing vector to the point in the real world. For example, a normalized image coordinate of (1, 0) would correspond to a bearing angle of 45 degrees from the optical axis of the camera in the x direction and 0 degrees in the y direction.
After using RANSAC to obtain a good hypothesis, the best estimate can be improved by using iterative robust non-linear least-squares. This is mentioned in the paper and described in more detail in "Bundle Adjustment - A Modern Synthesis". Unfortunately, it appears that the OpenCV implementation of the 5-point algorithm does not use any iterative refinement methods.
Even if your algorithm is correct, 8 point F matrix computation is very error prone due to image noise. The lesser correspondences you use the better. The best you can do is doing 5 point Essential (E) matrix computation, but that would require you to pre-calibrate the camera and convert the detected pixel image points after SIFT/SURF to normalized pixels (metric pixel locations). Then apply Nister's 5-point algorithm either from the freely available Matlab implementation or from Bundler (c++ implementation by Noah Snavely). In my experience with SfM, 5-point E matrix is much much better/stable than 7 or 8 point F matrix computation. And ofcourse do RANSAC after 5 point to get more robust estimates. Hope this helps.
The 8-point algorithm is the simplest method of computing fundamental matrix, but if care is taken you can perform it well. The key to obtain the good results is proper careful normalization of the input data before constructing the equations to solve. Many of algorithms can do it.
Pixels point coordinate must be changed to camera coordinates, I don't see that you are doing these. As I understand, your
vector<int> pointIndexes1; is expressed in the pixel coordinates.
You must known the intrinsic camera parameters, if you want get more stable results. You may find them by many methods: tutorial openCV. Then you have two options of normalize it. You may apply for your fundamental matrix,
Mat E = K.t() * F * K; where K is Intrinsic Camera Parameters.[see on Wiki]
However this assumption is not accurate. If camera calibration matrix K is known, then you may apply inverse to the point x to obtain the point expressed in camera normalized coordinates.
pointNormalize1= K.inv()*pointIndexes1 where pointIndexes1(2), z is equal 1.
In the case of the 8PA, a simple transformation of points improve and hence in the stability of the results. The suggested normalization is a translation and scaling of each image so that the centroid of the reference points is at origin of the coordinates and the RMS distance of the points from the origin is equal to ![sqrt{2}]. Note that it is recommended that the singularity condition should be enforced before denormalization.
Reference: check it if : you are still interested
I am working on image processing with OPENCV.
I want to find the x,y and the rotational displacement between two images in OPENCV.
I have found the features of the images using SURF and the features have been matched.
Now i want to find the displacement between the images. How do I do that? Can RANSAC be useful here?
regards,
shiksha
Rotation and two translations are three unknowns so your min number of matches is two (since each match delivers two equations or constraints). Indeed imagine a line segment between two points in one image and the corresponding (matched) line segment in another image. The difference between segments' orientations gives you a rotation angle. After you rotated just use any of the matched points to find translation. Thus this is 3DOF problem that requires two points. It is called Euclidean transformation or rigid body transformation or orthogonal Procrustes.
Using Homography (that is 8DOF problem ) that has no close form solution and relies on non-linear optimization is a bad idea. It is slow (in RANSAC case) and inaccurate since it adds 5 extra DOF. RANSAC is only needed if you have outliers. In the case of pure noise and overdetrmined system (more than 2 points) your optimal solution that minimizes the sum of squares of geometric distance between matched points is given in a close form by:
Problem statement: min([R*P+t-Q]2), R-rotation, t-translation
Solution: R = VUT, t = R*Pmean-Qmean
where X=P-Pmean; Y=Q-Qmean and we take SVD to get X*YT=ULVT; all matrices have data points as columns. For a gentle intro into rigid transformations see this
The quality of calibration is measured by the reprojection error (is there an alternative?), which requires a knowledge world coordinates of some 3d point(s).
Is there a simple way to produce such known points? Is there a way to verify the calibration in some other way (for example, Zhang's calibration method only requires that the calibration object be planar and the geometry of the system need not to be known)
You can verify the accuracy of the estimated nonlinear lens distortion parameters independently of pose. Capture images of straight edges (e.g. a plumb line, or a laser stripe on a flat surface) spanning the field of view (an easy way to span the FOV is to rotate the camera keeping the plumb line fixed, then add all the images). Pick points on said line images, undistort their coordinates, fit mathematical lines, compute error.
For the linear part, you can also capture images of multiple planar rigs at a known relative pose, either moving one planar target with a repeatable/accurate rig (e.g. a turntable), or mounting multiple planar targets at known angles from each other (e.g. three planes at 90 deg from each other).
As always, a compromise is in order between accuracy requirements and budget. With enough money and a friendly machine shop nearby you can let your fantasy run wild with rig geometry. I had once a dodecahedron about the size of a grapefruit, machined out of white plastic to 1/20 mm spec. Used it to calibrate the pose of a camera on the end effector of a robotic arm, moving it on a sphere around a fixed point. The dodecahedron has really nice properties in regard to occlusion angles. Needless to say, it's all patented.
The images used in generating the intrinsic calibration can also be used to verify it. A good example of this is the camera-calib tool from the Mobile Robot Programming Toolkit (MRPT).
Per Zhang's method, the MRPT calibration proceeds as follows:
Process the input images:
1a. Locate the calibration target (extract the chessboard corners)
1b. Estimate the camera's pose relative to the target, assuming that the target is a planar chessboard with a known number of intersections.
1c. Assign points on the image to a model of the calibration target in relative 3D coordinates.
Find an intrinsic calibration that best explains all of the models generated in 1b/c.
Once the intrinsic calibration is generated, we can go back to the source images.
For each image, multiply the estimated camera pose with the intrinsic calibration, then apply that to each of the points derived in 1c.
This will map the relative 3D points from the target model back to the 2D calibration source image. The difference between the original image feature (chessboard corner) and the reprojected point is the calibration error.
MRPT performs this test on all input images and will give you an aggregate reprojection error.
If you want to verify a full system, including both the camera intrinsics and the camera-to-world transform, you will probably need to build a jig that places the camera and target in a known configuration, then test calculated 3D points against real-world measurements.
On Engine's question: the pose matrix is a [R|t] matrix where R is a pure 3D rotation and t a translation vector. If you have computed a homography from the image, section 3.1 of Zhang's Microsoft Technical Report (http://research.microsoft.com/en-us/um/people/zhang/Papers/TR98-71.pdf) gives a closed form method to obtain both R and t using the known homography and the intrinsic camera matrix K. ( I can't comment, so I added as a new answer)
Should be just variance and bias in calibration (pixel re-projection) errors given enough variability in calibration rig poses. It is better to visualize these errors rather than to look at the values. For example, error vectors pointing to the center would be indicative of wrong focal length. Observing curved lines can give intuition about distortion coefficients.
To calibrate the camera one has to jointly solve for extrinsic and intrinsic. The latter can be known from manufacturer, the solving for extrinsic (rotation and translation) involves decomposition of calculated homography: Decompose Homography matrix in opencv python
Calculate a Homography with only Translation, Rotation and Scale in Opencv
The homography is used here since most calibration targets are flat.
I am trying to extract the curvature of a pulse along its profile (see the picture below). The pulse is calculated on a grid of length and height: 150 x 100 cells by using Finite Differences, implemented in C++.
I extracted all the points with the same value (contour/ level set) and marked them as the red continuous line in the picture below. The other colors are negligible.
Then I tried to find the curvature from this already noisy (due to grid discretization) contour line by the following means:
(moving average already applied)
1) Curvature via Tangents
The curvature of the line at point P is defined by:
So the curvature is the limes of angle delta over the arclength between P and N. Since my points have a certain distance between them, I could not approximate the limes enough, so that the curvature was not calculated correctly. I tested it with a circle, which naturally has a constant curvature. But I could not reproduce this (only 1 significant digit was correct).
2) Second derivative of the line parametrized by arclength
I calculated the first derivative of the line with respect to arclength, smoothed with a moving average and then took the derivative again (2nd derivative). But here I also got only 1 significant digit correct.
Unfortunately taking a derivative multiplies the already inherent noise to larger levels.
3) Approximating the line locally with a circle
Since the reciprocal of the circle radius is the curvature I used the following approach:
This worked best so far (2 correct significant digits), but I need to refine even further. So my new idea is the following:
Instead of using the values at the discrete points to determine the curvature, I want to approximate the pulse profile with a 3 dimensional spline surface. Then I extract the level set of a certain value from it to gain a smooth line of points, which I can find a nice curvature from.
So far I could not find a C++ library which can generate such a Bezier spline surface. Could you maybe point me to any?
Also do you think this approach is worth giving a shot, or will I lose too much accuracy in my curvature?
Do you know of any other approach?
With very kind regards,
Jan
edit: It seems I can not post pictures as a new user, so I removed all of them from my question, even though I find them important to explain my issue. Is there any way I can still show them?
edit2: ok, done :)
There is ALGLIB that supports various flavours of interpolation:
Polynomial interpolation
Rational interpolation
Spline interpolation
Least squares fitting (linear/nonlinear)
Bilinear and bicubic spline interpolation
Fast RBF interpolation/fitting
I don't know whether it meets all of your requirements. I personally have not worked with this library yet, but I believe cubic spline interpolation could be what you are looking for (two times differentiable).
In order to prevent an overfitting to your noisy input points you should apply some sort of smoothing mechanism, e.g. you could try if things like Moving Window Average/Gaussian/FIR filters are applicable. Also have a look at (Cubic) Smoothing Splines.