I've been trying to calculate the features in 2 images and then pass those features back to CameraParams.R without luck. The features are calculated and matched successfully, however, the problem is passing them back to R & t.
I understand that you must decompose the Homography in order for this to be possible, which I've done using something like this: https://github.com/syilma/homography-decomp, but am I really doing it right?
Right now I'm simply using:
Matching:
vector< vector<DMatch> > matches;
Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create(algorithmName);
matcher->knnMatch( descriptors_1, descriptors_2, matches, 50 );
vector< DMatch > good_matches; // Storing good matches here
I've noticed that the good_matches isn't used anywhere. So I guess my question is, how can I pass back good_matches to cameras.R/t?
Extracting Homography:
Mat K;
cameras[img_idx].K().convertTo(K, CV_32F);
findHomography -> decomposeHomography(H, K, outputR,outputT,noarray()).
Then by utilizing the library above, I pass in the values from R & t but the response is that the homography isn't found in the 4 possible outcomes.
Am I on the right path here? Seems like decomposeHomography is a 3D solution, but, findHomography is 2D?
Absolute Goal:
Refine CameraParam.R/t depending on the features found in the images.
Why? Because I'm currently passing in the .R from the devices rotation matrix but the rotation is slightly inaccurate. See more info about it on my previous question: Refining Camera parameters and calculating errors - OpenCV
If you are using the calculated R for image stitching, then there is no need to use decompose homography. Whole stitching pipeline assumes zero translation. So it gives perfect output for only rotation case and slight error is introduced with the introduction of translation in camera pose. If you look into opencv calculation of R from homography, it assumes 0 translation.
Mat R = K_from.inv() * pairwise_matches[pair_idx].H.inv() * K_to;
cameras[edge.to].R = cameras[edge.from].R * R;
You can find the source code in motion_estimators.cpp ->calcRotation function.
Coming to your question of using goodmatches for calculating R. goodmatches are actually used to calculate homography matrix, using findhomography function
So the whole process will be like
Find matches (as you mentioned)
Find homography matrix from these matches using findhomography
function
Use calcrotation function to find R
Find focal using findfocalfromhomography function and create intrinsic matrix
Use warper, seamfinder and blender for final stitching output
Related
I have correct correspondances between consecutive frames, and need to estimate the transformation between them to generate a trajectory. The following C++ pipeline, the generated trajectory goes no sense.
auto EssentialMatrix = cv::findEssentialMat(points_previous,
points_current,
camera_focal_length,
camera_principal_point,
cv::RANSAC,
0.999,
1.0,
mask);
auto inliers = cv::recoverPose(EssentialMatrix,
points_previous,
points_current,
CameraMatrix,
R,
t,
mask);
t_pos_ = t_pos_ + 1.0 *(R_pos_*t);
R_pos_ = R * R_pos_;
So, my question is: how to correctly recover the transformation between two consecutive frames with C++ OpenCV utilities? Are additional steps needed to do so?
I really have no idea whats is your problem.
Try to give like whats the input, whats "your expected output" whats the actual output.
But i do know where to find sample https://github.com/avisingh599/mono-vo
try to take a look at sample here. It is pure 2D to 2D opencv based estimation. only thing different is that they use dataset saclaing to adjust the translational vector scale. 2D to 2D alone only give a relative translation.
http://www.youtube.com/watch?v=homos4vd_Zs
We are trying continuosly process the image frames captured by the two cameras, process every two frames and then stitch them to get a complete view. In order to do so we have
1.extracted surf features.
2.Got the matches between the two images using Flann Matcher.
3.Computed the homography matrix using these matches.
4.Applied warpPerspective on the right image.
//To get the surf keypoints and descriptors:
cuda::SURD_CUDA surf(700);
surf(leftImgGpu, cuda::GpuMat(), keyPointsGpuA, descriptorsAGpu);
surf(rightImgGpu, cuda::GpuMat(), keyPointsGpuB, descriptorsBGpu);
surf.downloadKeypoints(keypointsAGpu, keypoiintsA);
surf.downloadKeypoints(keypointsBGpu, keypoiintsB);
//Flann based matcher:
FlannBasedMatcher matcher(new cv::flann::KDTreeIndexParams(4), new
cv::flann::SearchParams())
//To get the homography matrix:
vector<Point2f> imgPtsA, imgPtsB;
for(int i=0;i<matches.size();i++){
imgPtsB.push_back(keypointsB[matches[i].queryIdx].pt);
imgPtsA.push_back(keypointsA[matches[i].trainIdx].pt);
}
Mat H=findHomography(imgPtsA, imgPtsB, CV_RANSAC);
//To a warp right image:
warpPerspective(rightImg, warpRight, H, rightImg.size());
We have two issues:
Issue 1:
The warped image3 is moving around. The left and right cameras are fixed and the images( left, right) we are processing is almost the same everytime. We suspect there is some issue with the matches and the homography matrix becuase of which the warped image is not coming properly.
Issue 2:
We intially used BF Matcher to get the matches. When constructed the Homograpy mat using these matches, we were getting weird results. After using the Flann based matcher the result was comparatively better.
To create a proper "panorama" image with stitching the cameras need to be on nearly on the same position in space, otherwise parallax errors will occur (see here). In general case, a homography can only warp a single plane within the image such it is registered with its pendant. Thus, it would be possible e.g. to stitch just the floor (if it provides enough texture for features of course).
So, you can not expect a stable result, since the homography can not model this transformation. More vividly: The front side of the chair is only visible in the right image, so it is not possible to "match" this area with the left image.
I am totally new to OpenCV and I have started to dive into it. But I'd need a little bit of help.
So I want to combine these 2 images:
I would like the 2 images to match along their edges (ignoring the very right part of the image for now)
Can anyone please point me into the right direction? I have tried using the findTransformECC function. Here's my implementation:
cv::Mat im1 = [imageArray[1] CVMat3];
cv::Mat im2 = [imageArray[0] CVMat3];
// Convert images to gray scale;
cv::Mat im1_gray, im2_gray;
cvtColor(im1, im1_gray, CV_BGR2GRAY);
cvtColor(im2, im2_gray, CV_BGR2GRAY);
// Define the motion model
const int warp_mode = cv::MOTION_AFFINE;
// Set a 2x3 or 3x3 warp matrix depending on the motion model.
cv::Mat warp_matrix;
// Initialize the matrix to identity
if ( warp_mode == cv::MOTION_HOMOGRAPHY )
warp_matrix = cv::Mat::eye(3, 3, CV_32F);
else
warp_matrix = cv::Mat::eye(2, 3, CV_32F);
// Specify the number of iterations.
int number_of_iterations = 50;
// Specify the threshold of the increment
// in the correlation coefficient between two iterations
double termination_eps = 1e-10;
// Define termination criteria
cv::TermCriteria criteria (cv::TermCriteria::COUNT+cv::TermCriteria::EPS, number_of_iterations, termination_eps);
// Run the ECC algorithm. The results are stored in warp_matrix.
findTransformECC(
im1_gray,
im2_gray,
warp_matrix,
warp_mode,
criteria
);
// Storage for warped image.
cv::Mat im2_aligned;
if (warp_mode != cv::MOTION_HOMOGRAPHY)
// Use warpAffine for Translation, Euclidean and Affine
warpAffine(im2, im2_aligned, warp_matrix, im1.size(), cv::INTER_LINEAR + cv::WARP_INVERSE_MAP);
else
// Use warpPerspective for Homography
warpPerspective (im2, im2_aligned, warp_matrix, im1.size(),cv::INTER_LINEAR + cv::WARP_INVERSE_MAP);
UIImage* result = [UIImage imageWithCVMat:im2_aligned];
return result;
I have tried playing around with the termination_eps and number_of_iterations and increased/decreased those values, but they didn't really make a big difference.
So here's the result:
What can I do to improve my result?
EDIT: I have marked the problematic edges with red circles. The goal is to warp the bottom image and make it match with the lines from the image above:
I did a little bit of research and I'm afraid the findTransformECC function won't give me the result I'd like to have :-(
Something important to add:
I actually have an array of those image "stripes", 8 in this case, they all look similar to the images shown here and they all need to be processed to match the line. I have tried experimenting with the stitch function of OpenCV, but the results were horrible.
EDIT:
Here are the 3 source images:
The result should be something like this:
I transformed every image along the lines that should match. Lines that are too far away from each other can be ignored (the shadow and the piece of road on the right portion of the image)
By your images, it seems that they overlap. Since you said the stitch function didn't get you the desired results, implement your own stitching. I'm trying to do something close to that too. Here is a tutorial on how to implement it in c++: https://ramsrigoutham.com/2012/11/22/panorama-image-stitching-in-opencv/
You can use Hough algorithm with high threshold on two images and then compare the vertical lines on both of them - most of them should be shifted a bit, but keep the angle.
This is what I've got from running this algorithm on one of the pictures:
Filtering out horizontal lines should be easy(as they are represented as Vec4i), and then you can align the remaining lines together.
Here is the example of using it in OpenCV's documentation.
UPDATE: another thought. Aligning the lines together can be done with the concept similar to how cross-correlation function works. Doesn't matter if picture 1 has 10 lines, and picture 2 has 100 lines, position of shift with most lines aligned(which is, mostly, the maximum for CCF) should be pretty close to the answer, though this might require some tweaking - for example giving weight to every line based on its length, angle, etc. Computer vision never has a direct way, huh :)
UPDATE 2: I actually wonder if taking bottom pixels line of top image as an array 1 and top pixels line of bottom image as array 2 and running general CCF over them, then using its maximum as shift could work too... But I think it would be a known method if it worked good.
I'm trying to deskew an image that has an element of known size. Given this image:
I can use aruco:: estimatePoseBoard which returns rotation and translation vectors. Is there a way to use that information to deskew everything that's in the same plane as the marker board? (Unfortunately my linear algebra is rudimentary at best.)
Clarification
I know how to deskew the marker board. What I want to be able to do is deskew the other things (in this case, the cloud-shaped object) in the same plane as the marker board. I'm trying to determine whether or not that's possible and, if so, how to do it. I can already put four markers around the object I want to deskew and use the detected corners as input to getPerspectiveTransform along with the known distance between them. But for our real-world application it may be difficult for the user to place markers exactly. It would be much easier if they could place a single marker board in the frame and have the software deskew the other objects.
Since you tagged OpenCV:
From the image I can see that you have detected the corners of all the black box. So just get the most border for points in a way or another:
Then it is like this:
std::vector<cv::Point2f> src_points={/*Fill your 4 corners here*/};
std::vector<cv::Point2f> dst_points={cv:Point2f(0,0), cv::Point2f(width,0), cv::Point2f(width,height),cv::Point2f(0,height)};
auto H=v::getPerspectiveTransform(src_points,dst_points);
cv::Mat copped_image;
cv::warpPerspective(full_image,copped_image,H,cv::Size(width,height));
I was stuck on the assumption that the destination points in the call to getPerspectiveTransform had to be the corners of the output image (as they are in Humam's suggestion). Once it dawned on me that the destination points could be somewhere within the output image I had my answer.
float boardX = 1240;
float boardY = 1570;
float boardWidth = 1730;
float boardHeight = 1400;
vector<Point2f> destinationCorners;
destinationCorners(Point2f(boardX+boardWidth, boardY));
destinationCorners(Point2f(boardX+boardWidth, boardY+boardHeight));
destinationCorners(Point2f(boardX, boardY+boardHeight));
destinationCorners(Point2f(boardX, boardY));
Mat h = getPerspectiveTransform(detectedCorners, destinationCorners);
Mat bigImage(image.size() * 3, image.type(), Scalar(0, 50, 50));
warpPerspective(image, bigImage, h, bigImage.size());
This fixed the perspective of the board and everything in its plane. (The waviness of the board is due to the fact that the paper wasn't lying flat in the original photo.)
When using findHomography as used the OpenCV Features2D + Homography Documentation, it calls on CV_RANSAC as its third parameter:
Mat H = findHomography(obj, scene, CV_RANSAC);
But in some examples I've seen, you can add in a number after, like this:
Mat H = findHomography(obj, scene, CV_RANSAC, 5);
What does this do?
And is there a way of altering CV_RANSAC to perform better, or is it just called like this to do its job as best it can?
At the top of the page that you posted is the link to the API documentation of that function.
The fourth parameter is the allowed reprojection error in pixels, so that the reprojection still counts as a match.