I've been trying to refine my camera parameters with CvLevMarq but after reading about it, it seems to be causing mixed results - which is exactly what I am experiencing. I read about the alternatives and came upon EIGEN - and also found this library that utilizes it.
However, the library above seems to use a stitching class that doesn't support OpenCV and will probably require me to port it to OpenCV.
Before going ahead and doing so, which will probably not be an easy task, I figured I'd ask around first and see if anyone else had the same problem?
I'm currently using:
1. Calculating features with FASTFeatureDetector
Ptr<FeatureDetector> detector = new FastFeatureDetector(5,true);
detector->detect(firstGreyImage, features_global[firstImageIndex].keypoints); // Previous picture
detector->detect(secondGreyImage, features_global[secondImageIndex].keypoints); // New picture
2. Extracting features with SIFTDescriptorExtractor
Ptr<SiftDescriptorExtractor> extractor = new SiftDescriptorExtractor();
extractor->compute(firstGreyImage, features_global[firstImageIndex].keypoints, features_global[firstImageIndex].descriptors); // Previous Picture
extractor->compute(secondGreyImage, features_global[secondImageIndex].keypoints, features_global[secondImageIndex].descriptors); // New Picture
3. Matching features with BestOf2NearestMatcher
vector<MatchesInfo> pairwise_matches;
BestOf2NearestMatcher matcher(try_use_gpu, 0.50f);
matcher(features_global, pairwise_matches);
matcher.collectGarbage();
4. CameraParams.R quaternion passed from a device (slightly inaccurate which causes the issue)
5. CameraParams.Focal == 389.0f -- Played around with this value, 389.0f is the only value that matches the images horizontally but not vertically.
6. Bundle Adjustment (cvLevMarq, calcError & calcJacobian)
Ptr<BPRefiner> adjuster = new BPRefiner();
adjuster->setConfThresh(0.80f);
adjuster->setMaxIterations(5);
(*adjuster)(features,pairwise_matches,cameras);
7. ExposureCompensator (GAIN)
8. OpenCV MultiBand Blender
What works so far:
SeamFinder - works to some extent but it depends on the result of the cvLevMarq algoritm. I.e. if the algoritm is off, seamFinder is going to be off too.
HomographyBasedEstimator works beautifully. However, since it "relies" on the features, it's unfortunately not the method that I'm looking for.
I wouldn't want to rely on the features since I already have the matrix, if there's a way to "refine" the current matrix instead - then that would be the targeted result.
Results so far:
cvLevMarq "Russian roulette" 6/10:
This is what I'm trying to achieve 10/10 times. But 4/10 times, it looks like the picture below this one.
By simply just re-running the algorithm, the results change. 4/10 times it looks like this (or worse):
cvLevMarq "Russian roulette" 4/10:
Desired Result:
I'd like to "refine" my camera parameters with the features that I've matched - in hope that the images would align perfectly. Instead of hoping that cvLevMarq will do the job for me (which it won't 4/10 times), is there another way to ensure that the images will be aligned?
Update:
I've tried these versions:
OpenCV 3.1: Using CVLevMarq with 3.1 is like playing Russian roulette. Some times it can align them perfectly, and other times it estimates focal as NAN which causes segfault in the MultiBand Blender (ROI = 0,0,1,1 because of NAN)
OpenCV 2.4.9/2.4.13: Using CvLevMarq with 2.4.9 or 2.4.13 is unfortunately the same thing minus the NAN issue. 6/10 times it can align the images perfectly, but the other 4 times it's completely off.
My Speculations / Thoughts:
Template Matching using OpenCV. Maybe if I template match the ends of the images (i.e. x = 0, y = 0,height = image.height, width = 50). Any thoughts about this?
I found this interesting paper about Levenberg Marquardt applied in Homography. That looks like something that could solve my problem since the paper uses corner detection and whatnot to detect the features in the images. Any thoughts about this?
Maybe the problem isn't in CvLevMarq but instead in BestOf2NearestMatcher? However, I've searched for days and I couldn't find another method that returns the pairwise matches to pass to BPRefiner.
Hough Line Transform Detecting the lines in the first/second image and use that to align the images. Any thoughts on this? -- One thing might be, what if the images doesn't have any lines? I.e. empty wall?
Maybe I'm overkilling something so simple.. Or maybe I'm not? Basically, I'm trying to align a set of images so I can warp them without overlapping each-other. Drop a comment if it doesn't make sense :)
Update Aug 12:
After trying all kinds of combinations, the absolute best so far is CvLevMarq. The only problem with it is the mixed results shown in the images above. If anyone has any input, I'd be forever grateful.
It seems your parameter initialization is the problem. I would use a linear estimator first, i.e. ignore your noisy sensor, and then use this as the initial values for the non-linear optimizer.
A quick method is to use getaffinetransform, as you have mostly rotation.
Maybe you want to take a look at this library: https://github.com/ethz-asl/kalibr.
Cheers
If you want to stitch the images, you should see stitching_detailed.cpp. It will probably solve your problem.
In addition, I have used Graph Cut Seam Finding method with Canny Edge Detection for better stitching results in this code. If you want to optimize this code, see here.
Also, if you are going to use it for personal use, SIFT is good. You should know, SIFT is patented and will cost you if you use it for commercial purposes. Use ORB instead.
Hope it helps!
I am looking to align two face images using the 68 landmarks learnt by the dlib detector. I know I can convert the images to OpenCV Mat and then use the warpAffine method. I am having certain memory leaks with it and decided to see if using only dlib is possible. I noticed that the dlib library has some methods in geometry.h for this purpose.
For this, I am using find_affine_transform method to get a point_transform_affine object where I find the affine transformation between 3 points obtained using the shape and template. In the documentation, it is written that we can use this object to apply the transformation on vector of points. However, I have been unable to find some example for it.
Could you tell me how I can apply the transformation learnt ?
Secondly, I have an image loaded in array2d object. Is there a way to go from array2d to vector of points ?
Some initial code is listed down below for the find_affine_transform.
std::vector<dlib::vector<double,2>> TemplateLandmarks;
std::vector<dlib::vector<double,2>> ObtainedLandmarks;
// push_back the specific coordinates in the above vectors
array2d<bgr_pixel> img;
// read the image from a file path using load_image
// learning the best transformation map
point_transform_affine H = find_affine_transform ( ObtainedLandmarks , TemplateLandmarks );
This can be done using the extract_image_chips function in dlib. There is even an example of its use in the face landmarking example program to align faces.
I am trying a different Pipeline without success until now.
The Idea is to use the classic pipeline (as in the Explorer Example) but additionally to use the last ColorImage for the texutre.
So the idea (after clicking SAVE MESH):
Save current Image as BMP
Get the current transformation [m_pVolume->GetCurrentWorldToCameraTransform(&m_worldToCameraTransform);] .. lets call it M
Transform all Mesh vertices v in the last Camera Space Coordinate System ( M * v )
Now the current m_pMapper refers to the latest Frame which we want to use [ m_pMapper->MapCameraPointToColorSpace(camPoint, &colorPoint); ]
In theory I should have now every Point of the fusion mesh as a texture coordinate.. I want to use them to export as OBJ File (with texture and not only color).
What am I doing wrong?
The 3D Transformations seem to be correct.. when I visualize the resulting OBJ file in MeshLab I can see that the transformation is correct.. the WorldCoordinateSystem is Equal to the latest recorded position.
Only the texture is not set correctly.
I would be very very very very happy if anyone could help me. I am trying already for a long time :/
Thank you very much :)
Sorry I'm a bit new with SDL and C++ development. Right now I've created a tile mapper that reads from my map.txt file. So far it works, but I want to add editing the map now.
SDL_Texture *texture;
texture= IMG_LoadTexture(G_Renderer,"assets/tile_1.png");
SDL_RenderCopy(G_Renderer, texture, NULL, &destination);
SDL_RenderPresent(G_Renderer);
The above is the basic way I'm showing my tiles, but if I want to go in and change the texture in real time it's kind of buggy and doesn't work well. Is there a method that is best for editing a texture? Thanks for the help I appreciate everything.
The most basic way is to set up a storage container with some textures which you will use repeatedly; for example a vector or dictionary/map. Using the map approach for example you could do something like:
// remember to #include <map>
map<string, SDL_Texture> myTextures;
// assign using array-like notation:
myTextures["texture1"] = IMG_LoadTexture(G_Renderer,"assets/tile_1.png");
myTextures["texture2"] = IMG_LoadTexture(G_Renderer,"assets/tile_2.png");
myTextures["texture3"] = IMG_LoadTexture(G_Renderer,"assets/tile_3.png");
myTextures["texture4"] = IMG_LoadTexture(G_Renderer,"assets/tile_4.png");
then to utilise a different texture, all you have to do is use something along the lines of:
SDL_RenderCopy(G_Renderer, myTextures["texture1"], NULL, &destination);
SDL_RenderPresent(G_Renderer);
which can be further controlled by changing the first line to
SDL_RenderCopy(G_Renderer, myTextures[textureName], NULL, &destination);
where textureName is a string variable which you can alter in code in realtime.
This approach means you can load all the textures you will need before-hand and simply utilise them as needed later, meaning there's no loading from file system whilst rendering:)
There is a nice explanation of map here.
Hopefully this gives you a nudge in the right direction. Let me know if you need more info:)
I need help related to following matlab code
[labelMap_1,num] = bwlabel(labelMap == 1);
labelMap1Stat = imfeature(labelMap_1,'Area','Centroid');
Inside opencv i found few threads that i must use bloblib for it.
But suppose if i dont want to use it for the sake of code because i need to port this code into android and i am concern about the size. How can i achieve the same thing without using blob library overhead.
If there is no solution then what are the methods inside bloblib that will produce the same result as these two functions??
Thanks in advance.
Try using functions related to contours like cvFindContours() .
This article provides some insights on how to use opencv for blobs.
You can calculate centroid information my using cvMoments() function.
Then the center of mass is given by yc = M01 / M00, where M01 and M00 are fields in the structure returned by the Moments call.
Use cvContourArea() to find area.