My objective is to rotate the map around a point at the center of the bottom of the screen. To do this, I need to know the map coordinates of that (screen) point, then do a double transformation: first - move the center and then rotate around the new center , but that all requires the knowledge of the map coordinates of the rotation point.
I have the details of the map: center, orientation, zoom level, but I can't find any reference mapping a screen position to map coordinates.
Can someone point me in the right direction please?
Answered...but bet warned, it's not simple.
I had to dig into the arcane world of Google Maps tiling, and with a bit of help from mapTiler and some re-writing of the globalMercatorClass from php to java This is how I approached it:
Convert the center of the screen (via meters) to pixels in the current zoom
Using trig, find the pixel position of the bottom, center of the screen from the screen size and the map orientation from the screen center.
Convert that back to Lat/Lon and use it as the camera position
It uses an extraordinary number of CPU cycles, so I have had to slow the refresh rate down to 1.5 seconds, but, hey, it's working.
It would be nice to have some of this functionality available from the API, it would probably reduce the horsepower (and brainpower) required for the job. But I'm delighted that it's possible.
Related
I want to get the value of the center of the marker. as a picture below.
I have tried to calculate the center of the marker by calculating from the corner of the marker. So, I already got the center of the marker but it was not accurate values as I want. When the marker is not parallel to a camera, the calculated center has shifted from the center of the marker as a picture below.
I would appreciate if you could give me some suggestions to find the center of the marker in an accuracy way.
I would like to apologize if my question is not clear enough.
Please let me know what I have to improve.
Since your image is at an angle, you can use a perspective transform to obtain a bird's eye view of the image. To obtain the center of the marker, you can first find the corners using the Shi-Tomasi Corner Detector or the Harris corner detector and then use geometry calculations based on the found corners to obtain your center point.
For my undergraduate paper I am working on a iPhone Application using openCV to detect domino tiles. The detection works well in close areas, but when the camera is angled the tiles far away are difficult to detect.
My approach to solve this I would want to do some spacial calculations. For this I would need to convert a 2D Pixel value into world coordinates, calculate a new 3D position with a vector and convert these coordinates back to 2D and then check the colour/shape at that position.
Additionally I would need to know the 3D positions for Augmented Reality additions.
The Camera Matrix i got trough this link create opencv camera matrix for iPhone 5 solvepnp
The Rotationmatrix of the Camera I get from the Core Motion.
Using Aruco markers would be my last resort, as I woulnd't get the decided effect that I would need for the paper.
Now my question is, can i not make calculations when I know the locations and distances of the circles on a lets say Tile with a 5 on it?
I wouldn't need to have a measurement in mm/inches, I can live with vectors without measurements.
The camera needs to be able to be rotated freely.
I tried to invert the calculation sm'=A[R|t]M' to be able to calculate the 2D coordinates in 3D. But I am stuck with inverting the [R|t] even on paper, and I don't know either how I'd do that in swift or c++.
I have read so many different posts on forums, in books etc. and I am completely stuck and appreciate any help/input you can give me. Otherwise I'm screwed.
Thank you so much for your help.
Update:
By using the solvePnP that was suggested by Micka I was able to get the Rotation and Translation Vectors for the angle of the camera.
Meaning that if you are able to identify multiple 2D Points in your image and know their respective 3D World coordinates (in mm, cm, inch, ...), then you can get the mechanisms to project points from known 3D World coordinates onto the respective 2D coordinates in your image. (use the opencv projectPoints function).
What is up next for me to solve is the translation from 2D into 3D coordinates, where I need to follow ozlsn's approach with the inverse of the received matrices out of solvePnP.
Update 2:
With a top down view I am getting along quite well to being able to detect the tiles and their position in the 3D world:
tile from top Down
However if I am now angling the view, my calculations are not working anymore. For example I check the bottom Edge of a 9-dot group and the center of the black division bar for 90° angles. If Corner1 -> Middle Edge -> Bar Center and Corner2 -> Middle Edge -> Bar Center are both 90° angles, than the bar in the middle is found and the position of the tile can be found.
When the view is Angled, then these angles will be shifted due to the perspective to lets say 130° and 50°. (I'll provide an image later).
The Idea I had now is to make a solvePNP of 4 Points (Bottom Edge plus Middle), claculate solvePNP and then rotate the needed dots and the center bar from 2d position to 3d position (height should be irrelevant?). Then i could check with the translated points if the angles are 90° and do also other needed distance calculations.
Here is an image of what I am trying to accomplish:
Markings for Problem
I first find the 9 dots and arrange them. For each Edge I try to find the black bar. As said above, seen from Top, the angle blue corner, green middle edge to yellow bar center is 90°.
However, as the camera is angled, the angle is not 90° anymore. I also cannot check if both angles are 180° together, that would give me false positives.
So I wanted to do the following steps:
Detect Center
Detect Edges (3 dots)
SolvePnP with those 4 points
rotate the edge and the center points (coordinates) to 3D positions
Measure the angles (check if both 90°)
Now I wonder how I can transform the 2D Coordinates of those points to 3D. I don't care about the distance, as I am just calculating those with reference to others (like 1.4 times distance Middle-Edge) etc., if I could measure the distance in mm, that would even be better though. Would give me better results.
With solvePnP I get the rvec which I could change into the rotation Matrix (with Rodrigues() I believe). To measure the angles, my understanding is that I don't need to apply the translation (tvec) from solvePnP.
This leads to my last question, when using the iPhone, can't I use the angles from the motion detection to build the rotation matrix beforehand and only use this to rotate the tile to show it from the top? I feel that this would save me a lot of CPU Time, when I don't have to solvePnP for each tile (there can be up to about 100 tile).
Find Homography
vector<Point2f> tileDots;
tileDots.push_back(corner1);
tileDots.push_back(edgeMiddle);
tileDots.push_back(corner2);
tileDots.push_back(middle.Dot->ellipse.center);
vector<Point2f> realLivePos;
realLivePos.push_back(Point2f(5.5,19.44));
realLivePos.push_back(Point2f(12.53,19.44));
realLivePos.push_back(Point2f(19.56,19.44));
realLivePos.push_back(Point2f(12.53,12.19));
Mat M = findHomography(tileDots, realLivePos, CV_RANSAC);
cout << "M = "<< endl << " " << M << endl << endl;
vector<Point2f> barPerspective;
barPerspective.push_back(corner1);
barPerspective.push_back(edgeMiddle);
barPerspective.push_back(corner2);
barPerspective.push_back(middle.Dot->ellipse.center);
barPerspective.push_back(possibleBar.center);
vector<Point2f> barTransformed;
if (countNonZero(M) < 1)
{
cout << "No Homography found" << endl;
} else {
perspectiveTransform(barPerspective, barTransformed, M);
}
This however gives me wrong values, and I don't know anymore where to look (Sehe den Wald vor lauter Bäumen nicht mehr).
Image Coordinates https://i.stack.imgur.com/c67EH.png
World Coordinates https://i.stack.imgur.com/Im6M8.png
Points to Transform https://i.stack.imgur.com/hHjBM.png
Transformed Points https://i.stack.imgur.com/P6lLS.png
You see I am even too stupid to post 4 images here??!!?
The 4th index item should be at x 2007 y 717.
I don't know what I am doing wrongly here.
Update 3:
I found the following post Computing x,y coordinate (3D) from image point which is doing exactly what I need. I don't know maybe there is a faster way to do it, but I am not able to find it otherwise. At the moment I can do the checks, but still need to do tests if the algorithm is now robust enough.
Result with SolvePnP to find bar Center
The matrix [R|t] is not square, so by-definition, you cannot invert it. However, this matrix lives in the projective space, which is nothing but an extension of R^n (Euclidean space) with a '1' added as the (n+1)st element. For compatibility issues, the matrices that multiplies with vectors of the projective space are appended by a '1' at their lower-right corner. That is : R becomes
[R|0]
[0|1]
In your case [R|t] becomes
[R|t]
[0|1]
and you can take its inverse which reads as
[R'|-Rt]
[0 | 1 ]
where ' is a transpose. The portion that you need is the top row.
Since the phone translates in the 3D space, you need the distance of the pixel in consideration. This means that the answer to your question about whether you need distances in mm/inches is a yes. The answer changes only if you can assume that the ratio of camera translation to the depth is very small and this is called weak perspective camera. The question that you're trying to tackle is not an easy one. There is still people researching on this at PhD degree.
I've created an openCV application for human detection on images.
I run my algorithm on the same image over different scales, and when detections are made, at the end I have information about the bounding box position and at which scale it was taken from. Then I want to transform that rectangle to the original scale, given that position and size will vary.
I've wrapped my head around this and I've gotten nowhere. This should be rather simple, but at the moment I am clueless.
Help anyone?
Ok, got the answer elsewhere
"What you should do is store the scale where you are at for each detection. Then transforming should be rather easy right. Imagine you have the following.
X and Y coordinates (center of bounding box) at scale 1/2 of the original. This means that you should multiply with the inverse of the scale to get the location in the original, which would be 2X, 2Y (again for the bounxing box center).
So first transform the center of the bounding box, than calculate the width and height of your bounding box in the original, again by multiplying with the inverse. Then from the center, your box will be +-width_double/2 and +-height_double/2."
I am working on a simple mesh viewer implementation in C++ with basic functionality such as translation, rotation, scaling.
I'm stuck with with implementing the rotation of the object along z-axis using the mouse. What I want to implement is the following:
Click and drag the mouse vertically (almost vertical will do, as I use a simple threshold to filter slight deviations along the horizontal axis) to rotate the object along y-axis (this part is done).
Click and drag the mouse horizontally just as described above to rotate the object along x-axis (this part is done too).
For z-axis rotation, I want to detect a circular (or along an arc) mouse movement. I'm stuck with this part, and don't know how to implement this.
For the above two, i just use atan2() to determine the angle of movement. But how do I detect circular movements?
The only way to deal with this is to have a delay between the user starting to make the motion and the object rotating:
When user clicks and begins to move the mouse you need to determine if its going to become a straight line movement, or a circular one. This will require a certain amount of data to be collected before that judgement can be made.
The most extreme case would be requiring the user to make one complete circle first, then the rotation begins (in reality you could do much better than this). Just how small you are able to cut this period down to will depend on a) how precise you dictate your users actions must be, and b) how good you are with pattern recognition algorithms.
To get you started heres an outline of an extremely poor algorithm:
On user click store the x and y coordinates.
Every 1/10 of a second store the new coordinates and process_for_pattern.
in process_for_pattern you're looking for:
A period where the x coordinates and the y coordinates regularly both increase, both decrease, or one increases and one decreases. Over time if this pattern changes such that either the x or the y begins to reverse whilst the other continues as it was, then at that moment you can be fairly sure you've got a circle.
This algorithm would require the user to draw a quarter circle before it was detected, and it does not account for size, direction, or largely irregular movements.
If you really want to continue with this method you can get a much better algorithm, but you might want to reconsider your control method.
Perhaps, you should define a screen region (e.g. at window boundaries), which, when was clicked, will initiate arc movement - or use some other modifier, a button or whatever.
Then at a mouse click you capture the coordinates and center of rotation (mesh axis) in 2D screen space. This gets you a vector (mesh center, button down pos)
On every mouse move you calculate a new vector (mesh center, mouse pos) and the angle between the two vectors is the angle of rotation.
I don't think it works like that...
You could convert mouse wheel rotation to z-axis, or use quaternion camera orientation, which is able to rotate along every axis almost intuitively...
The opposite is true for quarternion camera: if one tries to rotate the mesh along a straight line, the mesh appears to rotate slightly around some other weird axis -- and to compensate that, one intuitively tries to follow some slightly curved trajectory.
It's not exactly what you want, but should come close enough.
Choose a circular region within which your movements numbered 1 and 2 work as described (in the picture this would be some region that is smaller than the red circle. However, when the user clicks outside the circular region, you save the initial click position (shown in green). This defines a point which has a certain angle relative to the x-axis of your screen (you can find this easily with some trig), and it also defines the radius of the circle on which the user is working (in red). The release of the mouse adds a second point (blue). You then find the angle this point has relative to the center of the screen and the x-axis (just like before). You then project that angle onto your circle with the radius determined by the first click. The dark red arc defines the amount of rotation of the model.
This should be enough to get you started.
That will not be a good input method, I think. Because you will always need some travel distance to discriminate between a line and a curve, which means some input delay. Here is an alternative:
Only vertical mouse having their line crossing the center of the screen are considered vertical. Same for horizontal. In other cases it's considered a rotation, and to calculate its amplitude, calculate the angle between the last mouse location and the current location relatively to the center of the screen.
Alternatively you could use the center of the selected mesh if your application works like that.
You can't detect the "circular, along an arc" mouse movement with anywhere near the precision needed for 3d model viewing. What you want is something like this: http://thetechartist.com/?p=80
You nominate an axis (x, y, or z) using either keyboard shortcuts or on-screen axis indicators that you can grab with the mouse.
This will be much more precise than trying to detect an "arc" gesture. Any "arc" recognition would necessarily involve a delay while you accumulate enough mouse samples to decide whether an arc gesture has begun or not. Gesture recognition like this is non-trivial (I've done some gesture work with the Wii-mote). Similarly, even your simple "vertical" and "horizontal" mouse movement detection will require a delay for the same reason. Any "simple threshold to filter slight deviations" will make it feel dampened and weird.
For 3d viewing you want 1:1 mouse responsiveness, and that means just explicitly nominating an axis with a shortcut key or UI etc. For x-axis rotation, just restrict it to mouse x, y-axis to mouse y if you like. For z you could similarly restrict to x or y mouse input, or just take the total 2d mouse distance travelled. It depends what feels nicest to you.
As an alternative, you could try coding up support for a 3D mouse like the 3dConnexion SpaceExplorer.
I have a webcam pointed at a table at a slant and with it I track markers.
I have a transformationMatrix in OpenSceneGraph and its translation part contains the relative coordinates from the tracked Object to the Camera.
Because the Camera is pointed at a slant, when I move the marker across the table the Y and Z axis is updated, although all I want to be updated is the Z axis, because the height of the marker doesnt change only its distance to the camera.
This has the effect when when project a model on the marker in OpenSceneGraph, the model is slightly off and when I move the marker arround the Y and Z values are updated incorrectly.
So my guess is I need a Transformation Matrix with which I multiply each point so that I have a new coordinate System which lies orthogonal on the table surface.
Something like this: A * v1 = v2 v1 being the camera Coordinates and v2 being my "table Coordinates"
So what I did now was chose 4 points to "calibrate" my system. So I placed the marker at the top left corner of the Screen and defined v1 as the current camera coordinates and v2 as (0,0,0) and I did that for 4 different points.
And then taking the linear equations I get from having an unknown Matrix and two known vectors I solved the matrix.
I thought the values I would get for the matrix would be the values I needed to multiply the camera Coordinates with so the model would updated correctly on the marker.
But when I multiply the known Camera Coordinates I gathered before with the matrix I didnt get anything close to what my "table coordinates" were suposed to be.
Is my aproach completely wrong, did I just mess something up in the equations? (solved with the help of wolframalpha.com) Is there an easier or better way of doing this?
Any help would be greatly appreciated, as I am kind of lost and under some time pressure :-/
Thanks,
David
when I move the marker across the table the Y and Z axis is updated, although all I want to be updated is the Z axis, because the height of the marker doesnt change only its distance to the camera.
Only true when your camera's view direction is aligned with your Y axis (or Z axis). If the camera is not aligned with Y, it means the transform will apply a rotation around the X axis, hence modifying both the Y and Z coordinates of the marker.
So my guess is I need a Transformation Matrix with which I multiply each point so that I have a new coordinate System which lies orthogonal on the table surface.
Yes it is. After that, you will have 2 transforms:
T_table to express marker's coordinates in the table referential,
T_camera to express table coordinates in the camera referential.
Finding T_camera from a single 2d image is hard because there's no depth information.
This is known as the Pose problem -- it has been studied by -among others- Daniel DeMenthon. He developed a fast and robust algorithm to find the pose of an object:
articles available on its research homepage, section 4 "Model Based Object Pose" (and particularly "Model-Based Object Pose in 25 Lines of Code", 1995);
code at the same place, section "POSIT (C and Matlab)".
Note that the OpenCv library offers an implementation of the DeMenthon's algorithm. This library also offers a convenient and easy-to-use interface to grab images from a webcam. It's worth a try: OpenCv homepage
If you know the location in the physical world of your four markers and you've recorded the positions as they appear on the camera, you ought to be able to derive some sort of transform.
When you do the calibration, surely you'd want to put the marker at the four corners of the table not the screen? If you're just doing the corners of the screen, I imagine you're probably not taking into acconut the slant of the table.
Is the table literally just slanted relative to the camera or is it also rotated at all?