So I'm trying to place an image into a scene using opencv. I can rotate it about its centre (z axis) using GetRotationMatrix2d and warpaffine. I'm wondering to I need to use a perspective transform? Or can I just create a 3d rotation matrix and plug it into warpaffine? And is there any getRotationMatrix equivalent for 3d?
Thanks!
Edit: I found this post but it does not work for me.
My code is below And I have gotten it to rotate, although I'm not sure what focal length I should be using? everytime I rotate about a different angle I need to play around with the focal length to get it right. Say if I rotate my image 45' it looks like the pic below but that's clearly warped off in some way...
void rotateImage(const Mat &input, Mat &output, double alpha, double beta, double gamma, double dx, double dy, double dz, double f)
{
alpha = (alpha - 90.)*CV_PI/180.;
beta = (beta - 90.)*CV_PI/180.;
gamma = (gamma - 90.)*CV_PI/180.;
// get width and height for ease of use in matrices
double w = (double)input.cols;
double h = (double)input.rows;
// Projection 2D -> 3D matrix
Mat A1 = (Mat_<double>(4,3) <<
1, 0, -w/2,
0, 1, -h/2,
0, 0, 0,
0, 0, 1);
// Rotation matrices around the X, Y, and Z axis
Mat RX = (Mat_<double>(4, 4) <<
1, 0, 0, 0,
0, cos(alpha), -sin(alpha), 0,
0, sin(alpha), cos(alpha), 0,
0, 0, 0, 1);
Mat RY = (Mat_<double>(4, 4) <<
cos(beta), 0, -sin(beta), 0,
0, 1, 0, 0,
sin(beta), 0, cos(beta), 0,
0, 0, 0, 1);
Mat RZ = (Mat_<double>(4, 4) <<
cos(gamma), -sin(gamma), 0, 0,
sin(gamma), cos(gamma), 0, 0,
0, 0, 1, 0,
0, 0, 0, 1);
// Composed rotation matrix with (RX, RY, RZ)
Mat R = RX * RY * RZ;
// Translation matrix
Mat T = (Mat_<double>(4, 4) <<
1, 0, 0, dx,
0, 1, 0, dy,
0, 0, 1, dz,
0, 0, 0, 1);
// 3D -> 2D matrix
Mat A2 = (Mat_<double>(3,4) <<
f, 0, w/2, 0,
0, f, h/2, 0,
0, 0, 1, 0);
// Final transformation matrix
Mat trans = A2 * (T * (R * A1));
// Apply matrix transformation
warpPerspective(input, output, trans, input.size(), INTER_LANCZOS4);
}
Does translation work? Try the following code instead.
Mat T = (Mat_<double>(4, 4) <<
1, 0, 0, 0,
0, 1, 0, 0,
0, 0, 1, 0,
dx, dy, dz, 1);
Related
So, the first problem I ran into was that OpenCV defines its origin about the top left corner rather than the center of the window. This was a problem because I want to just apply a transformation matrix to an image (say a reflection about the x-axis for example) and see it applied "in place", so it stays in the same spot, but is still reflected.
My solution to fix this was to first translate the image to OpenCV's origin, apply my transformation matrix to the image, then translate it back to its original location.
This works fine, however, any part of the image that goes off screen at any point is permanently deleted/cropped. I cannot figure out how to prevent this, I figured there was maybe a flag or something and I tried utilizing BORDER_WRAP rather than BORDER_CONSTANT which almost fixed my problem, but not quite and now I am completely stuck on where to go next.
Here is what I have so far:
int main()
{
// Read in and display input image
Mat src = imread("myImage.png");
imshow("Input Image", src);
// Translate the image to the origin
Mat M = (Mat_<double>(2, 3) << 1, 0, -(src.rows / 2), 0, 1, -(src.cols / 2));
Size dsize = Size(src.rows, src.cols);
warpAffine(src, src, M, dsize, INTER_LINEAR, BORDER_CONSTANT, Scalar());
// Apply the affine transformation
Mat M2 = (Mat_<double>(2, 3) << 1, 0, 0, 0, -1, 0);
dsize = Size(src.rows, src.cols);
warpAffine(src, src, M2, dsize, INTER_LINEAR, BORDER_CONSTANT, Scalar());
// Translate the resulting image back to its original location and display
Mat M3 = (Mat_<double>(2, 3) << 1, 0, (src.rows / 2), 0, 1, (src.cols / 2));
dsize = Size(src.rows, src.cols);
warpAffine(src, src, M3, dsize, INTER_LINEAR, BORDER_CONSTANT, Scalar());
// This is an extremely cropped version of the input image because
// it gets cropped when translated to the origin
imshow("Output Image", src);
waitKey();
return 0;
}
NEW CODE:
// Read in and display input image
Mat src = imread("umichLogo.png");
imshow("Input Image", src);
Mat M = (Mat_<double>(3, 3) << 1, 0, -(src.rows / 2), 0, 1, -(src.cols / 2), 0, 0, 1);
Mat M2 = (Mat_<double>(3, 3) << 1, 0, 0, 0, -1, 0, 0, 0, 1);
Mat M3 = (Mat_<double>(3, 3) << 1, 0, (src.rows / 2), 0, 1, (src.cols / 2), 0, 0, 1);
Mat Composition = M3 * (M2 * M);
Size dsize = Size(src.rows, src.cols);
warpPerspective(src, src, Composition, dsize, INTER_LINEAR, BORDER_CONSTANT, Scalar());
imshow("Output Image", src);
waitKey();
return 0;
To avoid undesired cropping, transform once only (simultaneously).
Mat M = (Mat_<double>(3, 3) << 1, 0, -(src.rows / 2), 0, 1, -(src.cols / 2), 0,0,1);
Mat M2 = (Mat_<double>(3, 3) << 1, 0, 0, 0, -1, 0, 0,0,1);
Mat M3 = (Mat_<double>(3, 3) << 1, 0, (src.rows / 2), 0, 1, (src.cols / 2), 0,0,1);
Mat Composition = M3 * (M2 * M);
Size dsize = Size(src.rows, src.cols);
warpPerspective(src, src, Composition, dsize, INTER_LINEAR, BORDER_CONSTANT, Scalar());
I found that, rows and cols are mistaken (not only for size but also M and M3).
Fixed code is :
Mat M = (Mat_<double>(3, 3) << 1, 0, -(src.cols / 2), 0, 1, -(src.rows / 2), 0,0,1);
Mat M2 = (Mat_<double>(3, 3) << 1, 0, 0, 0, -1, 0, 0,0,1);
Mat M3 = (Mat_<double>(3, 3) << 1, 0, (src.cols / 2), 0, 1, (src.rows / 2), 0,0,1);
Mat Comp = M3 * (M2 * M);
warpPerspective(src, src, Comp, src.size(), INTER_LINEAR, BORDER_CONSTANT, Scalar());
Currently I am working on a program that is able to draw a spline based on four points. As can be seen below, I currently have it so that you need to manually enter these points into the code. I am hoping to change that using mouse clicks. What I intend to do is have the user be able to click on the image, which draws a dot immediately. This will be able to be done as many times as the user wants, but it will draw a spline when every four points are drawn. I am running into two problems. Firstly when trying to implement a simple mouse click to just draw a circle in onMouse, it is not working. Secondly, I am wondering how I will be able to store the coordinates of each mouse click as they are happening, so I can create a loop that plugs these coordinates into the below matrix equations to draw a spline while the program is running. I have an idea of how I can store the first, but I am not sure how to handle multiple coordinates as well as how to have a definite variable name for each that I can put into the equations. Apologies for the long post, I have some big aspirations for this project despite having limited experience, so I would appreciate some guidance.
#include <iostream>
#include<opencv2/core/core.hpp> //Mat is defined there
#include<opencv2/imgproc/imgproc.hpp> //resize an image
#include<opencv2/highgui/highgui.hpp> //input or output: imread(), imshow()
using namespace std;
using namespace cv;
Mat img;
void onMouse(int event, int x, int y, int flags, void* param)
{
if (event == EVENT_LBUTTONDOWN)
{
printf("(%d, %d)\n", x, y);
int r = 1;
circle(img, Point(x, y), r, Scalar(0, 255, 100), 5);
}
}
int main(int argc, char** argv)
{
img.create(600, 800, CV_8UC3);
img = Scalar(255, 255, 255);
double a_values[3][2] = { { 2.0, 1 }, { 0.0, 3.0 }, { -2.0, -4.0 } };
//A.create(3, 2, CV_64FC1); //create a matrix 3x2 with double value
Mat A = Mat(3, 2, CV_64FC1, a_values); //Constructor: pass the values directly using a 2D array.
printf("matrix A:\n");
cout << A << endl;
Mat B;
B.create(2, 2, CV_64FC1); //2x2 matrix
B.ptr<double>(0)[0] = 1.0;
B.ptr<double>(0)[1] = 2.0;
B.ptr<double>(1)[0] = 0.0;
B.ptr<double>(1)[1] = -2.0;
printf("matrix B:\n");
cout << B << endl;
Mat C = A * B; //Matrix product
printf("matrix C:\n");
cout << C << endl;
Mat B_inv = B.inv();
printf("matrix B inverse:\n");
cout << B_inv << endl;
double m_values[4][4] = { { 0, 0, 0, 1 }, { 1, 1, 1, 1 }, { 0, 0, 1, 0 }, { 3, 2, 1, 0 } };
Mat M = Mat(4, 4, CV_64FC1, m_values);
Mat M_inv = M.inv();
printf("matrix M inverse:\n");
cout << M_inv << endl;
double h_values[4][4] = { { 2, -2, 1, 1 }, { -3, 3, -2, -1 }, { 0, 0, 1, 0 }, { 1, 0, 0, 0 } };
Mat Hermite = Mat(4, 4, CV_64FC1, h_values);
double point_values[4][2] = { { 200, 350 }, { 220, 400 }, { 600, 300 }, { 390, 300 } };
Mat Points = Mat(4, 2, CV_64FC1, point_values);
Mat Final = Hermite * Points;
printf("Final matrix:\n");
cout << Final << endl;
/* If there are two points P1(30, 50) and P2(80, 120), I want to draw a spline between
them and also make sure the speed of the spline at point P1 equals(500, 2) and at P2 equals(10, 1000). */
//Draw 1st point
circle(img, Point(200, 350), 1, Scalar(0, 0, 255), 3);
//Draw 2nd point
circle(img, Point(220, 400), 1, Scalar(0, 0, 255), 3);
circle(img, Point(400, 450), 1, Scalar(0, 0, 255), 3);
circle(img, Point(350, 500), 1, Scalar(0, 0, 255), 3);
//Draw the spline between 1st and 2nd points
//Use a loop on t [0, 1], for different t values, compute x(t), y(t); then use circle() to draw it
// x(t) = axt3 + bxt2 + cxt + dx
// y(t) = ayt3 + byt2 + cyt + dy
double ax = (int)(Final.at<double>(0, 0));
double ay = (int)(Final.at<double>(0, 1));
double bx = (int)(Final.at<double>(1, 0));
double by = (int)(Final.at<double>(1, 1));
double cx = (int)(Final.at<double>(2, 0));
double cy = (int)(Final.at<double>(2, 1));
double dx = (int)(Final.at<double>(3, 0));
double dy = (int)(Final.at<double>(3, 1));
printf("ax:\n");
cout << ax << endl;
printf("dx:\n");
cout << dx << endl;
for (double t = 0.0; t <= 1.0; t += 0.001)
{
int x = ax * t * t * t + bx * t * t + cx * t + dx;
int y = ay * t * t * t + by * t * t + cy * t + dy;
circle(img, Point(x, y), 1, Scalar(0, 0, 0), 1);
}
while (1)
{
imshow("Spline", img);
char c = waitKey(1);
if (c == 27)
break;
}
return 1;
}
You defined onMouse function, but you did not register it to any window. You need to create window with cv::namedWindow and then register your callback for mouse with cv::setMouseCallback. In your case, just add this before anything in your main function:
cv::namedWindow("Spline");
cv::setMouseCallback("Spline", onMouse);
Here is the simple program to draw circle/point to provided image. It will store point where you clicked and it will draw all points to the image.
#include <opencv2/opencv.hpp>
std::vector<cv::Point> points;
void onMouse(int action, int x, int y, int, void*) {
if (action == cv::EVENT_LBUTTONDOWN) {
points.push_back(cv::Point{x, y});
}
}
int main(int argc, char** argv) {
const auto mainWindow = "Main Window";
cv::namedWindow(mainWindow);
cv::setMouseCallback(mainWindow, onMouse);
cv::Mat image {600, 800, CV_8UC3, cv::Scalar{255, 255, 255}};
while (true) {
for (const auto& point : points) {
cv::circle(image, point, 5, cv::Scalar{0, 200, 0}, -1);
}
cv::imshow(mainWindow, image);
cv::waitKey(25);
}
cv::waitKey();
cv::destroyAllWindows();
return 0;
}
EDIT:
Be aware that this solution will re-render old points. You can improve this by drawing just new points.
When you click, add that point to temporary vector and when you draw all points from temporary vector, clear that vector.
For more improvement, you can check if vector is not empty and than call cv::imshow.
I wrote a kalman Filter implementation using the Eigen Library in C++ and also using the implementation at this link to test my filter: My prediction step looks like this:
void KalmanFilter::Predict()
{
// state Estimate = state transition matrix * previous state
// No control input present.
x = A * x;
// State Covariance Matrix = (State Transition Matrix * Previous State
Covariance matrix * (State Transition Matrix)^T ) + Process Noise
P = A * P * A.transpose() + Q;
}
while my update step is:
void KalmanFilter::Update(VectorXd z)
{
//Kalman Gain = (State Covariance Matrix * Measurement matrix.transpose) * (H*P*H^T + Measurement Noise)^-1
K = (P * H.transpose()) * (H * P * H.transpose()+ R).inverse();
//Estimated Stated = Estimated state + Kalman Gain (Measurement Innovation)
x = x + K*(z - H * x);
//State Covariance matrix = (Identity Matrix of the size of x.size * x.size) - K* H * P;
long x_size = x.size();
MatrixXd I = MatrixXd::Identity(x_size, x_size);
P = (I - K * H) * P ;
}
My initial values are:
pos_x = 0.0;
pos_y = 0.0;
pos_z = 1.0;
vel_x = 10.0;
vel_y = 0.0;
vel_z = 0.0;
acc_x = 0.0;
acc_y = 0.0;
acc_z = -9.81;
and I'm generating "fake data" by doing the following in a loop:
double c = 0.1; // Drag resistance coefficient
double damping = 0.9 ; // Damping
double sigma_position = 0.1 ; // position_noise
// Create simulated position data
for (int i = 0; i < N; i ++)
{
acc_x = -c * pow(vel_x, 2); // calculate acceleration ( Drag Resistance)
vel_x += acc_x * dt; // Integrate acceleration to give you velocity in the x axis.
pos_x += vel_x * dt; // Integrate velocity to return the position in the x axis
acc_z = -9.806 + c * pow(vel_z, 2); // Gravitation + Drag
vel_z += acc_z * dt; // z axis velocity
pos_z += vel_z * dt; // position in z axis
// generate y position here later.
if(pos_z < 0.01)
{
vel_z = -vel_z * damping;
pos_z += vel_z * dt;
}
if (vel_x < 0.1)
{
acc_x = 0.0;
acc_z = 0.0;
}
// add some noise
pos_x = pos_x + sigma_position * process_noise(generator);
pos_y = pos_y + sigma_position * process_noise(generator);
pos_z = pos_z + sigma_position * process_noise(generator);
I then run my prediction and update step by:
// Prediction Step
kalmanFilter.Predict();
// Correction Step
kalmanFilter.Update(z);
where z is a 3 x 1 vector containing pos_x, pos_y and pos_z
My State Transition Matrix A looks like this:
A << 1, 0, 0, dt, 0, 0, dt_squared, 0 , 0,
0, 1, 0, 0, dt, 0, 0, dt_squared, 0,
0, 0, 1, 0, 0, dt, 0, 0, dt_squared,
0, 0, 0, 1, 0, 0, dt, 0, 0,
0, 0, 0, 0, 1, 0, 0 , dt, 0,
0, 0, 0, 0, 0, 1, 0, 0, dt,
0, 0, 0, 0, 0, 0, 1, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1, 0,
0, 0, 0, 0, 0, 0, 0, 0, 1;
where dt_squared is (dt * dt) /2;
P is
P<< 100, 0, 0, 0, 0, 0, 0, 0, 0,
0, 100, 0, 0, 0, 0, 0, 0, 0,
0, 0, 100, 0, 0, 0, 0, 0, 0,
0, 0, 0, 100, 0, 0, 0, 0, 0,
0, 0, 0, 0, 100, 0, 0, 0, 0,
0, 0, 0, 0, 0, 100, 0, 0, 0,
0, 0, 0, 0, 0, 0, 100, 0, 0,
0, 0, 0, 0, 0, 0, 0, 100, 0,
0, 0, 0, 0, 0, 0, 0, 0, 100;
and
R << 1, 0, 0,
0, 1, 0,
0, 0, 1;
and
Q = G * G.transpose()* a * a;
where G is a 9 x 1 Matrix of
G << dt_squared, dt_squared, dt_squared, dt, dt, dt, 1, 1, 1;
a = 0.1 //( acceleration process noise)
My issue is my estimated position for y and z are off and diverge from the "real" positions. If you look at the following graphs,
This is what pos_x looks like:
This is what pos_y looks like:
And finally Z:
This is my first foray with Kalman filters and I'm not sure what I'm doing wrong here. My final goal would be to use this to estimate the position of a drone. Additionally, I have the following questions:
In a real life situation for a drone for example, how do you about choosing your Process Noise if you can't directly observe the process? Do you simply just pick arbitrary values?
My apologies for the long post. Any help is appreciated.
I am not sure if it's a code related issue, an algorithm implementation issue, or an expectation issue.
You do realize that a filter like this will NOT reproduce truth data, or even anything close to it, if the fake data has too much maneuvering in it.
Also, your graphs are not present.
I know my response doesn't follow community standards but I cannot comment or I'd do that.
Until you provide the plots and check the curvature of the path against the update rate I would not attempt to go into detail. Also filters need to be "tuned" to a specific system. You may need to play with noise parameters to tune it better. For maneuvering tracks one may need to go to higher order filters, Singer, or Jerk filters... The filter needs to model the system well enough. Based on your update matrix it looks like you have a parabolic (second order) estimate. You may also want to ask about this in other forums that are not s/w or code specific.
Every System has variances. Let's say the Filter has a variance of +-1% and the real value has +-5%; If you predict a value you have to make a choice for the update to use the predicted or the meassures value. Depending on which one you believe more.
Otherwise your filter does develop always based on it's own values...
If I apply a Sobel filter to an image in Python using scipy.ndimage.filters.convole I get meaningful results, for example, for this simple input image img
0 255 0
0 255 0
0 255 0
the convolution
dimage.filters.convolve(img, Kx)
with Kx
-1 0 1
-2 0 2
-1 0 1
returns a meaningful gradient in x-direction:
-1020 0 1020
-1020 0 1020
-1020 0 1020
I don't know how to get a equivalent result using openCV2 in C++ though. When I define the input image by
int image_data[9] = {0, 255, 0, 0, 255, 0, 0, 255, 0};
cv::Mat image = cv::Mat(3, 3, CV_32F, image_data);
and apply the kernel by
cv::Mat gradientx;
double sobelx_data[9] = {-1, 0, 1, -2, 0, 2, -1, 0, 1};
cv::Mat sobelx = cv::Mat(3, 3, CV_32F, sobelx_data);
cv::filter2D(image, gradientx, -1, sobelx);
I get the following result from
for(int row=0; row<gradientx.rows; row++) {
for(int col=0; col<gradientx.cols; col++) {
std::cout << gradientx.at<int>(row,col) << std::endl;
}
}
it returns the following image
478 -2147482660 478
478 -2147482660 478
478 -2147482660 478
There seems to be an overflow problem, but I don't know why. Trying to get values from gradientx.at<double>(row,col) produces
-1.68911e-311 8.10602e-312 8.11663e-312
-1.68911e-311 8.10602e-312 8.11663e-312
-1.68911e-311 2.122e-314 8.54412e-72
Can someone tell me why this is? Isn't filter2D supposed to do a 2D convolution on the image, and why do I get weird values when addressing the output pixels with <double>? Thank you.
Okay, here's your code with the types corrected (I've also added more parameters to filter2D):
float image_data[9] = {0, 255, 0, 0, 255, 0, 0, 255, 0};
cv::Mat image = cv::Mat(3, 3, CV_32F, image_data);
std::cout << "image = " << std::endl << image << std::endl;
cv::Mat gradientx;
float sobelx_data[9] = {-1, 0, 1, -2, 0, 2, -1, 0, 1};
cv::Mat sobelx = cv::Mat(3, 3, CV_32F, sobelx_data);
std::cout << "sobelx = " << std::endl << sobelx << std::endl;
cv::filter2D(image, gradientx, -1, sobelx, cv::Point(-1, -1), 0,
cv::BORDER_DEFAULT);
std::cout << "gradientx = " << std::endl << gradientx << std::endl;
The result is:
image =
[0, 255, 0;
0, 255, 0;
0, 255, 0]
sobelx =
[-1, 0, 1;
-2, 0, 2;
-1, 0, 1]
gradientx =
[0, 0, 0;
0, 0, 0;
0, 0, 0]
If you look at the top of the documentation page on filtering, you'll see all of the border types that OpenCV uses. By default, filter2D uses BORDER_REFLECT_101. This is probably not what we want, so let's change it to BORDER_REPLICATE.
cv::filter2D(image, gradientx, -1, sobelx, cv::Point(-1, -1), 0,
cv::BORDER_REPLICATE);
Result:
image =
[0, 255, 0;
0, 255, 0;
0, 255, 0]
sobelx =
[-1, 0, 1;
-2, 0, 2;
-1, 0, 1]
gradientx =
[1020, 0, -1020;
1020, 0, -1020;
1020, 0, -1020]
That's better, but the values are flipped. If you look at the bottom of the function description for filter2D you'll see that it actually computes the cross correlation rather than the convolution. So we need to flip the kernel to get the correct results.
cv::Mat sobelxflip;
cv::flip(sobelx, sobelxflip, -1);
cv::filter2D(image, gradientx, -1, sobelxflip, cv::Point(-1, -1), 0,
cv::BORDER_REPLICATE);
std::cout << "gradientx = " << std::endl << gradientx << std::endl;
Result:
gradientx =
[-1020, 0, 1020;
-1020, 0, 1020;
-1020, 0, 1020]
I'm trying to implement the core detection algorithm on fingerprint from the paper "Singular Point Detection in Fingerprint Image".
1) I have defined the direction field from part 2 (original and Theta, W=2x2)
2) The next step is to find the gradient of field orientation (over x and y respectively):
Mat Jx, Jy;
Sobel(theta, Jx, CV_32F, 1, 0, 3, 1, 0, BORDER_DEFAULT);
Sobel(theta, Jy, CV_32F, 0, 1, 3, 1, 0, BORDER_DEFAULT);
3) Apply next formula to image (disctete Green's theoreme):
I do so, but I think it's wrong:
Mat Jx_dy, Jy_dx;
Sobel(Jy, Jy_dx, CV_32F, 1, 0, 3, 1, 0, BORDER_DEFAULT);
Sobel(Jx, Jx_dy, CV_32F, 0, 1, 3, 1, 0, BORDER_DEFAULT);
Mat G, _G;
cv::subtract(Jy_dx, Jx_dy, _G);
integral(_G,G);
imshow("Jy_dx - Jx_dy", _G); //totally black image
How to apply this formula on image?
Thanks.