Understanding OpenCV's undistort function - c++

I'm looking to undistort an image using the distortion coefficients that I've computed for my camera, without changing the camera matrix. This is exactly what undistort() does, but I wanted to draw the output to a larger canvas image.
When I tried this:
Mat drawtransform = getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, size, 1.0, size * 2);
undistort(inputimage, undistorted, cameraMatrix, distCoeffs, drawtransform);
It still wrote out the same sized image, but only the top left quarter of the scaled-up-by-two undistorted result. Like the documentation says, undistort writes into a target image of the same size.
It's pretty obvious that I can just go copy out and reimplement a slightly tweaked version of undistort() but I am having some trouble understanding what it is doing. Here's the source:
void cv::undistort( InputArray _src, OutputArray _dst, InputArray _cameraMatrix,
InputArray _distCoeffs, InputArray _newCameraMatrix )
{
Mat src = _src.getMat(), cameraMatrix = _cameraMatrix.getMat();
Mat distCoeffs = _distCoeffs.getMat(), newCameraMatrix = _newCameraMatrix.getMat();
_dst.create( src.size(), src.type() );
Mat dst = _dst.getMat();
CV_Assert( dst.data != src.data );
int stripe_size0 = std::min(std::max(1, (1 << 12) / std::max(src.cols, 1)), src.rows);
Mat map1(stripe_size0, src.cols, CV_16SC2), map2(stripe_size0, src.cols, CV_16UC1);
Mat_<double> A, Ar, I = Mat_<double>::eye(3,3);
cameraMatrix.convertTo(A, CV_64F);
if( distCoeffs.data )
distCoeffs = Mat_<double>(distCoeffs);
else
{
distCoeffs.create(5, 1, CV_64F);
distCoeffs = 0.;
}
if( newCameraMatrix.data )
newCameraMatrix.convertTo(Ar, CV_64F);
else
A.copyTo(Ar);
double v0 = Ar(1, 2);
for( int y = 0; y < src.rows; y += stripe_size0 )
{
int stripe_size = std::min( stripe_size0, src.rows - y );
Ar(1, 2) = v0 - y;
Mat map1_part = map1.rowRange(0, stripe_size),
map2_part = map2.rowRange(0, stripe_size),
dst_part = dst.rowRange(y, y + stripe_size);
initUndistortRectifyMap( A, distCoeffs, I, Ar, Size(src.cols, stripe_size),
map1_part.type(), map1_part, map2_part );
remap( src, dst_part, map1_part, map2_part, INTER_LINEAR, BORDER_CONSTANT );
}
}
About half of the lines here are for sanity checking and initializing input parameters. What I'm confused about is what's going on with map1 and map2. These names are sadly less descriptive than most. I must be missing some explanation, maybe it's tucked away in some introduction page, or under the doc for another function.
map1 is a two channel signed short integer matrix and map2 is an unsigned short integer matrix, both are of dimension (height, max(4096/width, 1)). The question is, why? What will these maps contain? What is the significance and purpose of this striping? What is the significance and purpose of the strange dimension of the stripes?

Use initUndistortRectifyMap to obtain the transformation to the scale you desire , then apply its output (the two matrices you mention) to remap .
The first map is used to compute the transform the x coordinate at each pixel position, the second is used to transform the y coordinate.

You might want to read the description for the function remap. The map represents the pixel X,Y location in the source image for every pixel in the destination image. Map1_part is every X location in the source, and Map2_part is every Y location in the source.
Without reading into it much, the striping could be a method of speeding up the transformation process.
EDIT:
Also, if you are looking to just scale your image to a larger dimension you could just re-size the output image.
double scaleX = 2.0;
double scaleY = 2.0;
cv::Mat undistortedScaled;
cv::resize(undistorted, undistortedScaled, cv::Size(0,0), scaleX, scaleY);

Related

Farneback optical flow - dealing with pixels out of view, pixels with wrong flow result, different size image

I am writing my thesis and one part of the task is to interpolate between images to create intermediate images. The work has to be done in c++ using openCV 2.4.13.
The best solution I've found so far is computing optical flow and remapping. But this solution has two problems that I am unable to solve on my own:
There are pixels that should go out of view (bottom of image for example), but they do not.
Some pixels do not move, creating a distorted result (upper right part of the couch)
What has made the flow&remap approach better:
Equalizing the intensity. This i'm allowed to do. You can check the result by comparing the couch form (centre of remapped image and original).
Reducing size of image. This i'm NOT allowed to do, as I need the same size output. Is there a way to rescale the optical flow result to get the bigger remapped image?
Other approaches tried and failed:
cuda::interpolateFrames. Creates incredible ghosting.
blending images with cv::addWeighted. Even worse ghosting.
Below is the code I am using at the moment. And images: dropbox link with input and result images
int main(){
cv::Mat second, second_gray, cutout, cutout_gray, flow_n;
second = cv::imread( "/home/zuze/Desktop/forstack/second_L.jpg", 1 );
cutout = cv::imread("/home/zuze/Desktop/forstack/cutout_L.png", 1);
cvtColor(second, second_gray, CV_BGR2GRAY);
cvtColor(cutout, cutout_gray, CV_RGB2GRAY );
///----------COMPUTE OPTICAL FLOW AND REMAP -----------///
cv::calcOpticalFlowFarneback( second_gray, cutout_gray, flow_n, 0.5, 3, 15, 3, 5, 1.2, 0 );
cv::Mat remap_n; //looks like it's drunk.
createNewFrame(remap_n, flow_n, 1, second, cutout );
cv::Mat cflow_n;
cflow_n = cutout_gray;
cvtColor(cflow_n, cflow_n, CV_GRAY2BGR);
drawOptFlowMap(flow_n, cflow_n, 10, CV_RGB(0,255,0));
///--------EQUALIZE INTENSITY, COMPUTE OPTICAL FLOW AND REMAP ----///
cv::Mat cutout_eq, second_eq;
cutout_eq= equalizeIntensity(cutout);
second_eq= equalizeIntensity(second);
cv::Mat flow_eq, cutout_eq_gray, second_eq_gray, cflow_eq;
cvtColor( cutout_eq, cutout_eq_gray, CV_RGB2GRAY );
cvtColor( second_eq, second_eq_gray, CV_RGB2GRAY );
cv::calcOpticalFlowFarneback( second_eq_gray, cutout_eq_gray, flow_eq, 0.5, 3, 15, 3, 5, 1.2, 0 );
cv::Mat remap_eq;
createNewFrame(remap_eq, flow_eq, 1, second, cutout_eq );
cflow_eq = cutout_eq_gray;
cvtColor(cflow_eq, cflow_eq, CV_GRAY2BGR);
drawOptFlowMap(flow_eq, cflow_eq, 10, CV_RGB(0,255,0));
cv::imshow("remap_n", remap_n);
cv::imshow("remap_eq", remap_eq);
cv::imshow("cflow_eq", cflow_eq);
cv::imshow("cflow_n", cflow_n);
cv::imshow("sec_eq", second_eq);
cv::imshow("cutout_eq", cutout_eq);
cv::imshow("cutout", cutout);
cv::imshow("second", second);
cv::waitKey();
return 0;
}
Function for remapping, to be used for intermediate image creation:
void createNewFrame(cv::Mat & frame, const cv::Mat & flow, float shift, cv::Mat & prev, cv::Mat &next){
cv::Mat mapX(flow.size(), CV_32FC1);
cv::Mat mapY(flow.size(), CV_32FC1);
cv::Mat newFrame;
for (int y = 0; y < mapX.rows; y++){
for (int x = 0; x < mapX.cols; x++){
cv::Point2f f = flow.at<cv::Point2f>(y, x);
mapX.at<float>(y, x) = x + f.x*shift;
mapY.at<float>(y, x) = y + f.y*shift;
}
}
remap(next, newFrame, mapX, mapY, cv::INTER_LANCZOS4);
frame = newFrame;
cv::waitKey();
}
Function to display optical flow in vector form:
void drawOptFlowMap (const cv::Mat& flow, cv::Mat& cflowmap, int step, const cv::Scalar& color) {
cv::Point2f sum; //zz
std::vector<float> all_angles;
int count=0; //zz
float angle, sum_angle=0; //zz
for(int y = 0; y < cflowmap.rows; y += step)
for(int x = 0; x < cflowmap.cols; x += step)
{
const cv::Point2f& fxy = flow.at< cv::Point2f>(y, x);
if((fxy.x != fxy.x)||(fxy.y != fxy.y)){ //zz, for SimpleFlow
//std::cout<<"meh"; //do nothing
}
else{
line(cflowmap, cv::Point(x,y), cv::Point(cvRound(x+fxy.x), cvRound(y+fxy.y)),color);
circle(cflowmap, cv::Point(cvRound(x+fxy.x), cvRound(y+fxy.y)), 1, color, -1);
sum +=fxy;//zz
angle = atan2(fxy.y,fxy.x);
sum_angle +=angle;
all_angles.push_back(angle*180/M_PI);
count++; //zz
}
}
}
Function to equalize intensity of images, for better results:
cv::Mat equalizeIntensity(const cv::Mat& inputImage){
if(inputImage.channels() >= 3){
cv::Mat ycrcb;
cvtColor(inputImage,ycrcb,CV_BGR2YCrCb);
std::vector<cv::Mat> channels;
cv::split(ycrcb,channels);
cv::equalizeHist(channels[0], channels[0]);
cv::Mat result;
cv::merge(channels,ycrcb);
cvtColor(ycrcb,result,CV_YCrCb2BGR);
return result;
}
return cv::Mat();
}
So to recap, my questions:
Is it possible to resize Farneback optical flow to apply to 2xbigger image?
How to deal with pixels that go out of view like at the bottom of my images (the brown wooden part should disappear).
How to deal with distortion that is created because optical flow wasn't computed for those pixels, while many pixels around there have motion? (couch upper right, & lion figurine has a ghost hand in the remapped image).
With OpenCV's Farneback optical flow, you will only get a rough estimation of pixel displacement, hence the distortions that appear on the result images.
I don't think optical flow is the way to go for what you are trying to achieve IMHO. Instead I'd recommend you to have a look at Image / Pixel Registration for instace here : http://docs.opencv.org/trunk/db/d61/group__reg.html
Image / Pixel Registration is the science of matching pixels of two images. Active research is ongoing about this complex non-trivial subject that is not yet accurately resolved.

OpenCV Applying Affine transform to single points rather than entire image

I've got a Affine transform matrix in OpenCV from the KeypointBasedMotionEstimator class.
It comes in a form like:
[1.0008478, -0.0017408683, -10.667297;
0.0011812132, 1.0009096, -3.3626099;
0, 0, 1]
I would now like to apply the transform to a vector< Pointf >, so that it will transform each point as it would be if they were in the image.
The OpenCV does not seem to allow transforming points only, the function:
void cv::warpAffine ( InputArray src,
OutputArray dst,
InputArray M,
Size dsize,
int flags = INTER_LINEAR,
int borderMode = BORDER_CONSTANT,
const Scalar & borderValue = Scalar()
)
Only seems to take images as inputs and outputs.
Is there a way I can apply an affine transform to single points in OpenCV?
you can use
void cv::perspectiveTransform(InputArray src, OutputArray dst, InputArray m)
e.g.
cv::Mat yourAffineMatrix(3,3,CV_64FC1);
[...] // fill your transformation matrix
std::vector<cv::Point2f> yourPoints;
yourPoints.push_back(cv::Point2f(4,4));
yourPoints.push_back(cv::Point2f(0,0));
std::vector<cv::Point2f> transformedPoints;
cv::perspectiveTransform(yourPoints, transformedPoints, yourAffineMatrix);
not sure about Point datatype, but the transformation must have double type, e.g. CV_64FC1
see http://docs.opencv.org/modules/core/doc/operations_on_arrays.html#perspectivetransform too
it's a bit clumsy, but you can matrix-multiply your points manually:
// the transformation matrix
Mat_<float> M(3,3);
M << 1.0008478, -0.0017408683, -10.667297,
0.0011812132, 1.0009096, -3.3626099,
0, 0, 1;
// a point
Point2f p(4,4);
// make a Mat for multiplication,
// must have same type as transformation mat !
Mat_<float> pm(3,1);
pm << p.x,p.y,1.0;
// now , just multiply:
Mat_<float> pr = M * pm;
// retrieve point:
Point2f pt(pr(0), pr(1));
cerr << pt << endl;
[-6.67087, 0.645753]

OpenCV : homomorphic filter

i want to use a homomorphic filter to work on underwater image. I tried to code it with the codes found on the internet but i have always a black image... I tried to normalized my result but didn't work.
Here my functions :
void HomomorphicFilter::butterworth_homomorphic_filter(Mat &dft_Filter, int D, int n, float high_h_v_TB, float low_h_v_TB)
{
Mat single(dft_Filter.rows, dft_Filter.cols, CV_32F);
Point centre = Point(dft_Filter.rows/2, dft_Filter.cols/2);
double radius;
float upper = (high_h_v_TB * 0.01);
float lower = (low_h_v_TB * 0.01);
//create essentially create a butterworth highpass filter
//with additional scaling and offset
for(int i = 0; i < dft_Filter.rows; i++)
{
for(int j = 0; j < dft_Filter.cols; j++)
{
radius = (double) sqrt(pow((i - centre.x), 2.0) + pow((double) (j - centre.y), 2.0));
single.at<float>(i,j) =((upper - lower) * (1/(1 + pow((double) (D/radius), (double) (2*n))))) + lower;
}
}
//normalize(single, single, 0, 1, CV_MINMAX);
//Apply filter
mulSpectrums( dft_Filter, single, dft_Filter, 0);
}
void HomomorphicFilter::Shifting_DFT(Mat &fImage)
{
//For visualization purposes we may also rearrange the quadrants of the result, so that the origin (0,0), corresponds to the image center.
Mat tmp, q0, q1, q2, q3;
/*First crop the image, if it has an odd number of rows or columns.
Operator & bit to bit by -2 (two's complement : -2 = 111111111....10) to eliminate the first bit 2^0 (In case of odd number on row or col, we take the even number in below)*/
fImage = fImage(Rect(0, 0, fImage.cols & -2, fImage.rows & -2));
int cx = fImage.cols/2;
int cy = fImage.rows/2;
/*Rearrange the quadrants of Fourier image so that the origin is at the image center*/
q0 = fImage(Rect(0, 0, cx, cy));
q1 = fImage(Rect(cx, 0, cx, cy));
q2 = fImage(Rect(0, cy, cx, cy));
q3 = fImage(Rect(cx, cy, cx, cy));
/*We reverse each quadrant of the frame with its other quadrant diagonally opposite*/
/*We reverse q0 and q3*/
q0.copyTo(tmp);
q3.copyTo(q0);
tmp.copyTo(q3);
/*We reverse q1 and q2*/
q1.copyTo(tmp);
q2.copyTo(q1);
tmp.copyTo(q2);
}
void HomomorphicFilter::Fourier_Transform(Mat frame_bw, Mat &image_phase, Mat &image_mag)
{
Mat frame_log;
frame_bw.convertTo(frame_log, CV_32F);
/*Take the natural log of the input (compute log(1 + Mag)*/
frame_log += 1;
log( frame_log, frame_log); // log(1 + Mag)
/*2. Expand the image to an optimal size
The performance of the DFT depends of the image size. It tends to be the fastest for image sizes that are multiple of 2, 3 or 5.
We can use the copyMakeBorder() function to expand the borders of an image.*/
Mat padded;
int M = getOptimalDFTSize(frame_log.rows);
int N = getOptimalDFTSize(frame_log.cols);
copyMakeBorder(frame_log, padded, 0, M - frame_log.rows, 0, N - frame_log.cols, BORDER_CONSTANT, Scalar::all(0));
/*Make place for both the complex and real values
The result of the DFT is a complex. Then the result is 2 images (Imaginary + Real), and the frequency domains range is much larger than the spatial one. Therefore we need to store in float !
That's why we will convert our input image "padded" to float and expand it to another channel to hold the complex values.
Planes is an arrow of 2 matrix (planes[0] = Real part, planes[1] = Imaginary part)*/
Mat image_planes[] = {Mat_<float>(padded), Mat::zeros(padded.size(), CV_32F)};
Mat image_complex;
/*Creates one multichannel array out of several single-channel ones.*/
merge(image_planes, 2, image_complex);
/*Make the DFT
The result of thee DFT is a complex image : "image_complex"*/
dft(image_complex, image_complex);
/***************************/
//Create spectrum magnitude//
/***************************/
/*Transform the real and complex values to magnitude
NB: We separe Real part to Imaginary part*/
split(image_complex, image_planes);
//Starting with this part we have the real part of the image in planes[0] and the imaginary in planes[1]
phase(image_planes[0], image_planes[1], image_phase);
magnitude(image_planes[0], image_planes[1], image_mag);
}
void HomomorphicFilter::Inv_Fourier_Transform(Mat image_phase, Mat image_mag, Mat &inverseTransform)
{
/*Calculates x and y coordinates of 2D vectors from their magnitude and angle.*/
Mat result_planes[2];
polarToCart(image_mag, image_phase, result_planes[0], result_planes[1]);
/*Creates one multichannel array out of several single-channel ones.*/
Mat result_complex;
merge(result_planes, 2, result_complex);
/*Make the IDFT*/
dft(result_complex, inverseTransform, DFT_INVERSE|DFT_REAL_OUTPUT);
/*Take the exponential*/
exp(inverseTransform, inverseTransform);
}
and here my main code :
/**************************/
/****Homomorphic filter****/
/**************************/
/**********************************************/
//Getting the frequency and magnitude of image//
/**********************************************/
Mat image_phase, image_mag;
HomomorphicFilter().Fourier_Transform(frame_bw, image_phase, image_mag);
/******************/
//Shifting the DFT//
/******************/
HomomorphicFilter().Shifting_DFT(image_mag);
/********************************/
//Butterworth homomorphic filter//
/********************************/
int high_h_v_TB = 101;
int low_h_v_TB = 99;
int D = 10;// radius of band pass filter parameter
int order = 2;// order of band pass filter parameter
HomomorphicFilter().butterworth_homomorphic_filter(image_mag, D, order, high_h_v_TB, low_h_v_TB);
/******************/
//Shifting the DFT//
/******************/
HomomorphicFilter().Shifting_DFT(image_mag);
/*******************************/
//Inv Discret Fourier Transform//
/*******************************/
Mat inverseTransform;
HomomorphicFilter().Inv_Fourier_Transform(image_phase, image_mag, inverseTransform);
imshow("Result", inverseTransform);
If someone can explain me my mistakes, I would appreciate a lot :). Thank you and sorry for my poor english.
EDIT : Now, i have something but it's not perfect ... I modified 2 things in my code.
I applied log(mag + 1) after dft and not on the input image.
I removed exp() after idft.
here the results (i can post only 2 links ...) :
my input image :
final result :
after having seen several topics, i find similar results on my butterworth filter and on my magnitude after dft/shifting.
Unfortunately, my final result isn't very good. Why i have so much "noise" ?
I was doing this method to balance illumination when camera was changed caused the Image waw dark!
I tried to FFT to the frequency to filter the image! it's work.but use too much time.(2750*3680RGB image).so I do it in Spatial domain.
here is my code!
//IplImage *imgSrcI=cvLoadImage("E:\\lean.jpg",-1);
Mat imgSrcM(imgSrc,true);
Mat imgDstM;
Mat imgGray;
Mat imgHls;
vector<Mat> vHls;
Mat imgTemp1=Mat::zeros(imgSrcM.size(),CV_64FC1);
Mat imgTemp2=Mat::zeros(imgSrcM.size(),CV_64FC1);
if(imgSrcM.channels()==1)
{
imgGray=imgSrcM.clone();
}
else if (imgSrcM.channels()==3)
{
cvtColor(imgSrcM, imgHls, CV_BGR2HLS);
split(imgHls, vHls);
imgGray=vHls.at(1);
}
else
{
return -1;
}
imgGray.convertTo(imgTemp1,CV_64FC1);
imgTemp1=imgTemp1+0.0001;
log(imgTemp1,imgTemp1);
GaussianBlur(imgTemp1, imgTemp2, Size(21, 21), 0.1, 0.1, BORDER_DEFAULT);//imgTemp2是低通滤波的结果
imgTemp1 = (imgTemp1 - imgTemp2);//imgTemp1是对数减低通的高通
addWeighted(imgTemp2, 0.7, imgTemp1, 1.4, 1, imgTemp1, -1);//imgTemp1是压制低频增强高频的结构
exp(imgTemp1,imgTemp1);
normalize(imgTemp1,imgTemp1,0,1,NORM_MINMAX);
imgTemp1=imgTemp1*255;
imgTemp1.convertTo(imgGray, CV_8UC1);
//imwrite("E:\\leanImgGray.jpg",imgGray);
if (imgSrcM.channels()==3)
{
vHls.at(1)=imgGray;
merge(vHls,imgHls);
cvtColor(imgHls, imgDstM, CV_HLS2BGR);
}
else if (imgSrcM.channels()==1)
{
imgDstM=imgGray.clone();
}
cvCopy(&(IplImage)imgDstM,imgDst);
//cvShowImage("jpg",imgDst);
return 0;
I took your code corrected it at a few places and got decent results as the homographic filter output.
Here are the corrections that I made.
1)
Instead of working just on the image_mag, work on the full output of the FFT.
2)
your filter values of high_h_v_TB = 101 and low_h_v_TB = 99 virtually made little effect in filtering.
Here are the values I used.
int high_h_v_TB = 100;
int low_h_v_TB = 20;
int D = 10;// radius of band pass filter parameter
int order = 4;
Here is my main code
//float_img == grayscale image in 0-1 scale
Mat log_img;
log(float_img, log_img);
Mat fft_phase, fft_mag;
Mat fft_complex;
HomomorphicFilter::Fourier_Transform(log_img, fft_complex);
HomomorphicFilter::ShiftFFT(fft_complex);
int high_h_v_TB = 100;
int low_h_v_TB = 30;
int D = 10;// radius of band pass filter parameter
int order = 4;
//get a butterworth filter of same image size as the input image
//dont call mulSpectrums yet, just get the filter of correct size
Mat butterWorthFreqDomain;
HomomorphicFilter::ButterworthFilter(fft_complex.size(), butterWorthFreqDomain, D, order, high_h_v_TB, low_h_v_TB);
//this should match fft_complex in size and type
//and is what we will be using for 'mulSpectrums' call
Mat butterworth_complex;
//make two channels to match fft_complex
Mat butterworth_channels[] = {Mat_<float>(butterWorthFreqDomain.size()), Mat::zeros(butterWorthFreqDomain.size(), CV_32F)};
merge(butterworth_channels, 2, butterworth_complex);
//do mulSpectrums on the full fft
mulSpectrums(fft_complex, butterworth_complex, fft_complex, 0);
//shift back the output
HomomorphicFilter::ShiftFFT(fft_complex);
Mat log_img_out;
HomomorphicFilter::Inv_Fourier_Transform(fft_complex, log_img_out);
Mat float_img_out;
exp(log_img_out, float_img_out);
//float_img_out is gray in 0-1 range
Here is my output.

reprojectImageTo3D() in OpenCV

I've been trying to compute real world coordinates of points from a disparity map using the reprojectImageTo3D() function provided by OpenCV, but the output seems to be incorrect.
I have the calibration parameters, and compute the Q matrix using
stereoRectify(left_cam_matrix, left_dist_coeffs, right_cam_matrix, right_dist_coeffs, frame_size, stereo_params.R, stereo_params.T, R1, R2, P1, P2, Q, CALIB_ZERO_DISPARITY, 0, frame_size, 0, 0);
I believe this first step is correct, since the stereo frames are being rectified properly, and the distortion removal I'm performing also seems all right. The disparity map is being computed with OpenCV's block matching algorithm, and it looks good too.
The 3D points are being calculated as follows:
cv::Mat XYZ(disparity8U.size(),CV_32FC3);
reprojectImageTo3D(disparity8U, XYZ, Q, false, CV_32F);
But for some reason they form some sort of cone, and are not even close to what I'd expect, considering the disparity map. I found out that other people had a similar problem with this function, and I was wondering if someone has the solution.
Thanks in advance!
[EDIT]
stereoRectify(left_cam_matrix, left_dist_coeffs, right_cam_matrix, right_dist_coeffs,frame_size, stereo_params.R, stereo_params.T, R1, R2, P1, P2, Q, CALIB_ZERO_DISPARITY, 0, frame_size, 0, 0);
initUndistortRectifyMap(left_cam_matrix, left_dist_coeffs, R1, P1, frame_size,CV_32FC1, left_undist_rect_map_x, left_undist_rect_map_y);
initUndistortRectifyMap(right_cam_matrix, right_dist_coeffs, R2, P2, frame_size, CV_32FC1, right_undist_rect_map_x, right_undist_rect_map_y);
cv::remap(left_frame, left_undist_rect, left_undist_rect_map_x, left_undist_rect_map_y, CV_INTER_CUBIC, BORDER_CONSTANT, 0);
cv::remap(right_frame, right_undist_rect, right_undist_rect_map_x, right_undist_rect_map_y, CV_INTER_CUBIC, BORDER_CONSTANT, 0);
cv::Mat imgDisparity32F = Mat( left_undist_rect.rows, left_undist_rect.cols, CV_32F );
StereoBM sbm(StereoBM::BASIC_PRESET,80,5);
sbm.state->preFilterSize = 15;
sbm.state->preFilterCap = 20;
sbm.state->SADWindowSize = 11;
sbm.state->minDisparity = 0;
sbm.state->numberOfDisparities = 80;
sbm.state->textureThreshold = 0;
sbm.state->uniquenessRatio = 8;
sbm.state->speckleWindowSize = 0;
sbm.state->speckleRange = 0;
// Compute disparity
sbm(left_undist_rect, right_undist_rect, imgDisparity32F, CV_32F );
// Compute world coordinates from the disparity image
cv::Mat XYZ(disparity32F.size(),CV_32FC3);
reprojectImageTo3D(disparity32F, XYZ, Q, false, CV_32F);
print_3D_points(disparity32F, XYZ);
[EDIT]
Adding the code used to compute 3D coords from disparity:
cv::Vec3f *StereoFrame::compute_3D_world_coordinates(int row, int col,
shared_ptr<StereoParameters> stereo_params_sptr){
cv::Mat Q_32F;
stereo_params_sptr->Q_sptr->convertTo(Q_32F,CV_32F);
cv::Mat_<float> vec(4,1);
vec(0) = col;
vec(1) = row;
vec(2) = this->disparity_sptr->at<float>(row,col);
// Discard points with 0 disparity
if(vec(2)==0) return NULL;
vec(3)=1;
vec = Q_32F*vec;
vec /= vec(3);
// Discard points that are too far from the camera, and thus are highly
// unreliable
if(abs(vec(0))>10 || abs(vec(1))>10 || abs(vec(2))>10) return NULL;
cv::Vec3f *point3f = new cv::Vec3f();
(*point3f)[0] = vec(0);
(*point3f)[1] = vec(1);
(*point3f)[2] = vec(2);
return point3f;
}
Your code seems fine to me. It could be a bug with the reprojectImageTo3D. Try to replace it with the following code (which has the same role):
cv::Mat_<cv::Vec3f> XYZ(disparity32F.rows,disparity32F.cols); // Output point cloud
cv::Mat_<float> vec_tmp(4,1);
for(int y=0; y<disparity32F.rows; ++y) {
for(int x=0; x<disparity32F.cols; ++x) {
vec_tmp(0)=x; vec_tmp(1)=y; vec_tmp(2)=disparity32F.at<float>(y,x); vec_tmp(3)=1;
vec_tmp = Q*vec_tmp;
vec_tmp /= vec_tmp(3);
cv::Vec3f &point = XYZ.at<cv::Vec3f>(y,x);
point[0] = vec_tmp(0);
point[1] = vec_tmp(1);
point[2] = vec_tmp(2);
}
}
I never used reprojectImageTo3D, however I am using successfully code similar to the snippet above.
[Initial answer]
As it is explained in the documentation for StereoBM, if you request a CV_16S disparity map, you have to divide each disparity value by 16 before using them.
Hence, you should convert the disparity map as follows before using it:
imgDisparity16S.convertTo( imgDisparity32F, CV_32F, 1./16);
You can also directly request a CV_32F disparity map from the StereoBM structure, in which case you directy get the true disparities.

cv::warpPerspective only shows part of warped image

Im changing an image from front perspective to a bids eye view by using getHomography and warpPerspective.
It works in that the image warps to the desired perspective but the crop is off. It moves the warped image largely outside the image box. I assume the reason is because the operation results in negative coordinates.
I have calculated the points for calculation of the translation matrix manually and not by using any of opencv:s functions for doing that since i.e. the chessboard functions failed to detect the proper points.
I guess this can be fixed by doing additional changes to the transformation matrix. But how is that done? Also, is there a way to make sure the transformed image is centered along the x-axis and then let the y-axis be adjusted to a desired position?
Code snippet that does the job now:
cv::Mat image; // image is loaded with the original image
cv::Mat warpPers; // The container for the resulting image
cv::Mat H;
std::vector<cv::Point2f> src;
std::vector<cv::Point2f> dst;
// In reality several more points.
src.push_back(cv::Point2f(264,301));
src.push_back(cv::Point2f(434,301));
src.push_back(cv::Point2f(243,356));
src.push_back(cv::Point2f(476,356));
dst.push_back(cv::Point2f(243,123));
dst.push_back(cv::Point2f(476,123));
dst.push_back(cv::Point2f(243,356));
dst.push_back(cv::Point2f(476,356));
H = cv::findHomography(src, dst, CV_RANSAC);
cv::warpPerspective(image,
newPers,
H,
cv::Size(3000,3000),
cv::INTER_NEAREST | CV_WARP_FILL_OUTLIERS
);
cv::namedWindow("Warped persp", cv::WINDOW_AUTOSIZE );
cv::imshow( "Warped persp", newPers);
Opencv gives very convenient way to do perpective transform. The only thing you have to do is take care of the homography return by findHomography.
Indeed, maybe some points of the image you provide go in the negative part of the x or y axis.
So you have to do some check before warp the image.
step 1: find the homography H with findHomography
you will get a classic structure for homography
H = [ h00, h01, h02;
h10, h11, h12;
h20, h21, 1];
step 2: search the position of image's corners after warping
So let me define the order for the corner:
(0,0) ________ (0, w)
| |
|________|
(h,0) (h,w)
To do that, just create a matrix like that:
P = [0, w, w, 0;
0, 0, h, h;
1, 1, 1, 1]
Make the product with H and get the warped coordinates:
P' = H * P
step 3: check the minimum in x and y with these new 4 points and get the size of warped image
After, you have done the product you will receive something like that:
P' = [s1*x1, s2*x2, s3*x3, s4*x4;
s1*y1, s2*y2, s3*y3, s4*y4;
s1 , s2 , s3 , s4]
So to obtain, new valid coordinate just divide line 1 and 2 by the line 3
After that check the minimum for the column on the first line, and the minimum for the row on the second line (use cvReduce)
to find the bounding box that will contains the image (ie the dimension of the dst matrix for the warpPerspective function) just find with cvReduce the maximum over each line
let minx be the minimum on the first row (ie for column), maxx (the maximum for the 1 row)
miny and maxy for the second row.
So the size of the warped image should be cvSize(maxx-minx, maxy-miny)
step 4: add a correction to the homography
Check if minx and/or miny is/are negative, if minx < 0 then add -minx to h02 and if miny < 0, then add -miny to h12
so H should be:
H = [ h00, h01, h02-minx; //if minx <0
h10, h11, h12-miny; //if miny <0
h20, h21, 1];
step 5: warp the image
I think this question OpenCV warpperspective is similar to the current question cv::warpPerspective only shows part of warped image
So i give you my answer https://stackoverflow.com/a/37275961/15485 also here:
Try the below homography_warp.
void homography_warp(const cv::Mat& src, const cv::Mat& H, cv::Mat& dst);
src is the source image.
H is your homography.
dst is the warped image.
homography_warp adjust your homography as described by https://stackoverflow.com/users/1060066/matt-freeman in his answer https://stackoverflow.com/a/8229116/15485
// Convert a vector of non-homogeneous 2D points to a vector of homogenehous 2D points.
void to_homogeneous(const std::vector< cv::Point2f >& non_homogeneous, std::vector< cv::Point3f >& homogeneous)
{
homogeneous.resize(non_homogeneous.size());
for (size_t i = 0; i < non_homogeneous.size(); i++) {
homogeneous[i].x = non_homogeneous[i].x;
homogeneous[i].y = non_homogeneous[i].y;
homogeneous[i].z = 1.0;
}
}
// Convert a vector of homogeneous 2D points to a vector of non-homogenehous 2D points.
void from_homogeneous(const std::vector< cv::Point3f >& homogeneous, std::vector< cv::Point2f >& non_homogeneous)
{
non_homogeneous.resize(homogeneous.size());
for (size_t i = 0; i < non_homogeneous.size(); i++) {
non_homogeneous[i].x = homogeneous[i].x / homogeneous[i].z;
non_homogeneous[i].y = homogeneous[i].y / homogeneous[i].z;
}
}
// Transform a vector of 2D non-homogeneous points via an homography.
std::vector<cv::Point2f> transform_via_homography(const std::vector<cv::Point2f>& points, const cv::Matx33f& homography)
{
std::vector<cv::Point3f> ph;
to_homogeneous(points, ph);
for (size_t i = 0; i < ph.size(); i++) {
ph[i] = homography*ph[i];
}
std::vector<cv::Point2f> r;
from_homogeneous(ph, r);
return r;
}
// Find the bounding box of a vector of 2D non-homogeneous points.
cv::Rect_<float> bounding_box(const std::vector<cv::Point2f>& p)
{
cv::Rect_<float> r;
float x_min = std::min_element(p.begin(), p.end(), [](const cv::Point2f& lhs, const cv::Point2f& rhs) {return lhs.x < rhs.x; })->x;
float x_max = std::max_element(p.begin(), p.end(), [](const cv::Point2f& lhs, const cv::Point2f& rhs) {return lhs.x < rhs.x; })->x;
float y_min = std::min_element(p.begin(), p.end(), [](const cv::Point2f& lhs, const cv::Point2f& rhs) {return lhs.y < rhs.y; })->y;
float y_max = std::max_element(p.begin(), p.end(), [](const cv::Point2f& lhs, const cv::Point2f& rhs) {return lhs.y < rhs.y; })->y;
return cv::Rect_<float>(x_min, y_min, x_max - x_min, y_max - y_min);
}
// Warp the image src into the image dst through the homography H.
// The resulting dst image contains the entire warped image, this
// behaviour is the same of Octave's imperspectivewarp (in the 'image'
// package) behaviour when the argument bbox is equal to 'loose'.
// See http://octave.sourceforge.net/image/function/imperspectivewarp.html
void homography_warp(const cv::Mat& src, const cv::Mat& H, cv::Mat& dst)
{
std::vector< cv::Point2f > corners;
corners.push_back(cv::Point2f(0, 0));
corners.push_back(cv::Point2f(src.cols, 0));
corners.push_back(cv::Point2f(0, src.rows));
corners.push_back(cv::Point2f(src.cols, src.rows));
std::vector< cv::Point2f > projected = transform_via_homography(corners, H);
cv::Rect_<float> bb = bounding_box(projected);
cv::Mat_<double> translation = (cv::Mat_<double>(3, 3) << 1, 0, -bb.tl().x, 0, 1, -bb.tl().y, 0, 0, 1);
cv::warpPerspective(src, dst, translation*H, bb.size());
}
If I understood correctly, basically question demands the method to calculate the correct offset for translation of the warped image. I will explain how to get the right offset for translation. Idea is that matching features in two images should have the same coordinate in the final stitched image.
Let's say we refer images as follows:
'source image' (si): the image which needs to be warped
'destination image' (di): the image to whose perspective 'source image' will be warped
'warped source image'(wsi): source image
after warping it to the destination image perspective
Following is what you need to do in order to calculate offset for translation:
After you have sampled the good matches and found the mask from homography, store the best match's keypoint(one with a minimum distance and being an inlier (should get the value of 1 in mask obtained from homography calculation)) in si and di. Let's say best match's keypoint in si and diisbm_siandbm_di` respectively..
bm_si = [x1, y1,1]
bm_di = [x2, y2, 1]
Find the position of bm_si in wsi by simply multiplying it with the homography matrix (H).
bm_wsi = np.dot(H,bm_si)
bm_wsi = [x/bm_wsi[2] for x in bm_wsi]
Depending on where you will be placing the di on the output of si warping (=wsi), adjust the bm_di
Let's say if you are warping from the left image to right image (such that left image is si and the right image is di) then you will placing di on the right side wsi and hence bm_di[0] += si.shape[0]
Now after the above steps
x_offset = bm_di[0] - bm_si[0]
y_offset = bm_di[1] - bm_si[1]
Using calculated offset find the new homography matrix and warp the si.
T = np.array([[1, 0, x_offset], [0, 1, y_offset], [0, 0, 1]])
translated_H = np.dot(T.H)
wsi_frame_size = tuple(2*x for x in si.shape)
stitched = cv2.warpPerspective(si, translated_H, wsi_frame_size)
stitched[0:si.shape[0],si.shape[1]:] = di