Affine transform in C++ - c++

I am currently making a project for school on image processing in visual Studio 2013, using Open CV 3.1. My goal (for now) is to transform an image, using affine transform, so that the trapezoidal board will be transformed into a rectangle.
To do that I have substracted certain channels and thresholded the image so that now I have a binary image with white blocks in the corners of the board.
Now I need to pick 4 white points that are closest to each corner and (using affine transform) set them as corners of the transformed image.
And since this is my first time using Open CV, I am stuck.
Here's my code:
#include <iostream>
#include <opencv2\core.hpp>
#include <opencv2\highgui.hpp>
#include<opencv2/imgproc.hpp>
#include <stdlib.h>
#include <stdio.h>
#include <vector>
int main(){
double dist;
cv::Mat image;
image = cv::imread("C:\\Users\\...\\ideal.png");
cv::Mat imagebin;
imagebin = cv::imread("C:\\Users\\...\\ideal.png");
cv::Mat imageerode;
//cv::imshow("Test", image);
cv::Mat src = cv::imread("C:\\Users\\...\\ideal.png");
std::vector<cv::Mat>img_rgb;
cv::split(src, img_rgb);
//cv::imshow("ideal.png", img_rgb[2] - img_rgb[1]);
cv::threshold(img_rgb[2] - 0.5*img_rgb[1], imagebin , 20, 255, CV_THRESH_BINARY);
cv::erode(imagebin, imageerode, cv::Mat(), cv::Point(1, 1), 2, 1, 1);
cv::erode(imageerode, imageerode, cv::Mat(), cv::Point(1, 1), 2, 1, 1);
// cv::Point2f array[4];
// std::vector<cv::Point2f> array;
for (int i = 0; i < imageerode.cols; i++)
{
for (int j = 0; j < imageerode.rows; j++)
{
if (imageerode.at<uchar>(i,j) > 0)
{
dist = std::min(dist, i + j);
}
}
}
//cv::imshow("Test binary", imagebin);
cv::namedWindow("Test", CV_WINDOW_NORMAL);
cv::imshow("Test", imageerode);
cv::waitKey(0);
std::cout << "Hello world!";
return 0;
}
As you can see I don't know how to loop over each white pixel using image.at and save the distance to each corner.
I would appreciate some help.
Also: I don't want to just do this. I really want to learn how to do that. But I'm currently having some mindstuck.
Thank you
EDIT:
I think I'm done with finding the coordinates of the 4 points. But I can't really get the idea of the warpAffine syntax.
Code:
for (int i = 0; i < imageerode.cols; i++)
{
for (int j = 0; j < imageerode.rows; j++)
{
if (imageerode.at<uchar>(i, j) > 0)
{
if (i + j < distances[0])
{
distances[0] = i + j;
coordinates[0] = i;
coordinates[1] = j;
}
if (i + imageerode.cols-j < distances[1])
{
distances[1] = i + imageerode.cols-j;
coordinates[2] = i;
coordinates[3] = j;
}
if (imageerode.rows-i + j < distances[2])
{
distances[2] = imageerode.rows - i + j;
coordinates[4] = i;
coordinates[5] = j;
}
if (imageerode.rows-i + imageerode.cols-j < distances[3])
{
distances[3] = imageerode.rows - i + imageerode.cols - j;
coordinates[6] = i;
coordinates[7] = j;
}
}
}
Where I set all of the distances values to imageerode.cols+imageerode.rows since it's the maximum value it can get.
Also: note that I'm using taxicab geometry. I was told it's faster and the results are pretty much the same.
If anyone could help me with warpAffine it would be great. I don't understand where do I put the coordinates I have found.
Thank you

I am not sure how your "trapezoidal board" looks like but if it has a perspective transform like when you capture a rectangle with a camera, then an affine transform is not enough. Use perspective transform. I think Features2D + Homography to find a known object is very close to what you want to do.

Related

Calculate 1DPlot, determine the maxima and their distances between each other

I want to create a 1D plot from an image. Then I want to determine the maxima and their distances to each other in c++.
I am looking for some tips on how I could approach this.
I load the image as cv::Mat. In opencv I have searched, but only found the histogram function, which is wrong. I want to get a cross section of the image - from left to right.
does anyone have an idea ?
Well I have the following picture:
From this I want to create a 1D plot like in the following picture (I created the plot in ImageJ).
Here you can see the maxima (I could refine it with "smooth").
I want to determine the positions of these maxima and then the distances between them.
I have to get to the 1D plot somehow. I suppose I can get to the maxima with a derivation?
++++++++++ UPDATE ++++++++++
Now i wrote this to get an 1D Plot:
cv::Mat img= cv::imread(imgFile.toStdString(), cv::IMREAD_ANYDEPTH | cv::IMREAD_COLOR);
cv::cvtColor(img, img, cv::COLOR_BGR2GRAY);
uint8_t* data = img.data;
int width = img.cols;
int height = img.rows;
int stride = img.step;
std::vector<double> vPlot(width, 0);
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
uint8_t val = data[ i * stride + j];
vPlot[j]=vPlot[j] + val;
}
}
std::ofstream file;
file.open("path\\plot.csv");
for(int i = 0; i < vPlot.size(); i++){
file << vPlot[i];
file << ";";
}
file.close();
When i plot this in excel i got this:
Thats looks not so smooth as in ImageJ. Did i something wrong?
I need it like in the Plot of ImageJ - more smooth.
ok I got it:
for (int i = 0; i < vPlot.size(); i++) {
vPlot[i] = vPlot[i] / height;
}
Ok but i don't know how to get the maxima an distances.
When i have the local maxima (i don't know how), i can calculate the distance between them with the index of the vetcor elements.
Has anybody an idea to get the local Maxima out of the vector, that I plot above ?
Now o wrote this to find the maxima:
// find maxima
std::vector<int> idxMax;
int flag = 0;
for(int i = 1; i < avg.size(); i++){
double diff = avg[i] - avg[i-1];
if(diff < 0){
if(flag>0){
idxMax.push_back(i);
flag = -1;
}
}
if(diff >= 0){
if(flag<=0){
flag = 1;
}
}
}
But more maxima are found than wanted. The length of the vector varies and also the number of peaks. These can be close together or far away. They are also not always the same height, as can be seen in the picture

Slow motion in C++

I want to do slow motion. I've seen an implementation here: https://github.com/vaibhav06891/SlowMotion
I modified the code to generate only one frame.
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/video/tracking.hpp>
#include <opencv2/opencv.hpp>
#include <iostream>
#include <fstream>
#include <string>
using namespace cv;
using namespace std;
#define CLAMP(x,min,max) ( ((x) < (min)) ? (min) : ( ((x) > (max)) ? (max) : (x) ) )
int main(int argc, char** argv)
{
Mat frame,prevframe;
prevframe = imread("img1.png");
frame = imread("img2.png");
Mat prevgray, gray;
Mat fflow,bflow;
Mat flowf(frame.rows,frame.cols ,CV_8UC3); // the forward co-ordinates for interpolation
flowf.setTo(Scalar(255,255,255));
Mat flowb(frame.rows,frame.cols ,CV_8UC3); // the backward co-ordinates for interpolation
flowb.setTo(Scalar(255,255,255));
Mat final(frame.rows,frame.cols ,CV_8UC3);
int fx,fy,bx,by;
cvtColor(prevframe,prevgray,COLOR_BGR2GRAY); // Convert to gray space for optical flow calculation
cvtColor(frame, gray, COLOR_BGR2GRAY);
calcOpticalFlowFarneback(prevgray, gray, fflow, 0.5, 3, 15, 3, 3, 1.2, 0); // forward optical flow
calcOpticalFlowFarneback(gray, prevgray, bflow, 0.5, 3, 15, 3, 3, 1.2, 0); //backward optical flow
for (int y=0; y<frame.rows; y++)
{
for (int x=0; x<frame.cols; x++)
{
const Point2f fxy = fflow.at<Point2f>(y,x);
fy = CLAMP(y+fxy.y*0.5,0,frame.rows);
fx = CLAMP(x+fxy.x*0.5,0,frame.cols);
flowf.at<Vec3b>(fy,fx) = prevframe.at<Vec3b>(y,x);
const Point2f bxy = bflow.at<Point2f>(y,x);
by = CLAMP(y+bxy.y*(1-0.5),0,frame.rows);
bx = CLAMP(x+bxy.x*(1-0.5),0,frame.cols);
flowb.at<Vec3b>(by,bx) = frame.at<Vec3b>(y,x);
}
}
final = flowf*(1-0.5) + flowb*0.5; //combination of frwd and bckward martrix
cv::medianBlur(final,final,3);
imwrite( "output.png",final);
return 0;
}
But the result is not as expected.
For the images:
The result is :
Does anyone know what is the problem?
The optical flow algorithm won't work for your test images.
The first problem is that your test images have very little difference in neighbour pixel values. That completely black lines and a single color square give no clues to optical flow algorithm where the image areas moved as the algorithm is not able to process the whole image at once and calculates optical flow with a small 15x15 (as you set it in calcOpticalFlowFarneback) pixels window.
The second problem is that your test images differ too much. The distance between positions of brown square is too big. Again Farneback is not able to detect it.
Try the code with some real life video frames or edit your tests to be less monotonous (set some texture to the square, background and rectangle lines) and bring the squares closer to each other on the images (try 2-10 px distance). You can also play with calcOpticalFlowFarneback arguments (read here) to suit your conditions.
You can use this code to save the optical flow you get to an image for debugging:
Mat debugImage = Mat::zeros(fflow.size(), CV_8UC3);
float hsvHue, magnitude;
for (int x = 0; x < fflow.cols; x++)
{
for (int y = 0; y < fflow.rows; y++)
{
auto& item = fflow.at<Vec2f>(y, x);
magnitude = sqrtf(item[0] * item[0] + item[1] * item[1]);
hsvHue = atan2f(item[1], item[0]) / static_cast<float>(CV_PI)* 180.f;
// div 2 to fit 0..255 range
hsvHue = (hsvHue >= 0. ? hsvHue : (360.f + hsvHue)) / 2.f;
debugImage.at<Vec3b>(y, x)[0] = static_cast<uchar>(hsvHue);
debugImage.at<Vec3b>(y, x)[1] = 255;
debugImage.at<Vec3b>(y, x)[2] = static_cast<uchar>(255.f * magnitude);
}
}
cvtColor(debugImage, debugImage, CV_HSV2BGR);
imwrite("OpticalFlow.png", debugImage);
Here pixel flow direction will be represented with color (hue), and pixel move distance will be represented with brightness.
Try to use this images I created:
.
Also note that
for (int y = 0; y < frame.rows; y++)
{
for (int x = 0; x < frame.cols; x++)
{
const Point2f fxy = fflow.at<Point2f>(y, x);
fy = CLAMP(y + fxy.y*0.5, 0, frame.rows);
fx = CLAMP(x + fxy.x*0.5, 0, frame.cols);
flowf.at<Vec3b>(fy, fx) = prevframe.at<Vec3b>(y, x);
...
code won't color some flowf pixels that have no corresponding target positions they moved to, and optical flow algorithm can produce such situations. I would change it to:
for (int y = 0; y < frame.rows; y++)
{
for (int x = 0; x < frame.cols; x++)
{
const Point2f fxy = fflow.at<Point2f>(y, x);
fy = CLAMP(y - fxy.y*0.5, 0, frame.rows);
fx = CLAMP(x - fxy.x*0.5, 0, frame.cols);
flowf.at<Vec3b>(y, x) = prevframe.at<Vec3b>(fy, fx);
const Point2f bxy = bflow.at<Point2f>(y, x);
by = CLAMP(y - bxy.y*(1 - 0.5), 0, frame.rows);
bx = CLAMP(x - bxy.x*(1 - 0.5), 0, frame.cols);
flowb.at<Vec3b>(y, x) = frame.at<Vec3b>(by, bx);
}
}
With this changed code and my tests I get this output:

3D reconstruction from multiple images with one camera

So, I've been trying to get a 3D cloud point from a sequence of images of an object. I have successfully obtained a decent point cloud with two images. I got that from matching features on both images, finding the fundamental matrix and from that, extracting P' (the camera matrix for the second view). For the first view, I set P = K(I | 0), where K is the matrix for the camera intrinsics. But I haven't been able to extend this approach to several images. My idea was to do this sliding the two image window through the sequence of images(e.g. match image1 with image2, find 3d points, match image2 with image3 and then find the more 3d points, and so on). For the following image pairs, P would be made of a cumulative rotation matrix and a cumulative translation vector (this would allow me to keep bringing the points to the first camera coordinate system). But this is not working at all. I'm using OpenCV. What I wanna know is if this approach makes sense at all.
In the code, P_prev is P and Pl is P'. This is just the part that I think it's relevant.
Mat combinedPointCloud;
Mat P_prev;
P_prev = (Mat_<double>(3,4) << cameraMatrix.at<double>(0,0), cameraMatrix.at<double>(0,1), cameraMatrix.at<double>(0,2), 0,
cameraMatrix.at<double>(1,0), cameraMatrix.at<double>(1,1), cameraMatrix.at<double>(1,2), 0,
cameraMatrix.at<double>(2,0), cameraMatrix.at<double>(2,1), cameraMatrix.at<double>(2,2), 0);
for(int i = 1; i < images.size(); i++) {
Mat points3D;
image1 = images[i-1];
image2 = images[i];
matchTwoImages(image1, image2, imgpts1, imgpts2);
P = findSecondProjectionMatrix(cameraMatrix, imgpts1, imgpts2);
P.col(0).copyTo(R.col(0));
P.col(1).copyTo(R.col(1));
P.col(2).copyTo(R.col(2));
P.col(3).copyTo(t.col(0));
if(i == 1) {
Pl = P;
triangulatePoints(P_prev, Pl, imgpts1, imgpts2, points3D); //points3D is 4xN
//Transforming to euclidean by hand, because couldn't make
// opencv's convertFromHomogeneous work
aux.create(3, points3D.cols, CV_64F);// aux is 3xN
for(int i = 0; i < points3D.cols; i++) {
aux.at<float>(0, i) = points3D.at<float>(0, i)/points3D.at<float>(3, i);
aux.at<float>(1, i) = points3D.at<float>(1, i)/points3D.at<float>(3, i);
aux.at<float>(2, i) = points3D.at<float>(2, i)/points3D.at<float>(3, i);
}
points3D.create(3, points3D.cols, CV_64F);
aux.copyTo(points3D);
}
else {
R_aux = R_prev * R;
t_aux = t_prev + t;
R_aux.col(0).copyTo(Pl.col(0));
R_aux.col(1).copyTo(Pl.col(1));
R_aux.col(2).copyTo(Pl.col(2));
t_aux.col(0).copyTo(Pl.col(3));
triangulatePoints(P_prev, Pl, imgpts1, imgpts2, points3D);
//Transforming to euclidean by hand, because couldn't make
// opencv's convertFromHomogeneous work
aux.create(3, points3D.cols, CV_64F);// aux is 3xN
for(int i = 0; i < points3D.cols; i++) {
aux.at<float>(0, i) = points3D.at<float>(0, i)/points3D.at<float>(3, i);
aux.at<float>(1, i) = points3D.at<float>(1, i)/points3D.at<float>(3, i);
aux.at<float>(2, i) = points3D.at<float>(2, i)/points3D.at<float>(3, i);
}
points3D.create(3, points3D.cols, CV_64F);
aux.copyTo(points3D);
}
Pl.col(0).copyTo(R_prev.col(0));
Pl.col(1).copyTo(R_prev.col(1));
Pl.col(2).copyTo(R_prev.col(2));
Pl.col(3).copyTo(t_prev.col(0));
P_prev = Pl;
if(i==1) {
points3D.copyTo(combinedPointCloud);
} else {
hconcat(combinedPointCloud, points3D, combinedPointCloud);
}
}
show3DCloud(comninedPointCloud);

Convert Matlab based Ridge Segment Function into C++

I am going to perform ridge segmentation on an input image using OpenCV. From the internet, I found a Matlab code as follows, which fits quite well with my goal:
function [normim, mask, maskind] = ridgesegment(im, blksze, thresh)
im = normalise(im,0,1); % normalise to have zero mean, unit std dev
fun = inline('std(x(:))*ones(size(x))');
stddevim = blkproc(im, [blksze blksze], fun);
mask = stddevim > thresh;
maskind = find(mask);
% Renormalise image so that the *ridge regions* have zero mean, unit
% standard deviation.
im = im - mean(im(maskind));
normim = im/std(im(maskind));
end
So I tried to convert it to C++. Up to now, I can only finish these parts:
cv::Mat ridgeSegment(cv::Mat inputImg, int blockSize, double thresh)
{
cv::normalize(inputImg, inputImg, 0, 1.0, cv::NORM_MINMAX, CV_8UC1);
blkproc(inputImg, cv::Size(blockSize, blockSize), thresh);
...// how to do the next steps ????
}
cv::Mat blkproc(cv::Mat img, cv::Size size, double thresh)
{
cv::Mat croppedImg;
for (int i = 0; i < im.cols; i += size.width)
{
for (int j = 0; j < im.rows; j += size.height)
{
croppedImg = im(cv::Rect(i, j, size.width, size.height)).clone();
//perform standard deviation calculation here???
}
}
return croppedImg;
}
I don't know how to proceed further here. Especially that stddevim and its later parts. Could someone explain and show me the rest? Thank you in advance.

OpenCV VLFeat Slic function call

I am trying to use the vl_slic_segment function of the VLFeat library using an input image stored in an OpenCV Mat. My code is compiling and running, but the output superpixel values do not make sense. Here is my code so far :
Mat bgrUChar = imread("/pathtowherever/image.jpg");
Mat bgrFloat;
bgrUChar.convertTo(bgrFloat, CV_32FC3, 1.0/255);
cv::Mat labFloat;
cvtColor(bgrFloat, labFloat, CV_BGR2Lab);
Mat labels(labFloat.size(), CV_32SC1);
vl_slic_segment(labels.ptr<vl_uint32>(),labFloat.ptr<const float>(),labFloat.cols,labFloat.rows,labFloat.channels(),30,0.1,25);
I have tried not converting it to the Lab colorspace and setting different regionSize/regularization, but the output is always very glitchy. I am able to retrieve the label values correctly, the thing is the every labels is usually scattered on a little non-contiguous area.
I think the problem is the format of my input data is wrong but I can't figure out how to send it properly to the vl_slic_segment function.
Thank you in advance!
EDIT
Thank you David, as you helped me understand, vl_slic_segment wants data ordered as [LLLLLAAAAABBBBB] whereas OpenCV is ordering its data [LABLABLABLABLAB] for the LAB color space.
In the course of my bachelor thesis I have to use VLFeat's SLIC implementation as well. You can find a short example applying VLFeat's SLIC on Lenna.png on GitHub: https://github.com/davidstutz/vlfeat-slic-example.
Maybe, a look at main.cpp will help you figuring out how to convert the images obtained by OpenCV to the right format:
// OpenCV can be used to read images.
#include <opencv2/opencv.hpp>
// The VLFeat header files need to be declared external.
extern "C" {
#include "vl/generic.h"
#include "vl/slic.h"
}
int main() {
// Read the Lenna image. The matrix 'mat' will have 3 8 bit channels
// corresponding to BGR color space.
cv::Mat mat = cv::imread("Lenna.png", CV_LOAD_IMAGE_COLOR);
// Convert image to one-dimensional array.
float* image = new float[mat.rows*mat.cols*mat.channels()];
for (int i = 0; i < mat.rows; ++i) {
for (int j = 0; j < mat.cols; ++j) {
// Assuming three channels ...
image[j + mat.cols*i + mat.cols*mat.rows*0] = mat.at<cv::Vec3b>(i, j)[0];
image[j + mat.cols*i + mat.cols*mat.rows*1] = mat.at<cv::Vec3b>(i, j)[1];
image[j + mat.cols*i + mat.cols*mat.rows*2] = mat.at<cv::Vec3b>(i, j)[2];
}
}
// The algorithm will store the final segmentation in a one-dimensional array.
vl_uint32* segmentation = new vl_uint32[mat.rows*mat.cols];
vl_size height = mat.rows;
vl_size width = mat.cols;
vl_size channels = mat.channels();
// The region size defines the number of superpixels obtained.
// Regularization describes a trade-off between the color term and the
// spatial term.
vl_size region = 30;
float regularization = 1000.;
vl_size minRegion = 10;
vl_slic_segment(segmentation, image, width, height, channels, region, regularization, minRegion);
// Convert segmentation.
int** labels = new int*[mat.rows];
for (int i = 0; i < mat.rows; ++i) {
labels[i] = new int[mat.cols];
for (int j = 0; j < mat.cols; ++j) {
labels[i][j] = (int) segmentation[j + mat.cols*i];
}
}
// Compute a contour image: this actually colors every border pixel
// red such that we get relatively thick contours.
int label = 0;
int labelTop = -1;
int labelBottom = -1;
int labelLeft = -1;
int labelRight = -1;
for (int i = 0; i < mat.rows; i++) {
for (int j = 0; j < mat.cols; j++) {
label = labels[i][j];
labelTop = label;
if (i > 0) {
labelTop = labels[i - 1][j];
}
labelBottom = label;
if (i < mat.rows - 1) {
labelBottom = labels[i + 1][j];
}
labelLeft = label;
if (j > 0) {
labelLeft = labels[i][j - 1];
}
labelRight = label;
if (j < mat.cols - 1) {
labelRight = labels[i][j + 1];
}
if (label != labelTop || label != labelBottom || label!= labelLeft || label != labelRight) {
mat.at<cv::Vec3b>(i, j)[0] = 0;
mat.at<cv::Vec3b>(i, j)[1] = 0;
mat.at<cv::Vec3b>(i, j)[2] = 255;
}
}
}
// Save the contour image.
cv::imwrite("Lenna_contours.png", mat);
return 0;
}
In addition, have a look at README.md within the GitHub repository. The following figures show some example outputs of setting the regularization to 1 (100,1000) and setting the region size to 30 (20,40).
Figure 1: Superpixel segmentation with region size set to 30 and regularization set to 1.
Figure 2: Superpixel segmentation with region size set to 30 and regularization set to 100.
Figure 3: Superpixel segmentation with region size set to 30 and regularization set to 1000.
Figure 4: Superpixel segmentation with region size set to 20 and regularization set to 1000.
Figure 5: Superpixel segmentation with region size set to 20 and regularization set to 1000.