How to use CNN in openCV via C++? - c++

Here https://stackoverflow.com/a/49817506/1277317
there is an example of how to use a convolution network in OpenCV. But this example is in Python.
How to do the same in C++?
Namely, how to do this in C++:
net = cv.dnn.readNetFromTensorflow('model.pb')
net.setInput(inp.transpose(0, 3, 1, 2))
cv_out = net.forward()
?
And how to create Mat for the setInput function for an image size: 60x162x1? I use float for the data just like in the python example.
Now I have this code and it gives incorrect results:
Net net = readNet("e://xor.pb");
float x0[60][162];
for(int i=0;i<60;i++)
{
for(int j=0;j<162;j++)
{
x0[i][j]=0;
}
}
x0[5][59]=0.5;
x0[5][60]=1;
x0[5][61]=1;
x0[5][62]=0.5;
Mat aaa = cv::Mat(60,162, CV_32F, x0);
Mat inputBlob = dnn::blobFromImage(aaa, 1.0, Size(60,162));
net.setInput(inputBlob , "conv2d_input");
Mat prob = net.forward("activation_2/Softmax");
for(int i=0;i<prob.cols;i++)
{
qDebug()<<i<<prob.at<float>(0,i);
}

In openCV almost all functions are designed to work with 3D matrices. So the easiest way for me to work with CV_32F 4D matrices is to work with them directly. The following code works correctly and quickly:
Net net = readNet("e://xor.pb");
const int sizes[] = {1,1,60,162};
Mat tenz = Mat::zeros(4, sizes, CV_32F);
float* dataB=(float*)tenz.data;
int x=1;
int y=2;
dataB[y*tenz.size[2]+x]=0.5f;
x=1;
y=3;
dataB[y*tenz.size[2]+x]=1.0f;
try
{
net.setInput(tenz , "input_layer_my_input_1");
Mat prob = net.forward("output_layer_my/MatMul");
}
catch( cv::Exception& e )
{
const char* err_msg = e.what();
qDebug()<<"err_msg"<<err_msg;
}

Related

cv::getPerspectiveTransform throws Error after cv::eigen2cv

My Code:
cv::Mat
getPerspectiveTransform(Eigen::MatrixXd quadrangle, Eigen::MatrixXd warpedQuadrangle) {
cv::Mat transMat;
cv::Mat quad(4,2,CV_32FC1);
cv::Mat warpedQuad(4,2,CV_32FC1);
cv::eigen2cv(quadrangle,quad);
cv::eigen2cv(warpedQuadrangle,warpedQuad);
std::cout << "[ ] quadrangle in cv::Mat " << quad << std::endl;
transMat = cv::getPerspectiveTransform(quad,warpedQuad);
return transMat;
}
Error:
C++ exception with description "OpenCV(4.6.0) /home/ci/opencv/modules/imgproc/src/imgwarp.cpp:3392: error: (-215:Assertion failed) src.checkVector(2, CV_32F) == 4 && dst.checkVector(2, CV_32F) == 4 in function 'getPerspectiveTransform'
Suspected Issue:
eigen2cv is converting my CV_32FC1 to CV_64F. getPerspectiveTransform is expecting CV32F as its input.
What should be the ideal solution to this?
eigen2cv changes the layout of output matrix according to type of the input matrix, as can be seen from its source. So, e.g., if your Eigen matrices use 64-bit floats, output Mat's will have CV_64F depth. In this case, the simplest solution is to convert output matrices to CV_32F using Mat::convertTo (documentation):
Mat quadF, warpedQuadF;
quad.convertTo(quadF, CV_32F);
warpedQuad.convertTo(warpedQuadF, CV_32F);
transMat = cv::getPerspectiveTransform(quadF, warpedQuadF);
Since matrices are pretty small, this conversion is unlikely to be a performance issue, but it's possible to avoid it by rewriting 2 overloads of getPerspectiveTransform function (source1 and source2) to work with 64-bit floats. Second overload just delegates to the first, and the first works with double's internally anyway, so it's pretty trivial:
cv::Mat getPerspectiveTransform64(const Point2d src[], const Point2d dst[], int solveMethod)
{
CV_INSTRUMENT_REGION();
Mat M(3, 3, CV_64F), X(8, 1, CV_64F, M.ptr());
double a[8][8], b[8];
Mat A(8, 8, CV_64F, a), B(8, 1, CV_64F, b);
for( int i = 0; i < 4; ++i )
{
a[i][0] = a[i+4][3] = src[i].x;
a[i][1] = a[i+4][4] = src[i].y;
a[i][2] = a[i+4][5] = 1;
a[i][3] = a[i][4] = a[i][5] = a[i+4][0] = a[i+4][1] = a[i+4][2] = 0;
a[i][6] = -src[i].x*dst[i].x;
a[i][7] = -src[i].y*dst[i].x;
a[i+4][6] = -src[i].x*dst[i].y;
a[i+4][7] = -src[i].y*dst[i].y;
b[i] = dst[i].x;
b[i+4] = dst[i].y;
}
solve(A, B, X, solveMethod);
M.ptr<double>()[8] = 1.;
return M;
}
cv::Mat getPerspectiveTransform64(InputArray _src, InputArray _dst, int solveMethod)
{
Mat src = _src.getMat(), dst = _dst.getMat();
CV_Assert(src.checkVector(2, CV_64F) == 4 && dst.checkVector(2, CV_64F) == 4);
return getPerspectiveTransform64((const Point2d*)src.data, (const Point2d*)dst.data, solveMethod);
}
This can now be used directly without additional conversion:
transMat = getPerspectiveTransform64(quad, warpedQuad);

convert octave Matrix to cv::Mat in oct file

I wrote a simple Oct file to wrap an OpenCV function. This is my code:-
#include <octave/oct.h>
#include <opencv2/imgproc.hpp>
DEFUN_DLD (cornerHarris, args, , "Harris Corner Detector")
{
// Processing arguments
if(args.length()<4){
print_usage();
}
Matrix octInMat = args(0).matrix_value();
int blockSize = args(1).int_value();
int kSize = args(2).int_value();
double k = args(3).double_value();
int borderType = args(4).int_value();
// Dimentions
dim_vector dims = octInMat.dims();
int h = dims.elem(0);
int w = dims.elem(1);
// OpenCV Matrix
cv::Mat cvInMat = cv::Mat::zeros(h,w, CV_8U);
cv::Mat cvOutMat = cv::Mat::zeros(h,w, CV_32FC1);
// Converting Octave Matrix to OpenCV Matrix
for (int r=0;r<h;r++)
{
for(int s=0;s<w;s++)
{
cvInMat.at<int>(r,s) = octInMat(r,s);
}
}
cv::cornerHarris( cvInMat, cvOutMat, blockSize, kSize, k, borderType );
// Converting OpenCV Matrix to Octave Matrix
Matrix octOutMat = Matrix(dim_vector(h,w));
for (int r=0;r<h;r++)
{
for(int s=0;s<w;s++)
{
octOutMat(r,s) = cvOutMat.at<double>(r,s);
}
}
return octave_value(octOutMat);
}
But I am getting a segmentation error when the value of w variable increased. Is there any short way to convert the matrices without looping? Or is there a way to resolve the segmentation error?
Documentations:-
octave::Matrix
cv::Mat
I figured it out by commenting line by line in my code. The issue was occurred from this line because of a type casting issue.
cvInMat.at<int>(r,s) = octInMat(r,s);
I changed this as following.
cvInMat.at<uchar>(r,s) = (uchar)octInMat(r,s);
This answer helped me to fix it.

Convolution using FFT gives a bad result

I'm trying to convolve an image using FFT. I use openCV so images are in Mat containers. I convert color image to gray image, then add a second channel for imaginary numbers that is all zero. Then I take this 2-channel Mat and convolve it with Prewitt's kernel. I get a result very different from the result I get when I use normal convolution algorithm. Left image is the output I get using FFT and right image is the output of normal convolution.
Below is the pseudo algorithm of how I do the operation;
Convert image Mat and kernel Mat to complex Mats by adding second channel (Result Mat type is CV_32FC2)
Assign all Mat elements to complex vectors
Zero pad the vectors to the same next power of 2
FFT the vectors
Signal multiply both vectors elementwise and assign result to result vector
Inverse FFT the result vector
Convert result vector to Mat
I think FFT algorithm is not the problem because when I take an image, FFT it, then inverse FFT it, I get the original image just fine. But I could be wrong. So here is the FFT algorithm. Notice how there are two of them. I use the second one. I also tried other FFT algorithms and they all output the same. FFT'ing and IFFT'ing same image only skips the signal multiplication step above. So I think that's where the problem is. Here is the code of the operation;
std::vector<cf> signalMultiplication(std::vector<cf> lh, std::vector<cf> rh) {
std::vector<cf> imVec = lh, kerVec = rh, resultVec;
resultVec.resize(imVec.size());
std::transform(imVec.begin(), imVec.end(), kerVec.begin(), resultVec.begin(), std::multiplies<cf>());
return resultVec;
}
I tried multiplying them using for loop but result was the same. I don't know the problem and I can't type the whole code here since it is too long, so tell me where you think the problem is and I'll give the code of that part.
#Paul below is the main body of the code;
cv::Mat convolution2D(cv::Mat image, cv::Mat kernel) {
cv::Mat imMat, kerMat;
imMat = convertToComplexMat(image);
kerMat = convertToComplexMat(kernel);
std::vector<cf> imVec, kerVec, resultVec;
imVec = matElementsToVector<cf>(imMat);
kerVec = matElementsToVector<cf>(kerMat);
float power = log2f(imVec.size());
if (abs(power - (int)power) == 0)
power++;
else
power = ceil(power);
zeroPadding(imVec, power);
zeroPadding(kerVec, power);
//FFT code I linked takes valarray as argument so I convert vectors to valarray and back
std::valarray<cf> imCArr(imVec.data(), imVec.size());
std::valarray<cf> kerCArr(kerVec.data(), kerVec.size());
fftRosetta(imCArr);
fftRosetta(kerCArr);
imVec.assign(std::begin(imCArr), std::end(imCArr));
kerVec.assign(std::begin(kerCArr), std::end(kerCArr));
resultVec = signalMultiplication(imVec, kerVec);
std::valarray<cf> resCArr(resultVec.data(), resultVec.size());
ifftRosetta(resCArr);
resultVec.assign(std::begin(resCArr), std::end(resCArr));
cv::Mat resultMat;
resultMat = vectorToMatElementsRowMajor(resultVec, imMat.rows, imMat.cols, imMat.type());
std::vector<cv::Mat> matVec;
cv::split(resultMat, matVec);
return matVec[0]; }
These are the custom functions;
convertToComplexMat, matElementsToVector, zeroPadding, fftRosetta, ifftRosetta, signalMultiplication, vectorToMatElementsRowMajor
signalMultiplication is posted, fftRosetta and ifftRosetta are linked so here, the rest of the functions;
using cf = std::complex<float>;
cv::Mat convertToComplexMat(cv::Mat imageMat) {
cv::Mat matOper;
if (imageMat.channels() == 3)
cv::cvtColor(imageMat, matOper, cv::COLOR_BGR2GRAY);
else
matOper = imageMat.clone();
matOper.convertTo(matOper, CV_32FC1);
cv::Mat compChannel = cv::Mat::zeros(matOper.rows, matOper.cols, CV_32FC1);
std::vector<cv::Mat> channels;
channels.push_back(matOper);
channels.push_back(compChannel);
cv::merge(channels, matOper);
return matOper;
}
template <typename T>
std::vector<T> matElementsToVector(cv::Mat operand) {
std::vector<T> vecOper;
int cn = operand.channels();
int lele = operand.total();
for (int i = 0; i < operand.total(); i++) {
if (cn == 1)
vecOper.push_back(operand.at<cv::Vec<T, 1>>(i)[0]);
else if (cn == 2) {
if (typeid(T) == typeid(cf)) {
T xd = operand.at<T>(i);
vecOper.push_back(xd);
}
else
for (int k = 0; k < cn; k++)
vecOper.push_back(operand.at<cv::Vec<T, 2>>(i)[k]);
}
else if (cn == 3)
for (int k = 0; k < cn; k++)
vecOper.push_back(operand.at<cv::Vec<T,3>>(i)[k]);
}
return vecOper;
}
void zeroPadding(std::vector<cf>& a, int power) {
int p, ioper;
if (power == -1)
p = ceil(log2f(a.size()));
else
p = power;
ioper = pow(2, p);
int size = a.size();
for (int i = 0; i < ioper - size; i++) {
a.push_back(0);
}
}
template <typename T>
cv::Mat vectorToMatElementsRowMajor(std::vector<T> operand, int mrows, int mcols, int mtype) {
cv::Mat matoper(mrows, mcols, mtype);
for (int j = 0; j < matoper.total(); j++) {
matoper.at<T>(j) = operand[j];
}
return matoper;
}
#Cris I tried it again with openCV DFT like you said, following the directions here. I applied DFT to image and kernel, then element-wise multiplied them, then applied IDFT. But result is something very different now. I can see resemblance of original image in there, but there are multiple shadows of it in different angles. I think the problem is how I do signal multiplication, but I can't find any answers on how to multiply 2D signals. Here is the code, output image is below it;
cv::Mat convolution2DopenCV(cv::Mat image, cv::Mat kernel) {
cv::Mat paddedImage, paddedKernel, imgOper, kerOper;
if (image.channels() == 3)
cv::cvtColor(image, imgOper, cv::COLOR_BGR2GRAY);
else
imgOper = image.clone();
kerOper = kernel;
int m = cv::getOptimalDFTSize(imgOper.rows);
int n = cv::getOptimalDFTSize(imgOper.cols);
cv::copyMakeBorder(imgOper, paddedImage, 0, m - imgOper.rows, 0, n - imgOper.cols, cv::BORDER_CONSTANT, cv::Scalar::all(0));
cv::copyMakeBorder(kerOper, paddedKernel, 0, m - kerOper.rows, 0, n - kerOper.cols, cv::BORDER_CONSTANT, cv::Scalar::all(0));
cv::Mat planesImage[] = { cv::Mat_<float>(paddedImage), cv::Mat::zeros(paddedImage.size(), CV_32F) };
cv::Mat cmpImgMat;
cv::merge(planesImage, 2, cmpImgMat);
cv::dft(cmpImgMat, cmpImgMat);
cv::Mat planesKernel[] = { cv::Mat_<float>(paddedKernel), cv::Mat::zeros(paddedKernel.size(), CV_32F) };
cv::Mat cmpKerMat;
cv::merge(planesKernel, 2, cmpKerMat);
cv::dft(cmpKerMat, cmpKerMat);
cv::Mat resultMat = cmpImgMat.mul(cmpKerMat);
cv::Mat planes[2];
cv::idft(resultMat, resultMat);
cv::split(resultMat, planes);
cv::normalize(planes[0], planes[0], 0, 255, cv::NORM_MINMAX);
return planes[0];
}
That's everything, if there is something I'm missing, let me know.

OpenCV Mat Problem: Difference between Histogram and this loop

i am working on image processing project that i want to implement it on cuda with opencv (opencv 4.0 with cuda suport)and i am not good at c++.
for color correction between two images, i am using code from this link: (https://answers.opencv.org/question/178127/matching-colors-between-two-pictures-in-opencv/)
my goal is to implement this code on GPU. for that i tried to rewrite that code. i faced two problems:
1- Is there any Cuda implemented library for this purpose? (Same Functionality)
2- in rewriting function ((do1ChnHist)), it seams that this loop calculates 1D histogram (Is that true?) :
for (size_t p = 0; p<img.total(); p++)
{
if (mask(p) > 0)
{
uchar c = img(p);
h(c) += 1.0;
}
}
but i can't replace it with :
int histSize = 256;
float range[] = { 0, 256 }; //the upper boundary is exclusive
const float* histRange = { range };
bool uniform = false, accumulate = false;
calcHist(&img, 1, 0, Mat(), h, 1, &histSize, &histRange, uniform, accumulate);
or rewrite it with this loop (For changing Mat >> GpuMat in future. unfortunately Opencv_cuda does not support GpuMat_<>, due to that i tried to rewrite loop with Mat):
Mat h;
h = Mat::zeros(cv::Size(256, 1), CV_16U);
uchar x;
for (size_t m = 0; m < img.size().width; m++)
{
for (size_t n = 0; n < img.size().width; n++)
{
x = img.at<int>(Point(m, n));
h.at<int>(Point(int(x),0)) += 1;
}
}
because ether of two options return different answer from main loop in do1ChnHist function...
thanks...
Opencv has all the function u want
virtual void cv::cuda::TemplateMatching::match ( InputArray image,
InputArray templ,
OutputArray result,
Stream & stream = Stream::Null()
)
void cv::cuda::calcHist (InputArray src, OutputArray hist, Stream &stream=Stream::Null())
Calculates histogram for one channel 8-bit image. More...
void cv::cuda::calcHist (InputArray src, InputArray mask, OutputArray hist, Stream &stream=Stream::Null())
Calculates histogram for one channel 8-bit image confined in given mask. More...
depends, could be 1D array, and could be 2D array, depends on color. You should learn some basic image processing principle first.

Exception thrown for cvCalcOpticalFlowHS() opencv

Heey, I'm trying to sort out the function of Optical Flow of openCV, but for some reason I'm getting an exception in visual studio:
Unhandled exception at 0x772615de in Optical_flow.exe: Microsoft C++ exception: cv::Exception at memory location 0x0036f334..
With breakpoints I found out that the error lies within the cvCalcOpticalFlowHS function.
I'm using openCV 2.1
#include <cv.h>
#include <highgui.h>
using namespace cv;
int init() {
return 0;
}
int main(int argc, char **args) {
CvCapture* capture = cvCaptureFromFile("Video/Wildlife.wmv");
double fps = cvGetCaptureProperty(capture, CV_CAP_PROP_FPS);
CvSize size;
size.width = (int)cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH);
size.height = (int)cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT);
CvVideoWriter* writer = cvCreateVideoWriter("result.avi", 0, fps,size, 1);
IplImage* curFrame = cvQueryFrame(capture);
Mat u = Mat(size, CV_32FC2);
Mat v = Mat(size, CV_32FC2);
CvTermCriteria IterCriteria;
IterCriteria.type = CV_TERMCRIT_ITER | CV_TERMCRIT_EPS;
IterCriteria.max_iter = 500;
IterCriteria.epsilon = 0.01;
while(1) {
IplImage* nextFrame = cvQueryFrame(capture);
if(!nextFrame) break;
u = Mat::zeros(size, CV_32FC2);
v = Mat::zeros(size, CV_32FC2);
/* Do optical flow computation */
cvCalcOpticalFlowHS(&curFrame, &nextFrame, 0, &u, &v, 0.01, IterCriteria);
cvWriteFrame(writer, curFrame);
curFrame = nextFrame;
}
cvReleaseVideoWriter(&writer);
cvReleaseCapture(&capture);
return 0;
}
Anyone has seen this problem before or sees the mistake I made?
Best Regards
Remco
From the documentation, curFrame and nextFrame should be 8-bit single channel. You are currently just pulling these from the loaded file without checking/converting them as necessary. Can you confirm that the input is of the right type?
Also you have a nasty mix of C++ style cv::Mat with C style IplImage*. I'd suggest you upgrade to a more recent version of OpenCV (2.4 has recently been released), and try to stick with the one or other of the C++ or C style methods.
Note also that this optical flow method is classed as obsolete with a recommendation to use either calcOpticalFlowPyrLK() for sparse features or calcOpticalFlowFarneback() for dense features.
Below is some example code demonstrating calcOpticalFlowFarneback(), which is what I believe you are trying to achieve. It takes data from the webcam rather than a file.
#include <opencv2/opencv.hpp>
using namespace cv;
void drawOptFlowMap(const cv::Mat& flow,
cv::Mat& cflowmap,
int step,
const cv::Scalar& color
)
{
for(int y = 0; y < cflowmap.rows; y += step)
for(int x = 0; x < cflowmap.cols; x += step)
{
const cv::Point2f& fxy = flow.at<cv::Point2f>(y, x);
cv::line(cflowmap,
cv::Point(x,y),
cv::Point(cvRound(x+fxy.x),cvRound(y+fxy.y)),
color);
cv::circle(cflowmap, cv::Point(x,y), 2, color, -1);
}
}
int main(int argc, char **args) {
VideoCapture cap(0); // open the default camera
if(!cap.isOpened()) // check if we succeeded
return -1;
Mat newFrame, newGray, prevGray;
cap >> newFrame; // get a new frame from camera
cvtColor(newFrame, newGray, CV_BGR2GRAY);
prevGray = newGray.clone();
double pyr_scale = 0.5;
int levels = 3;
int winsize = 5;
int iterations = 5;
int poly_n = 5;
double poly_sigma = 1.1;
int flags = 0;
while(1) {
cap >> newFrame;
if(newFrame.empty()) break;
cvtColor(newFrame, newGray, CV_BGR2GRAY);
Mat flow = Mat(newGray.size(), CV_32FC2);
/* Do optical flow computation */
calcOpticalFlowFarneback(
prevGray,
newGray,
flow,
pyr_scale,
levels,
winsize,
iterations,
poly_n,
poly_sigma,
flags
);
drawOptFlowMap(flow, newFrame, 20, CV_RGB(0,255,0));
namedWindow("Output",1);
imshow("Output", newFrame);
waitKey(1);
prevGray = newGray.clone();
}
return 0;
}
The above code is pretty similar to the fback.cpp sample code which comes with OpenCV.