I am trying to project an image to eigenface convariance matrix that EigenFacesRecognizer of opencv returns. I use the following code to load eigenfaces parameters loading an image and trying to project the sample image to pca subspace.
Ptr<FaceRecognizer> model = createEigenFaceRecognizer();
model->load("eigenfaces.yml"); // Load eigenfaces parameters
Mat eigenvalues = model->getMat("eigenvalues"); // Eigen values of PCA
Mat convMat = model->getMat("eigenvectors"); //Convariance matrix
Mat mean = model->getMat("mean"); // Mean value
string path = fileName;
Mat sample ,pca_ed_sample;
sample = imread(path, CV_LOAD_IMAGE_GRAYSCALE); //size 60x60
Mat nu = sample.reshape(1,3600 ).t(); //1x3600
pca_ed_sample = (nu - mean)*(convMat);
I am keeping 5 eigenvectors, so the size of eigenvalues 5x1, convMat3600x5 mean 1x3600. When I am trying to calculate pca_ed_sample it returns me:
cv::Exception at memory location 0x0011d300.Dimensionality reduction using default opencv eigenfaces...
OpenCV Error: Assertion failed (type == B.type() && (type == CV_32FC1 || type ==
CV_64FC1 || type == CV_32FC2 || type == CV_64FC2)) in unknown function, file .\
src\matmul.cpp, line 711`
The problem stands in nu Mat since when I trying to calculate nu*.nu.t();(1x3600* 3600x1) it returns the same issue. Am I having troubles due to reshape function?? I am trying to transform my sample mat to a vector, it seems to work but I cant understand why I cant even multiply nu with nu_transposed.
Matrix multiplication is only possible with floating point data, which is what the assertion error is trying to tell you.
Your image is loaded as type CV_8U, and you must first rescale that to float using the convertTo member.
sample = imread(path, CV_LOAD_IMAGE_GRAYSCALE); //size 60x60
cv::Mat sample_float;
sample.convertTo(sample_float, CV_32F); // <-- Convert to CV_32F for matrix mult
Mat nu = sample_float.reshape(1,3600 ).t(); //1x3600
pca_ed_sample = (nu - mean)*(convMat);
Related
I am loading a pre-trained TensorFlow model in the opencv dnn module using the following code -
cv::dnn::Net net = cv::dnn::readNetFromTensorflow("frozen_inference_graph.pb",
"graph.pbtxt");
net.setPreferableBackend(cv::dnn::DNN_BACKEND_CUDA); //Run model on GPU
net.setPreferableTarget(cv::dnn::DNN_TARGET_CUDA);
Mat image = imread("img.jpeg");
Mat resized;
cv::resize(image, resized, cv::Size(300, 300));
cout<<resized.size()<<endl;
cout<<"Resized"<<endl;
auto input_image = dnn::blobFromImage(image, 1.0, cv::Size(300, 300),
cv::Scalar(127.5, 127.5, 127.5),
false, false, CV_32F);
cout<<"Now setting Input";
net.setInput(input_image);
auto detections = net.forward();
cout<<detections;
return 0;
However the I get the following error as mentioned in the question -
what(): OpenCV(4.4.0) /home/atharva/opencv-4.4.0/modules/core/src/out.cpp:87: error: (-215:Assertion failed) m.dims <= 2 in function 'FormattedImpl'
Could someone please point out what the mistake is?. I believe there is some problem in BlobFromImage as nothing after it is getting printed.
TIA
This error occurs because you are trying to print a cv::Mat to standard output that has more than 2 dimensions. With cv::dnn, the output after using net.forward() is 4-dimensional. However I have no idea what model you are using because the output structure of the blob is different depending on what task you are trying to do. If I had to guess you are doing some sort of object detection given your choice of variable names. In that case, usually the first dimension is the batch size and since you are using only one image, the batch size is 1. The second dimension is the number of channels in the output. As you're doing object detection on the image, this will also be size 1. The third and fourth dimensions are the number of rows and columns for the final output layer.
Going on faith, you can extract a 2D version of this cv::Mat to print out to standard output by doing:
cv::Mat output(detections.size[2], detections.size[3], CV_32F, detection.ptr<float>());
Now that this is a 2D matrix, you can print out this instead by doing std::cout << output << std::endl;.
I'm trying to convert an OpenCV 3-channel Mat to a 3D Eigen Tensor.
So far, I can convert 1-channel grayscale Mat by:
cv::Mat mat = cv::imread("/image/path.png", cv::IMREAD_GRAYSCALE);
Eigen::MatrixXd myMatrix;
cv::cv2eigen(mat, myMatrix);
My attempt to convert a BGR mat to a Tensor have been:
cv::Mat mat = cv::imread("/image/path.png", cv::IMREAD_COLOR);
Eigen::MatrixXd temp;
cv::cv2eigen(mat, temp);
Eigen::Tensor<double, 3> myTensor = Eigen::TensorMap<Eigen::Tensor<double, 3>>(temp.data(), 3, mat.rows, mat.cols);
However, I'm getting the following error :
libc++abi.dylib: terminating with uncaught exception of type cv::Exception: OpenCV(4.1.0) /tmp/opencv-20190505-12101-14vk1fh/opencv-4.1.0/modules/core/src/matrix_wrap.cpp:1195:
error: (-215:Assertion failed) !fixedType() || ((Mat*)obj)->type() == mtype in function 'create'
in the line: cv::cv2eigen(mat, temp);
Any help is appreciated!
The answer might be disappointing for you.
After going through 12 pages, My conclusion is you have to separate the RGB to individual channel MAT and then convert to eigenmatrix. Or create your own Eigen type and opencv convert function
In OpenCV it is tested like this. It only allows a single channel greyscale image
https://github.com/daviddoria/Examples/blob/master/c%2B%2B/OpenCV/ConvertToEigen/ConvertToEigen.cxx
And in OpenCV it is implemented like this. Which dont give you much option for custom type aka cv::scalar to eigen std::vector
https://github.com/stonier/opencv2/blob/master/modules/core/include/opencv2/core/eigen.hpp
And according to this post,
https://stackoverflow.com/questions/32277887/using-eigen-array-of-arrays-for-rgb-images
I think Eigen was not meant to be used in this way (with vectors as
"scalar" types).
they also have the difficulting in dealing with RGB image in eigen.
Take note that Opencv Scalar and eigen Scalar has a different meaning
It is possible to do so if and only if you use your own datatype aka matrix
So you either choose to store the 3 channel info in 3 eigen matrix and you can use default eigen and opencv routing.
Mat src = imread("img.png",CV_LOAD_IMAGE_COLOR); //load image
Mat bgr[3]; //destination array
split(src,bgr);//split source
//Note: OpenCV uses BGR color order
imshow("blue.png",bgr[0]); //blue channel
imshow("green.png",bgr[1]); //green channel
imshow("red.png",bgr[2]); //red channel
Eigen::MatrixXd bm,gm,rm;
cv::cv2eigen(bgr[0], bm);
cv::cv2eigen(bgr[1], gm);
cv::cv2eigen(bgr[2], rm);
Or you can define your own type and write you own version of the opencv cv2eigen function
custom eigen type follow this. and it wont be pretty
https://eigen.tuxfamily.org/dox/TopicCustomizing_CustomScalar.html
https://eigen.tuxfamily.org/dox/TopicNewExpressionType.html
Rewrite your own cv2eigen_custom function similar to this
https://github.com/stonier/opencv2/blob/master/modules/core/include/opencv2/core/eigen.hpp
So good luck.
Edit
Since you need tensor. forget about cv function
Mat image;
image = imread(argv[1], CV_LOAD_IMAGE_COLOR);
Tensor<float, 3> t_3d(image.rows, image.cols, 3);
// t_3d(i, j, k) where i is row j is column and k is channel.
for (int i = 0; i < image.rows; i++)
for (int j = 0; j < image.cols; j++)
{
t_3d(i, j, 0) = (float)image.at<cv::Vec3b>(i,j)[0];
t_3d(i, j, 1) = (float)image.at<cv::Vec3b>(i,j)[1];
t_3d(i, j, 2) = (float)image.at<cv::Vec3b>(i,j)[2];
//cv ref Mat.at<data_Type>(row_num, col_num)
}
watch out for i,j as em not sure about the order. I only write the code based on reference. didnt compile for it.
Also watch out for image type to tensor type cast problem. Some times you might not get what you wanted.
this code should in principle solve your problem
Edit number 2
following the example of this
int storage[128]; // 2 x 4 x 2 x 8 = 128
TensorMap<Tensor<int, 4>> t_4d(storage, 2, 4, 2, 8);
Applied to your case is
cv::Mat frame=imread('myimg.ppm');
TensorMap<Tensor<float, 3>> t_3d(frame.data, image.rows, image.cols, 3);
problem is I'm not sure this will work or not. Even it works, you still have to figure out how the inside data is being organized so that you can get the shape correctly. Good luck
Updated answer - OpenCV now has conversion functions for Eigen::Tensor which will solve your problem. I needed this same functionality too so I made a contribution back to the project for everyone to use. See the documentation here:
https://docs.opencv.org/3.4/d0/daf/group__core__eigen.html
Note: if you want RGB order, you will still need to reorder the channels in OpenCV before converting to Eigen::Tensor
I'm trying to implement color conversion from RGB-LMS and LMS-RGB back and using reshape for multiplication matrix, following answer from this question : Fastest way to apply color matrix to RGB image using OpenCV 3.0?
My ori Mat object is from an image with 3 channel (RGB), and I need to multiply them with matrix of 1 channel (lms), it seems like I have an issue with the matrix type. I've read reshape docs and questions related to this issue, like Issues multiplying Mat matrices, and I believe I have followed the instructions.
Here's my code : [UPDATED : Convert into flat image]
void test(const Mat &forreshape, Mat &output, Mat &pic, int rows, int cols)
{
Mat lms(3, 3, CV_32FC3);
Mat rgb(3, 3, CV_32FC3);
Mat intolms(rows, cols, CV_32F);
lms = (Mat_<float>(3, 3) << 1.4671, 0.1843, 0.0030,
3.8671, 27.1554, 3.4557,
4.1194, 45.5161 , 17.884 );
/* switch the order of the matrix according to the BGR order of color on OpenCV */
Mat transpose = (3, 3, CV_32F, lms).t(); // this will do transpose from matrix lms
pic = forreshape.reshape(1, rows*cols);
Mat flatFloatImage;
pic.convertTo(flatFloatImage, CV_32F);
rgb = flatFloatImag*transpose;
output = rgb.reshape(3, cols);
}
I define my Mat object, and I have converted it into float using convertTo
Mat ori = imread("ori.png", CV_LOAD_IMAGE_COLOR);
int rows = ori.rows;
int cols = ori.cols;
Mat forreshape;
ori.convertTo(forreshape, CV_32F);
Mat pic(rows, cols, CV_32FC3);
Mat output(rows, cols, CV_32FC3);
Error is :
OpenCV Error: Assertion failed (type == B.type() && (type == CV_32FC1 || type == CV_64FC1 || type == CV_32FC2 || type == CV_64FC2)) ,
so it's the type issue.
I tried to change all type into either 32FC3 of 32FC1, but doesn't seem to work. Any suggestion ?
I believe what you need is to convert your input to a flat image and than multiply them
float lms [] = {1.4671, 0.1843, 0.0030,
3.8671, 27.1554, 3.4557,
4.1194, 45.5161 , 17.884};
Mat lmsMat(3, 3, CV_32F, lms );
Mat flatImage = ori.reshape(1, ori.rows * ori.cols);
Mat flatFloatImage;
flatImage.convertTo(flatFloatImage, CV_32F);
Mat mixedImage = flatFloatImage * lmsMat;
Mat output = mixedImage.reshape(3, imData.rows);
I might have messed up with lms matrix there, but I guess you will catch up from here.
Also see 3D matrix multiplication in opencv for RGB color mixing
EDIT:
Problem with distortion is that you got overflow after float to 8U conversion. This would do the trick:
rgb = flatFloatImage*transpose;
rgb.convertTo(pic, CV_32S);
output = pic.reshape(3, rows)
Output:
;
Also I'm not sure but quick google search gives me different matrix for LMS see here. Also note that opencv stores colors in B-G-R format instead of RGB so change your mix mtraixes recordingly.
I'm trying to make a vector matrix, which is the same as MNIST image dataset.
Each image from the webcam is captured and store into the vector. However the matrix i created is different from the MNIST dataset. So the main code doesn't work for matrix I created.
I was thinking that maybe its because the pixel type is different.
What I noticed is, when I looked up a single matrix from MNIST data it had 15 decimal points. However I was not able to set 15 decimal points. When I set the image to be CV_FC64. It shows following error message.
"Assertion failed in cv::cvtColor, file C:\file path. "
The main code works for MNIST dataset.. I'm not sure what to do..
please advice. me.
while (1)
{
cap >> src;
src.convertTo(src, CV_64FC1);
src = src / 256;
cvtColor(src, src_gray, CV_RGB2GRAY);
resize(src_gray, src_N, size);
testX.push_back(src_N);
}
cvtColor only allows 8U, 16U and 32F bit-depths. So after you convertTo(..., CV_64FC1), the bit-depth is 64F and the assertion fails: https://github.com/opencv/opencv/blob/84699e0e1860a3485e3dfc12230fbded955dba13/modules/imgproc/src/color.cpp#L8676:
CV_Assert( depth == CV_8U || depth == CV_16U || depth == CV_32F );
If you really need 64F, it'd make sense to first cvtColor and then increase bit-depth to 64F using convertTo.
As an aside: Apologies if I'm flooding SO with OpenCV questions :p
I'm currently trying to port over my old C code to use the new C++ interface and I've got to the point where I'm rebuilidng my Eigenfaces face recogniser class.
Mat img = imread("1.jpg");
Mat img2 = imread("2.jpg");
FaceDetector* detect = new HaarDetector("haarcascade_frontalface_alt2.xml");
// convert to grey scale
Mat g_img, g_img2;
cvtColor(img, g_img, CV_BGR2GRAY);
cvtColor(img2, g_img2, CV_BGR2GRAY);
// find the faces in the images
Rect r = detect->getFace(g_img);
Mat img_roi = g_img(r);
r = detect->getFace(g_img2);
Mat img2_roi = g_img2(r);
// create the data matrix for PCA
Mat data;
data.create(2,1, img2_roi.type());
data.row(0) = img_roi;
data.row(1) = img2_roi;
// perform PCA
Mat averageFace;
PCA pca(data, averageFace, CV_PCA_DATA_AS_ROW, 2);
//namedWindow("avg",1); imshow("avg", averageFace); - causes segfault
//namedWindow("avg",1); imshow("avg", Mat(pca.mean)); - doesn't work
I'm trying to create the PCA space, and then see if it's working by displaying the computed average image. Are there any other steps to this?
Perhaps I need to project the images onto the PCA subspace first of all?
Your error is probably here:
Mat data;
data.create(2,1, img2_roi.type());
data.row(0) = img_roi;
data.row(1) = img2_roi;
PCA expects a matrix with the data vectors as rows. However, you never scale the images to the same size so that they have the same number of pixels (so the dimension is the same), also data.create(2,1,...) - the 1 needs to be the dimension of your vector, i.e. the number of your pixels. Then copy the pixels from the crop to your matrix.