OpenCV Mat Problem: Difference between Histogram and this loop - c++

i am working on image processing project that i want to implement it on cuda with opencv (opencv 4.0 with cuda suport)and i am not good at c++.
for color correction between two images, i am using code from this link: (https://answers.opencv.org/question/178127/matching-colors-between-two-pictures-in-opencv/)
my goal is to implement this code on GPU. for that i tried to rewrite that code. i faced two problems:
1- Is there any Cuda implemented library for this purpose? (Same Functionality)
2- in rewriting function ((do1ChnHist)), it seams that this loop calculates 1D histogram (Is that true?) :
for (size_t p = 0; p<img.total(); p++)
{
if (mask(p) > 0)
{
uchar c = img(p);
h(c) += 1.0;
}
}
but i can't replace it with :
int histSize = 256;
float range[] = { 0, 256 }; //the upper boundary is exclusive
const float* histRange = { range };
bool uniform = false, accumulate = false;
calcHist(&img, 1, 0, Mat(), h, 1, &histSize, &histRange, uniform, accumulate);
or rewrite it with this loop (For changing Mat >> GpuMat in future. unfortunately Opencv_cuda does not support GpuMat_<>, due to that i tried to rewrite loop with Mat):
Mat h;
h = Mat::zeros(cv::Size(256, 1), CV_16U);
uchar x;
for (size_t m = 0; m < img.size().width; m++)
{
for (size_t n = 0; n < img.size().width; n++)
{
x = img.at<int>(Point(m, n));
h.at<int>(Point(int(x),0)) += 1;
}
}
because ether of two options return different answer from main loop in do1ChnHist function...
thanks...

Opencv has all the function u want
virtual void cv::cuda::TemplateMatching::match ( InputArray image,
InputArray templ,
OutputArray result,
Stream & stream = Stream::Null()
)
void cv::cuda::calcHist (InputArray src, OutputArray hist, Stream &stream=Stream::Null())
Calculates histogram for one channel 8-bit image. More...
void cv::cuda::calcHist (InputArray src, InputArray mask, OutputArray hist, Stream &stream=Stream::Null())
Calculates histogram for one channel 8-bit image confined in given mask. More...
depends, could be 1D array, and could be 2D array, depends on color. You should learn some basic image processing principle first.

Related

Convolution using FFT gives a bad result

I'm trying to convolve an image using FFT. I use openCV so images are in Mat containers. I convert color image to gray image, then add a second channel for imaginary numbers that is all zero. Then I take this 2-channel Mat and convolve it with Prewitt's kernel. I get a result very different from the result I get when I use normal convolution algorithm. Left image is the output I get using FFT and right image is the output of normal convolution.
Below is the pseudo algorithm of how I do the operation;
Convert image Mat and kernel Mat to complex Mats by adding second channel (Result Mat type is CV_32FC2)
Assign all Mat elements to complex vectors
Zero pad the vectors to the same next power of 2
FFT the vectors
Signal multiply both vectors elementwise and assign result to result vector
Inverse FFT the result vector
Convert result vector to Mat
I think FFT algorithm is not the problem because when I take an image, FFT it, then inverse FFT it, I get the original image just fine. But I could be wrong. So here is the FFT algorithm. Notice how there are two of them. I use the second one. I also tried other FFT algorithms and they all output the same. FFT'ing and IFFT'ing same image only skips the signal multiplication step above. So I think that's where the problem is. Here is the code of the operation;
std::vector<cf> signalMultiplication(std::vector<cf> lh, std::vector<cf> rh) {
std::vector<cf> imVec = lh, kerVec = rh, resultVec;
resultVec.resize(imVec.size());
std::transform(imVec.begin(), imVec.end(), kerVec.begin(), resultVec.begin(), std::multiplies<cf>());
return resultVec;
}
I tried multiplying them using for loop but result was the same. I don't know the problem and I can't type the whole code here since it is too long, so tell me where you think the problem is and I'll give the code of that part.
#Paul below is the main body of the code;
cv::Mat convolution2D(cv::Mat image, cv::Mat kernel) {
cv::Mat imMat, kerMat;
imMat = convertToComplexMat(image);
kerMat = convertToComplexMat(kernel);
std::vector<cf> imVec, kerVec, resultVec;
imVec = matElementsToVector<cf>(imMat);
kerVec = matElementsToVector<cf>(kerMat);
float power = log2f(imVec.size());
if (abs(power - (int)power) == 0)
power++;
else
power = ceil(power);
zeroPadding(imVec, power);
zeroPadding(kerVec, power);
//FFT code I linked takes valarray as argument so I convert vectors to valarray and back
std::valarray<cf> imCArr(imVec.data(), imVec.size());
std::valarray<cf> kerCArr(kerVec.data(), kerVec.size());
fftRosetta(imCArr);
fftRosetta(kerCArr);
imVec.assign(std::begin(imCArr), std::end(imCArr));
kerVec.assign(std::begin(kerCArr), std::end(kerCArr));
resultVec = signalMultiplication(imVec, kerVec);
std::valarray<cf> resCArr(resultVec.data(), resultVec.size());
ifftRosetta(resCArr);
resultVec.assign(std::begin(resCArr), std::end(resCArr));
cv::Mat resultMat;
resultMat = vectorToMatElementsRowMajor(resultVec, imMat.rows, imMat.cols, imMat.type());
std::vector<cv::Mat> matVec;
cv::split(resultMat, matVec);
return matVec[0]; }
These are the custom functions;
convertToComplexMat, matElementsToVector, zeroPadding, fftRosetta, ifftRosetta, signalMultiplication, vectorToMatElementsRowMajor
signalMultiplication is posted, fftRosetta and ifftRosetta are linked so here, the rest of the functions;
using cf = std::complex<float>;
cv::Mat convertToComplexMat(cv::Mat imageMat) {
cv::Mat matOper;
if (imageMat.channels() == 3)
cv::cvtColor(imageMat, matOper, cv::COLOR_BGR2GRAY);
else
matOper = imageMat.clone();
matOper.convertTo(matOper, CV_32FC1);
cv::Mat compChannel = cv::Mat::zeros(matOper.rows, matOper.cols, CV_32FC1);
std::vector<cv::Mat> channels;
channels.push_back(matOper);
channels.push_back(compChannel);
cv::merge(channels, matOper);
return matOper;
}
template <typename T>
std::vector<T> matElementsToVector(cv::Mat operand) {
std::vector<T> vecOper;
int cn = operand.channels();
int lele = operand.total();
for (int i = 0; i < operand.total(); i++) {
if (cn == 1)
vecOper.push_back(operand.at<cv::Vec<T, 1>>(i)[0]);
else if (cn == 2) {
if (typeid(T) == typeid(cf)) {
T xd = operand.at<T>(i);
vecOper.push_back(xd);
}
else
for (int k = 0; k < cn; k++)
vecOper.push_back(operand.at<cv::Vec<T, 2>>(i)[k]);
}
else if (cn == 3)
for (int k = 0; k < cn; k++)
vecOper.push_back(operand.at<cv::Vec<T,3>>(i)[k]);
}
return vecOper;
}
void zeroPadding(std::vector<cf>& a, int power) {
int p, ioper;
if (power == -1)
p = ceil(log2f(a.size()));
else
p = power;
ioper = pow(2, p);
int size = a.size();
for (int i = 0; i < ioper - size; i++) {
a.push_back(0);
}
}
template <typename T>
cv::Mat vectorToMatElementsRowMajor(std::vector<T> operand, int mrows, int mcols, int mtype) {
cv::Mat matoper(mrows, mcols, mtype);
for (int j = 0; j < matoper.total(); j++) {
matoper.at<T>(j) = operand[j];
}
return matoper;
}
#Cris I tried it again with openCV DFT like you said, following the directions here. I applied DFT to image and kernel, then element-wise multiplied them, then applied IDFT. But result is something very different now. I can see resemblance of original image in there, but there are multiple shadows of it in different angles. I think the problem is how I do signal multiplication, but I can't find any answers on how to multiply 2D signals. Here is the code, output image is below it;
cv::Mat convolution2DopenCV(cv::Mat image, cv::Mat kernel) {
cv::Mat paddedImage, paddedKernel, imgOper, kerOper;
if (image.channels() == 3)
cv::cvtColor(image, imgOper, cv::COLOR_BGR2GRAY);
else
imgOper = image.clone();
kerOper = kernel;
int m = cv::getOptimalDFTSize(imgOper.rows);
int n = cv::getOptimalDFTSize(imgOper.cols);
cv::copyMakeBorder(imgOper, paddedImage, 0, m - imgOper.rows, 0, n - imgOper.cols, cv::BORDER_CONSTANT, cv::Scalar::all(0));
cv::copyMakeBorder(kerOper, paddedKernel, 0, m - kerOper.rows, 0, n - kerOper.cols, cv::BORDER_CONSTANT, cv::Scalar::all(0));
cv::Mat planesImage[] = { cv::Mat_<float>(paddedImage), cv::Mat::zeros(paddedImage.size(), CV_32F) };
cv::Mat cmpImgMat;
cv::merge(planesImage, 2, cmpImgMat);
cv::dft(cmpImgMat, cmpImgMat);
cv::Mat planesKernel[] = { cv::Mat_<float>(paddedKernel), cv::Mat::zeros(paddedKernel.size(), CV_32F) };
cv::Mat cmpKerMat;
cv::merge(planesKernel, 2, cmpKerMat);
cv::dft(cmpKerMat, cmpKerMat);
cv::Mat resultMat = cmpImgMat.mul(cmpKerMat);
cv::Mat planes[2];
cv::idft(resultMat, resultMat);
cv::split(resultMat, planes);
cv::normalize(planes[0], planes[0], 0, 255, cv::NORM_MINMAX);
return planes[0];
}
That's everything, if there is something I'm missing, let me know.

Matrix assignement value error in opencv C++ with mat.at<uchar>(i,j)

I am learning image processing with OpenCV in C++. To implement a basic down-sampling algorithm I need to work on the pixel level -to remove rows and columns. However, when I assign values with mat.at<>(i,j) other values are assign - things like 1e-38.
Here is the code :
Mat src, dst;
src = imread("diw3.jpg", CV_32F);//src is a 479x359 grayscale image
//dst will contain src low-pass-filtered I checked by displaying it works fine
Mat kernel;
kernel = Mat::ones(3, 3, CV_32F) / (float)(9);
filter2D(src, dst, -1, kernel, Point(-1, -1), 0, BORDER_DEFAULT);
// Now I try to remove half the rows/columns result is stored in downsampled
Mat downsampled = Mat::zeros(240, 180, CV_32F);
for (int i =0; i<downsampled.rows; i ++){
for (int j=0; j<downsampled.cols; j ++){
downsampled.at<uchar>(i,j) = dst.at<uchar>(2*i,2*j);
}
}
Since I read here OpenCV outputing odd pixel values that for cout I needed to cast, I wrote downsampled.at<uchar>(i,j) = (int) before dst.at<uchar> but it does not work also.
The second argument to cv::imread is cv::ImreadModes, so the line:
src = imread("diw3.jpg", CV_32F);
is not correct; it should probably be:
cv::Mat src_8u = imread("diw3.jpg", cv::IMREAD_GRAYSCALE);
src_8u.convertTo(src, CV_32FC1);
which will read the image as 8-bit grayscale image, and will convert it to floating point values.
The loop should look something like this:
Mat downsampled = Mat::zeros(240, 180, CV_32FC1);
for (int i = 0; i < downsampled.rows; i++) {
for (int j = 0; j < downsampled.cols; j++) {
downsampled.at<float>(i,j) = dst.at<float>(2*i,2*j);
}
}
note that the argument to cv::Mat::zeros is CV_32FC1 (1 channel, with 32-bit floating values), so Mat::at<float> method should be used.

Why do these histogram functions differ, and why is one nondeterministic?

NOTE: This is a homework problem and the professor explicitly forbids soliciting answers from StackOverflow, so please limit your response to the specific question I have asked and do not attempt to provide a working solution.
I am asked to implement a function that computes the histogram of a single-channel 8-bit image represented as an OpenCV Mat with type CV_U8.
In this case, the histogram uses 256 uniformly-distributed buckets. This is the reference we are intended to replicate (using OpenCV 3.4):
Mat reference;
/// Establish the number of bins
int histSize = 256;
/// Set the ranges ( for B,G,R) )
float range[] = { 0, 256 } ;
const float* histRange = { range };
bool uniform = true;
bool accumulate = false;
cv::calcHist(&bgr_planes[0], 1, 0, Mat(), reference, 1, &histSize, &histRange,
uniform, accumulate);
// reference now contains the canonical histogram of the input image's
// blue channel
I wrote the following function to calculate the histogram, which produces the correct results 45-69% of the time (p<0.05, n=66). Once when it failed, I examined the results and found no discernable pattern. All trials were conducted on the same test image.
Mat myCalcHist(const Mat& input) {
assert(input.isContinuous());
Mat res(256, 1, CV_32F);
for (const uint8_t* it = input.datastart; it != input.dataend; ++it) {
++res.at<float>(*it);
}
return res;
}
The following function, on the other hand, more closely matches OpenCV's internal implementation in that it uses the idiomatic accessors and converts the float result from an int work matrix, but in n=66 trials it did not produce the correct result a single time. Again, I found no discernable pattern in the data.
Mat myCalcHist(const Mat& input) {
Mat ires(256, 1, CV_32S);
for (int i = 0; i < input.total(); ++i) {
++ires.at<int>(input.at<uint8_t>(i));
}
Mat res(256, 1, CV_32F);
ires.convertTo(res, CV_32F);
return res;
}
Why are the results for my first implementation different than those from my second implementation, and where is nondeterminism introduced to the first implementation?
initializing the histogram matrix should work:
Mat myCalcHist(const Mat& input)
{
Mat ires = cv::Mat::zeros(256, 1, CV_32S);
for (int i = 0; i < input.total(); ++i)
{
++ires.at<int>(input.at<uint8_t>(i));
}
Mat res(256, 1, CV_32F);
ires.convertTo(res, CV_32F);
return res;
}

EXC_BAD_ACCESS error OpenCV?

I'm trying write a code for image segmentation in OpenCV. As a part of the image processing, I'm trying to detect the edges of a test image using Sobel filter.
In order to find the magnitude of gradient on both dX and dY direction, I'm computing the euclidean distance of both the gradients. But when I run the code I get the above error. I do know that the above error occurs when I am trying "ACCESS" an unavailable location in memory, but I am sure I have defined all Mat in my code.
This is part of my code.
//Blur the raw image to remove noise
GaussianBlur(src, src, kernel, 2);
//Run sobel edge detector
Sobel(src, edgeX, src.depth(), 1, 0);
Sobel(src, edgeY, src.depth(), 0, 1);
edge = Mat::zeros(317,554,CV_8UC1);
for (int r = 0; r < edgeX.rows; r++)
{
for (int c = 0; c < edgeY.cols; c++)
{
edge.at<double>(r,c) = sqrt((edgeX.at<double>(r,c)*edgeX.at<double>(r,c)) + (edgeY.at<double>(r,c)*edgeY.at<double>(r,c)));
}
}
where:
src: the RGB test image
edgeX: sobel output with dX gradient
edgeY: sobel output with dY
edge: is the Mat with the euclidean distance.
I get the error at this line
edge.at<double>(r,c) = sqrt((edgeX.at<double>(r,c)*edgeX.at<double>(r,c)) + (edgeY.at<double>(r,c)*edgeY.at<double>(r,c)));
when trying to access edge.at<double>(316,395)
How do I debug this?
What am I doing wrong?
edge is a matrix of type CV_8UC1, which means a matrix of uchar, not of double.
You need to access it with at<uchar>:
edge.at<uchar>(r,c) = sqrt((edgeX.at<uchar>(r,c)*edgeX.at<uchar>(r,c)) + (edgeY.at<uchar>(r,c)*edgeY.at<uchar>(r,c)));
You can avoid this kind of problems using Mat_<Tp>, that allows also easier access without using the .at function:
Mat1b edge(317,554,uchar(0));
for (int r = 0; r < edgeX.rows; r++) {
for (int c = 0; c < edgeY.cols; c++) {
edge(r,c) = sqrt((edgeX(r,c)*edgeX(r,c)) + (edgeY(r,c)*edgeY(r,c)));
}
}
In this case, you can also use cv::magnitude which performs the same operation you're doing with your for loops (but it needs matrices of float):
Sobel(src, edgeX, CV_32F, 1, 0);
Sobel(src, edgeY, CV_32F, 0, 1);
Mat edge;
magnitude(edgeX, edgeY, edge);
// Convert to CV_8UC1
edge.convertTo(edge, CV_8UC1);

OpenCV: color extraction based on Gaussian mixture model

I am trying to use opencv EM algorithm to do color extraction.I am using the following code based on example in opencv documentation:
cv::Mat capturedFrame ( height, width, CV_8UC3 );
int i, j;
int nsamples = 1000;
cv::Mat samples ( nsamples, 2, CV_32FC1 );
cv::Mat labels;
cv::Mat img = cv::Mat::zeros ( height, height, CV_8UC3 );
img = capturedFrame;
cv::Mat sample ( 1, 2, CV_32FC1 );
CvEM em_model;
CvEMParams params;
samples = samples.reshape ( 2, 0 );
for ( i = 0; i < N; i++ )
{
//from the training samples
cv::Mat samples_part = samples.rowRange ( i*nsamples/N, (i+1)*nsamples/N);
cv::Scalar mean (((i%N)+1)*img.rows/(N1+1),((i/N1)+1)*img.rows/(N1+1));
cv::Scalar sigma (30,30);
cv::randn(samples_part,mean,sigma);
}
samples = samples.reshape ( 1, 0 );
//initialize model parameters
params.covs = NULL;
params.means = NULL;
params.weights = NULL;
params.probs = NULL;
params.nclusters = N;
params.cov_mat_type = CvEM::COV_MAT_SPHERICAL;
params.start_step = CvEM::START_AUTO_STEP;
params.term_crit.max_iter = 300;
params.term_crit.epsilon = 0.1;
params.term_crit.type = CV_TERMCRIT_ITER|CV_TERMCRIT_EPS;
//cluster the data
em_model.train ( samples, Mat(), params, &labels );
cv::Mat probs;
probs = em_model.getProbs();
cv::Mat weights;
weights = em_model.getWeights();
cv::Mat modelIndex = cv::Mat::zeros ( img.rows, img.cols, CV_8UC3 );
for ( i = 0; i < img.rows; i ++ )
{
for ( j = 0; j < img.cols; j ++ )
{
sample.at<float>(0) = (float)j;
sample.at<float>(1) = (float)i;
int response = cvRound ( em_model.predict ( sample ) );
modelIndex.data [ modelIndex.cols*i + j] = response;
}
}
My question here is:
Firstly, I want to extract each model, here totally five, then store those corresponding pixel values in five different matrix. In this case, I could have five different colors seperately. Here I only obtained their indexes, is there any way to achieve their corresponding colors here? To make it easy, I can start from finding the dominant color based on these five GMMs.
Secondly, here my sample datapoints are "100", and it takes about nearly 3 seconds for them. But I want to do all these things in no more than 30 milliseconds. I know OpenCV background extraction, which is using GMM, performs really fast, below 20ms, that means, there must be a way for me to do all these within 30 ms for all 600x800=480000 pixels. I found predict function is the most time consuming one.
First Question:
In order to do color extraction you first need to train the EM with your input pixels. After that you simply loop over all the input pixels again and use predict() to classify each of them. I've attached a small example that utilizes EM for foreground/background separation based on colors. It shows you how to extract the dominant color (mean) of each gaussian and how to access the original pixel color.
#include <opencv2/opencv.hpp>
int main(int argc, char** argv) {
cv::Mat source = cv::imread("test.jpg");
//ouput images
cv::Mat meanImg(source.rows, source.cols, CV_32FC3);
cv::Mat fgImg(source.rows, source.cols, CV_8UC3);
cv::Mat bgImg(source.rows, source.cols, CV_8UC3);
//convert the input image to float
cv::Mat floatSource;
source.convertTo(floatSource, CV_32F);
//now convert the float image to column vector
cv::Mat samples(source.rows * source.cols, 3, CV_32FC1);
int idx = 0;
for (int y = 0; y < source.rows; y++) {
cv::Vec3f* row = floatSource.ptr<cv::Vec3f > (y);
for (int x = 0; x < source.cols; x++) {
samples.at<cv::Vec3f > (idx++, 0) = row[x];
}
}
//we need just 2 clusters
cv::EMParams params(2);
cv::ExpectationMaximization em(samples, cv::Mat(), params);
//the two dominating colors
cv::Mat means = em.getMeans();
//the weights of the two dominant colors
cv::Mat weights = em.getWeights();
//we define the foreground as the dominant color with the largest weight
const int fgId = weights.at<float>(0) > weights.at<float>(1) ? 0 : 1;
//now classify each of the source pixels
idx = 0;
for (int y = 0; y < source.rows; y++) {
for (int x = 0; x < source.cols; x++) {
//classify
const int result = cvRound(em.predict(samples.row(idx++), NULL));
//get the according mean (dominant color)
const double* ps = means.ptr<double>(result, 0);
//set the according mean value to the mean image
float* pd = meanImg.ptr<float>(y, x);
//float images need to be in [0..1] range
pd[0] = ps[0] / 255.0;
pd[1] = ps[1] / 255.0;
pd[2] = ps[2] / 255.0;
//set either foreground or background
if (result == fgId) {
fgImg.at<cv::Point3_<uchar> >(y, x, 0) = source.at<cv::Point3_<uchar> >(y, x, 0);
} else {
bgImg.at<cv::Point3_<uchar> >(y, x, 0) = source.at<cv::Point3_<uchar> >(y, x, 0);
}
}
}
cv::imshow("Means", meanImg);
cv::imshow("Foreground", fgImg);
cv::imshow("Background", bgImg);
cv::waitKey(0);
return 0;
}
I've tested the code with the following image and it performs quite good.
Second Question:
I've noticed that the maximum number of clusters has a huge impact on the performance. So it's better to set this to a very conservative value instead of leaving it empty or setting it to the number of samples like in your example. Furthermore the documentation mentions an iterative procedure to repeatedly optimize the model with less-constrained parameters. Maybe this gives you some speed-up. To read more please have a look at the docs inside the sample code that is provided for train() here.