I've been playing around with opencv2 implemented in C++ for a couple of days and noticed that the lookup tables are the fastest way to apply changes to an image. However, I've been having some troubles with using them for my purposes.
The code below shows an example of inverting pixels' values:
bool apply(Image& img) {
int dim(256);
Mat lut(1, &dim, CV_8U);
for (int i=0; i<256; i++)
lut.at<uchar>(i)= 255-i;
LUT(img.final,lut,img.final);
return true;
}
class Image {
public:
const Mat& original;
Mat final;
...
};
As it's very efficient, much more efficient than changing each pixel by one (verified by my own tests), I'd like to use this method for other operations. However to do this, I have to access each layer (each color, the picture is in BGR) separately. So for example, I'd like to change blue to 255-i, green to 255-i/2 and red to 255-i/3.
I've been searching the net for a while, but couldn't come up with a correct solution. As far as I know, it's possible (documentation) but I can't find a way to implement it.
The key is this paragraph in the docs:
the table should either have a single channel (in this case the same table is used for all channels) or the same number of channels as in the source array
So, you must create a multichannel LUT:
bool apply(Image& img) {
int dim(256);
Mat lut(1, &dim, CV_8UC(img.final.channels()));
if( img.final.channels() == 1)
{
for (int i=0; i<256; i++)
lut.at<uchar>(i)= 255-i;
}
else // stupid idea that all the images are either mono either multichannel
{
for (int i=0; i<256; i++)
{
lut.at<Vec3b>(i)[0]= 255-i; // first channel (B)
lut.at<Vec3b>(i)[1]= 255-i/2; // second channel (G)
lut.at<Vec3b>(i)[2]= 255-i/3; // ... (R)
}
}
LUT(img.final,lut,img.final); // are you sure you are doing final->final?
// if yes, correct the LUT allocation part
return true;
}
Related
I have a very large Mat which is actually a ROI of another Mat (obtained by otherMat(cv::Rect(x,y,w,h))). I want to go through all the pixels of the Mat, do some pixelwise computation and write the result to another Mat by using a pointer.
Going through all pixels, including the ones outside the ROI is working fine so far, but I am wondering what the fastest way of skipping the pixels outside the ROI is. I want to have as few cache misses as possible and also I don't want to have an inefficient branch prediction. What would be the best way to go about this?
Edit: I am not interested in getting a submatrix for a specifitc region of interest. I am interested in iterating through the pixel by pointer in an maximally efficient way without accessing data outside the submatrix' region.
Just use a submatrix:
cv::Mat largeMat
cv::Rect roi(yourROI);
cv::Mat submatrix = largeMat(roi);
// now iterate over all the pixels of submatrix
you will have cache misses at the end of each row
Here's the actual code example which shows, that the pixels outside of the submat are skipped (you'll get an additional cache miss at the end of each row but that should be all).
int main(int argc, char* argv[])
{
cv::Mat input = cv::imread("C:/StackOverflow/Input/Lenna.png");
cv::Rect roi(128, 128, 256, 256);
cv::Mat submat = input(roi);
cv::MatIterator_<cv::Vec3b> it; // = src_it.begin<cv::Vec3b>();
for (it = submat.begin<cv::Vec3b>(); it != submat.end<cv::Vec3b>(); ++it)
{
(*it)[0] = 0;
(*it)[1] = 0;
}
cv::imshow("input", input);
cv::imwrite("C:/StackOverflow/Output/submatIter.png", input);
cv::waitKey(0);
return 0;
}
giving this result:
If you want it a little faster you can use row-Pointers: http://docs.opencv.org/2.4/doc/tutorials/core/how_to_scan_images/how_to_scan_images.html
Please mind, that in the link they compared debug mode runtime speed, that's why the random access is so slow. In release mode it should be as fast (or maybe faster) than the iterator verson.
But here's the row-Ptr version (which spares to compute the row-offset on each pixel access) which gives the same result and should be the fastest way (if openCV's LUT function can't be used for your task):
int main(int argc, char* argv[])
{
cv::Mat input = cv::imread("C:/StackOverflow/Input/Lenna.png");
cv::Rect roi(128, 128, 256, 256);
cv::Mat submat = input(roi);
cv::Vec3b * currentRow;
for (int j = 0; j < submat.rows; ++j)
{
currentRow = submat.ptr<cv::Vec3b>(j);
for (int i = 0; i < submat.cols; ++i)
{
currentRow[i][0] = 0;
currentRow[i][1] = 0;
}
}
cv::imshow("input", input);
cv::imwrite("C:/StackOverflow/Output/submatIter.png", input);
cv::waitKey(0);
return 0;
}
As the OtherMat is a subset of Original mat and you want to do operation over the original mat but only inside the otherMat Region
//As otherMat(cv::Rect(x,y,w,h)));
for(int j=x;j<x+w;j++)
{
for (int i=y;i<y+h;i++)
{
original.at<uchar>(j,i) = 255;
}
}
cv::Mat in = cv::imread("SegmentedImage.png");
// vector with all non-white point positions
std::vector<Point> nonWhiteList;
nonWhiteList.reserve(in.rows*in.cols);
// add all non-white points to the vector
for(int j=0; j< in.rows; ++j)
{
for(int i=0; i<in.cols; ++i)
{
// if not white: add to the list
if(in.at<cv::Vec3b>(j,i) != cv::Vec3b(255,255,255))
{
nonWhiteList.push_back(cv::Point(i,j));
}
}
}
cv::Mat BKGR = imread("photo_booth_Cars.png", CV_LOAD_IMAGE_COLOR); //1529x736
I need to write the vector<Point> nonWhiteList to image BKGR, How to do it?
Basically, need to remove the white background from the image and put non-white points on another background image. Researched very much on grabcut and findcontours.
I am completely new to Opencv. Thanks so much for help.
cv::Mat BKGR = imread("photo_booth_Cars.png", CV_LOAD_IMAGE_COLOR); //1529x736
for(int j=0; j<in.rows; ++j)
for(int i=0; i<in.cols; ++i)
{
if(in.at<cv::Vec3b>(j,i) != cv::Vec3b(255,255,255))
{
BKGR.at<cv::Vec3b>(j,i) = in.at<cv::Vec3b>(j,i);
}
}
cv::imwrite("newFinalImage.png", BKGR);
If the images are of same dimensionality than it makes sense else it is difficult to copy directly unless you know the 2 image camera parameters. However, you may use Interpolation to get the two images to same size.
If the images are same size, than why do you need to create a std::vector than copy it to another cv::Mat. You can achieve this in the same loop without extra computation overhead. Just simple as filling an array. However your question is ambiguous.
I am having some issues with my sobel_y (and sobel_x, but I figure they are having the same issue) filter in that it keeps giving me an image that it basically only black and white. I am having to rewrite this function for a class, so no I cannot use the built-in one, and had it working, minus some minor tweaks because the output image looked a little strange with still being black and white even though it was supposed to be converted back. I figured out how to fix that, and in the process I messed with something and broke it and cannot seem to get it back to working even with the black and white image output only. I keep getting a black image, with some white lines here and there near the top. I have tried changing the Mat grayscale type (third parameter) to all different values, as my professor mentioned in the class that we are using 32 bit floating point images, but that did not help either.
Even though the issue occurs after running the Studentfilter2D, I think it is a problem with the grayscaling of the image, although whenever I debug, it seems to work just fine. This is also because I have 2 other filtering functions I had to write that use Studentfilter2D, and they both give me the expected results. My sobel_y function is shown below:
// Convert the image in bgr to grayscale OK to use the OpenCV function.
// Find the coefficients used by the OpenCV function, and give a link where you found it.
// Note: This student function expects the matrix gray to be preallocated with the same width and
// height, but with 1 channel.
void BGR2Gray(Mat& bgr, Mat& gray)
{
// Y = .299 * R + .587 * G + .114 * B, from http://docs.opencv.org/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor
// Some extra assistance, for the third parameter for the InputArray, from http://docs.opencv.org/trunk/modules/core/doc/basic_structures.html#inputarray
// Not sure about the fourth parameter, but was just trying it to see if that may be the issue as well
cvtColor(bgr, gray, CV_BGR2GRAY, 1);
return;
}
// Convolve image with kernel - this routine will be called from the other
// subroutines! (gaussian, sobel_x and sobel_y)
// image is single channel. Do not use the OpenCV filter2D!!
// Implementation can be with the .at or similar to the
// basic method found in the Chapter 2 of the OpenCV tutorial in CANVAS,
// or online at the OpenCV documentation here:
// http://docs.opencv.org/doc/tutorials/core/mat-mask-operations/mat-mask operations.html
// In our code the image and the kernel are both floats (so the sample code will need to change)
void Studentfilter2D (Mat& image, Mat& kernel)
{
int kCenterX = kernel.cols / 2;
int kCenterY = kernel.rows / 2;
// Algorithm help from http://www.songho.ca/dsp/convolution/convolution.html
for (int iRows = 0; iRows < image.rows; iRows++)
{
for (int iCols = 0; iCols < image.cols; iCols++)
{
float result = 0.0;
for (int kRows = 0; kRows < kernel.rows; kRows++)
{
// Flip the rows for the convolution
int kRowsFlipped = kernel.rows - 1 - kRows;
for (int kCols = 0; kCols < kernel.cols; kCols++)
{
// Flip the columns for the convolution
int kColsFlipped = kernel.cols - 1 - kCols;
// Indices of shifting around the convolution
int iRowsIndex = iRows + kRows - kCenterY;
int iColsIndex = iCols + kCols - kCenterX;
// Check bounds using the indices
if (iRowsIndex >= 0 && iRowsIndex < image.rows && iColsIndex >= 0 && iColsIndex < image.cols)
{
result += image.at<float>(iRowsIndex, iColsIndex) * kernel.at<float>(kRowsFlipped, kColsFlipped);
}
}
}
image.at<float>(iRows, iCols) = result;
}
}
return;
}
void sobel_y (Mat& image, int)
{
// Note, the filter parameter int is unused.
Mat mask = (Mat_<float>(3, 3) << 1, 2, 1,
0, 0, 0,
-1, -2, -1) / 3;
//Mat grayscale(image.rows, image.cols, CV_32FC1);
BGR2Gray(image, image);
Studentfilter2D(image, mask);
// Here is the documentation on normalize http://docs.opencv.org/modules/core/doc/operations_on_arrays.html#normalize
normalize(image, image, 0, 1, NORM_MINMAX);
cvtColor(image, image, CV_GRAY2BGR);
return;
}
Like I said, I had this working before, just looking for some fresh eyes to look at it and see what I may be missing. I have been looking at this same code so much for the past 4 days that I think I am just missing things. In case anyone is wondering, I have also tried changing the mask values of the filter, but to no avail.
There are two things that are worth mentioning.
The first is that you are not taking proper care of the type of your matrices/images.
The input to Studentfilter2D in sobel_y is an 8-bit grayscale image of type CV_8UC1 meaning that the data is an array of unsigned char.
Your Studentfilter2D function, however, is indexing this input image as though it was of type float. This means it is picking the wrong pixels to work with.
If the above does not immediately solve your problem, you should consider the range of your final derivative image. Since it is a derivative it will no longer be in the range [0, 255]. Instead, it might even contain negative numbers. When you try to visualize this, you will run into problems unless you first normalize your image.
There are built in functions to do this in OpenCV if you look around in the documentation.
I'm learning Neural Networks from this bytefish machine learning guide and code. I understand it well but I would like to update the code at the previous link to use image pixel data instead of random values as the input data. In this section of the aforementioned code:
cv::randu(trainingData,0,1);
cv::randu(testData,0,1);
the training and test matrices are filled with random data. Then label data is added to the classes matrices here:
cv::Mat trainingClasses = labelData(trainingData, eq);
cv::Mat testClasses = labelData(testData, eq);
using this function:
// label data with equation
cv::Mat labelData(cv::Mat points, int equation) {
cv::Mat labels(points.rows, 1, CV_32FC1);
for(int i = 0; i < points.rows; i++) {
float x = points.at<float>(i,0);
float y = points.at<float>(i,1);
labels.at<float>(i, 0) = f(x, y, equation);
// the f() function used above
//is only a case statement with 5
//switches in it eg on of the switches is:
//case 0:
//return y > sin(x*10) ? -1 : 1;
//break;
}
return labels;
}
Then points are plotted in a window here:
plot_binary(trainingData, trainingClasses, "Training Data");
plot_binary(testData, testClasses, "Test Data");
with this function:
;; Plot Data and Class function
void plot_binary(cv::Mat& data, cv::Mat& classes, string name) {
cv::Mat plot(size, size, CV_8UC3);
plot.setTo(cv::Scalar(255.0,255.0,255.0));
for(int i = 0; i < data.rows; i++) {
float x = data.at<float>(i,0) * size;
float y = data.at<float>(i,1) * size;
if(classes.at<float>(i, 0) > 0) {
cv::circle(plot, Point(x,y), 2, CV_RGB(255,0,0),1);
} else {
cv::circle(plot, Point(x,y), 2, CV_RGB(0,255,0),1);
}
}
imshow(name, plot);
}
The plotted points, as I understand it, represent the input data multiplied by the equations in the f() function and is used by the predict functions to predict which point to plot in the mlp, knn, svm etc. functions. How do I update what is going on here to do something with Image pixel data. Any advice to get me farther would be appreciated.
"How do I update what is going on here to do something with Image pixel data" is a broad and generic question. May I ask in exchange: what do you want to do with "Image pixel data"?
Do you want an answer to: what can be done with "Image pixel data" on machine learning algorithms like ANN, SVM etc. ?
The answer is a loooong list of things encompassing thousands of research papers and hundreds of PhD theses. Some examples include: supervised and/or un-supervised classification of images into labels/tags/categories based on features like image content, objects in image, patterns in image etc. The possibilities are endless. You may perhaps want to take a look at this: http://stuff.mit.edu/afs/athena/course/urop/profit/PDFS/EdwardTolson.pdf
Now, coming back to you original objective: "I would like to update the code at the previous link to use image pixel data instead of random values as the input data"...
The implementation technique would depend largely on what you want to do. I can cite one/two easy techniques for extracting feature vectors from image, which can be fed into any machine learning algorithm of your choice...
Example 1:
You may start with using pixel intensity data as a feature vector. Here's how you may go ahead with it:
Load image using
Mat image = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE);
Resize image into a smaller area using resize. You may want to begin with small image sizes, like 8x8 or 10x10 pixels.
Loop through the image matrix, somewhat like this:
for(int row = 0; row < img.rows; ++row)
{
uchar* p = img.ptr(row);
for(int col = 0; col < img.cols; ++col)
{
*p++ //points to each pixel value in turn assuming a CV_8UC1 greyscale image
}
}
A collection of all the pixel values will give you a feature vector for that image.
Now suppose you have two classes of image. For each set of feature vector you generate, you'll have to prepare (for supervised classification) a corresponding label Mat (somewhat like the example you've mentioned). It needs to contain the class label (say, 0 and 1) for all the feature vectors present in your feature Mat.
Now feed the feature vectors and label Mat to your machine learning code and see what happens.
However, the ability of image classification based on image pixel data alone is quite limited. There are thousands of techniques for extracting image features, most of which are dependent on the application area.
Example 2:
I'll finish off with one more example for extracting feature vectors, which, in some cases, will prove to be more effective than simple image pixel values.
You may use the Histogram of Oriented Gradients descriptor for slightly better results, use this:
cv::HOGDescriptor hog;
vector<float> descriptors;
hog.compute(mat, descriptors);
The vector descriptors is your feature vector.
HOGDescriptors, when used with SVM, provides a decent classification mechanism.
You can put the pixel data of an image into a Mat called trainingData using something similar to this:
cv::Mat labelData(cv::Mat points, int equation)
{
cv::Mat labels(points.rows, 1, CV_32FC1);
for(int i = 0; i < points.rows; i++)
{
float x = points.at<float>(i,0);
float y = points.at<float>(i,1);
labels.at<float>(i, 0) = f(x, y, equation);
}
return labels;
}
Now, instead of labelData, we're going to return a Mat of pixel data. One obvious way is to use the image itself as a feature vector. However, some machine learning algorithms in openCV, including ANN, SVM etc., required special formatting of input data.
You may try something like this:
cv::Mat trainingData(cv::Mat image)
{
cv::Mat trainingVector(image.rows*image.cols, 1, CV_32FC1);
for(int i = 0; i < image.rows; i++)
{
for(int j = 0; j < image.cols; j++)
{
float valueOfPixel = image.at<float>(i,j);
trainingVector.at<float>((i*image.cols)+j, 0) = valueOfPixel;
}
}
return trainingVector;
}
(Please recheck the syntax of the code before using, I just typed it out here)
So, what the above block effectively does is change the 2D matrix of the image into a 1D array. Now, how and where you use it depends on your requirements.
Please make necessary modifications before invoking the machine learning modules.
Hope this answers your question.
Thanks.
I am trying to pass in a HSV frame from a video to the function, but the function does not seem to do anything to it. What am I doing wrong? The function is supposed to go through each pixel, and depending on its hue range supposed to make it black or white, leaving me with a binary image. Instead it doesn't seem to affect the HSV image at all....
Thanks
PS sorry for the bad code formatting, StackOverflow isn't allowing me to post the original format.
void sort (IplImage *skinmask)
{
for (int row=0; row<=skinmask->height;row++)
{
uchar* pixelrow=(uchar*)(skinmask->imageData+(row*(skinmask->widthStep)));
for (int column=0; column<=skinmask->width; column++)
{
if (6<pixelrow[3*column]<36)
{
pixelrow[3*column]=256;
pixelrow[(3*column)+1]=256;
pixelrow[(3*column)+2]=256;
}
else
{
pixelrow[3*column]=0;
pixelrow[(3*column)+1]=0;
pixelrow[(3*column)+2]=0;
}
column++;
}
row++;
}
cvMorphologyEx(skinmask,skinmask,NULL,NULL,CV_MOP_CLOSE,1);
}
Doing an operation like thresholding pixel-by-pixel is usually the wrong way to go about achieving this in OpenCV - there are functions that work on whole image arrays that are simpler and are already optimized for speed.
In this case try first splitting the image to separate out the H/S/V channels, then threshold on the Hue channel to get a mask (you may have to use the intersection of two masks, which you can do using a multiply or "bitwise and") - the resulting mask is your black and white image.
(I realise I've linked to the C++ documentation, but I'm sure you can find the equivalent functions in the old-style OpenCV docs)
Update
Ok, I'll try to write some code to show what I mean. I also found the function I was looking for, which is better than Threshold, it is InRangeS. This lets you put upper and lower bounds on all the channels at once, and it applies them all into your mask for you.
void HSVImageToMask(IplImage * image, cvMat * mask)
/* mask should be the same size as image, and of type CV_8UC1 */
/* e.g. cvMat * mask = cvCreateMat(image->width, image->height, CV8UC1); */
{
double hMin = 6;
double hMax = 36;
double sMin = 10; /* not sure what value you need */
double sMax = 245; /* not sure what value you need */
double vMin = 0;
double vMax = 255;
CvScalar hsvMin = cvScalar(hMin, sMin, vMin);
CvScalar hsvMax = cvScalar(hMax, sMax, vMax);
cvInRangeS(image, hsvMin, hsvMax, mask);
}
PS. I figured out the problem with your original code - you should be using 255 instead of 256 as your "white" value. This method is still better though :)
PPS. We didn't need them after all but for future reference:
"bitwise and":
cvAnd(const CvArr* src1, const CvArr* src2, CvArr* dst)
If you have two black and white masks, this will give you the intersection. Use cvOr to get the Union.