I am doing a real-time shapes and colors classification system with very high accuracy. It seems like my preprocessing phase is not good enough so that the result is not as accurate as I expected. Here is what I'm doing:
Take data from the Camera can crop it to receive ROI.
Convert ROI Image from RGB to HSV space.
Using a median filter to reduce noise in HSV image.
Threshold the image
Using dilate and erode to remove small holes and small objects in Image
Using findContours and approxPolyDP to detect square objects.
This is my preprocessing phase:
image_cv = cv::cvarrToMat(image_camera);
Mat cropped = image_cv(cv::Rect(0, 190, 640, 110));
imshow("origin", cropped);
Mat croppedCon = CropConveyor(cropped);
cv::cvtColor(croppedCon, croppedCon, CV_RGB2HSV);
medianBlur(croppedCon, croppedCon, 3);
cv::Mat binRect;
cv::inRange(croppedCon, Scalar(iLowH, iLowS, iLowV), Scalar(iHighH, iHighS, iHighV), binRect);
This is the code for detecting squares:
vector<vector<Point>> contours;
findContours(binarizedIm, contours, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE);
vector<Point> approx;
for (size_t i = 0; i < contours.size(); i++)
{
//double arclength = arcLength(Mat(contours[i]), true);
approxPolyDP(Mat(contours[i]), approx, 3.245 , true); //0.04 for wood
if (approx.size() != 4) continue;
if (isContourConvex(Mat(approx)) && contourArea(Mat(approx)) > 250)
{
double MaxCos = 0;
for (int j = 2; j < 5; j++)
{
double cos = angle(approx[j % 4], approx[j - 1], approx[j - 2]);
MaxCos = MAX(cos, MaxCos);
}
if (MaxCos < 0.2)
squares.push_back(approx);
}
}
I think noise in HSV Image is the main reason. Here is some images illustrating my problems. I saw a lot of noise in HSV Image, that's why I use a media filter to it to reduce noise but preserve the edges becase I think that edges information is very important when using findContours function.
HSV and HSV in separate channels
My question is:
What is the noise in HSV Image, refer to the above Image, how can I
enhance my Image's quality?
The reason for noise in your saturation image is noise in your input image. Caused by a bad camera / optics and further increased by JPEG compression.
That's by far the worst image I have seen in years. You shouldn't invest another second into processing that, unless you live on Mars and need results tomorrow.
Your input image is super noisy, undersampled, defocussed, underexposed, full of aliasing and compression artifacts and pretty much anything else you can do wrong with an image.
First rule of signal processing:
crap in = crap out
You can get much better cameras basically for free. Find and use one.
Part of the problem is that you're doing the noise reduction in HSV space. In your example you can see the V channel is better-behaved than H and S. It would be better to do noise-reduction in RGB (which is more linear and closer, though not identical, to the camera's native colour space where the noise originates; of course there's also gamma-correction).
Maybe consider a stronger edge-preserving noise-reducing filter such as Bilateral Filter.
I don't get it why are you using HSV for segmenting the objects, the RGB image is good enough. Separate the image into 3 channels (r,g,b) and apply an adaptive threshold on them. dilate and erode the images then add (not merging) those 3 binary images to have one binary image. Finally do level 6 of your recipe to extract the objects. If the noise still effects the result, apply a bilateral filter on r,g,b channels before the threshold.
Related
According to this post OCR: Difference between two frames, I now know how to find pixel differences between two images with OpenCV.
I would like to improve this solution and use it with high resolution images (from a video) with rich content. The example above is not applicable with big images because the process is to slow (too much differences found, the "findCountours method" fills the tab with 250k elements which takes a huge time to process).
My application uses a RLE decoder to decode the compressed frames of the video. Once the frame is decoded, I would like to compare the current frame with the previous one in order to store the differences between the two frames in a "Mat" tab for example.
The goal of all of this is to be able to perform an analysis on the different pixels and to check if there is any latin character. This allows me to reduce the amount of pixels to analyze and to save precious time.
If anyone has other ideas instead of this one to perform such operations, feel free to propose it please.
Thank you for your help.
EDIT 1:
Example of two high resolution images of a computer screen. These are for the moment the perfect example of what I'm trying to analyse. As we can see there is just a window as difference between the two big images and I would like to analyze just the new "Challenge" window for any character.
EDIT 2:
I'm trying to tune the algorithm depending on the data analyzed. Typically on the two following pictures I only get the green lines as differences and no text at all (which is what is the most interesting). I'm trying to understand better how things work for this.
1st image:
2nd image:
3rd image:
As you can see I only have those green lines and never the text (at the best I can have just ONE letter when decreasing the countours[i].size())
In addition to the post you mentioned, you need to:
When you binarize the mask, use a threshold higher then 0 to remove small differences.
Remove some noise. You can find all connected components, and remove smaller ones.
Find the area of the bigger connected components. You can use convexHull and fillConvexPoly to get the mask of the different objects on screen
Copy the second image to a new image, with the given mask.
The result will look like:
Code:
#include <opencv2/opencv.hpp>
#include <vector>
using namespace std;
using namespace cv;
int main()
{
Mat3b img1 = imread("path_to_image_1");
Mat3b img2 = imread("path_to_image_2");
Mat3b diff;
absdiff(img1, img2, diff);
// Split each channel
vector<Mat1b> masks;
split(diff, masks);
// Create a black mask
Mat1b mask(diff.rows, diff.cols, uchar(0));
// OR with each channel of the N channels mask
for (int i = 0; i < masks.size(); ++i)
{
mask |= masks[i];
}
// Binarize mask
mask = mask > 100;
// Results images
vector<Mat3b> difference_images;
// Remove small blobs
//Mat kernel = getStructuringElement(MORPH_RECT, Size(5,5));
//morphologyEx(mask, mask, MORPH_OPEN, kernel);
// Find connected components
vector<vector<Point>> contours;
findContours(mask.clone(), contours, CV_RETR_EXTERNAL, CHAIN_APPROX_NONE);
for (int i = 0; i < contours.size(); ++i)
{
if (contours[i].size() > 1000)
{
Mat1b mm(mask.rows, mask.cols, uchar(0));
vector<Point> hull;
convexHull(contours[i], hull);
fillConvexPoly(mm, hull, Scalar(255));
Mat3b difference_img(img2.rows, img2.cols, Vec3b(0,0,0));
img2.copyTo(difference_img, mm);
difference_images.push_back(difference_img.clone());
}
}
return 0;
}
Is there any algorithm, which can remove outliers, but do not blur other part of image?
Only for example, when we use cv::StereoBM/SBGM or cv::gpu::StereoConstantSpaceBP from opencv, then we can have outliers, as shown in relevant question: opencv sgbm produces outliers on object edges Also, we can get large bursts of intensity (strong variations) in local area of image with similar colors:
And many other cases...
The simplest solution is using cv::medianBlur(), but it will smooth all image, not only outliers: Median filter example video
Is there any algorithm which smoothes only outliers, and It does not affect the rest of the image?
Is there anything better than this?
// get cv::Mat src_frame ...
int outliers_size = 10;
int outliers_intensive = 100;
int ksize = outliers_size*2 + 1; // smooth all outliers smaller than 11x11
cv::Mat smoothed;
cv::medianBlur( src_frame, smoothed, ksize );
cv::Mat diff;
cv::absdiff( src_frame, smoothed, diff );
cv::Mat mask = diff > Scalar( outliers_intensive );
smoothed.copyTo( src_frame, mask );
// we have smoothed only small outliers areas in src_frame
Perhaps you are looking for the bilateral filter?
OpenCV says:
we have explained some filters which main goal is to smooth an input
image. However, sometimes the filters do not only dissolve the noise,
but also smooth away the edges. To avoid this (at certain extent at
least), we can use a bilateral filter.
OpenCV has this built-in: http://docs.opencv.org/modules/imgproc/doc/filtering.html?highlight=bilateralfilter#bilateralfilter
I'm fairly new to OpenCV, and very excited to learn more. I've been toying with the idea of outlining edges, shapes.
I've come across this code (running on an iOS device), which uses Canny. I'd like to be able to render this in color, and circle each shape. Can someone point me in the right direction?
Thanks!
IplImage *grayImage = cvCreateImage(cvGetSize(iplImage), IPL_DEPTH_8U, 1);
cvCvtColor(iplImage, grayImage, CV_BGRA2GRAY);
cvReleaseImage(&iplImage);
IplImage* img_blur = cvCreateImage( cvGetSize( grayImage ), grayImage->depth, 1);
cvSmooth(grayImage, img_blur, CV_BLUR, 3, 0, 0, 0);
cvReleaseImage(&grayImage);
IplImage* img_canny = cvCreateImage( cvGetSize( img_blur ), img_blur->depth, 1);
cvCanny( img_blur, img_canny, 10, 100, 3 );
cvReleaseImage(&img_blur);
cvNot(img_canny, img_canny);
And example might be these burger patties. OpenCV would detect the patty, and outline it.
Original Image:
Color information is often handled by conversion to HSV color space which handles "color" directly instead of dividing color into R/G/B components which makes it easier to handle same colors with different brightness etc.
if you convert your image to HSV you'll get this:
cv::Mat hsv;
cv::cvtColor(input,hsv,CV_BGR2HSV);
std::vector<cv::Mat> channels;
cv::split(hsv, channels);
cv::Mat H = channels[0];
cv::Mat S = channels[1];
cv::Mat V = channels[2];
Hue channel:
Saturation channel:
Value channel:
typically, the hue channel is the first one to look at if you are interested in segmenting "color" (e.g. all red objects). One problem is, that hue is a circular/angular value which means that the highest values are very similar to the lowest values, which results in the bright artifacts at the border of the patties. To overcome this for a particular value, you can shift the whole hue space. If shifted by 50° you'll get something like this instead:
cv::Mat shiftedH = H.clone();
int shift = 25; // in openCV hue values go from 0 to 180 (so have to be doubled to get to 0 .. 360) because of byte range from 0 to 255
for(int j=0; j<shiftedH.rows; ++j)
for(int i=0; i<shiftedH.cols; ++i)
{
shiftedH.at<unsigned char>(j,i) = (shiftedH.at<unsigned char>(j,i) + shift)%180;
}
now you can use a simple canny edge detection to find edges in the hue channel:
cv::Mat cannyH;
cv::Canny(shiftedH, cannyH, 100, 50);
You can see that the regions are a little bigger than the real patties, that might be because of the tiny reflections on the ground around the patties, but I'm not sure about that. Maybe it's just because of jpeg compression artifacts ;)
If you instead use the saturation channel to extract edges, you'll end up with something like this:
cv::Mat cannyS;
cv::Canny(S, cannyS, 200, 100);
where the contours aren't completely closed. Maybe you can combine hue and saturation within preprocessing to extract edges in the hue channel but only where saturation is high enough.
At this stage you have edges. Regard that edges aren't contours yet. If you directly extract contours from edges they might not be closed/separated etc:
// extract contours of the canny image:
std::vector<std::vector<cv::Point> > contoursH;
std::vector<cv::Vec4i> hierarchyH;
cv::findContours(cannyH,contoursH, hierarchyH, CV_RETR_TREE , CV_CHAIN_APPROX_SIMPLE);
// draw the contours to a copy of the input image:
cv::Mat outputH = input.clone();
for( int i = 0; i< contoursH.size(); i++ )
{
cv::drawContours( outputH, contoursH, i, cv::Scalar(0,0,255), 2, 8, hierarchyH, 0);
}
you can remove those small contours by checking cv::contourArea(contoursH[i]) > someThreshold before drawing. But you see the two patties on the left to be connected? Here comes the hardest part... use some heuristics to "improve" your result.
cv::dilate(cannyH, cannyH, cv::Mat());
cv::dilate(cannyH, cannyH, cv::Mat());
cv::dilate(cannyH, cannyH, cv::Mat());
Dilation before contour extraction will "close" the gaps between different objects but increase the object size too.
if you extract contours from that it will look like this:
If you instead choose only the "inner" contours it is exactly what you like:
cv::Mat outputH = input.clone();
for( int i = 0; i< contoursH.size(); i++ )
{
if(cv::contourArea(contoursH[i]) < 20) continue; // ignore contours that are too small to be a patty
if(hierarchyH[i][3] < 0) continue; // ignore "outer" contours
cv::drawContours( outputH, contoursH, i, cv::Scalar(0,0,255), 2, 8, hierarchyH, 0);
}
mind that the dilation and inner contour stuff is a little fuzzy, so it might not work for different images and if the initial edges are placed better around the object border it might 1. not be necessary to do the dilate and inner contour thing and 2. if it is still necessary, the dilate will make the object smaller in this scenario (which luckily is great for the given sample image.).
EDIT: Some important information about HSV: The hue channel will give every pixel a color of the spectrum, even if the saturation is very low ( = gray/white) or if the color is very low (value) so often it is desired to threshold the saturation and value channels to find some specific color! This might be much easier and much more stavle to handle than the dilation I've used in my code.
I'm using a canny edge detection and a finding contours function (both OpenCV) to create markers for the watershed transform. Everything works fine but I'm not 100% satisfied with the results. The reason is that some edges are missing and therefore important information is lost. In more detail, I got a bunch of windows (front views), which are rectangles, after the watershed transform I end up with something like this:
but I would rather have nice rectangles, that are complete and not open to one side. While maintaining irregular shapes (bushes in front of the house, cars..) Any ideas how I could solve this problem?I thought about overlaying the whole image with a grid, but I can't make it work.
Thank you very much.
Here is my code:
Mat gray;
cvtColor(im, gray, CV_BGR2GRAY);
// Use Canny instead of threshold to catch squares with gradient shading
Mat bw;
Canny(gray, bw, 0, 100, 5, true);
// Find contours
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;
findContours( bw, contours, hierarchy,
CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE );
// watershed
Mat markers(bw.size(), CV_32S);
markers = Scalar::all(0);
int idx = 0;
int compCount = 0;
for( ; idx >= 0; idx = hierarchy[idx][0], compCount++ ) {
if (fabs(contourArea(contours[compCount])) < min_size )
continue;
drawContours(markers, contours, idx, Scalar::all(compCount+1), 1, 8, hierarchy, INT_MAX);
}
watershed( im, markers );
As requested, here is the original image, the image I would like to get and my output:
And I would like to have a segmentation like this (although over segmentation does not hurt, I just need to make sure, I get all the details):
While I get something like this:
(please ignore the colours, they are not important for this question and are just a result of my overall program). This is only one example, if you want, I can show you more, also please have a look at the etrims dataset, all my pictures are from there.
Two things -
1) As already mentioned, edge detection results in spurious edges being picked up.
2) Using these edges as markers for watershed segmentation results in over-segmentation because every marker produces a segmented region in the output.
Strategy -
(i) Preprocessing: Smooth the image heavily (morphological opening by reconstruction can be used for homogenizing the intensities without significantly affecting edges you are interested in).
(ii) Markers: Instead of using edges as seeds, I'd use the local extrema. Ideally, we want one marker for every region we want segmented.
(iii) Segmentation: Find the gradient magnitude (range filtering is also a good option) of the image from step (i) and use that as the segmentation function.
Using this strategy, I get the following segmentation.
Alternatively, after step (i), you can use Canny edge detection and do some morphological cleanup (to fill contours and remove edges that remain). This is what I get.
These are not exactly the expected segmentation (some objects like the car are not detected), but are a good start.
Edit: The MATLAB code used to generate the images -
% convert to grayscale
img = rgb2gray(origImg);
% create an appropriate structuring element
w_size = 20;
seSquare = strel('square', w_size);
% opening by reconstruction - to smooth dark regions
imgEroded = imerode(img, seSquare);
imgRecon = imreconstruct(imgEroded, img);
% invert and repeat - to smooth bright regions
imgReconComp = imcomplement(imgRecon);
imgEroded2 = imerode(imgReconComp, seSquare);
imgRecon2 = imreconstruct(imgEroded2, imgReconComp);
% get foreground markers
fgm = imregionalmax(imgRecon2);
% get background markers - this step can be skipped
% in which case only fgm would be the marker image
% and the segmentation would be different
distTrans = bwdist(fgm);
wLines= watershed(distTrans);
bgm = wLines == 0;
% get the segmentation function and impose markers
% perform watershed segmentation
seSquare3 = strel('square', 3);
rangeImg = rangefilt(imgRecon2, getnhood(seSquare3));
segFunc = imimposemin(rangeImg, fgm | bgm);
grayLabel = watershed(segFunc);
rgbLabel= label2rgb(grayLabel);
figure, imshow(rgbLabel); title('Output using Watershed')
% alternatively, extract edges from the preprocessed image
% perform morph cleanup
bwEdges = edge(imgRecon2, 'canny');
bwFilled = imfill(bwEdges, 'holes');
bwRegions = imopen(bwFilled, seSquare3);
grayLabel = bwlabel(bwRegions);
rgbLabel = label2rgb(grayLabel, 'jet', 'k');
figure, imshow(rgbLabel); title('Output using Canny')
from the looks of the desired output and the program's output, it seems that the edge detector is finding spurious edges. Canny edge detector contains a low-pass filter, but it might help for you to do a separate Gaussian low-pass filtering step before you actually run the Canny edge detector.
Other than that, it is difficult to achieve the desired result. For e.g., look at the top-most windows in the picture. They have distinct colors --- the frame, the shadow of the frame, and the window. The boundaries of these colors will be detected as edges by the Edge detector.
I setup an area of interest somewhere near the center of my image using:
Mat frame;
//frame has been initialized as a frame from a camera input
Rect roi= cvRect(frame.cols*.45, frame.rows*.45, 10, 8);
image_roi= frame(roi);
//I stoped here not knowing what to do next
I'm using a camera and at any time when I grab a frame, the ROI will be anywhere between 30% to 100% filled with my desired color, which is Red in this case. What is the most efficient method to know if Red is present in my current frame?
Solution:
image_roi= frame(roi);// a frame from my camera as a cv::Mat
cvtColor(image_roi, image_roi, CV_BGR2HSV);
thrs= new Mat(image_roi.rows, image_roi.cols, CV_8UC1);//allocate space for new img
inRange(image_roi, Scalar(0,100,100), Scalar(12,255,255), *thrs);//do hsv thresholding for red
for(int i= 0; i < thrs->rows; i++)//sum up
{
for(int j=0; j < thrs->cols; j++)
{
sum= sum+ thrs->data[(thrs->rows)* i + j];
}
}
if(sum> 100)//my application only cares about red
cout<<"Red"<<endl;
else
cout<<"White"<<endl;
sum=0;
This solution should address not only red but any color distribution:
Get a color histogram for your ROI, a bidimensional hue and saturation histogram (follow the example here).
Use calcBackProject to project the histogram back in the full image. You will get larger values in pixels presenting a color near the modes of the histogram (in this case, reds).
Threshold the result to get the pixels that better match the distribution (in this case, the "best reds").
This solution can be used, for example, to get a simple but very functional skin detector.
I'm assuming you just want to know the percentage of red in the ROI. If that's not correct, please clarify.
I'd scan the ROI and convert each pixel into a better color space for color comparison, such as YCbCr, or HSV. I'd then count the number of pixels where the hue is within some delta of red's hue (usually 0 degrees on the color wheel). You might need to deal with some edge cases where the brightness or saturation are too low for a human to think they're red, even though technically they are, depending on what you're trying to achieve.