I am aiming to develop an iOS app that captures an image > extracts Braille dots represented in a specific color (Blue) > translates Braille letters into text using image processing techniques.
My approach was to use OpenCV/C++ to have the image be processed to extract the blue colored dots as shown in this photo into this photo
The next step is to recognize Braille in the image to translate it into text, one solution was to put a grid on the image to find the intersection points pixel color value then classify them into (1 if white , 0 if black) as represented in this photo
The problem with the proposed solution was:
how to position the grid rows/columns at the wanted positions?
how to get the coordinates and value(0 or 1)of the intersection points?
If you have any suggestion/solution about the proposed solution or any other solution please share them It would be appreciated since I don't have an experience in OpenCV/C++ field.
*note that python solutions cannot be used in iOS (as far as I know).
I attached my code for reference
+ (UIImage *)detectRedShapesInImage:(UIImage *)image{
cv::Mat mat;
UIImageToMat(image, mat);
cv::medianBlur(mat, mat, 3);
// Convert input image to HSV
cv::Mat hsv_image;
cv::cvtColor(mat, hsv_image, cv::COLOR_BGR2HSV);
// Threshold the HSV image, keep only the red (replaced it with blue) pixels
cv::Mat lower_red_hue_range;
cv::Mat upper_red_hue_range;
cv::inRange(hsv_image, cv::Scalar(0, 100, 100), cv::Scalar(10, 255, 255), lower_red_hue_range);
cv::inRange(hsv_image, cv::Scalar(160, 100, 100), cv::Scalar(179, 255, 255), upper_red_hue_range);
// Combine the above two images
cv::Mat red_hue_image;
cv::addWeighted(lower_red_hue_range, 1.0, upper_red_hue_range, 1.0, 0.0, red_hue_image);
cv::GaussianBlur(red_hue_image, red_hue_image, cv::Size(9, 9), 2, 2);
// detect circules, for now it doesnot take all circles
std::vector<cv::Vec4f> circles;
cv::HoughCircles(red_hue_image, circles, cv::HOUGH_GRADIENT, 1.0, 20, 150, 40, 0, 0);
// Loop over all detected circles and outline them on the original image
if(circles.size() == 0) std::exit(-1);
for(size_t current_circle = 0; current_circle < circles.size(); ++current_circle) {
cv::Point center(std::round(circles[current_circle][0]), std::round(circles[current_circle][1]));
int radius = std::round(circles[current_circle][2]);
cv::circle(red_hue_image, center, radius, cv::Scalar(0, 255, 0), 5);
}
UIImage *maskedShapesImg = MatToUIImage(red_hue_image);
return maskedShapesImg;}
Hints:
Assuming that the text is fairly horizontal and the spacing between lines is sufficient:
get the centers of the dots;
find the shortest horizontal and vertical distances between dots; these give you the horizontal and vertical pitch;
cluster the dots that are no more than one horizontal or two vertical pitches apart (with a safety margin); a cluster should correspond to a single character;
find the top-left corner of the characters;
find the median horizontal and vertical distance between the characters;
from this information (dot and character spacing), map the dots to the grid by predicting the grid nodes and using the nearest-neighbor rule.
This work is made a little tricky by the fact that not all character occupy the two columns and three rows of dots.
Related
]2
I would like to get a square shape from the right image above. But when I try to get it, it also includes other protruding parts because they have similar color. Are there any solutions to get the result like below? (The square lines are not 100 % straight. They are little distorted.)
This is the code I wrote.
cv::Mat img_gray, img, clahe_img, threshold_img, bitwise_img, morph_img;
cv::Mat rectified_CCD_img = cv::imread('img.png')
cv::Mat kernel = cv::Mat::ones(99, 99, CV_8U);
clahe = cv::createCLAHE(10, cv::Size(100, 100));
cv::cvtColor(rectified_CCD_img, img_gray, cv::COLOR_BGR2GRAY);
cv::medianBlur(img_gray, img, 33);
clahe->apply(img, clahe_img);
cv::threshold(clahe_img, threshold_img, 0, 255, cv::THRESH_OTSU);
cv::bitwise_not(threshold_img, bitwise_img);
cv::morphologyEx(bitwise_img, morph_img, cv::MORPH_OPEN, kernel);
That's the original image:
Google Drive link
For this specific image my pipeline would be very simple:
Binary threshold the image with a fixed threshold. The rectangle is quite dark compared to the rest of the image.
Morphological opening with a large rectangular kernel to get rid of the "noise".
To get a perfect rectangle, determine the bounding rectangle of the remaining part, and draw a white rectangle.
That'd be the whole code:
// Read image
cv::Mat img = cv::imread("OTH61.png", cv::IMREAD_GRAYSCALE);
// Binary threshold image at fixed threshold
cv::Mat img_thr;
cv::threshold(img, img_thr, 32, 255, cv::THRESH_BINARY_INV);
// Morphological opening with large rectangular kernel
cv::Mat img_mop;
cv::morphologyEx(img_thr, img_mop, cv::MORPH_OPEN, cv::Mat::ones(51, 51, CV_8UC1));
// Draw rectangle w.r.t. to the bounding rectangle of the remaining part
cv::rectangle(img_mop, cv::boundingRect(img_mop), 255, cv::FILLED);
The thresholded image:
The morphological opened image:
The cleaned image:
I have a camera under a glass with IR light to detect objects. I can find the contours and draw them using the following code (I just found some examples online and modified it to my need so I am not a master at all!).
using namespace cv;
cvtColor(mat, mat, COLOR_BGR2GRAY);
blur(mat, mat, Size(3,3));
erode(mat, mat, NULL, Point(-1,-1), 2);
dilate(mat, mat, NULL, Point(-1,-1), 2);
Canny(mat, mat, 100, 200);
auto contours = std::vector<std::vector<Point>>();
auto hierarchy = std::vector<Vec4i>();
findContours(mat, contours, hierarchy, CV_RETR_TREE,
CV_CHAIN_APPROX_SIMPLE, Point(0, 0));
Mat drawing = Mat::zeros(mat.size(), CV_8UC3);
for( int i = 0; i< contours.size(); i++ ) {
Scalar color = Scalar(rng.uniform(0, 255), rng.uniform(0,255),
rng.uniform(0,255));
drawContours(drawing, contours, i, color, 2, 8, hierarchy, 0, Point());
}
putText(mat,
(QString("Blobs: %1").arg(contours.size())).toStdString(),
Point(25,175), cv::FONT_HERSHEY_PLAIN, 10, CV_RGB(0, 0, 255), 2);
This code results in a nice finding of the contours that I am quite happy with. Except the fact that my IR light somehow makes artifacts at the corners and bottom of the image.
You can see that I have used gimp to highlight the areas that I want to ignore while searching for contours. Under the gray shade you see the white pixels that my original code detects as contours. These areas are problematic and I want to exclude them from the either contour search or contour drawing (whichever is easier!)
I was thinking of cropping the image to get the ROI but the cropping is a rectangle while I (for example) could have things to be detected i.e. exactly at leftmost area.
I think there should be some data in the contour that tells me where are the pixels but I could not figure it out yet...
The easiest way would be to simply crop the image. Areas of the image are known as ROIs in OpenCV, which stands for Region of Interest.
So, you could simply say
cv::Mat image_roi = image(cv::Rect(x, y, w, h));
This basically makes a rectangular crop, with the top left corner at x,y, width w and height h.
Now, you might not want to reduce the size of the image. The next easiest way is to remove the artifacts is to set the borders to 0. Using ROIs, of course:
image(cv::Rect(x, y, w, h)).setTo(cv::Scalar(0, 0, 0));
This sets a rectangular region to black. You then have to define the 4 rectangular regions on the borders of your image that you want dark.
Note that all of the above is based on manual tuning and some experimentation, and it would work provided that your system is static.
I am trying to make an average of two blobs in OpenCV. To achieve that I was planning to use watershed algorithm on the image preprocessed in the following way:
cv::Mat common, diff, processed, result;
cv::bitwise_and(blob1, blob2, common); //calc common area of the two blobs
cv::absdiff(blob1, blob2, diff); //calc area where they differ
cv::distanceTransform(diff, processed, CV_DIST_L2, 3); //idea here is that the highest intensity
//will be in the middle of the differing area
cv::normalize(processed, processed, 0, 255, cv::NORM_MINMAX, CV_8U); //convert floats to bytes
cv::Mat watershedMarkers, watershedOutline;
common.convertTo(watershedMarkers, CV_32S, 1. / 255, 1); //change background to label 1, common area to label 2
watershedMarkers.setTo(0, processed); //set 0 (unknown) for area where blobs differ
cv::cvtColor(processed, processed, CV_GRAY2RGB); //watershed wants 3 channels
cv::watershed(processed, watershedMarkers);
cv::rectangle(watershedMarkers, cv::Rect(0, 0, watershedMarkers.cols, watershedMarkers.rows), 1); //remove the outline
//draw the boundary in red (for debugging)
watershedMarkers.convertTo(watershedOutline, CV_16S);
cv::threshold(watershedOutline, watershedOutline, 0, 255, CV_THRESH_BINARY_INV);
watershedOutline.convertTo(watershedOutline, CV_8U);
processed.setTo(cv::Scalar(CV_RGB(255, 0, 0)), watershedOutline);
//convert computed labels back to mask (blob), less relevant but shows my ultimate goal
watershedMarkers.convertTo(watershedMarkers, CV_8U);
cv::threshold(watershedMarkers, watershedMarkers, 1, 0, CV_THRESH_TOZERO_INV);
cv::bitwise_not(watershedMarkers * 255, result);
My problem with the results is that the calculated boundary is (almost) always adjacent to the area common to both blobs. Here are the pictures:
Input markers (black = 0, gray = 1, white = 2)
Watershed input image (distance transform result) with resulting outline drawn in red:
I would expect the boundary to go along the maximum intensity region of the input (that is, along the middle of the differing area). Instead (as you can see) it mostly goes around the area marked as 2, with a bit shifted to touch the background (marked as 1). Do I do something wrong here, or did I misunderstand how watershed works?
Starting from this image:
You can get the correct result simply passing an all-zero image to watershed algorithm. The "basin" is then equally filled of "water" starting from each "side" (then just remember to remove the outer border which is set by default to -1 by watershed algorithm):
Code:
#include <opencv2\opencv.hpp>
using namespace cv;
using namespace std;
int main()
{
Mat1b img = imread("path_to_image", IMREAD_GRAYSCALE);
Mat1i markers(img.rows, img.cols, int(0));
markers.setTo(1, img == 128);
markers.setTo(2, img == 255);
Mat3b image(markers.rows, markers.cols, Vec3b(0,0,0));
markers.convertTo(markers, CV_32S);
watershed(image, markers);
Mat3b result;
cvtColor(img, result, COLOR_GRAY2BGR);
result.setTo(Scalar(0, 0, 255), markers == -1);
imshow("Result", result);
waitKey();
return(0);
}
I am trying to find triangles (blue contours) and trapezoids (yellow contours) in real time. In general it's okay.
But there is some problems. First it's a false positives. Triangles become trapezoids and vice versa. And I don't know how to how to solve this problem.
Second it's "noise". . I tried to check area of the figure, but the noise can be equal to the area. So it did not help so much. The noise depends on the thresholding parameters. cv::adaptiveThresholddoes not help at all. It's adds even more noise (and it so SLOW) erode and dilate cant fix it in a proper way
And here is my code.
cv::Mat detect(cv::Mat imageRGB)
{
//RGB -> GRAY
cv::Mat imageGray;
cv::cvtColor(imageRGB, imageGray, CV_BGR2GRAY);
//Bluring it
cv::Mat image;
cv::GaussianBlur(imageGray, image, cv::Size(5,5), 2);
//Thresholding
cv::threshold(image, image, 100, 255, CV_THRESH_BINARY_INV);
//SLOW and NOISE
//cv::adaptiveThreshold(image, image, 255.0, CV_ADAPTIVE_THRESH_GAUSSIAN_C, CV_THRESH_BINARY, 21, 0);
//Calculating canny params.
cv::Scalar mu;
cv::Scalar sigma;
cv::meanStdDev(image, mu, sigma);
cv::Mat imageCanny;
cv::Canny(image,
imageCanny,
mu.val[0] + sigma.val[0],
mu.val[0] - sigma.val[0]);
//Detecting conturs.
std::vector<std::vector<cv::Point> > contours;
std::vector<cv::Vec4i> hierarchy;
cv::findContours(imageCanny, contours, hierarchy,CV_RETR_TREE, CV_CHAIN_APPROX_NONE);
//Hierarchy is not needed here so clear it.
hierarchy.clear();
for (std::size_t i = 0; i < contours.size(); i++)
{
//fitEllipse need at last 5 points.
if (contours.at(i).size() < 5)
{
continue;
}
//Skip small contours.
if (std::fabs(cv::contourArea(contours.at(i))) < 800.0)
{
continue;
}
//Calculating RotatedRect from contours NOT from hull
//because fitEllipse need at last 5 points.
cv::RotatedRect bEllipse = cv::fitEllipse(contours.at(i));
//Finds the convex hull of a point set.
std::vector<cv::Point> hull;
cv::convexHull(contours.at(i), hull, true);
//Approx it, so we'll get 3 point for triangles
//and 4 points for trapez.
cv::approxPolyDP(hull, hull, 15, true);
//Is our contour convex. It's mast be.
if (!cv::isContourConvex(hull))
{
continue;
}
//Triangle
if (hull.size() == 3)
{
cv::drawContours(imageRGB, contours, i, cv::Scalar(255, 0, 0), 2);
cv::circle(imageRGB, bEllipse.center, 3, cv::Scalar(0, 255, 0), 2);
}
//trapez
if (hull.size() == 4)
{
cv::drawContours(imageRGB, contours, i, cv::Scalar(0, 255, 255), 2);
cv::circle(imageRGB, bEllipse.center, 3, cv::Scalar(0, 0, 255), 2);
}
}
return imageRGB;
}
So... In general all problems coused by wrong thresholding parameters, how can I calculete it in a proper way (automatically, of course)? And how can I can (lol, sorry for my english) prevent false positives?
Thesholding - i think that you should try Otsu binarization - here is some theory and a nice picture and here is documentation. This kind of thresholding generally is trying to find 2 most common values in image and use average value of them as a threshold value.
Alternatively consider using HSV color space, it might be easier to distinguish black and white regions from other regions. Another idea is to use inRange function (in RGB or in HSV color space - should work in woth situations) - you need to find 2 ranges (one from black regions and one for white) and search only for those regions (using inRange function) - look at this post.
Another way to accomplish this task might be using some library for blob extraction like this one or blob extractor which is part of OpenCV.
Distinguish triangle from trapezoid - i see 2 basic ways to improve you solution here:
in this line cv::approxPolyDP(hull, hull, 15, true); make third parameter (15 in this situation) not a constant value, but some part of contour area or length. Definitely it should adapt to contour size, it can't be just a canstant value. It's hard to say how to calculate it without some testing - try to start with 1-5% of contour area or length (i would start with length, but this is just my guess) and see whether this value is fine/to big/to small an check other values if needed. Unfortunetely there is no other way, but finding this equation manually shouldn't take very long time.
when you have 4 or 5 points calculate the equations of lines which join consecutive points (point 1 with point 2, point 2 with point 3, etc don't forget to calculate line between first point and last point), than check whether any 2 of those lines are parallel (or at least are close to being parallel - angle between them is close to 0 degress) - if you find any parallel lines than this contour is trapezoid, otherwise it's a triangle.
I want to identify and extract the contour of the largest leaf of the following image using OpenCV and C++.
I applied Canny edge detector to the image and got the following result.
Canny(img_src, img_edge_detected, 20, 60, 3);
Now I want to extract the largest contour (largest leaf) form the image and draw the contour line, but the problem here is the edge line of the largest leaf is not continuous. So I looked in to dialate and morphological close but using those functions I couldn't get a good result to extract the area. Is there any way to get the largest contour in such image?
Note that here I cannot use template matching or any masking kind of things because my final intention is to built a system where a user can upload an image and get the species of the plant. So the system doesn't have any prior idea about the shape of the leaf that user is going to upload.
Please tell me how to find and draw the largest contour here if it is possible.
Thanks.
cant you use hsv color threshoding to track only that leaf and then you can straight away use minmaxloc function to get the area of the largest contour.just an idea try doing it like that.it will work.good luck
Same thing i will do in java please convert it into c++, here BGR to convert HSV then after apply the combination of the yellow, green and brown with specified range and simply perfom bitwise or operation. it will be give to you not zero pixles using opencv function Core.findNonZero(Mat src, Mat dst);
Imgproc.cvtColor(mRgba, mHSV, Imgproc.COLOR_BGR2HSV, 4);
//Yellow
Core.inRange(mHSV, new Scalar(25, 80, 80), new Scalar(36, 255, 255), yellow);
//Green
Core.inRange(mHSV, new Scalar(37, 80, 80), new Scalar(70, 255, 255), green);
//Brown
Core.inRange(mHSV, new Scalar(10, 80, 80), new Scalar(30, 200, 200), brown);
// logical OR mask
Core.bitwise_or(yellow, green, green);
Core.bitwise_or(green, brown, mask);
Imgproc.dilate(mask, mask, new Mat());
// Find non zero pixels
pts = Mat.zeros(mask.size(), mask.type());
Core.findNonZero(mask, pts);
return mask;