OCR & OpenCV: Difference between two frames on high resolution images - c++

According to this post OCR: Difference between two frames, I now know how to find pixel differences between two images with OpenCV.
I would like to improve this solution and use it with high resolution images (from a video) with rich content. The example above is not applicable with big images because the process is to slow (too much differences found, the "findCountours method" fills the tab with 250k elements which takes a huge time to process).
My application uses a RLE decoder to decode the compressed frames of the video. Once the frame is decoded, I would like to compare the current frame with the previous one in order to store the differences between the two frames in a "Mat" tab for example.
The goal of all of this is to be able to perform an analysis on the different pixels and to check if there is any latin character. This allows me to reduce the amount of pixels to analyze and to save precious time.
If anyone has other ideas instead of this one to perform such operations, feel free to propose it please.
Thank you for your help.
EDIT 1:
Example of two high resolution images of a computer screen. These are for the moment the perfect example of what I'm trying to analyse. As we can see there is just a window as difference between the two big images and I would like to analyze just the new "Challenge" window for any character.
EDIT 2:
I'm trying to tune the algorithm depending on the data analyzed. Typically on the two following pictures I only get the green lines as differences and no text at all (which is what is the most interesting). I'm trying to understand better how things work for this.
1st image:
2nd image:
3rd image:
As you can see I only have those green lines and never the text (at the best I can have just ONE letter when decreasing the countours[i].size())

In addition to the post you mentioned, you need to:
When you binarize the mask, use a threshold higher then 0 to remove small differences.
Remove some noise. You can find all connected components, and remove smaller ones.
Find the area of the bigger connected components. You can use convexHull and fillConvexPoly to get the mask of the different objects on screen
Copy the second image to a new image, with the given mask.
The result will look like:
Code:
#include <opencv2/opencv.hpp>
#include <vector>
using namespace std;
using namespace cv;
int main()
{
Mat3b img1 = imread("path_to_image_1");
Mat3b img2 = imread("path_to_image_2");
Mat3b diff;
absdiff(img1, img2, diff);
// Split each channel
vector<Mat1b> masks;
split(diff, masks);
// Create a black mask
Mat1b mask(diff.rows, diff.cols, uchar(0));
// OR with each channel of the N channels mask
for (int i = 0; i < masks.size(); ++i)
{
mask |= masks[i];
}
// Binarize mask
mask = mask > 100;
// Results images
vector<Mat3b> difference_images;
// Remove small blobs
//Mat kernel = getStructuringElement(MORPH_RECT, Size(5,5));
//morphologyEx(mask, mask, MORPH_OPEN, kernel);
// Find connected components
vector<vector<Point>> contours;
findContours(mask.clone(), contours, CV_RETR_EXTERNAL, CHAIN_APPROX_NONE);
for (int i = 0; i < contours.size(); ++i)
{
if (contours[i].size() > 1000)
{
Mat1b mm(mask.rows, mask.cols, uchar(0));
vector<Point> hull;
convexHull(contours[i], hull);
fillConvexPoly(mm, hull, Scalar(255));
Mat3b difference_img(img2.rows, img2.cols, Vec3b(0,0,0));
img2.copyTo(difference_img, mm);
difference_images.push_back(difference_img.clone());
}
}
return 0;
}

Related

Image Segmentation - Remove Unwanted Pixels in C++

I'm working on image processing. Firstly, I have to make image segmentation and extract only boundary of image. Then, This image is converted to freeman chain code. The part of freeman chain code is Okay. But, When I make a segmentation of image, inside of the image remains some unwanted white pixels. And thus, the next step,which is freeman chain code, is not being succesfull. I mean, It gives incorrect chain code because of unwanted pixels. So, I have to remove unwanted pixels from inside of image. I will share my code and can you tell me how i can change in this code or what kind of a correct code can i should write for this filter ? Code is here :
#include <opencv2/opencv.hpp>
#include <vector>
#include <iostream>
#include <opencv2/imgproc/imgproc_c.h>
using namespace cv;
using namespace std;
int main(){
Mat img = imread("<image-path>");
Mat gray;
cvtColor(img,gray,CV_BGR2GRAY);
Mat binary;
threshold(gray,binary, 200, 255, CV_THRESH_BINARY);
Mat kernel = (Mat_<float>(3,3) <<
1, 1, 1,
1, -8, 1,
1, 1, 1);
Mat imgLaplacian;
Mat sharp= binary;
filter2D(binary, imgLaplacian, CV_32F, kernel);
binary.convertTo(sharp, CV_32F);
Mat imgResult = sharp - imgLaplacian;
imgResult.convertTo(imgResult, CV_8UC1);
imgLaplacian.convertTo(imgLaplacian, CV_8UC1);
//Find contours
vector<vector<Point>> contours;
vector <uchar> chaincode;
vector <char> relative;
findContours(imgLaplacian,contours, CV_RETR_LIST, CHAIN_APPROX_NONE);
for (size_t i=0; i<contours.size();i++){
chain_freeman(contours[i],chaincode);
FileStorage fs("<file-path>", 1);
fs << "chain" << chaincode;
}
for (size_t i=0; i<chaincode.size()-1; i++){
int relative1 = 0;
relative1 = abs(chaincode[i]-chaincode[i+1]);
cout << relative1;
for (int j=0; j<relative1; j++){
}
relative.push_back(relative1);
FileStorage fs("<file-path>", 1);
fs << "chain" << relative;
}
imshow("binary",imgLaplacian);
cvWaitKey();
return 0;
}
original image
Result
In this result, I want to remove white pixel inside of the image. I tried all fiter in opencv but I could not achieve. It's very important because of chain code.
Okay, now I see it. As said, you can ignore small contours simply by their length. For the rest, you need maximally thin contours (seems like 4-connected is the case). There you have couple options:
1) thinning of the current. If you can grab Matlab's lookup table, you can then load it into OpenCV as How to use Matlab's 512 element lookup table array in OpenCV?
2) it's pretty simple to label the boundary pixels by hand after binarization. To make it more efficient, you can first fill small cavities (islets) by applying connected component labeling on the background (using opposite connectivity this time, 8 it is).
2i & 2ii) If you do the labeling by hand, you can either continue collecting the contour vector by hand or switch to cv::findContours
Hope this helps

Real-time Image Processing: Noise in HSV image (openCV)

I am doing a real-time shapes and colors classification system with very high accuracy. It seems like my preprocessing phase is not good enough so that the result is not as accurate as I expected. Here is what I'm doing:
Take data from the Camera can crop it to receive ROI.
Convert ROI Image from RGB to HSV space.
Using a median filter to reduce noise in HSV image.
Threshold the image
Using dilate and erode to remove small holes and small objects in Image
Using findContours and approxPolyDP to detect square objects.
This is my preprocessing phase:
image_cv = cv::cvarrToMat(image_camera);
Mat cropped = image_cv(cv::Rect(0, 190, 640, 110));
imshow("origin", cropped);
Mat croppedCon = CropConveyor(cropped);
cv::cvtColor(croppedCon, croppedCon, CV_RGB2HSV);
medianBlur(croppedCon, croppedCon, 3);
cv::Mat binRect;
cv::inRange(croppedCon, Scalar(iLowH, iLowS, iLowV), Scalar(iHighH, iHighS, iHighV), binRect);
This is the code for detecting squares:
vector<vector<Point>> contours;
findContours(binarizedIm, contours, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE);
vector<Point> approx;
for (size_t i = 0; i < contours.size(); i++)
{
//double arclength = arcLength(Mat(contours[i]), true);
approxPolyDP(Mat(contours[i]), approx, 3.245 , true); //0.04 for wood
if (approx.size() != 4) continue;
if (isContourConvex(Mat(approx)) && contourArea(Mat(approx)) > 250)
{
double MaxCos = 0;
for (int j = 2; j < 5; j++)
{
double cos = angle(approx[j % 4], approx[j - 1], approx[j - 2]);
MaxCos = MAX(cos, MaxCos);
}
if (MaxCos < 0.2)
squares.push_back(approx);
}
}
I think noise in HSV Image is the main reason. Here is some images illustrating my problems. I saw a lot of noise in HSV Image, that's why I use a media filter to it to reduce noise but preserve the edges becase I think that edges information is very important when using findContours function.
HSV and HSV in separate channels
My question is:
What is the noise in HSV Image, refer to the above Image, how can I
enhance my Image's quality?
The reason for noise in your saturation image is noise in your input image. Caused by a bad camera / optics and further increased by JPEG compression.
That's by far the worst image I have seen in years. You shouldn't invest another second into processing that, unless you live on Mars and need results tomorrow.
Your input image is super noisy, undersampled, defocussed, underexposed, full of aliasing and compression artifacts and pretty much anything else you can do wrong with an image.
First rule of signal processing:
crap in = crap out
You can get much better cameras basically for free. Find and use one.
Part of the problem is that you're doing the noise reduction in HSV space. In your example you can see the V channel is better-behaved than H and S. It would be better to do noise-reduction in RGB (which is more linear and closer, though not identical, to the camera's native colour space where the noise originates; of course there's also gamma-correction).
Maybe consider a stronger edge-preserving noise-reducing filter such as Bilateral Filter.
I don't get it why are you using HSV for segmenting the objects, the RGB image is good enough. Separate the image into 3 channels (r,g,b) and apply an adaptive threshold on them. dilate and erode the images then add (not merging) those 3 binary images to have one binary image. Finally do level 6 of your recipe to extract the objects. If the noise still effects the result, apply a bilateral filter on r,g,b channels before the threshold.

OCR: Difference between two frames

I am trying to find an easy solution to implement the OCR algorithm from OPenCV. I am very new to Image Processing !
I am playing a video that is decoded with specific codec using RLE algorithm.
What I would like to do is that for each decoded frame, I would like to compare it with the previous one and store the pixels that have changed between the two frames.
Most of the existing solutions gives a difference between the two frames but I would like to just keep the new pixels that have changed and store it in a table and then be able to analyze every group of pixels that have changed instead of analyzing the whole image each time.
I planned to use the "blobs detection" algoritm mais I'm stuck before being able to implement it.
Today, I'm trying this:
char *prevFrame;
char *curFrame;
QVector DiffPixel<LONG>;
//for each frame
DiffPixel.push_back(curFrame-prevFrame);
I really want to have the "Only changed pixel result" solution. Could anyone give me some tips or correct me if I'm going to a wrong way ?
EDIT:
New question, what if there are multiple areas of changed pixels ? Will it be possible to have one table per blocs of changed pixels or will it be only one unique table ? Take the example below:
The best thing as a result would be to have 2 mat matrices. The first matrix with the first orange square and the second matrix with the second orange square. This way, it avoids having to "scan" almost the entire frame if we store the result in one matrix only with a resolution being almost the same as the full frame.
The main goal here is to minimize the area (aka the resolution) to analyze to find text.
After loading your images:
img1
img2
you can apply XOR operation to get the differences. The result has the same number of channels of the input images:
XOR
You can then create a binary mask OR-ing all channels:
mask
The you can copy the values of img2 that correspond to non-zero elements in the mask to a white image:
diff
UPDATE
If you have multiple areas where pixel changed, like this:
You'll find a difference mask (after binarization all non-zero pixels are set to 255) like:
You can then extract connected components and draw each connected component on a new black-initialized mask:
Then, as before, you can copy the values of img2 that correspond to non-zero elements in each mask to a white image.
The complete code for reference. Note that this is the code for the updated version of the answer. You can find the original code in the revision history.
#include <opencv2\opencv.hpp>
#include <vector>
using namespace cv;
using namespace std;
int main()
{
// Load the images
Mat img1 = imread("path_to_img1");
Mat img2 = imread("path_to_img2");
imshow("Img1", img1);
imshow("Img2", img2);
// Apply XOR operation, results in a N = img1.channels() image
Mat maskNch = (img1 ^ img2);
imshow("XOR", maskNch);
// Create a binary mask
// Split each channel
vector<Mat1b> masks;
split(maskNch, masks);
// Create a black mask
Mat1b mask(maskNch.rows, maskNch.cols, uchar(0));
// OR with each channel of the N channels mask
for (int i = 0; i < masks.size(); ++i)
{
mask |= masks[i];
}
// Binarize mask
mask = mask > 0;
imshow("Mask", mask);
// Find connected components
vector<vector<Point>> contours;
findContours(mask.clone(), contours, RETR_LIST, CHAIN_APPROX_SIMPLE);
for (int i = 0; i < contours.size(); ++i)
{
// Create a black mask
Mat1b mask_i(mask.rows, mask.cols, uchar(0));
// Draw the i-th connected component
drawContours(mask_i, contours, i, Scalar(255), CV_FILLED);
// Create a black image
Mat diff_i(img2.rows, img2.cols, img2.type());
diff_i.setTo(255);
// Copy into diff only different pixels
img2.copyTo(diff_i, mask_i);
imshow("Mask " + to_string(i), mask_i);
imshow("Diff " + to_string(i), diff_i);
}
waitKey();
return 0;
}

OpenCV Canny + Watershed

I'm using a canny edge detection and a finding contours function (both OpenCV) to create markers for the watershed transform. Everything works fine but I'm not 100% satisfied with the results. The reason is that some edges are missing and therefore important information is lost. In more detail, I got a bunch of windows (front views), which are rectangles, after the watershed transform I end up with something like this:
but I would rather have nice rectangles, that are complete and not open to one side. While maintaining irregular shapes (bushes in front of the house, cars..) Any ideas how I could solve this problem?I thought about overlaying the whole image with a grid, but I can't make it work.
Thank you very much.
Here is my code:
Mat gray;
cvtColor(im, gray, CV_BGR2GRAY);
// Use Canny instead of threshold to catch squares with gradient shading
Mat bw;
Canny(gray, bw, 0, 100, 5, true);
// Find contours
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;
findContours( bw, contours, hierarchy,
CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE );
// watershed
Mat markers(bw.size(), CV_32S);
markers = Scalar::all(0);
int idx = 0;
int compCount = 0;
for( ; idx >= 0; idx = hierarchy[idx][0], compCount++ ) {
if (fabs(contourArea(contours[compCount])) < min_size )
continue;
drawContours(markers, contours, idx, Scalar::all(compCount+1), 1, 8, hierarchy, INT_MAX);
}
watershed( im, markers );
As requested, here is the original image, the image I would like to get and my output:
And I would like to have a segmentation like this (although over segmentation does not hurt, I just need to make sure, I get all the details):
While I get something like this:
(please ignore the colours, they are not important for this question and are just a result of my overall program). This is only one example, if you want, I can show you more, also please have a look at the etrims dataset, all my pictures are from there.
Two things -
1) As already mentioned, edge detection results in spurious edges being picked up.
2) Using these edges as markers for watershed segmentation results in over-segmentation because every marker produces a segmented region in the output.
Strategy -
(i) Preprocessing: Smooth the image heavily (morphological opening by reconstruction can be used for homogenizing the intensities without significantly affecting edges you are interested in).
(ii) Markers: Instead of using edges as seeds, I'd use the local extrema. Ideally, we want one marker for every region we want segmented.
(iii) Segmentation: Find the gradient magnitude (range filtering is also a good option) of the image from step (i) and use that as the segmentation function.
Using this strategy, I get the following segmentation.
Alternatively, after step (i), you can use Canny edge detection and do some morphological cleanup (to fill contours and remove edges that remain). This is what I get.
These are not exactly the expected segmentation (some objects like the car are not detected), but are a good start.
Edit: The MATLAB code used to generate the images -
% convert to grayscale
img = rgb2gray(origImg);
% create an appropriate structuring element
w_size = 20;
seSquare = strel('square', w_size);
% opening by reconstruction - to smooth dark regions
imgEroded = imerode(img, seSquare);
imgRecon = imreconstruct(imgEroded, img);
% invert and repeat - to smooth bright regions
imgReconComp = imcomplement(imgRecon);
imgEroded2 = imerode(imgReconComp, seSquare);
imgRecon2 = imreconstruct(imgEroded2, imgReconComp);
% get foreground markers
fgm = imregionalmax(imgRecon2);
% get background markers - this step can be skipped
% in which case only fgm would be the marker image
% and the segmentation would be different
distTrans = bwdist(fgm);
wLines= watershed(distTrans);
bgm = wLines == 0;
% get the segmentation function and impose markers
% perform watershed segmentation
seSquare3 = strel('square', 3);
rangeImg = rangefilt(imgRecon2, getnhood(seSquare3));
segFunc = imimposemin(rangeImg, fgm | bgm);
grayLabel = watershed(segFunc);
rgbLabel= label2rgb(grayLabel);
figure, imshow(rgbLabel); title('Output using Watershed')
% alternatively, extract edges from the preprocessed image
% perform morph cleanup
bwEdges = edge(imgRecon2, 'canny');
bwFilled = imfill(bwEdges, 'holes');
bwRegions = imopen(bwFilled, seSquare3);
grayLabel = bwlabel(bwRegions);
rgbLabel = label2rgb(grayLabel, 'jet', 'k');
figure, imshow(rgbLabel); title('Output using Canny')
from the looks of the desired output and the program's output, it seems that the edge detector is finding spurious edges. Canny edge detector contains a low-pass filter, but it might help for you to do a separate Gaussian low-pass filtering step before you actually run the Canny edge detector.
Other than that, it is difficult to achieve the desired result. For e.g., look at the top-most windows in the picture. They have distinct colors --- the frame, the shadow of the frame, and the window. The boundaries of these colors will be detected as edges by the Edge detector.

Filter out only one contour in OpenCV C/C++

I'm trying to make a program to detect an object in any shape using a video camera/webcam based on Canny filter and contour finding function. Here is my program:
int main( int argc, char** argv )
{
CvCapture *cam;
CvMoments moments;
CvMemStorage* storage = cvCreateMemStorage(0);
CvSeq* contours = NULL;
CvSeq* contours2 = NULL;
CvPoint2D32f center;
int i;
cam=cvCaptureFromCAM(0);
if(cam==NULL){
fprintf(stderr,"Cannot find any camera. \n");
return -1;
}
while(1){
IplImage *img=cvQueryFrame(cam);
if(img==NULL){return -1;}
IplImage *src_gray= cvCreateImage( cvSize(img->width,img->height), 8, 1);
cvCvtColor( img, src_gray, CV_BGR2GRAY );
cvSmooth( src_gray, src_gray, CV_GAUSSIAN, 5, 11);
cvCanny(src_gray, src_gray, 70, 200, 3);
cvFindContours( src_gray, storage, &contours, sizeof(CvContour), CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE, cvPoint(0,0));
if(contours==NULL){ contours=contours2;}
contours2=contours;
cvMoments(contours, &moments, 1);
double m_00 = cvGetSpatialMoment( &moments, 0, 0 );
double m_10 = cvGetSpatialMoment( &moments, 1, 0 );
double m_01 = cvGetSpatialMoment( &moments, 0, 1 );
float gravityX = (m_10 / m_00)-150;
float gravityY = (m_01 / m_00)-150;
if(gravityY>=0&&gravityX>=0){
printf("center point=(%.f, %.f) \n",gravityX,gravityY); }
for (; contours != 0; contours = contours->h_next){
CvScalar color = CV_RGB(250,0,0);
cvDrawContours(img,contours,color,color,-1,-1, 8, cvPoint(0,0));
}
cvShowImage( "Input", img );
cvShowImage( "Contours", src_gray );
cvClearMemStorage(storage);
if(cvWaitKey(33)>=0) break;
}
cvDestroyWindow("Contours");
cvDestroyWindow("Source");
cvReleaseCapture(&cam);
}
This program will detect all contours captured by the camera and the average coordinate of the contours will be printed. My question is how to filter out only one object/contour so I can get more precise (x,y) position of the object? If possible, can anyone show me how to mark the center of the object by using (x,y) coordinates?
Thanks in advance. Cheers
p/s:Sorry I couldn't upload a screenshot yet but if anything helps, here's the link.
Edit: To make my question more clear:
For example, if I only want to filter out only the square from my screenshot above, what should I do?
The object I want to filter out has the biggest contour area and most importantly has a shape(any shape), not a straight or a curve line
I'm still experimenting with the smooth and canny values so if anybody have the problem to detect the contours using my program please alter the values.
I think it can be solved fairly easy. I would suggest some morphological operations before contour detection. Also, I would suggest filtering "out" smaller elements, and getting the biggest element as the only one still in the image.
I suggest:
for filtering out lines (straight or curved): you have to decide what do you yourself consider a border between a "line" and a "shape". Let's say you consider all the objects of a thickness 5 pixel or more to be objects, while the ones that are less than 5 pixels across to be lines. An morphological opening that uses a 5x5 square or a 3-pixel sized diamond shape as a structuring element would take care of this.
for filtering out small objects in general: if objects are of arbitrary shapes, purely morphological opening won't do: you have to do an algebraic opening. A special type of algebraic openings is an area opening: an operation that removes all the connected components in the image that have (pixel) area smaller than a given threshold. If you have an upper bound on the size of uninteresting objects, or a lower bound on the size of interesting ones, that value should be used as a threshold. You can probably get a similar effect with a larger morphological opening, but it will not be so flexible.
for filtering out all the objects except the largest: it sounds like removing connected components from the smallest one to the largest one should work. Try labeling the connected components. On a binary (black & white image), this image transformation works by creating a greyscale image, labeling the background as 0 (black), and each component with a different, increasing grey value. In the end, pixels of each object are marked by a different value. You can now simply look at the gray level histogram, and find the grey value with the most pixels. Set all the other grey levels to 0 (black), and the only object left in the image is the biggest one.
The suggestions are written from the simplest to the most complex ones. Still, I think OpenCV can be of help with any of these. Morphological erosion, dilation, opening and closing are implemented in OpenCV. I think you might need to construct an algebraic opening operator on your own (or play with combining OpenCV basic morphology), but I'm sure OpenCV can help you with both labeling the connected components and examining the histogram of the resulting greyscale image.
In the end, when only pixels from one object are left, you do the Canny contour detection.
This is a blob processing problem that can not be solved (easily) by OpenCV itself. Have a look at cvBlobsLib. This library is extends OpenCV with functions/classes for connected component labeling.
http://opencv.willowgarage.com/wiki/cvBlobsLib