Changing displacement over a sequence of frames - c++

guys.
I've written a code to describe motion on interest points over a .avi video file.
Here is the code:
#include "opencv2/video/tracking.hpp"
#include<opencv2/core/core.hpp>
#include<opencv2/highgui/highgui.hpp>
#include<opencv2/imgproc/imgproc.hpp>
#include<iostream>
using namespace cv;
using namespace std;
int main() {
VideoCapture capture("video.avi");
if (!capture.isOpened()) {
cout << "ERROR OPENING VIDEO\n\n";
return(0);
}
double rate = capture.get(CV_CAP_PROP_FPS);
unsigned int numberFrames = (unsigned int) capture.get(CV_CAP_PROP_FRAME_COUNT);
int width = (unsigned int) capture.get(CV_CAP_PROP_FRAME_WIDTH);
int height = (unsigned int) capture.get(CV_CAP_PROP_FRAME_HEIGHT);
unsigned int codec = (unsigned int) capture.get(CV_CAP_PROP_FOURCC);
Mat currentGray;
Mat previousGray;
vector< Point2f > points[2];
vector< Point2f > initial;
vector< Point2f > features;
vector< uchar > status;
vector< float > error;
int maxCorners = 500; // maximum number of features to detect
double qualityLevel = 0.01; // quality level for feature detection
double minDistance = 10; // min distance between two points
Mat frame, output;
VideoWriter createdVideo("output.avi", codec, rate, Size(width,height), 1);
for (unsigned frameCounter = 0; frameCounter < numberFrames; frameCounter++) {
capture >> frame;
if (frame.empty())
break;
imshow("Video", frame);
cvtColor(frame, currentGray, CV_BGR2GRAY);
frame.copyTo(output);
if (points[0].size() <= 10){
goodFeaturesToTrack(currentGray, // the image
features, // the output detected features
maxCorners, // the maximum number of features
qualityLevel, // quality level
minDistance); // min distance between two features
// add the detected features to
// the currently tracked features
points[0].insert(points[0].end(),
features.begin(), features.end());
initial.insert(initial.end(),
features.begin(), features.end());
}
if (previousGray.empty())
currentGray.copyTo(previousGray);
calcOpticalFlowPyrLK(previousGray, currentGray, // 2 consecutive images
points[0], // input point positions in first image
points[1], // output point positions in the 2nd image
status, // tracking success
error); // tracking error
int k = 0;
for (int i = 0; i < points[1].size(); i++) {
// do we keep this point?
if (status[i] && // if point has moved
(abs(points[0][i].x - points[1][i].x) +
(abs(points[0][i].y - points[1][i].y)) > 2))
initial[k] = initial[i];
points[1][k++] = points[1][i];
}
points[1].resize(k);
initial.resize(k);
for (int i = 0; i < points[1].size(); i++) {
// draw line and circle
line(output,
initial[i], // initial position
points[1][i],// new position
Scalar(0, 255, 0), 2);
circle(output,
points[1][i],
2,
Scalar(0, 0, 255), -1);
}
std::swap(points[1], points[0]);
cv::swap(previousGray, currentGray);
createdVideo.write(output);
}
waitKey(0);
return(0);
}
My code tracks displacement of points frame by frame and keeps the first location of them until the end of video.
However, I would like not the keep the location's points of the first frame, but change them over time, i.e. changing the first point location with the second point location so on and then huge lines will not appear but only the displacement between two points in two frames.
Is there any possibility of doing this?

Since you only want the position of points in two frames, just use two vectors; one holding the keypoints from the last frame, and one holding keypoints from the previous frame. At the end of each iteration, just set the previous points to the current points. Something like this pseudocode:
// first frame
// detect keypoints
prev_frame_points = keypoints
// rest of the frames
for frame in frames:
// detect keypoints
curr_frame_points = keypoints
line(..., prev_frame_points, curr_frame_points, ...)
prev_frame_points = curr_frame_points

Related

OpenCV Finding square center c++

To begin, I am a complete novice in OpenCV and am beginner/reasonable in c++ code.
But OpenCV is new to me and I try to learn by doing projects and stuff.
Now for my new project I am trying to find the centre of square in a picture.
In my case there is only 1 square in picture.
I would like to build further upon the square.cpp example of OpenCV.
For my project there are 2 things I need some help with,
1: The edge of the window is detected as a square, I do not want this. Example
2: How could I get the centre of 1 square from the squares array?
This is the code from the example "square.cpp"
// The "Square Detector" program.
// It loads several images sequentially and tries to find squares in
// each image
#include "opencv2/core.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/highgui.hpp"
#include <iostream>
using namespace cv;
using namespace std;
static void help(const char* programName)
{
cout <<
"\nA program using pyramid scaling, Canny, contours and contour simplification\n"
"to find squares in a list of images (pic1-6.png)\n"
"Returns sequence of squares detected on the image.\n"
"Call:\n"
"./" << programName << " [file_name (optional)]\n"
"Using OpenCV version " << CV_VERSION << "\n" << endl;
}
int thresh = 50, N = 11;
const char* wndname = "Square Detection Demo";
// helper function:
// finds a cosine of angle between vectors
// from pt0->pt1 and from pt0->pt2
static double angle(Point pt1, Point pt2, Point pt0)
{
double dx1 = pt1.x - pt0.x;
double dy1 = pt1.y - pt0.y;
double dx2 = pt2.x - pt0.x;
double dy2 = pt2.y - pt0.y;
return (dx1 * dx2 + dy1 * dy2) / sqrt((dx1 * dx1 + dy1 * dy1) * (dx2 * dx2 + dy2 * dy2) + 1e-10);
}
// returns sequence of squares detected on the image.
static void findSquares(const Mat& image, vector<vector<Point> >& squares)
{
squares.clear();
Mat pyr, timg, gray0(image.size(), CV_8U), gray;
// down-scale and upscale the image to filter out the noise
pyrDown(image, pyr, Size(image.cols / 2, image.rows / 2));
pyrUp(pyr, timg, image.size());
vector<vector<Point> > contours;
// find squares in every color plane of the image
for (int c = 0; c < 3; c++)
{
int ch[] = { c, 0 };
mixChannels(&timg, 1, &gray0, 1, ch, 1);
// try several threshold levels
for (int l = 0; l < N; l++)
{
// hack: use Canny instead of zero threshold level.
// Canny helps to catch squares with gradient shading
if (l == 0)
{
// apply Canny. Take the upper threshold from slider
// and set the lower to 0 (which forces edges merging)
Canny(gray0, gray, 0, thresh, 5);
// dilate canny output to remove potential
// holes between edge segments
dilate(gray, gray, Mat(), Point(-1, -1));
}
else
{
// apply threshold if l!=0:
// tgray(x,y) = gray(x,y) < (l+1)*255/N ? 255 : 0
gray = gray0 >= (l + 1) * 255 / N;
}
// find contours and store them all as a list
findContours(gray, contours, RETR_LIST, CHAIN_APPROX_SIMPLE);
vector<Point> approx;
// test each contour
for (size_t i = 0; i < contours.size(); i++)
{
// approximate contour with accuracy proportional
// to the contour perimeter
approxPolyDP(contours[i], approx, arcLength(contours[i], true) * 0.02, true);
// square contours should have 4 vertices after approximation
// relatively large area (to filter out noisy contours)
// and be convex.
// Note: absolute value of an area is used because
// area may be positive or negative - in accordance with the
// contour orientation
if (approx.size() == 4 &&
fabs(contourArea(approx)) > 1000 &&
isContourConvex(approx))
{
double maxCosine = 0;
for (int j = 2; j < 5; j++)
{
// find the maximum cosine of the angle between joint edges
double cosine = fabs(angle(approx[j % 4], approx[j - 2], approx[j - 1]));
maxCosine = MAX(maxCosine, cosine);
}
// if cosines of all angles are small
// (all angles are ~90 degree) then write quandrange
// vertices to resultant sequence
if (maxCosine < 0.3)
squares.push_back(approx);
}
}
}
}
}
int main(int argc, char** argv)
{
static const char* names[] = { "testimg.jpg", 0 };
help(argv[0]);
if (argc > 1)
{
names[0] = argv[1];
names[1] = "0";
}
for (int i = 0; names[i] != 0; i++)
{
string filename = samples::findFile(names[i]);
Mat image = imread(filename, IMREAD_COLOR);
if (image.empty())
{
cout << "Couldn't load " << filename << endl;
continue;
}
vector<vector<Point> > squares;
findSquares(image, squares);
polylines(image, squares, true, Scalar(0, 0, 255), 3, LINE_AA);
imshow(wndname, image);
int c = waitKey();
if (c == 27)
break;
}
return 0;
}
I would like some help to start off.
How could I get some information from 1 of the squares out of the array called "squares" (I am having a difficult time understand what exactly is in the array as well; is it an array of points?)
If something is not clear please let me know and I will try to re-explain.
Thank you in advance
Firstly, you are talking about squares but you are actually detecting rectangles. I provided a shorter code to be able to better answer your questions.
I read the image, apply a Canny filter for binarization and detect all contours. After that I iterate through the contours and find the ones which can be approximated by exactly four points and are convex:
int main(int argc, char** argv)
{
// Reading the images
cv::Mat img = cv::imread("squares_image.jpg", cv::IMREAD_GRAYSCALE);
cv::Mat edge_img;
std::vector <std::vector<cv::Point>> contours;
// Convert the image into a binary image using Canny filter - threshold could be automatically determined using OTSU method
cv::Canny(img, edge_img, 30, 100);
// Find all contours in the Canny image
findContours(edge_img, contours, cv::RETR_LIST, cv::CHAIN_APPROX_SIMPLE);
// Iterate through the contours and test if contours are square
std::vector<std::vector<cv::Point>> all_rectangles;
std::vector<cv::Point> single_rectangle;
for (size_t i = 0; i < contours.size(); i++)
{
// 1. Contours should be approximateable as a polygon
approxPolyDP(contours[i], single_rectangle, arcLength(contours[i], true) * 0.01, true);
// 2. Contours should have exactly 4 vertices and be convex
if (single_rectangle.size() == 4 && cv::isContourConvex(single_rectangle))
{
// 3. Determine if the polygon is really a square/rectangle using its properties (parallelity, angles etc.)
// Not necessary for the provided image
// Push the four points into your vector of squares (could be also std::vector<cv::Rect>)
all_rectangles.push_back(single_rectangle);
}
}
for (size_t num_contour = 0; num_contour < all_rectangles.size(); ++num_contour) {
cv::drawContours(img, all_rectangles, num_contour, cv::Scalar::all(-1));
}
cv::imshow("Detected rectangles", img);
cv::waitKey(0);
return 0;
}
1: The edge of the window is detected as a square, I do not want this.
There are several options depending on your applications. You can filter the outer boundary already using the Canny thresholds, using a different contour retrieval method for finding contours in findContours or by filtering single_rectangle using the area of the found contour (e.g. cv::contourArea(single_rectangle) < 1000).
2: How could I get the centre of 1 square from the squares array?
Since the code is already detecting the four corner points you could e.g. find the intersection of the diagonals. If you know that there are only rectangles in your image you could also try to detect all centroids of the detected contours using the Hu moments.
I am having a difficult time understand what exactly is in the array as well; is it an array of points?
One contour in OpenCV is always represented as a vector of single points. If you are adding multiple contours you are using a vector of vector of points. In the example you provided squares is a vector of a vector of exactly 4 points. You could also use a vector of cv::Rect in this case.

Object tracking delay color tracking OpenCV

I am trying to detect colored balls like ps3 move controller balls from 2 mt distance.I have 10 camera in same room hanging from the ceiling.Room is dark and balls have led inside.I have 4-5 balls.(red,blue,green,yellow,pink). I want track their position with opencv.Whats the right mehtod for doing this in opencv ? Can u give link , example for this ?
I use this code but i have delay problem.When i comment // my trackFilteredObject line there is no lag.But when using this code i have lot latency.I cant understand why happening because my normal cpu usage ~%15 ram usage 6.3GB/15GB (%40) when run this code cpu usage ~20-23 ram usage 6.4GB . I think its not about cpu-ram performance.What am i doing wrong ?
Video: https://www.youtube.com/watch?v=_BKtJpPrkO4 (You can see lag in first 10 sec.After 10 sen i comment tracking codes.)
Note:Kamerasayisi mean cameracount My Track Function:
void trackFilteredObject(Object theObject,Mat threshold,Mat HSV, Mat &cameraFeed){
//max number of objects to be detected in frame
const int FRAME_WIDTH = 5120;
const int FRAME_HEIGHT = 480;
const int MAX_NUM_OBJECTS=50;
//minimum and maximum object area
const int MIN_OBJECT_AREA = 10*10;
const int MAX_OBJECT_AREA = FRAME_HEIGHT*FRAME_WIDTH/1.5;
vector <Object> objects;
Mat temp;
threshold.copyTo(temp);
//these two vectors needed for output of findContours
vector< vector<Point> > contours;
vector<Vec4i> hierarchy;
//find contours of filtered image using openCV findContours function
findContours(temp,contours,hierarchy,CV_RETR_CCOMP,CV_CHAIN_APPROX_SIMPLE );
//use moments method to find our filtered object
double refArea = 0;
bool objectFound = false;
if (hierarchy.size() > 0) {
int numObjects = hierarchy.size();
//if number of objects greater than MAX_NUM_OBJECTS we have a noisy filter
if(numObjects<MAX_NUM_OBJECTS){
for (int index = 0; index >= 0; index = hierarchy[index][0]) {
Moments moment = moments((cv::Mat)contours[index]);
double area = moment.m00;
//if the area is less than 20 px by 20px then it is probably just noise
//if the area is the same as the 3/2 of the image size, probably just a bad filter
//we only want the object with the largest area so we safe a reference area each
//iteration and compare it to the area in the next iteration.
if(area>MIN_OBJECT_AREA){
Object object;
object.setXPos(moment.m10/area);
object.setYPos(moment.m01/area);
object.setType(theObject.getType());
object.setColor(theObject.getColor());
objects.push_back(object);
objectFound = true;
}else objectFound = false;
}
//let user know you found an object
if(objectFound ==true){
//draw object location on screen
drawObject(objects,cameraFeed,temp,contours,hierarchy);}
}else putText(cameraFeed,"TOO MUCH NOISE! ADJUST FILTER",Point(0,50),1,2,Scalar(0,0,255),2);
}
}
};
Main Code:
void Run()
{
int w, h;
_fps = 30;
IplImage *pCapImage[kameraSayisi];
IplImage *pDisplayImage;
PBYTE pCapBuffer = NULL;
// Create camera instance
for(int i = 0; i < kameraSayisi; i++)
{
_cam[i] = CLEyeCreateCamera(_cameraGUID[i], _mode, _resolution, _fps);
if(_cam[i] == NULL) return;
// Get camera frame dimensions
CLEyeCameraGetFrameDimensions(_cam[i], w, h);
// Create the OpenCV images
pCapImage[i] = cvCreateImage(cvSize(w, h), IPL_DEPTH_8U, 1);
// Set some camera parameters
CLEyeSetCameraParameter(_cam[i], CLEYE_GAIN, 0);
CLEyeSetCameraParameter(_cam[i], CLEYE_EXPOSURE, 511);
// Start capturing
CLEyeCameraStart(_cam[i]);
}
pDisplayImage = cvCreateImage(cvSize(w*kameraSayisi / 2, h * kameraSayisi/4 ), IPL_DEPTH_8U ,1);
if(_cam == NULL) return;
int iLastX = -1;
int iLastY = -1;
//Capture a temporary image from the camera
//program
bool trackObjects = true;
bool useMorphOps = true;
Mat HSV;
//Create a black image with the size as the camera output
Mat imgLines;
// imgLines = Mat::zeros( cvarrToMat(image).size(), CV_8UC3 );;
Mat threshold;
//x and y values for the location of the object
int x=0, y=0;
bool calibrationMode = false;
if(calibrationMode){
//create slider bars for HSV filtering
createTrackbars();
}
// image capturing loop
while(_running)
{
PBYTE pCapBuffer;
// Capture camera images
for(int i = 0; i < kameraSayisi; i++)
{
cvGetImageRawData(pCapImage[i], &pCapBuffer);
CLEyeCameraGetFrame(_cam[i], pCapBuffer, (i==0)?2000:0);
}
// Display stereo image
for(int i = 0; i < kameraSayisi; i++)
{
cvSetImageROI(pDisplayImage, cvRect(w * (i%4) ,i/4 * h, w, h));
cvCopy(pCapImage[i], pDisplayImage);
}
cvResetImageROI(pDisplayImage);
Mat imgOriginal;
Mat imgConverted = cvarrToMat(pDisplayImage);
if(calibrationMode==true)
{
//need to find the appropriate color range values
// calibrationMode must be false
//if in calibration mode, we track objects based on the HSV slider values.
//cvtColor(imgOriginal,imgOriginal,CV_BayerRG2RGB);
cvtColor(imgConverted,imgOriginal,CV_BayerGB2BGR);
cvtColor(imgOriginal,HSV,CV_BGR2HSV);
inRange(HSV,Scalar(H_MIN,S_MIN,V_MIN),Scalar(H_MAX,S_MAX,V_MAX),threshold);
morphOps(threshold);
imshow(_windowName + 'T',threshold);
//the folowing for canny edge detec
/// Create a matrix of the same type and size as src (for dst)
dst.create( imgOriginal.size(), src.type() );
/// Convert the image to grayscale
cvtColor( imgOriginal, src_gray, CV_BGR2GRAY );
/// Create a window
namedWindow( window_name, CV_WINDOW_AUTOSIZE );
/// Create a Trackbar for user to enter threshold
// createTrackbar( "Min Threshold:", window_name, &lowThreshold, max_lowThreshold, CannyThreshold );
/// Show the image
Object a = Object(H_MIN,S_MIN,V_MIN,H_MAX,S_MAX,V_MAX);
trackFilteredObject(a,threshold,HSV,imgOriginal);
}
else{
//we can use their member functions/information
Object blue("blue"), yellow("yellow"), red("red"), orange("orange"),white("white");
cvtColor(imgConverted,imgOriginal,CV_BayerGB2BGR);
//first find blue objects
cvtColor(imgOriginal,HSV,CV_RGB2HSV);
inRange(HSV,blue.getHSVmin(),blue.getHSVmax(),threshold);
morphOps(threshold);
//then yellows
inRange(HSV,yellow.getHSVmin(),yellow.getHSVmax(),threshold);
//then reds
inRange(HSV,red.getHSVmin(),red.getHSVmax(),threshold);
//then white
inRange(HSV,white.getHSVmin(),white.getHSVmax(),threshold);
//then orange
inRange(HSV,orange.getHSVmin(),orange.getHSVmax(),threshold);
trackFilteredObject(yellow,threshold,HSV,imgOriginal);
trackFilteredObject(white,threshold,HSV,imgOriginal);
trackFilteredObject(red,threshold,HSV,imgOriginal);
trackFilteredObject(blue,threshold,HSV,imgOriginal);
trackFilteredObject(orange,threshold,HSV,imgOriginal);
}
//delay 10ms so that screen can refresh.
//image will not appear without this waitKey() command
if (cvWaitKey(30) == 27) //wait for 'esc' key press for 30ms. If 'esc' key is pressed, break loop
{
cout << "esc key is pressed by user" << endl;
break;
}
// cvShowImage(_windowName, image);
imshow(_windowName,imgOriginal);
}
for(int i = 0; i < kameraSayisi; i++)
{
// Stop camera capture
CLEyeCameraStop(_cam[i]);
// Destroy camera object
CLEyeDestroyCamera(_cam[i]);
// Destroy the allocated OpenCV image
cvReleaseImage(&pCapImage[i]);
_cam[i] = NULL;
}
}

OpenCV splitting camera feed into grid and determining colour

I've written a piece of code to take my camera feed, split it into a grid (like a chess board) and evaluate each square for colour.
The code i currently have looks like this:
using namespace std;
using namespace cv;
//Standard Dilate and erode functions to improve white/black areas in Binary Image
// Pointer &thresh used so it affects threshImg so it can be used in tracking.
void morphOps(Mat &thresh){
//Increases size of black to remove unwanted white specks outside of object
Mat erodeElement = getStructuringElement( MORPH_RECT,Size(3,3));
//Increases white-area size to remove holes in object
Mat dilateElement = getStructuringElement( MORPH_RECT,Size(8,8));
erode(thresh,thresh,erodeElement);
erode(thresh,thresh,erodeElement);
dilate(thresh,thresh,dilateElement);
dilate(thresh,thresh,dilateElement);
}
//Tracking for the Filtered Object
void trackFilteredObject(int noteNum, string colourtype, Mat &thresh ,Mat HSVImage, Mat &cam){
vector<Brick> Bricks;
Mat temp;
thresh.copyTo(temp);
threshold(temp, thresh, 120, 255, 3); //3 = Threshold to Zero
int whitePixs = countNonZero(thresh);
int cols = thresh.cols;
int rows = thresh.rows;
int imgSize = (rows*cols)/0.75;
if(whitePixs > imgSize){
Brick Brick;
Brick.setColour(colourtype);
Brick.setnoteNum(noteNum);
Bricks.push_back(Brick);
}
int main(int argc, char* argv[])
{
/// Create a window
namedWindow("window", CV_WINDOW_AUTOSIZE );
while(1){
//initialtes camera, sets capture resolution
VideoCapture capture;
capture.open(1);
capture.set(CV_CAP_PROP_FPS, 30);
capture.set(CV_CAP_PROP_FRAME_WIDTH,640);
capture.set(CV_CAP_PROP_FRAME_HEIGHT,480);
Mat cam;
// Saves camera image to Matrix "cam"
capture.read(cam);
//Sets Widths and Heights based on camera resolution (cam.cols/cam.rows retrieves this)
int Width = cam.cols;
int gridWidth = Width/16;
int Height = cam.rows;
int gridHeight = Height/16;
//Splits image into 256 squares going left to right through rows and descending vertically. (16 squares per row for 4/4 pattern)
Mat BigImage;
Mat HSVImage;
// Converts cam to HSV pallete
cvtColor(cam, HSVImage, COLOR_BGR2HSV);
Size smallSize(gridWidth,gridHeight);
std::vector<Mat> smallImages;
for (int y = 0; y < HSVImage.rows; y += smallSize.height)
{
for (int x = 0; x < HSVImage.cols; x += smallSize.width)
{
cv::Rect rect = cv::Rect(x,y, smallSize.width, smallSize.height);
//Saves the matrix to vector
smallImages.push_back(cv::Mat(HSVImage, rect));
}
}
for (int i = 0; i < smallImages.size(); i++){
Mat HSV;
smallImages.at(i).copyTo(HSV);
int noteNum = i;
Mat threshImg;
inRange(HSV,Scalar(0,0,0),Scalar(255,255,255),threshImg);
morphOps(threshImg); //erodes image
string colour = "Red";
trackFilteredObject(noteNum,colour,threshImg,HSV,cam);
inRange(HSV,Scalar(0,0,0),Scalar(255,255,255),threshImg);
morphOps(threshImg); // threshold = mat after erosion/dilation
colour = "yellow";
trackFilteredObject(noteNum,colour,threshImg,HSV,cam);
inRange(HSV,Scalar(0,0,0),Scalar(255,255,255),threshImg);
morphOps(threshImg);
colour = "Black";
trackFilteredObject(noteNum,colour,threshImg,HSV,cam);
inRange(HSV,Scalar(0,0,0),Scalar(255,255,255),threshImg);
morphOps(threshImg); // threshold = mat after erosion/dilation
colour = "White";
trackFilteredObject(noteNum,colour,threshImg,HSV,cam);
inRange(HSV,Scalar(0,0,0),Scalar(255,255,255),threshImg);
morphOps(threshImg); // threshold = mat after erosion/dilation
colour = "Green";
trackFilteredObject(noteNum,colour,threshImg,HSV,cam);
}
imshow("window", cam);
}
return 0;
}
At the moment the code takes quite a long time to execute a full loop (about 1.5 seconds) but i ideally need it to run as close to real time as possible for a music application.
Could anyone suggest why it takes so long to execute? Is there a better way to evaluate the colour of each square?
My class is as follows:
//Brick.h
#include <string>
using namespace std;
class Brick{
public:
Brick(void);
~Brick(void);
string getColour();
void setColour(string whatColour);
int getnoteNum();
void setnoteNum(int whatnoteNum);
private:
int noteNum;
string colour;
};
///
Brick.cpp
#include <stdio.h>
#include <Brick.h>
Brick::Brick(void){
}
Brick::~Brick(void){
}
// get/set Colour
////////////////////////////////
string Brick::getColour(){
return Brick::colour;
}
void Brick::setColour(string whatColour){
Brick::colour = whatColour;
}
// get/set Note Number
////////////////////////////////
int Brick::getnoteNum(){
return Brick::noteNum;
}
void Brick::setnoteNum(int whatnoteNum){
Brick::noteNum = whatnoteNum;
}
I will be so grateful to anyone who replies!
Thank you.
Try hard to not use erode and dilate. These operations are extremely time intensive. I'm quite confident that they are the bottleneck in your program.
There are some measures you can take:
Downscaling(or downsampling) the image. Ideally, you want the downscaled image's pixel to be of the same order of magnitude of a grid square's size.
Remove dilate and erode.
Off-topic: Bugfix. Fix the inRange() parameters used. Consult the HSV color space diagram and normalize to your space. e.g. extracting "green pixels" would correspond to inRange(HSV,Scalar(80f*255/360,0.3*255,0.3*255),Scalar(160f*255/360,255,255),threshImg);

How to run findContours() on meanShiftSegmentation() output?

I'm trying to rewrite my very slow naive segmentation using floodFill to something faster. I ruled out meanShiftFiltering a year ago because of the difficulty in labelling the colours and then finding their contours.
The current version of opencv seems to have a fast new function that labels segments using mean shift: gpu::meanShiftSegmentation(). It produces images like the following:
(source: ekran.org)
So this looks to me pretty close to being able to generating contours. How can I run findContours to generate segments?
Seems to me, this would be done by extracting the labelled colours from the image, and then testing which pixel values in the image match each label colour to make a boolean image suitable for findContours. This is what I have done in the following (but its a bit slow and strikes me there should be a better way):
Mat image = imread("test.png");
...
// gpu operations on image resulting in gpuOpen
...
// Mean shift
TermCriteria iterations = TermCriteria(CV_TERMCRIT_ITER, 2, 0);
gpu::meanShiftSegmentation(gpuOpen, segments, 10, 20, 300, iterations);
// convert to greyscale (HSV image)
vector<Mat> channels;
split(segments, channels);
// get labels from histogram of image.
int size = 256;
labels = Mat(256, 1, CV_32SC1);
calcHist(&channels.at(2), 1, 0, Mat(), labels, 1, &size, 0);
// Loop through hist bins
for (int i=0; i<256; i++) {
float count = labels.at<float>(i);
// Does this bin represent a label in the image?
if (count > 0) {
// find areas of the image that match this label and findContours on the result.
Mat label = Mat(channels.at(2).rows, channels.at(2).cols, CV_8UC1, Scalar::all(i)); // image filled with label colour.
Mat boolImage = (channels.at(2) == label); // which pixels in labeled image are identical to this label?
vector<vector<Point>> labelContours;
findContours(boolImage, labelContours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// Loop through contours.
for (int idx = 0; idx < labelContours.size(); idx++) {
// get bounds for this contour.
bounds = boundingRect(labelContours[idx]);
// create ROI for bounds to extract this region
Mat patchROI = image(bounds);
Mat maskROI = boolImage(bounds);
}
}
}
Is this the best approach or is there a better way to get the label colours? Seems it would be logical for meanShiftSegmentation to provide this information? (vector of colour values, or vector of masks for each label, etc.)
Thank you.
Following is another way of doing this without thowing away the colour information in the meanShiftSegmentation results. I did not compare the two for performance.
// Loop through whole image, pixel and pixel and then use the colour to index an array of bools indicating presence.
vector<Scalar> colours;
vector<Scalar>::iterator colourIter;
vector< vector< vector<bool> > > colourSpace;
vector< vector< vector<bool> > >::iterator colourSpaceBIter;
vector< vector<bool> >::iterator colourSpaceGIter;
vector<bool>::iterator colourSpaceRIter;
// Initialize 3D Vector
colourSpace.resize(256);
for (int i = 0; i < 256; i++) {
colourSpace[i].resize(256);
for (int j = 0; j < 256; j++) {
colourSpace[i][j].resize(256);
}
}
// Loop through pixels in the image (should be fastish, look into LUT for faster)
uchar r, g, b;
for (int i = 0; i < segments.rows; i++)
{
Vec3b* pixel = segments.ptr<Vec3b>(i); // point to first pixel in row
for (int j = 0; j < segments.cols; j++)
{
b = pixel[j][0];
g = pixel[j][1];
r = pixel[j][2];
colourSpace[b][g][r] = true; // this colour is in the image.
//cout << "BGR: " << int(b) << " " << int(g) << " " << int(r) << endl;
}
}
// Get all the unique colours from colourSpace
// loop through colourSpace
int bi=0;
for (colourSpaceBIter = colourSpace.begin(); colourSpaceBIter != colourSpace.end(); colourSpaceBIter++) {
int gi=0;
for (colourSpaceGIter = colourSpaceBIter->begin(); colourSpaceGIter != colourSpaceBIter->end(); colourSpaceGIter++) {
int ri=0;
for (colourSpaceRIter = colourSpaceGIter->begin(); colourSpaceRIter != colourSpaceGIter->end(); colourSpaceRIter++) {
if (*colourSpaceRIter)
colours.push_back( Scalar(bi,gi,ri) );
ri++;
}
gi++;
}
bi++;
}
// For each colour
int segmentCount = 0;
for (colourIter = colours.begin(); colourIter != colours.end(); colourIter++) {
Mat label = Mat(segments.rows, segments.cols, CV_8UC3, *colourIter); // image filled with label colour.
Mat boolImage = Mat(segments.rows, segments.cols, CV_8UC3);
inRange(segments, *colourIter, *colourIter, boolImage); // which pixels in labeled image are identical to this label?
vector<vector<Point> > labelContours;
findContours(boolImage, labelContours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// Loop through contours.
for (int idx = 0; idx < labelContours.size(); idx++) {
// get bounds for this contour.
Rect bounds = boundingRect(labelContours[idx]);
float area = contourArea(labelContours[idx]);
// Draw this contour on a new blank image
Mat maskImage = Mat::zeros(boolImage.rows, boolImage.cols, boolImage.type());
drawContours(maskImage, labelContours, idx, Scalar(255,255,255), CV_FILLED);
Mat patchROI = frame(bounds);
Mat maskROI = maskImage(bounds);
}
segmentCount++;
}

OpenCv 2.3 C - How to isolate object inside image

i have an image like:
i want to remove the black rows and cols round the number.
So i want that the result is:
i try this:
void findX(IplImage* imgSrc,int* min, int* max){
int i;
int minFound=0;
CvMat data;
CvScalar maxVal=cvRealScalar(imgSrc->width * 255);
CvScalar val=cvRealScalar(0);
//For each col sum, if sum < width*255 then we find the min
//then continue to end to search the max, if sum< width*255 then is new max
for (i=0; i< imgSrc->width; i++){
cvGetCol(imgSrc, &data, i);
val= cvSum(&data);
if(val.val[0] < maxVal.val[0]){
*max= i;
if(!minFound){
*min= i;
minFound= 1;
}
}
}
}
void findY(IplImage* imgSrc,int* min, int* max){
int i;
int minFound=0;
CvMat data;
CvScalar maxVal=cvRealScalar(imgSrc->width * 255);
CvScalar val=cvRealScalar(0);
//For each col sum, if sum < width*255 then we find the min
//then continue to end to search the max, if sum< width*255 then is new max
for (i=0; i< imgSrc->height; i++){
cvGetRow(imgSrc, &data, i);
val= cvSum(&data);
if(val.val[0] < maxVal.val[0]){
*max=i;
if(!minFound){
*min= i;
minFound= 1;
}
}
}
}
CvRect findBB(IplImage* imgSrc){
CvRect aux;
int xmin, xmax, ymin, ymax;
xmin=xmax=ymin=ymax=0;
findX(imgSrc, &xmin, &xmax);
findY(imgSrc, &ymin, &ymax);
aux=cvRect(xmin, ymin, xmax-xmin, ymax-ymin);
//printf("BB: %d,%d - %d,%d\n", aux.x, aux.y, aux.width, aux.height);
return aux;
}
So i use:
IplImage *my_image = cvLoad....
CvRect bb = findBB(my_image);
IplImage *new_image = cvCreateImage(cvSize(bb.width,bb.height), my_image->depth, 1);
cvShowImage("test",new_image);
it doesn't work good, cause i try to check if in new image there are black rows or cols and they are present. what can i do? can someone help me? (sorry for my english!)
One way to do it is to simply execute the bounding box technique to detect the digit, as illustrated by the image below:
Since your image is already processed the bounding box technique I use is a lot simpler.
After that procedure, all you really need to do is set the ROI (Region of Interest) of the original image to the area defined by the box to achieve the crop effect and isolate the object:
Notice that in the resulting image there is one extra row/column of pixels in the border that are not white. Well, they are not black either. That's because I didn't performed any threshold method to binarize the image to black and white. The code below demonstrates the bounding box technique being executed on a grayscale version of the image.
This is pretty much the roadmap to achieve what you want. For educational purposes I'm sharing the code I wrote using the C++ interface of OpenCV. I'm sure you are capable of converting it to the C interface.
#include <cv.h>
#include <highgui.h>
#include <vector>
int main(int argc, char* argv[])
{
cv::Mat img = cv::imread(argv[1]);
// Convert RGB Mat to GRAY
cv::Mat gray;
cv::cvtColor(img, gray, CV_BGR2GRAY);
// Store the set of points in the image before assembling the bounding box
std::vector<cv::Point> points;
cv::Mat_<uchar>::iterator it = gray.begin<uchar>();
cv::Mat_<uchar>::iterator end = gray.end<uchar>();
for (; it != end; ++it)
{
if (*it) points.push_back(it.pos());
}
// Compute minimal bounding box
cv::RotatedRect box = cv::minAreaRect(cv::Mat(points));
// Draw bounding box in the original image (debug purposes)
//cv::Point2f vertices[4];
//box.points(vertices);
//for (int i = 0; i < 4; ++i)
//{
//cv::line(img, vertices[i], vertices[(i + 1) % 4], cv::Scalar(0, 255, 0), 1, CV_AA);
//}
//cv::imshow("box", img);
//cv::imwrite("box.png", img);
// Set Region of Interest to the area defined by the box
cv::Rect roi;
roi.x = box.center.x - (box.size.width / 2);
roi.y = box.center.y - (box.size.height / 2);
roi.width = box.size.width;
roi.height = box.size.height;
// Crop the original image to the defined ROI
cv::Mat crop = img(roi);
cv::imshow("crop", crop);
cv::imwrite("cropped.png", crop);
cvWaitKey(0);
return 0;
}