OpenCV Detecting Multiple, Rotated, Scaled objects - c++

Question

I've only been using OpenCV for 12 hours or so and haven't been able to solve this problem. The end goal is to take an image and store each character as an entry inside of 6 separate vector2 arrays (5 chars + bubbles)
Additionally, I need to know whether a character is "enlarged" or not.
Link to resources: https://imgur.com/a/lT5HA
As you can tell at any given moment there's a ton of stuff going on, making this a somewhat difficult task. I know that it's possible, though - Robotmon identifies each character with almost 100% accuracy - the only downfall is that the "enlarged" characters get identified 3 times (the distance when discarding duplicates just doesn't work on the big ones due to them being large enough to register multiple times).
All characters are tagged with a single color and characters from the same color group won't appear in the same match.
I'm sure I'm making a ton of errors - I'm not finding much useful information on OpenCV for this usecase. A decent amount is trial and error + looking inside of the files.
For instance, I'm sure that if I were to add all of the characters appearing in a screenshot, searched for all of them, and then compared the "scores" I'd be able to rule out a few false identifications (because the character would be accurately claimed).
To restate my question:
How do I identify every character within the images with no false positives (including small characters being "sent" to the score or transparent characters fading away), all characters identified accurately, and with the enlarged characters identified separately? (And using OpenCL perhaps?)
#include "stdafx.h"
#include <opencv2/opencv.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/core.hpp>
#include <opencv2/features2d.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/xfeatures2d.hpp>
#include <iostream>
#include <stdio.h>
#include <string>
using namespace cv;
using namespace std;
using namespace cv::xfeatures2d;
int MatchFunction();
int main()
{
MatchFunction();
waitKey(0);
return 0;
}
int MatchFunction()
{
Mat Image_Scene = imread("Bubbles.jpg");
Mat image_Object = imread("block_peterpan_s.png");
// Check for invalid input
if (Image_Scene.empty() || image_Object.empty())
{
cout << "Could not open or find the image" << endl;
return 0;
}
// Initiate ORB detector
Ptr<ORB> detector = ORB::create();
//detector->setMaxFeatures(50);
detector->setScaleFactor(1.1);
detector->setNLevels(12);
//detector->setPatchSize(16);
std::vector<KeyPoint> keypoints_object, keypoints_scene;
Mat descriptors_object, descriptors_scene;
// find the keypoints and descriptors with ORB
detector->detect(image_Object, keypoints_object);
detector->detect(Image_Scene, keypoints_scene);
detector->compute(image_Object, keypoints_object, descriptors_object);
detector->compute(Image_Scene, keypoints_scene, descriptors_scene);
//-- Step 3: Matching descriptor vectors with a brute force matcher
//BFMatcher matcher(NORM_HAMMING, true); //BFMatcher matcher(NORM_L2);
//Ptr<BFMatcher> matcher = BFMatcher::create(); //Ptr<ORB> detector = ORB::create();
Ptr<BFMatcher> matcher = BFMatcher::create(NORM_HAMMING, true);
vector<DMatch> matches;
matcher->match(descriptors_object, descriptors_scene, matches);
//matcher.match(descriptors_object, descriptors_scene, matches);
vector<DMatch> good_matches;
//vector<Point2f> featurePoints1;
//vector<Point2f> featurePoints2;
//Sort the matches by adding them 1 by 1 to good_matches
//for (int i = 0; i<int(matches.size()); i++) { //Size is basically length
// good_matches.push_back(matches[i]);
//}
string k = to_string((matches.size()));
cout << k << endl;
//cout << " Usage: ./SURF_FlannMatcher <img1> <img2>" << std::endl;
double max_dist = 0; double min_dist = 100;
//-- Quick calculation of max and min distances between keypoints
for (int i = 0; i < int(matches.size()); i++)
{
//cout << to_string(i) << endl;
double dist = matches[i].distance;
if (dist < min_dist) min_dist = dist;
if (dist > max_dist) max_dist = dist;
}
//printf("-- Max dist : %f \n", max_dist);
//printf("-- Min dist : %f \n", min_dist);
//-- Draw only "good" matches (i.e. whose distance is less than 2*min_dist,
//-- or a small arbitary value ( 0.02 ) in the event that min_dist is very
//-- small)
//-- PS.- radiusMatch can also be used here.
//std::vector< DMatch > good_matches;
for (int i = 0; i < int(matches.size()); i++)
{
if (matches[i].distance <= max(4 * min_dist, 0.02))
{
good_matches.push_back(matches[i]);
}
}
Mat img_matches;
drawMatches(image_Object, keypoints_object, Image_Scene, keypoints_scene,
good_matches, img_matches, Scalar::all(-1), Scalar::all(-1),
vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS);
imshow("Good Matches", img_matches);
}

Answer 1

Template matching would be the most naive approach, but you'd have to brute-force the same template at different scales and rotations for each object. If you have the information about the possible number of such scales/rotations in the game, this will narrow down the number of iterations drastically.
The image seems free from noise, distortion or occlusions, so machine learning approach is not really necessary in this case. If you want something more efficient and are familiar with scientific language and implementing algorithms, take a look at this study which uses radial and circular filters to narrow down the number of combinations:
http://pdfs.semanticscholar.org/c965/0f78bf9d18eba3850281841dc7ddd20e8d5e.pdf
The algorithm or more specifically the filters can be parallelized with OpenCL or any other library if needed. On modern machines this shouldn't be necessary as serial implementation works quite fast.
I successfully implemented it some time ago and it works well and is fast enough to solve your problem with near-realtime performance. RGB won't be neccessary. To my best knowledge it is not implemented anywhere as open-source code, but you can also try looking up for scale and rotation invariant template matching and see what comes up.