C++ Get GDI raw text from another Win32 application - c++

Question

I'm using the UI Automation C++ library to get text data from another GUI application.
If the GUI application is developed using a framework that support UI Automation (WinForm, WPF, ...), it is very easy to get the text.
This is an example using the Inspect tool of UI Automation SDK:
Here, test with CCleaner Free application, it can get the text "Microsoft Edge"
We can also using Win32 API to send some message (Send Message) to get text, example: WM_GETTEXT, LVM_GETITEMTEXT, ...
But, I can't get the text from following element:
The whole header area is just an element only, and I can't get the text inside it.
First, I think it is just an image, but it is not. This is the result of using Ranorex Spy:
The plugin that Ranorex using to get text is GDI RawText Plug-In:
For some older technologies like VB6.0, Delphi or MFC based
applications, Ranorex has only limited support in addressing UI
elements in a robust way. With the GDI RawText Plug-In Ranorex
increases object recognition for controls which use the Windows GDI
drawing mechanism to display text on screen.
So, is there any tutorial about getting text from Windows GDI drawing in another application? Currently, I can't found any resource about it
PS: First, I think it use some OCR library to recognize text from image, but it get the text very fast, not look like using OCR library.
Ranorex is a commercial tool, so I think they will not let me know what library that they implement the plugin if I ask them
Update 1:
I capture the header bar of CCleaner application to Paint, and use Ranorex Spy to get text from the image (if it use some OCR library):
There is no RawText inside the selected element, so it does not use OCR to get the text
Update 2:
I write some simple MFC application to test:
void CMFCApplication1Dlg::OnPaint()
{
CPaintDC dc(this);
CRect rcText(20, 20, 300, 300);
wchar_t text[36];
wsprintf(text, L"Hello world");
dc.SetBkColor(RGB(255, 255, 255));
dc.DrawText(text, &rcText, DT_LEFT);
CDialogEx::OnPaint();
}
Using Spy++:
Using Inspect:
Using Ranorex Spy:
Update 3
Try set background color to black as well:
void CMFCApplication1Dlg::OnPaint()
{
CPaintDC dc(this);
CRect rcText(20, 20, 300, 300);
wchar_t text[36];
wsprintf(text, L"Hello world");
dc.SetBkColor(RGB(0, 0, 0));
dc.DrawText(text, &rcText, DT_LEFT);
CDialogEx::OnPaint();
}
So, no OCR engine can be used here. But this is the result of using Ranorex Spy:

Answer 1

Here's how you can quickly figure out if Ranorex uses OCR. Take a screenshot of CCleanerFree and paste it to mspaint and then run Ranorex on mspaint window to see if it can read text.
Here's how to detect if Ranorex uses winapi hooks for DrawText to see what text some app is drawing. Create a simple windows app and DrawText manually (preferably draw black text on black background to eliminate OCR possibility). If Ranorex reads it then they use winapi hooks to read your text.
Here's how read text with OCR.
OCR is pretty fast, it doesn't take seconds to process simple screen text. You could take screenshot of that window and process specific area of interest for you. I used ocrad for similar task and it worked out great.
Here's sample code you could try:
#include <stdint.h>
#include <string>
#include "ocradlib.h"
std::string image_to_text(uint8_t *data, int width, int height)
{
OCRAD_Pixmap pixmap = { 0 };
pixmap.data = data;
pixmap.width = width;
pixmap.height = height;
pixmap.mode = OCRAD_greymap; // OCRAD_greymap --> 1 byte per pixel; 256 level greymap
std::string ret;
OCRAD_Descriptor * const ocrdes = OCRAD_open();
if (!ocrdes || OCRAD_get_errno(ocrdes) != OCRAD_ok)
{
OCRAD_close(ocrdes);
fprintf(stderr, "not enough memory.\n");
return ret;
}
if (OCRAD_set_image(ocrdes, &pixmap, false) < 0)
{
const OCRAD_Errno ocr_errno = OCRAD_get_errno(ocrdes);
OCRAD_close(ocrdes);
return ret;
}
if (OCRAD_set_threshold(ocrdes, -1) < 0 || // auto threshold
OCRAD_recognize(ocrdes, false) < 0) // no layout
{
const OCRAD_Errno ocr_errno = OCRAD_get_errno(ocrdes);
OCRAD_close(ocrdes);
return ret;
}
const int blocks = OCRAD_result_blocks(ocrdes);
for (int b = 0; b < blocks; ++b)
{
const int lines = OCRAD_result_lines(ocrdes, b);
for (int l = 0; l < lines; ++l)
{
const char * s = OCRAD_result_line(ocrdes, b, l);
if (s && s[0])
ret += s;
}
if (b + 1 < blocks)
ret += '\n';
}
OCRAD_close(ocrdes);
return ret;
}
note that this code uses grayscale pixmap, e.g. 1 byte per pixel rectangle. You can try to change the code to use other image types:
/* OCRAD_Pixmap.data is a pointer to image data formed by "height" rows
of "width" pixels each.
The format for each pixel depends on mode like this:
OCRAD_bitmap --> 1 byte per pixel; 0 = white, 1 = black
OCRAD_greymap --> 1 byte per pixel; 256 level greymap (0 = black)
OCRAD_colormap --> 3 bytes per pixel; 16777216 colors RGB (0,0,0 = black) */
enum OCRAD_Pixmap_Mode { OCRAD_bitmap, OCRAD_greymap, OCRAD_colormap };
There are also other libraries that may provide better quality for more complex images, but for simple text like that ocrad was perfect for me.
To read text using WinAPI hooks
Read some tutorials to understand how that works. For example Hooks Overview on MSDN.
Here's a tutorial that shows how that can be done using EasyHook: EasyHook - Installing a remote hook using EasyHook with C++. Note that there are multiple functions that draw text in WinAPI, perhaps you'll need to hook one or all of them depending on your needs: DrawText, TextOut, ExtTextOut, ExtTextOutWarp (and others).