how to do pixel analysis on the screen? - c++

I need to save to an array (regular or two-dimensional) the color of each pixel from a certain area. The area should change for example: from 100(x) and 100(y) to 200, 200. Another example: from 0, 0 to 1920, 1080. Then use if and for, where you can specify the desired color and search area. The area may coincide with the area placed in the array, or it may be smaller than in the array. If a color was found, for example 255 (red). Then you need to write down its coordinates, for example in x and y. Please leave comments in the code. I'm new to C++. I use Visual Studio. I need the function to be executed very often. About 30 ms. I understand that as the speed increases, the load increases. Is there any way to reduce the load on the CPU? I use windows.
This is not my code. It doesn't work for me. I don't have enough knowledge to write code that reads from the screen.
#include <Windows.h>
#include <iostream>
using namespace std;
int main() {
int width = GetSystemMetrics(SM_CXSCREEN);
int height = GetSystemMetrics(SM_CYSCREEN);
HDC hdcScreen = GetDC(NULL);
HDC hdcCompatible = CreateCompatibleDC(hdcScreen);
HBITMAP hBitmap = CreateCompatibleBitmap(hdcScreen, width, height);
HGDIOBJ hOld = SelectObject(hdcCompatible, hBitmap);
BitBlt(hdcCompatible, 0, 0, width, height, hdcScreen, 0, 0, SRCCOPY);
BITMAPINFO bmi;
ZeroMemory(&bmi, sizeof(BITMAPINFO));
bmi.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
bmi.bmiHeader.biWidth = width;
bmi.bmiHeader.biHeight = height;
bmi.bmiHeader.biPlanes = 1;
bmi.bmiHeader.biBitCount = 32;
bmi.bmiHeader.biCompression = BI_RGB;
int** pixels = new int*[height];
for (int i = 0; i < height; i++) {
pixels[i] = new int[width];
}
GetDIBits(hdcScreen, hBitmap, 0, height, (LPVOID*)pixels, &bmi, DIB_RGB_COLORS);
SelectObject(hdcCompatible, hOld);
DeleteObject(hBitmap);
DeleteDC(hdcCompatible);
ReleaseDC(NULL, hdcScreen);
return 0;
}
This is suggested to be used to analyze a pixel in an array.
int getPixelColor(int x, int y, int** pixels) {
return pixels[y][x];
}

Related

OpenCV image zoomed in

I'm using this function to make a screenshot of a window. It works fine but at games like Minecraft it's zoomed in.
Normal
Screenshot
cv::Mat getMat(HWND hWND) {
HDC deviceContext = GetDC(hWND);
HDC memoryDeviceContext = CreateCompatibleDC(deviceContext);
RECT windowRect;
GetClientRect(hWND, &windowRect);
int height = windowRect.bottom;
int width = windowRect.right;
HBITMAP bitmap = CreateCompatibleBitmap(deviceContext, width, height);
SelectObject(memoryDeviceContext, bitmap);
//copy data into bitmap
BitBlt(memoryDeviceContext, 0, 0, width, height, deviceContext, 0, 0, SRCCOPY);
//specify format by using bitmapinfoheader!
BITMAPINFOHEADER bi;
bi.biSize = sizeof(BITMAPINFOHEADER);
bi.biWidth = width;
bi.biHeight = -height;
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0; //because no compression
bi.biXPelsPerMeter = 1;
bi.biYPelsPerMeter = 2;
bi.biClrUsed = 3;
bi.biClrImportant = 4;
cv::Mat mat = cv::Mat(height, width, CV_8UC4); // 8 bit unsigned ints 4 Channels -> RGBA
//transform data and store into mat.data
GetDIBits(memoryDeviceContext, bitmap, 0, height, mat.data, (BITMAPINFO*)&bi, DIB_RGB_COLORS);
//clean up!
DeleteObject(bitmap);
DeleteDC(memoryDeviceContext); //delete not release!
ReleaseDC(hWND, deviceContext);
return mat;
}
According to the documentation, windowRect.bottom is not the height. Or windowRect.right is not the width. https://learn.microsoft.com/en-us/windows/win32/api/windef/ns-windef-rect
So, you should do something like this:
int height = windowRect.bottom - windowRect.top;
int width = windowRect.right - windowRect.left;
EDIT1:
I have found an issue similar to yours. Maybe it could help you as well.
ClientRect mysteriously smaller than WindowRect?
Cheers.

opencv + cuda how to record video from the screen at 64 fps?

Good afternoon.
At the moment I cannot find information on how to record video from the screen using opencv + cuda at 64 fps. I have some code that uses cpu but not gpu, so it's about 25 fps. Please help me find information.
#include <iostream>
#include <opencv2/opencv.hpp>
#include <opencv2/cudaarithm.hpp>
#include <opencv2/video.hpp>
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"
#include <Windows.h>
using namespace std;
using namespace cv;
using namespace cv::cuda;
Mat hwnd2mat(HWND hwnd)
{
HDC hwindowDC, hwindowCompatibleDC;
int height, width, srcheight, srcwidth;
HBITMAP hbwindow;
Mat src;
BITMAPINFOHEADER bi;
hwindowDC = GetDC(hwnd);
hwindowCompatibleDC = CreateCompatibleDC(hwindowDC);
SetStretchBltMode(hwindowCompatibleDC, COLORONCOLOR);
RECT windowsize; // get the height and width of the screen
GetClientRect(hwnd, &windowsize);
srcheight = windowsize.bottom;
srcwidth = windowsize.right;
height = windowsize.bottom / 1; //change this to whatever size you want to resize to
width = windowsize.right / 1;
src.create(height, width, CV_8UC4);
// create a bitmap
hbwindow = CreateCompatibleBitmap(hwindowDC, width, height);
bi.biSize = sizeof(BITMAPINFOHEADER); //http://msdn.microsoft.com/en-us/library/windows/window/dd183402%28v=vs.85%29.aspx
bi.biWidth = width;
bi.biHeight = -height; //this is the line that makes it draw upside down or not
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
// use the previously created device context with the bitmap
SelectObject(hwindowCompatibleDC, hbwindow);
// copy from the window device context to the bitmap device context
StretchBlt(hwindowCompatibleDC, 0, 0, width, height, hwindowDC, 0, 0, srcwidth, srcheight, SRCCOPY); //change SRCCOPY to NOTSRCCOPY for wacky colors !
GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.data, (BITMAPINFO*)&bi, DIB_RGB_COLORS); //copy from hwindowCompatibleDC to hbwindow
// avoid memory leak
DeleteObject(hbwindow);
DeleteDC(hwindowCompatibleDC);
ReleaseDC(hwnd, hwindowDC);
return src;
}
int main(int argc, char** argv)
{
HWND hwndDesktop = GetDesktopWindow();
//namedWindow("output", WINDOW_NORMAL);
int key = 0;
while (1)
{
Mat src = hwnd2mat(hwndDesktop);
// you can do some image processing here
imshow("output", src);
key = waitKey(1);
}
}
how to record video from desktop at 64 fps?
I tried to replace MAT with UMAT, which uses mostly gpu, but unfortunately it didn't work because of this function:
GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.data, (BITMAPINFO*)&bi, DIB_RGB_COLORS);
Because the UMAT class has no variable data.
Edit:
Good afternoon, I updated the hwnd2mat function for gpu, but it gives me a black screen for some reason. Help me please.
UMat hwnd2mat(HWND hwnd)
{
HDC hwindowDC, hwindowCompatibleDC;
int height, width, srcheight, srcwidth;
HBITMAP hbwindow;
UMat src;
BITMAPINFOHEADER bi;
hwindowDC = GetDC(hwnd);
hwindowCompatibleDC = CreateCompatibleDC(hwindowDC);
SetStretchBltMode(hwindowCompatibleDC, COLORONCOLOR);
RECT windowsize; // get the height and width of the screen
GetClientRect(hwnd, &windowsize);
srcheight = windowsize.bottom;
srcwidth = windowsize.right;
height = windowsize.bottom / 1; //change this to whatever size you want to resize to
width = windowsize.right / 1;
src.create(height, width, CV_8UC4);
// create a bitmap
hbwindow = CreateCompatibleBitmap(hwindowDC, width, height);
bi.biSize = sizeof(BITMAPINFOHEADER); //http://msdn.microsoft.com/en-us/library/windows/window/dd183402%28v=vs.85%29.aspx
bi.biWidth = width;
bi.biHeight = -height; //this is the line that makes it draw upside down or not
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
// use the previously created device context with the bitmap
SelectObject(hwindowCompatibleDC, hbwindow);
// copy from the window device context to the bitmap device context
StretchBlt(hwindowCompatibleDC, 0, 0, width, height, hwindowDC, 0, 0, srcwidth, srcheight, SRCCOPY); //change SRCCOPY to NOTSRCCOPY for wacky colors !
GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.u->data, (BITMAPINFO*)&bi, DIB_RGB_COLORS); //copy from hwindowCompatibleDC to hbwindow
// avoid memory leak
DeleteObject(hbwindow);
DeleteDC(hwindowCompatibleDC);
ReleaseDC(hwnd, hwindowDC);
return src;
}
CUDA helps to accelerate processing functions, but cannot help in accelerating capture of desktop. In this case, bottleneck is the blit API you are using to capture the screen (GetDIBits). Since you seem to be having a Nvidia GPU, you should look at OS/driver level alternatives like mentioned here (https://developer.nvidia.com/capture-sdk).

C++ Window capture output isn't the same size as said window

I have a short program in C++ to capture the pixels from this BlueStacks emulator window, and then I'll be manipulating the pixels using OpenCV and then sending mouse input (win32api) based on some decisions here and there. I'm working in Visual Studio Enterprise 2017.
Only problem is, the function seems to be capturing a smaller region of pixels compared to the full window.
Here's an example image. Original window is on the left, mirrored output is on the right.
How can I fix this? I've already enabled High DPI awareness in my project settings, not sure what other steps to take other than adding random magic numbers into the function.
I've done this program already in Python but I'd like to redo this in C++ for the performance boost.
Here's a video (warning: noise). And another one.
Here's my current code:
#include "stdafx.h"
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"
#include <Windows.h>
#include <iostream>
using namespace std;
using namespace cv;
Mat hwnd2mat(HWND hwnd);
int main(int argc, char **argv)
{
printf("Hello, world!\n");
Sleep(1000);
HWND hwndDesktop;
hwndDesktop = GetForegroundWindow();
// namedWindow("output", WINDOW_NORMAL);
int key = 0;
Mat src;
while (key != 27)
{
src = hwnd2mat(hwndDesktop);
// you can do some image processing here
imshow("output", src);
key = waitKey(1); // you can change wait time
}
}
Mat hwnd2mat(HWND hwnd)
{
HDC hwindowDC, hwindowCompatibleDC;
int height, width, srcheight, srcwidth;
HBITMAP hbwindow;
Mat src;
BITMAPINFOHEADER bi;
hwindowDC = GetDC(hwnd);
hwindowCompatibleDC = CreateCompatibleDC(hwindowDC);
SetStretchBltMode(hwindowCompatibleDC, COLORONCOLOR);
RECT windowsize; // get the height and width of the screen
GetClientRect(hwnd, &windowsize);
srcheight = windowsize.bottom;
srcwidth = windowsize.right;
height = windowsize.bottom / 1; //change this to whatever size you want to resize to
width = windowsize.right / 1;
src.create(height, width, CV_8UC4);
// create a bitmap
hbwindow = CreateCompatibleBitmap(hwindowDC, width, height);
bi.biSize = sizeof(BITMAPINFOHEADER); //http://msdn.microsoft.com/en-us/library/windows/window/dd183402%28v=vs.85%29.aspx
bi.biWidth = width;
bi.biHeight = -height; //this is the line that makes it draw upside down or not
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
// use the previously created device context with the bitmap
SelectObject(hwindowCompatibleDC, hbwindow);
// copy from the window device context to the bitmap device context
StretchBlt(hwindowCompatibleDC, 0, 0, width, height, hwindowDC, 0, 0, srcwidth, srcheight, SRCCOPY); //change SRCCOPY to NOTSRCCOPY for wacky colors !
GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.data, (BITMAPINFO *)&bi, DIB_RGB_COLORS); //copy from hwindowCompatibleDC to hbwindow
// avoid memory leak
DeleteObject(hbwindow);
DeleteDC(hwindowCompatibleDC);
ReleaseDC(hwnd, hwindowDC);
return src;
}
The stretching is due to DPI scaling. Your own program is not DPI aware, the other program seems to be DPI aware. The easiest way to make your program DPI aware is by calling SetProcessDPIAware(); at the start of the program.
Aside note, HBITMAP handle should not be selected in a device context when calling GetDIBits. You can rewrite your code as
SetProcessDPIAware();
...
Mat hwnd2mat(HWND hwnd)
{
RECT rc;
GetClientRect(hwnd, &rc);
int width = rc.right;
int height = rc.bottom;
Mat src;
src.create(height, width, CV_8UC4);
HDC hdc = GetDC(hwnd);
HDC memdc = CreateCompatibleDC(hdc);
HBITMAP hbitmap = CreateCompatibleBitmap(hdc, width, height);
HBITMAP oldbmp = (HBITMAP)SelectObject(memdc, hbitmap);
BitBlt(memdc, 0, 0, width, height, hdc, 0, 0, SRCCOPY);
SelectObject(memdc, oldbmp);
BITMAPINFOHEADER bi = { sizeof(BITMAPINFOHEADER), width, -height, 1, 32, BI_RGB };
GetDIBits(hdc, hbitmap, 0, height, src.data, (BITMAPINFO*)&bi, DIB_RGB_COLORS);
DeleteObject(hbitmap);
DeleteDC(memdc);
ReleaseDC(hwnd, hdc);
return src;
}
This code will only work for native Win32 programs. For other programs like Chrome, WPF, Qt apps, this will show blank screen. You would need to take screen shot of desktop window instead.
I think you want GetActiveWindow() instead of GetForegroundWindow()
See this SO answer

OPENCV desktop capture Part II

Continuing in previous post from here OPENCV Destop Capture and this hwnd2mat function, a method to capture desktop as source in opencv is create a bitmap from the screen image with this hwnd2mat function. done.
this function is already create a bitmap from the screen image.but it gives weird effect like a trailing inside the video, i just want a normal video source more likely like this one . i've already try to find what cause it, but i don't know anymore.
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"
#include <Windows.h>
#include <iostream>
using namespace std;
using namespace cv;
Mat hwnd2mat(HWND hwnd)
{
HDC hwindowDC, hwindowCompatibleDC;
int height, width, srcheight, srcwidth;
HBITMAP hbwindow;
Mat src;
BITMAPINFOHEADER bi;
hwindowDC = GetDC(hwnd);
hwindowCompatibleDC = CreateCompatibleDC(hwindowDC);
SetStretchBltMode(hwindowCompatibleDC, COLORONCOLOR);
RECT windowsize; // get the height and width of the screen
GetClientRect(hwnd, &windowsize);
srcheight = windowsize.bottom;
srcwidth = windowsize.right;
height = windowsize.bottom / 1; //change this to whatever size you want to resize to
width = windowsize.right / 1;
src.create(height, width, CV_8UC4);
// create a bitmap
hbwindow = CreateCompatibleBitmap(hwindowDC, width, height);
bi.biSize = sizeof(BITMAPINFOHEADER); //http://msdn.microsoft.com/en-us/library/windows/window/dd183402%28v=vs.85%29.aspx
bi.biWidth = width;
bi.biHeight = -height; //this is the line that makes it draw upside down or not
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
// use the previously created device context with the bitmap
SelectObject(hwindowCompatibleDC, hbwindow);
// copy from the window device context to the bitmap device context
StretchBlt(hwindowCompatibleDC, 0, 0, width, height, hwindowDC, 0, 0, srcwidth, srcheight, SRCCOPY); //change SRCCOPY to NOTSRCCOPY for wacky colors !
GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.data, (BITMAPINFO *)&bi, DIB_RGB_COLORS); //copy from hwindowCompatibleDC to hbwindow
// avoid memory leak
DeleteObject(hbwindow);
DeleteDC(hwindowCompatibleDC);
ReleaseDC(hwnd, hwindowDC);
return src;
}
int main(int argc, char **argv)
{
HWND hwndDesktop = GetDesktopWindow();
namedWindow("output", CV_WINDOW_NORMAL);
int key = 0;
while (key != 30 )
{
Mat src = hwnd2mat(hwndDesktop);
// you can do some image processing here
imshow("output", src);
key = waitKey(30); // you can change wait time
}
ReleaseCapture();
destroyAllWindows();
return 0;
}
Using GetDesktopWindow() will make infinite-mirror-effect, instead that, using what window you want capture, and me, i'm using FindWindowEx(), and use Spy++ to find what window you want to capture, that should works. but of course it will go to the next bug :).

Getting RGB data of a minimised window

I want to get RGB data of a particular part of a running window which is minimised and I'm not quite sure how to do it. Here is my piece of code for getting bitmap info of a window for a specified rectangle.
BYTE* LoadBMPFromHandle(HWND hwnd, int leftTopX, int leftTopY, int width, int height)
{
BITMAPINFO bmi;
BITMAPINFOHEADER bmpiHeader;
HDC dc, memdc;
HBITMAP hbWnd;
bmpiHeader.biSize = sizeof(BITMAPINFOHEADER);
bmpiHeader.biPlanes = 1;
bmpiHeader.biBitCount = 24;
bmpiHeader.biCompression = BI_RGB;
bmpiHeader.biSizeImage = 0;
bmpiHeader.biXPelsPerMeter = 3780;
bmpiHeader.biYPelsPerMeter = 3780;
bmpiHeader.biClrUsed = 0;
bmpiHeader.biClrImportant = 0;
bmpiHeader.biWidth = width;
bmpiHeader.biHeight = height;
bmi.bmiHeader = bmpiHeader;
dc = GetDC(hwnd);
memdc = CreateCompatibleDC(dc);
hbWnd = CreateCompatibleBitmap(dc, width, height);
SelectObject(memdc, hbWnd);
BitBlt(memdc, 0, 0, width, height, dc, leftTopX, leftTopY, SRCCOPY);
int paddedWidth = width + (4-(width%4))%4;
BYTE* BGRwnd = new BYTE [paddedWidth*height*3];
if( 0 == GetDIBits(dc, hbWnd, 0, height, BGRwnd, &bmi, DIB_RGB_COLORS) )
{
cout <<"GetDIBits function fail..." << endl;
DeleteDC(memdc);
DeleteObject(hbWnd);
ReleaseDC(hwnd, dc);
exit(0);
}
DeleteDC(memdc);
DeleteObject(hbWnd);
ReleaseDC(hwnd, dc);
return BGRwnd;
}
However, this function would only work if there is no another window overlapping with the one I have a handle to.
hwnd - handle to my window
LoadBMPFromHandle has to retrieve BGR (blue green red) data of a rectangle with upper left coordinates (leftTopX, leftTopY) and dimesions width x height.
Thank You!