High DPI scaling, mouse hooks and WindowFromPoint - c++

I have a process that sets a SetWindowsHookEx(WH_MOUSE_LL, , ,) right click hook. My process is set to DPI system aware on Window 10.0.10586 at 150% scaling on both monitors. This was set by calling SetProcessDpiAwareness(PROCESS_SYSTEM_DPI_AWARE).
My problem scenario is e.g. Office 2007 is DPI unaware, so let's place MS Word on the right quarter of the screen. Right click just above the task bar in the bottom right and the mouse hook sends coordinates of 1279, 675 - scaled to Word. Then I right click on visual studio (DPI aware), nearly three quarters across the screen and the mouse hook sends me coordinates of e.g. 1279, 1008 from Visual Studio. So if I clicked higher up the screen I would potentially get the same 1279, 675.
My process is trying to determine which window is at the point by calling the WindowFromPoint API, but this will clearly fail in this scenario as two applications "share" the same point.
Is it possible to force the mouse hook to always send raw physical coordinates instead of those scaled to the DPI unaware application? and if so, how? Alternately, is there some other way of determining the hWnd or processID from the mouse hooks?

Microsoft fixed it in 10.0.14393.
You should have nothing in your client's network now with a lower build number unless they are on LTSB 10.0.10240.
This is the solution: DPI aware screen capture

Because the process is DPI aware, calling GetCursorPos() in the mouse hook callback handler always fetches raw physical coordinates instead of logical coordinates scaled to the application. Simply discard the coordinates passed to the mouse callback.
Added 30/09/2016
It's worth nothing that whilst GetMessagePos seems a likely, candidate it only returns the correct coordinates if the process is not dpi virtualised.
e.g.
VOID MessagePump()
{
MSG messageGet = { 0 };
DWORD dwPos;
POINTS p;
while (GetMessage(&messageGet,NULL,0,0))
{
dwPos = GetMessagePos();
p = MAKEPOINTS( dwPos );
TranslateMessage( &messageGet );
DispatchMessage( &messageGet );
}
}
The Mouse callback handler gets called during the GetMessage() call, but this does not fetch correct physical coordinates where DPI virtualisation is active for the process. e.g. physical x = 1909, y = 1072 comes back as 1091, 612 with 175% scaling, which though arithmetically correct, is not what was required.

Related

How can I send mouse clicks and drags to another program that is in the background?

I am trying to write a program that takes in the visual output from another program (we'll call it P2) and displays it on a screen in a 3d world.
The flow of the image capture is as follows:
P2 -> BitBlt -> OpenGL Texture -> 3d World
I want the user to be able to look around and click on elements in P2 using a crosshair which would be stationary in the middle of the viewport of my program. However, in my implementation it seems that I would need to have 2 cursors - one to control the camera in my 3d world and one to control P2 - which I don't think is trivial.
Additionally, since a window only updates what part of it is visible and I want to have P2 constantly updating in as high resolution as possible, it makes sense to have P2 on a second virtual desktop. This makes the issue of interaction more difficult.
Some solutions I can think of with their downsides:
Make a Compositing Window Manager
Sounds like a lot of work and I haven't been able to find any documentation on how this might be done.
Have the actual cursor over P2 but get the change in cursor position and use this to move the camera in the 3d world
This wouldn't be a linear transformation from planar to spherical coordinates. Additionally, I'm not sure if the cursor could be on a different Desktop to the current Desktop.
I am open to suggestions of alternative capture methods if they would help.
One example would be hooking the DirectX or OpenGL output of P2 and, if necessary, tricking it into rendering while not being active. This might allow P2 to be minimized but would not solve the issue of input.
Maybe it would be possible to hook the input functions of P2? Would this even be advised?
Here are some images to illustrate my program.
Update:
I have implemented SendNotifyMessage() and found that when I sent a WM_LBUTTONDOWN message to the application it became the foreground window. I set WS_EX_NOACTIVATE on P2 to stop this behaviour however, it still steals focus. I then need to Sleep(), presumably until P2 processes the messages, and then use SetForegroundWindow(). Note that SetFocus() and SetActiveWindow() do not result in my program regaining focus (so maybe focus is the wrong word). Are there any methods to remove this delay while keeping the SendNotifyMessage() asynchronous?
Additionally, in trying to use PostMessage() I found that the coordinate transformation was not correct. However, it worked perfectly in SendNotifyMessage(). What could be causing this behaviour?
Relevant Code
After getting the handle to the window I want to capture I run this:
prevWndStyle = GetWindowLong(hwnd, GWL_EXSTYLE);
SetWindowLong(hwnd, GWL_EXSTYLE, prevWndStyle | WS_EX_NOACTIVATE);
After receiving a WM_INPUT message in my WndProc and processing the input I run this code:
if (raw->data.mouse.usButtonFlags & RI_MOUSE_LEFT_BUTTON_DOWN) {
int hitX, hitY;
if (gls->Click(vec3(0.0f, 0.0f, 1.0f), 0, hitX, hitY)) {
LPARAM lParam = MAKELPARAM(hitX, hitY);
SendNotifyMessage(gls->ic.GetHWND(), WM_MOUSEMOVE, 0, lParam);
SendNotifyMessage(gls->ic.GetHWND(), WM_LBUTTONDOWN, MK_LBUTTON, lParam);
SendNotifyMessage(gls->ic.GetHWND(), WM_LBUTTONUP, 0, lParam);
// I want to get rid of these
Sleep(100);
SetForegroundWindow(gls->aw.GetHWND());
}
}
Update 2
I have found that WS_EX_NOACTIVATE stops the switch to the second desktop however when P2 is on the same desktop, P2 is brought to the foreground.
I think you can use the WIN32API to simulate clicks and also to capture window contents, I've seen it done before in different 3d "window" environments and also I've seen it done in things like Termina Server.
So maybe when "they click you, you pass the click forward" (while doing coordinate transformation)?
Some links for your perusal:
https://msdn.microsoft.com/en-us/library/windows/desktop/gg153548%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
http://www.cplusplus.com/forum/windows/24162/

Trap cursor in Window

I wrote my own cameraclass in C++ using DX11.
At the moment I use WM_MOUSEMOVE events to look around in the scene. To prevent the cursur to get out of the window, I call the function SetCursorPos to center the mouse whenever a WM_MOUSEMOVE event occurs.
But if I move my mouse very fast, the cursor gets out of the window.
A solution to that was using the function ClipCursor, but this leads to falter the rotation of the camera when the cursor hits the border of the rect. So ClipCursor solved the original problem, but ended in another.
Do you guys have any solution to that?
For a Windows desktop app, just use ClipCursor giving the rectangle of your window:
RECT rect;
GetClientRect(mWindow, &rect);
POINT ul;
ul.x = rect.left;
ul.y = rect.top;
POINT lr;
lr.x = rect.right;
lr.y = rect.bottom;
MapWindowPoints(mWindow, nullptr, &ul, 1);
MapWindowPoints(mWindow, nullptr, &lr, 1);
rect.left = ul.x;
rect.top = ul.y;
rect.right = lr.x;
rect.bottom = lr.y;
ClipCursor(&rect);
Be sure to have a way to escape this mode so that users can choose to interact with other windows if desired. Typically you call ClipCursor(nullptr); when you go to a 'pause' menu to get out of 'mouse-look' behavior like this.
You can use "raw" input in desktop apps as well, see Taking Advantage of High-Definition Mouse Movement. Keep in mind that raw input works pretty well for relative-movement 'mouse-look' behavior, but (a) it won't work over remote desktop, and (b) you won't get 'pointer-ballistics' which is a non-linear movement rate that makes a mouse more agile so you should generally stick with traditional WM_MOUSE messages when dealing with absolute-movement.
For Universal Windows Platform, you can't use "raw" input as there's no equivalent to WM_INPUT, but you do get high-precision data out of the MouseDevice.MouseMoved event via the MouseDelta property. You don't need to use ClipCursor for relative movement in UWP, just turning off the cursor by setting the CoreWindow.PointerCursor property to nullptr will prevent the system mouse position from being tracked. Again, you should restore the system cursor when you are in a 'pause' menu. See Developing mouse controls (DirectX and C++).
See DirectX Tool Kit Mouse helper class and more importantly the implementation file. For relative mode, it makes use of both ClipCursor and WM_INPUT for the Windows desktop Win32 implementation.
The regular windows message are not the best solution to drive precise controls. They inherit from the OS acceleration system, clipping and depends on other shenanigans as you can see.
The best API to receive mouse inputs is Raw Input. It has the advantage to expose the better dpi and polling rate the hardware can provide and is free of any under the hood manipulation. Once you read the mouse with this, you are free to use SetCapture and clipCursor to prevent unwanted click to an other window.
You can find the documentation here : https://msdn.microsoft.com/en-us/library/windows/desktop/ms645536(v=vs.85).aspx

Direct2D WM_MOUSEMOVE message with scaled display

I am new to Direct2D programming and I have encountered an issue with the WM_MOUSEMOVE message handling.
As documented in MSDN, I should use this enum to handle the mouse move, and should use the LOWORD & HIWORD to extract the current x and y coordinates.
That works fine when I am working on a normal display, but when trying to run it on scaled displays (e.g. 125% in my case), the values of x and y aren't accurate, in other words, there is an "indentation" between the current position of the mouse and the values extracted from lparam.
I guess I should query the OS or the window to get the current scaling so I can calculate the right position, but don't know how!
any help please?
You can take control of the scaling by declaring your program as DPI aware. Then the automatic scaling will stop and you'll get the original coordinates. You'll need to scale the window yourself though.
Creating a DPI-Aware Application

Stop app window overlapping Windows start bar

On my Win7 PC I have the start-bar running vertically, it's about 60px wide. In my win32 application, the created window always appears overlapping the start bar, which looks bad - I just use (0,0) as the top-left position for the window.
How should I be doing it to get (0,0) relative to the desktop, taking the Start Bar into account? Is there a flag, or do I manually need to look up a metric?
There are a few problems here. You don't want to use a hard-coded value like (0,0). That might not even be visible on a multi-monitor system. As you have discovered, you should try to avoid overlapping with the taskbar or other appbars. If there are multiple monitors you should try and start on the monitor where the user has most recently interacted.
There is a simple way to make much of this happen for free. Pass CW_USEDEFAULT as the x and y coordinates when you call CreateWindow. This will let the window manager do the hard work of making your window appear in a sensible location.
You can get the system to tell you the coordinates of the work area. The work area is that part of the desktop that does not contain the taskbar or other appbars. Call SystemParametersInfo passing SPI_GETWORKAREA.
Retrieves the size of the work area on the primary display monitor. The work area is the portion of the screen not obscured by the system taskbar or by application desktop toolbars. The pvParam parameter must point to a RECT structure that receives the coordinates of the work area, expressed in virtual screen coordinates.
To get the work area of a monitor other than the primary display monitor, call the GetMonitorInfo function.
Use SetWindowPlacement. The (0,0) for that function excludes the taskbar and any other appbars.

How do I determine if a window is off-screen?

In Windows XP and above, given a window handle (HWND), how can I tell if the window position and size leaves the window irretrievably off screen? For example, if the title bar is available to the cursor, then the window can be dragged back on screen. I need to discover if the window is in fact visible or at least available to the user. I guess I also need to know how to detect and respond to resolution changes and how to deal with multiple monitors. This seems like a fairly big deal. I'm using C++ and the regular SDK, so please limit your answers to that platform rather than invoking C# or similar.
Windows makes it relatively simple to determine the size of a user's working area on the primary monitor (i.e., the area of the screen not obscured by the taskbar). Call the SystemParametersInfo function and specify the SPI_GETWORKAREA flag for the first parameter (uiAction). The pvParam parameter should point to a RECT structure that will receive the coordinates of the working area in virtual screen coordinates.
Once you've got the coordinates that describe the working area, it's a simple matter of comparing those to the current position of your application's window to determine if it lies within those bounds.
The desire to support multiple monitors makes things slightly more complicated. The documentation for SystemParametersInfo suggests that you need to call the GetMonitorInfo function instead to get the working area of a monitor other than the primary. It fills in a structure called MONITORINFOEX that contains the member rcWork that defines the working area of that monitor, again expressed in virtual screen coordinates as a RECT structure.
To do this right, you'll need to enumerate all of the monitors a user has connected to the system and retrieve the working area of each using GetMonitorInfo.
There are a few samples of this to be found around the Internet:
MSDN has some sample code for Positioning Objects on a Multiple Display Setup.
If you're using MFC, here's what looks to be an excellent example of multiple monitor support.
Even if you're not using MFC, that article refers the following link which looks be a real gem as far as explaining how multiple monitor supports works in Windows, even if it's a little bit old school. Like it or not, very little of this has changed in later versions of Windows.
Finally, you mentioned wanting to detect resolution changes. This is much simpler than you probably imagined. As you know if you've done any Windows programming, the primary way that the operating system communicates with your application is by sending messages to your WindowProc function.
In this case, you'll want to watch for the WM_DISPLAYCHANGE message, which is sent to all windows when the display resolution has changed. The wParam contains the new image depth in bits per pixel; the low-order word of the lParam specifies the horizontal resolution and the high-order word of the lParam specifies the vertical resolution of the screen.
You can use MonitorFromRect or MonitorFromPoint to check if window's top left point or bottom right point isn't contained within any display monitor (off screen).
POINT p;
p.x = x;
p.y = y;
HMONITOR hMon = MonitorFromPoint(p, MONITOR_DEFAULTTONULL);
if (hMon == NULL) {
// point is off screen
}
Visibility check is really easy.
RECT rtDesktop, rtView;
GetWindowRect( GetDesktopWindow(), &rtDesktop );
GetWindowRect( m_hWnd, &rtView );
HRGN rgn = CreateRectRgn( rtDesktop.left, rtDesktop.top, rtDesktop.right, rtDesktop.bottom );
BOOL viewIsVisible = RectInRegion( rgn, &rtView );
DeleteObject(rgn);
You don't have to use RectInRegion, I used for shorten code.
Display, resolution change monitoring is also easy if you handle WM_SETTINGCHANGE message.
http://msdn.microsoft.com/en-us/library/ms725497(v=vs.85).aspx
UPDATE
As #Cody Gray noted, I think WM_DISPLAYCHANGE is more appropriate than WM_SETTINGCHANGE. But MFC 9.0 library make use of WM_SETTINGCHANGE.