Mouse movements for the entire computer can be done using SendInput(MOUSEEVENTF_MOVE), which can be manipulated when the UE4 window is activated.
Given that there are multiple UE4 Windows, it may be necessary to have a transiting service to handle them uniformly, but this may require switching applications or even desktops.
It would be nice if I could send mouse movement information to the specified UE4 process. Unfortunately, keyboard input and mouse clicks can be done with WM_KEYDOWN/WM_LBUTTONDOWN, etc., but mouse movement alone cannot be done with WM_MOUSEMOVE
I read the source code of UE4 and found that it uses VM_INPUT + GetDeviceData for mouse movement events, because the definition of RAWINPUT is invisible, so I can't simulate the message of RAWINPUT. In addition, Microsoft has implemented GlobalAlloc as LocalAlloc So, even if it can be simulated, it cannot be injected into different processes. Even if it can be injected into the process, I can’t modify the usage method of GetDeviceData in the UE4 source code.
I heard that there is a technology called "pixel streaming" in UE4, we can use webrtc to directly operate remote UE4 applications, including mouse movement, I tested it, it is true, but I don’t know what message is sent in webrtc, If I know, maybe I can send it a similar message to operate it.
So, there is any way to operator multiple UE4 processes at same time without webrtc?
I have joined EpicGames and can clone the newest source code, I find WM_MOUSEMOVE has been ignore when "bUsingHighPrecisionMouseInput is true"
Related
I'm currently trying to develop an application to use two mice to perform completely different actions in Windows. However, after having spent couple days on it, I'm starting to wonder if what I want to do is even possible using Windows APIs. As I'm far from being an expert in Windows APIs, I would like to get your opinions to know whether I'm going in the right direction or whether I should try to do it completely differently (maybe developing a driver ?).
Here's what I want to do : Imagine two mice are plugged in my computer. I would like to use the first one as a regular mouse, while the second one would be used to perform completely different actions. For instance, by clicking the second left mouse button, it would open a new tab in Firefox (sending a CTRL+T command to FireFox app) and when clicking the right button, it would send a CTRL+C. Then, by moving the second mouse upwards, it would zoom in, and when moving it downwards, the firefox page would zoom out (so the mouse cursor on screen would remain fix while doing that !). The idea is to recognize as well which application is currently used (which one has mouse/keyboard focus) and perform different actions depending on it. So for instance, the second mouse left click would generate a CTRL+T in FireFox, a CTRL+B in WORD and a CTRL+S in Notepad (in fact, the idea is to parameterize those actions at will). All of that while the first mouse must continue to act just as a regular mouse.
So, it's important to understand that my application will run in the background and will never, per se, interact directly with the user (no GUI as it doesn't require the user to input anything). Its purpose is just to modify the mouse inputs coming from the second mouse and send other inputs(messages) to the application currently being used.
So far, I'm using raw input. I'm able to differentiate which mouse is being used and I'm able to send messages (application specific) to other applications when an action is performed on the second mouse. I'm even able to lock the cursor on screen when the second mouse is moved (so as only the corresponding message is sent to the application of interest !). However, I'm unable to block the button messages sent by the second mouse to the app with the mouse focus. Hence, when clicking on the second mouse right button in Notepad for instance, my specific command ("aaa" for the moment as I'm just trying with letters for sake of simplicity) is sent (and displayed in the notepad window) BUT the contextual Notepad menu opens as well… (hence it's received as well a WM_RBUTTONDOWN message).
My question is then : How can I block the mouse button messages ((WM_RBUTTONDOWN, and so on…) to be received by other applications when the second mouse is used? Is it even possible ? The problem is that (in my understanding) those messages have higher priority over the WM_input messages… So when I read the WM_input message in my application and detects that the button was pressed from the second mouse, it's already too late and the WM_xBUTTONDOWN was already sent !)
I know that using the mouse hooks, I could block those but then, there is no way to differentiate the origin of the message (and of course, detecting which mouse is used is the main point of my application).
I've tried as well using DirectInput8 but it doesn't support anymore the usage of several mice (Windows specifically says to use raw input to this effect).
So, I guess that by know you've gotten that I'm quite lost and have no idea whether what I want to do it even achievable. Any help would be more than welcome.
Looking forward to reading your replies.
I was about to suggest hooks, but then I read that you looked into that already. I guess, the last resort for your problem would be to write your own driver.
After Windows installed the second mouse in it's usual way, you can go to the Device Manager and change the driver of the mouse you want to "repurpose" to your own driver.
Although, developing a driver is probably nothing one will do as a side task in a project.
We have an application built on an older framework (Qt 3.3.5) that we would prefer not to have to attempt to upgrade to recognize touch events.
We recently upgraded it from win32-msvc2005 to win32-msvc2013. For the most part this has gone just fine, but tablets (for testing purposes, a Surface Pro on Windows 8.1) that sent mouse events when compiled against 2005 now send only WM_POINTER events when the touch screen is used in the 2013 compiled application.
I have been completely unable to find a way to get Windows to send me mouse events for touchscreen input again. My research implies that if I register for WM_TOUCH events I should also get mouse events (as indicated by many people on the internet angry about getting mouse events with their WM_TOUCH events), but my (supposedly) successful calls to RegisterTouchWindow don't seem to actually enable the WM_TOUCH events (or any mouse events), and I am still receiving pointer events.
I feel like I must be missing something obvious, especially with not being able to get WM_TOUCH events (which I don't even want, but would supposedly get me mouse events), but whatever it is continues to elude me. (Presumably it would have to be RegisterTouchWindow not being called for the specific hwnd I am actually touching on the screen, but I've gone so far as putting a RegisterTouchWindow call in specifically upon seeing any WM_POINTERUPDATE event for the hwnd that spawned the event, and outputting the result of the Register call upon it returning true and it is returning true, so that seems impossible as the cause.)
I'm also calling DefWindowProc on all WM_TOUCH/GESTURE/POINTER events, which is the only other thing the internet seems to think might be necessary for events to bubble into more basic events correctly. The framework does not call RegisterRawInputDevices and does not attempt to handle WM_INPUT events (which it wouldn't receive anyway thanks to not being registered for raw input). Any events not handled explicitly should fall through to a DefWindowProc call.
Is there even a way that older applications like ours can move to a newer msvc without going through the pain of teaching the framework to correctly handle the various touch protocols? How does an application that worked just fine on msvc2005 using the built in windows touch to mouse events conversion get that functionality back in msvc2013?
Is there any way to figure out where did a mouse event come from?
I mean, if I code a C/C++ program on Windows, and get a mouse click event on it, how can I find if this event come from a mouse driver, a touchpad, or if it was send by an application (mouse event simulation by sending appropriate message like WM_LBUTTONDOWN).
Thanks for any help :)
This is not possible for an application in user mode - mouse events generally don't provide documented info on event source. There is the way to obtain some message extra info by Win32 API function GetMessageExtraInfo but there is no safe way to interpret this data. It is very device specific, undocumented and never guaranteed to ever present.
To solve this task you need to develop your own Mouse Filter driver basing on Windows DDK sample.
Its callback has input parameter MOUSE_INPUT_DATA - structure containing mouse event info. There is the field UnitId:
UnitId Specifies the unit number of the mouse device. A mouse device name has the format \Device\PointerPortN, where the suffix N is the unit number of the device. For example, a device, whose name is \Device\PointerPort0, has a unit number of zero, and a device, whose name is \Device\PointerPort1, has a unit number of one.
GetAsyncKeyState function can be used to check if the button was pressed, and unfortunately SendInput cannot trick this function.
So you can simulate a mouse click, but the program can check if the button was really pressed.
So creating your own mouse driver is better.
I needed a safe way so simulate mouse/keyboard behavior for my bot, and I wrote a detailled article on my blog http://poker-botting.blogspot.fr/2012/11/how-to-simulate-mouse-and-keyboard.html
How to know if a mouse click is simulated or not? When mouse click send by a program or real mouse device
... I'm programming a system detection for a game to avoid bots, autoclicks,etc that only accept legit mouse clicks
This depends a bit on the kind of application you are writing, but if you can, I would watch the cursor movement, not the clicks.
Human mouse movement has non-uniform speeds, reaction times, imprecisions (clicks on different coordinates of your buttons, etc...).
Also, you can defend a gui against bots by randomly requiring an interaction that is hard to script. For example: If scripts depend upon buttons being always in the same position, I would make sure that, while trying to remain intuitive, the dialog should pop up in slightly different positions every time.
Otherwise: There is no way to detect if the mouse is a real one or a really well simulated one. The Windows HID/MacOS/Linux driver layer abstracts away the distinction between Mice, TrackPens, TrackBalls, draw-pads, touch screens... and of course script-mice...
Although the blog post itself is about a different issue, I refer you to Raymond Chen's excellent Old New Thing. In this specific blog post he talks about the validity of message parameters going into an application, but also makes the point that:
There's no point discussing the possibility that the sender of the message is playing tricks and lying to you because (1) your program should just go along with the ruse and respond to fake menu messages as if they were real menu messages, because (2) there's no way to tell that you're being lied to anyway. To detect lying, you'd have to be able to read into the mindset of the programmer who sent you the message.
Essentially the argument is that you should respond to mouse clicks as mouse clicks, regardless of how those clicks were generated.
Is mouse keys simulated mouse input or legit? The point of simulating mouse input is to make them look exactly like real mouse input. If the simulation is doing its job, then your job is impossible. Sorry, that's the blessing & curse of software for you. Here are some more imperfect ideas:
Use GetKeyboardState and verify that the button states are correct. If the message faker is using PostMessage, they will likely not be setting keyboard state and this would indicate fakery.
If you are targeting known applications that are doing the input simulation, detect them and complain. This is not perfect at all for many reasons.
Fuzzy logic, as many other people have suggested.
You need to be creative and figure out the difference between a simulated event and a real one to you, as there is no generalized answer.
It can't be done (reliably (with software alone anyway))
I've used WIN32API calls to read pixels/manipulate the mouse/send keystrokes to automate large portions of video games and other repetitive tasks. You could write a lot of code to analyze the input, but equally smart developers are just going to modify their code to match.
When I first try to automate a mouse click, that's all I'll do. Send a mouse click. And most of the time it works. You might have code that tracks the mouse movement and the entire stack of mouse events that would fire along with a legitimate click and say, 'That wasn't real - we ignore it' but nothing stops the developer from also implementing mouse movements.
The mouse events are more complex than keypresses; but it's essentially the same idea. If you write code that monitors the time between keypresses and determine that I'm sending the '2' key to your application in EXACTLY 250ms intervals, you might decide I'm a bot. But, all I'll do is modify my code to send the keystroke in 250ms + a random value between -25 and 25 ms.
It's a never-ending game of cat and mouse. The best solution is to make tasks non-trivial so simple forms of automation aren't applicable.
The question is a bit thin on details.
Events can be sent directly to controls without moving the mouse so find out where the mouse is when you get the click event and see if it's on the control. Keyboard input requires control focus, so check that too.
For situations where the mouse moves, you won't be able to tell if the mouse movement is recorded and played back. If its scripted then perhaps you could monitor the mouse behavior in the parent panel(s) of the control and use those events and movements to ascertain whether it is real or not. An automated click might appear from nowhere and cause an flurry of unlikely hover, focus events.
Only way it would be possible is with some specialist hardware and software on the mouse itself that sends evidence of the actual mechanical click. Via software this is not possible.
Although in my other answer I mention that you should ideally just respond to clicks as clicks, there is one possibility that could work, depending on how a "programmatic" click is generated.
I am assuming a Windows platform, due to the "vb.net" tag:
With the WinAPI you can send a message to any window in order to simulate, for example, a WM_LBUTTONDOWN event. In this message you would include the X and Y location of the mouse at the time the button was pressed - or where the receiving program expects it to be. When you handle the message you could use the GetCursorPos call to get the actual cursor position. Verify that the current position is close to that in the message, and handle it as a click, otherwise ignore it.
Bear in mind however that the nature of the message queue is such that it could take some time to handle the event, and the mouse can move a long way in a short space of time.
This solution would only work if the "click" is generated by a simple Send/PostMessage. If the application that is generating the click simulates the movement of the mouse also, then you should probably see the other answers :)
You can check the mouse event flags LLMHF_INJECTED and LLMHF_LOWER_IL_INJECTED or utilize input hooks to monitor mouse input data, where malformed data may indicated that input was injected.
https://learn.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-msllhookstruct
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwindowshookexa
I'm not sure there is a way to determine with perfect accuracy whether a mouse click is automated or not. I mean, you could write your own USB driver that sits in between the native mouse driver and the OS and relays only "real" clicks to you. However, even that can be defeated by plugging in a USB device (like a smartphone) that's programmed to send USB packets to host computer.
What are you trying to accomplish that requires you to distinguish between real mouse clicks and fake ones?
Create a statistical learning solution by logging the past X mouse events in your program. When the user clicks on the control, determine the probability based on the last X actions that it's a real click.
Train your solution using real clicks and a large variety of automated scenarios.
This is obviously not a guaranteed-to-work solution and is more for fun than anything else.
It is much more harder than what you think because input macro programs produce legit mouse and keyboard input messages to your game. I don't think there is a way to check if the input message is actually triggered by a physical hardware input (like mouse or keyboard) unless OS provides you with accessibility of input-driver-level.
Since this is specifically for a game, you can see how other games handle this situation. Some of the common methods are,
Check frequency of mouse clicks. (human beings cannot click as fast as programs.)
At random points or when it doubts, use CAPTCHA to verify. (Read this: http://www.threadmeters.com/v-1Vvd/CAPTCHA_The_Obvious_AntiBot_Solution/ )
Use outside monitoring tools to inspect all processes running in a machine to find out programs known for cheating purpose. Steam does this. Check Valve Anti-Cheat System Blizzard's WoW also does the same thing with Warden.
My own advice would be "Use your gameplay system". Since every game has its own rules and gameplay styles, it wouldn't be too hard to detect whether a player is cheating or not. This approach won't be a general solution and it could be silly but if it works for your game, why not? :)
There are two ways you could work around this problem.
Make a new Button with a Text proving that the user isn't AFK and if the user doesn't press the button, kick him or her out. Generate the button in random locations.
If the user isn't moving for a while, (for example, 10 mins) kick him or her out.
I am developing an application for PocketPC. When the application starts the custom function SetScreenOrientation(270) is called which rotates the screen. When the application closes the function SetScreenOrientation(0) is called which restores the screen orientation.
This way the screen orientation isn't restored if the user minimizes the application and this is not acceptable.
Does anyone know where (in which event handlers) should SetScreenOrientation(int angle) be called to set the screen orientation on application start, restore orientation on minimize, set the orientation on maximize and restore the orientation on close?
Actually I don't know which event handler handles the Minimize and Maximize event.
The correct message is WM_SIZE, but Daemin's answer points to the wrong WM_SIZE help topic. Check the wParam. Be careful as your window may be maximized but hidden.
Going from my Windows CE experience you should handle either the WM_SIZE or WM_WINDOWPOSCHANGED messages. If you're working on PocketPC I would suggest you take a look at the WM_WINDOWPOSCHANGED message first because I'm not sure the WM_SIZE has the right parameters that you need.
From the WM_WINDOWPOSCHANGED message's WINDOWPOS structure take a look at the flags member, specifically SWP_SHOWWINDOW and SWP_HIDEWINDOW.
The specific version of the messages that you need to look at vary with what operating system you're using. The Pocket PC OS is built on Windows CE 3.0 (and lower), while Windows Mobile is now built on Windows CE 5.0 (even Windows Mobile 6), but was also built on Windows CE 4. (Source)
So just look under the relevant section in MSDN for the OS that you're writing for.
I don't know what these are called in the C++ world, but in .NET Compact Framework your application form's Resize event would be called when you minimize/maximize a window, and then in the event code you would check the WindowState property of the form to see if its minimized or mazimized.
Altering the state of your PDA from within your application is risky (although there are lots of good reasons to do it), because if your app crashes it will leave the PDA in whatever state it was in. I've done a lot of kiosk-type (full-screen) apps in Windows Mobile, and one of the tricks to doing this effectively is to hide the WM title bar (the top row with the Windows start button) to keep it from flashing up for a split second every time you open a new form. If the app crashes, the windows bar remains invisible until you reset the device, which isn't good. At least with screen rotation the user can restore it manually.
It really depends on the platform, but I'd go with WM_WINDOWPOSCHANGED or the OnShow. It's not wm_size.. That one is not always thrown on all platforms. Casio's don't throw the size event when you'd expect them to. TDS and Symbol's do.
Even though the MSDN is a great sourse for info, remember not all OS's are created equal. In the PPC world the hardware provider gets to create their own OS and sometimes the miss things, or purposfully ignore things.
I've got a platform here (name withheld to protect... well me) that has left and right buttons.. When you press them, you'd expect to be able to catch VK_LEFT, VK_RIGHT.. You'd be wrong. You actually get ';' or ':'. How's that for a kick in the pants.