AllocConsole Questions - c++

I read the MSDN documentation on AllocConsole, I do not understand one word referring to its purpose. In a very minor DLL tutorial he attaches a DLL to a simple console application and in the DLL, AllocConsole is called. When I remove it from the DLL code, nothing changes during runtime. So I'm curious as to its main purpose:
case DLL_PROCESS_ATTACH:
AllocConsole();
printf("\nInjected Successfully!");
msgBox(true);
break;
This has no difference during runtime as to when I remove AllocConsole.

The MSDN documentation for AllocConsole says:
A process can be associated with only one console, so the AllocConsole
function fails if the calling process already has a console. A process
can use the FreeConsole function to detach itself from its current
console, then it can call AllocConsole to create a new console or
AttachConsole to attach to another console.
So, call it if the process doesn't have a console but you want it to have one. A common example of where you might do this is in a Windows (GUI) application, which does not automatically create and display a console. (Unlike a console application, which does, making AllocConsole rather useless.)

Related

Use PowerShell as a console, as if it was created with AllocConsole

Is this possible? I wish AllocConsole (or a set of function calls behaving the same way) to allocate a PowerShell window instead of a normal console window.
If it's not possible, can I CreateProcess and attach my input and output to a PowerShell instance?
PowerShell also uses a "normal console window" because there is only one console window type. Cmd.exe and PowerShell.exe are not special, they just use the normal Windows console API functions.
If you want to control stdin and stdout then you must create some pipes with CreatePipe and set STARTF_USESTDHANDLES and the handles in STARTUPINFO before calling CreateProcess. See this MSDN example for details. You can write the stdout data to a console window you create with AllocConsole if you also want to display the output. Note that when you redirect stdin/stdout the interactive console functions no longer work.

Win32 application with console output and without new window

I'd like to create a tool that can either act as command line (display some console output based on input parameters), or display a window, based on input parameters.
I'm using MSV2012 with C++, and it seems you have to 'choose' between console and window app.
I know the net is filled with samples which use AllocConsole() and redirect std::out, but it doesn't make it feel like a command line application : calling the exe from the windows console will open a new window with the console output...
Is there a way to have it use the current console window instead of allocating a new one?
If not possible, I'll make 2 applications instead, but that's a pity..
Someone else may have a more authoritative answer, but I don't believe it's supported.
The usual workaround is to create a Windows app, but have a command-line wrapper that launches it from the CLI (and provides a channel for communicating with the original console).
It's not technically supported but I found a good solution by getting a snapshot for the current process, finding the parent process, attaching to it's console if it's a console app or creating one with AllocConsole, redirecting the output, getting the thread of the parent process if it's cmd.exe and suspending it, resuming it just before I exit my app

How can I detect the subsystem used when my program was compiled?

How can I detect whether my Win32 / Win64 program was compiled with /SUBSYSTEM:WINDOWS, or /SUBSYSTEM:CONSOLE?
A preprocessor method would be best if it already exists, but I'd also like to know how to detect it at runtime. It would be sufficient to just detect if there is a console associated with the app, as long as it returns true for console apps.
I'm doing this because I have code to make a console window appear for debugging my GUI apps, but I also build them in console mode (in which case I do not spawn a new console window), and don't really want to make/manage a new #define for this when it seems like there has to already be a way to determine this.
Since in your setting you desire a console in both modes the simplest solution is to call AllocConsole all the time.
A process can be associated with only one console, so the AllocConsole function fails if the calling process already has a console. A process can use the FreeConsole function to detach itself from its current console, then it can call AllocConsole to create a new console or AttachConsole to attach to another console.

Console output window in DLL

I am trying to redirect the output from my DLL to an external console window for easy debugging.
I have been told about AllocConsole but I am not able to reproduce it, i.e. the console window does not appear.
My current environment is Visual Studio 2005.
I tried the following example which is gotten off the Internet,
AllocConsole();
HANDLE han = GetStdHandle(STD_OUTPUT_HANDLE);
WriteConsole(han,"hello",6,new DWORD,0);
yet nothing happens. Can someone point me in the right direction if creating a console window via DLL is possible in the first place.
Thanks in advance!
The proper way to output debug strings is via OutputDebugString(), with an appropriate debugging tool listening for output strings.
Once loaded, there is nothing special about DLLs, so there is no way that allocating consoles would be any different for a DLL than for the EXE that originally loaded it.
Having said that, a process can be associated with only one console at a time, so if there is already a console attached to the process, then allocating a new one is not going to do anything (I assume you're checking the return value of AllocConsole? What does it return? What does GetLastError return?)
There are some other possibilities. For example, if your DLL is loaded into a service, then the service will (likely) be running under a different window station to the currently logged-in user so if you create a console window, you won't be able to see it.

Handling Messages in Console Apps/DLLs in C++ Win32

I would like to have the ability to process Win32 messages in a console app and/or inside a standalone DLL.
I have been able to do it in .NET with the following article and it works great in C# inside a console app and standalone DLL
http://msdn.microsoft.com/en-us/magazine/cc163417.aspx
Is there a way to do the equivalent with C/C++ Win32 APIs? I have tried doing RegisterClassEx(...) and CreateWindow(...) even passing in HWND_MESSAGE to hWndParent but the trouble is that after the "invisible" window is created messages are not being processed probably due to the lack of a message pump.
Where would the message pump go if you had a DLL entry point? I have tried creating another thread in a DLL and put while(GetMesage(..)) there but that did not work either.
Any ideas?
You need a message pump yes. The window also has thread affinity so it needs to be created on the same thread that you're running the message pump on. The basic approach is sound, if you include more code it may become clear what the problem is.
In addition to what Logan Capaldo said, you also have the problem that, as a DLL, you don't know at compile time what kind of process is going to be loading you at runtime.
If you are being loaded by a console application (/SUBSYSTEM:CONSOLE), then creating a hidden window of your own and setting up a message pump on that same thread will work fine (as long as you are the first window created).
If you are being loaded by a windows app (/SUBSYSTEM:WINDOWS) then you might run into problems getting messages. They will be sent to the top-level window in the hierarchy, which you didn't create. You'll need to get the hWnd of the main process and subclass it (if you aren't already).
If you are being loaded by a service, then you aren't going to get window messages at all. You instead need to use the RegisterServiceCtrlHandlerEx Function