How to conditionally enable console without opening separate console window - c++

Question

I'd like to create a windows application that, under normal conditions, does not have any connected terminal, but may have one in some conditions. I tried two different routes. Option A, creating a normal console application, and conditionally calling FreeConsole():
int main()
{
if (someCondition) {
HANDLE stdOutHandle = GetStdHandle(STD_OUTPUT_HANDLE);
WriteConsoleA(stdOutHandle, "hello world\n", 12, NULL, NULL);
} else {
FreeConsole();
// normal operation
}
return 0;
}
And option B, creating a WinMain based application, and conditionally calling AllocConsole().
int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PSTR pCmdLine, int nCmdShow)
{
if (someCondition) {
AllocConsole();
HANDLE stdOutHandle = GetStdHandle(STD_OUTPUT_HANDLE);
WriteConsoleA(stdOutHandle, "hello world\n", 12, NULL, NULL);
}
else {
// normal operation
}
return 0;
}
The problem with option A, is that it doesn't act exactly like a windows application, in that you can very briefly see a console window open up before normal operation continues. This isn't a huge problem, but I'd prefer the program to act, as I said, exactly like a normal windows application.
The problem with option B is that, if the program is invoked from an existing terminal, it opens up a separate terminal window and outputs to that, instead of outputting to the terminal from which it was invoked. This is a much bigger problem.
What is the appropriate solution to conditionally behave as either a console program or a windows program, without either of the problems described above?
Edit
Based on a suggestion in the comments, I tried to use the AttachConsole function.
int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PSTR pCmdLine, int nCmdShow)
{
if (someCondition) {
AttachConsole(ATTACH_PARENT_PROCESS);
HANDLE stdOutHandle = GetStdHandle(STD_OUTPUT_HANDLE);
Sleep(3000);
WriteConsoleA(stdOutHandle, "hello world\n", 12, NULL, NULL);
FreeConsole();
}
else {
// normal operation
}
return 0;
}
This seems to be on the right track, but there is still something missing, because my shell prompt is immediately printed out without waiting for the program to finish.

Answer 1

Impossible.
AttachConsole does not work 100% because cmd.exe actually checks if the process it is about to start is a GUI app or not, and alters its behavior.
The only way to make it work is to have two programs; myapp.com and myapp.exe. %PathExt% lists .com before .exe so you can make a console .exe and rename it .com at it will be executed if the user runs "myapp" (but will not work if the user types "myapp.exe"). If your program is not very big you can just ship two versions. If the program is large you can move most of the code to a .dll or make the .com a small helper that calls the .exe with a command line parameter and some pipes for stdin/stdout. Visual Studio does this (devenv.com).

Answer 2

If you want to check if there is already a console, check for a NULL return from GetConsoleWindow(). If it returns null, then do the AllocConsole procedure and other setup (SetWindowLong, SetConsoleMode etc)
int main() programs don't make a console by default, they happen to attach into a console if run through cmd/ps.

Answer 3

One way is the following:
BOOL WINAPI
AttachOrCreateConsole()
{
if(AttachConsole(ATTACH_PARENT_PROCESS))
{
return TRUE;
}
return AllocConsole();
}
This will use the parent console if available and fall back to creating one if needed.
Note that if yo do this and want to use the standard C/C++ I/O mechanisms (stdin/std::cin, stdout/std::cout) you will need to do the work needed to associate those streams to the console handle. There is plenty of material available on how to do that.