WriteConsole doesn't work with PowerShell ISE? - c++

Question

WriteConsole does not work with PowerShell ISE.
Neither WriteConsoleW or WriteConsoleA do.
See, for example, this program:
#include <iostream>
#include <Windows.h>
void w() {
DWORD written;
BOOL const success = WriteConsoleW(GetStdHandle(STD_OUTPUT_HANDLE), L"Printed\n", 8, &written, nullptr);
std::wcout << (success ? L"Success" : L"Failure") << L". Wrote " << written << L" characters." << std::endl;
}
void a() {
DWORD written;
BOOL const success = WriteConsoleA(GetStdHandle(STD_OUTPUT_HANDLE), "Printed\n", 8, &written, nullptr);
std::cout << (success ? "Success" : "Failure") << ". Wrote " << written << " characters." << std::endl;
}
int main() {
w();
a();
return 0;
}
Ran from PowerShell (or Command Prompt, or Git Bash), it prints:
Printed
Success (wrote 8 characters)
Printed
Success (wrote 8 characters)
But from PowerShell ISE:
Failure (wrote 0 characters)
Failure (wrote 0 characters)

Answer 1

To provide background information on Bertie Wheen's own helpful answer:
Perhaps surprisingly, the Windows PowerShell ISE does not allocate a console by default. (The console-like UI that the ISE presents is not a true Windows console).
A console is allocated on demand, the first time a console-subsystem program is run in a session (e.g., cmd /c ver)
Even once that has happened, however, interactive console-subsystem programs are fundamentally unsupported (try choice /m "Prompt me", for instance).
Interactively, you can test if a console has been allocated or not with the following command: [Console]::WindowTop; if there's no console, you'll get a The handle is invalid error.
It follows from the above that your program cannot assume that a console is present when run in the ISE.
One option is to simply not support running in the ISE, given that it is:
no longer actively developed
and there are various reasons not to use it (bottom section), notably not being able to run PowerShell (Core) 6+, and the limitations with respect to console-subsystem programs mentioned above.
As for a successor environment: The actively developed, cross-platform editor that offers the best PowerShell development experience is Visual Studio Code with its PowerShell extension.
As for the potential reason for the poor console support in the ISE: zett42 notes:
A possible reason why ISE developers choose not to allocate a console could stem from the historic difficulties of creating a custom, embedded console within an app's own window. Developers had to resort to hackish, unsupported ways of doing that. Only recently (2018) Windows got a dedicated pseudo-console (ConPTY) API.

Answer 2

The reason why is shown by this program:
#include <iostream>
#include <Windows.h>
int main() {
DWORD const file_type = GetFileType(GetStdHandle(STD_OUTPUT_HANDLE));
if (file_type == FILE_TYPE_CHAR) {
std::cout << "char" << std::endl;
} else if (file_type == FILE_TYPE_PIPE) {
std::cout << "pipe" << std::endl;
} else {
std::cout << file_type << std::endl;
}
return 0;
}
When run from PowerShell (or Command Prompt, or Git Bash), it prints:
char
But from PowerShell ISE:
pipe
WriteConsole cannot write through a pipe, and thus fails. The same thing happens when run from PowerShell / Command Prompt / Git Bash if the output is piped.