QueueUserAPC function not working, reporting error 31 randomly - c++

Question

The following code uses the QueueUserAPC function to add commands to a dispatcher thread in order to synchronize console output.
#include <Windows.h>
#include <iostream>
constexpr auto fenceName = L"GlobalFence";
constexpr auto dispatchCloser = L"GlobalDispatchStop";
constexpr int threadCount = 5;
DWORD WINAPI pure(LPVOID lpThreadParameter)
{
const HANDLE dispatchCloseEvent = OpenEventW(EVENT_ALL_ACCESS, FALSE, dispatchCloser);
while(WaitForSingleObjectEx(dispatchCloseEvent, INFINITE, TRUE) != WAIT_IO_COMPLETION)continue;
return 0;
}
HANDLE dispatcher;
int main()
{
const HANDLE dispatchCloseEvent = CreateEventW(nullptr, TRUE, FALSE, dispatchCloser);
dispatcher = CreateThread(NULL, 1024, &pure, 0, 0, NULL);
const HANDLE g_FenceEvent = CreateEventW(nullptr, TRUE, FALSE, fenceName);
HANDLE threads[threadCount];
for (int i = 0; i < threadCount; i++)
{
threads[i] = CreateThread(NULL, 1024*1024,
[](LPVOID) -> DWORD
{
DWORD d = QueueUserAPC(([](ULONG_PTR) {std::cout << "thread opened\n"; }), dispatcher, NULL);
if(d == 0)std::cout << GetLastError() << std::endl;
HANDLE a = OpenEventW(EVENT_ALL_ACCESS, FALSE, fenceName);
WaitForSingleObject(a, INFINITE);
d = QueueUserAPC([](ULONG_PTR) {std::cout << "thread released\n"; }, dispatcher, NULL);
if (d == 0)std::cout << GetLastError() << std::endl;//often reports error 31
return 0;
},
0, 0, NULL);
}
Beep(300, 300);//the length of the delay effects the behavior, somehow.
SetEvent(g_FenceEvent);
SetEvent(dispatchCloseEvent);
WaitForMultipleObjects(threadCount, threads, TRUE, INFINITE);
WaitForSingleObject(dispatcher, INFINITE);
SetEvent(dispatchCloseEvent);
for (int i = 0; i < threadCount; i++)
CloseHandle(threads[i]);
CloseHandle(g_FenceEvent);
CloseHandle(dispatchCloseEvent);
}
The code executes correctly about 40% of the time. Sometimes (although this is somewhat rare) the "thread opened" text won't get written to the console the right amount of times, but no error is reported from getLastError()

Answer 1

As soon as the loop in pure() receives its 1st APC notification, the loop breaks and pure() exits, terminating the thread.
Error 31 is ERROR_GEN_FAILURE, and per the QueueUserAPC() documentation:
When the thread is in the process of being terminated, calling QueueUserAPC to add to the thread's APC queue will fail with (31) ERROR_GEN_FAILURE.
If you expect the dispatcher thread to process more than one APC notification, it needs to stay running. You meant to use == instead of != in your loop condition:
while (WaitForSingleObjectEx(dispatchCloseEvent, INFINITE, TRUE) == WAIT_IO_COMPLETION) continue;
That way, if the wait exits due to a queued APC, the loop will go back to waiting for the next APC. The loop will break, exiting pure() to terminate the thread, only when it receives a return value other than an APC notification, such as WAIT_OBJECT_0 when the close event is signaled.
Another problem I see is that you are signaling dispatchCloseEvent too soon, so the dispatcher thread can stop running while the fenced threads are still trying to queue APCs to it. That is why the 2nd call to QueueUserAPC() in each fenced thread fails randomly. You need to wait for all of the fenced threads to finish first, THEN signal the dispatcher to stop running.
SetEvent(g_FenceEvent);
WaitForMultipleObjects(threadCount, threads, TRUE, INFINITE);
SetEvent(dispatchCloseEvent); // <-- move here!
WaitForSingleObject(dispatcher, INFINITE);
Also, all of your threads are leaking the HANDLEs they open with OpenEventW(). You need to call CloseHandle() on them, per the OpenEventW() documentation:
Use the CloseHandle function to close the handle. The system closes the handle automatically when the process terminates. The event object is destroyed when its last handle has been closed.
For that matter, you don't really need OpenEventW() at all. You can pass the existing HANDLEs from main() to each thread via their LPVOID parameter instead:
DWORD WINAPI pure(LPVOID lpThreadParameter)
{
HANDLE dispatchCloseEvent = (HANDLE) lpThreadParameter;
...
return 0;
}
CreateThread(..., &pure, dispatchCloseEvent, ...);
CreateThread(...,
[](LPVOID param) -> DWORD
{
HANDLE a = (HANDLE) param;
...
},
g_fenceEvent, ...);
Or, just use global variables instead.
Either way, with OpenEventW() eliminated, there is no more need to assign names to your events when calling CreateEventW(), thus no longer exposing them to potential interference from outside code.
Also, you are also not closing the dispatcher thread's HANDLE either, like you are closing the fenced threads' HANDLEs.