Strange behaviour of GetQueuedCompletionStatus when used from thread pool worker threads - c++

Question

I've been testing to combine the IO Completion Ports with the worker threads from the Thread Pool and stumbled on a behaviour I can't explain. In particular, while the following code:
int data;
for (int i = 0; i < NUM; ++i)
PostQueuedCompletionStatus(cp, 1, NULL, reinterpret_cast<LPOVERLAPPED>(&data));
{
std::thread t([&] ()
{
LPOVERLAPPED aux;
DWORD cmd;
ULONG_PTR key;
for (int i = 0; i < NUM; ++i)
{
if (!GetQueuedCompletionStatus(cp, &cmd, &key, &aux, 0))
break;
++count;
}
});
t.join();
}
works perfectly fine and receives NUM status notifications (with NUM being large number, 100000 or more), the similar code that uses the thread pool work object that reads one status notification per work item and repost the work item after reading it, fails after reading couple of hundred status notifications. Having the following global variables (please don't mind the names):
HANDLE cport;
PTP_POOL pool;
TP_CALLBACK_ENVIRON env;
PTP_WORK work;
std::size_t num_calls;
std::mutex mutex;
std::condition_variable cv;
bool job_done;
and the callback function:
static VOID CALLBACK callback(PTP_CALLBACK_INSTANCE instance_, PVOID pv_, PTP_WORK work_)
{
LPOVERLAPPED aux;
DWORD cmd;
ULONG_PTR key;
if (GetQueuedCompletionStatus(cport, &cmd, &key, &aux, 0))
{
++num_calls;
SubmitThreadpoolWork(work);
}
else
{
std::unique_lock<std::mutex> l(mutex);
std::cout << "No work after " << num_calls << " calls.\n";
job_done = true;
cv.notify_one();
}
}
the following code:
{
job_done = false;
std::unique_lock<std::mutex> l(mutex);
num_calls = 0;
cport = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 1);
pool = CreateThreadpool(nullptr);
InitializeThreadpoolEnvironment(&env);
SetThreadpoolCallbackPool(&env, pool);
work = CreateThreadpoolWork(callback, nullptr, &env);
for (int i = 0; i < NUM; ++i)
PostQueuedCompletionStatus(cport, 1, NULL, reinterpret_cast<LPOVERLAPPED>(&data));
SubmitThreadpoolWork(work);
cv.wait_for(l, std::chrono::milliseconds(10000), [] { return job_done; } );
}
would report "No more work after ..." after 250 or so calls to GetQueuedCompletionStatus although the NUM was set to 1000000. Even more curious is that setting the wait from 0 to, way, 10 milliseconds would increase the number of successful calls to couple of hundred thousand and would occasionally read all 1000000 notifications. Which I don't really understand since all status notifications were posted before submitting the work object for the first time.
Is it possible that there really is a problem with combining completion ports and a thread pool or is there something wrong in my code? Please don't go into why would I want to do this - I was investigating the possibilities and stumbled on this. In my view it should work and can't figure put what's wrong. Thank you.

Answer 1

I've tried running this code, the issue seems to be the NumberOfConcurrentThreads parameters supplied to CreateIoCompletionPort. Passing 1 means that the first pool thread that executes callback becomes associated with io completion port but since thread pool may execute callback using different thread GetQueuedCompletionStatus will fail when this happens. From documentation:
The most important property of an I/O completion port to consider carefully is the concurrency value. The concurrency value of a completion port is specified when it is created with CreateIoCompletionPort via the NumberOfConcurrentThreads parameter. This value limits the number of runnable threads associated with the completion port. When the total number of runnable threads associated with the completion port reaches the concurrency value, the system blocks the execution of any subsequent threads associated with that completion port until the number of runnable threads drops below the concurrency value.
Although any number of threads can call GetQueuedCompletionStatus for a specified I/O completion port, when a specified thread calls GetQueuedCompletionStatus the first time, it becomes associated with the specified I/O completion port until one of three things occurs: The thread exits, specifies a different I/O completion port, or closes the I/O completion port. In other words, a single thread can be associated with, at most, one I/O completion port.
So to use io completion with thread pool you need to set the number of concurrent threads to the size of the thread pool (that you can set using SetThreadpoolThreadMaximum).
::DWORD const threads_count{1};
cport = ::CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, threads_count);
...
pool = ::CreateThreadpool(nullptr);
::SetThreadpoolThreadMaximum(pool, threads_count);