Winapi Timer callback thread, never returns - c++

Question

I've got to debug some code which is not from me.
This code implement a timer API using winapi Timer interface.
I'm not very used to this Winapi functionality, so i could use your help :)
From what I understand this code is done like this :
=> Init()
timerQueue = CreateTimerQueue();
=> CreateTimer()
CreateTimerQueueTimer(timerHandle, timerQueue, timerCallback, ..., WT_EXECUTEDEFAULT);
=> timerCallback()
DeleteTimerQueueTimer(timerQueue , timerHandle, NULL));
calback() //Launch user-defined callback
=> CleanUp() // to be called at the end
DeleteTimerQueueEx(timerQueue , INVALID_HANDLE_VALUE);
When we test that, user-defined callback are executed successfully after the desired amount of time. But after that timerCallback threads keep pending and never return, preventing the all process to returns. Using VS debugger I can see those threads (named TppWorkerThread#4) on the thread...
Perhaps we miss something to make callback returns properly or we created some sort of deadlocks... However I cannot figure it out ...
Please let me know if I forgot some relevant information.
Thank you for your help.
EDIT:
Further information :
- Blocking thread are at this state at the end of the process :
* Category :Worker Thread
* Name : _TppWorkerThread#4
* Location : _ZwWaitForWorkViaWorkerFactory#8
* Priotity : Normal
EDIT2:
Having some more time to work on that strange behavior, I am now able to reproduce it in a standalone code.
#include <windows.h>
#include <stdio.h>
HANDLE gDoneEvent;
HANDLE hTimer[5];
HANDLE hTimerQueue = NULL;
HANDLE g_threadHandle;
void PeriodicCallback(void)
{
printf("Periodic routine called.\n");
}
void SingleCallback(void)
{
printf("Single routine called.\n");
if (!DeleteTimerQueueTimer(hTimerQueue, hTimer[2], NULL))
printf("DeleteTimerQueueTimer() fail. Return value is %d.\n", GetLastError());
}
void CALLBACK CommonCallback(PVOID lpParam, BOOLEAN TimerOrWaitFired)
{
printf("Common routine called. Parameter is %d.\n", *(int *)lpParam);
((void (*)(void))lpParam)();
}
void MainTest(void)
{
// Use an event object to track the TimerRoutine execution
gDoneEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (NULL == gDoneEvent)
{
printf("CreateEvent failed (%d)\n", GetLastError());
return -1;
}
if(0 == SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_BELOW_NORMAL))
{
printf("SetThreadPriority failed (%d)\n", GetLastError());
return -2;
}
// Create the timer queue.
hTimerQueue = CreateTimerQueue();
if (NULL == hTimerQueue)
{
printf("CreateTimerQueue failed (%d)\n", GetLastError());
return -3;
}
/*
if (!CreateTimerQueueTimer( &hTimer[2], hTimerQueue,
(WAITORTIMERCALLBACK)CommonCallback, &SingleCallback, 1000, 0, WT_EXECUTEDEFAULT))
{
printf("CreateTimerQueueTimer failed (%d)\n", GetLastError());
return -4;
}
*/
if (!CreateTimerQueueTimer( &hTimer[4], hTimerQueue,
(WAITORTIMERCALLBACK)CommonCallback, &PeriodicCallback, 10, 500, WT_EXECUTEDEFAULT))
{
printf("CreateTimerQueueTimer failed (%d)\n", GetLastError());
return -5;
}
// TODO: Do other useful work here
printf("Call timer routine in 10 seconds...\n");
Sleep(4000);
CloseHandle(gDoneEvent);
if (!DeleteTimerQueueTimer(hTimerQueue, hTimer[4], INVALID_HANDLE_VALUE))
printf("DeleteTimerQueueTimer failed (%d)\n", GetLastError());
// Delete all timers in the timer queue.
if (!DeleteTimerQueueEx(hTimerQueue, INVALID_HANDLE_VALUE))
printf("DeleteTimerQueue failed (%d)\n", GetLastError());
Sleep(1000);
ExitThread(0);
}
int main(int argc, char **argv[])
{
if(g_threadHandle == CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)MainTest, NULL, 0, NULL))
printf("Creation fail");
ExitThread(0);
}
I'm compiling this code on VisualStudio 2010 Professional.
It appears that event after calling DeleteTimerQueueTimer() some threads remain pending on the threads pool, preventing my process to shutdown. I still cannot figured it out ...

Answer 1

When you call DeleteTimerQueueEx with an INVALID_HANDLE_VALUE as its second parameter it will block until all callbacks which are running have completed. The Error may be in one of your callback functions which never returns.
You are calling DeleteTimerQueueTimer(timerQueue , timerHandle, NULL); with NULL as the third parameter, this will not wait for the callback to complete if one is running at the time you delete the timer. I suggest using DeleteTimerQueueTimer(timerQueue , timerHandle, INVALID_HANDLE_VALUE) which will block until the call back completes (if one is running). Calling cleanUp() without using the blocking version of DeleteTimerQueueTimer is likely a bug as you may be cleaning up at the same time as the callback is executing.
It could also be a problem of calling DeleteTimerQueueEx or DeleteTimerQueueTimer from within a callback, which is forbidden. Break on execution of DeleteTimerQueueEx and look at what thread you are in, if its a TppWorkerThread than you have found your bug.
EDIT:
In your comment you say you do call DeleteTimerQueueTimer from within the callback but don't use INVALID_HANDLE_VALUE, reading the documentation again from http://msdn.microsoft.com/en-us/library/windows/desktop/ms682569%28v=vs.85%29.aspx this does seem to be legal but I distinctly remember us making design decisions to avoid this, I'm sorry this is so vague, I hope someone can give authoritative advice on this.
We send an event/message to the queue of a non timer thread which then removes the timer, you could even have a dedicated thread for this but that is probably overkill. At the end of the day you need to be sure that the timer is removed before doing cleanup so you have to either block on removal or have some other thread do it upon signaling of an event.

Answer 2

After some work on that issue I think I got to an answer.
I appears that this timerQueue API is coded on top of threadPool winAPI, and when we ask to create a timerQueue Windows create a thread pool from where all callback will be launched.
Until here no problem, but, when we ask for timerQueue deletion, it appears that this thread pool is not deleted...
This result in some thread keeping pending waiting to be used and preventing the process to returns.
After some time (timeout??) those threads returns and the process exit.
I don't really get why this pool is not closed... but, now, I use a workaround :
exit(0);
At the end of my program, it's a bit brutal but it does the job (ie: killing my process, whatever threads are still pending or not)