I have an application, which has registered handle routine via SetConsoleCtrlHandler . Thing is on break signal I have to do some cleanup before exiting, and that is when problems begin. Because, other threads keeps on running and using some of the resources which should freed I got access violation during cleanup. I tried to save thread instances which are using these resources and stop them before calling cleanup routine, but this did not help. I constantly get ERROR_INVALID_HANDLE. I tried to get thread instance via DuplicateHandle also I tried to save only thread id and then get thread handle via OpenThread function. But still invalid handle error is displayed.
Also, when I enumerate all thread instance and trying to terminate them I get the same error. It seems to me like there are some "magic" around handle routine, which does not allow to terminate other threads from it.
Listing for enumerating all threads
HANDLE h = CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, 0);
if (h != INVALID_HANDLE_VALUE) {
THREADENTRY32 te;
te.dwSize = sizeof(te);
if (Thread32First(h, &te)) {
do {
if (te.dwSize >= FIELD_OFFSET(THREADENTRY32, th32OwnerProcessID) +
sizeof(te.th32OwnerProcessID)) {
HANDLE thread = OpenThread(THREAD_TERMINATE, FALSE, te.th32ThreadID);
if (TerminateThread(thread , NUM_SIG_ACTION_1) == 0)
{
// printf("terminate failed (%d)\n", GetLastError());
if( SuspendThread(thread ) == -1)
{
// printf("suspend failed (%d)\n", GetLastError());
}
}
}
te.dwSize = sizeof(te);
} while (Thread32Next(h, &te));
}
CloseHandle(h);
}
Listing for get thread via DuplicateHandle
//code in working thread
DuplicateHandle(GetCurrentProcess(), GetCurrentThread(), GetCurrentProcess(), &g_hWorkerThread ,
DUPLICATE_SAME_ACCESS, FALSE, DUPLICATE_SAME_ACCESS);
//code in handler routine
if (TerminateThread( g_hWorkerThread , NUM_SIG_ACTION_1 ) == 0)
{
printf("Terminate thread failed (%d)\n", GetLastError());
}
Listing for OpenThread
//code from worker thread
g_iWorkerThreadId = GetThreadId(GetCurrentThread());
//code in routine handler
HANDLE handle = OpenThread(THREAD_TERMINATE, FALSE, g_iWorkerThreadId);
if (TerminateThread( handle , NUM_SIG_ACTION_1 ) == 0)
{
printf("Terminate thread failed (%d)\n", GetLastError());
}
Is there anyway to force handle routine run in to current worker thread?
Any help will be appreciated.
Thanks on advance.
You're trying to do something yourself that the OS already does. Don't fight the system.
Let me suggest another way of working, that could simplify your code considerably.
When your main application thread gets the Ctrl signal, it wants to shut down the application.
You can have Windows do that for you, but you might leave resources in an undefined state, so that's why you want to shut down gracefully.
Likely, each thread already runs an event loop - that's the basic Windows applications paradigm. Upon the ctrl-signal, post events to your other threads and catch them in each thread's event loop. Add custom event code that will make them gracefully end themselves. You now only have to wait for all the thread handles to ensure all threads have ended.
Using TerminateThread is not a good idea, it is exactly the opposite of the graceful end of a thread.
How one can sure calling TerminateThread will free the resource that is in use?
Only killing a process guarantees freeing of kernel resources used by the process.
In my humble experiences, I didn't found any multi-thread program runs good and stably without the clear and explicit exit mechanism.
Each thread must have an exit condition, when exiting it must free the resources, and the main thread (or controlling thread) must trigger the exit conditions and wait until all the other thread are exit.
Related
I'm doing some event handling with C++ and pthreads. I have a main thread that reads from event queue I defined, and a worker thread that fills the event queue. The queue is of course thread safe.
The worker thread have a list of file descriptors and create an epoll system call to get events on those file descriptors. It uses epoll_wait to wait for events on the fd's.
Now the problem. Assuming I want to terminate my application cleanly, how can I cancel the worker thread properly? epoll_wait is not one of the cancellation points of pthread(7) so it cannot react properly on pthread_cancel.
The worker thread main() looks like this
while(m_WorkerRunning) {
epoll_wait(m_EpollDescriptor, events, MAXEVENTS, -1);
//handle events and insert to queue
}
The m_WorkerRunning is set to true when the thread starts and it looks like I can interrupt the thread by settings m_WorkerRunning to false from the main thread. The problem is that epoll_wait theoretically can wait forever.
Other solution I though about is: instead of waiting forever (-1) I can wait for example X time slots, then handle properly no-events case and if m_WorkerRunning == false then exit the loop and terminate the worker thread cleanly. The main thread then sets m_WorkerRunning to false, and sleeps X. However I'm not sure about the performance of such epoll_wait and also not sure what would be the correct X? 500ms? 1s? 10s?
I'd like to hear some experienced advises!
More relevant information: the fd's I'm waiting events on, are devices in /dev/input so technically I'm doing some sort of input subsystem. The targeted OS is Linux (latest kernel) on ARM architecture.
Thanks!
alk's answer above is almost correct. The difference, however, is very dangerous.
If you are going to send a signal in order to wake up epoll_wait, never use epoll_wait. You must use epoll_pwait, or you might run into a race with your epoll never waking up.
Signals arrive asynchronously. If your SIGUSR1 arrives after you've checked your shutdown procedure, but before your loop returns to the epoll_wait, then the signal will not interrupt the wait (as there is none), but neither will the program exit.
This might be very likely or extremely unlikely, depending on how long the loop takes in relation to how much time is spent in the wait, but it is a bug one way or the other.
Another problem with alk's answer is that it does not check why the wait was interrupted. It might be any number of reasons, some unrelated to your exit.
For more information, see the man page for pselect. epoll_pwait works in a similar way.
Also, never send signals to threads using kill. Use pthread_kill instead. kill's behavior when sending signals is, at best, undefined. There is no guarantee that the correct thread will receive it, which might cause an unrelated system call to be interrupted, or nothing at all to happen.
You could send the thread a signal which would interupt the blocking call to epoll_wait(). If doing so modify your code like this:
while(m_WorkerRunning)
{
int result = epoll_wait(m_EpollDescriptor, events, MAXEVENTS, -1);
if (-1 == result)
{
if (EINTR == errno)
{
/* Handle shutdown request here. */
break;
}
else
{
/* Error handling goes here. */
}
}
/* Handle events and insert to queue. */
}
A way to add a signal handler:
#include <signal.h>
/* A generic signal handler doing nothing */
void signal_handler(int sig)
{
sig = sig; /* Cheat compiler to not give a warning about an unused variable. */
}
/* Wrapper to set a signal handler */
int signal_handler_set(int sig, void (*sa_handler)(int))
{
struct sigaction sa = {0};
sa.sa_handler = sa_handler;
return sigaction(sig, &sa, NULL);
}
To set this handler for the signal SIGUSR1 do:
if (-1 == signal_handler_set(SIGUSR1, signal_handler))
{
perror("signal_handler_set() failed");
}
To send a signal SIGUSR1 from another process:
if (-1 == kill(<target process' pid>, SIGUSR1))
{
perror("kill() failed");
}
To have a process send a signal to itself:
if (-1 == raise(SIGUSR1))
{
perror("raise() failed");
}
I am implementing multithreaded C++ program for Linux platform where I need a functionality similar to WaitForMultipleObjects().
While searching for the solution I observed that there are articles that describe how to achieve WaitForMultipleObjects() functionality in Linux with examples but those examples does not satisfy the scenario that I have to support.
The scenario in my case is pretty simple. I have a daemon process in which the main thread exposes a method/callback to the outside world for example to a DLL. The code of the DLL is not under my control. The same main thread creates a new thread "Thread 1". Thread 1 has to execute kind of an infinite loop in which it would wait for a shutdown event (daemon shutdown) OR it would wait on the data available event being signaled through the exposed method/callback mentioned above.
In short the thread would be waiting on shutdown event and data available event where if shutdown event is signaled the wait would satisfy and the loop would be broken or if data available event is signaled then also wait would satisfy and thread would do business processing.
In windows, it seems very straight forward. Below is the MS Windows based pseudo code for my scenario.
//**Main thread**
//Load the DLL
LoadLibrary("some DLL")
//Create a new thread
hThread1 = __beginthreadex(..., &ThreadProc, ...)
//callback in main thread (mentioned in above description) which would be called by the DLL
void Callbackfunc(data)
{
qdata.push(data);
SetEvent(s_hDataAvailableEvent);
}
void OnShutdown()
{
SetEvent(g_hShutdownEvent);
WaitforSingleObject(hThread1,..., INFINITE);
//Cleanup here
}
//**Thread 1**
unsigned int WINAPI ThreadProc(void *pObject)
{
while (true)
{
HANDLE hEvents[2];
hEvents[0] = g_hShutdownEvent;
hEvents[1] = s_hDataAvailableEvent;
//3rd parameter is set to FALSE that means the wait should satisfy if state of any one of the objects is signaled.
dwEvent = WaitForMultipleObjects(2, hEvents, FALSE, INFINITE);
switch (dwEvent)
{
case WAIT_OBJECT_0 + 0:
// Shutdown event is set, break the loop
return 0;
case WAIT_OBJECT_0 + 1:
//do business processing here
break;
default:
// error handling
}
}
}
I want to implement the same for Linux. According to my understanding when it would come to Linux, it has totally different mechanism where we need to register for signals. If the termination signal arrives, the process would come to know that it is about to shutdown but before that it is necessary for the process to wait for the running thread to gracefully shutdown.
The correct way to do this in Linux would be using condition variables. While this is not the same as WaitForMultipleObjects in Windows, you will get the same functionality.
Use two bools to determine whether there is data available or a shutdown must occur.
Then have the shutdown function and the data function both set the bools accordingly, and signal the condition variable.
#include <pthread.h>
pthread_cond_t cv = PTHREAD_COND_INITIALIZER;
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_t hThread1; // this isn't a good name for it in linux, you'd be
// better with something line "tid1" but for
// comparison's sake, I've kept this
bool shutdown_signalled;
bool data_available;
void OnShutdown()
{
//...shutdown behavior...
pthread_mutex_lock(&mutex);
shutdown_signalled = true;
pthread_mutex_unlock(&mutex);
pthread_cond_signal(&cv);
}
void Callbackfunc(...)
{
// ... whatever needs to be done ...
pthread_mutex_lock(&mutex);
data_available = true;
pthread_mutex_unlock(&mutex);
pthread_cond_signal(&cv);
}
void *ThreadProc(void *args)
{
while(true){
pthread_mutex_lock(&mutex);
while (!(shutdown_signalled || data_available)){
// wait as long as there is no data available and a shutdown
// has not beeen signalled
pthread_cond_wait(&cv, &mutex);
}
if (data_available){
//process data
data_available = false;
}
if (shutdown_signalled){
//do the shutdown
pthread_mutex_unlock(&mutex);
return NULL;
}
pthread_mutex_unlock(&mutex); //you might be able to put the unlock
// before the ifs, idk the particulars of your code
}
}
int main(void)
{
shutdown_signalled = false;
data_available = false;
pthread_create(&hThread1, &ThreadProc, ...);
pthread_join(hThread1, NULL);
//...
}
I know windows has condition variables as well, so this shouldn't look too alien. I don't know what rules windows has about them, but on a POSIX platform the wait needs to be inside of a while loop because "spurious wakeups" can occur.
If you wish to write unix or linux specific code, you have differenr APIs available:
pthread: provides threads, mutex, condition variables
IPC (inter process comunication) mechanisms : mutex, semaphore, shared memory
signals
For threads, the first library is mandatory (there are lower level syscalls on linux, but it's more tedious). For events, the three may be used.
The system shutdown event generate termination (SIG_TERM) and kill (SIG_KILL) signals broadcasted to all the relevant processes. Hence an individual daemon shutdown can also be initiated this way. The goal of the game is to catch the signals, and initiate process shutdown. The important points are:
the signal mechanism is made in such a way that it is not necessary to wait for them
Simply install a so called handler using sigaction, and the system will do the rest.
the signal is set to the process, and any thread may intercept it (the handler may execute in any context)
You need therefore to install a signal handler (see sigaction(2)), and somehow pass the information to the other threads that the application must terminate.
The most convenient way is probably to have a global mutex protected flag which all your threads will consult regularily. The signal handler will set that flag to indicate shutdown. For the worker thread, it means
telling the remote host that the server is closing down,
close its socket on read
process all the remaining received commands/data and send answers
close the socket
exit
For the main thread, this will mean initiating a join on the worker thread, then exit.
This model should not interfer with the way data is normally processed: a blocking call to select or poll will return the error EINTR if a signal was caught, and for a non blocking call, the thread is regularily checking the flag, so it does work too.
I am writing a multi-threaded networked application, and I'm using a separate thread with a blocking socket to receive data asynchronously from the server.
When I need to shutdown the socket I use a function which checks if the receive thread is still running and if it is calls TerminateThread to end it as follows:
DWORD dwExitCode = 0;
if( GetExitCodeThread( theApp.m_hRecvThread, &dwExitCode ) && dwExitCode == STILL_ACTIVE )
TerminateThread( theApp.m_hRecvThread, 0 );
However, GetExitCodeThread returns FALSE, and when polling GetLastError() it returns 6 (ERROR_INVALID_HANDLE). Which suggests that I do not have the THREAD_QUERY_INFORMATION or THREAD_QUERY_LIMITED_INFORMATION access rights on the m_hRecvThread handle.
My m_hRecvThread handle is set when creating the thread like so:
m_hRecvThread = AfxBeginThread( RecvThread, hWndMainFrame );
This successfully creates the thread, and the thread is running fine and exhibiting expected functionality. The TerminateThread and GetExitCodeThread are being called from the same thread which created the Receive thread in the first place.
My understanding was that when using AfxBeginThread, the HANDLE returned had THREAD_ALL_ACCESS access rights, is this the case, and if so, why am I still getting ERROR_INVALID_HANDLE?
Thanks in advance!
The thread created using:
m_hRecvThread = AfxBeginThread( RecvThread, hWndMainFrame )
will return a pointer to winthread. But GetExitCodeThread() requires the handle to the thread for example, you can pass RecvThread->m_hThread, which will solve the issue
In my main dialog I have a function that creates a process and waits for it to finish. It might take up to 15-20 seconds. If I simply wait using WaitForSingleObject my dialog becomes unresponsive.
I want to use a combination of EnableWindow(FALSE), and an internal message loop to make my dialog block, but without looking like the app freezes, the way MessageBox and DoModal do. But I'm not sure how to do that an internal message loop.
I'm afraid your approach won't work. Your app is single-threaded, or at least your UI is. After you call WaitForSingleObject your thread is put to sleep and it won't process windows messages. The fact that you have an internal message loop won't matter. You should probably start a new thread and use it to wait for the process to finish, then notify your UI thread and exit. Or something along those lines.
Running internal message loop is rather trivial coding.
Something like below is all:
EnableWindow(FALSE);
while ( /* check for my exit condition */ )
{
MSG msg;
if(::PeekMessage(&msg, NULL, 0, 0, PM_NOREMOVE))
{
if( !AfxGetApp()->PumpMessage() )
{
::PostQuitMessage(0);
}
}
}
EnableWindow(TRUE);
To wait for the process exit, you could use very short(<30ms) timeout WaitForSingleObject call in the message loop. Or MsgWaitForMultipleObjects. Or GetExitCodeProcess.
I'd like to recommend another approach.
1) Show new modal popup
2) Start the process in OnInitDialog handler and start a timer
3) Check if the process is still running in OnTimer handler, by GetExitCodeProcess
4) Call EndDialog when the process is no longer running
Try MsgWaitForMultipleObjects function; it can process Windows messages while waiting for the event object.
You could:
(a bit complicated) use MsgWaitForMultipleObjects (or MsgWaitForMultipleObjectsEx) to wait for the process to finish or for a message to arrive (processing it in the normal way).
(simple) use RegisterWaitForSingleObject to register a callback that is called in a separate thread when the process exits (and perhaps have that callback just post a message to your window).
(fairly simple) create your own thread to do the waiting in.
I'd go with the 2nd option.
DWORD ec;
if(CreateProcess( NULL, // No module name (use command line).
szExe, // Command line.
NULL, // Process handle not inheritable.
NULL, // Thread handle not inheritable.
FALSE, // Set handle inheritance to FALSE.
procFlags, // No creation flags.
NULL, // Use parent's environment block.
NULL, // Use parent's starting directory.
&si, // Pointer to STARTUPINFO structure.
&pi ) // Pointer to PROCESS_INFORMATION structure.
)
{
while(GetExitCodeProcess(pi.hProcess, &ec) && ec == STILL_ACTIVE)
{
MSG msg;
while(::PeekMessage(&msg, NULL, 0, 0, PM_NOREMOVE))
{
if(!AfxGetApp()->PumpMessage())
{
::PostQuitMessage(0);
break;
}
}
// let MFC do its idle processing
LONG lIdle = 0;
while(AfxGetApp()->OnIdle(lIdle++))
;
}
}
if(ec)
{
CloseHandle(pi.hProcess);
}
Having what appears to be a dead-lock situation with a multi-threaded logging application.
Little background:
My main application has 4-6 threads running. The main thread responsible for monitoring health of various things I'm doing, updating GUIs, etc... Then I have a transmit thread and a receive thread. The transmit and receive threads talk to physical hardware. I sometimes need to debug the data that the transmit and receive threads are seeing; i.e. print to a console without interrupting them due to their time critical nature of the data. The data, by the way, is on a USB bus.
Due to the threading nature of the application, I want to create a debug console that I can send messages to from my other threads. The debug consule runs as a low priority thread and implements a ring buffer such that when you print to the debug console, the message is quickly stored to a ring buffer and sets and event. The debug console's thread sits WaitingOnSingleObject events from the in bound messages that come in. When an event is detected, console thread updates a GUI display with the message. Simple eh? The printing calls and the console thread use a critical section to control access.
NOTE: I can adjust the ring buffer size if I see that I am dropping messages (at least that's the idea).
In a test application, the console works very well if I call its Print method slowly via mouse clicks. I have a button that I can press to send messages to the console and it works. However, if I put any sort of load (many calls to Print method), everything dead-locks. When I trace the dead-lock, my IDE's debugger traces to EnterCriticalSection and sits there.
NOTE: If I remove the Lock/UnLock calls and just use Enter/LeaveCriticalSection (see the code) I sometimes work but still find myself in a dead-lock situation. To rule out deadlocks to stack push/pops, I call Enter/LeaveCriticalSection directly now but this did not solve my issue.... What's going on here?
Here is one Print statement, that allows me to pass in a simple int to the display console.
void TGDB::Print(int I)
{
//Lock();
EnterCriticalSection(&CS);
if( !SuppressOutput )
{
//swprintf( MsgRec->Msg, L"%d", I);
sprintf( MsgRec->Msg, "%d", I);
MBuffer->PutMsg(MsgRec, 1);
}
SetEvent( m_hEvent );
LeaveCriticalSection(&CS);
//UnLock();
}
// My Lock/UnLock methods
void TGDB::Lock(void)
{
EnterCriticalSection(&CS);
}
bool TGDB::TryLock(void)
{
return( TryEnterCriticalSection(&CS) );
}
void TGDB::UnLock(void)
{
LeaveCriticalSection(&CS);
}
// This is how I implemented Console's thread routines
DWORD WINAPI TGDB::ConsoleThread(PVOID pA)
{
DWORD rVal;
TGDB *g = (TGDB *)pA;
return( g->ProcessMessages() );
}
DWORD TGDB::ProcessMessages()
{
DWORD rVal;
bool brVal;
int MsgCnt;
do
{
rVal = WaitForMultipleObjects(1, &m_hEvent, true, iWaitTime);
switch(rVal)
{
case WAIT_OBJECT_0:
EnterCriticalSection(&CS);
//Lock();
if( KeepRunning )
{
Info->Caption = "Rx";
Info->Refresh();
MsgCnt = MBuffer->GetMsgCount();
for(int i=0; i<MsgCnt; i++)
{
MBuffer->GetMsg( MsgRec, 1);
Log->Lines->Add(MsgRec->Msg);
}
}
brVal = KeepRunning;
ResetEvent( m_hEvent );
LeaveCriticalSection(&CS);
//UnLock();
break;
case WAIT_TIMEOUT:
EnterCriticalSection(&CS);
//Lock();
Info->Caption = "Idle";
Info->Refresh();
brVal = KeepRunning;
ResetEvent( m_hEvent );
LeaveCriticalSection(&CS);
//UnLock();
break;
case WAIT_FAILED:
EnterCriticalSection(&CS);
//Lock();
brVal = false;
Info->Caption = "ERROR";
Info->Refresh();
aLine.sprintf("Console error: [%d]", GetLastError() );
Log->Lines->Add(aLine);
aLine = "";
LeaveCriticalSection(&CS);
//UnLock();
break;
}
}while( brVal );
return( rVal );
}
MyTest1 and MyTest2 are just two test functions that I call in response to a button press. MyTest1 never causes a problem no matter how fast I click the button. MyTest2 dead locks nearly everytime.
// No Dead Lock
void TTest::MyTest1()
{
if(gdb)
{
// else where: gdb = new TGDB;
gdb->Print(++I);
}
}
// Causes a Dead Lock
void TTest::MyTest2()
{
if(gdb)
{
// else where: gdb = new TGDB;
gdb->Print(++I);
gdb->Print(++I);
gdb->Print(++I);
gdb->Print(++I);
gdb->Print(++I);
gdb->Print(++I);
gdb->Print(++I);
gdb->Print(++I);
}
}
UPDATE:
Found a bug in my ring buffer implementation. Under heavy load, when buffer wrapped, I didn't detect a full buffer properly so buffer was not returning. I'm pretty sure that issue is now resolved. Once I fixed the ring buffer issue, performance got much better. However, if I decrease the iWaitTime, my dead lock (or freeze up issue) returns.
So after further tests with a much heavier load it appears my deadlock is not gone. Under super heavy load I continue to deadlock or at least my app freezes up but no where near it use to since I fixed ring buffer problem. If I double the number of Print calls in MyTest2 I easily can lock up every time....
Also, my updated code is reflected above. I know make sure my Set & Reset event calls are inside critical section calls.
With those options closed up, I would ask questions about this "Info" object. Is it a window, which window is it parented to, and which thread was it created on?
If Info, or its parent window, was created on the other thread, then the following situation might occur:
The Console Thread is inside a critical section, processing a message.
The Main thread calls Print() and blocks on a critical section waiting for the Console Thread to release the lock.
The Console thread calls a function on Info (Caption), which results in the system sending a message (WM_SETTEXT) to the window. SendMessage blocks because the target thread is not in a message alertable state (isn't blocked on a call to GetMessage/WaitMessage/MsgWaitForMultipleObjects).
Now you have a deadlock.
This kind of #$(%^ can happen whenever you mix blocking routines with anything that interacts with windows. The only appropriate blocking function to use on a GUI thread is MSGWaitForMultipleObjects otherwise SendMessage calls to windows hosted on the thread can easily deadlock.
Avoiding this involves two possible approaches:
Never doing any GUI interaction in worker threads. Only use PostMessage to dispatch non blocking UI update commands to the UI thread, OR
Use kernel Event objects + MSGWaitForMultipleObjects (on the GUI thread) to ensure that even when you are blocking on a resource, you are still dispatching messages.
Without knowing where it is deadlocking this code is hard to figure out. Two comments tho:
Given that this is c++, you should be using an Auto object to perform the lock and unlock. Just in case it ever becomes non catastrophic for Log to throw an exception.
You are resetting the event in response to WAIT_TIMEOUT. This leaves a small window of opportunity for a 2nd Print() call to set the event while the worker thread has returned from WaitForMultiple, but before it has entered the critical section. Which will result in the event being reset when there is actually data pending.
But you do need to debug it and reveal where it "Deadlocks". If one thread IS stuck on EnterCriticalSection, then we can find out why. If neither thread is, then the incomplete printing is just the result of an event getting lost.
I would strongly recommend a lockfree implementation.
Not only will this avoid potential deadlock, but debug instrumentation is one place where you absolutely do not want to take a lock. The impact of formatting debug messages on timing of a multi-threaded application is bad enough... having locks synchronize your parallel code just because you instrumented it makes debugging futile.
What I suggest is an SList-based design (The Win32 API provides an SList implementation, but you can build a thread-safe template easily enough using InterlockedCompareExchange and InterlockedExchange). Each thread will have a pool of buffers. Each buffer will track the thread it came from, after processing the buffer, the log manager will post the buffer back to the source thread's SList for reuse. Threads wishing to write a message will post a buffer to the logger thread. This also prevents any thread from starving other threads of buffers. An event to wake the logger thread when a buffer is placed into the queue completes the design.