I have a very weird issue with a zombie/garbage thread crashing the process although this thread should have been joined a while ago.
Application written in C++, built for x86_64, runs on MacOS using Rosetta. Googletest runs app multiple times: initializes the engine, performs tests and uninitializes it for every test - no new processes are started - all happens within googletest process. In about one goggletest start of about ten, one of the threads that my application has not created during current run crashes with invalid resource access which triggers SIGABRT and crashes the complete process.
Threads in the pool are created like this during thread pool construction, done once during application initialization:
// threadsNum is a constant, e.g. 8
for (size_t i = 0; i < threadsNum; ++i){
// this->workers has type std::vector< std::thread > workers;
workers.emplace_back(
[this, i]
{
workerThreadFunction(i);
}
);
}
workerThreadFunction() sets thread name and enters an infinete loop for doing it's job while checking if it should stop in every iteration:
for (;;) {
// Mutex lock to access this->stop
std::unique_lock<std::mutex> lock(this->queue_mutex);
if (this->stop) {
break;
}
...
}
During thread pool destruction all threads are joined:
for(std::thread &worker: workers) {
worker.join();
}
To sum up:
Application starts 8 threads for a specfic task using a thread pool
Threads have their names for debugging purposes - e.g. "MyThread 1", "MyThread 2", etc.
I have verified that all threads are joined upon thread pool destruction (threadX.join() returns)
Application shutdown is clean after test iteration execution before the next test with the crash starts
In problematic runs with the crash there are more threads present than application has created - one of the threads is present two times (thread name duplicated). Crash dump also shows there are two threads with the same name and one of them has crashed.
That duplicated thread has corrupt stack and crashes due to invalid resource access (locking this->queue_mutex to be precise)
Additional main thread sleep for e.g. 100ms after engine uninitialization does not help - does not look like a timing issue
To me it looks like that thread has survived join() somehow and reappeared in the process. But I cannot imagine how could it be possible.
The question is, am I missing something here? Are there any tools to debug this issue besides what I have already done?
Related
I am trying to make a native app , and I need a separate thread freezing some values(constant overwriting with delay) in the background and I don't need any return from it to main. So after creating the thread when I detach from it , it does not do the freezing.
pthread_create(&frzTh, NULL, freezingNow, NULL);
pthread_detach(frzTh);
But if I join the thread then it performs freezing but my main thread gets blocked as it waits for the child thread to finish , and since the child runs infinitely , there is no coming out.
pthread_create(&frzTh, NULL, freezingNow, NULL);
pthread_join(frzTh,NULL);
So, I tried using fork() to create a child process instead of thread. Now , I am able to perform all tasks parallel to my main. But , this is causing a lot of memory usage and leads to heating of device.
pid_t pid_c = fork();
if (pid_c == 0 && freeze) {
while (freeze) {
Freeze();
usleep(delay);
}
}
So, what is the best way to do this ?
Best example is game guardian app and it's freezing mechanism.
To do this properly, you need to have a mechanism by which the main thread can cause the child thread to exit (a simple std::atomic<bool> pleaseQuitNow that the child thread tests periodically, and the main thread sets to true before calling pthread_join(), will do fine).
As for why you need to call pthread_join() before exiting, rather than just allowing the main thread to exit while the child thread remains running: there is often run-time-environment code that executes after main() returns that tears down various run-time data structures that are shared by all threads in the process. If any threads are still running while the main-thread is tearing down these data structures, it is possible that the still-running thread(s) will try to access one of these data structures while it is in a destroyed or half-destroyed state, causing an occasional crash-on-exit.
(Of course, if your program never exits at all, or if you don't care about an occasional crash-on-exit, you could skip the orderly shutdown of your child thread, but since it's not difficult to implement, you're better off doing things the right way and avoiding embarrassment later when your app crashes at the end of a demo)
If you wanna do Something as async with Mainthread untill end main ,
I recommand Promise - future in c++
this example :) good luck
#include <future>
#include <iostream>
#include <thread>
void DoWork(promise<int> p)
{
// do something (child thread)
// saved value in p
p.set_value(10);
}
int main(void)
{
promise<int> p;
auto future = p.get_future();
thread worker{ DoWork, std::move(p)};
// do something you want
// return result
int result = future.get();
std::cout<< result <<'\n'; // print 10
}
I am a beginner using multithreading in C++, so I'd appreciate it if you can give me some recommendations.
I have a function which receives the previous frame and current frame from a video stream (let's call this function, readFrames()). The task of that function is to compute Motion Estimation.
The idea when calling readFrames() would be:
Store the previous and current frame in a buffer.
I want to compute the value of Motion between each pair of frames from the buffer but without blocking the function readFrames(), because more frames can be received while computing that value. I suppose I have to write a function computeMotionValue() and every time I want to execute it, create a new thread and launch it. This function should return some float motionValue.
Every time the motionValue returned by any thread is over a threshold, I want to +1 a common int variable, let's call it nValidMotion.
My problem is that I don't know how to "synchronize" the threads when accessing motionValue and nValidMotion.
Can you please explain to me in some pseudocode how can I do that?
and every time I want to execute it, create a new thread and launch it
That's usually a bad idea. Threads are usually fairly heavy-weight, and spawning one is usually slower than just passing a message to an existing thread pool.
Anyway, if you fall behind, you'll end up with more threads than processor cores and then you'll fall even further behind due to context-switching overhead and memory pressure. Eventually creating a new thread will fail.
My problem is that I don't know how to "synchronize" the threads when accessing motionValue and nValidMotion.
Synchronization of access to a shared resource is usually handled with std::mutex (mutex means "mutual exclusion", because only one thread can hold the lock at once).
If you need to wait for another thread to do something, use std::condition_variable to wait/signal. You're waiting-for/signalling a change in state of some shared resource, so you need a mutex for that as well.
The usual recommendation for this kind of processing is to have at most one thread per available core, all serving a thread pool. A thread pool has a work queue (protected by a mutex, and with the empty->non-empty transition signalled by a condvar).
For combining the results, you could have a global counter protected by a mutex (but this is relatively heavy-weight for a single integer), or you could just have each task added to added to the thread pool return a bool via the promise/future mechanism, or you could just make your counter atomic.
Here is a sample pseudo code you may use:
// Following thread awaits notification from worker threads, detecting motion
nValidMotion_woker_Thread()
{
while(true) { message_recieve(msg_q); ++nValidMotion; }
}
// Worker thread, computing motion on 2 frames; if motion detected, notify uysing message Q to nValidMotion_woker_Thread
WorkerThread(frame1 ,frame2)
{
x = computeMotionValue(frame1 ,frame2);
if x > THRESHOLD
msg_q.send();
}
// main thread
main_thread()
{
// 1. create new message Q for inter-thread communication
msg_q = new msg_q();
// start listening thread
Thread a = new nValidMotion_woker_Thread();
a.start();
while(true)
{
// collect 2 frames
frame1 = readFrames();
frame2 = readFrames();
// start workre thread
Thread b = new WorkerThread(frame1 ,frame2);
b.start();
}
}
I am trying to manage the count of native threads in PPL by using its Scheduler class, here is my code:
for (int i = 0; i < 2000; i ++)
{
// configure concurrency count 16 to 32.
concurrency::SchedulerPolicy policy = concurrency::SchedulerPolicy(2, concurrency::MinConcurrency, 16,
concurrency::MaxConcurrency, 32);
concurrency::Scheduler *pScheduler = concurrency::Scheduler::Create(policy);
HANDLE hShutdownEvent = CreateEvent(NULL, FALSE, FALSE, NULL);
pScheduler->RegisterShutdownEvent(hShutdownEvent);
pScheduler->Attach();
//////////////////////////////////////////////////////////////////////////
//for (int i = 0; i < 2000; i ++)
{
concurrency::create_task([]{
concurrency::wait(1000);
OutputDebugString(L"Task Completed\n");
});
}
//////////////////////////////////////////////////////////////////////////
concurrency::CurrentScheduler::Detach();
pScheduler->Release();
WaitForSingleObject(hShutdownEvent, INFINITE);
CloseHandle(hShutdownEvent);
}
The usage of SchedulerPolicy is from MSDN, but it didn't work at all. The expected result of my code above is, PPL will launch 16 to 32 threads to execute the 2000 tasks, but the fact is:
By observing the speed of console output, only one task was processed within a second. I also tried to comment the outter for loop and uncomment the inner for loop, however, this will cause 300 threads being created, still incorrect. If I wait a longer time, the threads created will be even more.
Any ideas on what is the correct way to configure concurrency in PPL?
It has been proved that I should not do concurrency::wait within the task body, PPL works in work stealing mode, when the current task was suspended by wait, it will start to schedule the rest of tasks in queue to maximize the use of computing resources.
When I use concurrency::create_task in real project, since there are a couple of real calculations within the task body, PPL won't create hundreds of threads any more.
Also, SchedulePolicy can be used to configure the number of virtual processors that PPL may use to process the tasks, which is not always same as the number of native threads PPL will create.
Saying my CPU has 8 virtual processors, by default PPL will just create 8 threads in pool, but when some of those threads were suspended by wait or lock, and also there are more tasks pending in the queue, PPL will immediately create more threads to execute them (if the virtual processors were not fully loaded).
I am experimenting with multithreaded synchronization at the moment. For a backround I have a set of about 100000 objects - possibly more - I want to process in different ways multiple times per second.
Now the thing concerning me most is the performance of the synchronization.
This is what I think should work just fine (I omitted all security aspects as this is just a testprogram and in case of an error the program will just crash ..). I wrote two funktions, the first to be executed by the main thread of the program, the second to be run by all additional threads.
void SharedWorker::Start()
{
while (bRunning)
{
// Send the command to start task1
SetEvent(hTask1Event);
// Do task1 (on a subset of all objects) here
// Wait for all workers to finish task1
WaitForMultipleObjects(<NumberOfWorkers>, <ListOfTask1WorkerEvents>, TRUE, INFINITE);
// Reset the command for task1
ResetEvent(hTask1Event);
// Send the command to start task2
SetEvent(hTask2Event);
// Do task2 (on a subset of all objects) here
// Wait for all workers to finish task2
WaitForMultipleObjects(<NumberOfWorkers>, <ListOfTask2WorkerEvents>, TRUE, INFINITE);
// Reset the command for task2
ResetEvent(hTask2Event);
// Send the command to do cleanup
SetEvent(hCleanupEvent);
// Do some (on a subset of all objects) cleanup
// Wait for all workers to finish cleanup
WaitForMultipleObjects(<NumberOfWorkers>, <ListOfCleanupWorkerEvents>, TRUE, INFINITE);
// Reset the command for cleanup
ResetEvent(hCleanupEvent);
}
}
DWORD WINAPI WorkerThreads(LPVOID lpParameter)
{
while (bRunning)
{
WaitForSingleObject(hTask1Event, INFINITE);
// Unset finished cleanup
ResetEvent(hCleanedUp);
// Do task1 (on a subset of all objects) here
// Signal finished task1
SetEvent(hTask1);
WaitForSingleObject(hTask2Event, INFINITE);
// Reset task1 event
ResetEvent(hTask1);
// Do task2 (on a subset of all objects) here
// Signal finished task2
SetEvent(hTask2);
WaitForSingleObject(hCleanupEvent, INFINITE);
// Reset update event
ResetEvent(hTask2);
// Do cleanup (on a subset of all objects) here
// Signal finished cleanup
SetEvent(hCleanedUp);
}
return 0;
}
To point out my requirements, I'll just give you a little example:
Say we got the 100000 objects from above, split into 8 subsets of 12500 objects each, a modern multicore processor with 8 logical cores. The relevant part is the time. All tasks must be performed in about 8ms.
My questions are now, can I get a significant boost in time from split processing or is the synchronization via events too expensive? or is there maybe even another way of synchronizing threads with less effort or process time if all the tasks need to be done this way?
If your processing of a single object is fast, do not split it between threads. The thread synchronization on windows will eat well over 50 ms on every context switch. This time is not used by system, but just the time when something else is running on a system.
However, if every object processing will take around 8ms, there is a point of scheduling the work across pool of threads. However, object processing may vary a bit, and in large counts worker threads would complete the work in a different moment.
Better approach is to organize a synchronized object queue, to which you add objects to process, and from which you take them from processing. Furthermore, as processing of a single object considerably lower, than scheduling interval of a thread, it is good to take them into processing thread in batches (like 10-20). You can estimate the best number of worker threads in your pool and the best size of a batch with tests.
So the pseudocode can look like:
main_thread:
init queue
start workers
set counter to 100000
add 100000 objects to queue
while (counter) wait();
worker_thread:
while (!done)
get up to 10 objects from queue
process objects
counter -= processed count
if (counter == 0) notify done
I am a bit new to multi threading, so forgive me if these questions are too trivial.
My application needs to create multiple threads in a thread and perform actions from each thread.
For example, I have a set of files to read, say 50 and I create a thread to read these files using CreateThread() function.
Now this main thread creates 4 threads to access the file. 1st thread is given file 1, second file 2 and so on.
After 1st thread completed reading file 1 and gives main thread the required data, main thread needs to invoke it with file 5 and obtain data from it. Similar goes for all other threads until all 50 files are read.
After that, each thread is destroyed and finally my main thread is destroyed.
The issue I am facing is:
1) How to stop a thread to exit after file reading?
2) How to invoke the thread again with other file name?
3) How would my child thread give information to main thread?
4) After a thread completes reading the file and returns the main thread a data, how main thread would know which thread has provided the data?
Thanks
This is a very common problem in multi-threaded programming. You can view this as a producer-consumer problem: the main thread "produces" tasks which are "consumed" by the worker threads (s. e.g. http://www.mario-konrad.ch/blog/programming/multithread/tutorial-06.html) . You might also want to read about "thread pools".
I would highly recommend to read into boost's Synchronization (http://www.boost.org/doc/libs/1_50_0/doc/html/thread.html) and use boost's threading functionality as it is platform independent and good to use.
To be more specific to your question: You should create a queue with operations to be done (usually it's the same queue for all worker threads. If you really want to ensure thread 1 is performing task 1, 5, 9 ... you might want to have one queue per worker thread). Access to this queue must be synchronized by a mutex, waiting threads can be notified by condition_variables when new data is added to the mutex.
1.) don't exit the thread function but wait until a condition is fired and then restart using a while ([exit condition not true]) loop
2.) see 1.
3.) through any variable to which both have access and which is secured by a mutex (e.g. a result queue)
4.) by adding this information as the result written to the result queue.
Another advice: It's always hard to get multi-threading correct. So try to be as careful as possible and write tests to detect deadlocks and race conditions.
The typical solution for this kind of problem is using a thread pool and a queue. The main thread pushes all files/filenames to a queue, then starts a thread pool, ie different threads, in which each thread takes an item from the queue and processes it. When one item is processed, it goes on to the next one (if by then the queue is not yet empty). The main thread knows everything is processed when the queue is empty and all threads have exited.
So, 1) and 2) are somewhat conflicting: you don't stop the thread and invoke it again, it just keeps running as long as it finds items on the queue.
For 3) you can again use a queue in which the thread puts information, and from which the main thread reads. For 4) you could give each thread an id and put that together with the data. However normally the main thread should not need to know which thread exactly processed data.
Some very basic pseudocode to give you an idea, locking for threadsafety omitted:
//main
for( all filenames )
queue.push_back( filename );
//start some thread
threadPool.StartThreads( 4, CreateThread( queue ) );
//wait for threads to end
threadPool.Join();
//thread
class Thread
{
public:
Thread( queue q ) : q( q ) {}
void Start();
bool Join();
void ThreadFun()
{
auto nextQueueItem = q.pop_back();
if( !nextQueuItem )
return; //q empty
ProcessItem( nextQueueItem );
}
}
Whether you use a thread pool or not to execute your synchronies file reads, it boils down to a chain of functions or groups of functions that have to run serialized. So let's assume, you find a way to execute functions in parallel (be it be starting one thread per function or by using a thread pool), to wait for the first 4 files to read, you can use a queue, where the reading threads push there results into, the fifth function now pulls 4 results out of the queue (the queue blocks when empty) and processes. If there are more dependencies between functions, you can add more queues between them. Sketch:
void read_file( const std::string& name, queue& q )
{
file_content f= .... // read file
q.push( f )
}
void process4files( queue& q )
{
std::vector< file_content > result;
for ( int i = 0; i != 4; ++i )
result.push_back( q.pop() );
// now 4 files are read ...
assert( result.size() == 4u );
}
queue q;
thread t1( &read_file, "file1", q );
thread t2( &read_file, "file2", q );
thread t3( &read_file, "file3", q );
thread t4( &read_file, "file4", q );
thread t5( &process4files, q );
t5.join();
I hope you get the idea.
Torsten