I have two threads. The first creates a Logic object, detaching a second thread to spin, blocking on OpenSSL socket to receive messages:
struct Logic
{
Logic()
{
std::thread t1(&Logic::run, this);
t1.detach();
}
void run()
{
while(true)
{
// Gets data from SSL (blocking socket)
// Processes data
// Updates timestamp
}
}
uint64_t timestamp;
};
The first thread returns, enters a while loop and continually checks if the detached thread is still running (or whether its blocked permanently).
while(true)
{
Logic logic();
while(true)
{
if(timestamp_not_updated)
{
break; // Break, destroy current Logic object and create another
}
}
}
If the timestamp stops being updated, the inner while loop breaks, causing the Logic object to be destroyed and a new one created.
When this restart behaviour triggers I get a seg fault. thread apply all bt shows 3 threads, not 2. The original detached thread (blocking on OpenSSL) still exists. I thought this would get destroyed due to the object.
How do I stop a detached thread which is blocking/waiting on a resource, so I can restart my class? I need the blocking behaviour because I don't have anything else to do (besides receive the packet) and it's better for performance, than to keep calling in to OpenSSL.
Related
I have used below structure to create a threadpool, now the question is how do I let all the preallocate threads end properly?
std::vector<pthread_t> preallocatedThreadsPool; // threadpool
std::queue<int> tcpQueue; // a queue to hold my task
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t condition_var = PTHREAD_COND_INITIALIZER;
void* threadFunctionUsedByThreadsPool(void *arg);
main () {
preallocatedThreadsPool.resize(preallocatThreadsNumber);
for(pthread_t i : preallocatedThreadsPool) {
pthread_create(&i, NULL, threadFunctionUsedByThreadsPool, NULL);
}
pthread_mutex_lock(&mutex); // one thread mess with the queue at one time
tcpQueue.push(task);
pthread_cond_signal(&condition_var);
pthread_mutex_unlock(&mutex);
}
void* threadFunctionUsedByThreadsPool(void *arg) {
while (true) {
pthread_mutex_lock(&mutex);
if (tcpQueue.empty()) { // can't get work from the queue then just wait
pthread_cond_wait(&condition_var, &mutex); // wait for the signal from other thread to deal with client otherwise sleep
task = tcpQueue.front();
tcpQueue.pop();
}
pthread_mutex_unlock(&mutex);
if (task) {
// do task
}
}
return NULL;
}
I have been searching for days for this problem still can not find a decent solution, the closest one I have tried is , when the program wants to quit, push a special item into the queue, then inside threadFunctionUsedByThreadsPool, when detecting such item, I will call pthread_join, however, when I using gdb tool to debug it , those pre-allocated threads are still there, anyone could help, better with some code, for example, how do I modify the threadFunctionUsedByThreadsPool, so that I can quit all the pre-allocated threads properly?
Thanks so much!!!
TLDR: You just need a thread-safe variable that all threads can check for an exit condition in between work items. Use pthread_join to wait for a thread to exit.
First, let's get the while loop in your thread function correct with respect to condition variables.
Instead of this:
pthread_cond_wait(&condition_var, &mutex); // wait for the signal from other thread to deal with client otherwise sleep
task = tcpQueue.front();
tcpQueue.pop();
Check the state of the queue before before and after waking up on the condition variable. Spurious wake up is a real thing and there's no guarantee another thread didn't wake up and grab the last work item. You definitely don't want to be popping from an empty queue.
Better:
while (tcpQueue.empty()) {
pthread_cond_wait(&condition_var, &mutex); // wait for the signal from other thread to deal with client otherwise sleep
}
task = tcpQueue.front();
tcpQueue.pop();
With that addressed, we can introduce a new global bool that represents the stop condition:
bool stopCondition = false;
Whenever we want to tell all the threads in the pool to stop, we can set stopCondition to true and signal the condition var to alert all threads of a state change. Reading or writing stopCondition should be done under a lock. (I suppose you could also use std::atomic<bool>)
Putting it all together, your thread function becomes this:
void* threadFunctionUsedByThreadsPool(void* arg) {
pthread_mutex_lock(&mutex);
while (!stopCondition) {
// wait for a task to be queued
while (tcpQueue.empty() && !stopCondition) {
pthread_cond_wait(&condition_var, &mutex); // wait for the signal from other thread to deal with client otherwise sleep
}
if (stopCondition == false) {
task = tcpQueue.front();
tcpQueue.pop();
// exit lock while operating on a task
pthread_mutex_unlock(&mutex);
if (task) {
// do task
}
// re-acquire the lock
pthread_mutex_lock(&mutex);
}
}
// release the lock before exiting the function
pthread_mutex_unlock(&mutex);
return NULL;
}
And then a helper function to signal all the threads to exit and also waits for each thread to stop. notice that we're using pthread_cond_broadcast to notify all threads to wake up from their condition variable wait instead of pthread_cond_signal which only wakes up one thread.
void stopThreadPool()
{
// signal all threads to exit after they finish their current work item
pthread_mutex_lock(&mutex);
stopCondition = true;
pthread_cond_broadcast(&condition_var); // notify all threads
pthread_mutex_unlock(&mutex);
// wait for all threads to exit
for (auto& t : preAllocatedThreadsPool) {
pthread_join(t, nullptr);
}
preAllocatedThreadsPool.clear();
}
One last bug that I just caught - your main isn't property initializing your preAllocatedThreadsPool vector like you think it is. You're making a copy of the pthread_t, instead of using the handle actually in the vector.
Instead of this:
for(pthread_t i : preallocatedThreadsPool) {
Your loop needs to enumerate by reference:
Better:
for(pthread_t &i : preallocatedThreadsPool) {
Send a task that instructs the pool thread to requeue the task and then terminate. The poison-task will then run through all the threads in the pool, killing them all off. I have used a null as the poison, (ie, an illegal task) - it does not need to be destructed when it has killed the last thread. You may wish to purge the task queue before sending the null/whatever. If you use a null, you only need a null check in the threads, just after dequeing the task.
You need very little extra code, you don't need to know how many threads in the pool and it will work:)
I want to make function that when receive buffer from socket, thread make whole program freeze out of my function until my function is finished. I try these as below
Function Listen
void Listen(can* _c) {
while (true)
{
std::lock_guard<std::mutex>guard(_c->connection->mutex);
thread t(&connect_tcp::Recv_data,_c->connection,_c->s,ref(_c->response),_c->signals);
if (t.joinable())
t.join();
}
}
Function dataset_browseCan
void dataset_browseCan(can* _c) {
thread org_th(Listen, _c); // I call thread here
org_th.detach();
dataset_browse(_c->cotp, _c->mms_obj, _c->connection, _c->response, _c->list, _c->size_encoder, _c->s);
dataset_signals_browse(_c->cotp, _c->mms_obj, _c->connection, _c->response, _c->list, _c->size_encoder, _c->s);
Sleep(800);
_c->signals = new Signals[_c->response.real_signals_and_values.size()];
}
Function Recv Data
void connect_tcp::Recv_data(SOCKET s,mms_response &response,Signals *signals) {
LinkedList** list = new LinkedList * [1000];
uint8_t* buffer = new uint8_t [10000];
Sleep(800);
/*std::lock_guard<std::mutex>guard(mutex);*/
thread j(recv,s, (char*)buffer, 10000, 0);
j.join()
/*this->mutex.unlock();*/
decode_bytes(response,buffer, list,signals);
}
I tried mutex and this_thread::sleep_for() but everytime my main function keep running.
Is make program freeze possible ?
You use threads in order to allow things to keep running while something else is happening, so wanting to "stop main" seems counter-intuitive.
However, if you want to share data between threads (e.g. between the thread that runs main and a background thread) then you need to use some form of synchronization. One way to do that is to use a std::mutex. If you lock the mutex before every access, and unlock it afterwards (using std::lock_guard or std::unique_lock) then it will prevent another thread from locking the same mutex while you are accessing the data.
If you need to prevent concurrent access for a long time, then you should not hold a mutex for the whole time. Either consider whether threads are the best solution to your problem, or use a mutex-protected flag to indicate whether the data is ready, and then either poll or use std::condition_variable or similar to wait until the flag is set.
I read the source code of Boost ASIO, and I wanna find out it is only one thread for it to call epoll_wait(Of course,if I use epoll reactor).
I wanna find its solution about more than one thread to call epoll_wait, this may cause different threads doing the read for the same socket at the same time .
I read some key codes as follows:
// Prepare to execute first handler from queue.
operation* o = op_queue_.front();
op_queue_.pop();
bool more_handlers = (!op_queue_.empty());
if (o == &task_operation_)
{
task_interrupted_ = more_handlers;
if (more_handlers && !one_thread_)
wakeup_event_.unlock_and_signal_one(lock);
else
lock.unlock();
task_cleanup on_exit = { this, &lock, &this_thread };
(void)on_exit;
// Run the task. May throw an exception. Only block if the operation
// queue is empty and we're not polling, otherwise we want to return
// as soon as possible.
task_->run(!more_handlers, this_thread.private_op_queue);
}
task_ is epoll reactor and it will call epoll_wait in the run,
I guess it may only one thread to call it because only one "task_operation_" in the op_queue_, am I right ?
If I wanna use epoll in multi-threading, or I may use "EPOLLONESHOT" so that it can ensure that one thread handle one socket at one time.
First case, is when you are using a single instance of io_service and calling io_service::run method from multiple threads.
Lets see the schduler::run function (simplified):
std::size_t scheduler::run(asio::error_code& ec)
{
mutex::scoped_lock lock(mutex_);
std::size_t n = 0;
for (; do_run_one(lock, this_thread, ec); lock.lock())
if (n != (std::numeric_limits<std::size_t>::max)())
++n;
return n;
}
So, with the lock held, it calls the do_run_one method, which is something like:
std::size_t scheduler::do_run_one(mutex::scoped_lock& lock,
scheduler::thread_info& this_thread,
const asio::error_code& ec)
{
while (!stopped_)
{
if (!op_queue_.empty())
{
// Prepare to execute first handler from queue.
operation* o = op_queue_.front();
op_queue_.pop();
bool more_handlers = (!op_queue_.empty());
if (o == &task_operation_)
{
task_interrupted_ = more_handlers;
if (more_handlers && !one_thread_)
wakeup_event_.unlock_and_signal_one(lock);
else
lock.unlock();
task_cleanup on_exit = { this, &lock, &this_thread };
(void)on_exit;
task_->run(!more_handlers, this_thread.private_op_queue);
}
else
{
//......
}
}
else
{
wakeup_event_.clear(lock);
wakeup_event_.wait(lock);
}
}
return 0;
}
The interesting part of the code sre these lines:
if (more_handlers && !one_thread_)
wakeup_event_.unlock_and_signal_one(lock);
else
lock.unlock();
The case we are discussing now is the one with multiple threads, so the first condition will satisfy (assuming we have quite a number of pending tasks in op_queue_).
What wakeup_event_.unlock_and_signal_one ends up doing is release/unlock the lock and notify one of threads who is waiting on a conditional wait. So, with this, atleast one another thread (whoever gets the lock) can call do_run_one now.
The task_ in your case is epoll_reactor as you have said. And, in it's run method it calls epoll_wait (not holding the lock_ of scheduler).
The interesting thing here is what it does when it iterates over all the ready descriptors that epoll_wait returned. It pushes them back in the operational queue it received as reference in the argument. The operations pushed now have the run time type of descriptor_state instead of task_operation_:
for (int i = 0; i < num_events; ++i)
{
void* ptr = events[i].data.ptr;
if (ptr == &interrupter_)
{
// don't call work_started() here. This still allows the scheduler to
// stop if the only remaining operations are descriptor operations.
descriptor_state* descriptor_data = static_cast<descriptor_state*>(ptr);
descriptor_data->set_ready_events(events[i].events);
ops.push(descriptor_data);
}
}
So, in the next iteration of the while loop inside scheduler::do_run_one, for the completed tasks, it will hit the else branch (which I elided in my paste earlier):
else
{
std::size_t task_result = o->task_result_;
if (more_handlers && !one_thread_)
wake_one_thread_and_unlock(lock);
else
lock.unlock();
// Ensure the count of outstanding work is decremented on block exit.
work_cleanup on_exit = { this, &lock, &this_thread };
(void)on_exit;
// Complete the operation. May throw an exception. Deletes the object.
o->complete(this, ec, task_result);
return 1;
}
Which call the complete function pointer which inturn probably will call the user passed handle to the async_read or async_write API.
Second case, is where you create a pool of io_service objects and call its run method on 1 or more threads i.e the mapping between io_service and thread could be 1:1 or 1:N as may suit your application. This way you can assign an io_service object to a soucket object in round robin fashion.
Now, coming to your question:
If I wanna use epoll in multi-threading, or I may use "EPOLLONESHOT"
so that it can ensure that one thread handle one socket at one time.
If I understood this correctly, you want to handle all the events to a socket using 1 thread ? I think this is possible by following approach number 2, i.e to create a pool of io_service objects and map it to 1 thread. This way you can be sure that all the activity on a particular socket will be addressed by only one thread i.e the thread on which that io_service:run.
You do not have to worry about setting EPOLLONESHOT in the above case.
I am not so sure about getting the same behaviour using the first approach, which is multiple thread and 1 io_service.
But, if you are not using threads at all i.e your io_service runs on single thread, then you don't have to worry about all this, after all the purpose of asio is to abstract all these stuff.
Only a single thread will invoke epoll_wait. Once the thread receives event notifications for descriptors, it will demultiplex the descriptors to all threads running the io_service. Per the Platform-Specific Implementation Notes:
Threads:
Demultiplexing using epoll is performed in one of the threads that calls io_service::run(), io_service::run_one(), io_service::poll() or io_service::poll_one().
A single descriptor will be processed by a single thread that will perform the I/O. Hence, when using asynchronous operations, I/O will not be performed concurrently for a given socket.
The following process function reads data off a queue and processes it. The wait_and_pop function of masterQueue performs a blocking call. Therefore, control does not move ahead until there exists data on the queue that can be read.
class Context
{
void launch()
{
boost::thread thread1(boost::bind(&Context::push,this ) );
boost::thread thread2(boost::bind(&Context::process,this ) );
std::cout<<"Joining Thread1"<<std::endl;
thread1.join();
std::cout<<"Joining Thread2"<<std::endl;
thread2.join();
}
void process()
{
Data data;
while(status)
{
_masterQueue.wait_and_pop(data); //Blocking Call
//Do something with data
}
}
void push()
{
while(status)
{
//Depending on some internal logic, data is generated
_masterQueue.push(data);
}
}
};
status is a boolean(in global scope). This boolean is set to true by default. It is only changed to false when a signal is caught such as SIGINT, SIGSESV etc. In such a case, the while loop is exited and the program can be exited safely.
bool status = true;
void signalHandler(int signum)
{
std::cout<<"SigNum"<<signum;
status = false;
exit(signum);
}
int main()
{
signal(SIGABRT, signalHandler);
signal(SIGINT, signalHandler);
signal(SIGSEGV, signalHandler);
Context context;
context.launch();
}
Since, no new data is pushed by thread2 when a signal is thrown, control in thread1 is stuck at
_masterQueue.wait_and_pop(data);
How do I force this blocking call to be interrupted?
Is it possible to implement this without changing the internal workings of wait_and_pop
Placing a timeout is not an option, since data may arrive on the queue once in a couple of hours or multiple times a second
Do I push a specific type of data on receiving a signal, e.g INT_MAX/INT_MIN, which the process function is coded to recognise and it exits the loop.
Timeout actually is your answer
You break your loop on getting an answer or having an interrupt
You could also spoof things a bit by having the inturrupt push a noop to the queue
You can try to .interrupt() thread when need to finish it.
If .wait_and_pop() uses standard boost mechanism for wait(condition variable or like), it will definitely be interrupted even in blocked state via throwing boost::thread_interrupted exception. If your masterQueue class is reliable wrt exceptions, then such interruption is safe.
I've created a custom ThreadPool which starts a number of win32 threads with _beginthreadex(). The threads are running a simple loop that attempts to dequeue tasks from a blocking queue, but sometimes I need to stop the threads and if they're blocked on Dequeue then I don't know how to get the threads out of that blocking state.
void ThreadPool::Loop()
{
while(_running)
{
try
{
// Attempts to dequeue a task and run it
_taskQueue.Dequeue()->Run();
}
catch(BlockingQueueTerminate&)
{
// Eat the exception and check the running flag
continue;
}
}
}
My idea was to enqueue the same number of special tasks (let's call them "termination tasks") as there are threads in the pool and each "termination task" will call _endthreadex(0) in order to exit the thread. If there are other tasks in the blocking queue, then I won't really care because as soon as I dequeue a task, I will run it and I will check the _running flag to determine if the thread needs to dequeue any more tasks.
void TerminationTask::Run()
{
_endthreadex(0);
}
I have several concerns about this approach; mainly, if I processed a non-terminating task and the _running flag is set to false, then my thread will not call _endthreadex(0) when it exits the loop. I was wondering if I could call _endthreadex(0) at the end of the loop like this:
void ThreadPool::Loop()
{
while(_running)
{
try
{
// Attempts to dequeue a task and run it
_taskQueue.Dequeue()->Run();
}
catch(BlockingQueueTerminate&)
{
// Eat the exception and check the running flag
continue;
}
}
_endthreadex(0);
}
Will this cause a conflict with my TerminationTask or will the thread exit the loop directly after executing TerminationTask::Run() (i.e. it won't call _endthreadex(0) twice)? Furthermore, is there a better approach than this?
Calling _endthreadex(0) at the end of the thread method is fine. It is also optional. If you just leave the thread method normally, then _endthreadex(0) is called for you.
You can call _endthread or _endthreadex explicitly to terminate a thread; however, _endthread or _endthreadex is called automatically when the thread returns from the routine passed as a parameter to _beginthread or _beginthreadex.ref
Sending a termination task is the correct way to get a blocked thread pool thread to unblock and quit.
So, to summarise:
Your strategy is good and the implementation of TerminationTask::Run is correct.
You can remove the harmless call to _endthreadex(0) at the end of ThreadPool::Loop.
Putting a temination task in the Queue is correct. I would try a different approach to handling it, though:
class TerminateThreadNow {};
void TerminationTask::Run()
{
throw TerminateThreadNow();
}
void ThreadPool::Loop()
{
while(_running)
{
try
{
// Attempts to dequeue a task and run it
_taskQueue.Dequeue()->Run();
}
catch(TerminateThreadNow&)
{
_running = false;
}
catch(BlockingQueueTerminate&)
{
// Eat the exception and check the running flag
}
}
}