Is there a known algorithm for implementing a connection pool? If not what are the known algorithms and what are their trade-offs?
What design patterns are common when designing and programming a connection pool?
Are there any code examples implement a connection pool using boost.asio?
Is it a good idea to use a connection pool for persisting connections (not http)?
How is threading related to connection pooling? When do you need a new thread?
If you are looking for a pure thread-pooling policy (may be a connection or any resource) there are two simple approaches viz:-
Half Sync/Half Async Model (usually using using message queues to pass information).
Leaders/Followers Model (usually using request queues to pass information).
The first approach goes like this:-
You create a pool of threads to
handle a resource. Often this size
(number of threads) needs to be
configurable. Call these threads
'Workers'.
You then create a master thread that
will dispatch the work to the
Worker threads. The application program dispatches the task as a
message to the master thread.
The master thread puts the same on
the message Q of a chosen Worker
thread and the Worker thread removes itself from the
pool. Choosing and removing the
Worker thread needs synchronization.
After the Worker completes the
task, it returns to the thread-pool.
The master thread itself can consume the tasks it gets in FCFS or a prioritized manner. This will depend on your implementation.
The second model (Leader/Followers) goes something like this:-
Create a thread pool. Initially all
are Workers. Then elect a
Leader, automatically rest-all become followers. Note that electing
a Leader has to be synchronized.
Put all the data to be processed on a
single request Q.
The thread-pool Leader dequeues
the task. It then immediately
elects a new Leader and starts executing the task.
The new Leader picks up the next
task.
There may be other approaches as well, but the ones outlined above are simple that work with most use-cases.
Half Sync/Half Async Major Weakness:-
Higher context switching,
synchronization, and data copying
overhead.
Leader/Follwers Major Weakness:-
Implementation complexity of
Leader election in thread pool.
Now you can decide for yourself the more correct approach.
HTH,
Related
I'm trying to learn more about the async abstractions used by this codebase I'm working on.
I'm reading Folly's documentation for two async executor pools in the library, IOThreadPoolExecutor for io bound tasks, and CPUThreadPoolExecutor for cpu bound tasks (https://github.com/facebook/folly/blob/master/folly/docs/Executors.md).
I'm reading through the descriptions but I don't understand the main difference. It seems like IOThreadPoolExecutor is built around event_fd and epoll loop and CPUThreadPoolExecutor uses a queue and semaphore.
But that doesn't tell me that much about the benefits and trade-offs.
At a high level IPThreadPoolExecutors should be used only if you need a pool of EventBases. If you need a pool of workers, then use CPUThreadPoolExecutor.
CPUThreadPoolExecutor
Contains a series of priority queues which get constantly picked up by a series of workers. Each worker thread executes threadRun() after created. ThreadRun() is essentially an infinite loop which pulls one task from task queue and executes it. If the task is already expired when it is fetched, then the expire callback is executed instead of the task itself.
IOThreadPoolExecutor
Each IO thread runs its own EventBase. Instead of pulling task from task queue like the CPUThreadPoolExecutor, the IOThreadPoolExecutor registers an event to the EventBase of next IO thread. Each IO thread then calls loopForEver() for its EventBase, which essentially calls epoll() to perform async io.
So most of the time you should probably be using a CPUThreadPoolExecutor, as that is the usual use case for having a pool of workers.
I intend to implement a thread pool to manage threads in my project. The basic structure of thread pool come to my head is queue, and some threads generate tasks into this queue, and some thread managed by thread pool are waiting to handle those task. I think this is class producer and consumer problem. But when I google thread pool implementation on the web, I find those implementation seldom use this classic model, so my question is why they don't use this classic model, does this model has any drawbacks? why they don't use full semaphore and empty semaphore to sync?
If you have multiple threads waiting on a single resource (in this case the semaphores and queue) then you are creating a bottle neck. You are forcing all tasks through one queue, even though you have multiple workers. Logically this might make sense if the workers are usually idle, but the whole point of a thread pool is to deal with a heavily loaded scenario where the workers are kept busy (for maximum through-put). Using a single input queue will be particularly bad on a multi-processor system where all workers read and write the head of the queue when they are trying to get the next task. Even though the lock contention might be low, the queue head pointer will still need to be shared/communicated from one CPU cache to another each time it is updated.
Think about the ideal case: all workers are always busy. When a new task is enqueued you want it to be dispatched to the worker that will complete its current/pending task(s) first.
If, as a client, you had a contention-free oracle that could tell you which worker to enqueue a new task to, and each worker had its own queue, then you could implement each worker with its own multi-writer-single-reader queue and always dispatch new tasks to the best queue, thus eliminating worker contention on a single shared input queue. Of course you don't have such an oracle, but this mechanism still works pretty well until a worker runs out of tasks or the queues get imbalanced. "Work stealing" deals with these cases, while still reducing contention compared to the single queue case.
See also:
Is Work Stealing always the most appropriate user-level thread scheduling algorithm?
Why there's no Producer and Consumer model implementation
This model is very generic and could have lots of different explanations, one of the implementation could be a Queue:
Try Apache APR Queue:
It's documented as Thread Safe FIFO bounded queue.
http://apr.apache.org/docs/apr-util/1.3/apr__queue_8h.html
I know that it is not recommended to block in the receive method of an actor, but I believe it can be done (as long as it is not done in too many actors at once).
This post suggests blocking in preStart as one way to solve a problem, so presumably blocking in preStart is safe.
However, I tried to block in preRestart (not preStart) and everything seemed to just hang - no more messages were logged as received.
Also, in cases where it is not safe to block, what is a safe alternative?
It's relatively safe to block in receive when:
the number of blocked actors in total is much smaller than the number of total worker threads. By default there are ten worker threads, so 1-2 blocked actors are fine
blocking actor has its own, dedicated dispatcher (thread pool). Other actors are not affected
When it's not safe to block, good alternative is to... not block ;-). If you are working with legacy API that is inherently blocking you can either have a separate thread pool maintained inside some actor (feels wrong) or use approach 2. above - dedicate few threads to a subset of actors that need to block.
Never ever block an actor.
If your actor is part of an actor hierarchy (and it should be), the actor system is not able to stop it.
The actor's life-cycle (supervision, watchig etc.) is done by messaging.
Stopping a parent actor of a blocking child will not work.
Maybe there are ways to couple the blocking condition with the actor's lifecycle.
But this would lead to overload of complications and bad-style.
So, the best way is to do the blocking part outside of that actor.
E.g. you could run the blocking code via an executor service in a separate thread.
I have a class object which is acting as a server. It receives request from anywhere and pushes the request in its request queue (Producer). Now there is a consumer thread running which is popping the request from the request queue and based on the request calling appropriate class method to furnish the request. Now the consumption of the request from the queue and launching of appropriate function is being executed in a synchronous manner. What I want is the consumer thread pops up a request from the queue and launches the appropriate function in asynchronous manner so that the consumer can pop next request from the queue immediately.
One solution I have tried with this is the consumer pops up a request from the queue and create a boost::thread and start appropriate function in a new thread. I have saved thread pointers in std::vector as well as also tried boost::thread_group. So far so good. But there is a problem in this solution.
Once I have furnished more than 150 requests, there are more 150 threads and after that pthread does not create new thread giving error "pthread_create: Resource temporarily unavailable", which I believe means the stack of the current process has ran out so new threads cannot be created.
Question #1 My request handlers does not contain while (1), and those are just doing some work and exiting and are not waiting for anything at all, thats why I am expecting my initial threads has completed their processing and exited from the thread handler function. Considering this if the thread has completed its processing and exited shouldn't it got cleaned up its stuff from the stack?
One solution to this problem is I can set the stack size of the thread, but that will still raise this error after say 1000 threads.
So my requirement is I must clean up the completed threads after some time (i.e. say when the thread pointers vector has exceeded 100 or after every 1 minute or something like that).
Question #2 Besides launching the new thread as I have mentioned above what is the other asynchronous function call mechanism I should try. Is boost::function + boost::bind asynchronous? Is this a good solution to the situation I have mentioned? Say my system is supposed to be online 24/7/365 and receiving say >1000 request each day.
Update #1
So I found one problem in my design. I have mentioned in my question #1 that my request handler contains just plain calls which I found is not true. It is downloading a file from a server synchronously, which is essentially a blocking operation. I should download the file asynchronously.
There is no use of making threads which your underlying system can not handle concurrently if the request handler is not doing any blocking operation.
So as Alex has mentioned having more than one consumer threads (I think 5 are enough) to pop a request from the queue and have a asynchronous file download will solve my issue.
One solution is to have multiple consumer threads, which each pop a work item off the queue and deal with it synchronously. It enables you to manage concurrency (avoid over subscription), whilst still processing multiple items at a time. You also remove the overhead of launching a new thread on every item, which I'd predict is one of your bottlenecks.
You should ensure your queue is designed for multiple consumers.
Never used this implementation, but a threadpool might help.
You use already use Boost and you download files. So it would be pretty natural to use Boost.Asio for networking and for all other multithreading/async related stuff, like a central dispatcher.
First of all, I'd recommend to create a pool of threads and run Asio dispatcher over them: like here. Use Asio async networking to download files: example here. When a file is downloaded just process it.
This approach is pretty scalable and you won't worry about async networking or multithreading synchronization (rather tricky stuff). Boost.Asio provides good examples how to accomplish this.
I need a threadpool for my application, and I'd like to rely on standard (C++11 or boost) stuff as much as possible. I realize there is an unofficial(!) boost thread pool class, which basically solves what I need, however I'd rather avoid it because it is not in the boost library itself -- why is it still not in the core library after so many years?
In some posts on this page and elsewhere, people suggested using boost::asio to achieve a threadpool like behavior. At first sight, that looked like what I wanted to do, however I found out that all implementations I have seen have no means to join on the currently active tasks, which makes it useless for my application. To perform a join, they send stop signal to all the threads and subsequently join them. However, that completely nullifies the advantage of threadpools in my use case, because that makes new tasks require the creation of a new thread.
What I want to do is:
ThreadPool pool(4);
for (...)
{
for (int i=0;i<something;i++)
pool.pushTask(...);
pool.join();
// do something with the results
}
Can anyone suggest a solution (except for using the existing unofficial thread pool on sourceforge)? Is there anything in C++11 or core boost that can help me here?
At first sight, that looked like what I wanted to do, however I found out that all implementations I have seen have no means to join on the currently active tasks, which makes it useless for my application. To perform a join, they send stop signal to all the threads and subsequently join them. However, that completely nullifies the advantage of threadpools in my use case, because that makes new tasks require the creation of a new thread.
I think you might have misunderstood the asio example:
IIRC (and it's been a while) each thread running in the thread pool has called io_service::run which means that effectively each thread has an event loop and a scheduler. To then get asio to complete tasks you post tasks to the io_service using the io_service::post method and asio's scheduling mechanism takes care of the rest. As long as you don't call io_service::stop, the thread pool will continue running using as many threads as you started running (assuming that each thread has work to do or has been assigned a io_service::work object).
So you don't need to create new threads for new tasks, that would go against the concept of a threadpool.
Have each task class derive from a Task that has an 'OnCompletion(task)' method/event. The threadpool threads can then call that after calling the main run() method of the task.
Waiting for a single task to complete is then easy. The OnCompletion() can perform whatever is required to signal the originating thread, signaling a condvar, queueing the task to a producer-consumer queue, calling SendMessage/PostMessage API's, Invoke/BeginInvoke, whatever.
If an oringinating thread needs to wait for several tasks to all complete, you could extend the above and issue a single 'Wait task' to the pool. The wait task has its own OnCompletion to communicate the completion of other tasks and has a thread-safe 'task counter', (atomic ops or lock), set to the number of 'main' tasks to be issued. The wait task is issued to the pool first and the thread that runs it waits on a private 'allDone' condvar in the wait task. The 'main' tasks are then issued to the pool with their OnCompletion set to call a method of the wait task that decrements the task counter towards zero. When the task counter reaches zero, the thread that achieves this signals the allDone condvar. The wait task OnCompletion then runs and so signals the completion of all the main tasks.
Such a mechansism does not require the continual create/terminate/join/delete of threadpool threads, places no restriction on how the originating task needs to be signaled and you can issue as many such task-groups as you wish. You should note, however, that each wait task blocks one threadpool thread, so make sure you create a few extra threads in the pool, (not usually any problem).
This seems like a job for boost::futures. The example in the docs seems to demonstrate exactly what you're looking to do.
Joining a thread mean stop for it until it stop, and if it stop and you want to assign a new task to it, you must create a new thread. So in your case you should wait for a condition (for example boost::condition_variable) to indicate end of tasks. So using this technique it is very easy to implement it using boost::asio and boost::condition_variable. Each thread call boost::asio::io_service::run and tasks will be scheduled and executed on different threads and at the end, each task will set a boost::condition_variable or event decrement a std::atomic to indicate end of the job! that's really easy, isn't it?