Thread Pool join hangs when oversubscribing threads - c++

Question

I'm having an issue with a thread hanging when joining my threads in a thread pool I have created. The issue only occurs if I loop over the thread pool execution a large number of times.
I have a thread pool class like the following;
#include <queue>
#include <mutex>
#include <condition_variable>
#include <functional>
#include <atomic>
#include <vector>
#include <thread>
#include <iostream>
class ThreadPool
{
public:
ThreadPool()
{
m_shutdown.store(false, std::memory_order_relaxed);
createThreads(1);
}
ThreadPool(std::size_t numThreads)
{
m_shutdown.store(false, std::memory_order_relaxed);
createThreads(numThreads);
}
void add_job(std::function<void()> new_job)
{
{
std::scoped_lock<std::mutex> lock(m_jobMutex);
m_jobQueue.push(new_job);
}
m_notifier.notify_one();
}
void waitFinished()
{
{
std::unique_lock<std::mutex> lock(m_jobMutex);
m_finished.wait(lock, [this] {return m_jobQueue.empty(); }); //&& busy == 0
}
m_shutdown.store(true, std::memory_order_relaxed);
m_notifier.notify_all();
for (std::thread& th : m_threads)
{
th.join();
}
m_threads.clear();
}
private:
using Job = std::function<void()>;
std::vector<std::thread> m_threads;
std::queue<Job> m_jobQueue;
std::condition_variable m_notifier;
std::condition_variable m_finished;
std::mutex m_jobMutex;
std::atomic<bool> m_shutdown;
void createThreads(std::size_t numThreads)
{
// Settup threads
m_threads.reserve(numThreads);
for (int i = 0; i != numThreads; ++i)
{
m_threads.emplace_back(std::thread([this]()
{
// Infinite loop to consume tasks from queue and execute
while (true)
{
Job job;
{
std::unique_lock<std::mutex> lock(m_jobMutex);
m_notifier.wait(lock, [this] {return !m_jobQueue.empty() || m_shutdown.load(std::memory_order_relaxed); });
if (m_shutdown.load(std::memory_order_relaxed) || m_jobQueue.empty())
{
break;
}
job = std::move(m_jobQueue.front());
m_jobQueue.pop();
}
job();
m_finished.notify_one();
}
}));
}
}
};
I run this in a simple manner, like the following;
void threader (int x) {
std::cout<<"In threaded function: "<<x<<std::endl;
}
int main()
{
//outer loop
for (auto i = 0; i < 10000; i++) {
//Thread pool
int num_threads = std::thread::hardware_concurrency();
ThreadPool test_pool(num_threads);
// Assign work
for (int j = 0; j < 48; j++) {
test_pool.add_job(std::bind(threader, j));
}
test_pool.waitFinished();
std::cout<<"Thread Pool Done"<<std::endl;
}
}
After a number of outer loop iterations the join in the waitFinished hangs for a thread. The error only seems to occur after a number of iterations of the outer loop large enough. I have investigated this and can see that the threader function get's called 48 times, so looks like all threads complete. It seems to be the joining of threads in the waitFinished function of the ThreadPool that is causing the hang.
Is there something obvious I'm doing wrong ?
Many thanks!