I have simple code: first thread pushes std::strings to the std::list, and second thread pops std::strings from this std::list. All std::lists operations are protected with std::mutex m. This code permanently prints error to console: "Error: lst.begin() == lst.end()".
If I replace std::lock_guard with construction m.lock() and m.unlock() the code begins work correctly. What is wrong with std::lock_guard?
#include <iostream>
#include <thread>
#include <mutex>
#include <list>
#include <string>
std::mutex m;
std::list<std::string> lst;
void f2()
{
for (int i = 0; i < 5000; ++i)
{
std::lock_guard<std::mutex> { m };
lst.push_back(std::to_string(i));
}
m.lock();
lst.push_back("-1"); // last list's element
m.unlock();
}
void f1()
{
std::string str;
while (true)
{
m.lock();
if (!lst.empty())
{
if (lst.begin() == lst.end())
{
std::cerr << "Error: lst.begin() == lst.end()" << std::endl;
}
str = lst.front();
lst.pop_front();
m.unlock();
if (str == "-1")
{
break;
}
}
else
{
m.unlock();
std::this_thread::yield();
}
}
}
// tested in MSVS2017
int main()
{
std::thread tf2{ f2 };
f1();
tf2.join();
}
You did not obey CppCoreGuidelines CP.44: Remember to name your lock_guards and unique_locks :).
In
for (int i = 0; i < 5000; ++i)
{
std::lock_guard<std::mutex> { m };
lst.push_back(std::to_string(i));
}
you are only creating a temporary std::lock_guard object which is created and destroyed immediately. You need to name the object like in
{
std::lock_guard<std::mutex> lg{ m };
lst.push_back(std::to_string(i));
}
so that the lock guard lives until the end of the block.
And as you already recognized (CppCoreGuidelines):
Use RAII lock guards (lock_guard, unique_lock, shared_lock), never call mutex.lock and mutex.unlock directly (RAII)
If you are using Microsoft Visual Studio, I recommend using the code analysis and activating at least the Microsoft Native Recommended Rules. If you do this you will get a compiler analysis warning.
warning C26441: Guard objects must be named (cp.44).
Related
Currently working on a project, im struggeling with threading and queue at the moment, the issue is that all threads take the same item in the queue.
Reproduceable example:
#include <iostream>
#include <queue>
#include <thread>
using namespace std;
void Test(queue<string> queue){
while (!queue.empty()) {
string proxy = queue.front();
cout << proxy << "\n";
queue.pop();
}
}
int main()
{
queue<string> queue;
queue.push("101.132.186.39:9090");
queue.push("95.85.24.83:8118");
queue.push("185.211.193.162:8080");
queue.push("87.106.37.89:8888");
queue.push("159.203.61.169:8080");
std::vector<std::thread> ThreadVector;
for (int i = 0; i <= 10; i++){
ThreadVector.emplace_back([&]() {Test(queue); });
}
for (auto& t : ThreadVector){
t.join();
}
ThreadVector.clear();
return 0;
}
You are giving each thread its own copy of the queue. I imagine that what you want is all the threads to work on the same queue and for that you will need to use some synchronization mechanism when multiple threads work on the shared queue as std queue is not thread safe.
edit: minor note: in your code you are spawning 11 threads not 10.
edit 2: OK, try this one to begin with:
std::mutex lock_work;
std::mutex lock_io;
void Test(queue<string>& queue){
while (!queue.empty()) {
string proxy;
{
std::lock_guard<std::mutex> lock(lock_work);
proxy = queue.front();
queue.pop();
}
{
std::lock_guard<std::mutex> lock(lock_io);
cout << proxy << "\n";
}
}
}
Look at this snippet:
void Test(std::queue<std::string> queue) { /* ... */ }
Here you pass a copy of the queue object to the thread.
This copy is local to each thread, so it gets destroyed after every thread exits so in the end your program does not have any effect on the actual queue object that resides in the main() function.
To fix this, you need to either make the parameter take a reference or a pointer:
void Test(std::queue<std::string>& queue) { /* ... */ }
This makes the parameter directly refer to the queue object present inside main() instead of creating a copy.
Now, the above code is still not correct since queue is prone to data-race and neither std::queue nor std::cout is thread-safe and can get interrupted by another thread while currently being accessed by one. To prevent this, use a std::mutex:
// ...
#include <mutex>
// ...
// The mutex protects the 'queue' object from being subjected to data-race amongst different threads
// Additionally 'io_mut' is used to protect the streaming operations done with 'std::cout'
std::mutex mut, io_mut;
void Test(std::queue<std::string>& queue) {
std::queue<std::string> tmp;
{
// Swap the actual object with a local temporary object while being protected by the mutex
std::lock_guard<std::mutex> lock(mut);
std::swap(tmp, queue);
}
while (!tmp.empty()) {
std::string proxy = tmp.front();
{
// Call to 'std::cout' needs to be synchronized
std::lock_guard<std::mutex> lock(io_mut);
std::cout << proxy << "\n";
}
tmp.pop();
}
}
This synchronizes each thread call and prevents access from any other threads while queue is still being accessed by a thread.
Edit:
Alternatively, it'd be much faster in my opinion to make each thread wait until one of them receives a notification of your push to std::queue. You can do this through the use of std::condition_variable:
// ...
#include <mutex>
#include <condition_variable>
// ...
std::mutex mut1, mut2;
std::condition_variable cond;
void Test(std::queue<std::string>& queue, std::chrono::milliseconds timeout = std::chrono::milliseconds{10}) {
std::unique_lock<std::mutex> lock(mut1);
// Wait until 'queue' is not empty...
cond.wait(lock, [queue] { return queue.empty(); });
while (!queue.empty()) {
std::string proxy = std::move(queue.front());
std::cout << proxy << "\n";
queue.pop();
}
}
// ...
int main() {
std::queue<string> queue;
std::vector<std::thread> ThreadVector;
for (int i = 0; i <= 10; i++)
ThreadVector.emplace_back([&]() { Test(queue); });
// Notify the vectors of each 'push()' call to 'queue'
{
std::unique_lock<std::mutex> lock(mut2);
queue.push("101.132.186.39:9090");
cond.notify_one();
}
{
std::unique_lock<std::mutex> lock(mut2);
queue.push("95.85.24.83:8118");
cond.notify_one();
}
{
std::unique_lock<std::mutex> lock(mut2);
queue.push("185.211.193.162:8080");
cond.notify_one();
}
{
std::unique_lock<std::mutex> lock(mut2);
queue.push("87.106.37.89:8888");
cond.notify_one();
}
{
std::unique_lock<std::mutex> lock(mut2);
queue.push("159.203.61.169:8080");
cond.notify_one();
}
for (auto& t : ThreadVector)
t.join();
ThreadVector.clear();
}
For below program, thread Pool always picks the same thread ID 0x7000095f9000! Why so?
Should every push condi.notify_one() wake up all threads same time? What could be the reason same thread ID get picked?
Computer supports 3 threads.
Any other info on using function objects would be helpful!!
O/P
Checking if not empty
Not Empty
0x700009576000 0
Checking if not empty
Checking if not empty
Checking if not empty
Not Empty
0x7000095f9000 1
Checking if not empty
Not Empty
0x7000095f9000 2
Checking if not empty
Not Empty
0x7000095f9000 3
Checking if not empty
Not Empty
0x7000095f9000 4
Checking if not empty
Not Empty
0x7000095f9000 5
Checking if not empty
Code
#include <iostream>
#include <vector>
#include <queue>
#include <thread>
#include <condition_variable>
#include <chrono>
using namespace std;
class TestClass{
public:
void producer(int i) {
unique_lock<mutex> lockGuard(mtx);
Q.push(i);
cond.notify_all();
}
void consumer() {
{
unique_lock<mutex> lockGuard(mtx);
cout << "Checking if not empty" << endl;
cond.wait(lockGuard, [this]() {
return !Q.empty();
});
cout << "Not Empty" << endl;
cout << this_thread::get_id()<<" "<<Q.front()<<endl;
Q.pop();
}
};
void consumerMain() {
while(1) {
consumer();
std::this_thread::sleep_for(chrono::seconds(1));
}
}
private:
mutex mtx;
condition_variable cond;
queue<int> Q;
};
int main()
{
std::vector<std::thread> vecOfThreads;
std::function<void(TestClass&)> func = [&](TestClass &obj) {
while(1) {
obj.consumer();
}
};
unsigned MAX_THREADS = std::thread::hardware_concurrency()-1;
TestClass obj;
for(int i=0; i<MAX_THREADS; i++) {
std::thread th1(func, std::ref(obj));
vecOfThreads.emplace_back(std::move(th1));
}
for(int i=0; i<4*MAX_THREADS/2; i++) {
obj.producer(i);
}
for (std::thread & th : vecOfThreads)
{
if (th.joinable())
th.join();
}
return 0;
}
Any other info on using function objects would be helpful!! Thanks in advance!!
Any other pointers?
The very short unlocking of the mutex that happens in the consumer threads will in your case most probably let the running thread acquire the lock again, and again and again.
If you instead simulate some work being done after the workload has been picked from the queue by calling consumerMain (which sleeps a little) instead of consumer, you would likely see different threads picking up the workload.
while(1) {
obj.consumerMain();
}
Background: I have a list of files in a location and moveFile() function that will be used to move these files. my goal is to move all those files in parallel. So, I implemented multiple threads.
To avoid conflict initially I considered mutex lock before moveFile(). That will prevent threads to run in parallel.
Here's how it's been implemented:
std::mutex mtx;
enum class status
{ SUCCESS, FAILED };
status moveFile()
{ //function implementation }
void fileOperator()
{ // This is prevent parallel operation
mtx.lock;
moveFile();
mtx.unlock;
}
int main ()
{
int threadSize= 3; //generic size
std::thread fileProc[threadSize];
int index = 0;
// staring new threads
for (index; index < threadSize; index++)
{
fileProc[index] = std::thread (&fileOperator);
}
//joining all the threads
for (int i=0; i <threadSize; i++)
{
fileProc[i].join();
}
return 0;
}
Suggestion: I was wondering, if I remove mutex lock implement the moveFile() as in a class and call it as an object method, will it be a better way to implement parallel operation?
Not really sure what the problem here is, most probably it's located in the moveFile function but something like this should work:
#include <future>
#include <iostream>
#include <mutex>
#include <thread>
#include <vector>
std::mutex mtx;
enum class status { SUCCESS, FAILED };
status moveFile() {
std::cout << "Moving file" << std::endl;
return status::SUCCESS;
}
void fileOperator() {
std::lock_guard<std::mutex> lock(mtx);
moveFile();
}
int main(int argc, char** argv) {
std::vector<std::thread> threads;
int threadSize = 3;
for (int index = 0; index < threadSize; ++index) {
threads.push_back(std::thread(&fileOperator));
}
for (auto& th : threads) {
th.join();
}
return 0;
}
Could you also please post the contents of the moveFile to be able to help you with that? Thanks.
I need to run some number of threads to process an array of objects.
So I've written this piece of code :
unsigned int object_counter = 0;
while(object_counter != (obj_max - left))
{
thread genThread[thread_num];//create thread objects
///launch threads
int thread_index = 0;
for (; thread_index<thread_num; thread_index++)
{
genThread[thread_index] = thread(object[object_counter].gen_maps());//launch a thread
object_counter++;
if(object_counter == (obj_max - left)
{
break;
}
}
///finish threads
for (; thread_index>0; thread_index--)
{
genThread[thread_index].join();
}
}
Basically, there is an array of objects (number of objects = obj_max - left).
Each object has a function (void type function) called gen_maps() that generates a terrain.
What I want to do is running all gen_maps() functions from all objects using multithreading.
A maximum number of threads is stored in thread_num variable.
But when I'm trying to compile this code I'm getting an error:
error: invalid use of void expression
genThread[thread_index] = thread(object[object_counter].gen_maps(), thread_index);//launch a thread
^
How can I fix this issue?
A more extendable way to manage an arbitrarily large number of jobs with a smaller number of threads is to use a thread pool.
Here's a naive implementation (for better efficiency there would be 2 condition variables to manage control and state reporting) which allows the initiator to add an arbitrary number of jobs or threads and wait for all jobs to be complete.
#include <thread>
#include <condition_variable>
#include <mutex>
#include <vector>
#include <functional>
#include <deque>
#include <cassert>
#include <ciso646>
#include <iostream>
struct work_pool
{
std::mutex control_mutex;
std::condition_variable control_cv;
std::deque<std::function<void()>> jobs;
bool terminating = false;
std::size_t running = 0;
std::vector<std::thread> threads;
work_pool(std::size_t n = std::thread::hardware_concurrency())
{
add_threads(n);
}
work_pool(const work_pool&) = delete;
work_pool& operator=(const work_pool&) = delete;
~work_pool()
{
wait();
shutdown();
}
void add_threads(std::size_t n)
{
while (n--)
{
threads.emplace_back([this]{
run_jobs();
});
}
}
void run_jobs()
{
while (1)
{
auto lock = std::unique_lock(control_mutex);
control_cv.wait(lock, [this] {
return terminating or not jobs.empty();
});
if (terminating) return;
++running;
auto job = std::move(jobs.front());
jobs.pop_front();
lock.unlock();
job();
lock.lock();
--running;
lock.unlock();
control_cv.notify_one();
}
}
void shutdown()
{
auto lock = std::unique_lock(control_mutex);
terminating = true;
lock.unlock();
control_cv.notify_all();
for (auto&& t : threads) {
if (t.joinable()) {
t.join();
}
}
threads.clear();
}
void wait()
{
auto lock = std::unique_lock(control_mutex);
control_cv.wait(lock, [this] {
return jobs.empty() and not running;
});
}
template<class F>
void add_work(F&& f)
{
auto lock = std::unique_lock(control_mutex);
assert(not terminating);
jobs.emplace_back(std::forward<F>(f));
lock.unlock();
control_cv.notify_all();
}
};
// dummy function for exposition
void generate_map() {}
int main()
{
work_pool pool;
for(int i = 0 ; i < 100000 ; ++i)
pool.add_work(generate_map);
pool.wait();
// maps are now all generated
std::cout << "done" << std::endl;
}
With object[object_counter].gen_maps() you call the function gen_maps and use the returned value as the thread function. Apparently gen_maps is declared to return void which leads to the error you get.
You need to pass a pointer to the function, and then pass the object it should be called on as an argument to the thread:
thread(&SomeClass::gen_maps, object[object_counter])
I can't get code working reliably in a simple VS2012 console application consisting of a producer and consumer that uses a C++11 condition variable. I am aiming at producing a small reliable program (to use as the basis for a more complex program) that uses the 3 argument wait_for method or perhaps the wait_until method from code I have gathered at these websites:
condition_variable:
wait_for,
wait_until
I'd like to use the 3 argument wait_for with a predicate like below except it will need to use a class member variable to be most useful to me later. I am receiving "Access violation writing location 0x__" or "An invalid parameter was passed to a service or function" as errors after only about a minute of running.
Would steady_clock and the 2 argument wait_until be sufficient to replace the 3 argument wait_for? I've also tried this without success.
Can someone show how to get the code below to run indefinitely with no bugs or weird behavior with either changes in wall-clock time from daylight savings time or Internet time synchronizations?
A link to reliable sample code could be just as helpful.
// ConditionVariable.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
#include <condition_variable>
#include <mutex>
#include <thread>
#include <iostream>
#include <queue>
#include <chrono>
#include <atomic>
#define TEST1
std::atomic<int>
//int
qcount = 0; //= ATOMIC_VAR_INIT(0);
int _tmain(int argc, _TCHAR* argv[])
{
std::queue<int> produced_nums;
std::mutex m;
std::condition_variable cond_var;
bool notified = false;
unsigned int count = 0;
std::thread producer([&]() {
int i = 0;
while (1) {
std::this_thread::sleep_for(std::chrono::microseconds(1500));
std::unique_lock<std::mutex> lock(m);
produced_nums.push(i);
notified = true;
qcount = produced_nums.size();
cond_var.notify_one();
i++;
}
cond_var.notify_one();
});
std::thread consumer([&]() {
std::unique_lock<std::mutex> lock(m);
while (1) {
#ifdef TEST1
// Version 1
if (cond_var.wait_for(
lock,
std::chrono::microseconds(1000),
[&]()->bool { return qcount != 0; }))
{
if ((count++ % 1000) == 0)
std::cout << "consuming " << produced_nums.front () << '\n';
produced_nums.pop();
qcount = produced_nums.size();
notified = false;
}
#else
// Version 2
std::chrono::steady_clock::time_point timeout1 =
std::chrono::steady_clock::now() +
//std::chrono::system_clock::now() +
std::chrono::milliseconds(1);
while (qcount == 0)//(!notified)
{
if (cond_var.wait_until(lock, timeout1) == std::cv_status::timeout)
break;
}
if (qcount > 0)
{
if ((count++ % 1000) == 0)
std::cout << "consuming " << produced_nums.front() << '\n';
produced_nums.pop();
qcount = produced_nums.size();
notified = false;
}
#endif
}
});
while (1);
return 0;
}
Visual Studio Desktop Express had 1 important update which it installed and Windows Update has no other important updates. I'm using Windows 7 32-bit.
Sadly, this is actually a bug in VS2012's implementation of condition_variable, and the fix will not be patched in. You'll have to upgrade to VS2013 when it's released.
See:
http://connect.microsoft.com/VisualStudio/feedback/details/762560
First of all, while using condition_variables I personally prefer some wrapper classes like AutoResetEvent from C#:
struct AutoResetEvent
{
typedef std::unique_lock<std::mutex> Lock;
AutoResetEvent(bool state = false) :
state(state)
{ }
void Set()
{
auto lock = AcquireLock();
state = true;
variable.notify_one();
}
void Reset()
{
auto lock = AcquireLock();
state = false;
}
void Wait(Lock& lock)
{
variable.wait(lock, [this] () { return this->state; });
state = false;
}
void Wait()
{
auto lock = AcquireLock();
Wait(lock);
}
Lock AcquireLock()
{
return Lock(mutex);
}
private:
bool state;
std::condition_variable variable;
std::mutex mutex;
};
This may not be the same behavior as C# type or may not be as efficient as it should be but it gets things done for me.
Second, when I need to implement a producing/consuming idiom I try to use a concurrent queue implementation (eg. tbb queue) or write a one for myself. But you should also consider making things right by using Active Object Pattern. But for simple solution we can use this:
template<typename T>
struct ProductionQueue
{
ProductionQueue()
{ }
void Enqueue(const T& value)
{
{
auto lock = event.AcquireLock();
q.push(value);
}
event.Set();
}
std::size_t GetCount()
{
auto lock = event.AcquireLock();
return q.size();
}
T Dequeue()
{
auto lock = event.AcquireLock();
event.Wait(lock);
T value = q.front();
q.pop();
return value;
}
private:
AutoResetEvent event;
std::queue<T> q;
};
This class has some exception safety issues and misses const-ness on the methods but like I said, for a simple solution this should fit.
So as a result your modified code looks like this:
int main(int argc, char* argv[])
{
ProductionQueue<int> produced_nums;
unsigned int count = 0;
std::thread producer([&]() {
int i = 0;
while (1) {
std::this_thread::sleep_for(std::chrono::microseconds(1500));
produced_nums.Enqueue(i);
qcount = produced_nums.GetCount();
i++;
}
});
std::thread consumer([&]() {
while (1) {
int item = produced_nums.Dequeue();
{
if ((count++ % 1000) == 0)
std::cout << "consuming " << item << '\n';
qcount = produced_nums.GetCount();
}
}
});
producer.join();
consumer.join();
return 0;
}