I'm working with some multithreaded code (working with concurrent data structures), and part of it requires that I lock a set of mutexes. For my implementation I am utilizing a vector of lock_guards, since I don't necessarily know how many mutexes I'm going to need to lock, and I may run into exception conditions which will force me to unlock all of my mutexes and restart. Hence the reason for the vector.
The code I'm trying to use boils down to this:
#include <mutex>
#include <vector>
using namespace std;
int main( int argc, char** argv ) {
vector<recursive_mutex> vec(10);
vector<lock_guard<recursive_mutex>> lgv;
for( auto it = vec.begin(); it != vec.end(); ++it ) {
lgv.emplace_back( *it );
}
return 0;
}
When I try to compile this (G++ 5.3.1 using --std=c++11), I get the following error (somewhat distilled):
In file included from foo.cpp:1:0:
/usr/include/c++/5.3.1/mutex:385:7: note: declared here
lock_guard(const lock_guard&) = delete;
Based upon my understanding of emplace_back, the library should NOT be attempting to use the copy constructor for lock_guard -- it should be performing a construct-in-place. Am I understanding what SHOULD be happening properly, or is my understanding flawed?
FYI, I have attempted to use a unique_lock and this will compile just fine. However, I'm curious about this (apparent) discrepancy.
I assume, the question reflects an attemtpt to use unique_lock, which OP is saying works. The same example with lock_guard would not work, since std::vector::emplace_back requires the type to be both MoveInsertable and EmplaceConstructible, and std::lock_guard does not suit.
Related
I have a priority queue and the compare function references a value accessed by multiple threads. So it has to be protected by a mutex. Except I don't know when this compare function is ran. Is it ran when I push a value or when I pop a value? Example code below.
#include <iostream>
#include <queue>
#include <mutex>
using namespace std;
int main()
{
int compare = 7;
mutex compare_m;
auto cmp = [&](int a, int b) {return abs(compare - a)>=abs(compare-b);};
priority_queue<int, vector<int>, decltype(cmp)> x(cmp);
mutex x_m;
//in thread
{
scoped_lock m1(x_m);
//do I need this?
scoped_lock m(compare_m);
x.push(6);
}
//in thread
{
scoped_lock m1(x_m);
//do I need this?
scoped_lock m(compare_m);
x.pop();
}
}
To answer the question, if it is not documented anything can happen (and then we cannot then reason about when comparator is invoked).
If we take a look into cppreference, push is defined in terms of push_heap, which then reorganizes the elements into a heap. Given it then needs to reorganize, we can reason that it invokes the comparator. A similar situation happens with pop, that invokes pop_heap, which again modifies the underlying heap. So again, invoking comparator.
So the above implies you need a critical section on both (however please notice the comments regarding whether it is actually safe to change the behaviour of comparison function while the pq contains elements).
To my surprise, a C++11 std::thread object that has finished executing, but has not yet been joined is still considered an active thread of execution. This is illustrated in the following code example (built on Xubuntu 13.03 with g++ 4.7.3). Does anyone know if the C++11 standard provides a means to detect if a std::thread object is still actively running code?
#include <thread>
#include <chrono>
#include <iostream>
#include <pthread.h>
#include <functional>
int main() {
auto lambdaThread = std::thread([](){std::cout<<"Excuting lambda thread"<<std::endl;});
std::this_thread::sleep_for(std::chrono::milliseconds(250));
if(lambdaThread.joinable()) {
std::cout<<"Lambda thread has exited but is still joinable"<<std::endl;
lambdaThread.join();
}
return 0;
}
No, I don't think that this is possible. I would also try to think about your design and if such a check is really necessary, maybe you are looking for something like the interruptible threads from boost.
However, you can use std::async - which I would do anyway - and then rely on the features std::future provides you.
Namely, you can call std::future::wait_for with something like std::chrono::seconds(0). This gives you a zero-cost check and enables you to compare the std::future_status returned by wait_for.
auto f = std::async(foo);
...
auto status = f.wait_for(std::chrono::seconds(0));
if(status == std::future_status::timeout) {
// still computing
}
else if(status == std::future_status::ready) {
// finished computing
}
else {
// There is still std::future_status::defered
}
for what definition of "actively running code"? not that I know of, I'm not sure what state the thread is left in after it becomes joinable, in most cases I can think of you'd actually want fine grain control, like a flag set by the code running in that thread, anyway
for a platform specific solution, you could use GetThreadTimes
I am changing a single thread program into multi thread using boost:thread library. The program uses unordered_map as a hasp_map for lookups. My question is..
At one time many threads will be writing, and at another many will be reading but not both reading and writing at the same time i.e. either all the threads will be reading or all will be writing. Will that be thread safe and the container designed for this? And if it will be, will it really be concurrent and improve performance? Do I need to use some locking mechanism?
I read somewhere that the C++ Standard says the behavior will be undefined, but is that all?
UPDATE: I was also thinking about Intel concurrent_hash_map. Will that be a good option?
STL containers are designed so that you are guaranteed to be able to have:
A. Multiple threads reading at the same time
or
B. One thread writing at the same time
Having multiple threads writing is not one of the above conditions and is not allowed. Multiple threads writing will thus create a data race, which is undefined behavior.
You could use a mutex to fix this. A shared_mutex (combined with shared_locks) would be especially useful as that type of mutex allows multiple concurrent readers.
http://eel.is/c++draft/res.on.data.races#3 is the part of the standard which guarantees the ability to concurrently use const functions on different threads. http://eel.is/c++draft/container.requirements.dataraces specifies some additional non-const operations which are safe on different threads.
std::unordered_map meets the requirements of Container (ref http://en.cppreference.com/w/cpp/container/unordered_map). For container thread safety see: http://en.cppreference.com/w/cpp/container#Thread_safety.
Important points:
"Different elements in the same container can be modified concurrently by different threads"
"All const member functions can be called concurrently by different threads on the same container. In addition, the member functions begin(), end(), rbegin(), rend(), front(), back(), data(), find(), lower_bound(), upper_bound(), equal_range(), at(), and, except in associative containers, operator[], behave as const for the purposes of thread safety (that is, they can also be called concurrently by different threads on the same container)."
Will that be thread safe and the container designed for this?
No, the standard containers are not thread safe.
Do I need to use some locking mechanism?
Yes, you do. Since you're using boost, boost::mutex would be a good idea; in C++11, there's std::mutex.
I read somewhere that the C++ Standard says the behavior will be undefined, but is that all?
Indeed, the behaviour is undefined. I'm not sure what you mean by "is that all?", since undefined behaviour is the worst possible kind of behaviour, and a program that exhibits it is by definition incorrect. In particular, incorrect thread synchronisation is likely to lead to random crashes and data corruption, often in ways that are very difficult to diagnose, so you would be wise to avoid it at all costs.
UPDATE: I was also thinking about Intel concurrent_hash_map. Will that be a good option?
It sounds good, but I've never used it myself so I can't offer an opinion.
The existing answers cover the main points:
you must have a lock to read or write to the map
you could use a multiple-reader / single-writer lock to improve concurrency
Also, you should be aware that:
using an earlier-retrieved iterator, or a reference or pointer to an item in the map, counts as a read or write operation
write operations performed in other threads may invalidate pointers/references/iterators into the map, much as they would if they were done in the same thread, even if a lock is again acquired before an attempt is made to continue using them...
You can use concurrent_hash_map or employ an mutex when you access unordered_map. one of issue on using intel concurrent_hash_map is you have to include TBB, but you already use boost.thread. These two components have overlapped functionality, and hence complicate your code base.
std::unordered_map is a good fit for some multi-threaded situations.
There are also other concurrent maps from Intel TBB:
tbb:concurrent_hash_map. It supports fine-grained, per-key locking for insert/update, which is something that few other hashmaps can offer. However, the syntax is slightly more wordy. See full sample code. Recommended.
tbb:concurrent_unordered_map. It is essentially the same thing, a key/value map. However, it is much lower level, and more difficult to use. One has to supply a hasher, a equality operator, and an allocator. There is no sample code anywhere, even in the official Intel docs. Not recommended.
If you don't need all of the functionality of unordered_map, then this solution should work for you. It uses mutex to control access to the internal unordered_map. The solution supports the following methods, and adding more should be fairly easy:
getOrDefault(key, defaultValue) - returns the value associated with key, or defaultValue if no association exists. A map entry is not created when no association exists.
contains(key) - returns a boolean; true if the association exists.
put(key, value) - create or replace association.
remove(key) - remove association for key
associations() - returns a vector containing all of the currently known associations.
Sample usage:
/* SynchronizedMap
** Functional Test
** g++ -O2 -Wall -std=c++11 test.cpp -o test
*/
#include <iostream>
#include "SynchronizedMap.h"
using namespace std;
using namespace Synchronized;
int main(int argc, char **argv) {
SynchronizedMap<int, string> activeAssociations;
activeAssociations.put({101, "red"});
activeAssociations.put({102, "blue"});
activeAssociations.put({102, "green"});
activeAssociations.put({104, "purple"});
activeAssociations.put({105, "yellow"});
activeAssociations.remove(104);
cout << ".getOrDefault(102)=" << activeAssociations.getOrDefault(102, "unknown") << "\n";
cout << ".getOrDefault(112)=" << activeAssociations.getOrDefault(112, "unknown") << "\n";
if (!activeAssociations.contains(104)) {
cout << 123 << " does not exist\n";
}
if (activeAssociations.contains(101)) {
cout << 101 << " exists\n";
}
cout << "--- associations: --\n";
for (auto e: activeAssociations.associations()) {
cout << e.first << "=" << e.second << "\n";
}
}
Sample output:
.getOrDefault(102)=green
.getOrDefault(112)=unknown
123 does not exist
101 exists
--- associations: --
105=yellow
102=green
101=red
Note1: The associations() method is not intended for very large datasets as it will lock the table during the creation of the vector list.
Note2: I've purposefully not extended unordered_map in order to prevent my self from accidentally using a method from unordered_map that has not been extended and therefore might not be thread safe.
#pragma once
/*
* SynchronizedMap.h
* Wade Ryan 20200926
* c++11
*/
#include <unordered_map>
#include <mutex>
#include <vector>
#ifndef __SynchronizedMap__
#define __SynchronizedMap__
using namespace std;
namespace Synchronized {
template <typename KeyType, typename ValueType>
class SynchronizedMap {
private:
mutex sync;
unordered_map<KeyType, ValueType> usermap;
public:
ValueType getOrDefault(KeyType key, ValueType defaultValue) {
sync.lock();
ValueType rs;
auto value=usermap.find(key);
if (value == usermap.end()) {
rs = defaultValue;
} else {
rs = value->second;
}
sync.unlock();
return rs;
}
bool contains(KeyType key) {
sync.lock();
bool exists = (usermap.find(key) != usermap.end());
sync.unlock();
return exists;
}
void put(pair<KeyType, ValueType> nodePair) {
sync.lock();
if (usermap.find(nodePair.first) != usermap.end()) {
usermap.erase(nodePair.first);
}
usermap.insert(nodePair);
sync.unlock();
}
void remove(KeyType key) {
sync.lock();
if (usermap.find(key) != usermap.end()) {
usermap.erase(key);
}
sync.unlock();
}
vector<pair<KeyType, ValueType>> associations() {
sync.lock();
vector<pair<KeyType, ValueType>> elements;
for (auto it=usermap.begin(); it != usermap.end(); ++it) {
pair<KeyType, ValueType> element (it->first, it->second);
elements.push_back( element );
}
sync.unlock();
return elements;
}
};
}
#endif
I'm trying to implement what I think is a fairly simple design. I have a bunch of objects, each containing a std::map and there will be multiple processes accessing them. I want to make sure that there is only one insert/erase to each of these maps at a time.
So I've been reading about boost::thread and class member mutexes and using bind to pass to class member which are all new things to me. I started with a simple example from a Dr. Dobbs article and tried modifying that. I was getting all kinds of compiler errors due to my Threaded object having to be noncopyable. After reading up on that, I decided I can avoid the hassle by keeping a pointer to a mutex instead. So now I have code that compiles but results in the following error:
/usr/include/boost/shared_ptr.hpp:419:
T* boost::shared_ptr< <template-parameter-1-1> >::operator->() const
[with T = boost::mutex]: Assertion `px != 0' failed. Abort
Now I'm really stuck and would really appreciate help with the code as well as comments on where I'm going wrong conceptually. I realize there are some answered questions around these issues here already but I guess I'm still missing something.
#include <boost/thread/thread.hpp>
#include <boost/thread/mutex.hpp>
#include <boost/bind.hpp>
#include <boost/shared_ptr.hpp>
#include <iostream>
#include <map>
using namespace std;
class Threaded {
public:
std::map<int,int> _tsMap;
void count(int id) {
for (int i = 0; i < 100; ++i) {
_mx->lock();
//std::cout << id << ": " << i << std::endl;
_tsMap[i] ++;
_mx->unlock();
}
}
private:
boost::shared_ptr<boost::mutex> _mx;
};
int main(int argc, char* argv[]) {
Threaded th;
int i = 1;
boost::thread thrd1(boost::bind(&Threaded::count, &th, 1));
//boost::thread thrd2(boost::bind(&th.count, 2));
thrd1.join();
//thrd2.join();
return 0;
}
It looks like you're missing a constructor in your Threaded class that creates the mutex that _mx is intended to point at. In its current state (assuming you ran this code just as it is), the default constructor for Threaded calls the default constructor for shared_ptr, resulting in a null pointer (which is then dereferenced in your count() function.
You should add a constructor along the following lines:
Threaded::Threaded(int id)
: _mx(new boost::mutex())
, _mID(id)
{
}
Then you could remove the argument from your count function as well.
A mutex is non-copyable for good reasons. Trying to outsmart the compiler by using a pointer to a mutex is a really bad idea. If you succeed, the compiler will fail to notice the problems, but they will still be there and will turn round and bite you at runtime.
There are two solutions
store the mutex in your class as a static
store the mutex outside your class.
There are advantages for both - I prefer the second.
For some more discussion of this, see my answer here mutexes with objects
Conceptually, I think you do have a problem. Copying a std::shared_ptr will just increase its reference count, and the different objects will all use the same underlying mutex - meaning that whenever one of your objects is used, none of the rest of them can be used.
You, on the other hand, need each object to get its own mutex guard which is unrelated to other objects mutex guards.
What you need is to keep the mutex defined in the class private section as it is - but ensure that your copy constructor and copy assignment operator are overloaded to create a new one from scratch - one bearing no relation to the mutex in the object being copied/assigned from.
I posted a previous question "Seg Fault when using std::string on an embedded Linux platform" where I got some very useful advise. I have been away on other projects since then and have recently returned to looking at this issue.
To reiterate, I am restricted to using the arm-linux cross compiler (version 2.95.2) as this is what is supplied and supported by the embedded platform vendor. I understand that the issue is likely because the stdlib is very old, and not particularly thread safe.
The problem is that whenever I use the STL containers in multiple threads, I end up with a segmentation fault. The code below will consistently seg fault unless I use pthread_mutex_lock and scope operators around the container declarations (as in other post).
It is not feasible to use this approach in my application as I pass the containers around to different methods and classes. I would ideally like to solve this problem, or find a suitable alternative. I have tried STLPort and SGI's Standard Template Library with the same results. I can only assume that because they are being linked by the very old gcc, they cannot solve the problem.
Does anyone have any possible recommendations or solutions? Or perhaps you can suggest an implementation of vector (and string) that I can drop into my code?
Thanks in advance for any guidance.
#include <stdio.h>
#include <vector>
#include <list>
#include <string>
using namespace std;
/////////////////////////////////////////////////////////////////////////////
class TestSeg
{
static pthread_mutex_t _logLock;
public:
TestSeg()
{
}
~TestSeg()
{
}
static void* TestThread( void *arg )
{
int i = 0;
while ( i++ < 10000 )
{
printf( "%d\n", i );
WriteBad( "Function" );
}
pthread_exit( NULL );
}
static void WriteBad( const char* sFunction )
{
//pthread_mutex_lock( &_logLock );
//{
printf( "%s\n", sFunction );
string sKiller; // <----------------------------------Bad
//list<char> killer; // <----------------------------------Bad
//vector<char> killer; // <----------------------------------Bad
//}
//pthread_mutex_unlock( &_logLock );
return;
}
void RunTest()
{
int threads = 100;
pthread_t _rx_thread[threads];
for ( int i = 0 ; i < threads ; i++ )
{
pthread_create( &_rx_thread[i], NULL, TestThread, NULL );
}
for ( int i = 0 ; i < threads ; i++ )
{
pthread_join( _rx_thread[i], NULL );
}
}
};
pthread_mutex_t TestSeg::_logLock = PTHREAD_MUTEX_INITIALIZER;
int main( int argc, char *argv[] )
{
TestSeg seg;
seg.RunTest();
pthread_exit( NULL );
}
The issue is not with the containers, it's with your code.
It is completely unnecessary to make the containers themselves threadsafe, because what you need, first and foremost, is transaction-like semantics.
Let's assume, for the sake of demonstration, that you have a threadsafe implementation of vector, for example.
Thread 1: if (!vec.empty())
Thread 2: vec.clear();
Thread 1: foo = vec.front();
This leads to undefined behavior.
The issue is that having each operation on the container threadsafe is pretty much pointless because you are still required to be able to lock the container itself for several operations in a row. Therefore you would lock for your various operations, and then lock again on each and every operation ?
As I said: completely unnecessary.
Part of your query might be answered in another thread. The design of C++, including the standard library, is influenced by many factors. Efficiency is a repeated theme. Thread safety mechanisms often are at odds with an objective of efficiency. The age of the library is not really the issue.
For your situation, you may be able to wrap the STL vector in your own vector class (you might consider a Decorator) that contains the locking mechanism and provides the lock/unlock logic around accesses.