I am using a map<int, queue<string>>, where int refers to the source of a message, and the queue holds the message. One thread pushes messages into the queue, another thread pushes them out of the queue.
This is a client-server program - when the client sends a message, the message gets pushed into the queue.
I am currently using (pseudo code)
/*receive message in thread 1*/
map<int, queue<string>> test_map;
int client_id = 2;
string msg = received_from_client(client_id);
testmap[client_id].push(msg);
/*process message in thread 2*/
string msg_to_process testmap[client_id].front();
test_map[client_id].pop();
if (testmap[client_id].empty())
{
testmap.erase(client_id);
}
I know from this question that the difference is that insert will not overwrite an existing key - does this apply when I am pushing things into queues? Is it safer to use insert, or is what I'm doing with [] sufficient?
Also - while the system should only have one message in the queue at any one time, I am making expansion allowances by using map<int, queue> instead of using map<int,string>.
edit: I have a question about multiple threading as well - what happens when thread 1 attempts to insert into the map while thread 2 deletes the key because the queue is empty (after it has processed the message). Is that a quantitative answer to this, and does using [] or insert() help make it anymore threadsafe?
Queue's don't have keys or [] operators, so your first question can't really be answered. You insert into queue's by pushing onto the back. If there are elements there, it will go after them. You read off a queue by popping things off of the front, if there are any. You don't read or write anywhere other than that.
As for maps, like you said, insert will add a new key-value pair if it does not exist already. It will not overwrite an existing key. Find will find a value if it exists already, but will not insert it if it doesn't. And then the [] operator does both, and also allows you to change existing elements. The documentation here is very good.
One thing to be aware of is that using the map's [] operator to read from the map will also insert a default valuetype element into the map with that key, and is probably not what you would expect when first looking at it.
std::map<int, int> myMap;
if(myMap[1] == 0) //[] create's a key-value pair <1,0>
cout << "This will output";
if(myMap.size() == 1)
cout << "This too";
As for the thread safety aspect, no STL containers are thread safe based on the standard. You need to add proper locking in your code to prevent exactly what you asked about. If 2 threads tried to read and write from a queue at the same time, it will almost definitely cause an error. I would google around about writing thread safe programs for general help on how to do that.
Related
I have 8 threads that process an image in strips. The strips are ordered in raster order. When each thread is finished with a strip, the thread adds its strip id number to a blocking queue. I want the queue to only allow a pop when the number is in sequence from 0 to N. So, regardless of the order in which the threads add their IDs, the queue output will be 0,1,2,3,.....N.
Is there an existing construct in the STL that has this functionality ?
I suppose a simple implementation would be a vanilla queue with a counter starting at 0. When 0 gets added, it pops and moves the counter to 1, and keeps popping until it doesn't find a match. But this sounds inefficient.
Edit: if I wrap an STL priority queue to make it blocking, this could work.
The structure you want is a min heap (see std::priority_queue). This gives the element with the lowest ID.
Wake up the consumer thread every time the newly added element is at the beginning of the queue.
Consume all elements that are in sequence in one go.
This doesn't look like a queue at all! Queue should only support push_back and pop_front. There is no peeking inside.
I would suggest a map<ID,image>, and maintain the last processed image ID. Then you can quickly check if that map's front() is next in your sequence, and remove it.
I have found some similar questions on this subject, but I wanted to ask again in order to get a more clear answer. I am writing a graph matching algorithm, where each node on the graph assigned to a priority set depending on the matching of its neighbours. Details are not really important, but I am using an std::priority_queue in order to match the highest priority nodes first. Here is the tricky point: Each time a new match is introduced, the priority of the neighbours of the matching nodes shall be updated.
This algorithm is referenced from a paper and although I have implemented the exact same algorithm, I couldn't reach the same matching percentage. I naturally suspected that std::priority_queue may not be reordered as I wanted on priority updates, so I have run some tests and then I found out other questions asking the same thing:
How to tell a std::priority_queue to refresh its ordering?
Does changing a priority queue element result in resorting the queue?
My question naturally is, how can I update the order on new matchings? Can I enforce it? Or is there any other data structure (max heap for example) that can serve to this purpose? Note that, pushing new elements into the queue is not a valid solution for me. Here is the code piece I am using (matchFace() function updates the element priorities):
while (priorityQueue.size() != 0) {
// Take the face at the top of the queue and check if it is already matched
FaceData* currentFace = priorityQueue.top();
// Pop the face at the top in any case
priorityQueue.pop();
// If the face is not already matched, try to find a matching
if (!currentFace->matched) {
// Try to match the face with one of its neighbors, add it to the unmatched faces list if it fails
int neighborId = matchFace(currentFace);
if (neighborId == -1) {
unmatchedFaces.push_back(currentFace);
} else {
matchingMap[currentFace->id] = neighborId;
}
}
}
Using the comments that I received on the problem, I decided to answer it myself. I found out there are three possible ways to overcome this problem:
Implement your own updatable priority queue or use external libraries. Boost might have some additional data structures for this purpose. I also found an Updatable Priority Queue source code here.
Use a vector to store the values and use std::make_heap function provided in the algorithm library each time an update is received. This is the easiest way but it works very slow.
Remove and re-insert the elements. If this is not a valid approach, use a map to store the element ids and instead of removing the elements, mark the elements on the map so if you encounter them multiple times you can just ignore them. An alternative strategy is to alter the items by adding a flag and marking the elements by turning the flag on.
I have class:
class A{
//fields, methods
;
I need an efficient data structure that allows you to choose from a variety of pointers to objects of class A minima and maxima (it should work online, that is the choice of questions will alternate with requests for adding new poiters). This can be done by using two priority queues:
priority_queue<A*, vector<A*>, ComparatorForFindingLightestObjects>* qL;
priority_queue<A*, vector<A*>, ComparatorForFindingHardestObjects>* qH;
The problem is that if the object pointer is extracted from the first queue, then after a while the object is destroyed, but since a pointer to the object is still present in another queue there happens errors of reading data from the freed memory.
How solve this problem by means of the standard STL containers without writing own data structures?
I believe you're looking for boost::multi_index which is a single container accessible but multiple different "views": http://www.boost.org/doc/libs/1_59_0/libs/multi_index/doc/index.html
I think you can use std::set and delete the entry from the second set as soon as you extract the data from the first. Performance wise, both give O(log(n)) lookup and insertion. I'm not sure if this is what you want but i'll try
//Use std::set as your priority queue instead
set<A*, ComparatorForFindingLightestObjects> qL;
set<A*, ComparatorForFindingHardestObjects> qH;
auto it=qL.begin(); //The first element
if(it!=aL.end())
{
A* curr=*it;
qL.erase(curr); //Delete it from this
qH.erase(curr); //Delete this from the other queue as well
}
Also, I think you can merge your two queues or whatever and just maintain one container. You can access the minimum and maximum elements by *containerName.begin() and *containerName.rbegin() respectively
I am not very good at data structures, so this might be very silly question. I am looking for a way to implement a hybrid behavior of queue + maps.
I am currently using tbb::concurrent_bounded_queue (documented at Intel's developer zone) from www.threadingbuildingblocks.org in a multithreaded single producer single consumer process. The queue has market data quote objects and the producer side of the process is actually highly time sensitive, so what I need is a queue that is keyed on a market data identifier such as USDCAD, EURUSD. The Value points (through unique_ptr) to most latest market data quote that I received for this key.
So, let us say my queue has 5 elements for 5 unique identifiers and suddenly we get updated market data quote for the identifier at 3rd position in the queue, then I just store the most latest value and discard the value I previously had. So, essentially I just move my unique_ptr to the new market data quote for this key.
It's like it is similar to concurrent_bounded_queue<pair<string, unique_ptr<Quote>>> but is keyed on the first element of the pair.
I am not sure if this is already available in a third-party library (may be tbb itself) or what it is called if it is a standard data structure.
I would highly appreciate any help or guidance on this.
Thanks.
First, observe that we can easily write...
int idn_to_index(idn); // map from identifier to contiguous number sequence
...it doesn't matter much if that uses a std::map or std::unordered_map, binary search in a sorted std::vector, your own character-by-character hardcoded parser....
Then the producer could:
update (using a mutex) a std::vector<unique_ptr<Quote>> at [idn_to_index(idn)]
post the index to concurrent_bounded_queue<int>
The consumer:
pop an index
compares the pointer in std::vector<unique_ptr<Quote>> at [index] to its own array of last-seen pointers, and if they differ process the quote
The idea here is not to avoid having duplicate identifier-specific indices in the queue, but to make sure that the stalest of those still triggers processing of the newest quote, and that less-stale queue entries are ignored harmlessly until the data's genuinely been updated again.
TBB provides
concurrent_undordered_map: no concurrent erase, stable iterators, no element access protection;
concurrent_hash_map: has concurrent erase, concurrent operations invalidate iterators, per-element access management via 'accessors'
So, if the question
"It's like it is similar to concurrent_bounded_queue<pair<string, unique_ptr<Quote>>> but is keyed on the first element of the pair" means suggest a corresponding concurrent associative map container, these two are at your service. Basically, you have to choose between the ability to erase identifiers concurrently (hash_map) and the ability to traverse concurrently across all the elements (unordered_map). concurrent_hash_map also simplifies synchronization of accesses to the elements which looks useful for your case.
I was able to solve this problem as below:
I use a queue and a hashmap both from tbb library. Now, I push my unique identifiers on the queue and not the Quote's. My hashmap has my unique identifier as key and Quote as value
So, when I receive a Quote I iterate through the queue and check whether the queue contains that identifier, if it does, then I insert the corresponding Quote directly into the hashmap and do not add the unique identifier on the queue. If it does not, then I push the identifier on the queue and corresponding Quote in hashmap. This, ensures that my queue always as unique set of identifiers and my hashmap has the most latest Quote available for that identifier.
On the consumer side, I pop the queue to get my next identifier and get the Quote for that identifier from the hashmap.
This works pretty fast. Please let me know in case I am missing any hidden issues with this.
I have a map<uint, Order*> orders where Order is a defined class with applicable fields such as id, time, price and volume. I also have a thread which listens for incoming order additions and deletions for the map defined below.
void System::OrderDeleted_Thread(unsigned int order_id)
{
if(orders.find(order_id) != orders.end())
{
Order* order = orders[order_id];
orders.erase(order_id);
delete order;
}
}
My problem is very similar to this one:
Segmentation fault in std function std::_Rb_tree_rebalance_for_erase ()
My question is, how can I iterate through my orders map without the program giving me an error when it comes time to re balance the tree? Just like the solution in the link says, I have taken out the .erase(uint) method and gotten it to work. Unfortunately, I cannot keep a map of several tens of thousands keys around.
Thanks in advance!
I also have a thread which listens for incoming order additions and deletions for the map defined below.
You need to synchronize access to the map. STL containers are not thread-safe with multiple writers (and erasing elements is writing to the container) without some sort of external synchronization.
Queue up your additions and deletions in a seperate data structure, and then process them at a safe time, that is when you are guaranteed to not be iterating through the map. That safe time can be after you have acquired a mutex which protects the map, or some other way, depending on your program.
Apart from synchronisation issues, that's a costly way to write the loop. Instead, try this:
std::map<uint, Order*>::iterator it;
Order * p = NULL;
{ // enter critical section
if ((it = orders.find(id)) != orders.end())
{
p = it->second;
orders.erase(it);
}
} // leave critical section
delete p;