Note: This is not a question whether I should "use list or deque". It's a question about the validity of iterators in the face of insert().
This may be a simple question and I'm just too dense to see the right way to do this. I'm implementing (for better or worse) a network traffic buffer as a std::list<char> buf, and I'm maintaining my current read position as an iterator readpos.
When I add data, I do something like
buf.insert(buf.end(), newdata.begin(), newdata.end());
My question is now, how do I keep the readpos iterator valid? If it points to the middle of the old buf, then it should be fine (by the iterator guarantees for std::list), but typically I may have read and processed all data and I have readpos == buf.end(). After the insertion, I want readpos always to point to the next unread character, which in case of the insertion should be the first inserted one.
Any suggestions? (Short of changing the buffer to a std::deque<char>, which appears to be much better suited to the task, as suggested below.)
Update: From a quick test with GCC4.4 I observe that deque and list behave differently with respect to readpos = buf.end(): After inserting at the end, readpos is broken in a list, but points to the next element in a deque. Is this a standard guarantee?
(According to cplusplus, any deque::insert() invalidated all iterators. That's no good. Maybe using a counter is better than an iterator to track a position in a deque?)
if (readpos == buf.begin())
{
buf.insert(buf.end(), newdata.begin(), newdata.end());
readpos = buf.begin();
}
else
{
--readpos;
buf.insert(buf.end(), newdata.begin(), newdata.end());
++readpos;
}
Not elegant, but it should work.
From http://www.sgi.com/tech/stl/List.html
"Lists have the important property that insertion and splicing do not invalidate iterators to list elements, and that even removal invalidates only the iterators that point to the elements that are removed."
Therefore, readpos should still be valid after the insert.
However...
std::list< char > is a very inefficient way to solve this problem. Each byte you store in a std::list requires a pointer to keep track of the byte, plus the size of the list node structure, two more pointers usually. That is at least 12 or 24 bytes (32 or 64-bit) of memory used to keep track of a single byte of data.
std::deque< char> is probably a better container for this. Like std::vector it provides constant time insertions at the back however it also provides constant time removal at the front. Finally, like std::vector std::deque is a random-access container so you can use offsets/indexes instead of iterators. These three features make it an efficient choice.
I was indeed being dense. The standard gives us all the tools we need. Specifically, the sequence container requirements 23.2.3/9 say:
The iterator returned from a.insert(p, i, j) points to the copy of the first element inserted into a, or p if i == j.
Next, the description of list::insert says (23.3.5.4/1):
Does not affect the validity of iterators and references.
So in fact if pos is my current iterator inside the list which is being consumed, I can say:
auto it = buf.insert(buf.end(), newdata.begin(), newdata.end());
if (pos == buf.end()) { pos = it; }
The range of new elements in my list is [it, buf.end()), and the range of yet unprocessed elements is [pos, buf.end()). This works because if pos was equal to buf.end() before the insertion, then it still is after the insertion, since insertion does not invalidate any iterators, not even the end.
list<char> is a very inefficient way to store a string. It is probably 10-20 times larger than the string itself, plus you are chasing a pointer for every character...
Have you considered using std::dequeue<char> instead?
[edit]
To answer your actual question, adding and removing elements does not invalidate iterators in a list... But end() is still going to be end(). So you would need to check for that as a special case at the point where you insert the new element in order to update your readpos iterator.
Related
I have a simple database consisting of objects with strings containing unix time as keys and strings containing instructions as values
I want to iterate though the database and erase any object who's key is smaller that current time ( so erase objects with dates before current date)
for (auto it = m_jsonData.begin(); it != m_jsonData.end(); it++) {
if (std::stoi(it.key()) <= (std::time(NULL))) {
std::cout << "command is behind schedule, removing\n";
m_jsonData.erase(it);
} else {
/*
*/
}
}
this code works fine as long as m_jsonData.erase(it); isn't invoked. when it does, in the next iteration std::stoi(it.key()) causes a segfault, after a bit of playing with it I came to a conclusion that is somehow loses track of what it's actually iterating. Is my conclusion true? If not then what is? And how do I fix it?
It's extremely normal for mutating container operations to invalidate iterators. It's one of the first things you should check for.
Documentation for nlohnmann::json::erase():
Notes
Invalidates iterators and references at or after the point of the erase, including the end() iterator.
References and iterators to the erased elements are invalidated. Other references and iterators are not affected.
That means after this line:
m_jsonData.erase(it);
the iterator it can't be used for anything including incrementing it to the next element. It is invalid.
Fortunately, the documentation also points out that the successor to the removed element is returned, so you can just write
for (auto it = m_jsonData.begin(); it != m_jsonData.end(); ) {
if (std::stoi(it.key()) <= (std::time(NULL))) {
it = m_jsonData.erase(it);
} else {
++it;
}
}
Note that when I say this is extremely normal, it's because the standard containers often have similar behaviour. See the documentation for examples, but this is something everyone should be aware of:
std::vector::erase Iterator invalidation
std::unordered_map::erase Iterator invalidation
etc.
This is exactly the reason std::erase was added in C++20, and previously std::remove_if was provided to suppport the erase(remove_if(...), end) idiom, instead of writing fragile mutating loops.
The values of a std::unordered_set are not mutable, as they are both key and value. What is the correct way to modify an element of a std::unordered_set, if its element are first removed, modified and then reinserted? The erase does not invalidate any iterators, but the insert may. The obvious answer is to use the iterator, that the erase returns. One way, one could deal with this, I guess, is to reset the loop iterator to the beginning of an unordered_set, after a successful insert. I'd like to make sure, it is the only way.
The erase does not invalidate any iterators, but the insert may.
You can always check beforehand if insert is going to do that:
If rehashing occurs due to the insertion, all iterators are
invalidated. Otherwise iterators are not affected. References are not
invalidated. Rehashing occurs only if the new number of elements is
equal to or greater than max_load_factor()*bucket_count().
(from cppreference)
So, if you watch out for rehashing, your approach could work. Of course it leaves you with the problem of what to do when you detect that rehashing will occur.
You can reduce the probability of rehashing by increasing the capacity of the set before the loop.
The simplest way of dealing with rehashing is to start over after rehashing. Maybe there are other ways to deal with it but I wouldn't risk it.
Having said all these, what you are describing here seems to indicate that you probably need another container. If the unordered_set is really the best container for your application, I would most likely still go with Martin's solution, namely with an intermediate container. It's less messy and I can see what's going on; I can reason about correctness.
I think that you are better off using an intermediate container, like this:
unordered_set<int> original;
...
vector<int> temporary;
for (auto it = original.begin(), itEnd = original.end(); it != itEnd; ) {
if (...) {
int newValue = ...;
auto toDelete = it++;
original.erase(toDelete);
temporary.push_back(newValue);
} else {
...
++it;
}
}
original.insert(temporary.begin(), end.begin());
Use a temporary unordered_set that you will fill iterating on the first one (with an hint) then swap the original set with the temporary.
I would like to traverse a map in C++ with iterators but not all the way to the end.
The problem is that even if we can do basic operations with iterators, we cannot add or compare iterators with integers.
How can I write the following instructions? (final is a map; window, an integer)
for (it=final.begin(); it!=final.end()-window; it++)
You cannot subtract from a map iterator directly, because it is an expensive operation (in practice doing --iter the required number of times). If you really want to do it anyway, you can use the standard library function 'advance'.
map<...>::iterator end = final.end();
std::advance(end, -window);
That will give you the end of your window.
std::map<T1, T2>::iterator it = final.begin();
for (int i = 0; i < final.size()-window; ++i, ++it)
{
// TODO: add your normal loop body
}
Replace T1 and T2 with the actual types of the keys and values of the map.
Why don't you make 'it' an iterator as well ?
See the example here : http://www.cplusplus.com/reference/stl/map/begin/
Another solution:
size_t count=final.size();
size_t processCount=(window<count?count-window:0);
for (it=final.begin(); processCount && it!=final.end(); ++it, --processCount)
{
// loop body
}
This one is a bit safer:
It takes care of the case when your map is actually smaller than the value of window.
It will process at most processCount elements, even if you change the size of your map inside your loop (e.g. add new elements)
According to STL, size() can take O(n) time to compute, although usual implementations can do this in O(1). To be on the safe side, it is better not to call size() many times, if it is not necessary.
'end()' on the other hand has amortized constant time, so it should be OK to have it in the for-loop condition
++it may be faster than it++. The post-increment operator creates a temporary object, while the other - does not. When the variable is a simple integral type, compiler can optimise it out, but with iterators it is not always the case.
The STL standard defines that when an erase occurs on containers such as std::deque, std::list etc iterators are invalidated.
My question is as follows, assuming the list of integers contained in a std::deque, and a pair of indicies indicating a range of elements in the std::deque, what is the correct way to delete all even elements?
So far I have the following, however the problem here is that the assumed end is invalidated after an erase:
#include <cstddef>
#include <deque>
int main()
{
std::deque<int> deq;
for (int i = 0; i < 100; deq.push_back(i++));
// range, 11th to 51st element
std::pair<std::size_t,std::size_t> r(10,50);
std::deque<int>::iterator it = deq.begin() + r.first;
std::deque<int>::iterator end = deq.begin() + r.second;
while (it != end)
{
if (*it % 2 == 0)
{
it = deq.erase(it);
}
else
++it;
}
return 0;
}
Examining how std::remove_if is implemented, it seems there is a very costly copy/shift down process going on.
Is there a more efficient way of achieving the above without all the copy/shifts
In general is deleting/erasing an element more expensive than swapping it with the next nth value in the sequence (where n is the number of elements deleted/removed so far)
Note: Answers should assume the sequence size is quite large, +1mil elements and that on average 1/3 of elements would be up for erasure.
I'd use the Erase-Remove Idiom. I think the Wikipedia article linked even shows what you're doing -- removing odd elements.
The copying that remove_if does is no more costly than what happens when you delete elements from the middle of the container. It might even be more efficient.
Calling .erase() also results in "a very costly copy/shift down process going on.". When you erase an element from the middle of the container, every other element after that point must be shifted down one spot into the available space. If you erase multiple elements, you incur that cost for every erased element. Some of the non-erased elements will move several spots, but are forced to move one spot at a time instead of all at once. That is very inefficient.
The standard library algorithms std::remove and std::remove_if optimize this work. They use a clever trick to ensure that every moved element is only moved once. This is much, much faster than what you are doing yourself, contrary to your intuition.
The pseudocode is like this:
read_location <- beginning of range.
write_location <- beginning of range.
while read_location != end of range:
if the element at read_location should be kept in the container:
copy the element at the read_location to the write_location.
increment the write_location.
increment the read_location.
As you can see, every element in the original sequence is considered exactly once, and if it needs to be kept, it gets copied exactly once, to the current write_location. It will never be looked at again, because the write_location can never run in front of the read_location.
Remember that deque is a contiguous memory container (like vector, and probably sharing implementation), so removing elements mid-container necessarily means copying subsequent elements over the hole. You just want to make sure you're doing one iteration and copying each not-to-be-deleted object directly to its final position, rather than moving all objects one by one during each delete. remove_if is efficient and appropriate in this regard, your erase loop is not: it does massive amounts of unnecessary copying.
FWIW - alternatives:
add a "deleted" state to your objects and mark them deleted in place, but then every time you operate on the container you'll need to check yourself
use a list, which is implemented using pointers to previous and next elements, such that removing a list element alters the adjacent points to bypass that element: no copying, efficient iteration, just no random access, more small (i.e. inefficient) heap allocations and pointer overheads
What to choose depends on the nature, relative frequency, and performance requirements of specific operations (e.g. it may be that you can afford slow removals if they're done at non-critical times, but need fastest-possible iteration - whatever it is, make sure you understand your needs and the implications of the various data structures).
One alternative that hasn't been mentioned is to create a new deque, copy the elements that you want to keep into it, and swap it with the old deque.
void filter(std::deque<int>& in, std::pair<std::size_t,std::size_t> range) {
std::deque<int> out;
std::deque<int>::const_iterator first = in.begin();
std::deque<int>::const_iterator curr = first + range.first;
std::deque<int>::const_iterator last = first + range.second;
out.reserve(in.size() - (range.second-range.first));
std::copy(first, curr, std::back_inserter(out));
while (curr != last) {
if (*curr & 1) {
out.push_back(*curr);
}
++curr;
}
std::copy(last, in.end(), std::back_inserter(out));
in.swap(out);
}
I'm not sure if you have enough memory to create a copy, but it usually is faster and easier to make a copy instead of trying to inline erase elements from a large collection. If you still see memory thrashing, then figure out how many elements you are going to keep by calling std::count_if and reserve that many. This way you would have a single memory allocation.
How come that random deletion from a std::vector is faster than a std::list? What I'm doing to speed it up is swapping the random element with the last and then deleting the last.
I would have thought that the list would be faster since random deletion is what it was built for.
for(int i = 500; i < 600; i++){
swap(vector1[i], vector1[vector1.size()-1]);
vector1.pop_back();
}
for(int i = 0; i < 100; i++){
list1.pop_front();
}
Results (in seconds):
Vec swap delete: 0.00000909461232367903
List normal delete: 0.00011785102105932310
What you're doing is not random deletion though. You're deleting from the end, which is what vectors are built for (among other things).
And when swapping, you're doing a single random indexing operation, which is also what vectors are good at.
The difference between std::list and std::vector is not merely down to performance. They also have different iterator invalidation semantics. If you erase an item from a std::list, all iterators pointing to other items in the list remain valid. Not so with std::vector, where erasing an item invalidates all iterators pointing after that item. (In some implementations, they may still serve as valid iterators, but according to the standard they are now unusable, and a checking implementation ought to assert if you try to use them.)
So your choice of container is also to do with what semantics you require.
That's not random. Try vector1.erase(vector.begin() + rand() % vector.size()); instead.
The list erase will cause to delete the erased list element, this will invoke a call to the delete operator. The vector erase just causes a swap and then a integer decrement - this is a lot faster.
Actually if you want further speed-ups you should index elements in the vector via iterators. They are known to have better performance for some architectures.