My main question is how I can easily swap objects from one vector to another in C++. So adding an object to one vector and removing it from another.
To be more precise: I'm trying to iterate over a grid of cells in the following manner:
Add all cells to an unknownset (and one start cell to the knownset)
Add the neighbors of the known cell to the candidateset (and remove from the unknownset)
Pick the cell in the candidateset with the lowest value and add it to the knownset (and remove it from the candidateset)
Pick this lowest cell and return to step 2
If the unknownset is empty; quit.
Sloppy pseudocode:
vector<Cell> knownset = vector<Cell>();
vector<Cell> unknownset = vector<Cell>();
vector<Cell> candidateset = vector<Cell>();
Cell currentCell = some_cell;
// Iterate until all cells are known
while (unknownset.size() > 0){
for each (direction in directions){
c = currentCell+direction;
// Add cell to candidate set
candidateset.push_back(c);
// Remove cell from unknown set
unknownset.remove(c);
// Search the cell with the lowest value
for each( candidate in candidateset ){
if ( candidate.value < lowestValue ){
lowestValue = candidate.value;
lowestCell = candidate;
}
}
// Remove the cell with the lowest value
knownset.push_back(lowestCell);
candidateset.remove(lowestCell);
currentCell = lowestCell;
}
}
Does anyone have any suggestions how to easily swap cells in this way? (the grid is quite large, so any tips for the performance are also welcome)
The code you have is it. std::vector stores all its content contiguously, so your "swap" of necessity requires copying data around.
You might be eligible for a move optimization, except that you are keeping currentCell. Still, you could move from the location you are about to remove, to a local, and from there to the new location. It is unclear whether your Cell object has a move operation that is cheaper than a copy, so this may not matter to you.
If you want efficient element removal, you should use a datastructure that is optimized for element removal. If you also need to remember insertion order, then std::list might be good. If ordering is not needed, then std::unordered_set.
From your algorithm description, I suspect std::unordered_set is the best choice.
If you want to swap contents of two vectors v1 and v2, you can use
v1.swap(v2)
from algorithm library.
Related
Do you remember my prior question: What is causing data race in std::async here?
Even though I successfully parallelized this program, it still ran too slowly to be practical.
So I tried to improve the data structure representing a Conway's Game of Life pattern.
Brief explanation of the new structure:
class pattern {
// NDos::Lifecell represents a cell by x and y coordinates.
// NDos::Lifecell is equality comparable, and has std::hash specialization.
private:
std::unordered_map<NDos::Lifecell, std::pair<int, bool>> cells_coor;
std::unordered_set<decltype(cells_coor)::const_iterator> cells_neigh[9];
std::unordered_set<decltype(cells_coor)::const_iterator> cells_onoff[2];
public:
void insert(int x, int y) {
// if coordinate (x,y) isn't already ON,
// turns it ON and increases the neighbor's neighbor count by 1.
}
void erase(int x, int y) {
// if coordinate (x,y) isn't already OFF,
// turns it OFF and decreases the neighbor's neighbor count by 1.
}
pattern generate(NDos::Liferule rule) {
// this advances the generation by 1, according to the rule.
// (For example here, B3/S23)
pattern result;
// inserts every ON cell with 3 neighbors to result.
// inserts every OFF cell with 2 or 3 neighbors to result.
return result;
}
// etc...
};
In brief, pattern contains the cells. It contains every ON cells, and every OFF cells that has 1 or more ON neighbor cells. It can also contain spare OFF cells.
cells_coor directly stores the cells, by using their coordinates as keys, and maps them to their number of ON neighbor cells (stored as int) and whether they are ON (stored as bool).
cells_neigh and cells_onoff indirectly stores the cells, by the iterators to them as keys.
The number of ON neighbor of a cell is always 0 or greater and 8 or less, so cells_neigh is a size 9 array.
cells_neigh[0] stores the cells with 0 ON neighbor cells, cells_neigh[1] stores the cells with 1 ON neighbor cell, and so on.
Likewise, a cell is always either OFF or ON, so cells_onoff is a size 2 array.
cells_onoff[false] stores the OFF cells, and cells_onoff[true] stores the ON cells.
Cells must be inserted to or erased from all of cells_coor, cells_neigh and cells_onoff. In other words, if a cell is inserted to or erased from one of them, it must be so also for the others. Because of this, the elements of cells_neigh and cells_onoff is std::unordered_set storing the iterators to the actual cells, enabling fast access to the cells by a neighbor count or OFF/ON state.
If this structure works, the insertion function will have average time complexity of O(1), the erasure also O(1), and the generation O(cells_coor.size()), which are great improval of time complexity from the prior structure.
But as you see, there is a problem: How can I hash a std::unordered_map::const_iterator?
std::hash prohibits a specialization for them, so I have to make a custom one.
Taking their address won't work, as they are usually acquired as rvalues or temporaries.
Dereferencing them also won't work, as there are multiple cells that have 0 ON neighbor cells, or multiple cells that is OFF, etc.
So what can I do? If I can't do anything, cells_neigh and cells_onoff will be std::vector or something, sharply degrading the time complexity.
Short story: this won't work (really well)(*1). Most of the operations that you're likely going to perform on the map cells_coor will invalidate any iterators (but not pointers, as I learned) to its elements.
If you want to keep what I'd call different "views" on some collection, then the underlying container storing the actual data needs to be either not modified or must not invalidate its iterators (a linked list for example).
Perhaps I'm missing something, but why not keep 9 sets of cells for the neighbor counts and 2 sets of cells for on/off? (*2) Put differently: for what do you really need that map? (*3)
(*1): The map only invalidates pointers and iterators when rehashing occurs. You can check for that:
// Before inserting
(map.max_load_factor() * map.bucket_count()) > (map.size() + 1)
(*2): 9 sets can be reduced to 8: if a cell (x, y) is in none of the 8 sets, then it would be in the 9th set. Thus storing that information is unnecessary. Same for on/off: it's enough to store cells that are on. All other are off.
(*3): Accessing the number of neighbours without using the map but only with sets of cells, kind of pseudo code:
unsigned number_of_neighbours(Cell const & cell) {
for (unsigned neighbours = 9; neighbours > 0; --neighbours) {
if (set_of_cells_with_neighbours(neighbours).count() == 1) {
return neighbours;
}
}
return 0;
}
The repeated lookups in the sets could of course destroy actual performance, you'd need to profile that. (Asymptotic runtime is unaffected)
I want to sort an array with huge(millions or even billions) elements, while the values are integers within a small range(1 to 100 or 1 to 1000), in such a case, is std::sort and the parallelized version __gnu_parallel::sort the best choice for me?
actually I want to sort a vecotor of my own class with an integer member representing the processor index.
as there are other member inside the class, so, even if two data have same integer member that is used for comparing, they might not be regarded as same data.
Counting sort would be the right choice if you know that your range is so limited. If the range is [0,m) the most efficient way to do so it have a vector in which the index represent the element and the value the count. For example:
vector<int> to_sort;
vector<int> counts;
for (int i : to_sort) {
if (counts.size() < i) {
counts.resize(i+1, 0);
}
counts[i]++;
}
Note that the count at i is lazily initialized but you can resize once if you know m.
If you are sorting objects by some field and they are all distinct, you can modify the above as:
vector<T> to_sort;
vector<vector<const T*>> count_sorted;
for (const T& t : to_sort) {
const int i = t.sort_field()
if (count_sorted.size() < i) {
count_sorted.resize(i+1, {});
}
count_sorted[i].push_back(&t);
}
Now the main difference is that your space requirements grow substantially because you need to store the vectors of pointers. The space complexity went from O(m) to O(n). Time complexity is the same. Note that the algorithm is stable. The code above assumes that to_sort is in scope during the life cycle of count_sorted. If your Ts implement move semantics you can store the object themselves and move them in. If you need count_sorted to outlive to_sort you will need to do so or make copies.
If you have a range of type [-l, m), the substance does not change much, but your index now represents the value i + l and you need to know l beforehand.
Finally, it should be trivial to simulate an iteration through the sorted array by iterating through the counts array taking into account the value of the count. If you want stl like iterators you might need a custom data structure that encapsulates that behavior.
Note: in the previous version of this answer I mentioned multiset as a way to use a data structure to count sort. This would be efficient in some java implementations (I believe the Guava implementation would be efficient) but not in C++ where the keys in the RB tree are just repeated many times.
You say "in-place", I therefore assume that you don't want to use O(n) extra memory.
First, count the number of objects with each value (as in Gionvanni's and ronaldo's answers). You still need to get the objects into the right locations in-place. I think the following works, but I haven't implemented or tested it:
Create a cumulative sum from your counts, so that you know what index each object needs to go to. For example, if the counts are 1: 3, 2: 5, 3: 7, then the cumulative sums are 1: 0, 2: 3, 3: 8, 4: 15, meaning that the first object with value 1 in the final array will be at index 0, the first object with value 2 will be at index 3, and so on.
The basic idea now is to go through the vector, starting from the beginning. Get the element's processor index, and look up the corresponding cumulative sum. This is where you want it to be. If it's already in that location, move on to the next element of the vector and increment the cumulative sum (so that the next object with that value goes in the next position along). If it's not already in the right location, swap it with the correct location, increment the cumulative sum, and then continue the process for the element you swapped into this position in the vector.
There's a potential problem when you reach the start of a block of elements that have already been moved into place. You can solve that by remembering the original cumulative sums, "noticing" when you reach one, and jump ahead to the current cumulative sum for that value, so that you don't revisit any elements that you've already swapped into place. There might be a cleverer way to deal with this, but I don't know it.
Finally, compare the performance (and correctness!) of your code against std::sort. This has better time complexity than std::sort, but that doesn't mean it's necessarily faster for your actual data.
You definitely want to use counting sort. But not the one you're thinking of. Its main selling point is that its time complexity is O(N+X) where X is the maximum value you allow the sorting of.
Regular old counting sort (as seen on some other answers) can only sort integers, or has to be implemented with a multiset or some other data structure (becoming O(Nlog(N))). But a more general version of counting sort can be used to sort (in place) anything that can provide an integer key, which is perfectly suited to your use case.
The algorithm is somewhat different though, and it's also known as American Flag Sort. Just like regular counting sort, it starts off by calculating the counts.
After that, it builds a prefix sums array of the counts. This is so that we can know how many elements should be placed behind a particular item, thus allowing us to index into the right place in constant time.
since we know the correct final position of the items, we can just swap them into place. And doing just that would work if there weren't any repetitions but, since it's almost certain that there will be repetitions, we have to be more careful.
First: when we put something into its place we have to increment the value in the prefix sum so that the next element with same value doesn't remove the previous element from its place.
Second: either
keep track of how many elements of each value we have already put into place so that we dont keep moving elements of values that have already reached their place, this requires a second copy of the counts array (prior to calculating the prefix sum), as well as a "move count" array.
keep a copy of the prefix sums shifted over by one so that we stop moving elements once the stored position of the latest element
reaches the first position of the next value.
Even though the first approach is somewhat more intuitive, I chose the second method (because it's faster and uses less memory).
template<class It, class KeyOf>
void countsort (It begin, It end, KeyOf key_of) {
constexpr int max_value = 1000;
int final_destination[max_value] = {}; // zero initialized
int destination[max_value] = {}; // zero initialized
// Record counts
for (It it = begin; it != end; ++it)
final_destination[key_of(*it)]++;
// Build prefix sum of counts
for (int i = 1; i < max_value; ++i) {
final_destination[i] += final_destination[i-1];
destination[i] = final_destination[i-1];
}
for (auto it = begin; it != end; ++it) {
auto key = key_of(*it);
// while item is not in the correct position
while ( std::distance(begin, it) != destination[key] &&
// and not all items of this value have reached their final position
final_destination[key] != destination[key] ) {
// swap into the right place
std::iter_swap(it, begin + destination[key]);
// tidy up for next iteration
++destination[key];
key = key_of(*it);
}
}
}
Usage:
vector<Person> records = populateRecords();
countsort(records.begin(), records.end(), [](Person const &){
return Person.id()-1; // map [1, 1000] -> [0, 1000)
});
This can be further generalized to become MSD Radix Sort,
here's a talk by Malte Skarupke about it: https://www.youtube.com/watch?v=zqs87a_7zxw
Here's a neat visualization of the algorithm: https://www.youtube.com/watch?v=k1XkZ5ANO64
The answer given by Giovanni Botta is perfect, and Counting Sort is definitely the way to go. However, I personally prefer not to go resizing the vector progressively, but I'd rather do it this way (assuming your range is [0-1000]):
vector<int> to_sort;
vector<int> counts(1001);
int maxvalue=0;
for (int i : to_sort) {
if(i > maxvalue) maxvalue = i;
counts[i]++;
}
counts.resize(maxvalue+1);
It is essentially the same, but no need to be constantly managing the size of the counts vector. Depending on your memory constraints, you could use one solution or the other.
I'm writing something that needs to start with a list of numbers, already in order but possibly with gaps, and find the first gap, fill in a number in that gap, and return the number it filled in. The numbers are integers on the range [0, inf). I have this, and it works perfectly:
list<int> TestList = {0, 1, 5, 6, 7};
int NewElement;
if(TestList.size() == 0)
{
NewElement = 0;
TestList.push_back(NewElement);
}
else
{
bool Selected = false;
int Previous = 0;
for(auto Current = TestList.begin(); Current != TestList.end(); Current++)
{
if(*Current > Previous + 1)
{
NewElement = Previous + 1;
TestList.insert(Current, NewElement);
Selected = true;
break;
}
Previous = *Current;
}
if(!Selected)
{
NewElement = Previous + 1;
TestList.insert(TestList.end(), NewElement);
}
}
But I'm worried about efficiency, since I'm using an equivalent piece of code to allocate uniform block binding locations in OpenGL behind a class wrapper I wrote (but that isn't exactly relevant to the question :) ). Any suggestions to improve the efficiency? I'm not even sure if std::list is the best choice for it.
Some suggestions:
Try other containers and compare. A linked list may have good theoretical properties, but the real-life benefits of contiguous storage, such as in a sorted vector, can be dramatic.
Since your range is already ordered, you can perform binary search to find the gap: Start in the middle, and if the value there is equal to half the size, there's no gap, so you can restrict searching to the other half. Rinse and repeat. (This assumes that there are no repeated numbers in the range, which I suppose is a reasonable restriction given that you have a notion of "gap".)
This is more of a theoretical, separate suggestion. A binary search on a pure linked list cannot be implemented very efficiently, so a different sort of data structure would be needed to take advantage of this approach.
Some suggestions:
A cursor gap structure (a.k.a. “gap buffer”) based on std::vector is probably the fastest (regardless of system) for this. For the implementation, set the capacity at the outset (known from the largest number) so as to avoid costly dynamic allocations.
Binary search can be employed on a cursor gap structure, with then fast block move to move the insertion point, but do measure if you go this route: often methods that are bad for a very large number of items, such as linear search, turn out to be best for a small number of items.
Retain the position between calls, so you don't have to start from the beginning each time, reducing the total algorithmic complexity for filling in all, from O(n2) to O(n).
Update: based on the OP’s additional commentary information that this is a case of allocating and deallocating numbers, where the order (apparently) doesn't matter, the fastest is probably to use a free list, as suggested in a comment by j_random_hacker. Simply put, at the outset push all available numbers onto a stack, e.g. a std::stack, the free-list. To allocate a number simply pop it off the stack (choosing the number at top of the stack), to deallocate a number simply push it.
Suppose you have 2 vectors say v1 and v2 with the following values:
v1 = {8,4,9,9,1,3};
v2 = {9,4,3,8,1,9};
What is the most STL approach to check if they are "equal"? I am defining "equal" to mean the contents are the same regardless of the order. I would prefer to do this without sorting.
I was leaning towards building two std::map<double, int> to count up each of the vector's elements.
All, I need is a boolean Yes/No from the algorithm.
What say you?
Other conversations on Stack Overflow resort to sorting the vectors, I'd prefer to avoid that. Hence this new thread.
I was leaning towards building two std::map to count up each of the vector's elements.
This will be far slower than just creating sorted vectors. (Note also that std::map is powered by sorting; it just does so using red-black trees or AVL trees) Maps are data structures optimized for an even mix of inserts and lookups; but your use case is a whole bunch of inserts followed by a whole bunch of lookups with no overlap.
I would just sort the vectors (or make copies and sort those, if you are not allowed to destroy the source copies) and then use vector's built in operator ==.
Sorting the vectors and call set_difference is still the best way.
If the copy is heavy for you, the comparison between two unsorted arrays is even worse?
If you want current array untouched, you can make a copy of current arrays?
v1 = {8,4,9,9,1,3};
v2 = {9,4,3,8,1,9};
// can trade before copy/sort heavy work
if (v1.size() != v2.size()){
}
std::vector<int> v3(v1);
std::vector<int> v4(v2);
sort(v3.begin(), v3.end());
sort(v4.begin(), v4.end());
return v3 == v4;
I assume for some reason you can't sort the vectors, most likely because you still need them in their original order or they're expensive to copy. Otherwise, just sort them.
Create a "view" into each vector that allows you to see the vector in any order. You can do this with a vector of pointers that starts out pointing to the elements in order. Then sort the two views, producing a sorted view into each vector. Then compare the two views, comparing the two vectors in their view order. This avoids sorting the vectors themselves.
Was originally thinking of working in terms of sets since that's what you're actually thinking in terms of but that does necessitate sorting. This can be done in O(n) by converting both to hashmaps and checking for equality there.
just take the first vector and compare it with each element in the second vector.
If one value from the first one couldnt be find in the second the vectors are different.
In the worst case it takes O(n*m) time which n = size of first vector and m = size second vector.
This util method will help you to compare 2 int[], let me know in case of any issues
public static boolean compareArray(int[] v1, int[] v2){
boolean returnValue = false;
if(v1.length != v2.length)
returnValue = false;
if(v1.length == 0 || v2.length == 0)
returnValue = false;
List<Integer> intList = Ints.asList(v2);
for(int element : v1){
if(!intList.contains(element)){
returnValue = false;
break;
}else{
returnValue = true;
}
}
I have a data structure in sparse compressed column format.
For my given algorithm, I need to iterate over all the values in a "column" of data and do a bunch of stuff. Currently, it is working nicely using a regular for loop. The boss wants me to re-code this as a for_each loop for future parallelization.
For those not familiar with sparse compressed column, it use 2 (or 3) vectors to represent the data. One vector is just a long list of values. The second vector is the index of where each column starts.
The current version
// for processing data in column 5
vector values;
vector colIndex;
vector rowIndex;
int column = 5;
for(int i = conIndex[5]; i != colIndex[6]; i++){
value = values[i];
row = rowIndex[i];
// do stuff
}
The key is that I need to know the location(as an integer) in my values column in order to lookup the row position (And a bunch of other stuff I'm not bothering to list here.)
If I use the std::for_each() function, I get the value at the position, not the position. I need the position itself.
One thought, and clearly not efficient, would be to create a vector of integers the same length as my data. That way, I could pass an iterator over that dummy vector to the function in for_each and the value passed to my function would be the postion. However, this seems like the least efficient way.
Any thoughts?
My challenge is that I need to know the position in the vector. for_each takes an iterator and sends the value of that iterator to the function.
Use boost::counting_iterator<int>, or implement your own.
#n.m.'s answer is probably the best, but it is possible with only what the standard library provides, though painfully slow I assume:
void your_loop_func(const T& val){
iterator it = values.find(val);
std::ptrdiff_t index = it - values.begin();
value = val;
row = rowIndices[index];
}
And after writing that, I really can only recommend the Boost counting_iterator version. ;)