I'm trying to make a 3 dimensional array of booleans that tells me if I previously visited a location in 3d space for a simple navigation algorithm. The array could be quite large (something along the lines of 1,000,000 x 1,000,000 x 1,000,000 or maybe larger), so I'm wondering if it would be faster to declare an array of that size and set each boolean value to false, or to make a map with a key of coordinate (x, y, z) and a value of type bool.
From what I figure, the array would take O(1) to find or modify a coordinate, and the map would take O(log n) to find or insert a value. Obviously, for accessing values, the array is faster. However, does this offset the time it takes to declare such an array?
Thanks
Even at 1 bit per bool, your array will take over 2**39 bytes. I'd suggest a set if there aren't too many elements that will be true.
You can use a class to hide the implementation details, and use a 1D set.
Have you tried calculating how much memory would be needed for an array like this? A lot!
Use std::map if ordering of the points is important, or std::unordeded_map if not. Also the unordered map gives you a constant time insertion and lookup.
I guess that some kind of search tree is probably what you're looking for (k-d tree for example).
You're going to make an array that is one exabyte, assuming that you use 8 bits per point? Wow, you have a lot of RAM!
I think you should re-think your approach.
Related
pros, I need some performance-opinions with the following:
1st Question:
I want to store objects in a 3D-Grid-Structure, overall it will be ~33% filled, i.e. 2 out of 3 gridpoints will be empty.
Short image to illustrate:
Maybe Option A)
vector<vector<vector<deque<Obj>> grid;// (SizeX, SizeY, SizeZ);
grid[x][y][z].push_back(someObj);
This way I'd have a lot of empty deques, but accessing one of them would be fast, wouldn't it?
The Other Option B) would be
std::unordered_map<Pos3D, deque<Obj>, Pos3DHash, Pos3DEqual> Pos3DMap;
where I add&delete deques when data is added/deleted. Probably less memory used, but maybe less fast? What do you think?
2nd Question (follow up)
What if I had multiple containers at each position? Say 3 buckets for 3 different entities, say object types ObjA, ObjB, ObjC per grid point, then my data essentially becomes 4D?
Another illustration:
Using Option 1B I could just extend Pos3D to include the bucket number to account for even more sparse data.
Possible queries I want to optimize for:
Give me all Objects out of ObjA-buckets from the entire structure
Give me all Objects out of ObjB-buckets for a set of
grid-positions
Which is the nearest non-empty ObjC-bucket to
position x,y,z?
PS:
I had also thought about a tree based data-structure before, reading about nearest neighbour approaches. Since my data is so regular I had thought I'd save all the tree-building dividing of the cells into smaller pieces and just make a static 3D-grid of the final leafs. Thats how I came to ask about the best way to store this grid here.
Question associated with this, if I have a map<int, Obj> is there a fast way to ask for "all objects with keys between 780 and 790"? Or is the fastest way the building of the above mentioned tree?
EDIT
I ended up going with a 3D boost::multi_array that has fortran-ordering. It's a little bit like the chunks games like minecraft use. Which is a little like using a kd-tree with fixed leaf-size and fixed amount of leaves? Works pretty fast now so I'm happy with this approach.
Answer to 1st question
As #Joachim pointed out, this depends on whether you prefer fast access or small data. Roughly, this corresponds to your options A and B.
A) If you want fast access, go with a multidimensional std::vector or an array if you will. std::vector brings easier maintenance at a minimal overhead, so I'd prefer that. In terms of space it consumes O(N^3) space, where N is the number of grid points along one dimension. In order to get the best performance when iterating over the data, remember to resolve the indices in the reverse order as you defined it: innermost first, outermost last.
B) If you instead wish to keep things as small as possible, use a hash map, and use one which is optimized for space. That would result in space O(N), with N being the number of elements. Here is a benchmark comparing several hash maps. I made good experiences with google::sparse_hash_map, which has the smallest constant overhead I have seen so far. Plus, it is easy to add it to your build system.
If you need a mixture of speed and small data or don't know the size of each dimension in advance, use a hash map as well.
Answer to 2nd question
I'd say you data is 4D if you have a variable number of elements a long the 4th dimension, or a fixed large number of elements. With option 1B) you'd indeed add the bucket index, for 1A) you'd add another vector.
Which is the nearest non-empty ObjC-bucket to position x,y,z?
This operation is commonly called nearest neighbor search. You want a KDTree for that. There is libkdtree++, if you prefer small libraries. Otherwise, FLANN might be an option. It is a part of the Point Cloud Library which accomplishes a lot of tasks on multidimensional data and could be worth a look as well.
I noticed today that I could give a C++ Vector or Array a Float value as index.
(e.g. tab[0.5f])
This Float value will be converted into an Int value and then gives me the same result as tab[0].
This behavior is not interesting to me, as I'm searching for a method to access in the fastest way possible to an Object, depending on a Float key.
Is it possible to keep the access speed of an array/vector, with a Float index ?
I understand that my keys will have an inaccuracy problem, but I expect my Float values to keep a maximum of 3 digits of precision.
Would a Map<Float, Object> do the job ? I've read on the C++ reference documentation that the Map access was "logarithmic in size", which is way less appealing to me.
Thank you :).
Edit :
I need to transform a mesh M containing X numbers of shared vertices into a mesh M' containing X' number of NON shared vertices.
Indexes of vertices are set in M, and I know it's in TRIANGLE mode.
My current algorithm is :
for i in M.indexes, i+3
take 3indexes, and deducing the vertices they are pointing to (get 3vertices of a triangle)
calculate normal on these vertices
check, for each couple {Vertex_i, Normal} (i between 1 and 3, my 3vertices) if I already have this couple stored, and act accordingly
... Next steps
To check the couple {Vertex,Normal}, i use an Array[x][y][z] based on position of the vertice, which IS a Float, though i know it won't be more than 3digits precision.
Use an unordered_map. The find method has a complexity in average case: constant and in worst case: linear in container size.
Note : Since you were willing to use an array, I'm assuming you're not interested in having an ordered container
That been said, in any case, the performance depends on the input (mesh size) and its characteristics, and the only way to choose an optimal solution would be to implement any reasonable ones and benchmark against each other. In many cases theoretical complexity is irrelevant due to implementation specifics/intrinsics. I mean even if one told that a std::vector<std::pair<float, mapped_value>> would perform better in your case, I'd have to actually do some tests to prove him right/wrong
I have an assignment in which I must read a list of 4000 names from a text file and sort then into a C style array as they're being read in (rather than reading them all then sorting). Since this is going involve a lot elements changing indices would it be possible to use bitshifting to rearrange large quantities of elements simultaneously?For example,
declare a heap based array of 20 size
place variable x index 10
perform a bitshift on index 9 with the size of the array data type so that x is now in index 11
Also, if you have any tips on the task in general I'd appreciate it.
No, that doesn't sound at all like something you'd use bitshifting for.
You will have distinct elements (the names) stored in an array, and you need to change the order of entire elements. This is not what bitshifting is used for; it is used to move the bits in a single integer to the left or to the right.
You should just learn qsort().
Not sure about the "sort as they're being read in" requirement, but the easiest solution would be to just call qsort() as each name is added. If that's not allowed or deemed too expensive, think about how to do a "sorted insert" against an array.
By the way, a typical approach in C would be to work with an array of pointers to strings, rather than an array of actual strings. This is good, since sorting an array of pointers is much easier.
So you would have:
char *names[4000];
instead of
char names[4000][64 /* or whatever */];
This would require you to dynamically allocate space for each name as it's loaded though, which isn't to hard. Especially not if you have strdup(). :)
If using qsort() is not allowed(would be pretty stupid to do so after every insert), you could write your own insertion sort. It's not exactly a very efficient way of sorting large arrays but I suppose it's what your teacher is expecting for.
I want to allocate memory of 10^9*10^9 in a double dimension array but this is not possible.is their any way out?
I think vector could be solution to this but i dont know how to do it.
You cannot allocate 1018 bytes of memory in any computer today (that's roughly a million terabytes). However, if your data is mostly zeros (ie. is a sparse matrix), then you can use a different kind of data structure to store your data. It all depends on what kind of data you are storing and whether it has any redundant characteristics.
Assuming that the number of non-zero elements is much less than 10^18, you'll want to read up on sparse arrays. In fact, it's not even a requirement that most of the elements in a sparse array be zero -- they just need to be the same. The essential idea is to keep the non-default values in a structure like a list; any values not found in the list are assumed to be the default value.
I want to allocate memory of 10^9*10^9 in a double dimension array but this is not possible.is their any way out?
That's way beyond current hardware capabilities, and array this big is unsuitable for any practical purpose (you're free to calculate how many thousands of years it would take to walk through every element).
You need to create "sparse" array. Store only non-zero elements in memory, provide array-like interface to access them, but internally store them in something like std::map<std::pair<xcoord, ycoord>, value>, return zero for all elements not in map. As long as you don't do something reckless like trying to set every element to non-zero value, this should be sufficient array replacement.
so....
What do you need that much memory for?
I am just starting to learn hashtables, and so far, I know that you take the object you want to hash and put it through an hash function, then use the index it returns to get the corresponding object you want. There is something I don't understand though:
What structure do you use to store the objects in so you can quickly index them with the code returned by the hash function? The only thing I can think of is to use an array, but to handle all the keys, you'd have to allocate one that's 9999999999999 elements big or something ridiculous like that. Or is it as simple as iterating over a linked list or something and comparing the ID in each of the elements with the key from that hash function? And if so, that seems kind of inefficient doesn't it?
Normally, you use an array (or something similar like a vector). You pick a reasonable size (e.g., 20% larger than the number of items you expect) and some method of resolving collisions when/if two keys produce the same hash value (e.g., each of those locations is the head of a linked list of items that hashed to that value).
Yes, you usually use an array but then you do a couple of things:
You convert the hash code to an array index by using the remainder of the hash code divided by the array size.
You make the size of the array a prime number as that makes step #1 more efficient (some hash algorithms need this to get a uniform distribution)
You come up with a design to handle hash collisions. #JerryCoffin's answer gives you more detail.
Generally it's array. If the array size is N then use hash function that returns numbers in range 0..(N-1). For example apply modulo N on the hash function result.
And then use collision resolution in Wikipedia.