Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 months ago.
The community reviewed whether to reopen this question 9 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I have a list of unsigned shorts that act as local IDs for a database. I was wondering what is the most memory-efficient way to store allowed IDs. For the lifetime of my project, the allowed ID list will be dynamic, so it may have more true or more false allowed IDs as time goes on, with a range of none allowed or all allowed.
What would be the best method to store these? I've considered the following:
List of allowed IDs
Bool vector/array of true/false for allowed IDs
Byte array that can be iterated through, similar to 2
Let me know which of these would be best or if another, better method, exists.
Thanks
EDIT: If possible, can a vector have a value put at say, index 1234, without all 1233 previous values, or would this suit a map or similar type more?
I'm looking at using an Arduino with 2k total ram and using external storage to assist with managing a large block of data, but I'm exploring what my options are
"Best" is opinion-based, unless you are aiming for memory efficiency at the expense of all other considerations. Is that really what you want?
First of all, I hope we're talking <vector> here, not <list> -- because a std::list< short > would be quite wasteful already.
What is the possible value range of those ID's? Do they use the full range of 0..USHRT_MAX, or is there e.g. a high bit you could use to indicate allowed ones?
If that doesn't work, or you are willing to sacrifice a bit of space (no pun intended) for a somewhat cleaner implementation, go for a vector partitioned into allowed ones first, disallowed second. To check whether a given ID is allowed, find it in the vector and compare its position against the cut-off iterator (which you got from the partitioning). That would be the most memory-efficient standard container solution, and quite close to a memory-optimum solution either way. You would need to re-shuffle and update the cut-off iterator whenever the "allowedness" of an entry changes, though.
One suitable data structure to solve your problem is a trie (string tree) that holds your allowed or disallowed IDs.
Your can refer to the ID binary representation as the string. Trie is a compact way to store the IDs (memory wise) and the runtime access to it is bound by the longest ID length (which in your case is constant 16)
I'm not familiar with a standard library c++ implementation, but if efficiency is crucial you can find an implementation or implementat yourself.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have problem with me where I have to generate unique number throughout the system.Say an application 'X' generates value 'A' by using some inputs and this value 'A' will be used by some other application as input to generate some other value 'B'.
'A' and 'B' both values will be saved later in KDB. Purpose of doing this is to identify which value of 'A' triggered generation value of 'B'. 'A' are getting generated at very high speed so I am looking for algorithm which is fast and doesn't hamper the performance of application 'X'.
What you want is a UUID. See https://en.m.wikipedia.org/wiki/Universally_unique_identifier. They are generally based on things like MAC addresses, timestamps, hashes, and randomness. Their theoretical intent is to be globally unique. Depending on the platform there are often built in functions for generating them. I can expand on this more when I'm not on my phone if necessary, but start there.
You have likely run into them from time to time, https://www.uuidgenerator.net can give you some examples.
That said if you're inserting them in a database another strategy to investigate is using the databases auto assigned primary key ID numbers. Not always possible since you have to store them first to get an ID assigned, but philosophically sounds correct for your application.
You could also roll your own although there are many caveats. E.g. The time stamp of application startup concatenated with some internal counter. Just be aware of collision risks, even unlikely, e.g. two applications starting at the same time, or an incorrect system clock. I wouldn't consider this approach for serious usage given the presence of other more reliable strategies.
No matter what you use, I do recommend ultimately also using it as the primary key in your database. It will simplify things for you overall, and also having two unique ids in a database (e.g. UUID plus auto-generated primary key) denormalizes your database a bit (https://en.m.wikipedia.org/wiki/Database_normalization).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Fact: my code is in c++
Question:
I have a list of unsigned long long, which I use as reppresentation for some objects and the position in the list is the ID for the object.
Each unsigned long long is a sort of lis that mark if an object has the component j, so let say that I have
OBJ=1 | (1<<3)
that means that the object has the components 1 and 3.
I want to be fast when I insert, erease and when I retrieve objects from the list.
The retrieve usually is performed by components so I will look for objects with the same components.
I was using std::vector but as soon I started thinking about performance it seems to me not to be the best choice, since each time I have to delete an object from the list it will relocate all the object ( erease can be really frequent ) plus as soon as the underlying array is full vector will create a new bigger array and will copy all the elements on it.
I was thinking to have an "efficientArray" wich is a simple array but each time an element is removed I will "mark" the position as free an I will store that position inside a list of available positions, and anytime I have to add a new element I will store it in the first available position.
In this way all the elements will be stored in contiguos area, as using vector, but I avoid the "erease" problem.
In this way I'm not avoiding the "resize" problem (maybe I can't) plus the objects with the same components will not be closer (maybe).
Are there other ideas/structures wich I can use in order to have better performance?
Am I wrong when I say that I want "similar" object to be closer?
Thanks!
EDIT
Sorry maybe the title and the question was not write in a good way. I know vector is efficient and I don't want to write a better vector. Since I'm learning I would like to understand if vector IN THIS CASE is good or bad and why, if I'm wrong and if what I was thinking is bad and why, if there are better solutions and data structures (tree? map?), if yes why. I asked even if it is convinient to keep "similar" objects closer and if that MAYBE can influence things like branch prediction or something else (no answer about that) or if it is just nonsence. I just want to learn, even "wrong" answer can be useful for me and for others to learn something, but seems it was a bad idea like I asked *"my compiler works even if I write char ** which is wrong"* and I didn't understand why.
I recommend using either std::set or std::map. You want to know if an item exists in a container and both std::set and std::map have good performance for searches and "lookups".
You could use std::bitset and assign each object an ID that is a power of 2.
In any case, you need to profile. Profile your code without changes. Profile your code using different data structures. Choose the data structure with the best performance.
Some timing for different structures can be read here.
The problem with lists are that your always hunting after the link, where each link potentially is a cache miss (and maybe a TLB miss in addition).
The vector on the other hand will enjoy few cache misses and the hardware prefetcher will work optimally for this data structure.
If the data was much larger the results are not so clearcut.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Since language standards rarely mandate implementation methods, I'd like to know what is the real world hashing method used by C++ standard library implementations (libc++, libstdc++ and dinkumware).
In case it's not clear, I expect the answer to be a method like these :
Hashing with chaining
Hashing by Division / Multiplication
Universal hashing
Perfect hashing (static, dynamic)
Hashing with open addressing (linear/quadratic probing or double hashing)
Robin-Hood hashing
Bloom Filters
Cuckoo hashing
Knowing why a particular method was chosen over the others would be a good thing as well.
libstdc++: Chaining, only power-of-two table size, default (if it is even configurable) load threshold for rehashing is 1.0, buckets are all separate allocations. Outdated. I don't know current state of things.
Rust: Robin Hood, default load threshold for rehashing is 0.9 (too much for open addressing, BTW)
Go: table slots point to "bins" of 5(7?) slots, not sure what happens if bin is full, AFAIR it is growing in a vector/ArrayList manner
Java: chaining, only power-of-two table size, default load threshold is 0.75 (configurable), buckets (called entries) are all separate allocations. In recent versions of Java, above a certain threshold, chains are changed to binary search trees.
C#: chaining, buckets are allocated from a flat array of bucket structures. If this array is full, it is rehashed (with the table, I suppose) in a vector/ArrayList manner.
Python: open addressing, with own unique collision-resolution scheme (not very fortunate, IMHO), only power-of-two table sizes, load threshold for rehashing is 0.666.. (good). However, slot data in a separate array of structures (like in C#), i. e. hash table operations touch at least two different random memory locations (in the table and in the array of slot data)
If some points missed in descriptions, it doesn't mean they are absent, it means I don't know/remember details.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Here is a simplified example. Note that I chose simple types, but in an application I might have many more attribute
struct object
{
float a, b;
int c;
};
vector<object> objects;
or
vector<float> a, b;
vector<int> c;
In a game application where you access each attribute at least once, performance might vary. The game loop will run through the whole index, no matter what.
I know the first example is much more convenient (much easier to organize code with one single struct) and that it might depend on what the application is really doing, but isn't the second example more cache friendly ?
I know it's not the proper way to make an application at first with the Knuth premature optimization thing, but I'd just like to know... I've read this in a gamedev slide once and I'm wonder if it's true or not, and why.
"It depends!"
Really, there is not only one best data distribution for all the problems, otherwise the compilers and the programming languages would be built around it.
It depends on the algorithm where most of the time is spend in computations.
So, array-of-structs is more cache-friendly/page-miss-friendly; struct-of-arrays is better suited for SSE/AVX SIMD optimization, have better memory channel utilization (that's evident with CUDA for example).
My preferred approach is to start using array-of-objects, and using interfaces only (getter/setters). Than trying to optimize making the objects be an interface to the underlying arrays, in case I have the evidence that I am missing some important optimization by profiling.
and remember... "early optimization is evil of all programming" ;)
In my opinion it is better to have only one vector of class objects than three vectors of its data members. In the last case all operations are duplicated three times including memory allocation/deallocation.
Mpreover in the last case there is a logical inconsistence because these three vectors are not connected with objects. You will be unable to find an object that contains these attributes.
One vector of objects will probably perform better due to data locality and being more cache friendly, but it's always better to be sure by actually profiling your use scenario, which especially depends on your access pattern.
If a, b, c are related and will likely be read together then the struct is more cache friendly.
But packing that puts all the same datatypes together sequentially in memory has the advantage of possibly been (auto) vectorizable.
Anyway some related questions to delve deeper into the matter are:
Which is most cache friendly?
Structure of arrays and array of structures - performance difference
Layout in memory of a struct. struct of arrays and array of structs in C/C++
Structure of Arrays vs Array of Structures in cuda
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 months ago.
Improve this question
I am porting some very old c-code into c++ and I've run across a linked list implemented within an array. The element is a simple structure:
struct element
{
void *m_ptrData;
short m_nextEntry;
short m_prevEntry;
};
As an array, there is quick access to the data, if you know the index. The linked list aspect lets the elements be moved around, and "deleted" from the list. Elements can be moved in the list, based on frequency of use (up for MRU and down for LRU).
I like to find a better way to implement this than using another array. I'd like to use STL, but I'm not certain which container is best to use.
Any one have any thoughts?
Since this is a linked list, you should probably use std::list...
The rule of thumb is that you want to use a linked list when you need to insert elements into random positions in the list, or delete random elements from the list. If you mainly need to add/delete elements to/from the end of the list, then you should use std::vector. If you need to add/delete elements to/from either beginning or the end of the list, then you should use std::deque.
Keep in mind, we are talking about probabilities here. If you need to insert an element into the middle of an std::vector once in a blue moon, that will probably be ok. But if you need to do this all the time, it will have a major impact on performance, because the vector will need to constantly move its elements, and probably reallocate its memory too.
On the other hand, the advantage of using a vector is that its elements are contiguous in memory, which greatly improves performance if you simply need to traverse them in order because of caching.
Since the data in this list is pointers, why bother with a linked list at all? For small PODs, std::vector is usually the best first bet, and due to the better locality of its data playing nicely with processor caches it often out-performs a linked list even where, in theory, a linked list should be better. I'd pick std::vector until some profiling would show that there is a performance problem and std::list performs better.
See here:
http://linuxsoftware.co.nz/cppcontainers.html
There's a flow chart to help you choose the right container at the bottom.