So basically, what I need is to have 28,800 values which can be accessed by an index and can all be set to true or false. Using an array of bools or integers isn't an option, because the size needs to be set during rumtime using a parameter. Using a vector is way too slow and memory intensive. I'm new to C++ and therefore don't have any idea on how to solve this, can anyone help?
EDIT: Thank you to all who commented! Like I said, Im new to C++ programming and your answers really helped me to understand the functionality behind vectors.
So, after everyone said that vector isn't slow I checked again and it turns out that my program was running so slow because of another bug I had while filling the vector. But especially midor's and Some programmer dude's answers helped me to make the program run a bit faster than before, so thanks!
Using a vector is way too slow and memory intensive.
C++ specializes std::vector<bool> so it uses only as much memory as it needs. One bit per "flag" (+ bookkeeping overhead of course).
You could only optimize that implementation if you knew its size a priori (which you don't according to your question), or if you know that the bitmap will only contain very few set bits (e.g. 1bit in a 50'000, but you'd need to measure if a more complex implementation would be worth it). For sparse bitmaps an std::unordered_set<std::uint32_t> that stores the set bits could be an option.
But 28'800 is a very small number, so don't waste your time on optimizations. You won't get any benefits from it.
Related
I want to make a container with few elements and I will be just checking whether an element is part of that set or not.
I know a vector would not be the appropriate container if the set is big enough since each research would be worst-case O(n) and there are better options that use a hash function or binary trees.
However I was wondering what happens if my set has few elements (eg just 5) and I know it in advance, is it worth it to implement the container as a structure having a hash function?
Maybe if the set is not big enough the overhead introduced by having to apply the hash function is greater than having to iterate through 5 elements.
For example in C++ using std::unordered_set instead of std::vector.
As always, thank you
There are many factors affecting the point where the std::vector falls behind other approaches. See std::vector faster than std::unordered_set? and Performance of vector sort/unique/erase vs. copy to unordered_set for some of the reasons why this is the case. As a consequence, any calculation of this point would have to be rather involved and complicated. The most expedient way to find this point is performance testing.
Keep in mind that some of the factors are based on the hardware being used. So not only do you need performance testing on your development machine, but also performance testing on "typical" end-user machines. Your development machine can give you a gut check, though, (as can an online tool like Quick Bench) letting you know that you are not even in the ballpark yet. (The intuition of individual programmers is notoriously unreliable. Some people see 100 as a big number and worry about performance; others don't worry until numbers hit the billions. Either of these people would be blown away by the others' view.)
Given the difficulty in determining the point where std::vector falls behind, I think this is a good place to give a reminder that premature optimization is usually a waste of time. Investigating this performance out of curiosity is fine, but until this is identified as a performance bottleneck, don't hold up work on more important aspects of your project for this. Choose the approach that fits your code best.
That being said, I personally would assume the break point is well, well over 10 items. So for the 5-element collection of the question, I would be inclined to use a vector and not look back.
What's the need to go for defining and implementing data structures (e.g. stack) ourselves if they are already available in C++ STL?
What are the differences between the two implementations?
First, implementing by your own an existing data structure is a useful exercise. You understand better what it does (so you can understand better what the standard containers do). In particular, you understand better why time complexity is so important.
Then, there is a quality of implementation issue. The standard implementation might not be suitable for you.
Let me give an example. Indeed, std::stack is implementing a stack. It is a general-purpose implementation. Have you measured sizeof(std::stack<char>)? Have you benchmarked it, in the case of a million of stacks of 3.2 elements on average with a Poisson distribution?
Perhaps in your case, you happen to know that you have millions of stacks of char-s (never NUL), and that 99% of them have less than 4 elements. With that additional knowledge, you probably should be able to implement something "better" than what the standard C++ stack provides. So std::stack<char> would work, but given that extra knowledge you'll be able to implement it differently. You still (for readability and maintenance) would use the same methods as in std::stack<char> - so your WeirdSmallStackOfChar would have a push method, etc. If (later during the project) you realize or that bigger stack might be useful (e.g. in 1% of cases) you'll reimplement your stack differently (e.g. if your code base grow to a million lines of C++ and you realize that you have quite often bigger stacks, you might "remove" your WeirdSmallStackOfChar class and add typedef std::stack<char> WeirdSmallStackOfChar; ....)
If you happen to know that all your stacks have less than 4 char-s and that \0 is not valid in them, representing such "stack"-s as a char w[4] field is probably the wisest approach. It is fast and easy to code.
So, if performance and memory space matters, you might perhaps code something as weird as
class MyWeirdStackOfChars {
bool small;
union {
std::stack<char>* bigstack;
char smallstack[4];
}
Of course, that is very incomplete. When small is true your implementation uses smallstack. For the 1% case where it is false, your implemention uses bigstack. The rest of MyWeirdStackOfChars is left as an exercise (not that easy) to the reader. Don't forget to follow the rule of five.
Ok, maybe the above example is not convincing. But what about std::map<int,double>? You might have millions of them, and you might know that 99.5% of them are smaller than 5. You obviously could optimize for that case. It is highly probable that representing small maps by an array of pairs of int & double is more efficient both in terms of memory and in terms of CPU time.
Sometimes, you even know that all your maps have less than 16 entries (and std::map<int,double> don't know that) and that the key is never 0. Then you might represent them differently. In that case, I guess that I am able to implement something much more efficient than what std::map<int,double> provides (probably, because of cache effects, an array of 16 entries with an int and a double is the fastest).
That is why any developer should know the classical algorithms (and have read some Introduction to Algorithms), even if in many cases he would use existing containers. Be also aware of the as-if rule.
STL implementation of Data Structures is not perfect for every possible use case.
I like the example of hash tables. I have been using STL implementation for a while, but I use it mainly for Competitive Programming contests.
Imagine that you are Google and you have billions of dollars in resources destined to storing and accessing hash tables. You would probably like to have the best possible implementation for the company use cases, since it will save resources and make search faster in general.
Oh, and I forgot to mention that you also have some of the best engineers on the planet working for you (:
(This video is made by Kulukundis talking about the new hash table made by his team at Google )
https://www.youtube.com/watch?v=ncHmEUmJZf4
Some other reasons that justify implementing your own version of Data Structures:
Test your understanding of a specific structure.
Customize part of the structure to some peculiar use case.
Seek better performance than STL for a specific data structure.
Hating STL errors.
Benchmarking STL against some simple implementation.
This if my first post, I hope I'll meet the standards...
I'm translating into c++ (at which I'm quite new) a program originally written in MATLAB for reasons of efficiency. The piece of code I am actually working on resumes accesses to various indexes of a vector (matrix) in one step. For example, if M1 is a matrix of size, let's say, 10x15, the program would define a new one as follows:
idxs1 = [1 2 3];
idxs2 = [1 2 3 4 5];
M2 = M1 (idxs1 , idxs2);
resulting M2 as a matrix of size 3x5. Now, I guess what MATLAB actually does is access one by one the various places of M1 given by the indexes and then construct M2 by rearranging the many contents acquired, all very efficiently.
My question is, how can I reproduce such mechanism in c++? As far as I know there is no direct way to access in a row various indexes of an array, and the for loop I'm using seems rather cumbersome. Maybe there's some intelligent way to do it without demanding 'too much' processor time? Also, for the sake of educational purposes, I would be grateful if someone could explain what MATLAB actually does when such operation is performed.
Thanks in advance and sorry for the eventual inconveniences!
P.S: Just in case it adds anything to the question, I'm working with MEX files to link both languages.
P.S2: By the way, I found some related questions but regarding other languages:
python: Access multiple elements of an array
perl: How can I use an array slice to access several elements of an array simultaneously?
c: How to access multiple elements of array in one go?
"Armadillo is a high quality C++ linear algebra library, aiming towards a good balance between speed and ease of use
Useful for algorithm development directly in C++, or quick conversion of research code into production environments; the syntax (API) is deliberately similar to Matlab"
Link: http://arma.sourceforge.net/
Math program data structures can be some of the most elaborate out there. I can't even figure out what the 3rd line of your example actually does, so I can't even guess how MATLAB implements anything.
What I can tell you is, that one line of MATLAB is almost certainly hiding a ton of operations. If you want to recreate it you just need to make a utility function with a couple of for-loops that copy over all the correct indices one-by-one. Ultimately, this can't be too much different than one MATLAB does.
If you have a ton of matrix operations you need to support and you're on a large project, you might want to look into finding a C++ matrixes library. I don't have one to recommend, but Boost is a popular C++ library for many purposes including matrixes. (You can also make your own data structure, but not recommended for a novice.)
What MATLAB exactly does is left unspecified, and might very well differ from case to case depending on the indices, and even for a given set of indices it could differ from machine to machine. So let's not speculate.
In particular, it's left unspecified whether MATLAB physically copies M1. Such a copy can be faked, which saves time. The technique is known as "copy on write".
In C++, this would be possible as well, but harder. Furthermore, none of the existing container classes supports it.
If you're going to copy the elements, the CPU won't be the bottleneck. Instead, the memory bus will limit you. This is particularly the case when the indices aren't contiguous. For a 3x5 matrix, the time will be dominated by overhead - contiguity doesn't matter yet.
I've written a small program to calculate prime numbers using the naive division algorithm. In order to improve the performance, I thought it should only check the divisibility based on previously detected primes less than equal to the number's square root.
In order to do that, I need to keep track of the primes. I've implemented it using dynamic arrays. (e.g. Using new and delete). Should I use std::vector instead? Which is better in terms of performance? (Maintenance is not an issue.)
Any help would be appreciated. đ
The ideal answer:
How should any of us know? It depends on your compiler, your OS, your architecture, your standard library implementation, the alignment of the planets...
Benchmark it. Possibly with this. (Haven't used it, but it seems simple enough to use.)
The practical answer:
Use std::vector. Every new and delete you make is a chance for a memory leak, or a double-delete, or otherwise forgetting to do something. std::vector basically does this under the hood anyway. You're more likely to get a sizeable performance boost by maxing out your optimization flags (if you're using gcc, try -Ofast and -march=native).
Also:
Maintenance is not an issue.
Doubt it. Trust me on this one. If nothing else, at least comment your code (but that's another can of worms).
For your purpose, vector might be better, so that you don't need to worry about memory management (e.g. grow your array size and copy previous results), or reserve too much memory to store your results.
Working on some legacy code, I am running into memory issues due mainly (I believe) to the extensive use of STL maps (particularly âmaps-of-mapsâ.)
I am looking at Boost flat_map as a possible solution. Does anyone have any firsthand experience with flat_maps, in particular with regards improvements in speed and/or memory usage? I realize of course this can be very dependent on the types of data stored and the manner in which they are stored but still curious of folkâs actual experience.
Can anyone point me to some solid examples?
As an example: there are several cases in this code of a map-of-a-map; that is, a map where the value is another map.
By replacing the âinnerâ map with a pair of vectors, I reduced the memory footprint 10:1 (3G to 300M). Of course this can slow down searches but for this particular case it doesnât seem to matter much. And it involved about a day of refactoring and careful testing.
Boostâs flat_map sounds like it might be just what I need but I canât seem to find out much about it other than the class description on the Boost web site. Looking for some firsthand feedback.
Boost's flat_map is a binary-tree-based map implementation, except that that binary tree is stored as a (sorted) vector of key-value pairs.
You can basically figure out the answers regarding performance (relative to an std::map by yourself based on that fact:
Iterating the map or a large part of it should be super-fast, relatively
Lookup should typically be relatively fast
Adding or removing values is theoretically much slower, but in practice - assuming your key and value types are small and the number of map elements not very high - probably comparable in speed (or even better on small maps - often no allocation is necessary on insert)
etc.
In your case - maps-of-maps - you're going to lose some of the benefit of "flattening things out", since you'll have an outer map with a pointer to an inner map (if not more levels of indirection); but the flat map would at least help you reduce that. Also, supposing you have two levels of maps, you could arrange it so that you store all of the inner maps contiguously (either by constructing the inner maps appropriately or by instantiating them with your own allocator, a trickier affair); in that case, you could replace pointers to maps with map indices, reducing the amount of space they take up and making life easier for the compiler.
You might also want read Boost's documentation of flat_map; and you could also just use the force and read the source (and the source of the underlying flat_tree) - like I have; I dont actually have flat_map experience myself.
I know this is an old question, but this might be of use to someone finding this question.
I found that flat_map was a big improvement in searching, lookup and iterating large maps. The fact the map is using contiguous data in memory also makes inserting faster than you might expect due to great data locality. If you're doing more inserts than lookups in your map then it might not be for you.
Having said that, repeatedly inserting a random value into a sorted vector is faster than the same on a linked list because of the data locality - despite what Big O might tell you. (tested in VS2017 and G++ 4.8).