We are using c++ to develop an application that runs in Windows CE 4 on an embedded system.
One of our constraint is that all the memory used by the application shall be allocated during startup only. We wrote a lot of containers and algorithms that are using only preallocated memory instead of allocating new one.
Do you think it is possible for us to use the boost libraries instead of our own containers in these conditions?
Any comments and/or advice are welcomed!
Thanks a lot,
Nic
We use boost for embedded systems. With boost you can pick and choose what you use. We use smart_ptr and boost::bind in all of our projects. We write software for cheap cell phones.
And if Windows CE can run on your hardware I would expect that parts of boost would be applicable.
There are parts of boost that have no allocation and you might find them useful.
I would pick and choose based on your requirements.
Like anything that you use, you need to know the costs.
You could write your own allocator for the container, which allocates from a fixed size static buffer. Depending on the usage patterns of the container the allocator could be as simple as incrementing a pointer (e.g. when you only insert stuff into the container once at app startup, and don't continuously add/remove elements.)
Replacing your containers with Boost containers is NOT a good idea. The work to make appropriate custom allocators wouldn't be that bad, but you'd be violating the spirit of your 'allocate at startup' rule. The idea behind this rule (in my experience) is generally to make sure that you don't have to deal with out of memory type situations at run-time. The idea is to make sure that you have all the memory you could possibly need RIGHT AT THE START, so that there's no possibility of any part of the system coming up short of memory later on.
If you used the Boost containers with a custom allocator, you'd suddenly have to deal with the possibility that the pool the container is allocating from could go empty, thus eliminating the purpose of the 'allocate at startup' rule.
In the situation of a limited memory device, I would avoid any kind of container more complex than a statically allocated array.
Boost is a set of libraries. Some of them are focussed on template metaprogramming. Those don't even use any memory at runtime. But your question seems to be about replacing your containers. I'd doubt that is possible except using custom allocators. But even then, it's most likely you would be using plain STL containers and not boost. Boost only provides the TR1 containers, for those compilers that do not yet include TR1.
Do not use Boost.
It is a big library and your basic memory allocation requirements are very different from those of the libraries designers.
Even if you can get a current version of Boost to work according to your requirements with custom allocators it may break with a new version of Boost.
Feel free to look at the Boost source code though for some useful ideas but use your own implementation for what you need.
I'm looking into this right now — I would like to use circular buffers, lock-free containers, and asynchronous I/O, and instead of allocating dynamic memory, I'd prefer to use memory pools.
The biggest problem I've seen so far is that shared_ptr is used in a lot of places, with no easy way to replace it with intrusive_ptr. Since shared_ptr allocates dynamic memory to keep track of the reference count, I can't use it in an embedded system.
Fixing this looks doable, but a lot of work — I have to expand the template specification of any class that contains a shared_ptr so that the specific type of shared-pointer can be changed to intrusive_ptr if desired. So now I have to consider how much work that'll be, versus how much work it'll be to write my own version of the Boost features I need. Not a pleasant place to be.
I hope someone points out why I'm wrong about this.
Related
For a software I have to avoid any use of memory in the heap, and only rely on stack-allocated memory. Then, this prevents me from using any C++ standard containers, such as vector, map, string (well, basic_string), which I would really like to use, to ease development and data manipulation.
I found (many) implementations of stack allocators, such as this one which itself references two others, or this one from chromium.
Many of them are not fully compliant with the standard, or rely on C++11 (and I am stuck with C++03 at the moment, sadly). Do you have any feedback about a good already existing stack allocator for C++03 or should I adapt one of the above?
Thanks!
Howard Hinnant's short_alloc.h (see also here) is a pretty good start (you will need to add C++03 boilerplate, see here).
Of course, this will still go to the heap if it runs out of memory, the alternative is to throw std::bad_alloc.
For a piece of multiplatform c++ code I am writing, I need a shared pointer. Currently the project does not use boost, and pulling it in would be extremely difficult or impossible from an administrative view. However, I can use select C++11 features, including shared pointers.
There is a problem with the standard shared pointers, they guarantee thread safety. That means that on some platforms/compilers, like GCC ( http://tinyurl.com/GCCSharedPtrLockPolicy ) atomics and mutexes will be needlessly used, but at least I can check and work around issues incurred by this. Then for other platforms ( http://tinyurl.com/msvscSharedPtr ) there does not even appear to be a way to check, what thread safety mechanisms are used. The original boost pointer provides only the most basic of thread safety guarantees ( http://tinyurl.com/SharedPtrThreadSafety ).
My core issue here is that on some platforms Atomics can cause costly synchronizations between CPU caches and unneeded Mutexes can cause calls to the OS that that may delay for not entirely related reasons. This code will be multi-threaded, but we have other synchronization methods for moving data between threads. A thread-safe shared pointer is simply not needed or wanted.
Normally, I would prefer to benchmark and make my decision, but because of the platforms this will run on, and be ported too, I cannot practically do so. I would need to test on some of the less popular platforms, where less optimized compilers tend to exist, but I do not have that ability currently.
I will try to make a push to get Boost pointers, but that is unlikely, what are my other options for when that fails? In the mean time I will research trying to get just the Shared_ptr out of boost, but I do not think that will be easy.
I could roll my own. This seems like a terrible idea, why would I have to re-invent something this basic.
If there is a library with that is simple and has liberal enough licensing, then I could simply copy their shared_ptr code and simplify rolling my own.
Edit: Pulling in anything from boost other than header only libraries has been struck out. I will be researching Loki as one of the answerers suggested. If that fails and no answers materialize here, I will roll my own :( .
I'd take a look at the one in Loki. Loki is considerably smaller than boost, and the smart pointer implementation in Loki is highly configurable.
boost shared_ptr supports single threading usage
you can #define the macro BOOST_SP_DISABLE_THREADS on a project-wide basis to switch to ordinary non-atomic reference count updates
citation from boost shared_ptr
gcc has a class __shared_ptr which takes a lock-policy. shared_ptr derives off this.
One of the policies is _S_single, which is for single threaded code (ie: non locked/non atomic reference count)
In c++11 you can use template aliases which will allow you to use the non-standard __shared_ptr class which shared_ptr derives off
template<typename T>
using st_ptr = __shared_ptr<T, __gnu_cxx::_S_single>;
If you don't have a conformant compiler yet you can roll your own by just inheriting off __shared_ptr and exposing the interface (in fact this is how gcc currently does it because of the fact that template aliases were not available prior to 4.7)
Look in bits/shared_ptr.h to see how shared_ptr derives off __shared_ptr - it will be trivial to roll your own.
It is quite possible to write/adapt your own smart pointer class for use in legacy projects that the newer libraries do not support. I have written my own for just such a reason and it works well with MSVC++ 4.2 and onwards.
See ootips.org/yonat/4dev/smart-pointers.html which is what I based mine on. Definitely a possibility if you want a small solution. Just the header and .cpp file required.
The key point to watch out for is the lack of the explicit keyword in older compilers. Another is that you may want to allow implicit conversion to the raw pointer to allow your APIs to remain less affected (we did this) and in that case you should also take care to prevent conversion to other types of pointers as well.
I am using SFML, which handles most of it's memory through std::vector<Uint32> and other types similar to that. All declarations use the default Allocator. The SFML library is compiled into .lib files that are statically linked against in Visual Studio. (I possess all the source code to it however if needed.)
My question is, what would be the least painful method of forcing SFML to use my custom small-objects/thread-safe allocator for STL containers, instead of the default? The less I need to alter the library itself, the better of course!
Edit:
If it helps at all, forget the implementation of the library; I can recompile that on the whim. For the sake of the question, imagine it's my code I am using. My goal is to change the default Allocator that all STL containers use, essentially.
Edit2:
If that is not at all possible, would overriding new itself be a proper method? I have read around that the default allocator is nothing more than sugar coating on the new/delete operations.
Well, the most 'bruteforce' or foolproof method would seem to create a preload library that implements malloc/free (and friends?)
This is usually the way in which heap debuggers/bounds checkers operate.
However, I know nothing about the SFML allocator requirements, so it might not work if the space is very limited. In that case, I suggest 'marshaling' (fancy word for copying) the data over to the custom-allocated regions when needed
Tangentially related:
Eletronics Art has a 'port' of STL for game development. It is heavilty geared to custom allocators (in fact, it comes without a default one!). You can have a look at
EASTL http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html
at github
To change default new operator, just implement this functions (at global scope):
void* operator new (std::size_t size) throw (std::bad_alloc);
You may find more information there: http://www.cplusplus.com/reference/new/operator%20new/
I am not completely sure it would work with dynamic libraries.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
To STL or !STL, that is the question
Are there cases where one should avoid the use of the C++ STL in his / her project?
When you choose to use a framework like Qt you might consider using lists, vectors, etc from Qt rather that the STL. Not using STL in this case saves you from having to convert from STL to the Qt equivalents when you need to use them in your GUI.
This is debatable and not everyone wants to use everything from Qt
from http://doc.qt.nokia.com/latest/containers.html
These container classes are designed to be lighter, safer, and easier to use than the STL containers. If you are unfamiliar with the STL, or prefer to do things the "Qt way", you can use these classes instead of the STL classes.
If you cannot use RTTI and/or exceptions, you might experience that parts of STL won't work. This is the case e.g. for native Android apps. So if it doesn't give you what you need, it's a reason not to use it!
Not really. There's no excuse to ban the use of an entire library- unless that lib only serves one function, which is not the case with the Standard library. The provided facilities should be evaluated on a per-function basis- for example, you may well argue that you need a container that performs a more specific purpose than vector, but that is no excuse to ban the use of deque, iostream or for_each too.
More importantly, code generated via template will not be more bloated than the equivalent code written by hand. You won't save code bloat by refusing to use std::vector and then writing your equivalent vector for float and double. Especially in 2011, the size of an executable is pretty meaningless compared to the sizes of other things like media in the vast, vast majority of situations.
If you care a lot about executable size, then you might want to avoid using STL in your program.
For example, uTorrent doesn't use STL and that is one reason why it's so small.
Since STL does rely on templates a lot (it is Standard TEMPLATE Library, after all), many times you use templates, the compiler has to generate extra code for every type you use when dealing with STL.
This is compile time polymorphism and will increase your executable size the more you use it.
If you exclude STL from your project (and use templates sparingly or not at all), your code size will get smaller. Note that it won't necessarily be faster.
Also note that I'm not talking about a program's memory usage during execution, since that will depend on how many objects your allocating during your app's lifetime.
I'm talking about your binary's executable.
If you want an example, note that a simple Hello world program, when compiled, might be bigger than a cleverly code demo which can include an entire 3D engine (run time generated) in a very small executable.
Some info regarding uTorrent's size:
Official FAQ (from 2008), this question doesn't appear in recent FAQ.
How was uTorrent programmed to be so efficient?
Second post regarding this.
Third post regarding this.
Note that, even though uTorrent is >300kb and is compressed with UPX, it is still really small when you take into account what it's capable of doing.
I would say that there may be occasions where you do not use a particular feature of STL in your project for a given circumstance because you can custom write it better for your needs. STL collections are generic by nature.
You might want in your code:
Lock-free containers that are thread-safe. (STL ones are not).
A string class that is immutable by nature and copies the actual data "by reference" (with some mechanism).
An efficient string-building class that is not ostringstream (not part of STL anyway but you may mean all the standard library)
algorithms that use Map and Reduce (not to be confused with std::map. Map and Reduce is a way to iterate over a collection using multiple threads or processes, possibly even distributed on different machines).
Hey, look, so much of boost was written because what the Standard Library provided at the time did not really address the needs of the programmer and thus provided alternatives.
I am not sure if this is what you meant or if you particular meant STL should be "banned" at all times (eg device driver programming where templates are considered bloaty even though that is not always the case).
If you are working to particular standards that forbid it.
For example, the MISRA C/C++ guidelines are aimed at automotive and embedded systems and forbid using dynamic memory allocation, so you might choose to avoid STL containers altogether.
Note: The MISRA guideline is just an example of a standard that might influence your choice to use STL. That particular guideline doesn't rule out using all of the STL. But (I believe) it rules out using STL containers as they rely on runtime allocation of memory.
It can increase executable size. if you're running on an embedded platform you may wish to exclude the STL.
When you use something like the Qt library that implements identical functionality you may not need the STL. May depend on other needs, like performance.
The only reason is if you are working on embedded systems with low memory, or if your project coding guidelines explicitly forbid STL.
I can't other reasonable reason to manually roll your own incompatible, bug ridden implementation of some of the features on STL.
TR18015 deals with some of the limitation of the STL. It looks at it from a different angle - what compilers could do better - but still is an interesting (if in-depth) read.
I'd be careful in general with microprocessors and small embedded systems. First, compiler optimizations are not up to what you know from desktops, and you run into hardware limits much sooner.
Having said that, it depends a lot on the libraries you use. I/O streams are notoriously slow (and require a careful implementation to not be), whereas std::vector is merely a thin wrapper.
Previously, I used to use MFC collection classes such CArray and CMap. After a while I switched to STL containers and have been using them for a while. Although I find STL much better, I am unable to pin point the exact reasons for it. Some of the reasoning such as :
It requires MFC: does not hold because other parts of my program uses MFC
It is platform dependent: does not hold because I run my application only on windows.(No need for portability)
It is defined in the C++ standard: OK, but MFC containers still work
The only reason I could come up is that I can use algorithms on the containers. Is there any other reason that I am missing here - what makes STL containers better than MFC containers?
Ronald Laeremans, VC++ Product Unit Manager, even said to use STL in June 2006:
And frankly the team will give you the same answer. The MFC collection classes are only there for backwards compatibility. C++ has a standard for collection classes and that is the Standards C++ Library. There is no technical drawback for using any of the standard library in an MFC application.
We do not plan on making significant changes in this area.
Ronald LaeremansActing Product Unit ManagerVisual C++ Team
However, at one point where I was working on some code that ran during the installation phase of Windows, I was not permitted to use STL containers, but was told to use ATL containers instead (actually CString in particular, which I guess isn't really a container). The explanation was that the STL containers had dependecies on runtime bits that might not actually be available at the time the code had to execute, while those problems didn't exist for the ATL collections. This is a rather special scenario that shouldn't affect 99% of the code out there.
STL containers:
Have performance guarantees
Can be used in STL algorithms which also have performance guarantees
Can be leveraged by third-party C++ libraries like Boost
Are standard, and likely to outlive proprietary solutions
Encourage generic programming of algorithms and data structures. If you write new algorithms and data structures that conform to STL you can leverage what STL already provides at no cost.
Compatibility with other libraries (such as boost) in syntax, interoperability, and paradigm. It's a non-trivial benefit.
Using STL will develop a skillset that is more likely to be useful in other contexts. MFC isn't so widely used any more; STL is.
Using STL will develop a mindset that you may (or may not) find useful in code you write yourself.
Using something other than STL isn't inherently wrong though.
STL has more collection types than MFC
Visual Studio (2008+) debugger visualizes STL much better than MFC. (AUTOEXP.DAT magic can fix that - but it is a pain! Nothing like debugging your debugger when you screw it up...)
One good thing with MFC is that there is still a large corpus of MFC code out there. Other answers talk about third-party compatibility. Don't forget third party MFC-based stuff.
I always prefer using more standard/compatible libraries where I can since I may have future projects that I can reuse a portion of the code on. I have no idea what libraries future projects will use, but I have a better chance of making my code reusable if I use standard/compatible stuff.
Also, the more I use a library, I get more comfortable and quicker with it. If I am going to invest the time to learn a library, I want to make sure it's going to stay around and is not tied in with a specific platform or framework.
Of course, I say all of this assuming that my choices are rather similar when it comes to performance, features and ease of use. For instance, if the MFC classes are a significant enough improvement in these areas, I would use them instead.
In fact you can use STL algorithms on some of MFC containers as well. However, STL containers are preferred for another very practical reason: many third-party libraries (Boost, arabica, Crypto++, utf-cpp...) are designed to work with STL, but know nothing about MFC containers.
MFC containers derive from CObject and CObject has the assignment operator made private. I have found this very annoying in practice.
std::vector, unlinke CArray, guarantees that the memory block is contiguous, thus you can interop with C programming interfaces easily:
std::vector<char> buffer;
char* c_buffer = &*buffer.begin();
It is now assumed that C++ developers are at least passingly familiar with the STL. Not so for MFC containers. So if you're adding a new developer to your team, you will have an easier time finding one who know STL than the MFC containers, and thus will be able to contribute immediately.
I think it boils down to a simple question: Who do you trust more? If you trust Microsoft, then continue to use the MFC variants. If you trust the industry, then use STL.
I vote for STL because the code that runs on Windows today might need to be ported to another platform tomorrow. :)
This is a case of whichever tools work for the job you want to do, and 'better' being a subjective term.
If you need to use your containers with other standards-compliant code, or if it is ever going to be code that is shared across platforms, STL containers are probably a better bet.
If you're certain that your code will remain in MFC-land, and MFC containers work for you, why not continue to use them?
STL containers aren't inherently better than MFC containers, however as they are part of the standard they are more useful in a wider range of contexts.
Next to the mentioned aspects: well-supported, standard available, optimized for performance, designed for use with the algorithms, I might add one more aspect: type-safety, and loads of compile-time checks. You can't even imagine drawing a double out of a std::set<int>.
Because a library that uses iterators to combine sequences of any kind with algorithms so that A) all sensible permutations are possible and B) it's universally extensible, is (when you think about your concept of a container library as it was 15 years ago) such a mind-blowingly marvelous idea, that it has pretty much blown out of the water everything else within less than a decade?
Seriously, the STL has its flaws (why not ranges instead of iterators? and that erase-remove idiom is not exactly pretty), but it's based on a brilliant idea and there's very good reasons we don't need to learn a new container/algorithms library every time we start at a new job.
I wouldn't totally dismiss the portability argument. Just because you use MFC today doesn't mean that you always will. And even if you mostly write for MFC, some of your code could be re-used in other applications if it were more generic and standards-based.
I think the other reason to consider STL is that its design and evolution has benefited from libraries that have come before, include MFC and ATL. So many of the solutions are cleaner, more re-usable, and less error prone. (Though I wish they had a better naming convention.)