I am trying to write a container class which uses STL allocators. What I currently do is to have a private member
std::allocator<T> alloc_;
(this will later be templated so that the user can pick a different allocator) and then call
T* ptr = alloc_.allocate(1,0);
to get a pointer to a newly allocated 'T' object (and used alloc_.construct to call the constructor; see the answer below). This works with the GNU C++ library.
However, with STLPort on Solaris, this fails to do the right thing and leads to all sorts of bizarre memory corruption errors. If I instead do
std::allocator_interface<std::allocator<T> > alloc_;
then it is all working as it should.
What is the correct way to use the stl::allocator? The STLPort/Solaris version fails to compile with g++, but is g++ right?
You need to both allocate and construct with the allocator. Something like this:
T* ptr = alloc_.allocate(1,0);
alloc_.construct(ptr, value);
Lots of things are downright broken if you don't start with a properly constructed object. Imagine a std::string being allocated but not constructed. When you try to assign to it, it will first try to cleanup its old contents by freeing some data, which will of course be garbage values from the heap and crash.
Something you might want to do is have your own custom allocator that you can use to see how the standard containers interact wit allocators. Stephan T. Lavavej posted a nice, simple one called the mallocator. Drop it into a test program that uses various STL containers and you can easily see how the allocator is used by the standard containers:
http://blogs.msdn.com/vcblog/archive/2008/08/28/the-mallocator.aspx
Not all of the interface functions in the mallocator (such as construct() and destroy()) are instrumented with trace output, so you might want to drop trace statements in there to more easily see how the standard containers might use those functions without resorting to a debugger.
That should give you a good idea of how your containers might be expected to use a custom allocator.
Related
From what I read in http://en.cppreference.com/w/cpp/memory/allocator , most features of the allocators are now going to be deprecated. The question is, how is one supposed to use allocators in new code? What is the "right" way now?
From what I deduce in the documentation, construct is part of the allocator traits, rather than the allocator itself.
I am building a custom container, here it is a very simple version of the constructor, is this a good usage of the new design?
container::container(std::size_t size, T const& value, Allocator const& allocator) : allocator_(allocator){
data_ = std::allocator_traits<Alloc>::allocate(allocator_, size);
for(auto ptr = data_; ptr != data_ + size; ++ptr){
std::allocator_traits<Allocator>::construct(allocator_, ptr, value)
}
}
I tried to use an algorithm (like std::for_each) in the loop but I didn't manage to use one without taking addresses (operator&).
Where can I find a complete example of a modern allocator?
After some tweaking, I found a way to use an algorithm instead of the the raw loop (to which an execution policy can be passed). I am not very sure, but it could be this:
data_ = std::allocator_traits<Allocator>::allocate(allocator_, size);
std::for_each([policy? deduced from allocator?,]
boost::make_counting_iterator(data_),
boost::make_counting_iterator(data_ + size),
[&](auto ptr){std::allocator_traits<Allocator>::construct(allocator_, ptr, value);}
);
Yes, the current approach is through std::allocator_traits. You'll be able to support the "minimal allocator interface" that way.
http://en.cppreference.com/w/cpp/concept/Allocator
Some requirements are optional: the template std::allocator_traits supplies the default implementations for all optional requirements, and all standard library containers and other allocator-aware classes access the allocator through std::allocator_traits, not directly.
If you observe the std::allocator_traits member functions and typedefs, you'll see they're detecting the presence of appropriate function/types and dispatching through them if they can.
The deprecation and potential future removal will change nothing if you're already using std::allocator_traits since it only applies to std::allocator and their member functions/typedefs.
Now, if you ask me, there's nothing wrong with for-loops, and using std::for_each gains you nothing. There are several uninitialized_* functions, but they use placement new directly. If you really care you can extract this code to a separate construct_range function.
There's also an exception safety issue - in case one of the constructors throws, you need to destroy the earlier elements in order to satisfy the strong exception guarantee and free the memory too (destructor won't get called in case constructor throws)
I want to replace the standard allocator with a more robust allocator (the C++ standard only requires an overflow check on vector::resize). The various C++ allocators supplied with many libraries fall flat on their face when fed negative self tests.
I have access to a more robust allocator. ESAPI's allocator not only checks for overflow, it also has debug instrumentation to help find mistakes. http://code.google.com/p/owasp-esapi-cplusplus/source/browse/trunk/esapi/util/zAllocator.h.
Is there a standard way to replace the C++ allocator used in a program without too much effort? I also want to ensure its replaced in library code, which I may not have access to source code.
Unlike malloc which is a library function that can be replaced by another function with the same signature, std::allocator is a class template and template code is instantiated as needed and inlined into code that uses it. Some standard library code will have already been compiled into the library's object files and will contain instantiated std::allocator code which can't be replaced. So the only way is if the standard library provides some non-standard way to replace its std::allocator. Luckily, GCC's libstdc++ allows you to do just that, allowing you to select the implementation used for std::allocator when GCC is configured and built, with a few different choices
It wouldn't be too much work to add the ESAPI allocator to the GCC sources as one of the options, then rebuild GCC to use that allocator as the base class of std::allocator providing its implementation. You might need to tweak the ESAPI allocator code a bit, and maybe alter the libstdc++ configure script to allow you to say --enable-libstdcxx-allocator=esapi
If you want to modify allocation on a global basis instead of per-container, you probably want to replace ::operator new and ::operator delete. Conceivably, you'd also want to replace ::operator new[] and ::operator delete[] as well -- but these are only used for allocating arrays, which you should almost never use anyway (aside, in case it wasn't obvious: no, these are not used to allocate memory for a std::vector, despite its being rather similar to an array in some ways).
Although trying to replace most parts of the library is prohibited, the standard specifically allows replacing these.
Of course, if somebody is already specifying a different allocator for a particular container, and that allocator doesn't (eventually) get its memory via ::operator new (or ::operator new[]) this will not affect that container/those containers.
In C++0x, define a new template alias in namespace mystd that is a std::vector but with your custom allocator. Replace all std::vectors with mystd::vector. Get rid of all using namespace std and using std::vector in your code.
Rebuild. Replace the places where you used a raw vector<T> with mystd::vector<T>.
Oh, and use a better name than mystd.
I have a question very similar to
How do I allocate a std::string on the stack using glibc's string implementation?
but I think it's worth asking again.
I want an std::string with local storage that overflows into the free store. std::basic_string provides an allocator as a template parameter, so it seems like the thing to do is to write an allocator with local storage and use it to parameterize the basic_string, like so:
std::basic_string<
char,
std::char_traits<char>,
inline_allocator<char, 10>
>
x("test");
I tried to write the inline_allocator class that would work the way you'd expect: it reserves 10 bytes for storage, and if the basic_string needs more than 10 bytes, then it calls ::operator new(). I couldn't get it to work. In the course of executing the above line of code, my GCC 4.5 standard string library calls the copy constructor for inline_allocator 4 times. It's not clear to me that there's a sensible way to write the copy constructor for inline_allocator.
In the other StackOverflow thread, Eric Melski provided this link to a class in Chromium:
http://src.chromium.org/svn/trunk/src/base/stack_container.h
which is interesting, but it's not a drop-in replacement for std::string, because it wraps the std::basic_string in a container so that you have to call an overloaded operator->() to get at the std::basic_string.
I can't find any other solutions to this problem. Could it be that there is no good solution? And if that's true, then are the std::basic_string and std::allocator concepts badly flawed? I mean, it seems like this should be a very basic and simple use case for std::basic_string and std::allocator. I suppose the std::allocator concept is designed primarily for pools, but I think it ought to cover this as well.
It seems like the rvalue-reference move semantics in C++0x might make it possible to write inline_allocator, if the string library is re-written so that basic_string uses the move constructor of its allocator instead of the copy constructor. Does anyone know what the prospect is for that outcome?
My application needs to construct a million tiny ASCII strings per second, so I ended up writing my own fixed-length string class based on Boost.Array, which works fine, but this is still bothering me.
Andrei Alexandrescu, C++ programmer extraordinaire who wrote "Modern C++ Design" once wrote a great article about building different string implementations with customizable storage systems. His article (linked here) describes how you can do what you've described above as a special case of a much more general system that can handle all sorts of clever memory allocation requirements. This doesn't talk so much about std::string and focuses more on a completely customized string class, but you might want to look into it as there are some real gems in the implementation.
C++2011 is really going to help you here :)
The fact is that the allocator concept in C++03 was crippled. One of the requirement was that an allocator of type A should be able to deallocate memory from any other allocator from type A... Unfortunately this requirement is also at odds with stateful allocators each hooked to its own pool.
Howard Hinnant (who manages the STL subgroup of the C++ commitee and is implementing a new STL from scratch for C++0x) has explored stack-based allocators on his website, which you could get inspiration from.
This is generally unnecessary. It's called the "short string optimization", and most implementations of std::string already include it. It may be hard to find, but it's usually there anyway.
Just for example, here's the relevant piece of sso_string_base.h that's part of MinGW:
enum { _S_local_capacity = 15 };
union
{
_CharT _M_local_data[_S_local_capacity + 1];
size_type _M_allocated_capacity;
};
The _M_local_data member is the relevant one -- space for it to store (up to) 15 characters (plus a NUL terminator) without allocating any space on the heap.
If memory serves, the Dinkumware library included with VC++ allocates space for 20 characters, though it's been a while since I looked, so I can't swear to that (and tracking down much of anything in their headers tends to be a pain, so I prefer to avoid looking if I can).
In any case, I'd give good odds that you've been engaged in that all-too-popular pass-time known as premature optimization.
I believe the code from Chromium just wraps things into a nice shell. But you can get the same effect without using the Chromium wrapper container.
Because the allocator object gets copied so often, it needs to hold a reference or pointer to the memory. So what you'd need to do is create the storage buffer, create the allocator object, then call the std::string constructor with the allocator.
It will be a lot wordier than using the wrapper class but should get the same effect.
You can see an example of the verbose method (still using the chromium stuff) in my question about stack vectors.
I need to create a custom allocator for std:: objects (particularly and initially for std::vector) but it might eventually come to use others
The reason I need to create a custom allocator is that I need to track allocated (heap & stack) resources by individual components of the application (this is an inherent feature of the application). I will need the custom allocator to monitor the heap portion of the resources, so it is essential that I'm able to pass to the std::vector constructor something like
trackerId idToTrackUsage;
myAlloca<int> allocator(idToTrackUsage);
vector<int> Foo( allocator );
However, after reading a bit I found this little bomb about the STL / C++ standard (see references) saying that all allocator instances of a given type should be equivalent (that is that == should return true for any two instances) and, most terminal; any allocator should be able to deallocate memory allocated by any other instance (that is, without having a way to know what that other instance might be). In short, allocators cannot have state.
So I'm trying to find the best way around this. Any clever ideas? I really really REALLY don't want to have to keep a custom version of std::vector around.
EDIT: i read about scoped allocators for c++0x on http://www2.research.att.com/~bs/C++0xFAQ.html#scoped-allocator but i couldn't really get far into understanding how this applies to my problem. If anyone thinks c++0x alleviates this problem, please comment
References:
Allocator C++ article in Wikipedia
Some random further reading courtesy of Google
Aside from the obvious answer ("if you violate any requirement, that's undefined behavior, good night and thanks for playing"), I imagine the worst that would likely happen, is that the vector implementation can rely on the requirement that "all instances of the allocator class are interchangeable" in the obvious way:
vector(const Allocator &youralloc = Allocator()) {
const Allocator hawhaw;
// use hawhaw and ignore youralloc.
// They're interchangeable, remember?
}
Looking at the source, GCC's vector implementation (which I think is based eventually on SGI's original STL implementation) does sort-of store a copy of the allocator object passed into that constructor, so there's some hope that this won't happen.
I'd say try it and see, and document what you've done very carefully, so that anyone attempting to use your code on an implementation that you haven't checked, knows what's going on. Implementers are encouraged in the standard to relax the restrictions on allocators, so it would be a dirty trick to make it look as though they're relaxed when really they aren't. Which doesn't mean it won't happen.
If you're really lucky, there's some documentation for your container implementation that talks about allocators.
You could, of course, leave a pointer to whatever state you need in any allocated blocks. This does of course mean that any per-block state must be stored in that block, and the allocator instances would act more like handles than actual objects in as of themselves.
Making the allocator state static would do the trick, if you're able to work with that. It does mean that all allocators of that type will have to share their state, but from your requirements, that sounds like it could be acceptable
To respond to your edit: yes, in C++0x or C++11, allocators can have state.
Would it be possible in C++ to create a custom allocator that works simply like this:
{
// Limit memory to 1024 KB
ScopedMemoryPool memoryPool(1024 * 1024);
// From here on all heap allocations ('new', 'malloc', ...) take memory from the pool.
// If the pool is depleted these calls result in an exception being thrown.
// Examples:
std::vector<int> integers(10);
int a * = new int [10];
}
I couldn't find something like this in the boost libraries, or anywhere else.
Is there a fundamental problem that makes this impossible?
You would need to create a custom allocator that you pass in as a template param to vector. This custom allocator would essentially wrap the access to your pool and do whatever size validations that it wants.
Yes you can make such a construct, it's used in many games, but you'll basically need to implement your own containers and call memory allocation methods of that pool that you've created.
You could also experiment with writing a custom allocator for the STL containers, although it seems that that sort of work is generally advised against. (I've done it before and it was tedious, but I don't remember any specific problems.)
Mind- writing your own memory allocator is not for the faint of heart. You could take a look at Doug Lea's malloc, which provides "memory spaces", which you could use in your scoping construct somehow.
I will answer a different question. Look at 'efficient c++' book. One of the things they discuss is implementing this kind of thing. That was for a web server
For this particular thing you can either mess at the c++ layer by overriding new and supplying custom allocators to the STL.
Or you can mess at the malloc level, start with a custom malloc and work from there (like dmalloc)
Is there a fundamental problem that makes this impossible?
Arguing about program behavior would become fundamentally impossible. All sorts of weird issues will come up. Certain sections of the code may or may not execute though this will seeminly have no effect on the next sections which may work un-hindered. Certain sections may always fail. Dealing with the standard-library or any other third party library will become extremely difficult. There may be fragmentations at run-time at times and at times not.
If intent is that all allocations within that scope occur with that allocator object, then it's essentially a thread-local variable.
So, there will be multithreading issues if you use a static or global variable to implement it. Otherwise, not a bad workaround for the statelessness of allocators.
(Of course, you'll need to pass a second template argument eg vector< int, UseScopedPool >.)