I want to implement static_vector, i.e. container like std::vector, but having fixed size storage statically allocated (reserved) on construction and never exceeded during its lifetime.
Because allocator object is a data member of almost every (except std::array) container in C++ standard library, I decided to make raw storage a part of allocator. In the case std::vector virtually will contain whole storage it use.
#include <type_traits>
#include <new>
#include <cassert>
template<size_t N>
struct make
{
template<typename T>
struct static_allocator
{
using value_type = T;
using propagate_on_container_copy_assignment = std::false_type;
using propagate_on_container_move_assignment = std::false_type;
using propagate_on_container_swap = std::false_type;
[[nodiscard]] T * allocate(std::size_t n)
{
if (n != N)
throw std::bad_alloc();
return reinterpret_cast<T*>(&m_storage);
}
void deallocate(T * p, std::size_t n) noexcept
{
assert(p == reinterpret_cast<T*>(&m_storage));
assert(n == N);
}
private:
std::aligned_storage_t<sizeof(T) * N, alignof(T)> m_storage;
};
};
template<size_t N, typename T>
bool operator == (const typename make<N>::template static_allocator<T> & lhs, const typename make<N>::template static_allocator<T> & rhs) { return &lhs == &rhs; }
template<size_t N, typename T>
bool operator != (const typename make<N>::template static_allocator<T> & lhs, const typename make<N>::template static_allocator<T> & rhs) { return !(lhs == rhs); }
#include <vector>
#include <utility>
#include <iterator>
#include <algorithm>
#include <iostream>
int main()
{
std::vector<int, make<10>::static_allocator<int>> v;
v.reserve(10);
for (int i = 0; i < 10; ++i) {
v.push_back(i);
}
auto u = std::move(v);
std::copy(std::cbegin(u), std::cend(u), std::ostream_iterator<int>(std::cout, " "));
std::cout << std::endl;
}
On destruction the code gives an error:
prog.exe: prog.cc:29: void make<10>::static_allocator<int>::deallocate(T *, std::size_t) [N = 10, T = int]: Assertion `p == reinterpret_cast<T*>(&m_storage)' failed.
The cause of the error is that during move construction of std::vector underlying allocator is also move constructed. There is postcondition for allocators a and a1 of type A: a == a1 after A a1(std::move(a));, which breaks whole endeavour.
The postcondition makes allocators in Standard Library too restricted, I believe. Is there chance would it be removed in the future? Is there strong theoretical reason to keep this restriction in the Standard?
Here is my implementation of what you possibly try to achieve.
https://github.com/alekstheod/static_vector
Basically store the array -> data inside your own definition of the vector and replace the allocator to use this data when allocating a memory.
Based on your saying:
I want to implement static_vector, i.e. container like std::vector, but having fixed size storage statically allocated (reserved) on construction and never exceeded during its lifetime.
Looks like a static array would be the best, since you dont need to exceed the size ever, just do something like
int arr[your_size];
and intialize what you need to intialize, set all the other values to INT_MAX or INT_MIN.
Related
How to programatically find if a std::wstring is allocated with Short String Optimization? I was trying to detect such cases and use reserve to move out of the SSO.
The below code prints a small difference in the addresses:
#include <string>
#include <iostream>
int main ()
{
std::wstring str = L"H";
std::cout<<&str<<" "<<(void*)str.data()<<"\n";
}
Output example:
0x7ffc39465220 0x7ffc39465230
Although in windows console app the address comes exactly same:
There are two main ways to go about it:
Just straight-up test it.
Remember to use std::less to get a full order, and std::addressof to account for overloaded op& in weird classes.
template <class T>
constexpr bool uses_sbo(T&& t) noexcept {
auto less = std::less<const volatile void*>();
auto p = std::addressof(t);
return less(t.data(), p + 1) && !less(p, t.data()));
}
Compare the capacity.
This relies on the implementation being sensible, meaning never allocating dynamically when the current capacity suffices, and not dynamically allocating an initial buffer on principle.
template <class T>
constexpr bool is_minimal(T&& t) noexcept {
return t.capacity() == T().capacity();
}
Both were generalized for all contiguous containers.
template<typename T>
bool ssoTest1(const std::basic_string<T>& s)
{
return s.capacity() == std::basic_string<T>{}.capacity();
}
template<typename T>
bool ssoTest2(const std::basic_string<T>& s)
{
uintptr_t r = s.data() - reinterpret_cast<const T*>(&s);
return r <= sizeof(s);
}
https://godbolt.org/z/v3b57Y
I have a pretty simple custom/local allocator. My goal is to use an array on the stack as the allocating portion of memory. It appears to work in std::vector but when I try to plug it in to std::unordered_map it fails to compile. gcc 7.4.0's error messages are pretty impenetrable. Something along the lines of:
hashtable_policy.h:2083:26: error: no matching function for call to
‘MonotonicIncreasingAllocator<std::pair<const int, std::string>, 500>::
MonotonicIncreasingAllocator(std::__detail::_Hashtable_alloc<MonotonicIncreasingAllocator
<std::__detail::_Hash_node<std::pair<const int, std::string>, false>, 500> >::
__node_alloc_type&)’
__value_alloc_type __a(_M_node_allocator());
Clang 7.1.0 is a bit more manageable. Scrolling from an error like error: no matching conversion for functional-style cast from 'const std::_Hashtable . . . I find:
hashmap_custom_alloc.cpp:11:5: note: candidate constructor not viable: no known conversion from
'MonotonicIncreasingAllocator<std::__detail::_Hash_node<std::pair<const int,
std::__cxx11::basic_string<char> >, false>, [...]>' to 'const
MonotonicIncreasingAllocator<std::__detail::_Hash_node_base *, [...]>' for 1st argument
MonotonicIncreasingAllocator(const MonotonicIncreasingAllocator& rhs) = default;
^
Makes it a bit clearer this std::__detail::_Hash_node_base bit is getting in the way. Here is the code, neither unordered_map declaration compiles:
#include <array>
#include <stdexcept>
#include <unordered_map>
#include <vector>
template<class T, std::size_t max_size>
class MonotonicIncreasingAllocator
{
public:
MonotonicIncreasingAllocator() : _index{0} {}
using type = MonotonicIncreasingAllocator<T, max_size>;
using other = MonotonicIncreasingAllocator<T, max_size>;
using value_type = T;
using size_type = std::size_t;
using difference_type = std::ptrdiff_t;
using propagate_on_container_move_assignment = std::true_type;
using is_always_equal = std::true_type;
template<class U>
using rebind = MonotonicIncreasingAllocator<U, max_size>;
T* allocate(std::size_t n)
{
T* r = _data.begin() + _index;
_index += n;
return r;
}
constexpr void deallocate(T* p, std::size_t n)
{
throw std::runtime_error("MontonicIncreasingAllocator can never deallocate()!");
}
private:
std::size_t _index;
std::array<T, max_size> _data;
};
int main()
{
using namespace std;
using key = int;
using value = string;
using item = pair<key, value>;
using alloc = MonotonicIncreasingAllocator<item, 500>;
alloc a0;
alloc a1;
vector<item, alloc> v0(a0);
vector<int, alloc> v1;
// unordered_map<key, value, hash<key>, equal_to<key>, alloc> m; // doesn't compile
// unordered_map<key, value, hash<key>, equal_to<key>, alloc> m(500, a1); // doesn't compile
return 0;
}
An allocator of type T must be rebindable to an allocator of type U -- this is why there is the rebind template.
To do this you must offer a way to conversion-construct from a type U to a type T e.g. a constructor that constructs from MonotonicIncreasingAllocator<U, ...>&, such as:
template <typename U>
MonotonicIncreasingAllocator( const MonotonicIncreasingAllocator<U, max_size>& )
You might notice a problem that immediately comes from this: an array<U,max_size> cannot necessarily be copied to an array<T,max_size>; and due to this, you will want to rethink your allocator design.[1]
For legacy reasons, the C++ "Allocator" model is meant to be copyable. This requirement makes it difficult to work with allocators that itself contain state, rather than indirectly point to state.
Note: The reason this may have worked for vector is because an allocator of type T doesn't get rebound on a vector<T>, since it only needs to allocate n instances of T. This is not true for more complex data structures like a map, set, unordered_map, etc -- since there may be nodes of objects or other contiguous sequences internally used.
[1] Stateful allocators are stored directly into the containers that use them. This means that a vector<T,MonotonicIncreasingAllocator<T,N>> will now also store the allocator itself, containing an array<T,N>, directly inside of the vector class, in addition to its own data -- which is wasteful. Copying or even moving a container with this allocator would be an extremely expensive operation.
Additionally, by storing the data directly inside of the allocator, conversion-construction requires a copy of the entire internal std::array object, which means that the rebinding constructs a new object that refers to a different monotonic structure than the allocator that was being rebound -- which isn't ideal.
You should look into the architecture that's used in std::pmr::polymorphic_allocator for better inspiration. The std::pmr::polymorphic_allocator holds onto 1 data type: a std::memory_resource pointer, which makes rebinding cheap, and storage of this allocator cheap. The memory_resource is type-ambiguous and passed by indirection, which allows for allocators after being rebound to use and refer to the same memory pool.
As #Human-Compiler talked about in his answer, one should not couple the allocated data with the allocator. The solution is rather simple: pass in a pointer from the desired array-on-the-stack. You needn't bother with all the allocator-wrapper nonsense one finds in this thread and elsewhere. In SOLID terms, the data is injected as a dependency into the allocator.
I still find the rebind interface awfully curious. It's clearly poor design that we're stuck with. Beyond writing the struct rebind { other... archaic alias, one must also provide a copy constructor from a rebound type. The latter is hardly documented, if at all.
#include <array>
#include <unordered_map>
#include <vector>
struct SharedArray
{
uint8_t* data;
uint64_t index;
};
template<class T>
class MonotonicIncreasingAllocator
{
public:
MonotonicIncreasingAllocator(SharedArray& a) : _data{a} {}
template<class U>
MonotonicIncreasingAllocator(const MonotonicIncreasingAllocator<U>& rhs)
: _data{const_cast<MonotonicIncreasingAllocator<U>&>(rhs).data()} {}
using type = MonotonicIncreasingAllocator<T>;
using other = MonotonicIncreasingAllocator<T>;
using value_type = T;
using size_type = std::size_t;
using difference_type = std::ptrdiff_t;
using propagate_on_container_move_assignment = std::true_type;
using is_always_equal = std::true_type;
template<class U>
using rebind = MonotonicIncreasingAllocator<U>;
T* allocate(std::size_t n)
{
T* r = _data.data + _data.index;
_data.index += n * sizeof(T);
return r;
}
constexpr void deallocate(T* p, std::size_t n)
{
return;
}
SharedArray& data()
{
return _data;
}
private:
SharedArray& _data;
};
int main()
{
using namespace std;
using key = int;
using value = string;
using item = pair<key, value>;
std::array<uint8_t, 4096> arr; // allocate enough, here but a page
SharedArray sharr;
sharr.index = 0;
sharr.data = arr.begin();
using alloc = MonotonicIncreasingAllocator<item>;
alloc a0(sharr);
alloc a1(sharr);
vector<item, alloc> v0(a0);
unordered_map<key, value, hash<key>, equal_to<key>, alloc> m(500, a1);
return 0;
}
There is a good talk by Jason Turner and Ben Deane from C++Now 2017 called "Constexpr all the things" which also gives a constexpr vector implementation. I was dabbling with the idea myself, for educational purposes. My constexpr vector was pure in the sense that pushing back to it would return a new vector with added element.
During the talk, I saw a push_back implementation tat looks like more or less following:
constexpr void push_back(T const& e) {
if(size_ >= Size)
throw std::range_error("can't use more than Size");
else {
storage_[size_++] = e;
}
}
They were taking the element by value and moving it but, I don't think this is the source of my problems. The thing I want to know is, how this function could be used in a constexpr context? This is not a const member function, it modifies the state. I don think it is possible to do something like
constexpr cv::vector<int> v1;
v1.push_back(42);
And if this is not possible, how could we use this thing in constexpr context and achieve the goal of the task using this vector, namely compile-time JSON parsing?
Here is my version, so that you can see both my new vector returning version and the version from the talk. (Note that performance, perfect forwarding etc. concerns are omitted)
#include <cstdint>
#include <array>
#include <type_traits>
namespace cx {
template <typename T, std::size_t Size = 10>
struct vector {
using iterator = typename std::array<T, Size>::iterator;
using const_iterator = typename std::array<T, Size>::const_iterator;
constexpr vector(std::initializer_list<T> const& l) {
for(auto& t : l) {
if(size_++ < Size)
storage_[size_] = std::move(t);
else
break;
}
}
constexpr vector(vector const& o, T const& t) {
storage_ = o.storage_;
size_ = o.size_;
storage_[size_++] = t;
}
constexpr auto begin() const { return storage_.begin(); }
constexpr auto end() const { return storage_.begin() + size_; }
constexpr auto size() const { return size_; }
constexpr void push_back(T const& e) {
if(size_ >= Size)
throw std::range_error("can't use more than Size");
else {
storage_[size_++] = e;
}
}
std::array<T, Size> storage_{};
std::size_t size_{};
};
}
template <typename T>
constexpr auto make_vector(std::initializer_list<T> const& l) {
return cx::vector<int>{l};
}
template <typename T>
constexpr auto push_back(cx::vector<T> const& o, T const& t) {
return cx::vector<int>{o, t};
}
int main() {
constexpr auto v1 = make_vector({1, 2, 3});
static_assert(v1.size() == 3);
constexpr auto v2 = push_back(v1, 4);
static_assert(v2.size() == 4);
static_assert(std::is_same_v<decltype(v1), decltype(v2)>);
// v1.push_back(4); fails on a constexpr context
}
So, this thing made me realize there is probably something deep that I don' know about constexpr. So, recapping the question; how such a constexpr vector could offer a mutating push_back like that in a constexpr context? Seems like it is not working in a constexpr context right now. If push_back in a constexpr context is not intended to begin with, how can you call it a constexpr vector and use it for compile-time JSON parsing?
Your definition of vector is correct, but you can't modify constexpr objects. They are well and truly constant. Instead, do compile-time calculations inside constexpr functions (the output of which can then be assigned to constexpr objects).
For example, we can write a function range, which produces a vector of numbers from 0 to n. It uses push_back, and we can assign the result to a constexpr vector in main.
constexpr vector<int> range(int n) {
vector<int> v{};
for(int i = 0; i < n; i++) {
v.push_back(i);
}
return v;
}
int main() {
constexpr vector<int> v = range(10);
}
Your return cx::vector<int>{o, t}; will produce a compilation error when o and t are of types cx::vector<T> and T respectively, because those are different types, while all elements of std::initializer_list<T> should be of same type (o is not expanded into a list of its elements).
If you're merely after your 'pure' implementation of push_back, then you can make do with standard arrays:
#include <array>
template <typename T, std::size_t N>
constexpr auto push_back(std::array<T, N> const& oldArr, T const& el) {
std::array<T, N+1> newArr{};
std::copy(begin(oldArr), end(oldArr), begin(newArr));
newArr[N] = el;
return newArr;
}
int main() {
constexpr auto a1 = std::to_array({1, 2, 3});
static_assert(a1.size() == 3);
constexpr auto a2 = push_back(a1, 4);
static_assert(a2.size() == 4);
// This assert will still fail though, because push_back's implementation
// above not only returns new array, but also a new type.
// For example, std::array<int, 3> is not the same type as std::array<int, 4>
//static_assert(std::is_same_v<decltype(a1), decltype(a2)>);
}
I'd like to persist collection with data (unordered_map for instance) in memory mapped file, so I need something like offset kind of pointer to deal with (collection can be mapped to any random address later so I need offset from the beginning of mapped region)). The goal is to create some big dictionary the way I don't have to initialize it every time to use it effective way (just attaching/mapping only to any available mem location on app startup).
When I try to implement it in Visual Studio 2015 update 3 I receive compilation errors showing collections (vector for instance) use raw pointers internally instead of my class to represent offset pointers.
I know boost/interprocess has own specialized/custom collections to persist data in shared memory (or memory mapped file), but I need data persistence way that can work with any STL container.
The question is like in title:
Does anybody know any STL implementation supporting custom smart pointer on custom allocators?
Maybe somebody did something like this already with success. I'm not sure definition of offset smart pointer is sufficient in such case only.
Update 1:
Below are definitions of the allocator and offset pointer I have at the moment...
Allocator:
#include <memory>
#include "esoft_offset_pointer.hpp"
namespace esoft::mem_map_file_allocator
{
template<typename T>
class mmf_allocator
{
public:
using value_type = T;
using size_type = size_t;
using pointer = mmf_offset_pointer<T>;
using const_pointer = mmf_offset_pointer<const T>; // T*;
//using difference_type = ptrdiff_t;
// rebind allocator to type U
template <typename U>
struct rebind {
typedef mmf_allocator<U> other;
};
// constructors and destructor
mmf_allocator() noexcept {};
mmf_allocator(const mmf_allocator&) noexcept {};
template <typename U>
mmf_allocator(const mmf_allocator<U>&) noexcept {};
~mmf_allocator() {};
pointer allocate(size_type num, std::allocator<void>::const_pointer hint = 0)
{
return reinterpret_cast<pointer>(new char[num * sizeof(T)]);
}
// deallocate storage p of deleted elements
void deallocate(pointer p, size_type num)
{
delete[] reinterpret_cast<T*>(p);
}
//// initialize elements of allocated storage p with values args
//template <typename U, typename... Args>
//void construct(U* p, Args&&... args)
//{
//}
//// delete elements of initialized storage p
//template <typename U>
//void destroy(U* p)
//{
//}
};
}
Offset pointer:
namespace esoft::mem_map_file_allocator
{
template<typename T>
class mmf_offset_pointer
{
public:
operator T*();
T& operator ++();
//friend size_t operator -(const mmf_offset_pointer<T> p1, const mmf_offset_pointer<T> p2);
};
}
Usage:
#include <vector>
#include <memory>
#include "esoft_allocator.hpp"
using namespace std;
namespace ea = esoft::mem_map_file_allocator;
using vec_offset_alloc = ea::mmf_allocator<int>;
int main()
{
vector<int, vec_offset_alloc> vec;
for (int pos = 10; pos < 20; ++pos)
{
vec.push_back(pos);
}
int k = vec[4];
return 0;
}
I don't write body of methods yet. My goal is, first of all, to find the way (in reality set of methods and operators) to declare allocator and offset pointer the way it can be accepted by std::vector for instance.
Recently i have been looking for a pool/allocator mechanism.
Boost Pool seems to provide the solution, but there is still things, which it have not been able to deduce from the documentation.
What need to be allocated
Several small classes (~30 chars)
std::map (i want to ensure it do not perform dynamic allocator by itself)
allocation within pugi::xml
std::strings
How to control of address space for allocation (or just amount)
The object_pool seem the to provide a good way for allocating need 1)
However it would like to set a fixed size for the allocator to use. By default it grabs memory be ifself.
If possible i would like to give it the address space it can play within.
char * mem_for_class[1024*1024];
boost::object_pool<my_class,mem_for_class> q;
or:
const int max_no_objs=1024;
boost::object_pool<my_class,max_no_objs> q;
Although the UserAllocator is available in Boost::Pool; it seem to defeat the point. I am afraid the control needed would make it too inefficient... and it would be better to start from scratch.
It it possible to set a fixed area for pool_allocator ?
The question is a bit similar to the first.
Do boost pool provide any way of limiting how much / where there is allocated memory when giving boost::pool_allocator to a std-type-class (e.g. map)
My scenario
Embedded linux programming. The system must keep running for..ever. So we can not risk any memory segmentation. Currently i mostly either static allocation (stack), but also a few raw "new"s.
I would like an allocation scheme that ensure i use the same memory area each time the program loops.
Speed /space is important, but safety is still top priority.
I hope StackOverflow is the place to ask. I tried contacting the author of Boost::Pool "Stephen" without luck. I have not found any Boost-specific forum.
You can always create an allocator that works with STL. If it works with STL, it should work with boost as you are able to pass boost allocators to STL containers..
Considering the above, an allocator that can allocate at a specified memory address AND has a size limitation specified by you can be written as follows:
#include <iostream>
#include <vector>
template<typename T>
class CAllocator
{
private:
std::size_t size;
T* data = nullptr;
public:
typedef T* pointer;
typedef const T* const_pointer;
typedef T& reference;
typedef const T& const_reference;
typedef std::size_t size_type;
typedef std::ptrdiff_t difference_type;
typedef T value_type;
CAllocator() {}
CAllocator(pointer data_ptr, size_type max_size) noexcept : size(max_size), data(data_ptr) {};
template<typename U>
CAllocator(const CAllocator<U>& other) noexcept {};
CAllocator(const CAllocator &other) : size(other.size), data(other.data) {}
template<typename U>
struct rebind {typedef CAllocator<U> other;};
pointer allocate(size_type n, const void* hint = 0) {return &data[0];}
void deallocate(void* ptr, size_type n) {}
size_type max_size() const {return size;}
};
template <typename T, typename U>
inline bool operator == (const CAllocator<T>&, const CAllocator<U>&) {return true;}
template <typename T, typename U>
inline bool operator != (const CAllocator<T>& a, const CAllocator<U>& b) {return !(a == b);}
int main()
{
const int size = 1024 / 4;
int ptr[size];
std::vector<int, CAllocator<int>> vec(CAllocator<int>(&ptr[0], size));
int ptr2[size];
std::vector<int, CAllocator<int>> vec2(CAllocator<int>(&ptr2[0], size));
vec.push_back(10);
vec.push_back(20);
vec2.push_back(30);
vec2.push_back(40);
for (std::size_t i = 0; i < vec2.size(); ++i)
{
int* val = &ptr2[i];
std::cout<<*val<<"\n";
}
std::cout<<"\n\n";
vec2 = vec;
for (std::size_t i = 0; i < vec2.size(); ++i)
{
int* val = &ptr2[i];
std::cout<<*val<<"\n";
}
std::cout<<"\n\n";
vec2.clear();
vec2.push_back(100);
vec2.push_back(200);
for (std::size_t i = 0; i < vec2.size(); ++i)
{
int* val = &ptr2[i];
std::cout<<*val<<"\n";
}
}
This allocator makes sure that all memory is allocated at a specified address. No more than the amount you specify can be allocated with the freedom to allocate were you want whether it is on the stack or the heap.
You may create your own pool or use a std::unique_ptr as the pool for a single container.
EDIT: For strings, you need an offset of sizeof(_Rep_base). See: Why std::string allocating twice?
and http://ideone.com/QWtxWg
It is defined as:
struct _Rep_base
{
std::size_t _M_length;
std::size_t _M_capacity;
_Atomic_word _M_refcount;
};
So the example becomes:
struct Repbase
{
std::size_t length;
std::size_t capacity;
std::int16_t refcount;
};
int main()
{
typedef std::basic_string<char, std::char_traits<char>, CAllocator<char>> CAString;
const int size = 1024;
char ptr[size] = {0};
CAString str(CAllocator<char>(&ptr[0], size));
str = "Hello";
std::cout<<&ptr[sizeof(Repbase)];
}