Per suggestion from #BenVoigt in response to my question regarding stack allocated stringstream storage, I designed a stack_allocator (code follows below), and declared a basic_ostringstream type using it.
I am experiencing a strange bug though.
The first character I place into the stream is omitted when I print the resulting string!
Here is an example:
template<typename T, size_t capacity, size_t arr_size>
__thread bool stack_allocator<T, capacity, arr_size>::_used[arr_size] = {};
template<typename T, size_t capacity, size_t arr_size>
__thread T stack_allocator<T, capacity, arr_size>::_buf[capacity][arr_size] = {};
typedef std::basic_ostringstream<char,
std::char_traits<char>,
stack_allocator<char, 1024, 5> > stack_ostringstream;
int main()
{
stack_ostringstream _os;
_os << "hello world";
std::cout << _os.str() << std::endl;
return 0;
}
The resulting output is:
ello world
Can anyone elaborate on what is happening to the first character?
The stack_allocator impl follows: It's pretty simplistic, and I'm sure has lots of room for improvement (not withstanding fixing the bug!)
#include <cstddef>
#include <limits>
#include <bits/allocator.h>
template<typename T, size_t capacity = 1024, size_t arr_size = 5>
class stack_allocator
{
public:
typedef T value_type;
typedef value_type* pointer;
typedef const value_type* const_pointer;
typedef value_type& reference;
typedef const value_type& const_reference;
typedef std::size_t size_type;
typedef std::ptrdiff_t difference_type;
inline explicit stack_allocator() { }
template<typename U>
inline explicit stack_allocator(const stack_allocator<U, capacity, arr_size>& that) { }
inline ~stack_allocator() {}
template<typename U>
struct rebind
{
typedef stack_allocator<U, capacity, arr_size> other;
};
inline pointer allocate(size_type cnt, typename std::allocator<void>::const_pointer = 0)
{
if (cnt > capacity)
return reinterpret_cast<pointer>(::operator new(cnt * sizeof (T)));
for (size_t i = 0; i < arr_size; ++i)
{
if (!_used[i])
{
_used[i] = true;
return reinterpret_cast<pointer>(_buf[i]);
}
}
}
inline void deallocate(pointer p, size_type)
{
for (size_t i = 0; i < arr_size; ++i)
{
if (p != _buf[i])
continue;
_used[i] = false;
return;
}
::operator delete(p);
}
inline pointer address(reference r) { return &r; }
inline const_pointer address(const_reference r) { return &r; }
inline size_type max_size() const
{
return std::numeric_limits<size_type>::max() / sizeof(T);
}
inline void construct(pointer p, const T& t) { new(p) T(t); }
inline void destroy(pointer p) { p->~T(); }
inline bool operator==(const stack_allocator&) const { return true; }
inline bool operator!=(const stack_allocator& a) const { return !operator==(a); }
private:
static __thread bool _used[arr_size];
static __thread T _buf[capacity][arr_size];
};
Your allocate function can fall off the end if you allocate more than arr_size items. If you use g++ -Wall it will warn you about those sorts of things.
The other problem is that your _buf array indexes are backwards. It should be static T _buf[arr_size][capacity]; which has the arr_size as the row, not the other order that you have it in the original code which makes the capacity be the first index.
Also as a side note, just avoid identifiers that start with leading _ because some such identifiers are reserved for the implementation and it's easier to never use them than to remember the precise rules. Finally, never include the bits/ headers directly, just use the real headers. In this case, memory. I also had to add includes for <iostream> and <sstream> to get it to compile.
Related
#include <cstdlib>
#include <memory>
#include <unordered_map>
template <class T>
struct allocator {
typedef size_t size_type;
typedef ptrdiff_t difference_type;
typedef T* pointer;
typedef const T* const_pointer;
typedef T& reference;
typedef const T& const_reference;
typedef T value_type;
allocator() = default;
template <class U>
allocator(const allocator<U>&) {}
T* allocate(std::size_t n) const { return (T*)malloc(n); } // debugger breaks here
void deallocate(T* p, std::size_t) const { free(p); }
};
using allocations_map =
std::unordered_map<void*, std::size_t, std::hash<void*>,
std::equal_to<void*>,
allocator<std::pair<void* const, std::size_t>>>;
allocations_map allocations; // heap corruption in the constructor
void* operator new(std::size_t n) {
auto p = malloc(n);
allocations.emplace(p, n);
return p;
}
void operator delete(void* p) noexcept {
allocations.erase(p);
free(p);
}
int main() { std::vector<int> v(5); }
Why do i corrupt the heap in the constructor of allocations_map? The debugger detects the first heap corruption in a malloc call of the custom allocator, called inside the constructor.
Is there a more elegant solution then to write a non-logging custom allocator for allocations_map? The container shall obviously not log its own allocations.
I also tried two singleton approaches, as suggested in the comments, without success:
allocations_map& get_allocations_map()
{
static allocations_map* allocations_ptr = nullptr;
if (allocations_ptr == nullptr)
{
allocations_ptr = (allocations_map*) malloc(sizeof(allocations_map));
allocations_ptr = new(allocations_ptr)allocations_map;
}
return *allocations_ptr;
}
allocations_map& get_allocations_map()
{
static allocations_map allocations;
return allocations;
}
From std::allocator::allocate allocator allocates n "things" not n bytes. You should change:
T* allocate(std::size_t n) const { return (T*)malloc(n); }
to:
T* allocate(std::size_t n) const { return (T*)malloc(sizeof(T) * n); }
Why do i corrupt the heap in the constructor of allocations_map?
Because the constructor of elements stored in that map access allocated memory out-of-bounds.
So I'm currently in the process of writing a memory debugger and to do that I need stl container objects to use an untracked allocator.
I have std::string peppered throughout my entire codebase, so I typedef'd it to use my untracked allocator:
typedef std::basic_string<char, std::char_traits<char>, UntrackedAllocator<char>> String;
Now when I try to do this:
String str { "Some string" };
String copy = str;
I get this error:
/usr/local/include/c++/7.1.0/ext/alloc_traits.h:95:67: error: no matching function for call to 'UntrackedAllocator<char>::UntrackedAllocator(UntrackedAllocator<char>)' { return _Base_type::select_on_container_copy_construction(__a); }
This is what my Untracked Allocator looks like:
#pragma once
#define NOMINMAX
#undef max
template <typename T>
class UntrackedAllocator {
public:
typedef T value_type;
typedef value_type* pointer;
typedef const value_type* const_pointer;
typedef value_type& reference;
typedef const value_type& const_reference;
typedef std::size_t size_type;
typedef std::ptrdiff_t difference_type;
public:
template<typename U>
struct rebind {
typedef UntrackedAllocator<U> other;
};
public:
inline explicit UntrackedAllocator() {}
inline ~UntrackedAllocator() {}
inline explicit UntrackedAllocator(UntrackedAllocator const&) {}
template<typename U>
inline explicit UntrackedAllocator(UntrackedAllocator<U> const&) {}
// address
inline pointer address(reference r) {
return &r;
}
inline const_pointer address(const_reference r) {
return &r;
}
// memory allocation
inline pointer allocate(size_type cnt,
typename std::allocator<void>::const_pointer = 0) {
T *ptr = (T*)malloc(cnt * sizeof(T));
return ptr;
}
inline void deallocate(pointer p, size_type cnt) {
free(p);
}
// size
inline size_type max_size() const {
return std::numeric_limits<size_type>::max() / sizeof(T);
}
// construction/destruction
inline void construct(pointer p, const T& t) {
new(p) T(t);
}
inline void destroy(pointer p) {
p->~T();
}
inline bool operator==(UntrackedAllocator const& a) { return this == &a; }
inline bool operator!=(UntrackedAllocator const& a) { return !operator==(a); }
};
This is my first time working with custom allocators so I have no idea what's going on with it. It's incredibly annoyning that I can't do str1 = str2 if one of them uses a custom allocator.
The problem is the declaration of the copy c'tors as explicit.
Changing the UntrackedAllocator copy c'tor to:
inline UntrackedAllocator(UntrackedAllocator const&) {}
Solves the compilation issue and everything works just fine:
int main() {
String str { "13" };
String copy = str;
const char* cstr = str.c_str();
int out = atoi(cstr);
}
This happens because the assignment operator of the std::basic_string that accepts const std::basic_string & requires an implicit copy construction of the allocator.
I've created the following code for testing shared memory allocators and containers..
The allocator (basic allocator that just keeps a pointer to a memory block + the size:
template<typename T>
struct SharedMemoryAllocator
{
typedef std::size_t size_type;
typedef std::ptrdiff_t difference_type;
typedef T* pointer;
typedef const T* const_pointer;
typedef T& reference;
typedef const T& const_reference;
typedef T value_type;
void* memory;
std::size_t size;
SharedMemoryAllocator(void* memory, std::size_t size) noexcept : memory(memory), size(size) {};
SharedMemoryAllocator(const SharedMemoryAllocator& other) noexcept : memory(other.memory), size(other.size) {};
template<typename U>
SharedMemoryAllocator(const SharedMemoryAllocator<U>& other) noexcept : memory(other.memory), size(other.size) {};
template<typename U>
SharedMemoryAllocator& operator = (const SharedMemoryAllocator<U>& other) { return *this; }
SharedMemoryAllocator<T>& operator = (const SharedMemoryAllocator& other) { return *this; }
~SharedMemoryAllocator() {}
pointer address(reference value) const {return &value;}
const_pointer address(const_reference value) const {return &value;}
pointer allocate(size_type n, const void* hint = 0) {return static_cast<T*>(memory);}
void deallocate(T* ptr, size_type n) {}
template<typename U, typename... Args>
void construct(U* ptr, Args&&... args) {::new(static_cast<void*>(ptr)) U(std::forward<Args>(args)...);}
void construct(pointer ptr, const T& val) {new(static_cast<T*>(ptr)) T(val);}
template<typename U>
void destroy(U* ptr) {ptr->~U();}
void destroy(pointer ptr) {ptr->~T();}
size_type max_size() const {return size / sizeof(T);}
template<typename U>
struct rebind {typedef SharedMemoryAllocator<U> other;};
};
template <typename T, typename U>
inline bool operator == (const SharedMemoryAllocator<T>& a, const SharedMemoryAllocator<U>& b)
{
return (a == b);
}
template <typename T, typename U>
inline bool operator != (const SharedMemoryAllocator<T>& a, const SharedMemoryAllocator<U>& b)
{
return !(a == b);
}
The container (Just a container that allocates memory using the SharedMemory allocator):
template<typename T, typename allocator = std::allocator<T>>
class CVector
{
private:
T* memory;
std::size_t size, capacity;
allocator alloc;
public:
CVector() : memory(nullptr), size(0), capacity(0), alloc(allocator()) {}
CVector(const allocator &alloc) : memory(nullptr), size(0), capacity(0), alloc(alloc) {}
~CVector()
{
if(memory)
{
for(std::size_t i = 0; i < this->size; ++i)
{
alloc.destroy(memory + i);
}
alloc.deallocate(memory, capacity);
memory = nullptr;
}
}
void reserve(std::size_t size)
{
if(capacity < size)
{
capacity = size;
void* mem = alloc.allocate(capacity);
if(memory && memory != mem)
{
memcpy(static_cast<char*>(mem), memory, size * sizeof(T));
for(std::size_t i = 0; i < this->size; ++i)
{
alloc.destroy(memory + i);
}
alloc.deallocate(memory, capacity);
memory = nullptr;
}
memory = static_cast<T*>(mem);
}
}
void push_back(T&& value)
{
if(capacity == 0)
{
reserve(1);
}
if(size >= capacity)
{
reserve(capacity * 2);
}
alloc.construct(memory + size++, value);
}
T& operator[](std::size_t size)
{
return *(memory + size);
}
const T& operator[](std::size_t size) const
{
return *(memory + size);
}
};
Main:
int main()
{
MemoryMap mem{"Local\\Test", 5000, std::ios::in | std::ios::out};
mem.open();
mem.map();
typedef CVector<int, SharedMemoryAllocator<int>> SHMVec;
SHMVec* vec = ::new(mem.data()) SHMVec(SharedMemoryAllocator<int>(static_cast<char*>(mem.data()) + sizeof(SHMVec), 1024 - sizeof(SHMVec)));
vec->reserve(100);
vec->push_back(100);
vec->push_back(200);
vec->push_back(300);
std::cout<<"Address: "<<mem.data()<<"\n";
std::cin.get();
SHMVec* ptrVec = reinterpret_cast<SHMVec*>(mem.data());
std::cout<<(*ptrVec)[0];
vec->~SHMVec();
}
I read somewhere that std::vector cannot be placed in SharedMemory because it might do some tracking of its own in the current process's address space. So I decided to write my own "vector" which is just a cheap class.
Next, I allocate a shared memory block and I construct the container into that block as shown above in "main".
In the other program, I do (Main):
int main()
{
MemoryMap mem{"Local\\Test", 5000, std::ios::in};
mem.open();
mem.map();
typedef CVector<int, SharedMemoryAllocator<int>> SHMVec;
std::cout<<"Address: "<<mem.data()<<"\n";
SHMVec* ptrVec = reinterpret_cast<SHMVec*>(mem.data());
std::cout<<(*ptrVec)[0];
}
When both programs map the shared memory block at 0x370000 it works. However, if one program allocates the SharedMemoryBlock at 0x370000 and the second at 0x380000, it crashes (the second program crashes trying to access the container created by the first).
Any ideas why this happens? The container is IN the shared memory block. Why does it matter that the address of the blocks have to be the EXACT same?
Here is your problem:
template<typename T, typename allocator = std::allocator<T>>
class CVector
{
private:
T* memory;
^^^^^^^^^^^
Because in your programs the SHMVector object itself is stored in shared memory, you store its data members in shared memory. Thus the pointer to the elements (memory in this case) is stored in shared memory.
If the shared memory segment is loaded at a different address, then memory will point at an invalid address in the memory space of one of the two programs.
Maybe simple solution: don't put the SHMVec object itself in shared memory, only the elements.
boost::interprocess can help you keeping containers in shared memory - see http://www.boost.org/doc/libs/1_38_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.containers_explained.containers
I've started to code FixedAllocator class that allocates memory by chunks of fixed size and works as stack, so that it works in constant time to allocate/deallocate. Actually, I'll need this class to use it with std::vector, so that I have to implement all std::allocator methods.
Everything here is for learning purposes so that - I don't need any complete implementations or headers - the real ones have a lot of code over my problem.
And I got stuck on allocate/deallocate methods - I understand that I should somehow reserve some memory pool - for example using vector, I understand that I should use static_cast to convert char type into T-type, but I don't completely understand how to rebuild this two ideas into list. Deallocate takes pointer as argument, not TNode - that's maybe the main problem.
If someone already wrote this kind of allocator - answer with code will be perfect.
Any suggestions, links and other source of knowledge are welcome. Thank you.
Here is the skeleton of code:
template <typename T, unsigned int nodeSize>
class FixedAllocator : public std::allocator<T>{
private:
static size_t Used;
static const size_t MAX_SIZE = 100000;
struct TNode {
TNode* next;
char data[nodeSize];
};
TNode* head;
public:
typedef T* pointer;
typedef const T* const_pointer;
typedef T & reference;
typedef const T & const_reference;
typedef T value_type;
template <typename U> struct rebind { typedef allocator<U> other; };
FixedAllocator() {
if (Pool.empty()) {
Pool.resize(MAX_SIZE * sizeof(T));
Used = 0;
}
}
FixedAllocator(const FixedAllocator &) {
}
template<typename U>
FixedAllocator(const FixedAllocator<U> &) {
if (Pool.empty()) {
Pool.resize(MAX_SIZE * sizeof(T));
Used = 0;
}
}
pointer address(reference x) const {
return &x;
}
const_pointer address(const_reference x) const {
return &x;
}
pointer allocate(size_t n, FixedAllocator<void>::const_pointer = 0) {}
void deallocate(pointer, size_t) {}
size_t max_size() const throw() {
return MAX_SIZE - size;
}
void construct(pointer p, const_reference val) {
new (static_cast<void*>(p)) value_type(val);
}
void destroy(pointer p) {
p->~value_type();
}
};
Is there a malloc/free based allocator in the STL? If not, does anyone know of a simple copy/paste one? I need it for a map that must not call new/delete.
First, I'd note that changing the allocator for the map itself won't change the allocation used by the objects stored in the map. For example, if you do something like:
std::map<std::string, int, my_allocator<std::pair<const std::string, int> > m;
The map itself will allocate memory using the specified allocator, but when the std::strings in the map allocate memory, they'll still use the default allocator (which will use new and delete. So, if you need to avoid new and delete in general, you have to ensure that not only the map itself uses the right allocator, but that any objects it stores do the same (I know that's probably stating the obvious, but I've overlooked it, so maybe it's worth mentioning).
With that proviso, on with the code:
#ifndef ALLOCATOR_H_INC_
#define ALLOCATOR_H_INC_
#include <stdlib.h>
#include <new>
#include <limits>
namespace JVC {
template <class T>
struct allocator {
typedef size_t size_type;
typedef ptrdiff_t difference_type;
typedef T* pointer;
typedef const T* const_pointer;
typedef T& reference;
typedef const T& const_reference;
typedef T value_type;
template <class U> struct rebind { typedef allocator<U> other; };
allocator() throw() {}
allocator(const allocator&) throw() {}
template <class U> allocator(const allocator<U>&) throw(){}
~allocator() throw() {}
pointer address(reference x) const { return &x; }
const_pointer address(const_reference x) const { return &x; }
pointer allocate(size_type s, void const * = 0) {
if (0 == s)
return NULL;
pointer temp = (pointer)malloc(s * sizeof(T));
if (temp == NULL)
throw std::bad_alloc();
return temp;
}
void deallocate(pointer p, size_type) {
free(p);
}
size_type max_size() const throw() {
return std::numeric_limits<size_t>::max() / sizeof(T);
}
void construct(pointer p, const T& val) {
new((void *)p) T(val);
}
void destroy(pointer p) {
p->~T();
}
};
}
#endif
And, a little test code:
#include <map>
#include <vector>
#include <iostream>
#include <string>
#include <iterator>
#include "allocator.h"
// Technically this isn't allowed, but it's only demo code, so we'll live with it.
namespace std {
std::ostream &operator<<(std::ostream &os, std::pair<std::string, int> const &c) {
return os << c.first << ": " << c.second;
}
}
int main() {
std::map<std::string, int, std::less<std::string>,
JVC::allocator<std::pair<const std::string, int> > > stuff;
stuff["string 1"] = 1;
stuff["string 2"] = 2;
stuff["string 3"] = 3;
std::copy(stuff.begin(), stuff.end(),
std::ostream_iterator<std::pair<std::string, int> >(std::cout, "\n"));
return 0;
}
Indeed, as #MichaelBurr suggests, Stephen J, Lavavej's 'mallocator' is what you're looking for. I just got updated and prettied-up code for it in this answer by #Arnaud today, do have a look.