Not sure if "header" is the correct term, so please correct me if it isn't.
I'm trying to first allocate a memory, and then use an overloaded (placement) new[] operator to initialize an array of class objects (say, MyClass).
Say, the size of a MyClass object is 0x68, and I want an array of 0x20. So the total size is sizeof(MyClass) * 0x20 = 0xD00, or so I thought.
Now when I use my overloaded placement new[] operator :
pArr = new(pAllocatedMem)MyClass[0x20];
The compiler returned size_t to the new[] operator is actually 0xD08. There is an extra 8 bytes. Looking at the value of that 8 bytes, it's used to store the size of the array (0x20 in this case).
So is there a constant definition of what this header size is, say from WDK, that I can use? Does this size changes depends on compilers or what else?
That amount of that extra space, if any, is compiler dependent. For the languange definition [expr.new, paragraph 15]:
[It] is a non-negative unspecified value representing array allocation overhead; the result of the new-expression will be offset by this amount from the value returned by operator new[].
It is typically used by the runtime to know how many objects to destroy when delete is eventually called on the array.
Adding to the answer myself.
Did a little test with a colleague. It appears if the class to be initialized has a defined destructor function, the VS compiler will need that overhead storing the array size. No destructor, no overhead. Is that a bug or a feature?!.
#include <malloc.h>
#include <new>
class Foo
{
public:
//Foo() {}
//~Foo() {} // <-- will cause overhead even with user-allocated memory passed to placement new()
int a;
};
int main()
{
int n = 0x10;
size_t size = sizeof(int) * 2 + sizeof(Foo) * n;
void* p = malloc(size);
*((int*)p) = 0xaaaaaaaa;
*(int*)((char*)p + size - sizeof(int)) = 0xbbbbbbbb;
// Placement new with user-allocated memory
Foo* pf = new ((char*)p + sizeof(int)) Foo[n];
for (int i = 0; i < n; i++)
{
pf[i].a = i;
}
free(p);
return 0;
}
Related
I'm new to C++ and I've been playing around with memory allocation lately. I have found out that when you declare a class with a destructor, like so:
class B
{
public:
~B() { }
};
And then create a heap array of it like so:
B* arr = new B[8];
The allocator allocates 12 bytes but when I remove the destructor, it only allocates 8 bytes. This is how I'm measuring the allocation:
size_t allocated = 0;
void* operator new(size_t size)
{
allocated += size;
return malloc(size);
}
void deallocate(B* array, size_t size)
{
delete[] array;
allocated -= size * sizeof(B);
}
Of course I have to call deallocate manually while the new operator is called automatically.
I have found this problem while working with an std::string* and I realised that the deallocator worked fine with an int* but not with the former.
Does anyone know why that happens and more importantly: How to detect these programmatically at runtime?
Thanks in advance.
You are looking at an implementation detail of how your compiler treats new[] and delete[], so there isn't a definitive answer for the extra space being allocated since the answer will be specific to an implementation -- though I can provide a likely reason below.
Since this is implementation-defined, you cannot reliably detect this at runtime. More importantly, there shouldn't be any real reason to do this. Especially if you're new to C++, this fact is more of an interesting/esoteric thing to know of, but there should be no real benefit detecting this at runtime.
It's important to also be aware that this only happens with array-allocations, and not with object allocations. For example, the following will print expected numbers:
struct A {
~A(){}
};
struct B {
};
auto operator new(std::size_t n) -> void* {
std::cout << "Allocated: " << n << std::endl;
return std::malloc(n);
}
auto operator delete(void* p, std::size_t n) -> void {
std::free(p);
}
auto main() -> int {
auto* a = new A{};
delete a;
auto* b = new B{};
delete b;
}
Output:
Allocated: 1
Allocated: 1
Live Example
The extra storage only gets allocated for types with non-trivial destructors:
auto* a = new A[10];
delete[] a;
auto* b = new B[10];
delete[] b;
Outputs:
Allocated: 18
Allocated: 10
Live Example
The most likely reason why this happens is that extra bookkeeping of a single size_t is being kept at the beginning of allocated arrays containing non-trivial destructors. This would be done so that when delete is called, the language can know how many objects require their destructors invoked. For non-trivial destructors, its able to rely on the underlying delete mechanics of their deallocation functions.
This hypothesis is also supported by the fact that for the GNU ABI, the extra storage is sizeof(size_t) bytes. Building for x86_64 yields 18 for an allocation of A[10] (8 bytes for size_t). Building for x86 yields 14 for that same allocation (4 bytes for size_t).
Edit
I don't recommend doing this in practice, but you can actually view this extra data from arrays. The allocated pointer from new[] gets adjusted before being returned to the caller (which you can test by printing the address from the new[] operator).
If you read this data into a std::size_t, you can see that this data -- at least for the GNU ABI -- contains the exact count for the number of objects allocated.
Again, I do not recommend doing this in practice since this exploits implementation-defined behavior. But just for fun:
auto* a = new A[10];
const auto* bytes = reinterpret_cast<const std::byte*>(a);
std::size_t count;
std::memcpy(&count, bytes - sizeof(std::size_t), sizeof(std::size_t));
std::cout << "Count " << count << std::endl;
delete[] a;
The output:
Count 10
Live Example
I'm trying to allocate an array of struct and I want each struct to be aligned to 64 bytes.
I tried this (it's for Windows only for now), but it doesn't work (I tried with VS2012 and VS2013):
struct __declspec(align(64)) A
{
std::vector<int> v;
A()
{
assert(sizeof(A) == 64);
assert((size_t)this % 64 == 0);
}
void* operator new[] (size_t size)
{
void* ptr = _aligned_malloc(size, 64);
assert((size_t)ptr % 64 == 0);
return ptr;
}
void operator delete[] (void* p)
{
_aligned_free(p);
}
};
int main(int argc, char* argv[])
{
A* arr = new A[200];
return 0;
}
The assert ((size_t)this % 64 == 0) breaks (the modulo returns 16). It looks like it works if the struct only contains simple types though, but breaks when it contains an std container (or some other std classes).
Am I doing something wrong? Is there a way of doing this properly? (Preferably c++03 compatible, but any solution that works in VS2012 is fine).
Edit:
As hinted by Shokwav, this works:
A* arr = (A*)new std::aligned_storage<sizeof(A), 64>::type[200];
// this works too actually:
//A* arr = (A*)_aligned_malloc(sizeof(A) * 200, 64);
for (int i=0; i<200; ++i)
new (&arr[i]) A();
So it looks like it's related to the use of new[]... I'm very curious if anybody has an explanation.
I wonder why you need such a huge alignment requirement, moreover to store a dynamic heap allocated object in the struct. But you can do this:
struct __declspec(align(64)) A
{
unsigned char ___padding[64 - sizeof(std::vector<int>)];
std::vector<int> v;
void* operator new[] (size_t size)
{
// Make sure the buffer will fit even in the worst case
unsigned char* ptr = (unsigned char*)malloc(size + 63);
// Find out the next aligned position in the buffer
unsigned char* endptr = (unsigned char*)(((intptr_t)ptr + 63) & ~63ULL);
// Also store the misalignment in the first padding of the structure
unsigned char misalign = (unsigned char)(endptr - ptr);
*endptr = misalign;
return endptr;
}
void operator delete[] (void* p)
{
unsigned char * ptr = (unsigned char*)p;
// It's required to call back with the original pointer, so subtract the misalignment offset
ptr -= *ptr;
free(ptr);
}
};
int main()
{
A * a = new A[2];
printf("%p - %p = %d\n", &a[1], &a[0], int((char*)&a[1] - (char*)&a[0]));
return 0;
}
I did not have your align_malloc and free function, so the implementation I'm providing is doing this:
It allocates larger to make sure it will fit in 64-bytes boundaries
It computes the offset from the allocation to the closest 64-bytes boundary
It stores the "offset" in the padding of the first structure (else I would have required a larger allocation space each time)
This is used to compute back the original pointer to the free()
Outputs:
0x7fff57b1ca40 - 0x7fff57b1ca00 = 64
Warning: If there is no padding in your structure, then the scheme above will corrupt data, since I'll be storing the misalignement offset in a place that'll be overwritten by the constructor of the internal members.
Remember that when you do "new X[n]", "n" has to be stored "somewhere" so when calling delete[], "n" calls to the destructors will be done. Usually, it's stored before the returned memory buffer (new will likely allocate the required size + 4 for storing the number of elements). The scheme here avoid this.
Another warning: Because C++ calls this operator with some additional padding included in the size for storing the array's number of elements, you'll might still get a "shift" in the returned pointer address for your objects. You might need to account for it. This is what the std::align does, it takes the extra space, compute the alignment like I did and return the aligned pointer. However, you can not get both done in the new[] overload, because of the "count storage" shift that happens after returning from new(). However, you can figure out the "count storage" space once by a single allocation, and adjust the offset accordingly in the new[] implementation.
Do not ask me what I'm trying to do, this is just a quick test and its only purpose is to see if there is something wrong with placement new.
I've found an issue, or I just misunderstood something.
#include <vector>
using namespace std;
#define WORKS
int main(int argc, char** argv) {
vector<int>* pp = (vector<int>*)malloc(sizeof(vector<int>)*20);
#ifdef WORKS
for(int i = 0; i < 20; ++i)
new (pp+i) vector<int>;
#else
new (pp) vector<int>[20];
#endif
for(int i = 0; i < 20; ++i)
pp[i].~vector<int>();
}
when you remove the "#define WORKS" it will give you access violation, like
for(int i = 0; i < 20; ++i)
new (pp+i) vector<int>;
which works good, was different from
new (pp) vector<int>[20];
which is the cause of throwing exceptions at the destruction stage. What's going on here?
I'm working on Windows XP and building with VC++ Express 2010.
ยง5.3.4/12:
-- new T[5] results in a call of operator new[](sizeof(T)*5+x)
[ ... ]
Here, x and y are non-negative unspecified values representing array allocation overhead; the result of the new-expression will be offset by this amount from the value returned by operator new[]. This overhead may be applied in all array new-expressions, including those referencing the library function operator new[](std::size_t, void*) and other placement allocation functions. The amount of overhead may vary from one invocation of new to another. [ emphasis added ]
To summarize, trying to place the array may require some unspecified amount of overhead that you're not allocating. As long as you place the elements individually, no such overhead is allowed, so the placement new works.
The result of a new expression does not have to be at the same address passed to the placement new operator. And, you are not guaranteed that the size required to allocate an array is strictly the size of a single element times the number of elements.
5.3.4:
A new-expression passes the amount
of space requested to the allocation
function as the first argument of type
std::size_t. That argument shall be
no less than the size of the object
being created; it may be greater
than the size of the object being
created only if the object is an
array.
So, the more correct version of your code would be:
void *ppstorage= malloc(sizeof(vector<int>)*20);
pp= new (ppstorage) vector<int>[20];
for(int i = 0; i < 20; ++i)
pp[i].~vector<int>();
Although you will almost certainly write past the end of ppstorage. The compiler has to store the count of the array somewhere to properly destruct each element, and for MSVC that is stored before the address returned by the new expression.
In theory, you could overload operator new[] to get the actual allocation size of an array:
void *operator new[](size_t *allocation_size, size_t size)
{
*allocation_size= size;
return nullptr;
}
But I have never tried this.
When you use operator new[] you must deallocate with operator delete[]. You cannot allocate with new[] and then deallocate one by one. So instead of your deallocation loop, you would do this:
delete [] pp;
If I have a typedef of a struct
typedef struct
{
char SmType;
char SRes;
float SParm;
float EParm;
WORD Count;
char Flags;
char unused;
GPOINT2 Nodes[];
} GPATH2;
and it contains an uninitialized array, how can I create an instance of this type so that is will hold, say, 4 values in Nodes[]?
Edit: This belongs to an API for a program written in Assembler. I guess as long as the underlying data in memory is the same, an answer changing the struct definition would work, but not if the underlying memory is different. The Assembly Language application is not using this definition .... but .... a C program using it can create GPATH2 elements that the Assembly Language application can "read".
Can I ever resize Nodes[] once I have created an instance of GPATH2?
Note: I would have placed this with a straight C tag, but there is only a C++ tag.
You could use a bastard mix of C and C++ if you really want to:
#include <new>
#include <cstdlib>
#include "definition_of_GPATH2.h"
using namespace std;
int main(void)
{
int i;
/* Allocate raw memory buffer */
void * raw_buffer = calloc(1, sizeof(GPATH2) + 4 * sizeof(GPOINT2));
/* Initialize struct with placement-new */
GPATH2 * path = new (raw_buffer) GPATH2;
path->Count = 4;
for ( i = 0 ; i < 4 ; i++ )
{
path->Nodes[i].x = rand();
path->Nodes[i].y = rand();
}
/* Resize raw buffer */
raw_buffer = realloc(raw_buffer, sizeof(GPATH2) + 8 * sizeof(GPOINT2));
/* 'path' still points to the old buffer that might have been free'd
* by realloc, so it has to be re-initialized
* realloc copies old memory contents, so I am not certain this would
* work with a proper object that actaully does something in the
* constructor
*/
path = new (raw_buffer) GPATH2;
/* now we can write more elements of array */
path->Count = 5;
path->Nodes[4].x = rand();
path->Nodes[4].y = rand();
/* Because this is allocated with malloc/realloc, free it with free
* rather than delete.
* If 'path' was a proper object rather than a struct, you should
* call the destructor manually first.
*/
free(raw_buffer);
return 0;
}
Granted, it's not idiomatic C++ as others have observed, but if the struct is part of legacy code it might be the most straightforward option.
Correctness of the above sample program has only been checked with valgrind using dummy definitions of the structs, your mileage may vary.
If it is fixed size write:
typedef struct
{
char SmType;
char SRes;
float SParm;
float EParm;
WORD Count;
char Flags;
char unused;
GPOINT2 Nodes[4];
} GPATH2;
if not fixed then change declaration to
GPOINT2* Nodes;
after creation or in constructor do
Nodes = new GPOINT2[size];
if you want to resize it you should use vector<GPOINT2>, because you can't resize array, only create new one. If you decide to do it, don't forget to delete previous one.
also typedef is not needed in c++, you can write
struct GPATH2
{
char SmType;
char SRes;
float SParm;
float EParm;
WORD Count;
char Flags;
char unused;
GPOINT2 Nodes[4];
};
This appears to be a C99 idiom known as the "struct hack". You cannot (in standard C99; some compilers have an extension that allows it) declare a variable with this type, but you can declare pointers to it. You have to allocate objects of this type with malloc, providing extra space for the appropriate number of array elements. If nothing holds a pointer to an array element, you can resize the array with realloc.
Code that needs to be backward compatible with C89 needs to use
GPOINT2 Nodes[1];
as the last member, and take note of this when allocating.
This is very much not idiomatic C++ -- note for instance that you would have to jump through several extra hoops to make new and delete usable -- although I have seen it done. Idiomatic C++ would use vector<GPOINT2> as the last member of the struct.
Arrays of unknown size are not valid as C++ data members. They are valid in C99, and your compiler may be mixing C99 support with C++.
What you can do in C++ is 1) give it a size, 2) use a vector or another container, or 3) ditch both automatic (local variable) and normal dynamic storage in order to control allocation explicitly. The third is particularly cumbersome in C++, especially with non-POD, but possible; example:
struct A {
int const size;
char data[1];
~A() {
// if data was of non-POD type, we'd destruct data[1] to data[size-1] here
}
static auto_ptr<A> create(int size) {
// because new is used, auto_ptr's use of delete is fine
// consider another smart pointer type that allows specifying a deleter
A *p = ::operator new(sizeof(A) + (size - 1) * sizeof(char));
try { // not necessary in our case, but is if A's ctor can throw
new(p) A(size);
}
catch (...) {
::operator delete(p);
throw;
}
return auto_ptr<A>(p);
}
private:
A(int size) : size (size) {
// if data was of non-POD type, we'd construct here, being very careful
// of exception safety
}
A(A const &other); // be careful if you define these,
A& operator=(A const &other); // but it likely makes sense to forbid them
void* operator new(size_t size); // doesn't prevent all erroneous uses,
void* operator new[](size_t size); // but this is a start
};
Note you cannot trust sizeof(A) any where else in the code, and using an array of size 1 guarantees alignment (matters when the type isn't char).
This type of structure is not trivially useable on the stack, you'll have to malloc it. the significant thing to know is that sizeof(GPATH2) doesn't include the trailing array. so to create one, you'd do something like this:
GPATH2 *somePath;
size_t numPoints;
numPoints = 4;
somePath = malloc(sizeof(GPATH2) + numPoints*sizeof(GPOINT2));
I'm guessing GPATH2.Count is the number of elements in the Nodes array, so if it's up to you to initialize that, be sure and set somePath->Count = numPoints; at some point. If I'm mistaken, and the convention used is to null terminate the array, then you would do things just a little different:
somePath = malloc(sizeof(GPATH2) + (numPoints+1)*sizeof(GPOINT2));
somePath->Nodes[numPoints] = Some_Sentinel_Value;
make darn sure you know which convention the library uses.
As other folks have mentioned, realloc() can be used to resize the struct, but it will invalidate old pointers to the struct, so make sure you aren't keeping extra copies of it (like passing it to the library).
SomeObj<unsigned int>* Buffer;
char* BufferPtr = MemoryManager::giveMeSomeBytes(resX*resY*sizeof(SomeObj<unsigned int>));
Buffer = new(BufferPtr) SomeObj<unsigned int>[resX*resY];
when I step past these lines with the debugger, it shows me the values for the variables Buffer and BufferPtr:
BufferPtr: 0x0d7f004c
Buffer: 0x0d7f0050
I don't really understand why those values differ. The way I understand it, placement new should use the memory starting at address 'BufferPtr' to initialize the array elements using theyr default constructors on the allocated memory and return a pointer to the first byte of the first element in the array, which should be exactly the same byte as passed to the placement new operator.
Did I understand something wrong or can someone tell me why the values differ?
thanks!
//edit: ok - i investigated the issue further and got more confusing results:
int size = sizeof(matth_ptr<int>);
char* testPtr1 = (char*)malloc(a_resX*a_resY*sizeof(int));
int* test1 = new(testPtr1) int[a_resX*a_resY];
char* testPtr2 = mmgr::requestMemory(a_resX*a_resY*sizeof(int));
int* test2 = new(testPtr2) int[a_resX*a_resY];
char* testPtr3 = (char*)malloc(a_resX*a_resY*sizeof(matth_ptr<int>));
matth_ptr<int>* test3 = new(testPtr3)matth_ptr<int>[a_resX*a_resY];
char* testPtr4 = mmgr::requestMemory(a_resX*a_resY*sizeof(matth_ptr<int>));
matth_ptr<int>* test4 = new(testPtr4)matth_ptr<int>[a_resX*a_resY];
the debugger returns me the following values for my variables:
size: 4
testPtr1:0x05100418
test1: 0x05100418
testPtr2:0x0da80050
test2: 0x0da80050
testPtr3:0x05101458
test3: 0x0510145c
testPtr4:0x0da81050
test4: 0x0da81054
so it clearly must have something to do with my generic smartpointer class matth_ptr so here it is:
template <class X> class matth_ptr
{
public:
typedef X element_type;
matth_ptr(){
memoryOfst = 0xFFFFFFFF;
}
matth_ptr(X* p)
{
unsigned char idx = mmgr::getCurrentChunkIdx();
memoryOfst = (int)p-(int)mmgr::getBaseAddress(idx);
assert(memoryOfst<=0x00FFFFFF || p==0);//NULL pointer is not yet handled
chunkIdx = idx;
}
~matth_ptr() {}
X& operator*() {return *((X*)(mmgr::getBaseAddress(chunkIdx)+(memoryOfst&0x00FFFFFF)));}
X* operator->() {return ((X*)(mmgr::getBaseAddress(chunkIdx)+(memoryOfst&0x00FFFFFF)));}
X* get() {return ((X*)(mmgr::getBaseAddress(chunkIdx)+(memoryOfst&0x00FFFFFF)));}
template<typename T>
matth_ptr(const matth_ptr<T>& other) {memoryOfst=other.memoryOfst;}//put these two operators into the private part in order to prevent copying of the smartpointers
template<typename T>
matth_ptr& operator=(const matth_ptr<T>& other) {memoryOfst = other.memoryOfst; return *this;}
template<typename T>
friend class matth_ptr;
private:
union //4GB adressable in chunks of 16 MB
{
struct{
unsigned char padding[3]; //3 bytes padding
unsigned char chunkIdx; //8 bit chunk index
};
unsigned int memoryOfst; //24bit address ofst
};
};
can anyone explain me what's going on? thanks!
Be careful with placement new on arrays. In the current standard look to section 5.3.4.12, you'll find this:
new(2,f) T[5] results in a call of operator new[](sizeof(T)*5+y,2,f)
It is clear that it will expect the placement new operator to allocate it additional space beyond what the array contents need. "y" is specified only as a non-negative integral value. It will then offset the result of the new function by this amount.
Also look to 18.4.1.3.4 where it says the placement new operator simply returns the provided pointer. This is obviously the expected part.
Based on 5.3.4.12, since that offset may be different for every invocation of the array, the standard basically means there is no way to allocate the exact amount of size needed. In practice that value is probably constant and you could just add it to the allocation, but his amount may change per platform, and again, per invocation as the standard says.
You're using the array version of the new operator which in your implementation is storing information about the array size in the first few bytes of the memory allocation.
#Mat, This is actually a great question. When I've used placement new[], I've had trouble deleting the storage. Even if I call my own symmetrical placement delete[], the pointer address is not the same as was returned by my own placement new[]. This makes placement new[] completely useless, as you've suggested in the comments.
The only solution I've found was suggested by Jonathan#: Instead of placement new[], use placement new (non-array) on each of the elements of the array. This is fine for me as I store the size myself. The problem is that I have to worry about pointer alignments for elements, which new[] is supposed to do for me.
As others have said, this is due to your C++ implementation storing the size of the array at the start of the buffer you pass to array placement new.
An easy fix for this is to simply assign your array pointer to the buffer, then loop over the array and use regular (non-array) placement new to construct each object in the buffer.