I currently have a std::ofstream being created on the stack, and when it allocates off a global operator new that assigns it memory from a pre-allocated buffer the program will crash after main completes citing std::locale0 : line 59. Read Access Violation. nodeptr was 0x... as the program crash point. nodeptr's memory address is a real address. I have no idea why this is happening, and I can only assume it was because I misunderstand what the allocations are actually doing.
This behaviour happens on a release build tested on MSVC Version 19.10.25019 x86, building on debug has the program complete without a crash. Dr. Memory reports no leaks in debug mode.
Minimal Code:
#include <fstream>
#include <cstdlib>
std::uint8_t *byte = static_cast<std::uint8_t*>(malloc(134217728)); // 134217728 = 128 MiB
void *operator new(size_t bytes) throw(std::bad_alloc)
{
return static_cast<void*>(byte); // Tested empirically, and this is only called once so this shouldnt be a cause of a problem
}
void operator delete(void *memory) throw()
{}
int main()
{
std::ofstream out("abc");
free(byte);
}
There are two obvious bugs:
What if operator new is called more than once? What if the constructor of out constructs a sub-object?
You free byte while out is still in scope. When out's destructor runs, you've already returned its memory to the system.
Try this:
#include <fstream>
#include <cstdlib>
std::uint8_t *byte = static_cast<std::uint8_t*>(malloc(134217728)); // 134217728 = 128 MiB
void *operator new(size_t bytes) throw(std::bad_alloc)
{
static int offset = 0;
void * ret = static_cast<void*>(byte + offset);
offset += 16 * ((bytes + 15) / 16);
return ret;
}
void operator delete(void *memory) throw()
{}
int main()
{
{
std::ofstream out("abc");
}
free(byte);
}
It appears as it is a Visual Studio bug regarding custom allocations and std::locale.
Related
I have distilled by question down to a very simple example where I have a structure constructor that is initializing everything, yet valgrind complains about uninitialized bytes. The culprit seems to be the boolean member of the class, which causes padding bytes to be inserted before the size_t member. What is the right way to initialize those padding bytes so that valgrind doesn't complain?
#include <iostream>
#include <cstring>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#define MAX 128
typedef struct foo_struct
{
foo_struct(const std::string& name, bool q = true) : q(q)
{
if (name.size() > MAX)
{
throw std::runtime_error("too big");
}
point_name_size = name.size();
memset(n, 0, sizeof(n));
memcpy(n, name.c_str(), name.size());
}
bool q;
size_t point_name_size;
char n[MAX];
} foo_t;
int main()
{
int fd = open("/tmp/foo", O_WRONLY | O_CREAT, 0666);
if (-1 == fd)
{
throw std::runtime_error("Can't create File Descriptor: " + std::string(strerror(errno)));
}
const foo_t f("hello");
ssize_t written = write(fd, &f, sizeof(f));
std::cout << "wrote " << written << std::endl;
return 0;
}
Compile and run with
g++ try.cpp && valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 ./a.out
The valgrind error is
==11790== Syscall param write(buf) points to uninitialised byte(s)
==11790== at 0x54ED154: write (write.c:27)
==11790== by 0x1093DE: main (in /home/gri6507/tmp/a.out)
==11790== Address 0x1fff000251 is on thread 1's stack
==11790== in frame #1, created by main (???:)
The right way to serialize a data structure into a chunk of bytes to be written to mass storage is to write a serializer that encodes the data into the desired format. Relying on things not specified by the standard (such as the endianness or type sizes) is simply wrong.
There are plenty of serialization libraries available.
What is the right way to initialize those padding bytes so that valgrind doesn't complain?
I'm not sure about 'the right way' to do this, but there are several possibilities.
you create members for the padding and initialize them
struct bar
{
size_t point_name_size;
char n[MAX];
bool q;
bool unused[7] ={};
};
This 'typically' has the same size as your struct on a 64 bit system. See here.
you fill yourself with 0's if you are trivially copyable
struct bar
{
bar()
{
static_assert(std::is_trivially_copyable_v<bar>);
memset(this, 0, sizeof(bar));
}
size_t point_name_size;
char n[MAX];
bool q;
};
I am currently working on a large project of my own on an STM32F7 cortex-m7 microcontroller in C++ using GCC. I need to store a wide array in an external SDRAM (16 MB) containing vectors of notes structures (12 bytes each). I have already a working FMC and a custom ram region created
/* Specify the memory areas */
MEMORY
{
RAM (xrw) : ORIGIN = 0x20000000, LENGTH = 512K
FLASH (rx) : ORIGIN = 0x08000000, LENGTH = 1024K
SDRAM (xrw) : ORIGIN = 0xC0000000, LENGTH = 16M
}
/* Define output sections */
SECTIONS
{
/* Section créée pour l'allocation dans la SDRAM externe*/
.fmc :
{
. = ALIGN(8);
*(.fmc)
*(.fmc*)
. = ALIGN(8);
} >SDRAM
my array is declared like this :
std::vector<SequencerNoteEvent> NotesVectorArray[kMaxPpqn] __attribute__((section(".fmc")));
So far it's OK. I created an array of vectors in my external RAM. How can i proceed in order to make my vectors element creation
NotesVectorArray[position].push_back(note);
happening in the same external RAM dynamically ? I am currently only able to declare static data using the __attribute__(section)
I read lot of things about C++ allocators, memory pools, but i don't get where the allocation take place in the vector code and how i should replace it... I "just" need to have the same allocation system than the usual but in another part of my memory for this precise type.
It seems possible to have multiple heaps. Where is located the connection between scatter file and the effective memory allocation ?
Thanks in advance,
Ben
I was facing the same issue and got a working solution by redefining operator new. The benefit of this over using a custom allocator is that all dynamic allocations will automatically be in SDRAM, so you're free to use std::function or any STL container that requires the heap, without having to pass a custom allocator argument every time.
I did the following:
Remove the standard library by adding -nostdlib to the compilation flags. I also added this to the linker flags. Also remove and --specs=... from your compiler and linker flags. Bonus: you'll save ~60-80k of code space!
Write your own replacement for operator new, operator delete. I created a new file called new.cpp added the following: (taken from https://arobenko.gitbooks.io/bare_metal_cpp/content/compiler_output/dyn_mem.html)
//new.cpp
#include <cstdlib>
#include <new>
void *operator new(size_t size) noexcept { return malloc(size); }
void operator delete(void *p) noexcept { free(p); }
void *operator new[](size_t size) noexcept { return operator new(size); }
void operator delete[](void *p) noexcept { operator delete(p); }
void *operator new(size_t size, std::nothrow_t) noexcept { return operator new(size); }
void operator delete(void *p, std::nothrow_t) noexcept { operator delete(p); }
void *operator new[](size_t size, std::nothrow_t) noexcept { return operator new(size); }
void operator delete[](void *p, std::nothrow_t) noexcept { operator delete(p); }
Define a variable in the linker script that corresponds to the lowest SDRAM address (start of the heap). This isn't strictly necessary, as you could use a constant (0xC0000000) in your code in step 4, but it makes things easier to keep the SDRAM address in just one place. Just add one line: PROVIDE( _fmc_start = . );
//linker script
[snip]
/* Section créée pour l'allocation dans la SDRAM externe*/
.fmc :
{
. = ALIGN(8);
PROVIDE( _fmc_start = . );
*(.fmc)
*(.fmc*)
. = ALIGN(8);
} >SDRAM
The new implementation of new will directly call malloc(), which will call _sbrk() when it needs a block of memory in the heap.
So you have to define _sbrk() to keep track of the heap pointer, and simply start the pointer at the lowest SDRAM address that you just defined in the linker script.
Here's what I did:
//sbrk.cpp, or append this to new.cpp
#include <cstdlib>
extern "C" size_t _sbrk(int incr)
{
extern char _fmc_start; // Defined by the linker
static char *heap_end;
char *prev_heap_end;
if (heap_end == 0)
heap_end = &_fmc_start;
prev_heap_end = heap_end;
//Optionally can check for out-of-memory error here
heap_end += incr;
return (size_t)prev_heap_end;
}
At this point, if you try to compile you will probably get many linker errors. The exact errors will depend on what parts of the standard library your project uses besides just new and delete. In my case, I was using std::function and the compiler complained that it didn't have __throw_bad_function_call(). You probably will also see an error for things like _init() and _fini() and __errno. The basic strategy is to create your own empty function stub or variable declaration for each of these. It's worth doing a quick search for each of these to see what purpose they serve, you might find out your project or a library you're including is using some features you weren't aware of! When you create a stub, you'll need to match the function signature correctly, so that will require searching the internet as well. Many of the stubs are for handling errors/exceptions, so you can decide how to handle it in the function body (or ignore errors).
For example, here's some info on the (obsolete, but required) _init() and _fini() functions: https://tldp.org/HOWTO/Program-Library-HOWTO/miscellaneous.html
Here's some info on __dso_handle: https://lists.debian.org/debian-gcc/2003/07/msg00057.html
__cxa_pure_virtual(): What is the purpose of __cxa_pure_virtual?
Here's what my project required to work:
//syscalls.cpp
extern "C" {
void _init(void) {}
void _fini(void) {}
int __errno;
void *__dso_handle = (void *)&__dso_handle;
}
namespace std
{
//These are error handlers. They could alternatively restart, or log an error
void __throw_bad_function_call() { while (1); }
void __cxa_pure_virtual() { while (1); }
} // namespace std
After all that, it will finally compile and if you use a debugger you'll see that the address of your vectors start at 0xC0000000!
Why does valgrind say I've got memory leak at the fclose() call?
#include <stdio.h>
class Stream
{
public:
Stream();
~Stream();
private:
char* pszOutput;
size_t size;
FILE* file;
};
Stream::Stream()
{
file = open_memstream(&pszOutput, &size);
}
Stream::~Stream()
{
fclose(file);
}
int main()
{
Stream s;
return 0;
}
Valgrind report:
==52387== 1 bytes in 1 blocks are definitely lost in loss record 1 of 1
==52387== at 0x4C28CCE: realloc (vg_replace_malloc.c:632)
==52387== by 0x5639CA3: _IO_mem_finish (memstream.c:132)
==52387== by 0x5635956: fclose##GLIBC_2.2.5 (iofclose.c:66)
Does it matter to initialize pszOutput or size? Or maybe I need to add something else?
From : http://linux.die.net/man/3/open_memstream
The open_memstream() function opens a stream for writing to a buffer. The buffer is dynamically allocated (as with malloc(3)), and automatically grows as required. After closing the stream, the caller should free(3) this buffer.
So according to this you need to free(pszOutput) after your file descriptor is closed.
So I know I can track memory allocation with methods of overloading new globally like so: http://www.almostinfinite.com/memtrack.html
However, I was wondering if there was a good way to do this per function so I can get a report of how much is allocated per function. Right now I can get file and lines and what the typeid is as in the link I provided but I would like to find which function is allocating the most.
What about doing something like: http://ideone.com/Wqjkrw
#include <iostream>
#include <cstring>
class MemTracker
{
private:
static char func_name[100];
static size_t current_size;
public:
MemTracker(const char* FuncName) {strcpy(&func_name[0], FuncName);}
static void inc(size_t amount) {current_size += amount;}
static void print() {std::cout<<func_name<<" allocated: "<<current_size<<" bytes.\n";}
static void reset() {current_size = 0; memset(&func_name[0], 0, sizeof(func_name)/sizeof(char));}
};
char MemTracker::func_name[100] = {0};
size_t MemTracker::current_size = 0;
void* operator new(size_t size)
{
MemTracker::inc(size);
return malloc(size);
}
void operator delete(void* ptr)
{
free(ptr);
}
void FuncOne()
{
MemTracker(__func__);
int* i = new int[100];
delete[] i;
i = new int[200];
delete[] i;
MemTracker::print();
MemTracker::reset();
}
void FuncTwo()
{
MemTracker(__func__);
char* c = new char[1024];
delete[] c;
c = new char[2048];
delete[] c;
MemTracker::print();
MemTracker::reset();
}
int main()
{
FuncOne();
FuncTwo();
FuncTwo();
FuncTwo();
return 0;
}
Prints:
FuncOne allocated: 1200 bytes.
FuncTwo allocated: 3072 bytes.
FuncTwo allocated: 3072 bytes.
FuncTwo allocated: 3072 bytes.
What platform are you using? There might be platform specific solutions without changing the functions in your code base.
If you are using Microsoft Visual Studio, you can use compiler switches /Gh and /GH to let the compiler call functions _penter and _pexit that you can define. In those functions, you can query how much memory the program is using. There should be enough information in there to figure out how much memory is allocated in each function.
Example code for checking memory usage is provided in this MSDN article.
I'm working on the class that simply contains a character array and it's size (length in bytes). At the moment, I want to overload '+' operand for this class (to implement concatenation). Constructors work fine. Objects are created and I can see their fields and values in the debugger. I'm stuck at the point where '+' is used (main(line 13)). Code compiles well, without even warnings, but as I run it, my program fails with "invalid pointer message". And I found where exactly that invalid pointer is. It is in '+' implementation (BufferArray.cpp, line 39). When i call SetBuffer, char array is assigned properly (I saw it's value 'qwasd' in the operator implementation scope), but right at the next line it's vanished when I call SetSize. I have no idea why.
What is wrong with my setters and how can I implement '+' operand in this case?
Thanks in advance.
Here's the code I work with:
BufferArray.h:
#include <string.h>
#include <stdio.h>
#ifndef BUFFERARRAY_H
#define BUFFERARRAY_H
class BufferArray {
public:
BufferArray(char* reservedPlace);
BufferArray();
void SetSize(int sz);
int GetSize();
void SetBuffer(char* buf);
char* GetBuffer();
BufferArray operator+ (BufferArray bArr) const;
virtual ~BufferArray();
private:
int size;
char *buffer;
};
#endif /* BUFFERARRAY_H */
Implementation is in the next file BufferArray.cpp:
#include "BufferArray.h"
// Constructors.
BufferArray::BufferArray(){
size = 0;
strcpy(buffer, "");
}
BufferArray::BufferArray(char* reservedPlace) {
size = strlen(reservedPlace);
buffer = reservedPlace;
}
// Getters and setters.
void BufferArray::SetSize(int sz)
{
size = sz;
}
int BufferArray::GetSize()
{
return size;
}
void BufferArray::SetBuffer(char* buf)
{
buffer = buf;
}
char* BufferArray::GetBuffer()
{
return buffer;
}
// Operator +.
BufferArray BufferArray::operator+ (BufferArray bArr) const
{
char tempCharArray[strlen(buffer) + strlen(bArr.GetBuffer())];
strcpy(tempCharArray, buffer);
strcat(tempCharArray, bArr.GetBuffer());
BufferArray tempBA;
tempBA.SetBuffer(tempCharArray);
tempBA.SetSize(strlen(bArr.GetBuffer()) + strlen(buffer)); // Vanishes buffer field.
printf("%d",tempBA.GetSize());
return tempBA;
}
// Destructor.
BufferArray::~BufferArray() {
// Destroy the pointer.
delete [] buffer;
}
And the main function:
#include <cstdlib>
#include <iostream>
#include "BufferArray.h"
using namespace std;
int main(int argc, char** argv) {
BufferArray ba1;
char tmp1[3] = "qw";
char tmp2[4] = "asd";
ba1.SetSize(strlen(tmp1));
ba1.SetBuffer(tmp1);
BufferArray ba2(tmp2);
BufferArray ba3 = ba1 + ba2; // Runtime error is here.
cout << ba3.GetBuffer() << endl;
return 0;
}
in BufferArray::operator+, tempCharArray is a temporary buffer that will be destroyed when the function completes. There a basically two ways to handle this:
1/ allocate the temporary buffer with new[] in operator+, that way you'll make sure the buffer survives the call to operator+ but you'll either have a memory leak or require the caller to invoke delete[] later on, which is rather clumsy and error-prone
2/ or better yet, modify setBuffer so it does an internal copy of the buffer and add a call to delete[] in your own destructor :
BufferArray::~BufferArray() {
delete[] buffer;
}
void BufferArray::setBuffer(char *otherBuffer) {
buffer = new char[strlen(otherBuffer) + 1];
strcpy(buffer, otherBuffer);
}
Note that you'll have to modify the constructor so it also copies the input buffer (otherwise you'll have an illegal call to delete[] when the object is destroyed), and then you may want to overload the copy-constructor and assignment operator to prevent shallow copy which would result in double-deleting the buffer.
In actual production code, you'd want to use a managed pointer of some sort to avoid doing the delete yourself (e.g. std::vector or boost::shared_array), but for homework the above solution should be fine.
On a side note, don't forget to add +1 when using strlen to determine the size of your buffer ;)
You need to use new to create these char arrays, otherwise the temporaries (like tempBA) are destroyed when you exit the scope.
void BufferArray::SetBuffer(char* buf)
{
buffer = new char[strlen(buf)+1]; //edit, my size param wasn't necessary
strcpy(buffer,buf);
}