Debug assertion failure with stringstream::imbue and custom global operator new - c++

I apologize for this rather localized question, but I was hoping to get someone else's view on it to make sure I'm not doing something obviously wrong.
I believe I've run into a bug in either the Visual C++ Runtime library, or somewhere in Microsoft's implementation of std::stringstream. The issue only manifests itself with a combination of when:
imbue() is called to change the locale on a stringstream, and
A custom global operator new is used which returns a pointer offset from the base address returned by malloc() used to allocate the block.
I've been able to reproduce this with the following minimal test case:
#include <sstream>
static void *localMalloc(size_t bytes)
{
unsigned char *ptr = static_cast<unsigned char *>( malloc(bytes + 64) );
ptr += 64;
printf("malloc of %d bytes: %ph\n", bytes, ptr);
return ptr;
}
void *operator new(size_t bytes) { return localMalloc(bytes); }
void *operator new[](size_t bytes) { return localMalloc(bytes); }
void operator delete(void *ptr) throw() { /* do nothing */ }
void operator delete[](void *ptr) throw() { /* do nothing */ }
struct DecimalSeparator : std::numpunct<char>
{
char do_decimal_point() const
{
return '.';
}
};
int main()
{
std::stringstream ss;
ss.imbue(std::locale(std::locale(), new DecimalSeparator));
ss << 5; // <-- is_block_type_valid(header->_block_use) assertion failure here
return 0;
}
If either:
The ptr += 64; line in localMalloc() or
The ss.imbue() call in main()
are commented out, the code works as expected, and the assertion does not happen.
I have attempted to step into the code using the debugger as much as I possibly can, but I am currently unable to pinpoint the location at where the code fails in the STL as Visual Studio dumps me into raw disassembly mode after stepping out of basic_stringbuf::overflow() making debugging nearly impossible. As far as I can tell, I haven't seen any invalid memory writes outside of the memory being allocated, so I'm not completely sure where the CRT is checking the heap or why it thinks the pointer is invalid.
Note that operator delete is intentionally ignoring the free in this test case for brevity. It makes no difference if free() is properly called on the memory block or not.
So far, tests on the following compilers and platforms have resulted in:
VS2015 Update 2: fails
VS2015 Update 3: fails
VS2015 Update 3 + KB3165756: fails
clang-703.0.31 on OSX: OK
Does anyone see anything odd here that I missed?

This is more than likely not a bug, but instead your version of delete is not being called, and instead the Visual Studio's debug runtime library's version of the global delete is called. Having two or more versions of the global delete operator in the same program is undefined behavior.
From this reference (Global replacements), the behavior is stated to be undefined when this occurs.
From the C++ ISO standard:
3.7.4 Dynamic storage duration [basic.stc.dynamic]
//...
§2 The library provides default definitions for the global allocation
and deallocation functions. Some global allocation and deallocation
functions are replaceable (18.6.1). A C++ program shall provide at
most one definition of a replaceable allocation or deallocation
function.
Running with Visual Studio 2015 using the release version Visual Studio runtime library does not produce this error, and the replacement global delete does in fact get called.
Visual Studio 2015 online compiler results.

Related

What is the best memory management for GCC C++? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 months ago.
Improve this question
What's the problem of my memory management? Because it causes crash that I show in a comment in the code below ("A memory block could not be found when trying to free."). I know that my memory management is not thread safe because I use global variables g_numBlocks and g_blocks that can cause risk when using multiple threads.
Since my memory management code seems too complex, can anyone suggest a stable and better "Memory Management for C++" to avoid memory leaks.
The code that contains bug
#include "emc-memory.h" // <-- Declare the functions MALLOC() and FREE() from other library.
#include <vector>
int main() {
printf("HERE(1)\n");
{
std::vector<string> paths = { // <-- Problem, 'std::vector' & 'string' use internal malloc/free & operator new/delete that are overwritten with my own custom memory management.
"/foo/bar.txt",
"/foo/bar.",
"/foo/bar",
"/foo/bar.txt/bar.cc",
"/foo/bar.txt/bar.",
"/foo/bar.txt/bar",
"/foo/.",
"/foo/..",
"/foo/.hidden",
"/foo/..bar",
};
} // <-- It crashes here, error in FREE(): "A memory block could not be found when trying to free.".
printf("HERE(2)\n"); // The reason I know it crashes above is this line is not evaluated, only "HERE(1)" is printed. I'm using [RelWithDebInfo] with blurry debugging info.
return 0;
}
Compilers:
[Visual Studio 2015] [Debug]: No problem.
[Visual Studio 2015] [RelWithDebInfo]: No problem.
[GCC 12.1.0 x86_64-w64-mingw32] [Debug]: No problem.
[GCC 12.1.0 x86_64-w64-mingw32] [RelWithDebInfo]: Broken which means there's a bug.
In "emc-memory.h" in other library .so file
extern const char* __file;
extern int __line;
#define new (__file = __FILE__, __line = __LINE__, 0) ? 0 : new
enum MEMORYBLOCKTYPE {
MEMORYBLOCKTYPE_MALLOC,
MEMORYBLOCKTYPE_NEW,
};
void *MALLOC(size_t size, MEMORYBLOCKTYPE type);
void *REALLOC(void *block, size_t newSize);
void FREE(void *block, MEMORYBLOCKTYPE type);
#define malloc(size) ((__file = __FILE__, __line = __LINE__, 0) ? 0 : MALLOC(size, MEMORYBLOCKTYPE_MALLOC))
#define realloc(block, newSize) REALLOC(block, newSize)
#define free(block) FREE(block, MEMORYBLOCKTYPE_MALLOC)
In "emc-memory.cpp" in other library .so file
I use this code in a link to override the operator new & delete: https://codereview.stackexchange.com/questions/7216/custom-operator-new-and-operator-delete
typedef unsigned long long BlockId; // The reason it's 64-bit is a memory block can be freed and reallocated multiple times, which means that there can be a lot of ids.
BlockId g_blockId = 0;
BlockId newBlockId() {
return g_blockId++;
}
struct Block {
const char *file;
int line;
const char *scope;
char *hint;
size_t size;
BlockId id; // That id is used for comparison because it will never be changed but the block pointer can be changed.
void *block;
MEMORYBLOCKTYPE type;
};
bool g_blocks_initialized = false;
int g_numBlocks;
Block **g_blocks;
void *MALLOC(size_t size, MEMORYBLOCKTYPE type) {
if (g_blocks_initialized == false) {
g_blocks_initialized = true;
_initializeList(g_numBlocks, g_blocks);
}
Block *b = (Block *)malloc(sizeof(*b));
b->file = __file ; __file = nullptr;
b->line = __line ; __line = 0;
b->scope = __scope; __scope = nullptr;
b->hint = allocateMemoryHint(__hint);
b->size = size;
b->id = newBlockId();
b->block = malloc(size);
b->type = type;
_addListItem(g_numBlocks, g_blocks, b);
return b->block;
}
void FREE(void *block, MEMORYBLOCKTYPE type) {
if (block == nullptr) {
return; // 'free' can free a nullptr.
}
for (int i = 0; i < g_numBlocks; i++) {
Block *b = g_blocks[i];
if (b->block == block) {
if (b->type != type) {
switch (type) {
case MEMORYBLOCKTYPE_MALLOC: EMC_ERROR("The memory block type must be MALLOC."); break;
case MEMORYBLOCKTYPE_NEW: EMC_ERROR("The memory block type must be NEW."); break;
default: EMC_ERROR("Error"); break;
}
}
_removeListItem(g_numBlocks, g_blocks, b);
freeMemoryHint(b->hint); b->hint = nullptr;
SAFE_FREE(b->block);
SAFE_FREE(b);
return;
}
}
EMC_ERROR("A memory block could not be found when trying to free.\n\nExamples:\n - Calling free(pointer) where pointer was not set to zero after it's been called twice, the solution was to use SAFE_FREE(). And if possible, replace any free() with SAFE_FREE(). For example, see Lexer::read0() on the original line \"free(out.asIdentifier);\".\n - If an 'Engine' object is destroyed before destroying a Vulkan object then it can cause this error (It can happen with 'Release' or 'RelWithDebInfo' configuration but not with 'Debug' configuration), that problem happened to me and I stuck there for hours until I realized it.");
}
I would humbly suggest that without a very clear reason to think otherwise the best memory management for GCC C++ is the out-of-the-box default memory management for GCC C++.
That would mean your best solution would have been to do nothing or as it is now strip out your overrides of the global operators.
You may find in some area of a system the default memory management is sub-optimal but in 2022 the default options are very effective and if you find a general purpose strategy that is better it's a publishable paper.
However your question tells us nothing about the application in question or your motivations for thinking you should even try to change the memory management let alone give advice on what to.
Sure you can add a global allocation mutex to block memory management and make it thread-safe. I will be surprised if that doesn't turn out to more than throw away whatever advantage you're hoping to gain.
Not sure where SAFE_FREE is coming from.
If you see in MALLOC function, they use c runtime malloc. Meaning that if you want to free the block you need to use the corresponding free() function.
Make sure that SAFE_FREE is indeed using the c runtime free with the correct parameters.

Dynamic vector allocation in external RAM

I am currently working on a large project of my own on an STM32F7 cortex-m7 microcontroller in C++ using GCC. I need to store a wide array in an external SDRAM (16 MB) containing vectors of notes structures (12 bytes each). I have already a working FMC and a custom ram region created
/* Specify the memory areas */
MEMORY
{
RAM (xrw) : ORIGIN = 0x20000000, LENGTH = 512K
FLASH (rx) : ORIGIN = 0x08000000, LENGTH = 1024K
SDRAM (xrw) : ORIGIN = 0xC0000000, LENGTH = 16M
}
/* Define output sections */
SECTIONS
{
/* Section créée pour l'allocation dans la SDRAM externe*/
.fmc :
{
. = ALIGN(8);
*(.fmc)
*(.fmc*)
. = ALIGN(8);
} >SDRAM
my array is declared like this :
std::vector<SequencerNoteEvent> NotesVectorArray[kMaxPpqn] __attribute__((section(".fmc")));
So far it's OK. I created an array of vectors in my external RAM. How can i proceed in order to make my vectors element creation
NotesVectorArray[position].push_back(note);
happening in the same external RAM dynamically ? I am currently only able to declare static data using the __attribute__(section)
I read lot of things about C++ allocators, memory pools, but i don't get where the allocation take place in the vector code and how i should replace it... I "just" need to have the same allocation system than the usual but in another part of my memory for this precise type.
It seems possible to have multiple heaps. Where is located the connection between scatter file and the effective memory allocation ?
Thanks in advance,
Ben
I was facing the same issue and got a working solution by redefining operator new. The benefit of this over using a custom allocator is that all dynamic allocations will automatically be in SDRAM, so you're free to use std::function or any STL container that requires the heap, without having to pass a custom allocator argument every time.
I did the following:
Remove the standard library by adding -nostdlib to the compilation flags. I also added this to the linker flags. Also remove and --specs=... from your compiler and linker flags. Bonus: you'll save ~60-80k of code space!
Write your own replacement for operator new, operator delete. I created a new file called new.cpp added the following: (taken from https://arobenko.gitbooks.io/bare_metal_cpp/content/compiler_output/dyn_mem.html)
//new.cpp
#include <cstdlib>
#include <new>
void *operator new(size_t size) noexcept { return malloc(size); }
void operator delete(void *p) noexcept { free(p); }
void *operator new[](size_t size) noexcept { return operator new(size); }
void operator delete[](void *p) noexcept { operator delete(p); }
void *operator new(size_t size, std::nothrow_t) noexcept { return operator new(size); }
void operator delete(void *p, std::nothrow_t) noexcept { operator delete(p); }
void *operator new[](size_t size, std::nothrow_t) noexcept { return operator new(size); }
void operator delete[](void *p, std::nothrow_t) noexcept { operator delete(p); }
Define a variable in the linker script that corresponds to the lowest SDRAM address (start of the heap). This isn't strictly necessary, as you could use a constant (0xC0000000) in your code in step 4, but it makes things easier to keep the SDRAM address in just one place. Just add one line: PROVIDE( _fmc_start = . );
//linker script
[snip]
/* Section créée pour l'allocation dans la SDRAM externe*/
.fmc :
{
. = ALIGN(8);
PROVIDE( _fmc_start = . );
*(.fmc)
*(.fmc*)
. = ALIGN(8);
} >SDRAM
The new implementation of new will directly call malloc(), which will call _sbrk() when it needs a block of memory in the heap.
So you have to define _sbrk() to keep track of the heap pointer, and simply start the pointer at the lowest SDRAM address that you just defined in the linker script.
Here's what I did:
//sbrk.cpp, or append this to new.cpp
#include <cstdlib>
extern "C" size_t _sbrk(int incr)
{
extern char _fmc_start; // Defined by the linker
static char *heap_end;
char *prev_heap_end;
if (heap_end == 0)
heap_end = &_fmc_start;
prev_heap_end = heap_end;
//Optionally can check for out-of-memory error here
heap_end += incr;
return (size_t)prev_heap_end;
}
At this point, if you try to compile you will probably get many linker errors. The exact errors will depend on what parts of the standard library your project uses besides just new and delete. In my case, I was using std::function and the compiler complained that it didn't have __throw_bad_function_call(). You probably will also see an error for things like _init() and _fini() and __errno. The basic strategy is to create your own empty function stub or variable declaration for each of these. It's worth doing a quick search for each of these to see what purpose they serve, you might find out your project or a library you're including is using some features you weren't aware of! When you create a stub, you'll need to match the function signature correctly, so that will require searching the internet as well. Many of the stubs are for handling errors/exceptions, so you can decide how to handle it in the function body (or ignore errors).
For example, here's some info on the (obsolete, but required) _init() and _fini() functions: https://tldp.org/HOWTO/Program-Library-HOWTO/miscellaneous.html
Here's some info on __dso_handle: https://lists.debian.org/debian-gcc/2003/07/msg00057.html
__cxa_pure_virtual(): What is the purpose of __cxa_pure_virtual?
Here's what my project required to work:
//syscalls.cpp
extern "C" {
void _init(void) {}
void _fini(void) {}
int __errno;
void *__dso_handle = (void *)&__dso_handle;
}
namespace std
{
//These are error handlers. They could alternatively restart, or log an error
void __throw_bad_function_call() { while (1); }
void __cxa_pure_virtual() { while (1); }
} // namespace std
After all that, it will finally compile and if you use a debugger you'll see that the address of your vectors start at 0xC0000000!

Returning reference to local variable without trivial conversion

I am quite a newbie to the C++ programming, but this question keeps on spinning in my head. I understand that returning reference to a local variable in a function is illegal, i.e. compiling this code snippet:
inline int& funref() {
int a = 8;
return a; // not OK!
}
results in a warning from the compiler and then a runtime error. But then, why does this piece of code get compiled without any warnings and run without error?
inline int& funref() {
int a = 8;
int& refa = a;
return refa; // OK!
}
int main() {
int& refa = funref();
cout << refa;
}
My compiler is g++ on Linux Fedora platform.
It's still wrong, it just happens to be working by (un)happy coincidence.
This code has undefined behaviour with all the usual caveats (it might always work, it might always work until it's too late to fix, it might set fire to your house and run away with your betrothed).
The compiler isn't required to issue a diagnostic (warning or error message) for every possible mistake, just because it isn't always possible to do so. Here, at least your current version of g++ hasn't warned. A different compiler, or a different version of g++, or even the same version with different flags, might warn you.
The reason why you can't return a reference to a local variable is because the local variable will get wiped when your function returns. Simply put, the compiler prevents you from referencing garbage data.
However, the compiler isn't bulletproof (as shown in your example #2).
It does work for retrieving a singleton instance, though.
inline int& funref()
{
static int* p_a = nullptr;
if (nullptr == p_a)
p_a = new int(8);
return *p_a;
}
this case is valid because the memory pointed by p_a remains valid after the function returns.

destruction of std::stringbuf causing access violation

I have a c++ class that roughly looks like this (see code below). It has an input buffer and an output buffer using a std::stringbuf. Because I also want access to the raw buffer, I set the underlying buffer with my own user-defined byte array using std::stringbuf.pubsetbuf().
All things seem to work fine, until I no longer need the object but when the object is destructed, it causes my program to crash with an access violation.
I traced it down to the piece of code as illustrated below. It looks to me that the stringbuf is somehow cleaning up my user-defined byte array itself?? My program doesn't crash when I remove the code in the destrcutor of my class to free the byte array I previously allocated in the constructor.
Any advice on this, please? Am I wrongly using the std::stringbuf? I'm using the Borland 5.0 compiler (I know it's a very old and outdated compiler, but I have to stick to this compiler for a while).
class SomeClass {
private:
char *mIBuf;
char *mOBuf;
std::stringbuf mIBufStream;
std::stringbuf mOBufStream;
public:
SomeClass(int iBufSize, int oBufSize) :
mIBuf(), mOBuf(),
mIBufStream(), mOBufStream()
{
mIBuf = (char*)malloc(iBufSize);
mOBuf = (char*)malloc(oBufSize);
mIBufStream.pubsetbuf(mIBuf, iBufSize);
mIBufStream.pubseekpos(0);
mOBufStream.pubsetbuf(mOBuf, oBufSize);
mOBufStream.pubseekpos(0);
}
virtual ~SomeClass()
{
free(mIBuf);
free(mOBuf)
}
};
According to the Standard stringbuf doesn't have a destructor of its own and streambuf's destructor does nothing. Your old compiler and library may or may not be following that; the evidence is that it isn't.
Well, in principle you are doing something wrong. When you call pubsetbuf you are giving that object permission to use that buffer for as long as the object lives, or until you change its buffer again.
Looking at your destructor, you aren't keeping your side of the bargain.
virtual ~SomeClass()
{
free(mIBuf);
free(mOBuf); // <- missing semicolon in your code
// the stringbuf objects are still alive here
} // they get automatically destroyed here
One option is to arrange for the stringbuf objects to be destroyed before you free the buffers (char* buffer deallocation will need to be done by a helper class, either a base class or a member declared before the stringbuf -- std::vector<char> would be a good choice).
Or, you can let the stringbuf know that you're revoking its permission to use your memory:
virtual ~SomeClass()
{
mIBufStream.pubsetbuf(0, 0);
mOBufStream.pubsetbuf(0, 0);
// the stringbufs cannot use your memory any longer
free(mIBuf);
free(mOBuf);
}
Well, to answer my own question... (and thanks to Ben Voight for his useful advise). Until someone else would prove me wrong, I've come to the conclusion that my issue is implementation specific and that the Borland 5.0 STL library may not be conform the standard.
I tested my sample class. I extracted it and put it in a small test program as listed below (no other code that would possibly corrupt the memory). I compiled 1 version with Visual Studio 2013, the other with Borland 5.0. The one compiled with Visual Studio runs just fine. The one compiled with Borland crashes after the 1st iteration.
#include <iostream>
include
class SomeClass {
char *mIBuf;
char *mOBuf;
std::stringbuf mIBufStream;
std::stringbuf mOBufStream;
public:
SomeClass(int iBufSize, int oBufSize) : mIBuf(NULL), mOBuf(NULL), mIBufStream(), mOBufStream() {
mIBuf = (char*)malloc(iBufSize);
mOBuf = (char*)malloc(oBufSize);
mIBufStream.pubsetbuf(mIBuf, iBufSize);
mIBufStream.pubseekpos(0);
mOBufStream.pubsetbuf(mOBuf, oBufSize);
mOBufStream.pubseekpos(0);
}
virtual ~SomeClass() {
mIBufStream.pubsetbuf(0, 0);
mOBufStream.pubsetbuf(0, 0);
free(mIBuf);
free(mOBuf);
}
};
int main(int argc, char* argv[])
{
for (int x = 0; x < 1000; x++) {
SomeClass SomeClass(128 * 1024, 128 * 1024);
std::cout << "Pass " << x + 1 << std::endl;
}
return 0;
}

Stack unwinding in case of structured exceptions

This question provides more clarity on the problem described here. I did some more investigation and found that the stack unwinding is not happening in the following piece of code:
class One
{
public:
int x ;
};
class Wrapper
{
public:
Wrapper(CString csText):mcsText(csText)
{
CString csTempText;
csTempText.Format("Wrapper constructor :: %s\n", mcsText);
OutputDebugString(csTempText);
}
~Wrapper()
{
CString csTempText;
csTempText.Format("Wrapper destructor :: %s\n", mcsText);
OutputDebugString(csTempText);
}
CString mcsText;
};
class Test
{
public:
void notifyError()
{
try
{
int x = 10;
}
catch(...) {}
}
void OnRecvBuffer()
{
try
{
Wrapper a("AddRef");
One* p = NULL;
p->x = 10;
}
catch(...)
{
notifyError();
}
}
};
int main()
{
Test* pTest = new Test;
pTest->OnRecvBuffer();
OutputDebugString("Test");
}
I compiled this code using VC6 SP5 compiler and the output is "Wrapper constructor :: AddRef!!!" (i.e. the destructor of wrapper object which was constructed on stack is not called. Is this the expected behavior ? or is it a bug with VC compiler ? Can I use some compiler flags so that the stack unwinding happens in this case?
The C++ standard does not give anything to work with in case of Undefined Behavior. Even if MS does. That's a platform specific thing -- so beware. Some such floating point exceptions are turned to Win32 exceptions which you can attempt to catch with _set_se_translator(). The problem is you can catch Win32 exceptions but then your stack will not be unwound properly. At least that is not something you can bet your life on. Wherein lies the futility of the exercise.
Update: The exception is thrown intentionally to check the stack unwinding. The question is why Wrapper class's destructor is not getting called. – Naveen
If this is the case -- don't do it. There are better ways to throw exceptions than via Undefined Behavior.
E.g:
void OnRecvBuffer()
{
try
{
Wrapper a("AddRef");
throw 42; // this'll throw an exception without invoking UB
}
catch(...)
{
notifyError();
}
}
You cannot dereference a NULL pointer. You are invoking Undefined Behavior here:
One* p = NULL;
p->x = 10;
After that line all bets are off and you could have killed us all ;)
p is a pointer to a One object. It should contain the address of an One object. You have initialized it to 0 -- there is no object at address 0. 0 is not a valid address for any object (this is guranteed by the standard).
If you want to use SEH, you must use _set_se_translator function and /EHa compiler option.
Because C++ regular exception does not handle this kind of exception, you have to use SEH which does not know anything about the app and does not unwind.
This is undefined behavior:
One* p = NULL;
p->x = 10;
At this point the application is free to crash without unwinding the stack.
If you want to test the stack unwinding replace this with:
throw 42; // Life the Universe and Everything thrown away
You should not dynamically allocate all your objcts this is C++ not Java!
int main()
{
Test pTest; // Note the lack of new!
pTest.OnRecvBuffer();
OutputDebugString("Test");
}