We have a relatively large code base for a Linux server, it's dynamically linked-in libraries and server modules loaded during startup using dlopen(). The server as well as most of the other components are written in C++11, but some are in C99.
What approaches could one use to test whether the server, its dependencies and modules properly handle memory allocation failures, e.g.malloc/calloc returning NULL, operators new and new[] throwing std::bad_alloc etc, including allocation failures from std::string::resize() and such?
In the past, I've tried using memory allocation hooks to inject memory allocation failures into C applications, but I think these don't work for C++. What other options or approaches should I be looking at?
In fact, hooking into C malloc is enough, since under the hood the gcc C++ default implementation of operator new does call malloc and you confirmed you only needed a gcc compatible solution.
I could demonstrate it with that simple program:
mem.c++:
#include <iostream>
#include <string>
class A {
int ival;
std::string str;
public:
A(int i, std::string s): ival(i), str(s) {}
A(): ival(0), str("") {};
int getIval() const {
return ival;
}
std::string getStr() const {
return str;
}
};
int main() {
A a(2, "foo");
std::cout << &a << " : " << a.getIval() << " - " << a.getStr() << std::endl;
return 0;
}
memhook.c:
#include <stdio.h>
#include <stdlib.h>
extern void *__libc_malloc(size_t size);
void* malloc (size_t size) {
fprintf(stderr, "Allocating %u\n", size);
return NULL;
// return __libc_malloc(size);
}
When returning NULL (as above), the program displays:
Allocating 16
Allocating 100
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Abandon (core dumped)
That proves that returning NULL from the declared malloc function results in a std::bad_alloc exception in C++ code
When uncommenting the return __libc_malloc(size); the allocations are done by the libc malloc and the output becomes:
Allocating 16
0xbfe8d2e8 : 2 - foo
On linux you can hook into the operating system to force allocation to fail
man 2 mlockall
mlockall(MCL_CURRENT|MCL_FUTURE);
Should do what you want.
Related
Is calling this function creating a memory leak?
#include <iostream>
std::string somefunc()
{
std::string somestrng;
somestrng = "Fred";
return somestrng;
}
int main()
{
std::cout << "Hello World!\n";
std::string receiver = somefunc();
std::cout << "-->" << receiver.data() << "<--" << std::endl;
}
I've read about "value semantics" but I can't envision it.
No, there is no memory leak in the shown program.
In general, memory leaks happen when you allocate dynamic memory, and neglect to deallocate that memory.
You don't directly allocate any dynamic memory in the example program. std::string may potentially allocate some dynamic memory, but it will also deallocate it. For future learning, I would recommend studying the RAII pattern, which the string class follows.
I have currently a memory issue using the Botan library (version 2.15) for cryptography functions within a C++ project. My development environment is Solus Linux 4.1 (kernel-current), but I could observe this issue on Debian Buster too.
I observed that some memory allocated internally by Botan for calculations is not deallocated when going out of scope. When I called Botan::HashFunction, Botan::StreamCipher and Botan::scrypt multiple times, always going out of scope in between, the memory footprint increases steadily.
For example, consider this code:
#include <iostream>
#include <vector>
#include "botan/scrypt.h"
void pause() {
char ch;
std::cout << "Insert any key to proceed... ";
std::cin >> ch;
}
std::vector<uint8_t> get_scrypt_passhash(std::string const& password, std::string const& name) {
std::vector<uint8_t> key (32);
Botan::scrypt(key.data(), key.size(), password.c_str(), password.length(), salt.c_str(), salt.length(), 65536, 32, 1);
std::cout << "From function: before closing.\n";
pause();
return key;
}
int main(int argc, char *argv[]) {
std::cout << "Beginning test.\n";
pause();
auto pwhashed = get_scrypt_passhash(argv[1], argv[2]);
std::cout << "Test ended.\n";
pause();
}
I used the pause() function to observe the memory consumption (I called top/pmap and observed KSysGuard during the pause), when it is called from within get_scrypt_passhash before terminating, the used memory (both by top/pmap and KSysGuard) is about 2 MB more than at beginning, and after terminating the same.
I tried to dive into the Botan source code, but I cannot find memory leaks or the like. Valgrind also outputted that all allocated bytes have been freed, so no memory leaks were possible.
Just for information, I tried the same functionality with Crypto++ without observing this behavior.
Has anyone experienced the same issue? Is there a way to fix it?
I am trying to make code that throws a length_error exception. My goal is to detect and handle this exception condition. This is my attempt so far:
#include <iostream>
#include <string>
using namespace std;
int main(int argc, const char * argv[])
{
string buffer("hi");
cout<<buffer.max_size()<<endl;
try {
while(1) {
buffer.append(buffer);
}
}
catch(length_error &l) {
cout<<"Caught length_error exception: "<<l.what()<<endl;
}
catch(exception &e) {
cout<<"Caught exception: "<<e.what()<<endl;
}
return 0;
}
When I run the program I see the max size of the string is 18446744073709551599 bytes. The program continues running until all the memory is used up. Then it just goes quiet. No exception is thrown. The program is still running but its CPU usage went from 100% to around 0%.
Additional information:
OS: Mac OS 10.8.
Compiler: Clang 5.1
RAM: 16 GB
I believe your machine is going into virtual memory thrashing due to the memory consumption of growing your string by two characters a LOT of times.
A more effective way of getting this exception is to create a string of size max_size()+1 at the outset. Here's your code modified to do this, and (for me, at least) it throws the exception you expect instantly:
#include <iostream>
#include <string>
using namespace std;
int main(int argc, const char * argv[])
{
string buffer("hi");
cout<<buffer.max_size()<<endl;
try {
std::string blah(buffer.max_size()+1, 'X');
}
catch(length_error &l) {
cout<<"Caught length_error exception: "<<l.what()<<endl;
}
catch(exception &e) {
cout<<"Caught exception: "<<e.what()<<endl;
}
return 0;
}
std::length_error is only thrown in a case where a buffer size is known to exceed the container's max_size().
On a modern operating system with virtual memory, it's unlikely to request something that exceeds max_size() unless by accident (such as through negative unsigned values), and so the approach you are taking is unlikely to see this exception thrown. Instead, since you're using append, you're likely going to just use virtual memory by paging out to disk after you have exhausted your real memory -- which will slow the system down.
If you want to trigger a length_error, the easiest way would be to pass something greater than max_size() (assuming this is smaller than std::numeric_limits<std::string::size_type>::max()). You should be able to just do something like:
auto str = std::string{};
str.reserve(static_cast<std::string::size_type>(-1));
or
auto str = std::string{};
str.reserve(str.max_size()+1);
From this link, I know that we(application) should always don't delete a heap memory allocation from dll, because the heap memory manager are different.
I have few questions:
1.) How about .so file (Linux), is it the same case ?
2.) Is there anyway to make sure that application and library(.dll and .so) are using same heap memory manager or using same heap memory section ? So we can delete and new separately (new at .dll/.so, delete at application ).
Thank you.
1.) How about .so file (Linux), is it the same case ?
Yes, a library built using a different implementation of the Standard C++ Library than the program it's finally linked with may do allocations in a slightly different manner. g++'s libstdc++ and clang++'s libc++ are examples of two different implementations. They may be 100% ABI compatible - but a third unknown library may not be.
2.) Is there anyway to make sure that application and library(.dll and .so) are using same heap memory manager or using same heap memory section ? So we can delete and new separately (new at .dll/.so, delete at application ).
No, what's compiled into the library is what will be used by the library, unless there's a way to initialize the library upon loading it, telling it to use a specific heap manager.
Please explain in details. I wish to know for .so (Linux), is it using only one heap manager for both application and .so (library). Let's said, my application compiled by compiler version A, while my .so complied by compiler version B, is it still ok ?
Because of the reason mentioned above, no, you can't be sure. Since you are the library creator you could however make your API so that the new and delete memory allocations/deallocations for types in your library are delegated to member functions compiled into the library, which in turn does the real allocations/deallocations (described in operator new, operator new[]
and operator delete, operator delete[]). Pointers to objects of your types could then be newed and passed between the library and application and be deleted on either side.
Here's an (incomplete) example of how that could look using a class-specific allocation function:
void* T::operator new(std::size_t count);
and a class-specific usual deallocation function:
void T::operator delete(void* ptr);
It contains foo.hpp and foo.cpp used to create libfoo.so (or libfoo.a) and code for a program using the library.
foo.hpp
#pragma once
#include <new>
class Foo {
public:
// The "usual" part of your class definition:
Foo(int x);
~Foo();
// This part does NOT get compiled into your library.
// It'll only be used by users of your library:
#ifndef BUILDING_LIB
// Note: operator new and delete are static by default
// single object allocation/deallocation:
void* operator new(std::size_t /* byte_count */) { return Alloc(); }
void operator delete(void* addr) { Free(addr); }
// array allocation/deallocation:
// TODO: operator new[] and delete[]
#endif
private:
int value;
// the functions really doing the memory management
static void* Alloc();
static void Free(void* p);
};
foo.cpp
// Define BUILDING_LIB to disable the proxy operator new/delete functions when building
// the library.
#define BUILDING_LIB
#include "foo.hpp"
#include <cstdlib> // std::aligned_alloc
#include <iostream>
Foo::Foo(int x) : value(x) {
std::cout << "Foo:Foo(" << value << ")\n";
}
Foo::~Foo() {
std::cout << "Foo:~Foo() " << value << "\n";
}
void* Foo::Alloc() {
void* addr = std::aligned_alloc(alignof(Foo), sizeof(Foo));
std::cout << "Alloc() " << sizeof(Foo) << "\t# " << addr << '\n';
return addr;
}
void Foo::Free(void* addr) {
std::cout << "Free()\t\t# " << addr << '\n';
std::free(addr);
}
uselib.cpp
#include "foo.hpp"
#include <memory>
int main() {
auto a = std::make_unique<Foo>(123); // heap allocation
// An automatic variable will use the applications memory manager and will not
// use Alloc/Free.
Foo b(456);
}
Possible output:
Alloc() 4 # 0x1af7eb0
Foo:Foo(123)
Foo:Foo(456)
Foo:~Foo() 456
Foo:~Foo() 123
Free() # 0x1af7eb0
1) Symbol tables are shared across an entire process in Linux. malloc() for any part of the
process is the same as all the other parts. So yes, if all parts of a process access the
heap via malloc() et alia then they will share the same heap.
2) but second question i too have bit confusion.
The heap manager is located in static memory of the runtime library (msvcrXXX for Win and crt for Linux). You can use the runtime library in two ways: as static or dynamic. If you link the runtime as static to your own library, then you can't alloc in your library and free in another. But you link the runtime as dynamic, then you can alloc in your library and free in another.
Library developers take care about this by implementing they own methods for freeing memory allocated inside of library.
For example:
OpenIPMI calls ipmi_fru_node_get_field and ipmi_fru_data_free.
CryptoAPI calls CertCreateCertificateContext and CertFreeCertificateContext
I am maintaining some legacy code which was missing a copy assignment constructor in a managed Aligned Pointer class. I added one as follows (simplified view):
#include <iostream>
#include <cstring>
#include <stdlib.h>
template <class T, unsigned AlignB>
class AlignedPtr {
private:
T *mpBlock;
unsigned mBlkSize;
public:
// Size specific Ctor
AlignedPtr(unsigned uNum) :
mpBlock(static_cast<T*> (aligned_alloc(uNum*sizeof(T), AlignB))),
mBlkSize(uNum) {}
// Default, empty Ctor
AlignedPtr(void) : mpBlock(nullptr), mBlkSize(0) {}
// Copy Assignment Ctor
AlignedPtr& operator=(const AlignedPtr& x)
{
T *mpNewBlock(static_cast<T*>(aligned_alloc(x.mBlkSize*sizeof(T), AlignB)));
for (size_t index=0; index < x.mBlkSize; index++) {
mpNewBlock[index] = x.mpBlock[index];
}
free(mpBlock);
mpBlock = mpNewBlock;
mBlkSize = x.mBlkSize;
return *this;
}
// Destroy managed pointer
~AlignedPtr() {
free(mpBlock);
}
};
int main(int argc, char *argv[])
{
AlignedPtr<float, 16> first_ptr;
std::cout << "Pointer Initialized" << std::endl;
first_ptr = AlignedPtr<float, 16>(8);
std::cout << "Pointer Re-initialized" << std::endl;
return 0;
}
My expectation was that the program will terminate normally, however I saw AlignedPtr Dtor fail when first_ptr goes out of scope (main termination).
I compile above without any optimizations as:
g++ -std=c++11 -g aligned_alloc_bug.cpp -o aab
On a Ubuntu 14.04 with g++ 4.8.2 and get the following run-time error:
Pointer Initialized
Pointer Re-initialized
*** Error in `./aab': free(): invalid next size (fast): 0x0000000001cf9080 ***
Aborted (core dumped)
Interestingly when I replace aligned_alloc with malloc or posix_memalign for that matter program terminates correctly. Is this a bug in aligned_alloc or am I missing something basic?
P.S: 1) I did a brief search for a gcc bug which returned false. 2) Advise to avoid managing raw pointers is acknowledged in advance but I would appreciate any help for the problem at hand.
The problem is you have two objects pointing to the same memory: the anonymous object and the first_ptr after the assignment operator completed have the same address. When their respective destructors get called..... you can probably guess what is going to happen.
You are passing the arguments to aligned_alloc in the wrong order, "documentation".
void *aligned_alloc(size_t alignment, size_t size);
This then results in memory corruption, which is only detected when you call free.
Also, you should consider implementing a copy constructor. Here is a simple implementation:
AlignedPtr(const AlignedPtr& x) { *this = x; }