I started noticing that sometimes when deallocating memory in some of my programs, they would inexplicably crash. I began narrowing down the culprit and have come up with an example that illustrates a case that I am having difficulty understanding:
#include <iostream>
#include <stdlib.h>
using namespace std;
int main() {
char *tmp = (char*)malloc(16);
char *tmp2 = (char*)malloc(16);
long address = reinterpret_cast<long>(tmp);
long address2 = reinterpret_cast<long>(tmp2);
cout << "tmp = " << address << "\n";
cout << "tmp2 = " << address2 << "\n";
memset(tmp, 1, 16);
memset(tmp2, 1, 16);
char startBytes[4] = {0};
char endBytes[4] = {0};
memcpy(startBytes, tmp - 4, 4);
memcpy(endBytes, tmp + 16, 4);
cout << "Start: " << static_cast<int>(startBytes[0]) << " " << static_cast<int>(startBytes[1]) << " " << static_cast<int>(startBytes[2]) << " " << static_cast<int>(startBytes[3]) << "\n";
cout << "End: " << static_cast<int>(endBytes[0]) << " " << static_cast<int>(endBytes[1]) << " " << static_cast<int>(endBytes[2]) << " " << static_cast<int>(endBytes[3]) << "\n";
cout << "---------------\n";
free(tmp);
memcpy(startBytes, tmp - 4, 4);
memcpy(endBytes, tmp + 16, 4);
cout << "Start: " << static_cast<int>(startBytes[0]) << " " << static_cast<int>(startBytes[1]) << " " << static_cast<int>(startBytes[2]) << " " << static_cast<int>(startBytes[3]) << "\n";
cout << "End: " << static_cast<int>(endBytes[0]) << " " << static_cast<int>(endBytes[1]) << " " << static_cast<int>(endBytes[2]) << " " << static_cast<int>(endBytes[3]) << "\n";
free(tmp2);
return 0;
}
Here is the output that I am seeing:
tmp = 8795380
tmp2 = 8795400
Start: 16 0 0 0
End: 16 0 0 0
---------------
Start: 17 0 0 0
End: 18 0 0 0
I am using Borland's free compiler. I am aware that the header bytes that I am looking at are implementation specific, and that things like "reinterpret_cast" are bad practice. The question I am merely looking to find an answer to is: why does the first byte of "End" change from 16 to 18?
The 4 bytes that are considered "end" are 16 bytes after tmp, which are 4 bytes before tmp2. They are tmp2's header - why does a call to free() on tmp affect this place in memory?
I have tried the same example using new [] and delete [] to create/delete tmp and tmp2 and the same results occur.
Any information or help in understanding why this particular place in memory is being affected would be much appreciated.
You will have to ask your libc implementation why it changes. In any case, why does it matter? This is a memory area that libc has not allocated to you, and may be using to maintain its own data structures or consistency checks, or may not be using at all.
Basically you are looking at memory you didn't allocate. You can't make any supposition on what happens to the memory outside what you requested (ie the 16 bytes you allocated). There is nothing abnormal going on.
The runtime and compilers are free to do whatever they want to do with them so you should not use them in your programs. The runtime probably change the values of those bytes to keep track of its internal state.
Deallocating memory is very unlikely to crash a program. On the other hand, accessing memory you have deallocated like in your sample is big programming mistake that is likely to do so.
A good way to avoid this is to set any pointers you free to NULL. Doing so you'll force your program to crash when accessing freed variables.
It's possible that the act of removing an allocated element from the heap modifies other heap nodes, or that the implementation reserves one or more bytes of headers for use as guard bytes from previous allocations.
The memory manager must remember for example what is the size of the memory block that has been allocated with malloc. There are different ways, but probably the simplest one is to just allocate 4 bytes more than the size requested in the call and store the size value just before the pointer returned to the caller.
The implementation of free can then subtract 4 bytes from the passed pointer to get a pointer to where the size has been stored and then can link the block (for example) to a list of free reusable blocks of that size (may be using again those 4 bytes to store the link to next block).
You are not supposed to change or even look at bytes before/after the area you have allocated. The result of accessing, even just for reading, memory that you didn't allocate is Undefined Behavior (and yes, you really can get a program to really crash or behave crazily just because of reading memory that wasn't allocated).
Related
I am testing the following code with Visual Studio 2019 Diagnostic Tools.
It says that memory consumption is 55 KB instead of the 20 KB I previously calculated. As you can see, it is much more memory than I thought and I don't know why.
What I want to know is: what is happening or how could I calculate the correct memory consumption? (since I don't always have the "Diagnostic Tools" at hand.)
#include <iostream>
#define TEST_SIZE_ARR 1000
struct Node
{
Node(int)
: id(0),
time(0),
next(0),
back(0)
{}
int id;
int time;
Node* next;
Node* back;
};
int main()
{
int counter = 0;
std::cout << "= Node =" << std::endl;
std::cout << "Array size: " << sizeof(Node*) << " * " << TEST_SIZE_ARR << " = " << sizeof(Node*) * TEST_SIZE_ARR << std::endl;
std::cout << "Element size: " << sizeof(Node) << " * " << TEST_SIZE_ARR << " = " << sizeof(Node) * TEST_SIZE_ARR << std::endl;
Node **dataArr = new Node*[TEST_SIZE_ARR]; //break point
for (counter = 0; counter < TEST_SIZE_ARR; counter++) //break point
{
dataArr[counter] = new Node(counter);
}
counter++; //break point
return 0;
}
Console:
Array size: 4 * 1000 = 4000
Element size: 16 * 1000 = 16000
Diagnostic tool:
Array size: 3.94 KB
Element size: 50.78 KB
Your diagnostic tool is measuring an allocation overhead of 36 bytes per allocation.
50.78 KB is 52000 bytes, or 52 bytes per element allocation. Minus 16 is 36 bytes.
4000 bytes with 36 bytes overhead is 4036 bytes, which is 3.94 KB.
The heap has to track which blocks of memory are in use and which are not. Possibly your diagnostic tool has additional overhead and self measures stupidly; I don't know.
In your case, it appears to be adding an additional 36 bytes per value returned from new. Your system seems to be 32 bit pointers (ick), so that is enough room for 9 pointers. You probably want to include the size of each allocation in its block, which is 4 bytes on a 32 bit system. That leaves 8 pointers.
What your heap is using those 8 pointers for, I don't know. Maybe a skip list, or a red black tree, or even some buffers around each allocation to detect memory corruption because you profiled a debug build and heap.
In general, small heap allocations are inefficient and a bad idea. It is one of the many reasons why block containers, like std vector, are good idea, and node containers are iffy.
I wanted to check that if my string resized, will the address of string change or not. So I wrote the below program whereby initial capacity was 1, and then it changed to 30, I'd assume that on capacity change the string would've moved addresses, but that didnt happen.
Can someone explain why that is?
string s = "1";
string& s1 = s;
cout << &s << " capacity is " << s.capacity() << endl;
cout << &s1 << endl;
s = "sdhflshdgfljasdjflkasdfhalsjdf";
cout << &s << " capacity is " << s.capacity() << endl;
cout << &s1 << endl;
Output is
0x7ffc11fc08d0 capacity is 1
0x7ffc11fc08d0
0x7ffc11fc08d0 capacity is 30
0x7ffc11fc08d0
The string variable will not move, but the buffer it holds a pointer to internally may move to a new address as it allocates more memory. This is not observable by taking the address of the variable though. If you print the pointer returned by the .data() member (by casting it to a void pointer) you may see a change (assuming the new size is enough to trigger reallocation - many strings use a small string optimization with a pre-allocated buffer, so you need to grow beyond that).
I was trying to perform bitwise operations on a char array, as if it were an int, essentially treating bytes like a contiguous area in memory. The code below illustrates my problem.
char *cstr = new char[5];
std::strcpy(cstr, "abcd");
int *p = (int *)(void *)cstr;
std::cout << *p << " " << p << "\n";
std::cout << cstr << " " << (void *)cstr << "\n";
std::cout << sizeof(*p) << "\n";
(*p)++;
std::cout << *p << " " << p << "\n";
std::cout << cstr << " " << (void *)cstr << "\n";
The following output is produced:
1684234849 0x55f046e7de70
abcd 0x55f046e7de70
4
1684234850 0x55f046e7de70
bbcd 0x55f046e7de70
Quick explanation of the code and how it works (to my understanding):
I initialize cstr with "abcd"
char *cstr = new char[5];
std::strcpy(cstr, "abcd");
I point p to the address of cstr and specify that I want it to be an int
int *p = (int *)(void *)cstr;
I test that p is pointing where it should and that it occupies 4 bytes
std::cout << *p << " " << p << "\n";
std::cout << cstr << " " << (void *)cstr << "\n";
std::cout << sizeof(*p) << "\n";
I then increment the integer at the address p is pointing to
(*p)++;
So now, since "abcd" is a contiguous block of 32 bits in memory, incrementing by 1 should produce "abce". Instead, the code increments the integer successfully, but leaves the char array as "bbce". This last part checks the new values of the integer and cstr
std::cout << *p << " " << p << "\n";
std::cout << cstr << " " << (void *)cstr << "\n"
Is this expected behavior?
PS: I compiled the code on a linux machine using this command: g++ main.cpp -o main.
file main
produces the following output: "1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0"
x86-64 CPUs (like yours) store the least significant byte of multi-byte integers at the lowest memory address. So incrementing the integer that "abcd" corresponds to results in incrementing its least-significant byte, which is stored first in memory. This converted the "a" character into a "b". How code like this behaves is very dependent on how the CPU encodes integers and strings and your expectations of what this code will do have to take those details into account.
To expect the string "abce", you have to make lots of assumptions:
You have to expect integers to occupy 4 bytes.
You have to expect the least significant byte to be stored last.
You have to expect the encoding of the character "e" to be one more than the encoding of the character "d".
You have to expect that incrementing a "d" to an "e" won't overflow when viewed as a signed integer increment.
Some of these are reasonable assumptions and some of them aren't, but unless you have reasonable grounds for all these assumptions, your expectation isn't justified.
Is this expected behavior?
It is what people familiar with your platform would expect. But generally it's easy to avoid relying on these kinds of assumptions and so the best advice is not to rely on them. Assumption 3 is often unavoidable and reasonable on all modern platforms.
This is probably something fairly simple I'm missing, but could someone explain what's going on with my code here?
I'd expect arrayone[0] to be unchanged, as I'm never reassigning to it, but somewhere, it's being changed from 3 to 1.
int arrayone[1];
int arraytwo[1];
arrayone[0]=3;
cout << "expected: 3\n";
cout << arrayone[0] << "\n";
arraytwo[0] = int(arrayone[0]/4.0); //int (5/4) = 0
cout << "expected: 3 0\n";
cout << arrayone[0] << " " << arraytwo[0] << "\n";
arraytwo[1] = int(arrayone[0]/2.0); //int (3/2) = 1
cout << "expected: 3 0 1\n";
cout << arrayone[0] << " " << arraytwo[0] << " " << arraytwo[1] <<"\n";
(that final line is returning 1 0 1 instead of 3 0 1)
I've tried testing a few things, looking at where it gets changed; and I think it has to do with the int() function, but I don't understand why.
You use arraytwo[1], which is out of bounds. Because of that you will have undefined behavior.
Perhaps you meant to define arraytwo as an array of two elements:
int arraytwo[2];
The casting you do with int() have nothing to do with it.
On a note related to the casting: If you want an integer after the division, why not do integer division to start with? As in
arraytwo[0] = arrayone[0] / 4;
You need to declare arraytwo with a size of two, i.e., elements 0 and 1, like this
int arraytwo[2];
You don't say what toolchain you are using, but this is the sort of thing that both static and dynamic analysis will detect (for example clang static analyzer and sanitizer respectively).
arraytwo[1] = int(arrayone[0]/2.0); //int (3/2) = 1
Here you are writing to an area beyond arraytwo (it has size 1, so you can only write to arraytwo[0]). That's undefined behavior - what happens in practice is that it's writing to the calculated position anyway, and that's the memory where arrayone is (at least on your setup, it depends on the machine, the compiler used and many settings). This is called memory corruption, and depending on the magnitude of the error you could even get a stack corruption, or, of course, a segmentation fault.
To prevent such mistakes, most compilers will issue a warning on this line. If this isn't happening for you, try looking into whether you can configure your compiler to be more strict with warnings.
Okey, I have a weird (in my opinion) behaviour of my program, which is now reduced to just reading 3 arrays from pretty large (approximately 24GB and 48 GB) binary files. The structure of those files is pretty simple, they contain a small header, and 3 arrays after: of type int, int and float, all 3 of size N, where N is very large: 2147483648 for 28 GB file and 4294967296 for 48 GB one.
To track down the memory consumption, I'm using a simple function based on Linux sysinfo, to detect how much free memory I have on each stage of my program (for example after I allocated the arrays to store data and while reading the file). This is the code of the function:
#include <sys/sysinfo.h>
size_t get_free_memory_in_MB()
{
struct sysinfo info;
sysinfo(&info);
return info.freeram / (1024 * 1024);
}
Now straight to the problem: the strange part is that after reading each of 3 arrays from the file using standard C fread function or C++ read function (doesn't matter at all), and checking how much free memory we have after the read, I see that the amount of free memory is heavily reduced (approximately by edges_count * sizeof(int) for the next example).
fread(src_ids, sizeof(int), edges_count, graph_file);
cout << "1 test: " << get_free_memory_in_MB() << " MB" << endl;
So basically, after reading the whole file my memory consumption according to sysinfo is almost 2x times larger than expected. To illustrate the problem better, I provide the code of the whole function together with it's output; please, read it, it's very small and will illustrate the problem much better.
bool load_from_edges_list_bin_file(string _file_name)
{
bool directed = true;
int vertices_count = 1;
long long int edges_count = 0;
// open the file
FILE *graph_file = fopen(_file_name.c_str(), "r");
if(graph_file == NULL)
return false;
// just reading a simple header here
fread(reinterpret_cast<char*>(&directed), sizeof(bool), 1, graph_file);
fread(reinterpret_cast<char*>(&vertices_count), sizeof(int), 1, graph_file);
fread(reinterpret_cast<char*>(&edges_count), sizeof(long long), 1, graph_file);
cout << "edges count: " << edges_count << endl;
cout << "Before graph alloc free memory: " << get_free_memory_in_MB() << " MB" << endl;
// allocate the arrays to store the result
int *src_ids = new int[edges_count];
int *dst_ids = new int[edges_count];
_TEdgeWeight *weights = new _TEdgeWeight[edges_count];
cout << "After graph alloc free memory: " << get_free_memory_in_MB() << " MB" << endl;
memset(src_ids, 0, edges_count * sizeof(int));
memset(dst_ids, 0, edges_count * sizeof(int));
memset(weights, 0, edges_count * sizeof(_TEdgeWeight));
cout << "After memset: " << get_free_memory_in_MB() << " MB" << endl;
// add edges from file
fread(src_ids, sizeof(int), edges_count, graph_file);
cout << "1 test: " << get_free_memory_in_MB() << " MB" << endl;
fread(dst_ids, sizeof(int), edges_count, graph_file);
cout << "2 test: " << get_free_memory_in_MB() << " MB" << endl;
fread(weights, sizeof(_TEdgeWeight), edges_count, graph_file);
cout << "3 test: " << get_free_memory_in_MB() << " MB" << endl;
cout << "After actual load: " << get_free_memory_in_MB() << " MB" << endl;
delete []src_ids;
delete []dst_ids;
delete []weights;
cout << "After we removed the graph load: " << get_free_memory_in_MB() << " MB" << endl;
fclose(graph_file);
cout << "After we closed the file: " << get_free_memory_in_MB() << " MB" << endl;
return true;
}
So, nothing complicated. Straight to the output (with some comments form me after //). First, for 24GB file:
Loading graph...
edges count: 2147483648
Before graph alloc free memory: 91480 MB
After graph alloc free memory: 91480 MB // allocated memory here, but noting changed, why?
After memset: 66857 MB // ok, we put some data into the memory (memset) and consumed exactly 24 GB, seems correct
1 test: 57658 MB // first read and we have lost 9 GB...
2 test: 48409 MB // -9 GB again...
3 test: 39161 MB // and once more...
After actual load: 39161 MB // we lost in total 27 GB during the reads. How???
After we removed the graph load: 63783 MB // removed the arrays from memory and freed the memory we have allocated
// 24 GB freed, but 27 are still consumed somewhere
After we closed the file: 63788 MB // closing the file doesn't help
Complete!
After we quit the function: 63788 MB // quitting the function doesn't help too.
Similar for 48GB file:
edges count: 4294967296
Before graph alloc free memory: 91485 MB
After graph alloc free memory: 91485 MB
After memset: 42236 MB
1 test: 23784 MB
2 test: 5280 MB
3 test: 490 MB
After actual load: 490 MB
After we removed the graph load: 49737 MB
After we closed the file: 49741 MB
Complete!
After we quit the function: 49741 MB
So, what is happening inside my program?
1) Why so much memory is lost during the reads (both using fread from C and file streams from c++)?
2) Why closing the file doesn't free the memory consumed?
3) Maybe sysinfo is showing me incorrect info?
4) Can this problem be connected to memory fragmentation?
By the way, I'm launching my program on a supercomputer node, on which I have exclusive access (so other people can't influence it), and where are no side-application which can influence my program.
Thank you for reading this!
This is almost certainly the disk (/page) cache. When you read a file the operating system stores some or all of the contents in memory, thus decreasing the amount of free memory. This is to optimise future reads.
This however does not mean the memory is either used by the process or otherwise unavailable. If/when the memory is needed then it will be freed by the OS and made available.
You should be able to confirm this by tracking the value of the bufferram parameter in the sysinfo structure (https://www.systutorials.com/docs/linux/man/2-sysinfo/), or by looking at the output of the free -m command before and after running your program.
For more detailed information on this, see the following answer: https://superuser.com/questions/980820/what-is-the-difference-between-memfree-and-memavailable-in-proc-meminfo