I have been learning C++ for a while, but currently I know nearly nothing about assembly/machine language and how compiler and hardware work. Sorry if this question is really naive...
Consider the following very simple code:
#include <iostream>
using namespace std;
int main()
{
int x = 0;
cout << &x << endl;
}
In my current understanding, the first line in main() asks the compiler to reserve enough memory to hold an int, associate it with the identifier x, and put 0 into that memory location. And the second line in main() prints out the address of the start of that memory location.
I run the above code twice (consecutively), and I got different outputs as follows:
0000002EECAFFB84 // first time
0000007F1FAFF854 // second time
It is also in my current understanding that (please correct me if I have any misunderstandings):
when I first run the program, C++ compiler translate my source code directly into machine code (or also called object code, the code that runs directly on hardware).
since I modified nothing between my first run and my second run, the compiler will NOT generate machine code again, and thus the machine code used in the second run is the same as in the first run.
If my above understandings are correct, then the memory address of x is not determined in the machine code generated by the compiler (otherwise the outputs would be the same), and there must be some intermediate mechanism (between executing the machine code generated by the compiler and creating the int on memory) which decides on the exact memory address where the int will reside.
Is such mechanism done directly by the hardware (e.g. CPU?)? Or is it done by the operating system (so can I say OS is a kinda "virtual machine" for C++?)? May I ask who determines the exact memory address of x and how?
May I also ask how exactly does the compiler generates machine code for &x at compile-time, so that the memory address which hasn't been determined yet can be ensured to be retrieved successfully at a later point in runtime?
when I first run the program, C++ compiler translate my source code directly into machine code (or also called object code, the code that runs directly on hardware)
The compiler doesn't generate machine code when you run the program. It generates the machine code when you compile it. The compiler is also a program. C++ code is textual data. The C++ language is a standard. The compiler implements the C++ standard by writing code that can read the textual data of your C++ program and understand what it should do according to the standard. It will then write a file called an executable containing the machine code.
When you launch the executable, the desktop, which is also an executable but with a higher privilege, will use a system call to ask the os to create a new process which will run the code in that executable.
Assembly is also textual data. It is considered lower level than C++ because every line of code is almost on par with the CPU instructions (one line of code = one instruction) but not always. Assembly remains textual data that an assembler understands and can translate it to individual CPU instructions.
I don't think machine code is normally called object code. Object code normally refers to code that isn't yet linked. It means that, for some symbols that you call in higher level languages, the address to reach them isn't yet known. If the compiler cannot determine the address to reach for a certain symbol (like a function), then it will leave an unresolved symbol in the object code. For example, if you include an header and call a function in it, it contains only a declaration. The actual definition of the function is either in another object file or in a library. When you link your object files together, the linker looks at unresolved symbols and attempts to find them in the other object files you passed. If it doesn't find them, then it throws an error. Compilation is done in those two steps to allow for parallel compilation. Basically, your source code file doesn't need any other file to be compiled. It just needs that every symbol is declared so it can create an object file and leave unresolved symbols in it. Then, the linker patches them. It speeds up compilation by a lot because several threads can be used.
If my above understandings are correct, then the memory address of x is not determined in the machine code generated by the compiler (otherwise the outputs would be the same), and there must be some intermediate mechanism (between executing the machine code generated by the compiler and creating the int on memory) which decides on the exact memory address where the int will reside.
The memory address of x is not determined by the machine code but its relative position within the stack is. The stack's address is stored in a register called the stack pointer. The compiler doesn't know what is the address in advance and it doesn't care. It will access the content of the stack with a relative offset from the stack pointer register. This allows relative addressing for data local to your function.
When a function ends (if you call other functions from main let's say), the compiler puts an instruction to increment the stack pointer. The data that was there is still the same but the stack pointer is pointing above so, when the compiler accesses the stack relative to it, the data isn't in the way. The data is basically forgotten. If you call another function, then that data will most likely be overwritten by what this function initializes (its variables).
For data outside functions (global data), the executable contains a section called the data section which has room for it. The global data will thus have reserved space for the whole execution of your program.
Related
Quoting from C++ Primer:
The address of an object defined outside of any function is a constant expression, and so may be used to initialize a constexpr pointer.
In fact, each time I compile and run the following piece of code:
#include <iostream>
using namespace std;
int a = 1;
int main()
{
constexpr int *p = &a;
cout << "p = " << p << endl;
}
I always get the output:
p = 0x601060
Now, how is that possible? How can the address of an object (global or not) be known at compile time and be assigned to a constexpr? What if that part of the memory is being used for something else when the program is executed?
I always assumed that the memory is managed so that a free portion is allocated when a program is executed, but doesn't matter what particular part of the memory. However, since here we have a constexpr pointer, the program will always require a specific portion, that has to be free to allow the program execution. This doesn't make sense to me, could someone explain this behaviour please? Thanks.
EDIT: After reading your answers and a few articles online, I realized that I missed the whole concept of virtual memory... now it makes sense. It's quite surprising that neither C++ Primer nor Accelerated C++ mention this concept (maybe they will do it in later chapters, I'm still reading...).
However, quoting again C++ Primer:
A constant expression is an expression whose value cannot change and that can be evaluated at compile time.
Given that the linker has a major role in computing the fixed address of global objects, the book would have been more precise if it said "constant expression can be evaluated at link time", not "at compile time".
It's not actually true that the address of an object is known at compile time. What is known at compile time is the offset. When the program is compiled, the address is not emitted into the object file, but a marker to indicate the offset and the section.
To be simplistic about it, the linker then comes along, measures the size of each section, stitches them together and calculates the address of each marker in each object file now that it has a concrete 'base address' for each section.
Of course it's not quite that simple. A linker can also emit a map of the locations of all these adjusted values in its output, so that a loader or load-time linker can re-adjust them just prior to run time.
The point is, logically, for all intents and purposes, the address is a constant from the program's point of view. It's just that the constant isn't given a value until link/load time. When that value is available, every reference to that constant is overwritten by the linker/loader.
If your question is "why is it always the same address?" It's because your OS uses a standard virtual memory layout layered over the virtual memory manager. Addresses in a process are not real memory addresses - they are logical memory addresses. The piece of silicon at that 'address' is mapped in by the virtual memory management circuitry. Thus each process can use the "same" address, while actually using a different area of the memory chips.
I could go on about paging memory in and out, which is related, but it's a long topic. Further reading is encouraged.
It works because global variables are in static storage.
This is because the space for the global/static variable is allocated at compile time within the binary your compiler generates, in a region next to the program's machine code called the "data" segment. When the binary is copied and loaded into memory, the data segment becomes read-write.
This Wikipedia article includes a nice diagram of where the "data" segment fits into the virtual address space:
https://en.wikipedia.org/wiki/Data_segment
Automatic variables are not stored in the data segment because they may be instantiated as many times as their parent function is called. Moreover, they may be allocated at any depth of the stack. Thus it is not possible to know the address of an automatic variable at compile time in the general case.
This is not the case for global variables, which are clearly unique throughout the lifetime of the program. This allows the compiler to assign a fixed address for the variable which is separate from the stack.
In "Secure coding in C and C++", the author mentions,
"The W^X policy allows a memory segment to be writable or executable, but no both. This policy cannot prevent overwriting targets such as those required by atexit() that need to be both writable at runtime and executable. "
I have two questions:
The atexit needs to register a function by a function pointer as argument. The function pointed by the function pointer is either defined in the current program where linker will find the definition or the runtime loader will finds the function body. In either cases, we will know the function definition. Then it only needs to be executable. So why the memory segment for atexit() needs to be both writable at runtime and executable?
Can any C/C++ expert tells me that what other types of APIs that have this property (writable at runtime and executable)? (let's limit the scope to linux only)
Fundamentally, memory that can be written AND executed is very easily tempered with and can result in exploits more easily since there is no need to use ROP or other fancy methods, you can simply write anywhere in the segment the code to execute and branch to it.
In your quote, the meaning of targets in this context is very likely to be a list of function pointers called on exit. The list itself needs to be writable/mutable as per the C API. The code these functions point to need only be executable. Here again, because the list is mutable, you could exploit a program by simply modifying this list by inserting a pointer to your code and force the program to exit which would execute your code. In this context, keeping all memory segments writable OR executable will not save you since 2 different segments are used here (one writable with the function pointer list, the other executable with the code).
Writable & executable memory segments are required by anything that generates code dynamically at runtime: JIT, kernel, executable unpackers, etc. For each of these, there is no technical requirement that the segments hold both properties at the same time. The memory can be allocated writable first, the code copied/generated and with a call to mprotect(), be made executable (and removing the writable property). The only scenario I can see that would benefit from having both properties at the same time is perhaps in a memory constrained environment (eg: unpack an executable in place).
Note that some platforms do not support executable memory to be allocated in user space: Xbox360 & PS3 for example do not support JIT. (The kernel/api support it but you will not be able to release your software, Microsoft and Sony will refuse your submissions as such the feature can only be used in development.)
This is something that recently crossed my mind, quoting from wikipedia: "To initialize a function pointer, you must give it the address of a function in your program."
So, I can't make it point to an arbitrary memory address but what if i overwrite the memory at the address of the function with a piece of data the same size as before and than invoke it via pointer ? If such data corresponds to an actual function and the two functions have matching signatures the latter should be invoked instead of the first.
Is it theoretically possible ?
I apologize if this is impossible due to some very obvious reason that i should be aware of.
If you're writing something like a JIT, which generates native code on the fly, then yes you could do all of those things.
However, in order to generate native code you obviously need to know some implementation details of the system you're on, including how its function pointers work and what special measures need to be taken for executable code. For one example, on some systems after modifying memory containing code you need to flush the instruction cache before you can safely execute the new code. You can't do any of this portably using standard C or C++.
You might find when you come to overwrite the function, that you can only do it for functions that your program generated at runtime. Functions that are part of the running executable are liable to be marked write-protected by the OS.
The issue you may run into is the Data Execution Prevention. It tries to keep you from executing data as code or allowing code to be written to like data. You can turn it off on Windows. Some compilers/oses may also place code into const-like sections of memory that the OS/hardware protect. The standard says nothing about what should or should not work when you write an array of bytes to a memory location and then call a function that includes jmping to that location. It's all dependent on your hardware and your OS.
While the standard does not provide any guarantees as of what would happen if you make a function pointer that does not refer to a function, in real life and in your particular implementation and knowing the platform you may be able to do that with raw data.
I have seen example programs that created a char array with the appropriate binary code and have it execute by doing careful casting of pointers. So in practice, and in a non-portable way you can achieve that behavior.
It is possible, with caveats given in other answers. You definitely do not want to overwrite memory at some existing function's address with custom code, though. Not only is typically executable memory not writeable, but you have no guarantees as to how the compiler might have used that code. For all you know, the code may be shared by many functions that you think you're not modifying.
So, what you need to do is:
Allocate one or more memory pages from the system.
Write your custom machine code into them.
Mark the pages as non-writable and executable.
Run the code, and there's two ways of doing it:
Cast the address of the pages you got in #1 to a function pointer, and call the pointer.
Execute the code in another thread. You're passing the pointer to code directly to a system API or framework function that starts the thread.
Your question is confusingly worded.
You can reassign function pointers and you can assign them to null. Same with member pointers. Unless you declare them const, you can reassign them and yes the new function will be called instead. You can also assign them to null. The signatures must match exactly. Use std::function instead.
You cannot "overwrite the memory at the address of a function". You probably can indeed do it some way, but just do not. You're writing into your program code and are likely to screw it up badly.
How can I statically tell Visual C++ to place a global variable at a given absolute address in memory, like what __attribute__((at(address))) does?
It can be done but I don't believe there is a predefined way to do it so it will take some experimentation. Even though I don't see much benefit if you create your variable at run time just at the start of user code execution.
So first specify the section/segment where to init your variable using the allocate MS specific specifier. Then either start your application in real scenario, dump it or debug it and see where your variable appears. Watch for relocations (there is some ways to try to enforce no relocation but they are not guaranteed to be honored all the time). Another way is to use some code in your app like this one to find the address of the section you defined.
If you for some reason cannot get a consistent behavior you can use this utility to manipulate the virtual address of your object file. All in all except hurdles along the way but overall I don't see why you wouldn't be able to get it to work for your specific scenario if you are persistent enough.
I have a large inherited C/C++ project. Are there any good tools or techniques to produce a report on the "sizeof" of all the datatypes, and a breakdown of the stack footprints of each function in such a project.
I'm curious to know why you want to do this, but that's merely a curiosity.
Determining the sizeof for every class used should be simple, unless they've been templated, in which case you'd have to check every instantiation, also.
Likewise, determining the per call sizeof on a function is simple: it's a sizeof on each passed parameter plus some function overhead.
To determine the full memory usage of the whole program, if it's not all statically defined, couldn't be done without a runtime profiler.
Writing a shell scrip that would collect all the class names into a file would be pretty simple. That file could be constructed as a .cpp file that was a series of calls to sizeof on each class. If the file also #included each header file, it could be compiled and run to get an output of the memory footprint of just the classes.
Likewise, culling all of the function definitions to see when they're not using reference or pointer arguments (ie copying the entire class instance onto the stack) should be pretty straight-forward.
All this goes to say that I know of no existing tool, but writing one shouldn't be difficult.
I'm not aware of any tools, but if you're working under MSVC you can use DIA SDK to extract size information from .PDB files. Sadly, this wont work for stack footprints IIRC.
I'm not sure if the concept of the stack footprint actually exists with modern compilers. That is to say, I think that determining the amount of stack space used depends on the branches taken, which in turn depends on input parameters, and in general requires solving the halting problem.
I am looking for the same information about stack footprint for functions, and I dont believe what warren said is true. Yes, part of what impacts the stack in a function is the parameters, but I've also found that every local variable in a function, regardless of the scoping of said variable, is used to determine the amount of stack space to reserve for the function.
In the particular poor code example I am working with, there are >200 local class instances, each guarded by if (blah-blah) clauses, but the stack space reserved is modified by these guarded local variables.
I know what I need is to be able to read the function prologue for each method to determine the amount of space being reserved for the function, now how would I do that....?