I am currently in process of making our application Large Address Aware. As experience has shown, there are some unexpected gotchas when doing so. I create this post to make a complete list of steps which need to be taken.
The development considerations listed in the AMD Large Address Aware guide provide a good starting point, but are by no means complete:
The following considerations will help to make sure that the code can handle addresses larger than 2GB:
Avoid the use of signed pointer arithmetic (I.e. compares and adds)
Pointers use all 32-bits. Don’t use Bit31 for something else.
Some dll’s will be loaded just under the 2GB boundary. In this case, no consecutive memory can be allocated with VirtualAlloc().
Whenever possible, use GlobalMemoryStatusEx() (preferred) or GlobalMemoryStatus() to retrieve memory sizes.
Therefore, the question is: What is the complete list of things which need to be done when making C++ Win32 native application Large Address Aware?
(obvious) select Support Address Larger than 2 Gigabytes (/LARGEADDRESSAWARE) in the project properties: Linker / System / Enable Large Address
check all pointer subtractions and verify the result is stored in a type which can contain the possible difference, or replace them with comparisons or other constructs - see Detect pointer arithmetics because of LARGEADDRESSAWARE). Note: pointer comparison should be fine, contrary to AMD advice, there is no reason why it should cause 4 GB issues
make sure you are not assuming pointers have Bit31 zero, do not attempt to use Bit31 for something else.
replace all GetCursorPos calls with GetCursorInfo - see GetCursorPos fails with large addresses
for all assignments into PVOID64 use PtrToPtr64, needed e.g. when using ReadFileScatter, see ReadFileScatter remark section
Related
There are a number of garbage collection libraries for C++.
I am kind of confused how the pointer tracking works.
In particular, suppose we have a base pointer P and a list of other pointers who are computed as offsets from P using an array.
Ex,
P2 = P+offset[0]
How does the garbage collector know P2 is still in scope? It has no direct reference but it's still accessible.
Probably the most popular C++ gc is
https://en.m.wikipedia.org/wiki/Boehm_garbage_collector
But following their example syntax it seems very easy to break so I must not be understanding something.
This question cannot be answered in general. There are different systems that may be regarded as garbage collection for C++; for example, Herb Sutter's deferred_ptr is basically a garbage collecting smart pointer. I've personally implemented another version of this idea, similar to Sutter's but less fancy.
I can answer about Boehm, however. How the Boehm garbage collector recognizes pointers when it does its "mark" phase, is basically by scanning memory and assuming that things that look like pointers are pointers.
The garbage collector knows all the areas of memory where user data is and it knows all of the pointers that it has allocated and how big those allocations were. It just looks for chains of pointers starting from "root segments" defined as below, where by "look" we mean explicitly scanning memory for 64 bit values that are the same as one of the GC allocations it has done.
From here:
Since it cannot generally tell where pointer variables are located, it
scans the following root segments for pointers:
The registers. Depending on the architecture, this may be done using assembly code, or by calling a setjmp-like function which
saves register contents on the stack.
The stack(s). In the case of a single-threaded application, on most platforms this is done by scanning the memory between (an
approximation of) the current stack pointer and GC_stackbottom. (For
Itanium, the register stack scanned separately.) The GC_stackbottom
variable is set in a highly platform-specific way depending on the
appropriate configuration information in gcconfig.h. Note that the
currently active stack needs to be scanned carefully, since
callee-save registers of client code may appear inside collector
stack frames, which may change during the mark process. This is
addressed by scanning some sections of the stack "eagerly",
effectively capturing a snapshot at one point in time.
Static data region(s). In the simplest case, this is the region between DATASTART and DATAEND, as defined in gcconfig.h. However, in
most cases, this will also involve static data regions associated
with dynamic libraries. These are identified by the mostly
platform-specific code in dyn_load.c.
The address space for 64-bit pointers is huge so false positives will be rare, but even if they occur, false positives would just be leaks, that last as long as there happens to be some other variable in the memory the mark phase scans that is exactly the same value as some 64-bit pointer that was allocated by the garbage collector.
Would it be at all possible (I don't care about practicality or usefulness) to write a C or C++ program that monitored memory usage in the following, very basic way?
Given that declaring a variable without assigning it a value results in it having the value of whatever is already at its memory location, one could create a large array (thousands or millions of elements) and leave all the values unassigned. Then to see if any of these elements have been overwritten, we would simply need to repeatedly compare their current values to a previous value.
I highly doubt this would be as simple as I posited above. Assuming my doubt is well-founded, wherein would the problem lie and, more importantly, would it be something we could circumvent with some creative or esoteric code? I imagine that the problem would be attributable to something along the lines of the declared, uninitialized elements being not allowing other system processes to write to their memory address. Please give me some pointers! (heehee) Thanks.
Lets say your program is in C
Creating a large array is limited to the extent free memory is allowed and how the OS limits you.
So let's say you created a pretty large array (uninitialized).
Now that memory is given to your process(program you ran) and no other process can access it ! (It's OS role to avoid such things , basic requirements of Virtualization).
So as no other process can access its value won't be changed once its allocated to you.
In C++ when i do new (or even malloc) is there any guarantee that the return address will be greater than a certain value? Because... in this project i find it -very- useful to use 0-1k as a enum. But i wouldn't want to do that if its possible to get a value that low. My only target systems are 32 or 64bit CPUs with the OS window/linux and mac.
Does the standard say anything about pointers? Does windows or linux say anything about their C runtime and what the lowest memory address (for ram) is?
-edit- i end up modifying my new overload to check if the address is above >1k. I call std::terminate if it doesn't.
In terms of standard, there is nothing. But in reality, it depends on the target OS, windows for instance reserves the first 64kb of memory as a no-mans land (depending on the build it is read-only memory, else it is marked as PAGE_NOACCESS), while it uses the upper 0x80000000+ for kernel memory, but it can be changed, see this & this on MSDN.
On x64 you can also use the higher bits of the address (only 47bits are used for addresses currently), but its not such a good idea, as later on it will change and your program will break (AMD who set the standard also advise against it).
There's no such guarantee. You can try using placement new if you need very specific memory locations but it has certain problems that you'll have to work hard to avoid. Why don't you try using a map with an integer key that has the pointer as its value instead? That way you wouldn't have to rely on specific memory addresses and ranges.
In theory, no -- a pointer's not even guaranteed to be > 0. However, in practice, viewed as an unsigned integer (don't forget that a pointer may have a high-order "1" bit), no system that I know of would have a pointer value less than about 1000. But relying on that is relying on "undefined behavior".
There is no standard for where valid memory addresses come from; to write safe system-independent code, you cannot rely on certain addresses (and even with anecdotal support, you never know when that will change with a new system update).
It's very platform-specific, so I would discourage relying on this kind of information unless you have a very good reason and are aware of consequences for portability, maintainability etc.
NULL is guaranteed to be 0x0 always. If I recall correctly, x86 reserves the first 128 MB of address space as "NULL-equivalent", so that valid pointers can't take on values in this range. On x64 there are some additional addresses which you shouldn't encounter in practice, at least for now.
As for address space reserved for the operating system, it will clearly depend on the OS. On Linux, the kernel-user space division is configurable in the kernel, so at least the 3 splits: 1-3 GB, 2-2 GB and 3-1 GB are common on 32-bit systems. You can find more details on kerneltrap.
I had a need of a quick unique ID in one of my classes to differenciate one process from another. I decided to use the address of the instance to do so. I ended up with something like this (quintptr is a Qt defined type of integer to store addresses with the correct size, according to the platform):
Foo::Foo()
: _id(reinterpret_cast<quintptr>(this))
{
...
}
The idea is to compare the output of two different processes of the same exe. On Vista (my dev machine) there's no problem. But on XP, the value of _id is the same (!) in the two processes.
Can anyone explain why is that? and if it's a good idea to use pointers like that (I thought so, I'm not so sure anymore)?
Thanks.
Every process gets its own address space. On XP, they're all the same. Therefore it's very common to see what you saw: two objects that have the same address, but in two different address spaces.
It turns out that this contributes to security risks. Attackers were able to guess where vulnerable objects would be in memory, and exploit those. Vista randomizes address spaces (ASLR) which means that two processes are far more likely to put the same object at different addresses.
For your case, using pointers like that is not a smart idea. Just use the process ID
The reason is each process has its own address space and if two processes do the same they just use the same virtual addresses - maybe even heap allocations will be done at same virtual addresses.
You could call GetCurrentProcessId() once and store the result somewhere so that further retrieval is very fast. The process id persists and is unique for the lifetime of the process.
Each process gets its own address space. Unless something like ASLR kicks in, the memory layouts of two processes stemming from the same executable are likely to be very similar, if not identical.
So your idea is not a good one. Using the process ID sounds like a saner approach here, but keep in mind that those can be recycled too.
Or are those things are reserved for the operation system and things like that?
Thanks.
While it's unlikely that 0x00000001, etc. will be valid pointers (especially if you use odd numbers on many processors) using a pointer to store an integer value will be highly system dependent.
Are you really that strapped for space?
Edit:
You could make it portable like this:
char *base = malloc(NUM_MAGIC_VALUES);
#define MAGIC_VALUE_1 (base + 0)
#define MAGIC_VALUE_2 (base + 1)
...
Well the OS is going to give each program it's own virtual memory space, so when the application references memory spaces 0x0000001 or 0x0000002, it's actually referencing some other physical memory address. I would take a look at paging and virtual memory. So a program will never have access to memory the operating system is using. However I would stay away from manually assigning a memory address for a pointer rather than using malloc() because those memory addresses might be text or reserved space.
This depends on operating system layout. For User space applications running in general purpose operating systems, these are inaccessible addresses.
This problem is related to a architecture's virtual address space. Have a loot at this http://web.cs.wpi.edu/~cs3013/c07/lectures/Section09.1-Intel.pdf
Of course, you can do this:
int* myPointer1 = 0x000001;
int* myPointer2 = 0x000032;
But do not try to dereference addresses, cause it will end in an Access Violation.
The OS gives you the memory, by the way these addresses are just virtual
the OS hides the details and shows it like a big, continous stripe.
Maybe the 0x000000-0x211501 part is on a webserver and you read/write it through net,
and remaining is on your hard disk. Physical memory is just an illusion from your current viewpoint.
You tagged your question C++. I believe that in C++ the address at 0 is reserved and is normally referred to as NULL. Other than that you cannot assume anything. If you want to ask about a particular implementation on a particular OS then that would be a different question.
It depends on the compiler/platform, but many older compilers actually have something like the string "(null)" at address 0x00000000. This is a debug feature because that string will show up if a NULL pointer is ever used by accident. On newer systems like Windows, a pointer to this area will most likely cause a processor exception.
I can pretty much guarantee that address 1 and 2 will either be in use or will raise a processor exception if they're ever used. You can store any value you like in a pointer. But if you try and dereference a pointer with a random value, you're definitely asking for problems.
How about a nice integer instead?
Although the standard requires that NULL is 0, a pointer that is NULL does not have to consist of all zero bits, although it will do in many implementations. That is also something you have to beware of if you memset a POD struct that contains some pointers, and then rely on the pointers holding "NULL" as their value.
If you want to use the same space as a pointer you could use a union, but I guess what you really want is something that doubles up as a pointer and something else, and you know it is not a pointer to a real address if it contains low-numbered values. (With a union you still need to know which type you have).
I'd be interested to know what the magic other value is really being used for. Is this some lazy-evaluation issue where the pointer gives an indication of how to load the data when it is not yet loaded and a genuine pointer when it is?
Yes, on some platforms address 0x00000001 and 0x00000002 are valid addresses. On other platforms they are not.
In the embedded systems world, the validity depends on what resides at those locations. Some platforms may put interrupt or reset vectors at those addresses. Other embedded platforms may place Position Independent executable code there.
There is no standard specification for the layout of addresses. One cannot assume anything. If you want your code to be portable then forget about accessing specific addresses and leave that to the OS.
Also, the structure of a pointer is platform dependent. So is the conversion of the value in a pointer to a physical address. Some systems may only decode a portion of the pointer, others use the entire pointer value. Some may use indirection (a.k.a. virtual addressing) to access real objects. Still no standardization here either.