I have a Windows7 Maximal 64-bit computer with 8 Gb RAM. I have created a Win32 console aplication in MSVC and wrote as follows:
size_t const s_chunkSize = 1024 * 32;
size_t total = 0;
for (;;)
{
if (!::malloc(s_chunkSize))
{
break;
}
total += s_chunkSize;
}
printf("total = %li", total);
// yes, I do not free allocated memory for simplicity
It output me 2111668224 that is below 2Gb. How can I force my program to allocate more that 2Gb? Do I have to change some MSVC project settings? Or do I have to use not malloc but Windows-specific functions? Or do I have to configure Windows somehow?
As explained in the comments, you must use the /LARGEADDRESSAWARE linker flag to enable the use of >2GB of virtual address space on machines that provide it (typically, 32 bit machines with the /3GB flag or 64 bit machines). Notice that doing this requires you to exercise extra care when dealing with pointers ( http://blogs.msdn.com/b/oldnewthing/archive/2004/08/12/213468.aspx and articles linked from there), and won't allow you to access more than 4 GB of virtual address space anyway.
A better solution is to build a 64 bit version of your program: you are no longer restricted to a 32 bit address space, and you avoid the caveats of addresses with the high bit set. Obviously, the downside (beside the porting problems that may arise) is that the generated executable will run only on 64 bit machines.
Related
i can create this array:
int Array[490000000];
cout << "Array Byte= " << sizeof(Array) << endl;
Array byte = 1,960,000,000 byte and convert gb = 1,96 GB about 2 gb whatever.
but i cant create same time these:
int Array[490000000];
int Array2[490000000];
it give error why ? sorry for bad englisgh :)
Also i checked my compiler like this:
printf("%d\n", sizeof(char *));
it gives me 8.
C++ programs are not usually compiled to have 2Gb+ of stack space, regardless of whether it is compiled in 32-bit mode or 64-bit mode. Stack space can be increased as part of the compiler options, but even in the scenario where it is permissible to set the stack size that high, it's still not an ideomatic solution or recommended.
If you need an array of 2Gb, you should use std::vector<int> Array(490'000'000); (strongly recommended) or a manually created array, i.e. int* Array = new int[490'000'000]; (remember that manually allocated memory must be manually deallocated with delete[]), either of which will allocate dynamic memory. You'll still want to be compiling in 64-bit mode, since this will brush up against the maximum memory limit of your application if you don't, but in your scenario, it's not strictly necessary, since 2Gb is less than the maximum memory of a 32-bit application.
But still I can't use more than 2 GB :( why?
The C++ language does not have semantics to modify (nor report) how much automatic memory is available (or at least I have not seen it.) The compilers rely on the OS to provide some 'useful' amount. You will have to search (google? your hw documents, user's manuals, etc) for how much. This limit is 'machine' dependent, in that some machines do not have as much memory as you may want.
On Ubuntu, for the last few releases, the Posix function ::pthread_attr_getstacksize(...) reports 8 M Bytes per thread. (I am not sure of the proper terminology, but) what linux calls 'Stack' is the resource that the C++ compiler uses for the automatic memory. For this release of OS and compiler, the limit for automatic var's is thus 8M (much smaller than 2G).
I suppose that because the next machine might have more memory, the compiler might be given a bigger automatic memory, and I've seen no semantics that will limit the size of your array based on memory size of the machine performing the compile ...
there can can be no compile-time-report that the stack will overflow.
I see Posix has a function suggesting a way to adjust size of stack. I've not tried it.
I have also found Ubuntu commands that can report and adjust size of various memory issues.
From https://www.nics.tennessee.edu/:
The command to modify limits varies by shell. The C shell (csh) and
its derivatives (such as tcsh) use the limit command to modify limits.
The Bourne shell (sh) and its derivatives (such as ksh and bash) use
the ulimit command. The syntax for these commands varies slightly and
is shown below. More detailed information can be found in the man page
for the shell you are using.
One minor experiment ... the command prompt
& dtb_chimes
launches this work-in-progress app which uses Posix and reports 8 MByte stack (automatic var)
With the ulimit prefix command
$ ulimit -S -s 131072 ; dtb_chimes
the app now reports 134,217,728
./dtb_chimes_ut
default Stack size: 134,217,728
argc: 1
1 ./dtb_chimes_ut
But I have not confirmed the actual allocation ... and this is still a lot smaller than 1.96 GBytes ... but, maybe you can get there.
Note: I strongly recommend std::vector versus big array.
On my Ubuntu desktop, there is 4 GByte total dram (I have memory test utilities), and my dynamic memory is limited to about 3.5 GB. Again, the amount of dynamic memory is machine dependent.
64 bits address a lot more memory than I can afford.
I am using the Microsoft Visual Studio 2013 IDE. When I compile a program in C++ while using the header <climits>, I output the macro constant CHAR_BIT to the screen. It tells me there are 8-bits in my char data type (which is 1-byte in C++). However, Visual Studio is a 32-bit application and I am running it on a 64-bit machine (i.e. a machine whose processor has a 64-bit instruction set and operating system is 64-bit Windows 7).
I don't understand why my char data type uses only 8-bits. Shouldn't it be using at least 32-bits (since my IDE is a 32-bit application), let alone 64-bits (since I'm compiling on a 64-bit machine)?
I am told that the number of bits used in a memory address (1-byte) depends on the hardware and implementation. If that's the case, why does my memory address still only use 8-bits and not more?
I think you are confusing memory address bit-width with data value bit-width. Memory addresses (pointers) are 32 bits for 32-bit programs and 64 bits for 64-bit programs. But data types have different widths for their values depending on type (as governed by the standard). So a char is 8-bits, but a char* will be 32-bits if you are compiling as a 32-bit application (also note here it depends on how you compile the application and not what type of processor or OS you are running on).
Edit for questions:
However, what is the relationship between these two?
Memory addresses will always have the same bit width regardless of what data value is stored there.
For example, if I have a 32-bit address and I assign an 8-bit value to that address, does that mean there are 24-bits of unused address space?
Some code (assume 32-bit compilation):
char i_am_1_byte = 0x00; // an 8-bit data value that lives in memory
char* i_am_a_ptr = &i_am_1_byte; // pointer is 32-bits and points to an 8-bit data value
*i_am_a_ptr = 0xFF; // writes 0xFF to the location pointed to by the pointer
// that is, to i_am_1_byte
So we have i_am_1_byte which is a char and takes up 8 bits somewhere in memory. We can get this memory location using the address-of operator & and store it in the pointer variable i_am_a_ptr, which is your 32-bit address. We can write 8 bits of data to the location pointed to be i_am_a_ptr by dereferencing it.
If not, what is the bit-width for memory address actually used for
All the data that your program uses must be located somewhere in memory and each location has an address. Most programs probably will not use most of the memory available for them to use, but we need a way to address every possible location.
how can having more memory address bit-width be helpful?
That depends on how much data you need to work with. A 32-bit program, at most, can address 4GB of memory space (and this may be smaller depending on your OS). That used to be a very, very large amount of memory, but these days it is conceivable a program could run out. It is also a lot easier for the CPU to address more the 4GB of RAM if it is 64-bit (this gets into the difference between physical memory and virtual memory). Of course, 64-bit architecture means a lot more than just bigger addresses and brings many benefits that may be more useful to programs than the bigger memory space.
An interesting fact is that on some processors, such as 32-bit ARM, mostly of their instructions are word aligned. That is, compilers tend to allocate 32-bits (4 bytes) to any data type, even though the data type used needs less than 4 bytes unless otherwise stated in the source code. This happens because ARM architectures are optimized to memory access using word alignment.
I am using tricks to store extra information in pointers, At the moment some bits are not used in pointers(the highest 16 bits), but this will change in the future. I would like to have a way to detect if we are compiling or running on a platform that will use more than 48 bits for pointers.
related things:
Why can't OS use entire 64-bits for addressing? Why only the 48-bits?
http://developer.amd.com/wordpress/media/2012/10/24593_APM_v2.pdf
The solution is needed for x86-64, Windows, C/C++, preferably something that can be done compile-time.
Solutions for other platforms are also of interest but will not marked as correct answer.
Windows has exactly one switch for 32bit and 64bit programs to determine the top of their virtual-address-space:
IMAGE_FILE_LARGE_ADDRESS_AWARE
For both types, omitting it limits the program to the lower 2 GB of address-space, severely reducing the memory an application can map and thus also reducing effectiveness of Address-Space-Layout-Randomization (ASLR, an attack mitigation mechanism).
There is one upside to it though, and just what you seem to want: Only the lower 31 bits of a pointer can be set, so pointers can be safely round-tripped through int (32 bit integer, sign- or zero-extension).
At run-time the situation is slightly better:
Just use the cpuid-instruction from intel, function EAX=80000008H, and read the maximum number of used address bits for virtual addresses from bits 8-15.
The OS cannot use more than the CPU supports, remember that intel insists on canonical addresses (sign-extended).
See here for how to use cpuid from C++: CPUID implementations in C++
I wrote a directory information utility and (because I, and the people I wrote this for collect & use vintage hardware,) made it compatible with DOS and Windows 9x as well as Windows XP/Vista/7/8 64-bit (because we also use those.) The problem I'm running into is that in Windows 9x it's reporting available drive space and total drive space as 2G (well 1.9997 G) even on larger drives. On Windows XP and beyond (32-bit or 64-bit,) it reports the drive sizes correctly. In DOS, of course, this isn't an issue as the maximum size in DOS is 2G already.
The code I'm using is (DInfo.Path is the directory being accessed, with [0] being the drive letter - A, B, C, etc...):
_dos_getdiskfree(DInfo.Path[0] - 'A' + 1, &Free);
BlockSize = Free.sectors_per_cluster * Free.bytes_per_sector;
for (i = 0; i < BlockSize; i++) {
DriveBytes += Free.total_clusters;
if (DriveBytes < Free.total_clusters) ++DBOverflow;
FreeBytes += Free.avail_clusters;
if (FreeBytes < Free.avail_clusters) ++FBOverflow;
}
The only difference between the code in the DOS stub and the Windows portion of the executable is the _dos_getdiskfree is replaced with _getdiskfree instead. I use unsigned __int32 variables in the above code (or unsigned long for the DOS code.) I used 32 bit for compatibility and to reduce re-writing the code as much as possible when converting the DOS code to Windows code. In Windows XP+ I could probably have simplified things by using __int64 variables, but again, I wasn't sure if Windows 9x would provide those or not. I wasn't even sure if the 32-bit versions of Windows XP+ would allow it or not, and really didn't want to research it just streamline it a bit. Even on older HW it works fast enough with the loop.
With the Overflow & Byte variables both 32 bit integers, the size should max out at 8 exabytes (kilobytes, megabytes, gigabytes, terabytes, petabytes, exabytes in case you were wondering,) and since the largest drives currently available are measured in single digit terabytes, that limit shouldn't cause a problem for a while. At least it's doubtful to do so during my lifetime.
The answer provided by Raymond Chen in a comment has fixed the problem. Using GetDiskFreeSpaceEx rather than GetDiskFreeSpace produced the correct results.
I have programmed quite a bit in Matlab for graduate level numerical analysis courses and for stats simulations, but I am now trying to learn a more general purpose language. I am teaching myself C++, using Lippman's "C++ Primer". While working through the section on Pointers, I came across something that confused me. I compiled and ran the following code:
#include <iostream>
int main() {
int ival = 42;
int *p = &ival;
std::cout << p << std::endl;
}
I intended for this to give me the memory address where ival was held, and it seemed to work. Here is the output:
$ g++ pointer.cpp -o pointer
$ ./pointer
0x7fff5fbffa7c
This confused me for a couple of reasons.
1) This is a 12 digit Hex number. This seems to imply that there are at least 16^12 (=2^48) bits (or bytes) of available storage. But, I have 4 GB of DDR3 RAM installed, so I should only have 4*2^30 Bytes = 32*2^30 Bits = 2^35 Bits of storage available. So, my question is: Is this address actually in RAM, or is it in the Cache of the processor? If it is in RAM, how are the memory registers labeled in RAM? Or, am I just completely off the mark? It doesn't seem possible that there is a memory register corresponding to every 12 digit hex number (or even 11 digit, if the left-most digit is held constant) in my RAM.
2) Every time I run the program, ival is stored in the exact same place. This remains true if I rename the variable and change its value. If create a second variable, kval, it's location is at:
0x7fff5fbffa78
4 bytes away from ival's location. (I'm now assuming the address is labelled in bytes because an int uses 4 bytes.) Does the C++ compiler always start with the same memory address when it builds the program?
I am working on a MacBook Pro, with 4GB of DDR3 RAM and 6 MB of L2 Cache (I know, it's a 5 year old machine), and am using the g++ compiler. I hope to eventually learn assembly to really understand how these things work, that's why I'm digging a bit deeper into this.
I have tried to find this information on Google and on Stack Overflow, but haven't had any luck. Also, as far as I can tell, this is not addressed in the C++ book I'm working out of.
Thanks in advance for your time, and please let me know if there is any other info that would be helpful.
Joe
Two words - virtual memory. Memory addresses you see being used in code are not physical memory addresses, they are virtual memory addresses that the OS maps to physical memory addresses behind the scenes. This lets the OS allow different processes to use the same memory addresses internally but keep them physically isolated from each other, mapping each process's pool of virtual memory to different areas of physical memory, or even to HDD space when available RAM is exhausted.
The fact that your output contains more than 8 hex digits means you are likely compiling a 64bit program (which uses addresses with up to 16 hex digits). If you recompiled the program for 32bit, you would see a different output.
Your OS implements virtual memory. Memory is not contiguously allocated, so your pointer really does point to valid memory (from the point of view of your process).
The C++ compiler will likely put variables in similar places for similar programs. However, it's not only up to the compiler, the OS can also affect this (eg. with ASLR).
The address is the location of the pointer "p" in the "virtual" memory of the process. Read about "virtual memory management" to understand how the operating system manages the memory. Wikipedia has a very good article on that: Virtual Memory
The address of local variables depends on two things: 1. the compiler, and 2. where the OS loads the text segment of the process. Most compilers will perform some optimization techniques that may result in the variables being moved around. The absolute locations will also change if the operating system implements full ASLR.