Related
I'm looking to implement a simple timer mechanism in C++. The code should work in Windows and Linux. The resolution should be as precise as possible (at least millisecond accuracy). This will be used to simply track the passage of time, not to implement any kind of event-driven design. What is the best tool to accomplish this?
Updated answer for an old question:
In C++11 you can portably get to the highest resolution timer with:
#include <iostream>
#include <chrono>
#include "chrono_io"
int main()
{
typedef std::chrono::high_resolution_clock Clock;
auto t1 = Clock::now();
auto t2 = Clock::now();
std::cout << t2-t1 << '\n';
}
Example output:
74 nanoseconds
"chrono_io" is an extension to ease I/O issues with these new types and is freely available here.
There is also an implementation of <chrono> available in boost (might still be on tip-of-trunk, not sure it has been released).
Update
This is in response to Ben's comment below that subsequent calls to std::chrono::high_resolution_clock take several milliseconds in VS11. Below is a <chrono>-compatible workaround. However it only works on Intel hardware, you need to dip into inline assembly (syntax to do that varies with compiler), and you have to hardwire the machine's clock speed into the clock:
#include <chrono>
struct clock
{
typedef unsigned long long rep;
typedef std::ratio<1, 2800000000> period; // My machine is 2.8 GHz
typedef std::chrono::duration<rep, period> duration;
typedef std::chrono::time_point<clock> time_point;
static const bool is_steady = true;
static time_point now() noexcept
{
unsigned lo, hi;
asm volatile("rdtsc" : "=a" (lo), "=d" (hi));
return time_point(duration(static_cast<rep>(hi) << 32 | lo));
}
private:
static
unsigned
get_clock_speed()
{
int mib[] = {CTL_HW, HW_CPU_FREQ};
const std::size_t namelen = sizeof(mib)/sizeof(mib[0]);
unsigned freq;
size_t freq_len = sizeof(freq);
if (sysctl(mib, namelen, &freq, &freq_len, nullptr, 0) != 0)
return 0;
return freq;
}
static
bool
check_invariants()
{
static_assert(1 == period::num, "period must be 1/freq");
assert(get_clock_speed() == period::den);
static_assert(std::is_same<rep, duration::rep>::value,
"rep and duration::rep must be the same type");
static_assert(std::is_same<period, duration::period>::value,
"period and duration::period must be the same type");
static_assert(std::is_same<duration, time_point::duration>::value,
"duration and time_point::duration must be the same type");
return true;
}
static const bool invariants;
};
const bool clock::invariants = clock::check_invariants();
So it isn't portable. But if you want to experiment with a high resolution clock on your own intel hardware, it doesn't get finer than this. Though be forewarned, today's clock speeds can dynamically change (they aren't really a compile-time constant). And with a multiprocessor machine you can even get time stamps from different processors. But still, experiments on my hardware work fairly well. If you're stuck with millisecond resolution, this could be a workaround.
This clock has a duration in terms of your cpu's clock speed (as you reported it). I.e. for me this clock ticks once every 1/2,800,000,000 of a second. If you want to, you can convert this to nanoseconds (for example) with:
using std::chrono::nanoseconds;
using std::chrono::duration_cast;
auto t0 = clock::now();
auto t1 = clock::now();
nanoseconds ns = duration_cast<nanoseconds>(t1-t0);
The conversion will truncate fractions of a cpu cycle to form the nanosecond. Other rounding modes are possible, but that's a different topic.
For me this will return a duration as low as 18 clock ticks, which truncates to 6 nanoseconds.
I've added some "invariant checking" to the above clock, the most important of which is checking that the clock::period is correct for the machine. Again, this is not portable code, but if you're using this clock, you've already committed to that. The private get_clock_speed() function shown here gets the maximum cpu frequency on OS X, and that should be the same number as the constant denominator of clock::period.
Adding this will save you a little debugging time when you port this code to your new machine and forget to update the clock::period to the speed of your new machine. All of the checking is done either at compile-time or at program startup time. So it won't impact the performance of clock::now() in the least.
For C++03:
Boost.Timer might work, but it depends on the C function clock and so may not have good enough resolution for you.
Boost.Date_Time includes a ptime class that's been recommended on Stack Overflow before. See its docs on microsec_clock::local_time and microsec_clock::universal_time, but note its caveat that "Win32 systems often do not achieve microsecond resolution via this API."
STLsoft provides, among other things, thin cross-platform (Windows and Linux/Unix) C++ wrappers around OS-specific APIs. Its performance library has several classes that would do what you need. (To make it cross platform, pick a class like performance_counter that exists in both the winstl and unixstl namespaces, then use whichever namespace matches your platform.)
For C++11 and above:
The std::chrono library has this functionality built in. See this answer by #HowardHinnant for details.
Matthew Wilson's STLSoft libraries provide several timer types, with congruent interfaces so you can plug-and-play. Amongst the offerings are timers that are low-cost but low-resolution, and ones that are high-resolution but have high-cost. There are also ones for measuring pre-thread times and for measuring per-process times, as well as all that measure elapsed times.
There's an exhaustive article covering it in Dr. Dobb's from some years ago, although it only covers the Windows ones, those defined in the WinSTL sub-project. STLSoft also provides for UNIX timers in the UNIXSTL sub-project, and you can use the "PlatformSTL" one, which includes the UNIX or Windows one as appropriate, as in:
#include <platformstl/performance/performance_counter.hpp>
#include <iostream>
int main()
{
platformstl::performance_counter c;
c.start();
for(int i = 0; i < 1000000000; ++i);
c.stop();
std::cout << "time (s): " << c.get_seconds() << std::endl;
std::cout << "time (ms): " << c.get_milliseconds() << std::endl;
std::cout << "time (us): " << c.get_microseconds() << std::endl;
}
HTH
The StlSoft open source library provides a quite good timer on both windows and linux platforms. If you want it to implement on your own, just have a look at their sources.
The ACE library has portable high resolution timers also.
Doxygen for high res timer:
http://www.dre.vanderbilt.edu/Doxygen/5.7.2/html/ace/a00244.html
I have seen this implemented a few times as closed-source in-house solutions .... which all resorted to #ifdef solutions around native Windows hi-res timers on the one hand and Linux kernel timers using struct timeval (see man timeradd) on the other hand.
You can abstract this and a few Open Source projects have done it -- the last one I looked at was the CoinOR class CoinTimer but there are surely more of them.
I highly recommend boost::posix_time library for that. It supports timers in various resolutions down to microseconds I believe
SDL2 has an excellent cross-platform high-resolution timer. If however you need sub-millisecond accuracy, I wrote a very small cross-platform timer library here.
It is compatible with both C++03 and C++11/higher versions of C++.
I found this which looks promising, and is extremely straightforward, not sure if there are any drawbacks:
https://gist.github.com/ForeverZer0/0a4f80fc02b96e19380ebb7a3debbee5
/* ----------------------------------------------------------------------- */
/*
Easy embeddable cross-platform high resolution timer function. For each
platform we select the high resolution timer. You can call the 'ns()'
function in your file after embedding this.
*/
#include <stdint.h>
#if defined(__linux)
# define HAVE_POSIX_TIMER
# include <time.h>
# ifdef CLOCK_MONOTONIC
# define CLOCKID CLOCK_MONOTONIC
# else
# define CLOCKID CLOCK_REALTIME
# endif
#elif defined(__APPLE__)
# define HAVE_MACH_TIMER
# include <mach/mach_time.h>
#elif defined(_WIN32)
# define WIN32_LEAN_AND_MEAN
# include <windows.h>
#endif
static uint64_t ns() {
static uint64_t is_init = 0;
#if defined(__APPLE__)
static mach_timebase_info_data_t info;
if (0 == is_init) {
mach_timebase_info(&info);
is_init = 1;
}
uint64_t now;
now = mach_absolute_time();
now *= info.numer;
now /= info.denom;
return now;
#elif defined(__linux)
static struct timespec linux_rate;
if (0 == is_init) {
clock_getres(CLOCKID, &linux_rate);
is_init = 1;
}
uint64_t now;
struct timespec spec;
clock_gettime(CLOCKID, &spec);
now = spec.tv_sec * 1.0e9 + spec.tv_nsec;
return now;
#elif defined(_WIN32)
static LARGE_INTEGER win_frequency;
if (0 == is_init) {
QueryPerformanceFrequency(&win_frequency);
is_init = 1;
}
LARGE_INTEGER now;
QueryPerformanceCounter(&now);
return (uint64_t) ((1e9 * now.QuadPart) / win_frequency.QuadPart);
#endif
}
/* ----------------------------------------------------------------------- */-------------------------------- */
The first answer to C++ library questions is generally BOOST: http://www.boost.org/doc/libs/1_40_0/libs/timer/timer.htm. Does this do what you want? Probably not but it's a start.
The problem is you want portable and timer functions are not universal in OSes.
STLSoft have a Performance Library, which includes a set of timer classes, some that work for both UNIX and Windows.
I am not sure about your requirement, If you want to calculate time interval please see thread below
Calculating elapsed time in a C program in milliseconds
Late to the party here, but I'm working in a legacy codebase that can't be upgraded to c++11 yet. Nobody on our team is very skilled in c++, so adding a library like STL is proving difficult (on top of potential concerns others have raised about deployment issues). I really needed an extremely simple cross platform timer that could live by itself without anything beyond bare-bones standard system libraries. Here's what I found:
http://www.songho.ca/misc/timer/timer.html
Reposting the entire source here just so it doesn't get lost if the site ever dies:
//////////////////////////////////////////////////////////////////////////////
// Timer.cpp
// =========
// High Resolution Timer.
// This timer is able to measure the elapsed time with 1 micro-second accuracy
// in both Windows, Linux and Unix system
//
// AUTHOR: Song Ho Ahn (song.ahn#gmail.com) - http://www.songho.ca/misc/timer/timer.html
// CREATED: 2003-01-13
// UPDATED: 2017-03-30
//
// Copyright (c) 2003 Song Ho Ahn
//////////////////////////////////////////////////////////////////////////////
#include "Timer.h"
#include <stdlib.h>
///////////////////////////////////////////////////////////////////////////////
// constructor
///////////////////////////////////////////////////////////////////////////////
Timer::Timer()
{
#if defined(WIN32) || defined(_WIN32)
QueryPerformanceFrequency(&frequency);
startCount.QuadPart = 0;
endCount.QuadPart = 0;
#else
startCount.tv_sec = startCount.tv_usec = 0;
endCount.tv_sec = endCount.tv_usec = 0;
#endif
stopped = 0;
startTimeInMicroSec = 0;
endTimeInMicroSec = 0;
}
///////////////////////////////////////////////////////////////////////////////
// distructor
///////////////////////////////////////////////////////////////////////////////
Timer::~Timer()
{
}
///////////////////////////////////////////////////////////////////////////////
// start timer.
// startCount will be set at this point.
///////////////////////////////////////////////////////////////////////////////
void Timer::start()
{
stopped = 0; // reset stop flag
#if defined(WIN32) || defined(_WIN32)
QueryPerformanceCounter(&startCount);
#else
gettimeofday(&startCount, NULL);
#endif
}
///////////////////////////////////////////////////////////////////////////////
// stop the timer.
// endCount will be set at this point.
///////////////////////////////////////////////////////////////////////////////
void Timer::stop()
{
stopped = 1; // set timer stopped flag
#if defined(WIN32) || defined(_WIN32)
QueryPerformanceCounter(&endCount);
#else
gettimeofday(&endCount, NULL);
#endif
}
///////////////////////////////////////////////////////////////////////////////
// compute elapsed time in micro-second resolution.
// other getElapsedTime will call this first, then convert to correspond resolution.
///////////////////////////////////////////////////////////////////////////////
double Timer::getElapsedTimeInMicroSec()
{
#if defined(WIN32) || defined(_WIN32)
if(!stopped)
QueryPerformanceCounter(&endCount);
startTimeInMicroSec = startCount.QuadPart * (1000000.0 / frequency.QuadPart);
endTimeInMicroSec = endCount.QuadPart * (1000000.0 / frequency.QuadPart);
#else
if(!stopped)
gettimeofday(&endCount, NULL);
startTimeInMicroSec = (startCount.tv_sec * 1000000.0) + startCount.tv_usec;
endTimeInMicroSec = (endCount.tv_sec * 1000000.0) + endCount.tv_usec;
#endif
return endTimeInMicroSec - startTimeInMicroSec;
}
///////////////////////////////////////////////////////////////////////////////
// divide elapsedTimeInMicroSec by 1000
///////////////////////////////////////////////////////////////////////////////
double Timer::getElapsedTimeInMilliSec()
{
return this->getElapsedTimeInMicroSec() * 0.001;
}
///////////////////////////////////////////////////////////////////////////////
// divide elapsedTimeInMicroSec by 1000000
///////////////////////////////////////////////////////////////////////////////
double Timer::getElapsedTimeInSec()
{
return this->getElapsedTimeInMicroSec() * 0.000001;
}
///////////////////////////////////////////////////////////////////////////////
// same as getElapsedTimeInSec()
///////////////////////////////////////////////////////////////////////////////
double Timer::getElapsedTime()
{
return this->getElapsedTimeInSec();
}
and the header file:
//////////////////////////////////////////////////////////////////////////////
// Timer.h
// =======
// High Resolution Timer.
// This timer is able to measure the elapsed time with 1 micro-second accuracy
// in both Windows, Linux and Unix system
//
// AUTHOR: Song Ho Ahn (song.ahn#gmail.com) - http://www.songho.ca/misc/timer/timer.html
// CREATED: 2003-01-13
// UPDATED: 2017-03-30
//
// Copyright (c) 2003 Song Ho Ahn
//////////////////////////////////////////////////////////////////////////////
#ifndef TIMER_H_DEF
#define TIMER_H_DEF
#if defined(WIN32) || defined(_WIN32) // Windows system specific
#include <windows.h>
#else // Unix based system specific
#include <sys/time.h>
#endif
class Timer
{
public:
Timer(); // default constructor
~Timer(); // default destructor
void start(); // start timer
void stop(); // stop the timer
double getElapsedTime(); // get elapsed time in second
double getElapsedTimeInSec(); // get elapsed time in second (same as getElapsedTime)
double getElapsedTimeInMilliSec(); // get elapsed time in milli-second
double getElapsedTimeInMicroSec(); // get elapsed time in micro-second
protected:
private:
double startTimeInMicroSec; // starting time in micro-second
double endTimeInMicroSec; // ending time in micro-second
int stopped; // stop flag
#if defined(WIN32) || defined(_WIN32)
LARGE_INTEGER frequency; // ticks per second
LARGE_INTEGER startCount; //
LARGE_INTEGER endCount; //
#else
timeval startCount; //
timeval endCount; //
#endif
};
#endif // TIMER_H_DEF
If one is using the Qt framework in the project, the best solution is probably to use QElapsedTimer.
I am trying to write a function that behaves differently depending on the OS.
I got this code somewhere in my function:
#ifdef OS_WINDOWS
CONSOLE_SCREEN_BUFFER_INFO csbi;
int cols;
GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &csbi);
cols = csbi.srWindow.Right - csbi.srWindow.Left + 1;
#else
int cols;
#ifdef TIOCGSIZE
struct ttysize ts;
ioctl(STDIN_FILENO, TIOCGSIZE, &ts);
cols = ts.ts_cols;
#elif defined(TIOCGWINSZ)
struct winsize ts;
ioctl(STDIN_FILENO, TIOCGWINSZ, &ts);
cols = ts.ws_col;
#endif // TIOCGSIZE
#endif // OS_WINDOWS
So, I want to get the number of columns, but that needs to be done in two different ways for Windows and Linux... I then want to continue working with the cols variable. But I get variable 'cols' is uninitialized when used here
How can I "extract" the cols variable from the macro block?
What the compiler sees on Windows
CONSOLE_SCREEN_BUFFER_INFO csbi;
int cols; //This is the only place where `cols` is delcared!
GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &csbi);
cols = csbi.srWindow.Right - csbi.srWindow.Left + 1;
What the compiler sees not on Windows
struct ttysize ts;
ioctl(STDIN_FILENO, TIOCGSIZE, &ts);
cols = ts.ts_cols; //cols is not declared
or
struct winsize ts;
ioctl(STDIN_FILENO, TIOCGWINSZ, &ts);
cols = ts.ws_col; //cols is not declared
Note that there is no declaration of cols in the non-Windows code.
A simple solution would be to move the decalration of cols before the macro block.
int cols = 0; //or -1 or some other error value
#ifdef OS_WINDOWS
CONSOLE_SCREEN_BUFFER_INFO csbi;
GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &csbi);
//...
As a note on preprocessor macros:
The preprocessor runs before the code is compiled, dealing with #include, #define (replaces text), #ifdef (conditionally includes blocks of code), and so on. Editors and IDEs may show non-use blocks grayed out, but you can also get the preprocessor output seperately to look at (compiler flags depend on your compiler), and this may help you find why this is having errors.
When dealing with OS specific behavior, you'll notice that code is complicated and difficult to understand or maintain when such conditional code is included "inline" with the other, non conditional code.
One way to make this cleaner and clearer is to put conditional code outside the function, and choose among alternatives in a clearly separated function.
For example, let's say the concept is to get columns. Know that is different for each OS case, start by declaring a function to get columns:
int GetColumns();
Now, DEFINE the function depending on the operating system:
#ifdef OS_WINDOWS
int GetColumns()
{
int cols(0);
GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &csbi);
cols = csbi.srWindow.Right - csbi.srWindow.Left + 1;
return cols;
}
#else
#ifdef TIOCGSIZE
int GetColumns()
{
int cols;
struct ttysize ts;
ioctl(STDIN_FILENO, TIOCGSIZE, &ts);
cols = ts.ts_cols;
return cols;
}
#elif defined( TIOCGWINSZ )
int GetColumns()
{
int cols;
struct winsize ts;
ioctl(STDIN_FILENO, TIOCGWINSZ, &ts);
cols = ts.ws_col;
return cols;
}
#endif // elif
#endif // else to OS_WINDOWS
I have not reviewed all of your code within these blocks, but the point here is to separate these concepts to make the code clearer, cleaner and easier to use.
In your main function you now only need:
int cols = GetColumns();
What this does is acknowledge that you need a way to get columns, and that's it. It clearly separates the complications of getting the columns from the code that calls for it.
Now, the side effects of the various conditional code no longer corrupts the function you're writing. If there are complications, they are now isolated with the body of the various versions of GetColumns. The function "wraps" the issues of local variables, differences in function calls, inside the function.
This also makes it usable from elsewhere in the program without repeating conditional code.
Okay, thanks everyone. I changed a few more things and now it's working for me:
unsigned get_terminal_cols()
{
unsigned c = -1;
#ifdef _WIN32
CONSOLE_SCREEN_BUFFER_INFO csbi;
GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &csbi);
c = csbi.srWindow.Right - csbi.srWindow.Left - 3;
#else
struct winsize w;
ioctl(STDOUT_FILENO, TIOCGWINSZ, &w);
c = w.ws_col -3;
#endif // _WIN32
return c;
}
How do I get system up time since the start of the system? All I found was time since epoch and nothing else.
For example, something like time() in ctime library, but it only gives me a value of seconds since epoch. I want something like time() but since the start of the system.
It is OS dependant and already answered for several systems on stackoverflow.
#include<chrono> // for all examples :)
Windows ...
using GetTickCount64() (resolution usually 10-16 millisecond)
#include <windows>
// ...
auto uptime = std::chrono::milliseconds(GetTickCount64());
Linux ...
... using /proc/uptime
#include <fstream>
// ...
std::chrono::milliseconds uptime(0u);
double uptime_seconds;
if (std::ifstream("/proc/uptime", std::ios::in) >> uptime_seconds)
{
uptime = std::chrono::milliseconds(
static_cast<unsigned long long>(uptime_seconds*1000.0)
);
}
... using sysinfo (resolution 1 second)
#include <sys/sysinfo.h>
// ...
std::chrono::milliseconds uptime(0u);
struct sysinfo x;
if (sysinfo(&x) == 0)
{
uptime = std::chrono::milliseconds(
static_cast<unsigned long long>(x.uptime)*1000ULL
);
}
OS X ...
... using sysctl
#include <time.h>
#include <errno.h>
#include <sys/sysctl.h>
// ...
std::chrono::milliseconds uptime(0u);
struct timeval ts;
std::size_t len = sizeof(ts);
int mib[2] = { CTL_KERN, KERN_BOOTTIME };
if (sysctl(mib, 2, &ts, &len, NULL, 0) == 0)
{
uptime = std::chrono::milliseconds(
static_cast<unsigned long long>(ts.tv_sec)*1000ULL +
static_cast<unsigned long long>(ts.tv_usec)/1000ULL
);
}
BSD-like systems (or systems supporting CLOCK_UPTIME or CLOCK_UPTIME_PRECISE respectively) ...
... using clock_gettime (resolution see clock_getres)
#include <time.h>
// ...
std::chrono::milliseconds uptime(0u);
struct timespec ts;
if (clock_gettime(CLOCK_UPTIME_PRECISE, &ts) == 0)
{
uptime = std::chrono::milliseconds(
static_cast<unsigned long long>(ts.tv_sec)*1000ULL +
static_cast<unsigned long long>(ts.tv_nsec)/1000000ULL
);
}
+1 to the accepted answer. Nice survey. But the OS X answer is incorrect and I wanted to show the correction here.
The sysctl function with an input of { CTL_KERN, KERN_BOOTTIME } on OS X returns the Unix Time the system was booted, not the time since boot. And on this system (and every other system too), std::chrono::system_clock also measures Unix Time. So one simply has to subtract these two time_points to get the time-since-boot. Here is how you modify the accepted answer's OS X solution to do this:
std::chrono::milliseconds
uptime()
{
using namespace std::chrono;
timeval ts;
auto ts_len = sizeof(ts);
int mib[2] = { CTL_KERN, KERN_BOOTTIME };
auto constexpr mib_len = sizeof(mib)/sizeof(mib[0]);
if (sysctl(mib, mib_len, &ts, &ts_len, nullptr, 0) == 0)
{
system_clock::time_point boot{seconds{ts.tv_sec} + microseconds{ts.tv_usec}};
return duration_cast<milliseconds>(system_clock::now() - boot);
}
return 0ms;
}
Notes:
It is best to have chrono do your units conversions for you. If your code has 1000 in it (e.g. to convert seconds to milliseconds), rewrite it to have chrono do the conversion.
You can rely on implicit chrono duration unit conversions to be correct if they compile. If they don't compile, that means you're asking for truncation, and you can explicitly ask for truncation with duration_cast.
It's ok to use a using directive locally in a function if it makes the code more readable.
There is a boost example on how to customize logging messages.
In it the author is implementing a simple function unsigned int get_uptime() to get the system uptime for different platforms including Windows, OSx, Linux as well as BSD.
I am working on a project using Visual C++ /CLR in console mode.
How can I get the system clock in microseconds ?
I want to display hours:minutes:seconds:microseconds
The following program works well but is not compatible with other platforms:
#include <stdio.h>
#include <sys/time.h>
int main()
{
struct timeval tv;
struct timezone tz;
struct tm *tm;
gettimeofday(&tv, &tz);
tm=localtime(&tv.tv_sec);
printf(" %d:%02d:%02d %ld \n", tm->tm_hour, tm->tm_min,tm->tm_sec, tv.tv_usec);
return 0;
}
You could use ptime microsec_clock::local_time() from Boost.
The documentation is available here.
After that, you can use std::string to_iso_extended_string(ptime) to display the returned time as a string or you can use the members of ptime directly to format the output by yourself.
Anyway it is worth noting that:
Win32 systems often do not achieve microsecond resolution via this API. If higher resolution is critical to your application test your platform to see the achieved resolution.
So I guess it depends on how precise you require your "clock" to be.
thank you Mr ereOn
I followed your instructions and i have wrote this code ==> it works 100 %
#include <iostream>
#include "boost/date_time/posix_time/posix_time.hpp"
typedef boost::posix_time::ptime Time;
int main (){
int i;
Time t1;
for (int i=0;i<1000;i++)
{
t1=boost::posix_time::microsec_clock::local_time();
std::cout << to_iso_extended_string(t1) << "\n";
}
return 0;
}
I need to test code ported from 32bit to 64bit where pointers are cast around as integer handles, and I have to make sure that the correct sized types are used on 64 bit platforms.
Are there any flags for various compilers, or even flags at runtime which will ensure that malloc returns pointer values greater than the 32bit limit?
Platforms I'm interested in:
Visual Studio 2008 on Windows XP 64, and other 64 bit windows
AIX using xLC
64bit gcc
64bit HP/UX using aCC
Sample Application that allocates 4GB
So thanks to R Samuel Klatchko's answer, I was able to implement a simple test app that will attempt to allocate pages in the first 4GB of address space. Hopefully this is useful to others, and other SO users can give me an idea how portable/effective it is.
#include <stdlib.h>
#include <stdio.h>
#define UINT_32_MAX 0xFFFFFFFF
#ifdef WIN32
typedef unsigned __int64 Tuint64;
#include <windows.h>
#else
typedef unsigned long long Tuint64;
#include <sys/mman.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#endif
static void* Allocate(void* pSuggested, unsigned int PageSize)
{
#ifdef WIN32
void* pAllocated = ::VirtualAlloc(pSuggested, PageSize, MEM_RESERVE ,PAGE_NOACCESS);
if (pAllocated)
{
return pAllocated;
}
return (void*)-1;
#else
void* pAllocated = ::mmap(pSuggested,
PageSize,
PROT_NONE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE,
-1,
0);
if (pAllocated == MAP_FAILED)
{
pAllocated = (void*)-1;
}
return pAllocated;
#endif
}
static void Deallocate(void* pRegion, unsigned int PageSize)
{
#ifdef WIN32
::VirtualFree(pRegion,0,MEM_RELEASE);
#else
::munmap(pRegion,PageSize);
#endif
}
static void Gobble32bitAddressSpace()
{
#ifdef WIN32
SYSTEM_INFO SysInfo;
::GetSystemInfo(&SysInfo);
unsigned int PageSize = SysInfo.dwAllocationGranularity;
#else
unsigned int PageSize = ::sysconf(_SC_PAGE_SIZE);
#endif
unsigned int AllocatedPages = 0;
unsigned int SkippedPages = 0;
void *pStart = 0;
while( ((Tuint64)pStart) < UINT_32_MAX)
{
void* pAllocated = Allocate(pStart, PageSize);
if (pAllocated != (void*)-1)
{
if (pAllocated == pStart)
{
//Allocated at expected location
AllocatedPages++;
}
else
{
//Allocated at a different location
//unallocate and consider this page unreserved
SkippedPages++;
Deallocate(pAllocated,PageSize);
}
}
else
{
//could not allocate at all
SkippedPages++;
}
pStart = (char*)pStart + PageSize;
}
printf("PageSize : %u\n",PageSize);
printf("Allocated Pages : %u (%u bytes)\n",AllocatedPages,PageSize*AllocatedPages);
printf("Skipped Pages : %u (%u bytes)\n",SkippedPages,SkippedPages*PageSize);
}
int main()
{
Gobble32bitAddressSpace();
//Try to call malloc now and see if we get an
//address above 4GB
void* pFirstMalloc = ::malloc(1024);
if (((Tuint64)pFirstMalloc) >= UINT_32_MAX)
{
printf("OK\n");
}
else
{
printf("FAIL\n");
}
return 0;
}
One technique I have used in the past is to allocate enough memory at startup that all the address space below the 4GB limit is used up. While this technique does rely on malloc first using the lower parts of the address space, this was true on all the platforms I work on (Linux, Solaris and Windows).
Because of how Linux uses overcommit, if you don't touch the allocated space below the 4GB limit, you won't use up any virtual memory.
On Windows, you can use VirtualAlloc() with the MEM_RESERVE flag to use up address space without allocating any actual storage.
Not a compiler switch, but a boot-time switch for Windows can do what you want. There is a command called "nolomem" which forces everything to be loaded in address space > 4GB.
If you are using XP, you should be able to use /nolomem in boot.ini . See documentation on OSR.
For Vista/Win7 you can use NOLOMEM option. Documentation is here. This can be done as such:
bcdedit /set {current} NOLOMEM
Not that you asked specifically, but for others that might be curious, gcc on Mac OS X seems to allocate from the area above 4GB for 64-bit programs by default.
Here's a C program to verify this on whatever compiler/OS combination you might have:
#include <stdlib.h>
#include <stdio.h>
int main() {
void *p = malloc(1000);
printf("%p\n", p);
return 0;
}
You would do well to rewrite your code so that the intptr_t type were used, since that is intended exactly to render such practices safer. Unfortunately it is defined in the c99 header, and VC++ does not support C99. That would not however stop you from creating such a header for that platform.
You might also add asserts where such casts occur e.g.
assert( sizeof(integer_param) == sizeof(void*) ) ;
or you could cast the value back to the original pointer type, and then compare:
assert( (mytype*)integer_param == original_pointer ) ;