Problem
I intend to write a C++11 application for Linux which does some numerical simulation (not cryptography) based on approximately one million pseudorandom 32bit numbers. To speed things up, I'd like to perform the simulation in parallel threads using all cores of a desktop CPU. I'd like to use the Mersenne Twister mt19937 provided by boost as the PRNG, and I guess that for performance reasons I should have one such PRNG per thread. Now I'm unsure about how to seed them in order to avoid generating the same subsequence of random numbers in multiple threads.
Alternatives
Here are the alternatives that I have thought of so far:
Seed the PRNG for every thread independently from /dev/urandom.
I'm a bit worried about the case when the system entropy pool gets exhausted, as I don't know how the system internal PRNG operates. Could it happen that I accidentially get consecutive seeds which exactly identify consecutive states of the Mersenne Twister, due to the fact that /dev/urandom is using a Mersenne Twister itself? Probably strongly related to my concerns for the next point.
Seed one PRNG from /dev/urandom and the others from that first one.
Basically the same concern as well: is it good or bad to use one PRNG to seed another that uses the same algorithm? Or in other words, does reading 625 32bit integers from a mt19937 correspond directly to the internal state of the mt19937 generator at any point during this generation?
Seed others from first with non-Mersenne information.
As using the same algorithm to generate random numbers and to generate the initial seed feels somehow like it might be a bad idea, I thought about introducing some element which is not dependent on the Mersenne Twister algorithm. For example, I could XOR the thread id into each element of the initial seed vector. Does that make things any better?
Share one PRNG among threads.
This would make sure that there is only one sequence, with all the known and desirable properties of the Mersenne Twister. But the locking overhead required to control access to that generator does worry me somewhat. As I have found no evidence to the contrary, I assume that I as the library user would be responsible for preventing concurrent access to the PRNG.
Pre-generate all random numbers.
This would have one thread generate all the required 1M random numbers up front, to be used by the different threads later on. The memory requirement of 4M would be small compared to that of the overall application. What worries me most in this approach is that the generation of random numbers itself is not concurrent. This whole approach also doesn't scale too well.
Questions
Which of these approaches would you suggest, and why? Or do you have a different suggestion?
Do you know which of my concerns are justified and which are simply due to my lack of insight into how things actually work?
I would go with #1, seed every prng from urandom. This ensures that the states are totally independent (as far as the seed data is independent). Typically there will be plenty of entropy available unless you have many threads. Also, depending on the algorithm used for /dev/urandom you almost certainly don't need to worry about it.
So you might use something like the following to create each prng:
#include <random>
std::mt19937 get_prng() {
std::random_device r;
std::seed_seq seed{r(), r(), r(), r(), r(), r(), r(), r()};
return std::mt19937(seed);
}
You should verify that your implementation of std::random_device pulls from /dev/urandom under your configuration. And if it does use /dev/urandom by default then usually you can say std::random_device("/dev/random") if you want to use /dev/random instead.
You could use a PRNG with a different algebraic structure to seed the different PRNGs. E.g. some sequence of MD5 hashes.
However I would opt for #5. If it works then it is fine. If it does not you can still optimize it.
The point is creating a good PRNG is much harder than one might expect. A good PRNG for threaded applications is most probably something that is still subject to research.
If the number of CPUs is low enough you could get away with leap frogging. E.g. if you have 4 cores, initialize all with the same values, but advance core 1 PRNG by 1, #2 by and #3 by 3. Then advance always 4 steps when you need a new number.
I'd use one instance to seed the others. I'm pretty sure you can do this safely fairly easily.
Even small changes in the state space cause fairly large changes downstream - if you can ensure they don't have exactly the same starting space (and no identical state prefix), I wouldn't worry about producing identical numbers. For instance, using just the values 1,2,3 to seed three threads would work fine - you don't even need to seed the entire space. Another advantage: by using clearly predictable seeds you can easily discredit the idea that you're cherry-picking any runs (assuming you're trying to demonstrate something).
It's trivial to seed in a way that means the resultant "children" are highly un-correlated. Just iterate in a breadth-first fashion; i.e. if you want to seed N x 623 int values, don't seed 623 values sequentially, but pick the first N and distribute, then the next N etc. Even if there's some correlation between the seeder and the children, the correlation between the various children should be virtually non-existant - and that's all you care about.
I'd prefer an algorithm that allows deterministic execution whenever possible, so depending on urandom is not attractive. This makes debugging easier.
Finally, and obviously - test. These PRNG are fairly robust, but by all means eyeball the results and do a few correlation tests inspired by what you're simulating. Most problems should be obvious - either you've seeded badly and there are obvious repeating subsequences, you you've seeded well, and then the quality is dictated by the PRNG limitations.
For final executions, after you're done testing, you can seed the first of 623 state values using urandom for peace of mind and/or the thread ID.
Seed thread 1 with 1, seed thread 2 with 2, etc.
If you need monte carlo this will give you reproducible results, is easy to track and implement.
Take a look at the following paper: Dynamic Creation of Pseudorandom Number Generators and the accompanying implementation: Dynamic Creator. It tackles this exact problem.
If you really want to be mathematically correct, use the jump functions provided by the SFMT algorithm authors. Jump functions guarantee the minimum number of sequences between two different PRNG streams.
Practically speaking, however, a /dev/urandom initialization will suffice.
I'd say #3 is the winner. Seed each thread with something like the processID or threadID; while it's technically possible you could have overlap, it's highly unlikely. Even consecutive numbers shouldn't be related in terms of seeds once you get out of the single digits (I don't know the Twister algorithm, but the worst PRNG I've seen was fine above 7). One million PRNGs isn't that many compared to the scope of most PRNG equations.
Finally, you could check fairly easily. Check the last seed generated by each thread against all numbers in each other thread. If the seed appears in the thread, then check the previous number generated in each thread; if they also match, then you have a collision and need to re-seed your streams and try again.
There is an implementation (and published paper) specifically concerning the use of the Mersenne Twister for parallel computation. It is by the original authors of the MT. They refer to it as "Dynamic Creator", and it can be found here:
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dc.html
That would be a very good place to study your specific usage of MT19937, particularly the paper there.
Related
I hope you are well!
I use random numbers that depend on time for my encryption.
But this makes my encryption hackable...
My question is how can I make randoms that are not the same every time after running the randoms and also do not use seed time.
Most of the time we randomize like this:
srand ( ( unsigned ) time ( 0 ) );
cout << "the random number is: " << rand() % 11;
But we used time here and the hacker can understand the random numbers by having the program run time.
srand and/or rand aren't suited to your use at all.
Although it's not 100% guaranteed, the C++ standard library provides a random_device that's at least intended to provide access to true random number generation hardware. It has a member named entropy() that's intended to give an estimate of the actual entropy available--but it can be implemented as a pseudo-random number generator, in which case this should return 0. Assuming entropy() returns a non-zero result, use random_device to retrieve as much random data as you need.
Most operating systems provide their own access to random number generation hardware. On Windows you'd typically use BCryptGenRandom. On UNIXesque OSes, you can read an appropriate number of bytes from /dev/random/ (there's also /dev/urandom, but it's potentially non-blocking, so it could return data that's less random if there isn't enough entropy available immediately--that's probably fine if you're simulating rolling dice or shuffling cards in a game or something like that, but probably not acceptable for your purpose).
There are pseudo random number generators (PRNG) of very different quality and typical usage. Typical characteristics of PRNGs are:
The repeating period. This is the number of different numbers, that a PRNG can produce and for good generators this is almost always two to the power of the bitwidth.
Hidden patterns. Some bad PRNGs have hidden patterns. Obviously they decrease the quality of the random numbers. There are tests, that can be used to quantify the quality in this regard. This property, as well as the speed are mainly important for scientific usage like monte carlo simulation.
Cryptographic security. For cryptographic operations there is the need of special PRNGs with very specific use cases. You can reed more about them for example on wikipedia.
What I am trying to show you here, is that the choice of the right PRNG is just as important as the right choice of the seed. It turns out that the C rand() performs very bad in all three of the above categories (with exception maybe of the speed). That means if you seed rand() once and repeatedly call it and print say rand() % 11, an attacker will be able to synchronize to the PRNG after a short period of time even if you used the most secure and random seed. The rand() as well as most other, better random generators in the C++ standard library, were designed to be used for scientific calculations and are not suitable for cryptographic purposes.
If you need cryptographically secure random numbers I would suggest you to use a library that is build for exactly that purpose. A widely used crypto library, that can be used cross platform is OpenSSL. It includes a function called RAND_bytes() that can be used for secure random numbers. Do not forget to carefully read the manual page if you want to use it. You have to check the return value for errors!
I am trying to create a password generator. I used the random library, and after reading the documentation, I found out that rand() depends on an algorithm and a seed (srand()) to generate the random number. I tried using the current time as a seed (srand(time(0))), but that is not truly random. Is there a way to generate truly random numbers? Or maybe get the current time very accurately (like, in microseconds)? My platform is Windows 10 x64.
No, these are not truly random numbers. True randomness requires hardware support. Typically they work by sampling an analoge generated white noise signal.
Probably you want to seed your random generator with an always different value. A time() call would do it, or you can also hash it by the current pid ( getpid() ).
It is true that PCs cannot generate truly random numbers without dedicated hardware, however remember that each PC has at least one hardware random generator attached to it - that device is you, the user sitting in front of the computer. Human is a very random thing. Each one has its own speed of key presses, mouse movement patterns etc. This can be leveraged to your advantage.
On Linux you have a special device, /dev/random which uses exactly this. While you work on the PC, it collects random data (not security sensitive of course), such as how fast you tap the keyboard, and in addition hardware related data, such as intervals between interrupts, and some disk IO info. This all generates entropy, which is later used to generate pseudo random data, which is still not 100% random, but is based on much stronger random seed.
I am not an expert on Windows, but a quick search shows that Windows provides CryptGenRandom API, with somewhat similar functionality.
If you want to generate cryptographically strong random data, I suggest you start from this. Remember, this is still not as strong as dedicated hardware random generator. But this is good enough for most real world use cases.
If you are using rand() to generate a password as per your question then I suggest you use some kind of cryptographic method as they will be much stronger and collision-resistant than the numbers generated by rand(). Use something like Openssl or something similar to that kind.
If not, then truly random numbers as suggested by 'peterh' need hardware support. Yet you can use much better functions than rand() such as Mersenne Twister. It is a better generator than rand() as rand just uses a simple linear congruential generator. Read about mt_19937 here
I need to generate cryptographically secure random data in c++11 and I'm worried that using random_device for all the data would severely limit the performance (See slide 23 of Stephan T. Lavavej's "rand() Considered Harmful" where he says that when he tested it (on his system), random_device was 1.93 MB/s and mt19937 was 499 MB/s) as this code will be running on mobile devices (Android via JNI and iOS) which are probably slower than the numbers above.
In addition I'm aware that mt19937 is not cryptographically secure, from wikipedia: "observing a sufficient number of iterations (624 in the case of MT19937, since this is the size of the state vector from which future iterations are produced) allows one to predict all future iterations".
Taking all of the above information into account, can I generate cryptographically secure random data by generating a new random seed from random_device every 624 iterations of mt19937? Or (possibly) better yet, every X iterations where X is a random number (from random_device or mt19937 seeded by random_device) between 1 and 624?
Don't do this. Seriously, just don't. This isn't just asking for trouble--it's more like asking and begging for trouble by going into the most crime-ridden part of the most dangerous city you can find, carrying lots of valuables.
Instead of trying to re-seed MT 19937 often enough to cover up how insecure it is, I'd advise generating your random numbers by running AES in counter mode. This requires that you get one (but only one) good random number of the right size to use as your initial "seed" for your generator.
You use that as the key for AES, and simply use it to encrypt sequential numbers to get a stream of output that looks random, but is easy to reproduce.
This has many advantages. First, it uses an algorithm that's actually been studied heavily, and is generally believed to be quite secure. Second, it means you only need to distribute one (fairly small) random number as the "key" for the system as a whole. Third, it probably gives better throughput. Both Intel's and (seemingly) independent tests show a range of throughput that starts out competitive with what you're quoting for MT 19937 at the low end, and up to around 4-5 times faster at the top end. Given the way you're using it, I'd expect to see you get results close to (and possibly even exceeding1) the top end of the range they show.
Bottom line: AES in counter mode is obviously a better solution to the problem at hand. The very best you can hope for is that MT 19937 ends up close to as fast and close to as secure. In reality, it'll probably disappoint both those hopes, and end up both slower and substantially less secure.
1. How would it exceed those results? Those are undoubtedly based on encrypting bulk data--i.e., reading a block of data from RAM, encrypting it, and writing the result back to RAM. In your case, you don't need to read the result from RAM--you just have to generate consecutive numbers in the CPU, encrypt them, and write out the results.
The short answer is, no you cannot. The requirements for a cryptographically secure RNG are very stringent, and if you have to ask this question here, then you are not sufficiently aware of those requirements. As Jerry says, AES-CTR is one option, if you need repeatability. Another option, which does not allow repeatability, would be to look for an implementation of Yarrow or Fortuna for your system. In general it is much better to find a CSRNG in a library than to roll your own. Library writers are sufficiently aware of the requirements for a good CSRNG.
I tried using rand() function to create random numbers in each thread in a multi-threaded C++ program. I ended up getting much worse results as I was increasing the number of threads. Thanks to this post, I found that because it needs to keep track of the states, rand() uses a lock for every call.
Does using C++11 random library (this usage as an example) do the same procedure and I should expect the same observation if I use it? Or C++11 can provide a way around?
Yes and no. Most of the C++11 random number generators are objects that encapsulate their own state, so as long as you create a separate generator object for each thread, each should be able to operate independently of the others (so you don't need any locking).
The specific case of std::random_device is a little different though: this is intended (but not guaranteed) to obtain truly non-deterministic data from some sort of random number generation hardware. The driver for that device may well impose locking requirements of its own -- and it's often fairly low bandwidth as well, so if you want to obtain a lot of numbers quickly, it's usually a poor choice.
In a typical case, you want to use one generator (other than std::random_device) per thread, and use std::random_device only to provide the initial seeds for those other generators. This may impose locks during that initialization, but then allows each thread to generate its random numbers without any interlock with other threads.
I have a class that contains two sources of randomness.
std::random_device rd;
std::mt19937 random_engine;
I seed the std::mt19937 with a call to std::random_device. If I want to generate a number and I don't care about repeatability, should I call rd() or random_engine()?
In my particular case, I'm sure both would work just fine, because this is going to be called in some networking code where performance is not at a premium, and the results are not especially sensitive. However, I am interested in some "rules of thumb" on when to use hardware entropy and when to use pseudo-random numbers.
Currently, I am only using std::random_device to seed my std::mt19937 engine, and any random number generation I need for my program, I use the std::mt19937 engine.
edit: Here's an explanation for exactly what I am using this particular example for:
This is for a game playing program. This particular game allows the user to customize their 'team' prior to beginning a round against an opponent. Part of setting up a battle involves sending a team to the server. My program has several teams and uses the random number to determine which team to load. Each new battle makes a call to std::random_device to seed the pseudo-random number generator. I log the initial state of the battle, which includes this team that I'm randomly selecting and the initial seed.
The particular random number I'm asking about in this question is for the random team selection (where it is beneficial to not have the opponent know ahead of time what team I'm using, but not mission-critical), but what I'm really looking for is a rule of thumb. Is it fine to always use std::random_device if I don't need repeatability of my numbers, or is there a real risk of using up entropy faster than it can be collected?
The standard practice, as far as I am aware, is to seed the random number generator with a number that is not calculated by the computer (but comes from some external, unpredictable source). That should be the case with your rd() function. From then on, you call the pseudo-random number generator(PRNG) for each and every pseudo-random number that you need.
If you are worried about the numbers not being random enough, then you should pick a different PRNG. Entropy is a scarce and precious resource and should be treated as such. Although, you may not be needing that many random numbers right now, you may in the future; or other applications could need them. You want that entropy to be available whenever an application asks for it.
It sounds like, for your application, that the mersenne twister will suit your needs just fine. No one who plays your game will ever feel like the teams that are loaded aren't random.
If you are not using it for encryption it is fine and well to repeatedly use mt19937 which is seeded by random_engine.
For the rest of this answer, I assume you are using the random numbers for encryption in your networking code. In short, mt19937 is not suitable for that use.
http://en.wikipedia.org/wiki/Mersenne_twister#Disadvantages
There is a potential risk that you will leak information (perhaps indirectly) over time so that an attacker could start to predict the random numbers. At least in theory, but this is what it's about. From Wikipedia
...since this figure is the size of the state vector from
which future iterates are produced) allows one to predict all future iterates.
A simple means of preventing random number generation information to leak to the user is to use one-way hash functions, but there's much more to it. You should use a random number generator designed for that purpose:
http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator
Various examples (with code) are found here http://xlinux.nist.gov/dads/HTML/pseudorandomNumberGen.html
If you need randomness for a simulation or a game, then that you're doing is fine. Call the random device just once, and do everything else with a randomly seeded pseudo-RNG. As a bonus, you should store the seed value in a log file so you can later replay the pseudo-random sequence:
auto const seed = std::random_device()();
// save "seed" to log file
std::mt19937 random_engine(seed);
(For multiple threads, use the PRNG in the main thread to generate seeds for further PRNGs in the spawned threads.)
If you need a lot of true randomness for cryptographic purposes, then a PRNG is never a good idea, though, since a long sequence of output contains a lot less randomness than true randomness, i.e. you can predict all of it from a small subset. If you need true randomness, you should collect it from some unpredictable source (e.g. heat sensors, user keyboard/mouse activity, etc.). Unix's /dev/random may be such a "true randomness" source, but it may not fill up very quickly.
You may want to have a look at http://channel9.msdn.com/Events/GoingNative/2013/rand-Considered-Harmful that explains why you should use uniform_int_distribution, and the relatives strengths of random_device / mt19937.
In this video, Stephan T. Lavavej specifically states that on visual C++, random_device can be used for cryptographic purposes.
The answer is platform dependent. I seem to remember that with Visual C++ 2010, std::random_device is just mt19937 seeded in some undocumented way.
Of course you realize that any ad hoc encryption scheme based on a random number generator is likely to be very weak.
Assuming this is not for cryptographic purposes, the most important thing to remember about random number generation is to think of how you want the distribution of the random numbers to be and what is the range you are expecting.
Usually standard random number generators within libraries are designed to give out uniform distribution. So the numbers will range between 0 and some RAND_MAX ( say on 32 bit machine it is 2^31 -1 )
Now here is the thing to remember with pseudo random number generators. Most of them are designed to generate random numbers and not random bits. The difference is subtle. If you need numbers between 0 and 8 most programmers will say rand()%8
This is bad because the algorithm was for randomizing 32 bits. But you are using only the bottom 3 bits. No good. This will not give you a uniform distribution (assuming that is what you are looking for)
You should use 8 * (rand() + 1) / (RAND_MAX) which will now give you a uniformly random number between 0 and 8.
Now with hardware random number generators you may have random bits being produced. If that is indeed the case, then you have each bit independently being generated. Then it is more like a set of identical independent random bits. The modeling here would have to be a bit different. Keep that in mind, especially in simulations the distribution becomes important.