I certainly can't use the random generator for that. Currently I'm creating a CRC32 hash from unixtime()+microtime().
Are there any smarter methods than hashing time()+microtime() ?
I am not fully satisfied from the results though, I expected it to be more random, but I can see strong patterns in it, until I added more calls to MicroTime() but it gets a lot slower, so I'm looking for some optimal way of doing this.
This silly code generates the best output I could make so far, the calculations were necessary or I could see some patterns in the output:
starthash(crc32);
addtohash(crc32, MicroTime());
addtohash(crc32, time(NULL)); // 64bit
addtohash(crc32, MicroTime()/13.37f);
addtohash(crc32, (10.0f-MicroTime())*1337.0f);
addtohash(crc32, (11130.0f-MicroTime())/1313137.0f);
endhash(crc32);
MicroTime() returns microseconds elapsed from program start. I have overloaded the addtohash() to every possible type.
I would rather take non-library solutions, it's just ~10 lines of code probably anyways, I don't want to install huge library because of something I don't actually need that much, and I'm more interested in the code than just using it from a function call.
If in any doubt, get your seed from CryptGenRandom on Windows, or by reading from dev/random or dev/urandom on *NIX systems.
This might be overkill for your purposes, but unless it causes performance problems there's no point messing with low-entropy sources like the time.
It's unlikely to be underkill. And if you're writing code with a real need for high-quality secure random data, and didn't bother mentioning that in the question, well, you get what you deserve ;-)
you can check for lfsr & pseudorandom generators.. usually this is a hardwre solution but you can implement easily your own software lfsr
Related
Quoting from cppreference:
std::random_device is a non-deterministic random number engine, although implementations are allowed to implement std::random_device using a pseudo-random number engine if there is no support for non-deterministic random number generation.
Is there a way to check whether current implementation uses PRNG instead of RNG (and then say exit with an error) and if not, why not?
Note that little bit of googling shows that at least MinGW implements std::random_device in this way, and thus this is real danger if std::random_device is to be used.
---edit---
Also, if the answer is no and someone could give some insight as to why there is no such function/trait/something I would be quite interested.
Is there a way to check whether current implementation uses PRNG instead of RNG (and then say exit with an error) and if not, why not?
There is a way: std::random_device::entropy will return 0.0 if it is implemented in terms of a random number engine (that is, it's deterministic).
From the standard:
double entropy() const noexcept;
Returns: If the implementation employs a random number engine, returns 0.0. Otherwise, returns an entropy estimate for the random numbers returned by operator(), in the range min() to log_2(max() + 1).
There is no 100% safe way to determine real randomness for sure. With a black box approach the best you could do do is to show evidence if it's not fully random:
first you could verify that the distribution seems random, by generating a lot of random munmbers and making statistics about their distribution (e.g. generate 1 million random numbers between 0 and 1000). If it appears that some numbers come out significantly more often than other, then obviously it's not really random.
THe next you can is to run several time a programme generating random numbers after the same initial seed. If you obtain the same sequence of random numbers then it's definitively PRNG and not real randmness. However, if you don't obtain the same sequence it does not proove tanything: the library could use some kind of auto-seed (using clock ticks or something else) to hide/improve the pseudo-randmness.
If your application highly depends on randomness quality (e.g. cryptographic quality) you should consider some more tests, such as those recommended by NIST SP 800-22
Xarn stated above:
However, said pessimism also precludes this method from differentiating between RNG and PRNG based implementation, making it rather unhelpful. Also VC++ could be realistic, but to check that would probably require a lot of insider knowledge about Windows.
If you debug into the Windows implementation, then you will find that you end up in RtlGenRandom, which is one of the better sources of cryptographically random bytes. If you debug into the Linux implementation, then you should end up reading from dev/urandom, which is also OK. The fact that they don't tell us that we're not using something awful, like rand, is annoying.
PS - you don't have to have internal Windows knowledge, you just need to attach the symbols to the debugger.
Background: I use rand(), std::rand(), std::random_shuffle() and other functions in my code for scientific calculations. To be able to reproduce my results, I always explicitly specify the random seed, and set it via srand(). That worked fine until recently, when I figured out that libxml2 would also call srand() lazily on its first usage - which was after my early srand() call.
I filled in a bug report to libxml2 about its srand() call, but I got the answer:
Initialize libxml2 first then.
That's a perfectly legal call to be made from a library. You should
not expect that nobody else calls srand(), and the man page nowhere
states that using srand() multiple time should be avoided
This is actually my question now. If the general policy is that every lib can/should/will call srand(), and I can/might also call it here and there, I don't really see how that can be useful at all. Or how is rand() useful then?
That is why I thought, the general (unwritten) policy is that no lib should ever call srand() and the application should call it only once in the beginning. (Not taking multi-threading into account. I guess in that case, you anyway should use something different.)
I also tried to research a bit which other libraries actually call srand(), but I didn't find any. Are there any?
My current workaround is this ugly code:
{
// On the first call to xmlDictCreate,
// libxml2 will initialize some internal randomize system,
// which calls srand(time(NULL)).
// So, do that first call here now, so that we can use our
// own random seed.
xmlDictPtr p = xmlDictCreate();
xmlDictFree(p);
}
srand(my_own_seed);
Probably the only clean solution would be to not use that at all and only to use my own random generator (maybe via C++11 <random>). But that is not really the question. The question is, who should call srand(), and if everyone does it, how is rand() useful then?
Use the new <random> header instead. It allows for multiple engine instances, using different algorithms and more importantly for you, independent seeds.
[edit]
To answer the "useful" part, rand generates random numbers. That's what it's good for. If you need fine-grained control, including reproducibility, you should not only have a known seed but a known algorithm. srand at best gives you a fixed seed, so that's not a complete solution anyway.
Well, the obvious thing has been stated a few times by others, use the new C++11 generators. I'm restating it for a different reason, though.
You use the output for scientific calculations, and rand usually implements a rather poor generator (in the mean time, many mainstream implementations use MT19937 which apart from bad state recovery isn't so bad, but you have no guarantee for a particular algorithm, and at least one mainstream compiler still uses a really poor LCG).
Don't do scientific calculations with a poor generator. It doesn't really matter if you have things like hyperplanes in your random numbers if you do some silly game shooting little birds on your mobile phone, but it matters big time for scientific simulations. Don't ever use a bad generator. Don't.
Important note: std::random_shuffle (the version with two parameters) may actually call rand, which is a pitfall to be aware of if you're using that one, even if you otherwise use the new C++11 generators found in <random>.
About the actual issue, calling srand twice (or even more often) is no problem. You can in principle call it as often as you want, all it does is change the seed, and consequentially the pseudorandom sequence that follows. I'm wondering why an XML library would want to call it at all, but they're right in their response, it is not illegitimate for them to do it. But it also doesn't matter.
The only important thing to make sure is that either you don't care about getting any particular pseudorandom sequence (that is, any sequence will do, you're not interested in reproducing an exact sequence), or you are the last one to call srand, which will override any prior calls.
That said, implementing your own generator with good statistical properties and a sufficiently long period in 3-5 lines of code isn't all that hard either, with a little care. The main advantage (apart from speed) is that you control exactly where your state is and who modifies it.
It is unlikely that you will ever need periods much longer than 2128 because of the sheer forbidding time to actually consume that many numbers. A 3GHz computer consuming one number every cycle will run for 1021 years on a 2128 period, so there's not much of an issue for humans with average lifespans. Even assuming that the supercomputer you run your simulation on is a trillion times faster, your grand-grand-grand children won't live to see the end of the period.
Insofar, periods like 219937 which current "state of the art" generators deliver are really ridiculous, that's trying to improve the generator at the wrong end if you ask me (it's better to make sure they're statistically firm and that they recover quickly from a worst-case state, etc.). But of course, opinions may differ here.
This site lists a couple of fast generators with implementations. They're xorshift generators combined with an addition or multiplication step and a small (from 2 to 64 machine words) lag, which results in both fast and high quality generators (there's a test suite as well, and the site's author wrote a couple of papers on the subject, too). I'm using a modification of one of these (the 2-word 128-bit version ported to 64-bits, with shift triples modified accordingly) myself.
This problem is being tackled in C++11's random number generation, i.e. you can create an instance of a class:
std::default_random_engine e1
which allows you to fully control only random numbers generated from object e1 (as opposed to whatever would be used in libxml). The general rule of thumb would then be to use new construct, as you can generate your random numbers independently.
Very good documentation
To address your concerns - I also think that it would be a bad practice to call srand() in a library like libxml. However, it's more that srand() and rand() are not designed to be used in the context you are trying to use them - they are enough when you just need some random numbers, as libxml does. However, when you need reproducibility and be sure that you are independent on others, the new <random> header is the way to go for you. So, to sum up, I don't think it's a good practice on library's side, but it's hard to blame them for doing that. Also, I could not imagine them changing that, as billion other pieces of software probably depend on it.
The real answer here is that if you want to be sure that YOUR random number sequence isn't being altered by someone else's code, you need a random number context that is private to YOUR work. Note that calling srand is only one small part of this. For example, if you call some function in some other library that calls rand, it will also disrupt the sequence of YOUR random numbers.
In other words, if you want predictable behaviour from your code, based on random number generation, it needs to be completely separate from any other code that uses random numbers.
Others have suggested using the C++ 11 random number generation, which is one solution.
On Linux and other compatible libraries, you could also use rand_r, which takes a pointer to an unsigned int to a seed that is used for that sequence. So if you initialize that a seed variable, then use that with all calls to rand_r, it will be producing a unique sequence for YOUR code. This is of course still the same old rand generator, just a separate seed. The main reason I meantion this is that you could fairly easily do something like this:
int myrand()
{
static unsigned int myseed = ... some initialization of your choice ...;
return rand_r(&myseed);
}
and simply call myrand instead of std::rand (and should be doable to work into the std::random_shuffle that takes a random generator parameter)
I'm working on a cryptography project in C++ for school, and I'm going to need a way to generate random numbers that can't be regenerated by someone else (who "guessed" the seed).
To be precise, I'd need either a pure random generator, or a way to get 100% "secure" seed. I've already done some research and thinking, and I've found two ways I could do it, the first way of doing it would be initialising the seed with the current time, but this leaves me to worry that the "hacker" might find out the moment of the generation of the key, and then they'll have the seed, and therefore will be able to predict the next generated numbers. The second way of doing it I found was to ask the user for a seed.
Now, what if I don't want the user to generate the key ? And are my worries about the time-based seed founded or are they just pure paranoia ? Is there a chance anyone could get the execution moment for the code ? Or are there maybe other ways of doing it that I've missed ?
Sidenote: I'm using the random_default_engine from <random>
user1095108 had the right idea, but the comment probably was too short.
Ask the user to type something at random. Each character is about 1 bit of randomness. Users are pretty bad at choosing random characters. Yet, you'll need about 40-50 bits.
However, users are also pretty bad at typing at an exact rhythm. The timing of each keystroke adds an extra few bits of randomness, depending on how accurately your OS can report that. With millisecond resolution, 10 keystrokes should be enough.
Summary: I need a simple self-contained way to seed my RNG so that the seed is different every time the program is launched.
Details:
I often need to run the same program (which does calculations with random numbers, e.g. Monte Carlo simulation etc.) many times to have good statistics on the result. In this case it is important that the random number generator will have a different seed on each run.
I would like to have a simple, cross-platform solution for this that can be contained within the program itself. (I.e. I don't want to always go to the trouble of having a script that launches each instance of the program with a different seed parameter.)
Note that using time(0) as a seed is not a good solution because the timer resolution is bad: if several processes are launched in parallel, they are likely to get the same seed from time(0).
Requirements:
as simple as possible
cross platform (currently I need it to work on Windows & Linux, x86 & x64 only).
self contained: shouldn't rely on a special way of launching the program (passing the seed as a parameter from the launch script is too much trouble).
I'd like to wrap the whole thing into a small library that I can include in any new project with minimal effort and just do something like SeedMyRNG(getSeed());
EDIT:
Although my main question was about doing this in C (or C++), based on the pointers provided in the answer I found os.urandom() as a Python solution (which is also useful for me).
Related relevant question: How to use /dev/random or urandom in C?
"Cross-platform" is a subjective term. Do you mean "any platform" (you might encounter in the future) or "every platform" (on your list of supported platforms)? Here's a pragmatic approach that I usually take:
Check if you have /dev/urandom; if yes, seed from there.
On Windows, use CryptGenRandom().
If all else fails, seed from time().
You could use dev random on Linux and the crypto api on Windows. Write a small library to present a platform independent interface and it should do exactly what you want.
Check out RandomLib
which is a C++ random number library with good support for seeds. In
particular
Random r;
r.Reseed();
causes r to be seeded with a vector of numbers (from a call to
RandomSeed::SeedVector()) which is almost certainly unique. This
includes the time, microseconds, pid, hostid, year.
Less optimally, you can also seed with RandomSeed::SeedWord() which
reads from /dev/urandom if possible. However, you will typically get a
seed collision after 2^16 runs with a single 32-bit word as your seed.
So, if your application is run many times, you are better off using the
bigger seed space offered by a vector.
Of course, this supposes that you are using a random number generator
that can make use of a vector seed. RandomLib offers MT19937 and
SFMT19937, which both use vector seeds.
Update on 2014-08-04:
Boost has a cross-platform implementation now, random_device. Here's an example for seeding a pseudo-random generator from boost using an unpredictable seed:
#include <boost/random/mersenne_twister.hpp>
#include <boost/random/random_device.hpp>
boost::random::mt11213b rng( (boost::random_device())() );
I have this long and complex source code that uses a RNG with a fix seed.
This code is a simulator and the parameters of this simulator are the random values given by this RNG.
When I execute the code in the same machine, no matter how many attempts I do the output is the same. But when I execute this code on two different machines and I compare the outputs of both the machines, they are different.
Is it possible that two different machines gives somehow different output using the same random number generator and the same seed?
The compiler version, the libraries and the OS are the same.
It is certainly possible, as the RNG may be combining machine specific data with the seed, such as the network card address, to generate the random number. It is basically implementation specific.
As they do give different results it is obviously possible that they give different results. Easy-to-answer question, next!
Seriously: without knowing the source code to the RNG it’s hard to know whether you’re observing a bug or a feature. But it sounds like the RNG in question is using a second seed from somewhere else, e.g. the current time, or some hardware-dependent value like the network card’s MAC address.
If you need something that can be repeated from machine to machine, try the Boost Random Number Library.
If it's a pseudo random generator that uses nothing but the seed to produce a number sequence, then by definition they cannot be different. However, if the ones you're using are using something machine dependent to perturb the seed, or quite simply, a different algorithm, it's of course quite possible. Which implementation are you using, and if it's a standard library implementation, are they both the same version?
Yes. There are floating-point RNGs, for instance, whose results can depend on whether your CPU is properly implementing IEEE floats (not guranteed in ISO C++). Also, effects such as spilling 80 bits doubles to memory can influence results.
There is also some possibile confusion about the notion of a "seed". Some people define the seed as all input to set the initial state of the RNG. Others restrict it to only the explicit input in code, and exclude implicit input from e.g. HW sources or /dev/random.
Perhaps it's a little/big endian problem, or the code detects the processor in some way. Easiest way to do this would be to use breakpoints or similar debug routines to watch the seeding routines and the RNG itself at work.
It depends greatly on which RNG you are using. Things such like random(3) or the rand48(3) family are designed to return the same sequence when run with the same seed. Now, if the RNG you are using take /dev/random output, all bets are off and results will be different.