Why is it that the result of standard distributions isn't mandated to be consistent across implementations? The result of pseudo random number generators is on the other hand mandated to be identical.
For example, the following will almost certainly print something different for every different standard library implementation.
std::mt19937 random {100};
std::normal_distribution<> dist;
std::cout << dist(random);
Say I want to do procedural generation and would like identical starting seeds to result in identical results across platforms and compilers. I can't do it with the stl. I have to "regress" to using boost. Why isn't this a defect?
This is not a defect, it is by design. The rationale for this can be found in A Proposal to Add an Extensible Random Number Facility to the Standard Library (N1398) which says (emphasis mine):
On the other hand, the specifications for the distributions only
define the statistical result, not the precise algorithm to use. This
is different from engines, because for distribution algorithms,
rigorous proofs of their correctness are available, usually under the
precondition that the input random numbers are (truely) uniformly
distributed. For example, there are at least a handful of algorithms
known to produce normally distributed random numbers from uniformly
distributed ones. Which one of these is most efficient depends on at
least the relative execution speeds for various transcendental
functions, cache and branch prediction behaviour of the CPU, and
desired memory use. This proposal therefore leaves the choice of the
algorithm to the implementation. It follows that output sequences for
the distributions will not be identical across implementations. It is
expected that implementations will carefully choose the algorithms for
distributions up front, since it is certainly surprising to customers
if some distribution produces different numbers from one
implementation version to the next.
This point is reiterated in the implementation defined section which says:
The algorithms how to produce the various distributions are specified
as implementation-defined, because there is a vast variety of
algorithms known for each distribution. Each has a different trade-off
in terms of speed, adaptation to recent computer architectures, and
memory use. The implementation is required to document its choice so
that the user can judge whether it is acceptable quality-wise.
Related
Usage of rand() is usually frowned upon despite using a seed via srand(). Why would that be the case? What better alternatives are available?
There are two parts to this story.
First, rand is a pseudorandom number generator. This means it depends on a seed. For a given seed it will always give the same sequence (assuming the same implementation). This makes it not suitable for certain applications where security is of a great concern. But this is not specific to rand. It's an issue with any pseudo-random generator. And there are most certainly a lot of classes of problems where a pseudo-random generator is acceptable. A true random generator has its own issues (efficiency, implementation, entropy) so for problems that are not security related most often a pseudo-random generator is used.
So you analyzed your problem and you conclude a pseudo-random generator is the solution. And here we arrive to the real troubles with the C random library (which includes rand and srand) that are specific to it and make it obsolete (a.k.a.: the reasons you should never use rand and the C random library).
One issue is that it has a global state (set by srand). This makes it impossible to use multiple random engines at the same time. It also greatly complicates multithreaded tasks.
The most visible problem of it is that it lacks a distribution engine: rand gives you a number in interval [0 RAND_MAX]. It is uniform in this interval, which means that each number in this interval has the same probability to appear. But most often you need a random number in a specific interval. Let's say [0, 1017]. A commonly (and naive) used formula is rand() % 1018. But the issue with this is that unless RAND_MAX is an exact multiple of 1018 you won't get an uniform distribution.
Another issue is the Quality of Implementation of rand. There are other answers here detailing this better than I could, so please read them.
In modern C++ you should definitely use the C++ library from <random> which comes with multiple random well-defined engines and various distributions for integer and floating point types.
None of the answers here explains the real reason of being rand() bad.
rand() is a pseudo-random number generator (PRNG), but this doesn't mean it must be bad. Actually, there are very good PRNGs, which are statistically hard or impossible to distinguish from true random numbers.
rand() is completely implementation defined, but historically it is implemented as a Linear Congruential Generator (LCG), which is usually a fast, but notoriously bad class of PRNGs. The lower bits of these generators have much lower statistical randomness than the higher bits and the generated numbers can produce visible lattice and/or planar structures (the best example of that is the famous RANDU PRNG). Some implementations try to reduce the lower bits problem by shifting the bits right by a pre-defined amount, however this kind of solution also reduces the range of the output.
Still, there are notable examples of excellent LCGs, like L'Ecuyer's 64 and 128 bits multiplicative linear congruential generators presented in Tables of Linear Congruential Generators of Different Sizes and Good Lattice Structure, Pierre L'Ecuyer, 1999.
The general rule of thumb is that don't trust rand(), use your own pseudo-random number generator which fits your needs and usage requirements.
What is bad about rand/srand is that rand—
Uses an unspecified algorithm for the sequence of numbers it generates, yet
allows that algorithm to be initialized with srand for repeatable "randomness".
These two points, taken together, hamper the ability of implementations to improve on rand's implementation (e.g., to use a cryptographic random number generator [RNG] or an otherwise "better" algorithm for producing pseudorandom numbers). For example, JavaScript's Math.random and FreeBSD's arc4random don't have this problem, since they don't allow applications to seed them for repeatable "randomness" — it's for exactly this reason that the V8 JavaScript engine was able to change its Math.random implementation to a variant of xorshift128+ while preserving backward compatibility. (On the other hand, letting applications supply additional data to supplement "randomness", as in BCryptGenRandom, is less problematic; even so, however, this is generally seen only in cryptographic RNGs.)
Also:
The fact that the algorithm and the seeding procedure for rand and srand are unspecified means that even reproducible "randomness" is not guaranteed between rand/srand implementations, between versions of the same standard library, between operating systems, etc.
If srand is not called before rand is, rand behaves similarly as though srand(1) were first called. In practice, this means that rand can only be implemented as a pseudorandom number generator (PRNG) rather than as a nondeterministic RNG, and that rand's PRNG algorithm can't differ in a given implementation whether the application calls srand or not.
EDIT (Jul. 8, 2020):
There is one more important thing that's bad about rand and srand. Nothing in the C standard for these functions specifies a particular distribution that the "pseudo-random numbers" delivered by rand have to follow, including the uniform distribution or even a distribution that approximates the uniform distribution. Contrast this with C++'s uniform_int_distribution and uniform_real_distribution classes, as well as the specific pseudorandom generator algorithms specified by C++, such as linear_congruential_engine and mt19937.
EDIT (begun Dec. 12, 2020):
Yet another bad thing about rand and srand: srand takes a seed that can only be as big as an unsigned. unsigned must be at least 16 bits and in most mainstream C implementations, unsigned is either 16 or 32 bits depending on the implementation's data model (notably not 64 bits even if the C implementation adopts a 64-bit data model). Thus, no more than 2^N different sequences of numbers can be selected this way (where N is the number of bits in an unsigned), even if the underlying algorithm implemented by rand can produce many more different sequences than that (say, 2^128 or even 2^19937 as in C++'s mt19937).
Firstly, srand() doesn't get a seed, it sets a seed. Seeding is part of the use of any pseudo random number generator (PRNG). When seeded the sequence of numbers that the PRNG produces from that seed is strictly deterministic because (most?) computers have no means to generate true random numbers. Changing your PRNG won't stop the sequence from being repeatable from the seed and, indeed, this is a good thing because the ability to produce the same sequence of pseudo-random numbers is often useful.
So if all PRNGs share this feature with rand() why is rand() considered bad? Well, it comes down to the "psuedo" part of pseudo-random. We know that a PRNG can't be truly random but we want it to behave as close to a true random number generator as possible, and there are various tests that can be applied to check how similar a PRNG sequence is to a true random sequence. Although its implementation is unspecified by the standard, rand() in every commonly used compiler uses a very old method of generation suited for very weak hardware, and the results it produces fair poorly on these tests. Since this time many better random number generators have been created and it is best to choose one suited to your needs rather than relying on the poor quality one likely to provided by rand().
Which is suitable for your purposes depends on what you are doing, for example you may need cryptographic quality, or multi-dimensional generation, but for many uses where you simply want things to be fairly uniformly random, fast generation, and money is not on the line based on the quality of the results you likely want the xoroshiro128+ generator. Alternatively you could use one of the methods in C++'s <random> header but the generators offered are not state of the art, and much better is now available, however, they're still good enough for most purposes and quite convenient.
If money is on the line (e.g. for card shuffling in an online casino, etc.), or you need cryptogaphic quality, you need to carefully investigation appropriate generators and ensure they exactly much your specific needs.
rand is usually -but not always-, for historical reasons, a very bad pseudo-random number generator (PRNG). How bad is it is implementation specific.
C++11 has nice, much better, PRNGs. Use its <random> standard header. See notably std::uniform_int_distribution here which has a nice example above std::mersenne_twister_engine.
PRNGs are a very tricky subject. I know nothing about them, but I trust the experts.
Let me add another reason that makes rand() totally not usable: The standard does not define any characteristic of random numbers it generates, neither distribution nor range.
Without definition of distribution we can't even wrap it to have what distribution we want.
Even further, theorically I can implement rand() by simply return 0, and anounce that RAND_MAX of my rand() is 0.
Or even worse, I can let least significant bit always be 0, which doesn't violate the standard. Image someone write code like if (rand()%2) ....
Pratically, rand() is implementation defined and the standards says:
There are no guarantees as to the quality of the random sequence produced and some implementations
are known to produce sequences with distressingly non-random low-order bits. Applications with
particular requirements should use a generator that is known to be sufficient for their needs
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf p36
If you use rand(), you will basically have the same result after generating your random number.
So even after using srand(), it will be easy to predict the number generated if someone can guess the seed you use. This is because the function rand() uses a specific algorithm to produce such numbers
With some time to waste, you can figure out how to predict numbers generated by the function, given the seed. All you need now is to guess the seed. Some people refer to the seed as the current time. So if can guess the time at which you run the application, I ll be able to predict the number
IT IS BAD TO USE RAND()!!!!
C++ tries to use the concept of time complexity in the specification of many library functions, but asymptotic complexity is a mathematical construct based on asymptotic behavior when the size of inputs and the values of numbers tend to infinity.
Obviously the size of scalars in any given C++ implementation is finite.
What is the official formalization of complexity in C++, compatible with the finite and bounded nature of C++ operations?
Remark: It goes without saying that for a container or algorithm based on a type parameter (as in the STL), complexity can only be expressed in term of number of user provided operations (say a comparison for sorted stuff), not in term of elementary C++ language operations. This is not the issue here.
EDIT:
Standard quote:
4.6 Program execution [intro.execution]
1 The semantic descriptions in this International Standard define a
parameterized nondeterministic abstract machine. This International
Standard places no requirement on the structure of conforming
implementations. In particular, they need not copy or emulate the
structure of the abstract machine. Rather, conforming implementations
are required to emulate (only) the observable behavior of the abstract
machine as explained below.
2 Certain aspects and operations of the abstract machine are described
in this International Standard as implementation-defined (for example,
sizeof(int)). These constitute the parameters of the abstract machine. [...]
The C++ language is defined in term of an abstract machine based on scalar types like integer types with a finite, defined number of bits and only so many possible values. (Dito for pointers.)
There is no "abstract" C++ where integers would be unbounded and could "tend to infinity".
It means in the abstract machine, any array, any container, any data structure is bounded (even if possibly huge compared to available computers and their minuscule memory (compared to f.ex. a 64 bits number).
Obviously the size of scalars in any given C++ implementation is finite.
Of course, you are correct with this statement! Another way of saying this would be "C++ runs on hardware and hardware is finite". Again, absolutely correct.
However, the key point is this: C++ is not formalized for any particular hardware.
Instead, it is formalized against an abstract machine.
As an example, sizeof(int) <= 4 is true for all hardware that I personally have ever programmed for. However, there is no upper bound at all in the standard regarding sizeof(int).
What does the C++ standard state the size of int, long type to be?
So, on a particular hardware the input to some function void f(int) is indeed limited by 2^31 - 1. So, in theory one could argue that, no matter what it does, this is an O(1) algorithm, because it's number of operations can never exceed a certain limit (which is the definition of O(1)). However, on the abstract machine there literally is no such limit, so this argument cannot hold.
So, in summary, I think the answer to your question is that C++ is not as limited as you think. C++ is neither finite nor bounded. Hardware is. The C++ abstract machine is not. Hence it makes sense to state the formal complexity (as defined by maths and theoretical CS) of standard algorithms.
Arguing that every algorithm is O(1), just because in practice there are always hardware limits, could be justified by a purely theoretical thinking, but it would be pointless. Even though, strictly speaking, big O is only meaningful in theory (where we can go towards infinity), it usually turns out to be quite meaningful in practice as well, even if we cannot go towards infinity but only towards 2^32 - 1.
UPDATE:
Regarding your edit: You seem to be mixing up two things:
There is no particular machine (whether abstract or real) that has an int type that could "tend to infinity". This is what you are saying and it is true! So, in this sense there always is an upper bound.
The C++ standard is written for any machine that could ever possibly be invented in the future. If someone creates hardware with sizeof(int) == 1000000, this is fine with the standard. So, in this sense there is no upper bound.
I hope you understand the difference between 1. and 2. and why both of them are valid statements and don't contradict each other. Each machine is finite, but the possibilities of hardware vendors are infinite.
So, if the standard specifies the complexity of an algorithm, it does (must do) so in terms of point 2. Otherwise it would restrict the growth of hardware. And this growth has no limit, hence it makes sense to use the mathematical definition of complexity, which also assumes there is no limit.
asymptotic complexity is a mathematical construct based on asymptotic behavior when the size of inputs and the values of numbers tend to infinity.
Correct. Similarly, algorithms are abstract entities which can be analyzed regarding these metrics within a given computational framework (such as a Turing machine).
C++ tries to use the concept of time complexity in the specification of many library functions
These complexity specifications impose restrictions on the algorithm you can use. If std::upper_bound has logarithmic complexity, you cannot use linear search as the underlying algorithm, because that has only linear complexity.
Obviously the size of scalars in any given C++ implementation is finite.
Obviously, any computational resource is finite. Your RAM and CPU have only finitely many states. But that does not mean everything is constant time (or that the halting problem is solved).
It is perfectly reasonable and workable for the standard to govern which algorithms an implementation can use (std::map being implemented as a red-black-tree in most cases is a direct consequence of the complexity requirements of its interface functions). The consequences on the actual "physical time" performance of real-world programs are neither obvious nor direct, but that is not within scope.
Let me put this into a simple process to point out the discrepancy in your argument:
The C++ standard specifies a complexity for some operation (e.g. .empty() or .push_back(...)).
Implementers must select an (abstract, mathematical) algorithm that fulfills that complexity criterion.
Implementers then write code which implements that algorithm on some specific hardware.
People write and run other C++ programs that use this operation.
You argument is that determining the complexity of the resulting code is meaningless because you cannot form asymptotes on finite hardware. That's correct, but it's a straw man: That's not what the standard does or intends to do. The standard specifies the complexity of the (abstract, mathematical) algorithm (point 1 and 2), which eventually leads to certain beneficial effects/properties of the (real-world, finite) implementation (point 3) for the benefit of people using the operation (point 4).
Those effects and properties are not specified explicitly in the standard (even though they are the reason for those specific standard stipulations). That's how technical standards work: You describe how things have to be done, not why this is beneficial or how it is best used.
Computational complexity and asymptotic complexity are two different terms. Quoting from Wikipedia:
Computational complexity, or simply complexity of an algorithm is the amount of resources required for running it.
For time complexity, the amount of resources translates to the amount of operations:
Time complexity is commonly estimated by counting the number of elementary operations performed by the algorithm, supposing that each elementary operation takes a fixed amount of time to perform.
In my understanding, this is the concept that C++ uses, that is, the complexity is evaluated in terms of the number of operations. For instance, if the number of operations a function performs does not depend on any parameter, then it is constant.
On the contrary, asymptotic complexity is something different:
One generally focuses on the behavior of the complexity for large n, that is on its asymptotic behavior when n tends to the infinity. Therefore, the complexity is generally expressed by using big O notation.
Asymptotic complexity is useful for the theoretical analysis of algorithms.
What is the official formalization of complexity in C++, compatible with the finite and bounded nature of C++ operations?
There is none.
As I could understand, C++ 17 will come with Parallelism. However, what I could not understand is it a specific hardware parallelism (CPU by default)? Or it can be extended to any hardware with multiple computation units?
In other words, will we see something like,for example, "nVidia C++ standard compiler" which is going to compile the parallel parts to be executed on GPUs?
Will it be some more standardized alternative to OpenCL for example?
Note: Absolutely, I am not asking "Will nVidia do that?". I am asking if C++ 17 standards allow that and if it is theoretically possible.
The question provides a link to the paper proposing this change, and, with respect to the parallelism aspects, there haven't been substantial changes to what's proposed. Yes, the compiler can do whatever makes sense for the target hardware to parallelize the execution of various algorithms, provided only that it gets the right answer (with some reservations) and that it doesn't impose unneeded overhead (again, with some reservations).
There are a couple of important points to understand.
First, C++17 parallelism is not a general parallel programming mechanism. It provides parallel versions of many of the STL algorithms, nothing more. So it's not a replacement for more powerful mechanisms like OpenCL, TBB, etc.
Second, there are inherent limitations when you try to parallelize algorithms, and that's why I added those two parenthesized qualifications. For example, the parallel version of std::accumulate will produce the same result as the non-parallel version only if the function being applied to the input range is commutative and associative. The most obvious problem area here is floating-point values, where math operations are not associative, so the result might differ. Similarly, some algorithms actually impose more overhead when parallelized; you get a net speedup, but there is more total work done, so the speedup for those algorithms will not be linear in the number of processing units. std::partial_sum is an example: each output value depends on the preceding value, so it's not simple to parallelize the algorithm. There are ways to do it, but you end up applying the combiner function more times than the non-parallel algorithm would. In general, there are relaxations of the complexity requirements for algorithms in order to reflect this reality.
I'm working on a test suite for my package, and as part of the tests I would like to run my algorithm on a block of data. However, it occured to me that instead of hardcoding a particular block of data, I could use an algorithm to generate it. I'm wondering if the C++11 <random> facilities would be appropriate for this purpose.
From what I understand, the C++11 random number engines are required to implement specific algorithms. Therefore, given the same seed they should produce the same sequence of random integers in the range defined by the algorithm parameters.
However, as far as distributions are concerned, the standard specifies that:
The algorithms for producing each of the specified distributions are implementation-defined.
(26.5.8.1 Random number distribution class templates / In general)
Which — unless I'm mistaken — means that the output of a distribution is pretty much undefined. And from what I've tested, the distributions in GNU libstdc++ and LLVM project's libc++ produce different results given the same random engines.
The question would therefore be: what would be the most correct way of producing pseudo-random data that would be completely repeatable across different platforms?
what would be the most correct way of producing pseudo-random data that would be completely repeatable across different platforms?
That would be obvious: write your own distribution. As you yourself pointed out, the engines are cross-platform since they implement a specific algorithm. It's the distributions that are implementation-defined.
So write the distributions yourself.
Please see this answer: https://stackoverflow.com/a/34962942/1151329
I had exactly this problem and writing my own distributions worked perfectly. I got the same sequences across linux, OSx, windows, x86 and ARM.
I am using random_shuffle on sequence a 1000 times. I want to ensure that on every computer the final sequence is same. This might look undesirable, but still I want to achieve this. Does srand(x) ensure that ?
There are two sources of differences in the resulting shuffle: the algorithm for rand() is not specified, so different implementations produce different sequences of numbers; and the algorithm for random_shuffle is not specified, so, again, different implementations produce different results, even with the same sequence of pseudo-random numbers.
You can eliminate the first problem by using any of the random-number generators in C++11; they're all specified in detail, including, for some specializations, a requirement for the 10,000th value, which is a great help in debugging their implementation. However, there's no analog for shuffling. In particular, the algorithm for std::shuffle is not specified, so it won't give reproducible results. You'll have to write your own. That's not difficult (nowhere near as difficult as writing an engine), just do a little research; there are lots of discussion out there for you to start from.