I have read about many random number generators and all problems that most have (repeatable, non-uniform distribution, floating-point precision, modulus w/e).
I'm a game developer and I'm thinking why not generate 'random' numbers from time ? I know they won't be 'random', but at least they can't be predicted and I'm only happy for them to just feel random for the players.
For example let's say, at every frame we can take 5 digits out of the current time and use them to generate random numbers.
Let's say if we have the time as a float ss.mmmuuunnn where ss = seconds, mmm = miliseconds, uuu = microseconds and nnn = nanoseconds, we can take only the part muuun and use this to generate our very own random numbers. I have investigated them a bit, and they seem and feel pretty random. I can come up with so many formulas to play around with those 5 digits and get new numbers.
Anyone here seeing anything wrong or that can perform miserably ?
Reminder, I'm just looking for an easy way to generate numbers that 'FEEL' randomly distributed and are unpredictable.
Is this an easy and decent way to give players the sense of randomness ?
Let us assume for the sake of the argument that you are making, on average, one call to your random function every 0.1 milliseconds (since you need it to be fast, you are calling it often, right?) and that it is equally probable to fall anywhere into that time range. In other words, the uun part is assumed to be completely random, but everything higher only changes slowly from call to call and is thus mostly predictable.
That is 1000 possible outcomes or ~10 bits of randomness. There are 1,056,964,608 normal floats bewteen 0 and 1 - not equally distributed of course. That's three orders of magnitude more, which sounds like "poor randomness" to me. Similarly, spreading your 10 bits to the 32 bits of an int (no matter how fancy your function) won't actually improve the randomness.
Also note that none of this deals with the possibility (and very likely scenario) that your calls will probably be extremely periodic and/or in short sequences, as well as the fact that your system time function might not have high enough resolution (or significantly increase power consumption of the system). Both further reducing the randomness of the obtained time, and the side effect of the latter can be very undesirable.
Reminder, I'm just looking for an easy way to generate numbers that 'FEEL' randomly distributed and are unpredictable.
That is extremely unspecific. Humans are terrible at judging randomness and will likely "feel" a close-to-uniform distribution to be more random than a true, fully random one - especially when it comes to streaks.
Unpredictability also exists on way too many levels, from "the player can't manually predict what the enemy will do" to "cryptographically secure until the end of time". For example, given the above assumptions, it might be possible to predict the result of the second of two random calls that happen in quick succession with a success rate of anywhere from 0.1% to 100%. How this pattern emerges in a game is hard to tell, but humans are exceedingly good at spotting patterns.
Related
I'm writing performance benchmarks for some of my code. This is both to compare my own implementations as I develop/experiment, and to compare against "competing" implementations. I have no problem writing these, and getting usable results.
It's very well established that more samples are a good thing, as it reduces the impact of erroneous data and gives a more true result.
So, if I'm profiling a given function/procedure/whatever, how many samples does it seem reasonable to get?
I'm currently doing about 1 million samples for each test. These are individual operations, the results rarely take longer than 10s per item, even on an old laptop. Most are under a hundredth of a second.
Actually, it is not well established that more samples are a good thing.
It is nothing more than common wisdom.
I think you are sharing in a general confusion about the reason for profiling, whether the purpose is to measure performance or to find speedups.
For measuring performance, you don't need samples at all.
what you need is a stopwatch, whether in software or not.
If your process runs too quickly for the resolution of the stopwatch, just run your process 10^3 or 10^6 times, measure it, and divide by that number.
For finding speedups, sampling the call stack is very effective, provided the samples contain line-level or instruction-level call site information.
How many samples do you need?
Well, if you see it doing something that could be removed on one sample, that probably doesn't mean much.
But if you see it on two samples, that estimates it's costing time fraction F of about 2/N where N is the number of samples.
Example: if you see it twice in 10 samples, that means it costs roughly 20% of time.
In general, if the speedup is going to save you fraction F of time, it takes on average 2/F samples to see it twice.
Example: if it is going to save 30% of time (F = 0.3) you need on average 2/0.3 = 6.67 samples to see it twice.
Of course, if you see it more than twice, all the better.
Bottom line, for finding speedups, you don't need a lot of samples.
What you do need is to examine each one for activity that could be removed.
What you don't need is to mush them together into "statistics" (like most profilers do).
Many people understand this.
If you want a bit more rigorous explanation, look here.
I need to generate cryptographically secure random data in c++11 and I'm worried that using random_device for all the data would severely limit the performance (See slide 23 of Stephan T. Lavavej's "rand() Considered Harmful" where he says that when he tested it (on his system), random_device was 1.93 MB/s and mt19937 was 499 MB/s) as this code will be running on mobile devices (Android via JNI and iOS) which are probably slower than the numbers above.
In addition I'm aware that mt19937 is not cryptographically secure, from wikipedia: "observing a sufficient number of iterations (624 in the case of MT19937, since this is the size of the state vector from which future iterations are produced) allows one to predict all future iterations".
Taking all of the above information into account, can I generate cryptographically secure random data by generating a new random seed from random_device every 624 iterations of mt19937? Or (possibly) better yet, every X iterations where X is a random number (from random_device or mt19937 seeded by random_device) between 1 and 624?
Don't do this. Seriously, just don't. This isn't just asking for trouble--it's more like asking and begging for trouble by going into the most crime-ridden part of the most dangerous city you can find, carrying lots of valuables.
Instead of trying to re-seed MT 19937 often enough to cover up how insecure it is, I'd advise generating your random numbers by running AES in counter mode. This requires that you get one (but only one) good random number of the right size to use as your initial "seed" for your generator.
You use that as the key for AES, and simply use it to encrypt sequential numbers to get a stream of output that looks random, but is easy to reproduce.
This has many advantages. First, it uses an algorithm that's actually been studied heavily, and is generally believed to be quite secure. Second, it means you only need to distribute one (fairly small) random number as the "key" for the system as a whole. Third, it probably gives better throughput. Both Intel's and (seemingly) independent tests show a range of throughput that starts out competitive with what you're quoting for MT 19937 at the low end, and up to around 4-5 times faster at the top end. Given the way you're using it, I'd expect to see you get results close to (and possibly even exceeding1) the top end of the range they show.
Bottom line: AES in counter mode is obviously a better solution to the problem at hand. The very best you can hope for is that MT 19937 ends up close to as fast and close to as secure. In reality, it'll probably disappoint both those hopes, and end up both slower and substantially less secure.
1. How would it exceed those results? Those are undoubtedly based on encrypting bulk data--i.e., reading a block of data from RAM, encrypting it, and writing the result back to RAM. In your case, you don't need to read the result from RAM--you just have to generate consecutive numbers in the CPU, encrypt them, and write out the results.
The short answer is, no you cannot. The requirements for a cryptographically secure RNG are very stringent, and if you have to ask this question here, then you are not sufficiently aware of those requirements. As Jerry says, AES-CTR is one option, if you need repeatability. Another option, which does not allow repeatability, would be to look for an implementation of Yarrow or Fortuna for your system. In general it is much better to find a CSRNG in a library than to roll your own. Library writers are sufficiently aware of the requirements for a good CSRNG.
I have a very complicated and sophisticated data fitting program which uses the Levenverg-Marquardt algorithm to do fitting in double precision (basically the fitting class is templatized, but I use instantiate it to doubles). The fitting process involves:
Calculating an error function (chi-square)
Solving a system of linear equations (I use lapack for that)
calculating the derivatives of a function with respect to the parameters, which I want to fit to the data (usually 20+ parameters)
calculating the function value continuously: the function is a complicated combination of a sinusoidal and exponential functions with a few harmonics.
A colleague of mine has suggested that I use integers for at least 10 times faster at least. My questions are:
Is that true that I will get that kind of improvement?
Is it safe to convert everything to integers? And what are the drawbacks to this?
What advice would you have for this whole issue? What would you do?
The program is developed to calculate some parameters from the signal online, which means that the program must be as fast as possible, but I'm wondering whether it's worth it to start the project of converting everything to integers.
The amount of improvement depends on your platform. For example, if your platform has a fast floating point coprocessor, performing arithmetic in floating point may be faster than integral arithmetic.
You may be able to get more performance gain by optimizing your algorithms rather than switching to integer arithmetic.
Another method for boosting performance is to reduce data cache hits and also reducing branches and loops.
I would measure performance of the program to find out where the bottlenecks are and then review the sections that where most of the performance takes place. For example, in my embedded system, micro-optimizations like what you are suggesting, saved 3 microseconds. This gain is not worth the effort to retest the entire system. If it works, don't fix it. Concentrate on correctness and robustness first.
The bottom line here is that you have to test it and decide for yourself. Profile a release build using real data.
1- Is that true that I will get that kind of improvement?
Maybe yes, maybe no. It depends on a number of factors, such as
How long it takes to convert from double to int
How big a word is on your machine
What platform/toolset you're using and what optimizations you have enabled
(Maybe) how big a cache line is on your platform
How fast your memory is
How fast your platform computes floating-point versus integer.
And who knows what else. In short, too many complex variables for anyone to be able to say for sure if you will or will not improve performance.
But I would be highly skeptical about your friend's claim, "at least 10 times faster at least."
2- Is it safe to convert everything to integers? And what are the
drawbacks to this?
It depends on what you're converting and how. Obviously converting a value like 123.456 to an integer is decidedly unsafe.
Drawbacks include loss of precision, loss of accuracy, and the expense in terms of space and time to actually do the conversions. Another significant drawback is the fact that you have to write a substantial amount of code, and every line of code you write is a probable source of new bugs.
3- What advice would you have for this whole issue? What would you do?
I would step back & take a deep breath. Profile your code under real-world conditions. Identify the sources of the bottlenecks. Find out what the real problems are, and if there even are any.
Identify inefficiencies in your algorithms, and fix them.
Throw hardware at the problem.
Then you can endeavor to start micro-optimizing. This would be my last resort, especially if the optimization technique you are considering would require writing a lot of code.
First, this reeks of attempting to optimize unnecessarily.
Second, doubles are a minimum of 64-bits. ints on most systems are 32-bits. So you have a couple of choices: truncate the double (which reduces your precision to a single), or store it in the space of 2 integers, or store it as an unsigned long long (which is at least 64-bits as well). For the first 2 options, you are facing a performance hit as you must convert the numbers back and forth between the doubles you are operating on and the integers you are storing it as. For the third option, you are not gaining any performance increase (in terms of memory usage) as they are basically the same size - so you'd just be converting them to integers for no reason.
So, to get to your questions:
1) Doubtful, but you can try it to see for yourself.
2) The problem isn't storage as the bits are just bits when they get into memory. The problem is the arithmetic. Since you stated you need double precision, attempting to do those operations on an integer type will not give you the results you are looking for.
3) Don't optimize until it has been proven something needs to have a performance improvement. And always remember Amdahl's Law: Make the common case fast and the rare case correct.
What I would do is:
First tune it in single-thread mode (by the random-pausing method) until you can't find any way to reduce cycles. The kinds of things I've found are:
a large fraction of time spent in library functions like sin, cos, exp, and log where the arguments were often unchanged, so the answers would be the same. The solution for that is called "memoizing", where you figure out a place to store old values of arguments and results, and check there first before calling the function.
In calling library functions like DGEMM (lapack matrix-multiply) that one would assume are optimized to the teeth, they are actually spending a large fraction of time calling a function to determine if the matrices are upper or lower triangle, square, symmetric, or whatever, rather than actually doing the multiplication. If so, the answer is obvious - write a special routine just for your situation.
Don't say "but I don't have those problems". Of course - you probably have different problems - but the process of finding them is the same.
Once you've made it as fast as possible in single-thread, then figure out how to parallelize it. Multi-threading can have high overhead, so it's best not to tightly-couple the threads.
Regarding your question about converting from doubles to integers, the other answers are right on the money. It only makes sense in very particular situations.
It is well-known that the processor instruction for multiplication takes several times more time than addition, division is even worse (UPD: which is not true any more, see below). What about more complex operations like exponent? How difficult are they?
Motivation. I am interested because it would help in algorithm design to estimate performance-critical parts of algorithms on early stage. Suppose I want to apply a set of filters to an image. One of them operates on 3×3 neighborhood of each pixel, sums them and takes atan. Another one sums more neighbouring pixels, but does not use complicated functions. Which one would execute longer?
So, ideally I want to have approximate relative times of elementary operations execution, like multiplication typically takes 5 times more time than addition, exponent is about 100 multiplications. Of course, it is a deal of orders of magnitude, not the exact values. I understand that it depends on the hardware and on the arguments, so let's say we measure average time (in some sense) for floating-point operations on modern x86/x64. For operations that are not implemented in hardware, I am interested in typical running time for C++ standard libraries.
Have you seen any sources when such thing was analyzed? Does this question makes sense at all? Or no rules of thumb like this could be applied in practice?
First off, let's be clear. This:
It is well-known that processor instruction for multiplication takes
several times more time than addition
is no longer true in general. It hasn't been true for many, many years, and needs to stop being repeated. On most common architectures, integer multiplies are a couple cycles and integer adds are single-cycle; floating-point adds and multiplies tend to have nearly equal timing characteristics (typically around 4-6 cycles latency, with single-cycle throughput).
Now, to your actual question: it varies with both the architecture and the implementation. On a recent architecture, with a well written math library, simple elementary functions like exp and log usually require a few tens of cycles (20-50 cycles is a reasonable back-of-the-envelope figure). With a lower-quality library, you will sometimes see these operations require a few hundred cycles.
For more complicated functions, like pow, typical timings range from high tens into the hundreds of cycles.
You shouldn't be concerned about this. If I tell you that a typical C library implementation of transcendental functions tend to take around 10 times a single floating point addition/multiplication (or 50 floating point additions/multiplications), and around 5 times a floating point division, this wouldn't be useful to you.
Indeed, the way your processor schedules memory accesses will interfere badly with any premature optimization you'd do.
If after profiling you find that a particular implementation using transcendental functions is too slow, you can contemplate setting up a polynomial interpolation scheme. This will include a table and therefore will incur extra cache issues, so make sure to measure and not guess.
This will likely involve Chebyshev approximation. Document yourself about it, this is a particularly useful technique in this kind of domains.
I have been told that compilers are quite bad in optimizing floating point code. You may want to write custom assembly code.
Also, Intel Performance Primitives (if you are on Intel CPU) is something good to own if you are ready to trade off some accuracy for speed.
You could always start a second thread and time the operations. Most elementary operations don't have that much difference in execution time. The big difference is how many times the are executed. The O(n) is generally what you should be thinking about.
Profiling some C++ number crunching code with both gprof and kcachegrind gives similar results for the functions that contribute most to the execution time (50-80% depending on input) but for functions between 10-30% both these tools give different results. Does it mean one of them is not reliable? What would yo do here?
gprof is actually quite primitive. Here's what it does.
1) It samples the program counter at a constant rate and records how many samples land in each function (exclusive time).
2) It counts how many times any function A calls any function B.
From that it can find out how many times each function was called in total, and what it's average exclusive time was.
To get average inclusive time of each function it propagates exclusive time upward in the call graph.
If you're expecting this to have some kind of accuracy, you should be aware of some issues.
First, it only counts CPU-time-in-process, meaning it is blind to I/O or other system calls.
Second, recursion confuses it.
Third, the premise that functions always adhere to an average run time, no matter when they are called or who calls them, is very suspect.
Forth, the notion that functions (and their call graph) are what you need to know about, rather than lines of code, is simply a popular assumption, nothing more.
Fifth, the notion that accuracy of measurement is even relevant to finding "bottlenecks" is also just a popular assumption, nothing more.
Callgrind can work at the level of lines - that's good. Unfortunately it shares the other problems.
If your goal is to find "bottlenecks" (as opposed to getting general measurements), you should take a look at wall-clock time stack samplers that report percent-by-line, such as Zoom.
The reason is simple but possibly unfamiliar.
Suppose you have a program with a bunch of functions calling each other that takes a total of 10 seconds. Also, there is a sampler that samples, not just the program counter, but the entire call stack, and it does it all the time at a constant rate, like 100 times per second. (Ignore other processes for now.)
So at the end you have 1000 samples of the call stack.
Pick any line of code L that appears on more than one of them.
Suppose you could somehow optimize that line, by avoiding it, removing it, or passing it off to a really really fast processor.
What would happen to those samples?
Since that line of code L now takes (essentially) no time at all, no sample can hit it, so those samples would just disappear, reducing the total number of samples, and therefore the total time!
In fact the overall time would be reduced by the fraction of time L had been on the stack, which is roughly the fraction of samples that contained it.
I don't want to get too statistical, but many people think you need a lot of samples, because they think accuracy of measurement is important.
It isn't, if the reason you're doing this is to find out what to fix to get speedup.
The emphasis is on finding what to fix, not on measuring it.
Line L is on the stack some fraction F of the time, right?
So each sample has a probability F of hitting it, right? Just like flipping a coin.
There is a theory of this, called the Rule of Succession.
It says that (under simplifying but general assumptions), if you flip a coin N times, and see "heads" S times, you can estimate the fairness of the coin F as (on average) (S+1)/(N+2).
So, if you take as few as three samples, and see L on two of them, do you know what F is? Of course not.
But you do know on average it is (2+1)/(3+2) or 60%.
So that's how much time you could save (on average) by "optimizing away" line L.
And, of course, the stack samples showed you exactly where line L (the "bottleneck"**) is.
Did it really matter that you didn't measure it to two or three decimal places?
BTW, it is immune to all the other problems mentioned above.
**I keep putting quotes around "bottleneck" because what makes most software slow has nothing in common with the neck of a bottle.
A better metaphor is a "drain" - something that just needlessly wastes time.
gprof's timing data is statistical (read about it in details of profiling docs).
On the other hand, KCacheGrind uses valgrind which actually interprets all the code.
So KCacheGrind can be "more accurate" (at the expense of more overhead) if the CPU modeled by valgrind is close to your real CPU.
Which one to choose also depends on what type of overhead you can handle. In my experience, gprof adds less runtime overhead (execution time that is), but it is more intrusive (i.e. -pg adds code to each and every one of your functions). So depending on the situation, on or the other is more appropriate.
For "better" gprof data, run your code longer (and on as wide a range of test data you can). The more you have, the better the measurements will be statistically.