I need to evaluate the sum of the row: 1/1+1/2+1/3+...+1/n. Considering that in C++ evaluations are not complete accurate, the order of summation plays important role. 1/n+1/(n-1)+...+1/2+1/1 expression gives the more accurate result.
So I need to find out the order of summation, which provides the maximum accuracy.
I don't even know where to begin.
Preferred language of realization is C++.
Sorry for my English, if there are any mistakes.
For large n you'd better use asymptotic formulas, like the ones on http://en.wikipedia.org/wiki/Harmonic_number;
Another way is to use exp-log transformation. Basically:
H_n = 1 + 1/2 + 1/3 + ... + 1/n = log(exp(1 + 1/2 + 1/3 + ... + 1/n)) = log(exp(1) * exp(1/2) * exp(1/3) * ... * exp(1/n)).
Exponents and logarithms can be calculated pretty quickly and accuratelly by your standard library. Using multiplication you should get much more accurate results.
If this is your homework and you are required to use simple addition, you'll better add from the smallest one to the largest one, as others suggested.
The reason for the lack of accuracy is the precision of the float, double, and long double types. They only store so many "decimal" places. So adding a very small value to a large value has no effect, the small term is "lost" in the larger one.
The series you're summing has a "long tail", in the sense that the small terms should add up to a large contribution. But if you sum in descending order, then after a while each new small term will have no effect (even before that, most of its decimal places will be discarded). Once you get to that point you can add a billion more terms, and if you do them one at a time it still has no effect.
I think that summing in ascending order should give best accuracy for this kind of series, although it's possible there are some odd corner cases where errors due to rounding to powers of (1/2) might just so happen to give a closer answer for some addition orders than others. You probably can't really predict this, though.
I don't even know where to begin.
Here: What Every Computer Scientist Should Know About Floating-Point Arithmetic
Actually, if you're doing the summation for large N, adding in order from smallest to largest is not the best way -- you can still get into a situation where the numbers you're adding are too small relative to the sum to produce an accurate result.
Look at the problem this way: You have N summations, regardless of ordering, and you wish to have the least total error. Thus, you should be able to get the least total error by minimizing the error of each summation -- and you minimize the error in a summation by adding values as nearly close to each other as possible. I believe that following that chain of logic gives you a binary tree of partial sums:
Sum[0,i] = value[i]
Sum[1,i/2] = Sum[0,i] + Sum[0,i+1]
Sum[j+1,i/2] = Sum[j,i] + Sum[j,i+1]
and so on until you get to a single answer.
Of course, when N is not a power of two, you'll end up with leftovers at each stage, which you need to carry over into the summations at the next stage.
(The margins of StackOverflow are of course too small to include a proof that this is optimal. In part because I haven't taken the time to prove it. But it does work for any N, however large, as all of the additions are adding values of nearly identical magnitude. Well, all but log(N) of them in the worst not-power-of-2 case, and that's vanishingly small compared to N.)
http://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic
You can find libraries with ready for use implementation for C/C++.
For example http://www.apfloat.org/apfloat/
Unless you use some accurate closed-form representation, a small-to-large ordered summation is likely to be most accurate simple solution (it's not clear to me why a log-exp would help - that's a neat trick, but you're not winning anything with it here, as far as I can tell).
You can further gain precision by realizing that after a while, the sum will become "quantized": Effectively, when you have 2 digits of precision, adding 1.3 to 41 results in 42, not 42.3 - but you achieve almost a precision doubling by maintaining an "error" term. This is called Kahan Summation. You'd compute the error term (42-41-1.3 == -0.3) and correct that in the next addition by adding 0.3 to the next term before you add it in again.
Kahan Summation in addition to a small-to-large ordering is liable to be as accurate as you'll ever need to get. I seriously doubt you'll ever need anything better for the harmonic series - after all, even after 2^45 iterations (crazy many) you'd still only be dealing with a numbers that are at least 1/2^45 large, and a sum that's on the order of 45 (<2^6), for an order of magnitude difference of 51 powers-of-two - i.e. even still representable in a double precision variable if you add in the "wrong" order.
If you go small-to-large, and use Kahan Summation, the sun's probably going to extinguish before today's processors reach a percent of error - and you'll run into other tricky accuracy issues just due to the individual term error on that scale first anyhow (being that a number of the order of 2^53 or larger cannot be represented accurately as a double at all anyhow.)
I'm not sure about the order of summation playing an important role, I havent heard that before. I guess you want to do this in floating point arithmetic so the first thing is to think more inline of (1.0/1.0 + 1.0/2.0+1.0/3.0) - otherwise the compiler will do integer division
to determine order of evaluation, maybe a for loop or brackets?
e.g.
float f = 0.0;
for (int i=n; i>0; --i)
{
f += 1.0/static_cast<float>(i);
}
oh forgot to say, compilers will normally have switches to determine floating point evaluation mode. this is maybe related to what you say on order of summation - in visual C+ these are found in code-generation compile settings, in g++ there're options -float that handle this
actually, the other guy is right - you should do summation in order of smallest component first; so
1/n + 1/(n-1) .. 1/1
this is because the precision of a floating point number is linked to the scale, if you start at 1 you'll have 23 bits of precision relative to 1.0. if you start at a smaller number the precision is relative to the smaller number, so you'll get 23 bits of precision relative to 1xe-200 or whatever. then as the number gets bigger rounding error will occur, but the overall error will be less than the other direction
As all your numbers are rationals, the easiest (and also maybe the fastest, as it will have to do less floating point operations) would be to do the computations with rationals (tuples of 2 integers p,q), and then do just one floating point division at the end.
update to use this technique effectively you will need to use bigints for p & q, as they grow quite fast...
A fast prototype in Lisp, that has built in rationals shows:
(defun sum_harmonic (n acc)
(if (= n 0) acc (sum_harmonic (- n 1) (+ acc (/ 1 n)))))
(sum_harmonic 10 0)
7381/2520
[2.9289682]
(sum_harmonic 100 0)
14466636279520351160221518043104131447711/278881500918849908658135235741249214272
[5.1873775]
(sum_harmonic 1000 0)
53362913282294785045591045624042980409652472280384260097101349248456268889497101
75750609790198503569140908873155046809837844217211788500946430234432656602250210
02784256328520814055449412104425101426727702947747127089179639677796104532246924
26866468888281582071984897105110796873249319155529397017508931564519976085734473
01418328401172441228064907430770373668317005580029365923508858936023528585280816
0759574737836655413175508131522517/712886527466509305316638415571427292066835886
18858930404520019911543240875811114994764441519138715869117178170195752565129802
64067621009251465871004305131072686268143200196609974862745937188343705015434452
52373974529896314567498212823695623282379401106880926231770886197954079124775455
80493264757378299233527517967352480424636380511370343312147817468508784534856780
21888075373249921995672056932029099390891687487672697950931603520000
[7.485471]
So, the next better option could be to mantain the list of floating points and to reduce it summing the two smallest numbers in each step...
Related
Consider the two cases below:
// Case 1
double val { initial_value };
for (int i { 0 }; i < n; ++i) {
val += step;
foo(val);
}
// Case 2
for (int i { 0 }; i < n; ++i) {
double val = initial_value + i * step;
foo(val);
}
where n is the number of steps, initial_value is some given value of type double, step is some predetermined value of type double and val is a variable used in subsequent call for the function foo. Which of the cases produces less of floating-point error? My guess would be the second one, as there are only one addition and multiplication, while the first case incurs the floating-point representation error from all of the n additions. I am asking this question because I didn't know what to search for. Does there exists some good reference for cases like these?
In practice the variable val is to be used in the loop of the both cases. I didn't include any example for this as I'm only interested in the floating-point error.
Considering the comment by supercat (emphasis mine):
The point is that in many scenarios one might want a sequence of values that are uniformly spaced between specified start and end points. Using the second approach would yield values that are as uniformly spaced as possible between the start point and an end value that's near a desired one, but may not quite match.
And the one by Bathsheba:
Both are flawed. You should compute the start and end, then compute each value as a function of those. The problem with the second way is you multiply the error in step. The former accumulates errors.
I'd suggest a couple of alternatives.
Since C++20, the Standard Library provides std::lerp where std::lerp(a, b, t) returns "the linear interpolation between a and b for the parameter t (or extrapolation, when t is outside the range [0,1])".
A formula like value = (a * (n - i) + b * i) / n; may result in a more uniform1 distribution of the intermediate values.
(1) Here I tried to test all those approaches for different extremes and number of sample points. The program compares the values generated by each algorithm when applied in the opposite directions (first from left to right, then from right to left). It shows the average and variance of the sum of the absolute difference between the values of the intermediate points.
Other metrics may yield different results.
Option 2 has significantly lower error.
How much? Well, let's assume an initial_value of 0 for simplicity sake at first. You have 53 significant bits, and how quickly you will see rounding errors depends how quickly we can manage to shift these off the far end during addition.
So let's pick step such that the significant bits are ideally all 1s: 0.999999999999999999999999.
Now the rounding error is log2(val/step) bits from the far end of step during each single addition. Not much during the first iteration, but the error becomes noticable rather quickly.
Picking a huge initial_value and the error can become quite extreme. For initial_value >= pow(2, 53) * step, your first loop even fails to change val at all in between iterations.
Your second loop still handles that correctly.
Consider an extreme case. Suppose that initial_value is much larger than step. Much, much larger. So large that initial_value + step == initial_value due to the limits of floating point representation. However, we do not want this "extreme" case to get too extreme. Put a cap on initial_value, say keep it small enough to have initial_value + (2*step) != initial_value. (Some people might call this putting step between a certain epsilon and half that epsilon, but I would get the terminology mixed up.) Now run through your code.
In the first loop, val will equal initial_value every iteration as no operation is performed that will change its value. In contrast, the second loop will eventually have a different value for val, if there are enough iterations. Therefore, the second option, the one that calculates initial_value + i * step is more accurate in this extreme case.
We should also look at the opposite extremity. Suppose that initial_value is so small relative to step that initial_value + step == step. In this case, initial_value might as well be zero, and the question simplifies to asking if there is a more accurate way to calculate i*step than by multiplying i and step. (If there is, I might want a new compiler.) Therefore, the second option is not worse than the first in this extreme case.
Extreme case analysis is not conclusive, but it often reveals trends. I pushed the calculation to opposite extremes, and the second option varied from definitely better to definitely not worse. I'd be willing to conclude that the second option produces less error.
Caveats: It might be that the size of the error is negligible and not worth coding around. Also, the question has limited scope, ignoring other considerations (such as from where step came; if it is the result of dividing by n, there might be even better alternatives). Still, in the narrow scenario presented by the question, calculating initial_value + i*step each iteration looks like the way to get minimal numerical error.
Including <cmath> and using std::fma(i, step, initial_value) will always produce the best result, presuming i is not so large that converting it to the floating-point type has a rounding error. This is because fma is specified to produce a result equivalent to computing the real-arithmetic value of i•step + initial_value and then rounding it to the nearest representable value. It does not have an internal rounding after the multiplication and before the addition, so it produces the best result that is representable in the floating-point type.
Between the multiplication method and the addition method, multiplication is generally preferred. It is possible for addition to produce a better result. Presuming IEEE-754 double precision binary, an example is easily constructed as initial_value = -1./3, i = 3, and step = 1./3. Then in initial_value + step + step + step, initial_value + step produces exactly zero (so there is no rounding error), adding step has no error, and the second add merely doubles step, which also has no error. So addition produces a final result with no error. In contrast, in initial_value + 3*step, 3*step has a rounding error, and it persists through the addition.
However, outside of deliberately constructed examples, multiplication will commonly produce better results than addition, simply because it uses fewer operations, many fewer in most cases. Typically, the rounding errors in repeated additions will act like a random walk, sometimes increasing the accumulated error and sometimes decreasing it. A random walk can sometimes return to the origin but does so rarely. So it is rare that a sequence with many additions will have accumulated error closer to the origin (zero error) than an expression with one multiplication and one addition.
so the problem I have is that there is two integers (a, b) that is in [1, 10^16] interval and I need to do find out how many digits will number a^b have? Those numbers are too big for saving them on single variables, and if I write them on Array it would take a lot of time.
Is there a way to count the number a^b number of digits with some kind of formula or any simpler way then Arrays?
after fixing the one-off error suggested in the comments
number of digits of a^b = floor( b * log(a) ) + 1
karakfa has it right.
The base-k logarithm of a number n, rounded up to the nearest whole number, will give you the number of digits required to represent n in base k.
EDIT: as pointed out in comments, it should not be rounded up, but rounded down and then incremented by one. This accounts for round powers of 10 having an extra digit.
If your number is a^b then take the base-10 logarithm, log a^b and use the laws of logarithms to simplify as b log a. Note that this simplification happens inside the ceiling function so the simplification is valid. Computing log a should not be an issue (it will be between 0 and 16) and b is known. Just make sure to round after multiplying, not before.
Note that limited precision of floating-point numbers may introduce some errors into this method. If the true value of b x log a is different from the nearest floating-point representation of b x log a in such a way that they fall on different sides of an integer, the method fails. You can possibly detect when you are close to this condition and remediate it somehow.
You could use a library that supports arbitrarily large numbers, like GMP .
The core C++ language itself offers no types to work with such large numbers. So either you use a pre-existing library or write one yourself (I suggest the former - don't re-invent the wheel).
In C++ programming, when do I need to worry about the precision issue? To take a small example (it might not be a perfect one though),
std::vector<double> first (50000, 0.0);
std::vector<double> second (first);
Could it be possible that second[619] = 0.00000000000000000000000000001234 (I mean a very small value). Or SUM = second[0]+second[1]+...+second[49999] => 1e-31? Or SUM = second[0]-second[1]-...-second[49999] => -7.987654321e-12?
My questions:
Could it be some small disturbances in working with the double type numbers?
What may cause these kind of small disturbances? i.e. rounding errors become large? Could you please list them? How to take precautions?
If there could be small disturbance in certain operations, does it then mean after these operations, using if (SUM == 0) is dangerous? One should then always use if (SUM < SMALL) instead, where SMALL is defined as a very small value, such as 1E-30?
Lastly, could the small disturbances result into a negative value? Because if it is possible, then I should be better use if (abs(SUM) < SMALL) instead.
Any experiences?
This is a good reference document for floating point precision: What Every Computer Scientist Should Know About Floating-Point Arithmetic
One of the more important parts is catastrophic cancellation
Catastrophic cancellation occurs when the operands are subject to
rounding errors. For example in the quadratic formula, the expression
b2 - 4ac occurs. The quantities b2 and 4ac are subject to rounding
errors since they are the results of floating-point multiplications.
Suppose that they are rounded to the nearest floating-point number,
and so are accurate to within .5 ulp. When they are subtracted,
cancellation can cause many of the accurate digits to disappear,
leaving behind mainly digits contaminated by rounding error. Hence the
difference might have an error of many ulps. For example, consider b =
3.34, a = 1.22, and c = 2.28. The exact value of b2 - 4ac is .0292. But b2 rounds to 11.2 and 4ac rounds to 11.1, hence the final answer
is .1 which is an error by 70 ulps, even though 11.2 - 11.1 is exactly
equal to .16. The subtraction did not introduce any error, but rather
exposed the error introduced in the earlier multiplications.
Benign cancellation occurs when subtracting exactly known quantities.
If x and y have no rounding error, then by Theorem 2 if the
subtraction is done with a guard digit, the difference x-y has a very
small relative error (less than 2).
A formula that exhibits catastrophic cancellation can sometimes be
rearranged to eliminate the problem. Again consider the quadratic
formula
For your specific example, 0 has an exact representation as a double, and adding exactly 0 to a double does not change its value.
Also, like any other values you put in variables, numbers that you initialize in the array are not going to mysteriously change. You only get rounding when the result of a calculation cannot be exactly represented as a floating point number.
To give a better opinion about "disturbances" I would need to know the kinds of calculations that your code performs.
I have two integer variables, partial and total. It is a progress, so partial starts at zero and goes up one-by-one to the value of total.
If I want to get a fraction value indicating the progress(from 0.0 to 1.0) I may do the following:
double fraction = double(partial)/double(total);
But if total is too big, the conversion to double may lose information.
Actually, the amount of lost information is tolerable, but I was wondering if there is a algorithm or a std function to get the fraction between two values losing less information.
The obvious answer is to multiply partial by some scaling factor;
100 is a frequent choice, since the division then gives the percent as
an integral value (rounded down). The problem is that if the values are
so large that they can't be represented precisely in a double, there's
also a good chance that the multiplication by the scaling factor will
overflow. (For that matter, if they're that big, the initial values
will overflow an int on most machines.)
Yes, there is an algorithm losing less information. Assuming you want to find the double value closest to the mathematical value of the fraction, you need an integer type capable of holding total << 53. You can create your own or use a library like GMP for that. Then
scale partial so that (total << 52) <= numerator < (total << 53), where numerator = (partial << m)
let q be the integer quotient numerator / total and r = numerator % total
let mantissa = q if 2*r < total, = q+1 if 2*r > total and if 2*r == total, mantissa = q+1 if you want to round half up, = q if you want to round half down, the even of the two if you want round-half-to-even
result = scalbn(mantissa, -m)
Most of the time you get the same value as for (double)partial / (double)total, differences of one least significant bit are probably not too rare, two or three LSB difference wouldn't surprise me either, but are rare, a bigger difference is unlikely (that said, somebody will probably give an example soon).
Now, is it worth the effort? Usually not.
If you want a precise representation of the fraction, you'd have some sort of structure containing the numerator and the denominator as integers, and, for unique representation, you'd just factor out the greatest common divisor (with a special case for zero). If you are just worried that after repeated operations the floating point representation might not be accurate enough, you need to just find some courses on numerical analysisas that issue isn't strictly a programming issue. There are better ways than others to calculate certain results, but I can't really go into them (I've never done the coursework, just read about it).
I am writing a simulation program that proceeds in discrete steps. The simulation consists of many nodes, each of which has a floating-point value associated with it that is re-calculated on every step. The result can be positive, negative or zero.
In the case where the result is zero or less something happens. So far this seems straightforward - I can just do something like this for each node:
if (value <= 0.0f) something_happens();
A problem has arisen, however, after some recent changes I made to the program in which I re-arranged the order in which certain calculations are done. In a perfect world the values would still come out the same after this re-arrangement, but because of the imprecision of floating point representation they come out very slightly different. Since the calculations for each step depend on the results of the previous step, these slight variations in the results can accumulate into larger variations as the simulation proceeds.
Here's a simple example program that demonstrates the phenomena I'm describing:
float f1 = 0.000001f, f2 = 0.000002f;
f1 += 0.000004f; // This part happens first here
f1 += (f2 * 0.000003f);
printf("%.16f\n", f1);
f1 = 0.000001f, f2 = 0.000002f;
f1 += (f2 * 0.000003f);
f1 += 0.000004f; // This time this happens second
printf("%.16f\n", f1);
The output of this program is
0.0000050000057854
0.0000050000062402
even though addition is commutative so both results should be the same. Note: I understand perfectly well why this is happening - that's not the issue. The problem is that these variations can mean that sometimes a value that used to come out negative on step N, triggering something_happens(), now may come out negative a step or two earlier or later, which can lead to very different overall simulation results because something_happens() has a large effect.
What I want to know is whether there is a good way to decide when something_happens() should be triggered that is not going to be affected by the tiny variations in calculation results that result from re-ordering operations so that the behavior of newer versions of my program will be consistent with the older versions.
The only solution I've so far been able to think of is to use some value epsilon like this:
if (value < epsilon) something_happens();
but because the tiny variations in the results accumulate over time I need to make epsilon quite large (relatively speaking) to ensure that the variations don't result in something_happens() being triggered on a different step. Is there a better way?
I've read this excellent article on floating point comparison, but I don't see how any of the comparison methods described could help me in this situation.
Note: Using integer values instead is not an option.
Edit the possibility of using doubles instead of floats has been raised. This wouldn't solve my problem since the variations would still be there, they'd just be of a smaller magnitude.
I've worked with simulation models for 2 years and the epsilon approach is the sanest way to compare your floats.
Generally, using suitable epsilon values is the way to go if you need to use floating point numbers. Here are a few things which may help:
If your values are in a known range you and you don't need divisions you may be able to scale the problem and use exact operations on integers. In general, the conditions don't apply.
A variation is to use rational numbers to do exact computations. This still has restrictions on the operations available and it typically has severe performance implications: you trade performance for accuracy.
The rounding mode can be changed. This can be use to compute an interval rather than an individual value (possibly with 3 values resulting from round up, round down, and round closest). Again, it won't work for everything but you may get an error estimate out of this.
Keeping track of the value and a number of operations (possible multiple counters) may also be used to estimate the current size of the error.
To possibly experiment with different numeric representations (float, double, interval, etc.) you might want to implement your simulation as templates parameterized for the numeric type.
There are many books written on estimating and minimizing errors when using floating point arithmetic. This is the topic of numerical mathematics.
Most cases I'm aware of experiment briefly with some of the methods mentioned above and conclude that the model is imprecise anyway and don't bother with the effort. Also, doing something else than using float may yield better result but is just too slow, even using double due to the doubled memory footprint and the smaller opportunity of using SIMD operations.
I recommend that you single step - preferably in assembly mode - through the calculations while doing the same arithmetic on a calculator. You should be able to determine which calculation orderings yield results of lesser quality than you expect and which that work. You will learn from this and probably write better-ordered calculations in the future.
In the end - given the examples of numbers you use - you will probably need to accept the fact that you won't be able to do equality comparisons.
As to the epsilon approach you usually need one epsilon for every possible exponent. For the single-precision floating point format you would need 256 single precision floating point values as the exponent is 8 bits wide. Some exponents will be the result of exceptions but for simplicity it is better to have a 256 member vector than to do a lot of testing as well.
One way to do this could be to determine your base epsilon in the case where the exponent is 0 i e the value to be compared against is in the range 1.0 <= x < 2.0. Preferably the epsilon should be chosen to be base 2 adapted i e a value that can be exactly represented in a single precision floating point format - that way you know exactly what you are testing against and won't have to think about rounding problems in the epsilon as well. For exponent -1 you would use your base epsilon divided by two, for -2 divided by 4 and so on. As you approach the lowest and the highest parts of the exponent range you gradually run out of precision - bit by bit - so you need to be aware that extreme values can cause the epsilon method to fail.
If it absolutely has to be floats then using an epsilon value may help but may not eliminate all problems. I would recommend using doubles for the spots in the code you know for sure will have variation.
Another way is to use floats to emulate doubles, there are many techniques out there and the most basic one is to use 2 floats and do a little bit of math to save most of the number in one float and the remainder in the other (saw a great guide on this, if I find it I'll link it).
Certainly you should be using doubles instead of floats. This will probably reduce the number of flipped nodes significantly.
Generally, using an epsilon threshold is only useful when you are comparing two floating-point number for equality, not when you are comparing them to see which is bigger. So (for most models, at least) using epsilon won't gain you anything at all -- it will just change the set of flipped nodes, it wont make that set smaller. If your model itself is chaotic, then it's chaotic.