I intend to use half-precision floating-point in my code but I am not able to figure out how to declare them. For Example, I want to do something like the following-
fp16 a_fp16;
bfloat a_bfloat;
However, the compiler does not seem to know these types (fp16 and bfloat are just dummy types, for demonstration purposes)
I remember reading that bfloat support was added into GCC-10, but I am not able to find it in the manual.I am especially interested in bfloat floating numbers
Additional Questions -
FP16 now has hardware support on Intel / AMD support as today? I think native hardware support was added since Ivy Bridge itself. (https://scicomp.stackexchange.com/questions/35187/is-half-precision-supported-by-modern-architecture)
I wanted to confirm whether using FP16 will indeed increase FLOPs. I remember reading somewhere that all arithmetic operations on fp16 are internally converted to fp32 first, and only affect cache footprint and bandwidth.
SIMD intrinsic support for half precision float, especially bfloat(I am aware of intrinsics like _mm256_mul_ph, but not sure how to pass the 16bit FP datatype, would really appreciate if someone could highlight this too)
Are these types added to Intel Compilers as well ?
PS - Related Post - Half-precision floating-point arithmetic on Intel chips , but it does not cover on declaring half precision floating point numbers.
TIA
Neither C++ nor C language has arithmetic types for half floats.
The GCC compiler supports half floats as a language extension. Quote from the documentation:
On x86 targets with SSE2 enabled, GCC supports half-precision (16-bit) floating point via the _Float16 type. For C++, x86 provides a builtin type named _Float16 which contains same data format as C.
...
On x86 targets with SSE2 enabled, without -mavx512fp16, all operations will be emulated by software emulation and the float instructions. The default behavior for FLT_EVAL_METHOD is to keep the intermediate result of the operation as 32-bit precision. This may lead to inconsistent behavior between software emulation and AVX512-FP16 instructions. Using -fexcess-precision=16 will force round back after each operation.
Using -mavx512fp16 will generate AVX512-FP16 instructions instead of software emulation. The default behavior of FLT_EVAL_METHOD is to round after each operation. The same is true with -fexcess-precision=standard and -mfpmath=sse. If there is no -mfpmath=sse, -fexcess-precision=standard alone does the same thing as before, It is useful for code that does not have _Float16 and runs on the x87 FPU.
Related
I'm asking this question because I'm trying to achieve bitwise (hash) equality between Visual Studio 2017 (cl.exe) and gcc 5.4.0. The problematic function makes use of sin() and cos(). All variables are double, and FMAD is also relevant.
I've been reading extensively on SO and the web about floating point determinism, reproducibility, and lock-stock MP game design. I'm aware that single-compiler, single-build determinism is not hard, but I am attempting 2-compiler, single-build determinism.
Efficiency is not a concern here. I just want the results to match.
I ask because I hope to narrow my concerns for what to test/try.
Are these things relevant for x86_64 processors and builds?
functions that control the x87 fpu
XPFPA_{DECLARE,RESTORE,etc}
"<"fpu_control.h>, _FPU_SINGLE, _FPU_DOUBLE, etc.
_controlfp_s(), _PC24, _PC53, _PC_64
I ask because I have read that platforms with SSE (x86_64) default to using SSE for floating point, so fpu control functions should be irrelevant?
I have found this and this to be most informative. This MSDN article says setting the floating point precision mask is not supported on x64 arch. And this SO post says SSE has fixed precision.
My testing has shown that /fp:{strict,precise,fast} are not changing the hashes. Neither is optimization level. So I'm hoping to narrow my focus to sin, cos.
Most floating point functions have to perform rounding one way or an other. The C/C++ standard is rather vague on the subject, and IEEE conformance is not strict enough on trigonometric functions. Which means that in practice it is useless to try to squeeze correct rounding out of your compilers default math implementation in a portable way.
For instance, the libm implementation (used by gcc) of sin/cos is written in assembly and the algorithm is different for different architectures and most probably depends on the version of the library.
You therefore have two possibilities:
implement your own sin/cos using only floating point operations with exact rounding (fused multiply-accumulate + Taylor series)
use a 3rd party library with strong rounding considerations
I personally use the MPFR library as a gold standard when dealing with rounding errors. There will be a runtime cost, although I never tried to benchmark it against libm performance.
Custom Implementation
Note that if you decide to implement it yourself, you need to choose the rounding mode and inform the compiler that it matter to you.
In C++ it is done this way:
#include <cfenv>
#pragma STDC FENV_ACCESS ON
#pragma STDC FP_CONTRACT OFF
int main(int, char**) {
...
if(!std::fesetround(FE_TONEAREST))
throw std::runtime_error("fesetround failed!");
...
}
The question is in the title. It seems that the software I deliver to my customer has varying behaviors depending on wether some parameters passed as integers or as floats. I build a DLL for my customer with MinGW and he integrates it in his Visual Studio project, which uses some other compiler (no idea which, I guess the standard one for VS).
Could it be that floats are represented differently by him than by me ?
Thanks for the heads up,
Charles
Yes, floating point representation is compiler dependent.
In theory you can use std::numeric_limits to determine the main aspects of the representation, such as whether it's IEEE 754, or whether it's binary or decimal.
In practice you can't rely on that except for the memory layout, because with a main compiler, g++, the semantics of floating point operations are influenced strongly by the optimization options (e.g., whether NaN compares equal to itself or not).
I.e. in practice it's not only compiler dependent but also option-dependent.
Happily compilers for a given platform will generally conform to that platform's standard memory layout for floating point, e.g. IEEE 754 in Windows (the standard originated on the PC platform). And so floating point values should in general work OK when exchanged between g++ and Visual C++, say. One exception is that with g++ long double maps to 80-bit IEEE 754, while with Visual C++ it maps to ordinary double, i.e. 64-bit, and that could conceivably be what makes trouble for you.
If you are programming with the C language for a microprocessor that does not have an FPU, does the compiler signal errors when floating point literals and keywords are encountered (0.75, float, double, etc)?
Also, what happens if the result of an expression is fractional?
I understand that there are software libraries that are used so you can do floating-point math, but I am specifically wondering what the results will be if you did not use one.
Thanks.
A C implementation is required to implement the types float and double, and arithmetic expressions involving them. So if the compiler knows that the target architecture doesn't have floating-point ops then it must bring in a software library to do it. The compiler is allowed to link against an external library, it's also allowed to implement floating point ops in software by itself as intrinsics, but it must somehow generate code to get it done.
If it doesn't do so [*] then it is not a conforming C implementation, so strictly speaking you're not "programming with the C language". You're programming with whatever your compiler docs tell you is available instead.
You'd hope that code involving float or double types will either fail to compile (because the compiler knows you're in a non-conforming mode and tells you) or else fails to link (because the compiler emits calls to emulation routines in the library, but the library is missing). But you're on your own as far as C is concerned, if you use something that isn't C.
I don't know the exact details (how old do I look?), but I imagine that back in the day if you took some code compiled for x87 then you might be able to link and load it on a system using an x86 with no FPU. Then the CPU would complain about an illegal instruction when you tried to execute it -- quite possibly the system would hang depending what OS you were running. So the worst possible case is pretty bad.
what happens if the result of an expression is fractional?
The actual result of an expression won't matter, because the expression itself was either performed with integer operations (in which case the result is not fractional) or else with floating-point operations (in which case the problem arises before you even find out the result).
[*] or if you fail to specify the options to make it do so ;-)
Floating-point is a required part of the C language, according to the C standard. If the target hardware does not have floating-point instructions, then a C implementation must provide floating-point operations in some other way, such as by emulating them in software. (All calculations are just functions of bits. If you have elementary operations for manipulating bits and performing tests and branches, then you can compute any function that a general computer can.)
A compiler could provide a subset of C without floating-point, but then it would not be a standard-compliant C compiler.
Software floating point can take two forms:
a compiler may generate calls to built-in floating point functions directly - for example the operation 1.2 * 2.5 may invoke (for example) fmul( 1.2, 2.5 ),
alternatively for architectures that support an FPU, but for which some device variants may omit it, it is common to use FPU emulation. When an FP instruction is encountered an invalid instruction exception will occur and the exception handler will vector to code that emulates the instruction.
FPU emulation has the advantage that when the same code is executed on a device with a real FPU, it will be used automatically and accelerate execution. However without an FPU there is usually a small overhead compared with direct software implementation, so if the application is never expected to run on an FPU, emulation might best be avoided is the compiler provides the option.
Software floating point is very much slower that hardware supported floating point. Use of fixed-point techniques can improve performance with acceptable precision in many cases.
Typically, such microprocessor comes along either with a driver-package or even with a complete BSP (board-support-package, consisting of drivers and OS linked together), both of which contain FP library routines.
The compiler replaces every floating-point operation with an equivalent function call. This should be taken into consideration, especially when invoking such operations iteratively (inside a for / while loop), since the compiler cannot apply loop-unrolling optimization as a result.
The result of not including the required libraries within the project would be linkage errors.
I have a file spec (here: http://www.septentrio.com/secure/asterx1v_2_1/SBF%20Reference%20Guide.pdf) that has fields marked as both 32-bit and 64-bit floats (see page 8). How can I use both widths in my program? I am developing on Mac OSX right now but I will also deploy on a Linux machine.
More details:
I know I could tell the compiler the width, but how could I distinguish two different float widths? Maybe someone also has a suggestion for changing the way I parse, which is to reinterpret_cast(buffer+offset) and then use the values. These file sizes are huge (4GB) so I need performance.
This might seem obvious, nevertheless:
On Intel platform and many others float is 32-bit floating point value, and double is 64-bit floating point value. Try this approach. Most likely it will work.
To be absolutely sure check sizeof of your types at the start of your program or statically during compilation if your compiler allows this.
Once again, try the simple solution first.
Float and double arithmetic is both implemented on Intel and it is fast. In any case native arithmetic is the fastest of what you can get from the CPU.
IEEE 754 (http://en.wikipedia.org/wiki/IEEE_floating_point) defines not one floating point format, but several, like 4, 8, 16 bytes, etc. They all have different range and precision but they are all still IEEE values.
As per c99 standard, size of long long should be minimum 64 bits. How is this implemented in a 32 bit machine (eg. addition or multiplication of 2 long longs). Also, What is the equivalent of long long in C++.
The equivalent in C++ is long long as well. It's not required by the standard, but most compilers support it because it's so usefull.
How is it implemented? Most computer architectures already have built-in support for multi-word additions and subtractions. They don't do 64 bit addititions directly but use the carry flag and a special add-instruction to build a 64 bit add from two 32 bit adds.
The same extension exists for subtraction as well (the carry is called borrow in these cases).
Longword multiplications and divisions can be built from smaller multiplications without the help of carry-flags. Sometimes simply doing the operations bit by bit is faster though.
There are architectures that don't have any flags at all (some DSP chips and simple micros). On these architectures the overflow has to be detected with logic operations. Multi-word arithmetic tend to be slow on these machines.
On the IA32 architecture, 64-bit integer are implemented in using two 32-bit registers (eax and edx).
There are platform specific equivalents for C++, and you can use the stdint.h header where available (boost provides you with one).
As everyone has stated, a 64-bit integer is typically implemented by simply using two 32-bit integers together. Then clever code generation is used to keep track of the carry and/or borrow bits to keep track of overflow, and adjust accordingly.
This of course makes such arithmetic more costly in terms of code space and execution time, than the same code compiled for an architecture with native support for 64-bit operations.
If you care about bit-sizes, you should use
#include <stdint.h>
int32_t n;
and friends. This works for C++ as well.
64-bit numbers on 32-bit machines are implemented as you think,
by 4 extra bytes. You could therefore implement your own 64-bit
datatype by doing something like this:
struct my_64bit_integer {
uint32_t low;
uint32_t high;
};
You would of course have to implement mathematical operators yourself.
There is an int64_t in the stdint.h that comes with my GCC version,
and in Microsoft Visual C++ you have an __int64 type as well.
The next C++ standard (due 2009, or maybe 2010), is slated to include the "long long" type. As mentioned earlier, it's already in common use.
The implementation is up to the compiler writers, although computers have always supported multiple precision operations. Some languages, like Python and Common Lisp, require support for indefinite-precision integers. Long ago, I wrote 64-bit multiplication and division routines for a computer (the Z80) that could manage 16-bit addition and subtraction, with no hardware multiplication at all.
Probably the easiest way to see how an operation is implemented on your particular compiler is to write a code sample and examine the assembler output, which is available from all the major compilers I've worked with.