Let's say that you are using <cstdint> and types like std::uint8_t and std::uint16_t, and want to do operations like += and *= on them. You'd like arithmetic on these numbers to wrap around modularly, like typical in C/C++. This ordinarily works, and you find experimentally works with std::uint8_t, std::uint32_t and std::uint64_t, but not std::uint16_t.
Specifically, multiplication with std::uint16_t sometimes fails spectacularly, with optimized builds producing all kinds of weird results. The reason? Undefined behavior due to signed integer overflow. The compiler is optimizing based upon the assumption that undefined behavior does not occur, and so starts pruning chunks of code from your program. The specific undefined behavior is the following:
std::uint16_t x = UINT16_C(0xFFFF);
x *= x;
The reason is C++'s promotion rules and the fact that you, like almost everyone else these days, are using a platform on which std::numeric_limits<int>::digits == 31. That is, int is 32-bit (digits counts bits but not the sign bit). x gets promoted to signed int, despite being unsigned, and 0xFFFF * 0xFFFF overflows for 32-bit signed arithmetic.
Demo of the general problem:
// Compile on a recent version of clang and run it:
// clang++ -std=c++11 -O3 -Wall -fsanitize=undefined stdint16.cpp -o stdint16
#include <cinttypes>
#include <cstdint>
#include <cstdio>
int main()
{
std::uint8_t a = UINT8_MAX; a *= a; // OK
std::uint16_t b = UINT16_MAX; b *= b; // undefined!
std::uint32_t c = UINT32_MAX; c *= c; // OK
std::uint64_t d = UINT64_MAX; d *= d; // OK
std::printf("%02" PRIX8 " %04" PRIX16 " %08" PRIX32 " %016" PRIX64 "\n",
a, b, c, d);
return 0;
}
You'll get a nice error:
main.cpp:11:55: runtime error: signed integer overflow: 65535 * 65535
cannot be represented in type 'int'
The way to avoid this, of course, is to cast to at least unsigned int before multiplying. Only the exact case where the number of bits of the unsigned type exactly equals half the number of bits of int is problematic. Any smaller would result in the multiplication being unable to overflow, as with std::uint8_t; any larger would result in the type exactly mapping to one of the promotion ranks, as with std::uint64_t matching unsigned long or unsigned long long depending on platform.
But this really sucks: it requires knowing which type is problematic based upon the size of int on the current platform. Is there some better way by which undefined behavior with unsigned integer multiplication can be avoided without #if mazes?
Some template metaprogramming with SFINAE, perhaps.
#include <type_traits>
template <typename T, typename std::enable_if<std::is_unsigned<T>::value && (sizeof(T) <= sizeof(unsigned int)) , int>::type = 0>
T safe_multiply(T a, T b) {
return (unsigned int)a * (unsigned int)b;
}
template <typename T, typename std::enable_if<std::is_unsigned<T>::value && (sizeof(T) > sizeof(unsigned int)) , int>::type = 0>
T safe_multiply(T a, T b) {
return a * b;
}
Demo.
Edit: simpler:
template <typename T, typename std::enable_if<std::is_unsigned<T>::value, int>::type = 0>
T safe_multiply(T a, T b) {
typedef typename std::make_unsigned<decltype(+a)>::type typ;
return (typ)a * (typ)b;
}
Demo.
Here's a relatively simple solution, which forces a promotion to unsigned int instead of int for unsigned type narrower than an int. I don't think any code is generated by promote, or at least no more code than the standard integer promotion; it will just force multiplication etc. to use unsigned ops instead of signed ones:
#include <type_traits>
// Promote to unsigned if standard arithmetic promotion loses unsignedness
template<typename integer>
using promoted =
typename std::conditional<std::numeric_limits<decltype(integer() + 0)>::is_signed,
unsigned,
integer>::type;
// function for template deduction
template<typename integer>
constexpr promoted<integer> promote(integer x) { return x; }
// Quick test
#include <cstdint>
#include <iostream>
#include <limits>
int main() {
uint8_t i8 = std::numeric_limits<uint8_t>::max();
uint16_t i16 = std::numeric_limits<uint16_t>::max();
uint32_t i32 = std::numeric_limits<uint32_t>::max();
uint64_t i64 = std::numeric_limits<uint64_t>::max();
i8 *= promote(i8);
i16 *= promote(i16);
i32 *= promote(i32);
i64 *= promote(i64);
std::cout << " 8: " << static_cast<int>(i8) << std::endl
<< "16: " << i16 << std::endl
<< "32: " << i32 << std::endl
<< "64: " << i64 << std::endl;
return 0;
}
This article regarding a C solution to the case of uint32_t * uint32_t multiplication on a system in which int is 64 bits has a really simple solution that I hadn't thought of: 32 bit unsigned multiply on 64 bit causing undefined behavior?
That solution, translated to my problem, is simple:
// C++
static_cast<std::uint16_t>(1U * x * x)
// C
(uint16_t) (1U * x * x)
Simply involving 1U in the left side of the chain of arithmetic operations like that will promote the first parameter to the larger rank of unsigned int and std::uint16_t, then so on down the chain. The promotion will ensure that the answer is both unsigned and that the requested bits remain present. The final cast then reduces it back to the desired type.
This is really simple and elegant, and I wish I had thought of it a year ago. Thank you to everyone who responded before.
Related
I want to use the following code in my program but gcc won't allow me to left shift my 1 beyond 31.
sizeof(long int) displays 8, so doesn't that mean I can left shift till 63?
#include <iostream>
using namespace std;
int main(){
long int x;
x=(~0 & ~(1<<63));
cout<<x<<endl;
return 0;
}
The compiling outputs the following warning:
left shift `count >= width` of type [enabled by default] `x=(~0 & ~(1<<63))`;
^
and the output is -1. Had I left shifted 31 bits I get 2147483647 as expected of int.
I am expecting all bits except the MSB to be turned on thus displaying the maximum value the datatype can hold.
Although your x is of type long int, the 1 is not. 1 is an int, so 1<<63 is indeed undefined.
Try (static_cast<long int>(1) << 63), or 1L << 63 as suggested by Wojtek.
You can't use 1 (int by default) to shift it beyond the int boundaries.
There's an easier way to get the "all bits except the MSB turned on" for a specific datatype
#include <iostream>
#include <limits>
using namespace std;
int main(){
unsigned long int max = std::numeric_limits<unsigned long int>::max();
unsigned long int max_without_MSB = max >> 1;
cout<< max_without_MSB <<endl;
return 0;
}
note the unsigned type. Without numeric_limits:
#include <iostream>
using namespace std;
int main() {
long int max = -1;
unsigned long int max_without_MSB = ((unsigned long int)max) >> 1;
cout << max_without_MSB << endl;
return 0;
}
Your title is misleading; a long can shift beyond 31 bits if a long is indeed that big. However your code shifts 1, which is an int.
In C++, the type of an expression is determined by the expression itself. An expression XXXXX has the same type regardless; if you later go double foo = XXXXX; it doesn't mean XXXXX is a double - it means a conversion happens from whatever XXXXX was, to double.
If you want to left-shift a long, then do that explicitly, e.g. 1L << 32, or ((long)1) << 32. Note that the size of long varies between platforms, so if you don't want your code to break when run on a different system then you'll have to take further measures, such as using fixed-width types, or shifting by CHAR_BIT * sizeof(long) - 1.
There is another issue with your intended code: 1L << 63 causes undefined behaviour if long is 64-bit or less. This is because of signed integer overflow; left-shift is defined the same as repeated multiplication of two, so attempting to "shift into the sign bit" causes an overflow.
To fix this, use unsigned types where it is fine to shift into the MSB, e.g. 1ul << 63.
Technically there is another issue in that ~0 doesn't do what you want if you are not on a 2's complement system, but these days it's pretty safe to ignore that case.
Looking at your overall intention with long x = ~0 & ~(1 << 63). A shorter way to write this is:
long x = LONG_MAX;
which is defined by <climits>. If you wanted 64-bit on all platforms then
int64_t x = INT64_MAX;
NB. If you do not intend to work with negative values then use unsigned long x and uint64_t respectively.
First let me state a few things about the shift, which is the source of your problem:
There is no guarantee that long int is actually 64 bit wide.
The most generic way I can think of is using std::numeric_limits:
static_cast<long int>(1) << (std::numeric_limits<long int>::digits - 1);
Now you can even make that a constexpr templated function:
template <typename Integer>
constexpr Integer foo()
{
return static_cast<Integer>(1) << (std::numeric_limits<Integer>::digits - 1);
}
So replacing the shift with static_cast<long int>(1) << (std::numeric_limits<long int>::digits - 1) will fix your issue, however there is a far better way:
std::numeric_limits includes a bunch of useful stuff, including:
std::numeric_limits<T>::max(); // the maximum value T can hold
std::numeric_limits<T>::min(); // the minimum value T can hold
std::numeric_limits<T>::digits; // the number of binary digits
std::numeric_limits<T>::is_signed(); // well, do I have to explain? ;-)
See cppreference.com for a complete list. You should prefer the facilities provided by the standard library, because it will most likely have fewer mistakes and other developers immediately know it.
The default datatype for a numeric value in C is integer unless explicitly mentioned.
Here you have to type cast the 1 as long int which would otherwise be an int.
I want to use the following code in my program but gcc won't allow me to left shift my 1 beyond 31.
sizeof(long int) displays 8, so doesn't that mean I can left shift till 63?
#include <iostream>
using namespace std;
int main(){
long int x;
x=(~0 & ~(1<<63));
cout<<x<<endl;
return 0;
}
The compiling outputs the following warning:
left shift `count >= width` of type [enabled by default] `x=(~0 & ~(1<<63))`;
^
and the output is -1. Had I left shifted 31 bits I get 2147483647 as expected of int.
I am expecting all bits except the MSB to be turned on thus displaying the maximum value the datatype can hold.
Although your x is of type long int, the 1 is not. 1 is an int, so 1<<63 is indeed undefined.
Try (static_cast<long int>(1) << 63), or 1L << 63 as suggested by Wojtek.
You can't use 1 (int by default) to shift it beyond the int boundaries.
There's an easier way to get the "all bits except the MSB turned on" for a specific datatype
#include <iostream>
#include <limits>
using namespace std;
int main(){
unsigned long int max = std::numeric_limits<unsigned long int>::max();
unsigned long int max_without_MSB = max >> 1;
cout<< max_without_MSB <<endl;
return 0;
}
note the unsigned type. Without numeric_limits:
#include <iostream>
using namespace std;
int main() {
long int max = -1;
unsigned long int max_without_MSB = ((unsigned long int)max) >> 1;
cout << max_without_MSB << endl;
return 0;
}
Your title is misleading; a long can shift beyond 31 bits if a long is indeed that big. However your code shifts 1, which is an int.
In C++, the type of an expression is determined by the expression itself. An expression XXXXX has the same type regardless; if you later go double foo = XXXXX; it doesn't mean XXXXX is a double - it means a conversion happens from whatever XXXXX was, to double.
If you want to left-shift a long, then do that explicitly, e.g. 1L << 32, or ((long)1) << 32. Note that the size of long varies between platforms, so if you don't want your code to break when run on a different system then you'll have to take further measures, such as using fixed-width types, or shifting by CHAR_BIT * sizeof(long) - 1.
There is another issue with your intended code: 1L << 63 causes undefined behaviour if long is 64-bit or less. This is because of signed integer overflow; left-shift is defined the same as repeated multiplication of two, so attempting to "shift into the sign bit" causes an overflow.
To fix this, use unsigned types where it is fine to shift into the MSB, e.g. 1ul << 63.
Technically there is another issue in that ~0 doesn't do what you want if you are not on a 2's complement system, but these days it's pretty safe to ignore that case.
Looking at your overall intention with long x = ~0 & ~(1 << 63). A shorter way to write this is:
long x = LONG_MAX;
which is defined by <climits>. If you wanted 64-bit on all platforms then
int64_t x = INT64_MAX;
NB. If you do not intend to work with negative values then use unsigned long x and uint64_t respectively.
First let me state a few things about the shift, which is the source of your problem:
There is no guarantee that long int is actually 64 bit wide.
The most generic way I can think of is using std::numeric_limits:
static_cast<long int>(1) << (std::numeric_limits<long int>::digits - 1);
Now you can even make that a constexpr templated function:
template <typename Integer>
constexpr Integer foo()
{
return static_cast<Integer>(1) << (std::numeric_limits<Integer>::digits - 1);
}
So replacing the shift with static_cast<long int>(1) << (std::numeric_limits<long int>::digits - 1) will fix your issue, however there is a far better way:
std::numeric_limits includes a bunch of useful stuff, including:
std::numeric_limits<T>::max(); // the maximum value T can hold
std::numeric_limits<T>::min(); // the minimum value T can hold
std::numeric_limits<T>::digits; // the number of binary digits
std::numeric_limits<T>::is_signed(); // well, do I have to explain? ;-)
See cppreference.com for a complete list. You should prefer the facilities provided by the standard library, because it will most likely have fewer mistakes and other developers immediately know it.
The default datatype for a numeric value in C is integer unless explicitly mentioned.
Here you have to type cast the 1 as long int which would otherwise be an int.
I found a rather strange but working square root approximation for floats; I really don't get it. Can someone explain me why this code works?
float sqrt(float f)
{
const int result = 0x1fbb4000 + (*(int*)&f >> 1);
return *(float*)&result;
}
I've test it a bit and it outputs values off of std::sqrt() by about 1 to 3%. I know of the Quake III's fast inverse square root and I guess it's something similar here (without the newton iteration) but I'd really appreciate an explanation of how it works.
(nota: I've tagged it both c and c++ since it's both valid-ish (see comments) C and C++ code)
(*(int*)&f >> 1) right-shifts the bitwise representation of f. This almost divides the exponent by two, which is approximately equivalent to taking the square root.1
Why almost? In IEEE-754, the actual exponent is e - 127.2 To divide this by two, we'd need e/2 - 64, but the above approximation only gives us e/2 - 127. So we need to add on 63 to the resulting exponent. This is contributed by bits 30-23 of that magic constant (0x1fbb4000).
I'd imagine the remaining bits of the magic constant have been chosen to minimise the maximum error across the mantissa range, or something like that. However, it's unclear whether it was determined analytically, iteratively, or heuristically.
It's worth pointing out that this approach is somewhat non-portable. It makes (at least) the following assumptions:
The platform uses single-precision IEEE-754 for float.
The endianness of float representation.
That you will be unaffected by undefined behaviour due to the fact this approach violates C/C++'s strict-aliasing rules.
Thus it should be avoided unless you're certain that it gives predictable behaviour on your platform (and indeed, that it provides a useful speedup vs. sqrtf!).
1. sqrt(a^b) = (a^b)^0.5 = a^(b/2)
2. See e.g. https://en.wikipedia.org/wiki/Single-precision_floating-point_format#Exponent_encoding
See Oliver Charlesworth’s explanation of why this almost works. I’m addressing an issue raised in the comments.
Since several people have pointed out the non-portability of this, here are some ways you can make it more portable, or at least make the compiler tell you if it won’t work.
First, C++ allows you to check std::numeric_limits<float>::is_iec559 at compile time, such as in a static_assert. You can also check that sizeof(int) == sizeof(float), which will not be true if int is 64-bits, but what you really want to do is use uint32_t, which if it exists will always be exactly 32 bits wide, will have well-defined behavior with shifts and overflow, and will cause a compilation error if your weird architecture has no such integral type. Either way, you should also static_assert() that the types have the same size. Static assertions have no run-time cost and you should always check your preconditions this way if possible.
Unfortunately, the test of whether converting the bits in a float to a uint32_t and shifting is big-endian, little-endian or neither cannot be computed as a compile-time constant expression. Here, I put the run-time check in the part of the code that depends on it, but you might want to put it in the initialization and do it once. In practice, both gcc and clang can optimize this test away at compile time.
You do not want to use the unsafe pointer cast, and there are some systems I’ve worked on in the real world where that could crash the program with a bus error. The maximally-portable way to convert object representations is with memcpy(). In my example below, I type-pun with a union, which works on any actually-existing implementation. (Language lawyers object to it, but no successful compiler will ever break that much legacy code silently.) If you must do a pointer conversion (see below) there is alignas(). But however you do it, the result will be implementation-defined, which is why we check the result of converting and shifting a test value.
Anyway, not that you’re likely to use it on a modern CPU, here’s a gussied-up C++14 version that checks those non-portable assumptions:
#include <cassert>
#include <cmath>
#include <cstdint>
#include <cstdlib>
#include <iomanip>
#include <iostream>
#include <limits>
#include <vector>
using std::cout;
using std::endl;
using std::size_t;
using std::sqrt;
using std::uint32_t;
template <typename T, typename U>
inline T reinterpret(const U x)
/* Reinterprets the bits of x as a T. Cannot be constexpr
* in C++14 because it reads an inactive union member.
*/
{
static_assert( sizeof(T)==sizeof(U), "" );
union tu_pun {
U u = U();
T t;
};
const tu_pun pun{x};
return pun.t;
}
constexpr float source = -0.1F;
constexpr uint32_t target = 0x5ee66666UL;
const uint32_t after_rshift = reinterpret<uint32_t,float>(source) >> 1U;
const bool is_little_endian = after_rshift == target;
float est_sqrt(const float x)
/* A fast approximation of sqrt(x) that works less well for subnormal numbers.
*/
{
static_assert( std::numeric_limits<float>::is_iec559, "" );
assert(is_little_endian); // Could provide alternative big-endian code.
/* The algorithm relies on the bit representation of normal IEEE floats, so
* a subnormal number as input might be considered a domain error as well?
*/
if ( std::isless(x, 0.0F) || !std::isfinite(x) )
return std::numeric_limits<float>::signaling_NaN();
constexpr uint32_t magic_number = 0x1fbb4000UL;
const uint32_t raw_bits = reinterpret<uint32_t,float>(x);
const uint32_t rejiggered_bits = (raw_bits >> 1U) + magic_number;
return reinterpret<float,uint32_t>(rejiggered_bits);
}
int main(void)
{
static const std::vector<float> test_values{
4.0F, 0.01F, 0.0F, 5e20F, 5e-20F, 1.262738e-38F };
for ( const float& x : test_values ) {
const double gold_standard = sqrt((double)x);
const double estimate = est_sqrt(x);
const double error = estimate - gold_standard;
cout << "The error for (" << estimate << " - " << gold_standard << ") is "
<< error;
if ( gold_standard != 0.0 && std::isfinite(gold_standard) ) {
const double error_pct = error/gold_standard * 100.0;
cout << " (" << error_pct << "%).";
} else
cout << '.';
cout << endl;
}
return EXIT_SUCCESS;
}
Update
Here is an alternative definition of reinterpret<T,U>() that avoids type-punning. You could also implement the type-pun in modern C, where it’s allowed by standard, and call the function as extern "C". I think type-punning is more elegant, type-safe and consistent with the quasi-functional style of this program than memcpy(). I also don’t think you gain much, because you still could have undefined behavior from a hypothetical trap representation. Also, clang++ 3.9.1 -O -S is able to statically analyze the type-punning version, optimize the variable is_little_endian to the constant 0x1, and eliminate the run-time test, but it can only optimize this version down to a single-instruction stub.
But more importantly, this code isn’t guaranteed to work portably on every compiler. For example, some old computers can’t even address exactly 32 bits of memory. But in those cases, it should fail to compile and tell you why. No compiler is just suddenly going to break a huge amount of legacy code for no reason. Although the standard technically gives permission to do that and still say it conforms to C++14, it will only happen on an architecture very different from we expect. And if our assumptions are so invalid that some compiler is going to turn a type-pun between a float and a 32-bit unsigned integer into a dangerous bug, I really doubt the logic behind this code will hold up if we just use memcpy() instead. We want that code to fail at compile time, and to tell us why.
#include <cassert>
#include <cstdint>
#include <cstring>
using std::memcpy;
using std::uint32_t;
template <typename T, typename U> inline T reinterpret(const U &x)
/* Reinterprets the bits of x as a T. Cannot be constexpr
* in C++14 because it modifies a variable.
*/
{
static_assert( sizeof(T)==sizeof(U), "" );
T temp;
memcpy( &temp, &x, sizeof(T) );
return temp;
}
constexpr float source = -0.1F;
constexpr uint32_t target = 0x5ee66666UL;
const uint32_t after_rshift = reinterpret<uint32_t,float>(source) >> 1U;
extern const bool is_little_endian = after_rshift == target;
However, Stroustrup et al., in the C++ Core Guidelines, recommend a reinterpret_cast instead:
#include <cassert>
template <typename T, typename U> inline T reinterpret(const U x)
/* Reinterprets the bits of x as a T. Cannot be constexpr
* in C++14 because it uses reinterpret_cast.
*/
{
static_assert( sizeof(T)==sizeof(U), "" );
const U temp alignas(T) alignas(U) = x;
return *reinterpret_cast<const T*>(&temp);
}
The compilers I tested can also optimize this away to a folded constant. Stroustrup’s reasoning is [sic]:
Accessing the result of an reinterpret_cast to a different type from the objects declared type is still undefined behavior, but at least we can see that something tricky is going on.
Update
From the comments: C++20 introduces std::bit_cast, which converts an object representation to a different type with unspecified, not undefined, behavior. This doesn’t guarantee that your implementation will use the same format of float and int that this code expects, but it doesn’t give the compiler carte blanche to break your program arbitrarily because there’s technically undefined behavior in one line of it. It can also give you a constexpr conversion.
Let y = sqrt(x),
it follows from the properties of logarithms that log(y) = 0.5 * log(x) (1)
Interpreting a normal float as an integer gives INT(x) = Ix = L * (log(x) + B - σ) (2)
where L = 2^N, N the number of bits of the significand, B is the exponent bias, and σ is a free factor to tune the approximation.
Combining (1) and (2) gives: Iy = 0.5 * (Ix + (L * (B - σ)))
Which is written in the code as (*(int*)&x >> 1) + 0x1fbb4000;
Find the σ so that the constant equals 0x1fbb4000 and determine whether it's optimal.
Adding a wiki test harness to test all float.
The approximation is within 4% for many float, but very poor for sub-normal numbers. YMMV
Worst:1.401298e-45 211749.20%
Average:0.63%
Worst:1.262738e-38 3.52%
Average:0.02%
Note that with argument of +/-0.0, the result is not zero.
printf("% e % e\n", sqrtf(+0.0), sqrt_apx(0.0)); // 0.000000e+00 7.930346e-20
printf("% e % e\n", sqrtf(-0.0), sqrt_apx(-0.0)); // -0.000000e+00 -2.698557e+19
Test code
#include <float.h>
#include <limits.h>
#include <math.h>
#include <stddef.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
float sqrt_apx(float f) {
const int result = 0x1fbb4000 + (*(int*) &f >> 1);
return *(float*) &result;
}
double error_value = 0.0;
double error_worst = 0.0;
double error_sum = 0.0;
unsigned long error_count = 0;
void sqrt_test(float f) {
if (f == 0) return;
volatile float y0 = sqrtf(f);
volatile float y1 = sqrt_apx(f);
double error = (1.0 * y1 - y0) / y0;
error = fabs(error);
if (error > error_worst) {
error_worst = error;
error_value = f;
}
error_sum += error;
error_count++;
}
void sqrt_tests(float f0, float f1) {
error_value = error_worst = error_sum = 0.0;
error_count = 0;
for (;;) {
sqrt_test(f0);
if (f0 == f1) break;
f0 = nextafterf(f0, f1);
}
printf("Worst:%e %.2f%%\n", error_value, error_worst*100.0);
printf("Average:%.2f%%\n", error_sum / error_count);
fflush(stdout);
}
int main() {
sqrt_tests(FLT_TRUE_MIN, FLT_MIN);
sqrt_tests(FLT_MIN, FLT_MAX);
return 0;
}
I faced an interesting scenario in which I got different results depending on the right operand type, and I can't really understand the reason for it.
Here is the minimal code:
#include <iostream>
#include <cstdint>
int main()
{
uint16_t check = 0x8123U;
uint64_t new_check = (check & 0xFFFF) << 16;
std::cout << std::hex << new_check << std::endl;
new_check = (check & 0xFFFFU) << 16;
std::cout << std::hex << new_check << std::endl;
return 0;
}
I compiled this code with g++ (gcc version 4.5.2) on Linux 64bit: g++ -std=c++0x -Wall example.cpp -o example
The output was:
ffffffff81230000
81230000
I can't really understand the reason for the output in the first case.
Why at some point would any of the temporal calculation results be promoted to a signed 64bit value (int64_t) resulting in the sign extension?
I would accept a result of '0' in both cases if a 16bit value is shifted 16 bits left in the first place and then promoted to a 64bit value. I also do accept the second output if the compiler first promotes the check to uint64_t and then performs the other operations.
But how come & with 0xFFFF (int32_t) vs. 0xFFFFU (uint32_t) would result in those two different outputs?
That's indeed an interesting corner case. It only occurs here because you use uint16_t for the unsigned type when you architecture use 32 bits for ìnt
Here is a extract from Clause 5 Expressions from draft n4296 for C++14 (emphasize mine):
10 Many binary operators that expect operands of arithmetic or enumeration type cause conversions ...
This pattern is called the usual arithmetic conversions, which are defined as follows:
...(10.5.3) — Otherwise, if the operand that has unsigned integer type has rank greater than or equal to the
rank of the type of the other operand, the operand with signed integer type shall be converted to
the type of the operand with unsigned integer type.
(10.5.4) — Otherwise, if the type of the operand with signed integer type can represent all of the values of
the type of the operand with unsigned integer type, the operand with unsigned integer type shall
be converted to the type of the operand with signed integer type.
You are in the 10.5.4 case:
uint16_t is only 16 bits while int is 32
int can represent all the values of uint16_t
So the uint16_t check = 0x8123U operand is converted to the signed 0x8123 and result of the bitwise & is still 0x8123.
But the shift (bitwise so it happens at the representation level) causes the result to be the intermediate unsigned 0x81230000 which converted to an int gives a negative value (technically it is implementation defined, but this conversion is a common usage)
5.8 Shift operators [expr.shift]...Otherwise, if E1 has a signed type and non-negative value, and E1×2E2 is representable
in the corresponding unsigned type of the result type, then that value, converted to the result type, is the
resulting value;...
and
4.7 Integral conversions [conv.integral]...
3 If the destination type is signed, the value is unchanged if it can be represented in the destination type;
otherwise, the value is implementation-defined.
(beware this was true undefined behaviour in C++11...)
So you end with a conversion of the signed int 0x81230000 to an uint64_t which as expected gives 0xFFFFFFFF81230000, because
4.7 Integral conversions [conv.integral]...
2 If the destination type is unsigned, the resulting value is the least unsigned integer congruent to the source
integer (modulo 2n where n is the number of bits used to represent the unsigned type).
TL/DR: There is no undefined behaviour here, what causes the result is the conversion of signed 32 bits int to unsigned 64 bits int. The only part part that is undefined behaviour is a shift that would cause a sign overflow but all common implementations share this one and it is implementation defined in C++14 standard.
Of course, if you force the second operand to be unsigned everything is unsigned and you get evidently the correct 0x81230000 result.
[EDIT] As explained by MSalters, the result of the shift is only implementation defined since C++14, but was indeed undefined behaviour in C++11. The shift operator paragraph said:
...Otherwise, if E1 has a signed type and non-negative value, and E1×2E2 is representable
in the result type, then that is the resulting value; otherwise, the behavior is undefined.
Let's take a look at
uint64_t new_check = (check & 0xFFFF) << 16;
Here, 0xFFFF is a signed constant, so (check & 0xFFFF) gives us a signed integer by the rules of integer promotion.
In your case, with 32-bit int type, the MSbit for this integer after the left shift is 1, and so the extension to 64-bit unsigned will do a sign extension, filling the bits to the left with 1's. Interpreted as a two's complement representation that gives the same negative value.
In the second case, 0xFFFFU is unsigned, so we get unsigned integers and the left shift operator works as expected.
If your toolchain supports __PRETTY_FUNCTION__, a most-handy feature, you can quickly determine how the compiler perceives expression types:
#include <iostream>
#include <cstdint>
template<typename T>
void typecheck(T const& t)
{
std::cout << __PRETTY_FUNCTION__ << '\n';
std::cout << t << '\n';
}
int main()
{
uint16_t check = 0x8123U;
typecheck(0xFFFF);
typecheck(check & 0xFFFF);
typecheck((check & 0xFFFF) << 16);
typecheck(0xFFFFU);
typecheck(check & 0xFFFFU);
typecheck((check & 0xFFFFU) << 16);
return 0;
}
Output
void typecheck(const T &) [T = int]
65535
void typecheck(const T &) [T = int]
33059
void typecheck(const T &) [T = int]
-2128412672
void typecheck(const T &) [T = unsigned int]
65535
void typecheck(const T &) [T = unsigned int]
33059
void typecheck(const T &) [T = unsigned int]
2166554624
The first thing to realize is that binary operators like a&b for built-in types only work if both sides have the same type. (With user-defined types and overloads, anything goes). This might be realized via implicit conversions.
Now, in your case, there definitely is such a conversion, because there simply isn't a binary operator & that takes a type smaller than int. Both sides are converted to at least int size, but what exact types?
As it happens, on your GCC int is indeed 32 bits. This is important, because it means that all values of uint16_t can be represented as an int. There is no overflow.
Hence, check & 0xFFFF is a simple case. The right side is already an int, the left side promotes to int, so the result is int(0x8123). This is perfectly fine.
Now, the next operation is 0x8123 << 16. Remember, on your system int is 32 bits, and INT_MAX is 0x7FFF'FFFF. In the absence of overflow, 0x8123 << 16 would be 0x81230000, but that clearly is bigger than INT_MAX so there is in fact overflow.
Signed integer overflow in C++11 is Undefined Behavior. Literally any outcome is correct, including purple or no output at all. At least you got a numerical value, but GCC is known to outright eliminate code paths which unavoidably cause overflow.
[edit]
Newer GCC versions support C++14, where this particular form of overflow has become implementation-defined - see Serge's answer.
0xFFFF is a signed int. So after the & operation, we have a 32-bit signed value:
#include <stdint.h>
#include <type_traits>
uint64_t foo(uint16_t a) {
auto x = (a & 0xFFFF);
static_assert(std::is_same<int32_t, decltype(x)>::value, "not an int32_t")
static_assert(std::is_same<uint16_t, decltype(x)>::value, "not a uint16_t");
return x;
}
http://ideone.com/tEQmbP
Your original 16 bits are then left-shifted which results in 32-bit value with the high-bit set (0x80000000U) so it has a negative value. During the 64-bit conversion sign-extension occurs, populating the upper words with 1s.
This is the result of integer promotion. Before the & operation happens, if the operands are "smaller" than an int (for that architecture), compiler will promote both operands to int, because they both fit into a signed int:
This means that the first expression will be equivalent to (on a 32-bit architecture):
// check is uint16_t, but it fits into int32_t.
// the constant is signed, so it's sign-extended into an int
((int32_t)check & (int32_t)0xFFFFFFFF)
while the other one will have the second operand promoted to:
// check is uint16_t, but it fits into int32_t.
// the constant is unsigned, so the upper 16 bits are zero
((int32_t)check & (int32_t)0x0000FFFFU)
If you explicitly cast check to an unsigned int, then the result will be the same in both cases (unsigned * signed will result in unsigned):
((uint32_t)check & 0xFFFF) << 16
will be equal to:
((uint32_t)check & 0xFFFFU) << 16
Your platform has 32-bit int.
Your code is exactly equivalent to
#include <iostream>
#include <cstdint>
int main()
{
uint16_t check = 0x8123U;
auto a1 = (check & 0xFFFF) << 16
uint64_t new_check = a1;
std::cout << std::hex << new_check << std::endl;
auto a2 = (check & 0xFFFFU) << 16;
new_check = a2;
std::cout << std::hex << new_check << std::endl;
return 0;
}
What's the type of a1 and a2?
For a2, the result is promoted to unsigned int.
More interestingly, for a1 the result is promoted to int, and then it gets sign-extended as it's widened to uint64_t.
Here's a shorter demonstration, in decimal so that the difference between signed and unsigned types is apparent:
#include <iostream>
#include <cstdint>
int main()
{
uint16_t check = 0;
std::cout << check
<< " " << (int)(check + 0x80000000)
<< " " << (uint64_t)(int)(check + 0x80000000) << std::endl;
return 0;
}
On my system (also 32-bit int), I get
0 -2147483648 18446744071562067968
showing where the promotion and sign-extension happens.
The & operation has two operands. The first is an unsigned short, which will undergo the usual promotions to become an int. The second is a constant, in one case of type int, in the other case of type unsigned int. The result of the & is therefore int in one case, unsigned int in the other case. That value is shifted to the left, resulting either in an int with the sign bit set, or an unsigned int. Casting a negative int to uint64_t will give a large negative integer.
Of course you should always follow the rule: If you do something, and you don't understand the result, then don't do that!
REALLY BRIEF
How do you create an unsigned constant that has all bits set?
...that you can use to initialize a field with { }s,
...that does not get a -Wnarrowing warning from GCC 4.7.2.
The following are not satisfactory:
struct U { unsigned ufield; };
struct Uc { unsigned char ufield; };
struct ULL { unsigned long long ufield; };
struct U32 { unsigned ufield; };
struct U64 { uint64_t ufield; }
typedef
//any of the above U Uc ULL U32 U64, we will arbitrarily choose:
U Ueg;
// somewhere far away
Ueg u = {-1}; // well defined by C standard, GCC 4.7.2 -Wnarrowing warning
Ueg u = {~0U}; // finit width constant, warnings in some places, silent non-all-1s-mask others
Ueg u = {~0ULL}; // ditto
Ueg u = {-1ULL}; // ditto
Basically, the user, the guy writing the {} initialization,
does not know the type of the ufield.
He only knows that it is an unsigned type, but not how wide.
Not exactly which unsigned type it is.
* Another Reason Why I want as simple and elegant a syntax as possible *
I might as well mention something else: the "user" here is not actually writing a C or C++ program.
He is editing a config file.
A program, a simple Perl or Python script, processes the config file, and generates C code.
This program is not very sophisticated, and at the moment passes through chunks of text that look like
Foo: {-1,2,3};
to generate
typedef
struct Some_Struct { unsigned a; unsigned b, unsigned c; }
Some_Struct = {-1,2,3}; // ditto
Basically, I want to be able to a nice user friendly syntax for a literal that says
"All of the bits in this unsigned value are set".
Without having to know how big of an unsigned.
And without the program that handles the config file getting too complicated.
Lest the potential answer-provider complain that this is a new constraint, not realistic, etc:
I have had exactly the same problem with templates.
I.e. with template types, where I want to write a literal that is "unsigned of any width, all 1s".
In a template I might be more willing to do some of the ugly, Ugly, UGLY syntax
that is obviously able to do this:
but I really wish there was a simple, elegant, syntax.
* The Real Question *
Q: is there any way to create a constant that is "all 1s set"
without triggering the GCC 4.7.2 warning?
BRIEF
I ran across a program that was using the literal constant -1 to initialize a field of a struct, e.g.
> cat ./-1u.cpp
#include <stdio.h>
struct U { unsigned ufield; } ustruct = { -1 };
int main(int argc, char** argv)
{
printf("ustruct.ufield = %08x\n",ustruct.ufield);
}
Although earlier versions of GCC accepted this without warning,
the fairly recent version GCC 4.7.2 provides a warning:
> /SOME-PATH/import/gcc/gcc-4.7.2.pkg/installdir/bin/g++ -Wall ./-1u.cpp
./-1u.cpp:3:46: warning: narrowing conversion of '-1' from 'int' to 'unsigned int' inside { } is ill-formed in C++11 [-Wnarrowing]
Note: this is only a warning. The result of converting -1 to unsigned is well defined in the C/C++ standards:
> ./a.out
ustruct.ufield = ffffffff
I dislike warnings, so I would like to silence this annoying warning. I prefer not to use #pragmas that apply to the entire file, since that may disable a warning for real bugs.
(By the way, you get this warnng only when initializing a field. Not when initializing a non-field
unsigned u = -1; // no cmpiler warning.
Doing
struct U { unsigned ufield; } ustruct = { ~0U };
silences the bug.
But it was pointed out that if the type of the field is not unsigned, but, instead, uint64_t,
then ~0U provides a different result than -1: 0x00000000FFFFFFFF rather than 0xFFFFFFFFFFFFFFFF.
(I.e. 32 bits of 1s, rather than 64 bits of 1s.)
The struct U and the initializaton code may live in completely different places,
and we would want to be able to increase the size of the field, a bitmask, without informing users.
And the intent is to get a "mask of all 1s" of whatever unsigned type is being used.
Similarly
struct U { unsigned ufield; } ustruct = { -1u };
silences the bug. (To my surprise - I did not know that -1 could be considered an unisgned.)
But is also a finite-width constant.
DETAIL
Here's a test program.
(By the way, all I am asking about is the use of the signed literal constant -1
to initialize an unsigned member. The other warnings are just tests.
You don;t need to explain to me that a 64 bit number doesn't fit in 32 bits.)
sh-3.2$ cat ./-1u.cpp
#include <stdio.h>
unsigned um1 = -1;
unsigned un0u = ~0u;
unsigned un0ull = ~0ull;
struct Foo {
unsigned um1;
unsigned un0u;
unsigned un0ull;
};
Foo foo = { -1, ~0u, ~0ull };
int main(int argc, char** argv)
{
printf("um1 = %08x\n",um1);
printf("un0u = %08x\n",un0u);
printf("un0ull = %08x\n",un0ull);
printf("foo.um1 = %08x\n",foo.um1);
printf("foo.un0u = %08x\n",foo.un0u);
printf("foo.un0ull = %08x\n",foo.un0ull);
}
sh-3.2$ /mips/proj/performance/import/gcc/gcc-4.7.2.pkg/installdir/bin/gcc -Wall ./-1u.cpp
./-1u.cpp:7:20: warning: large integer implicitly truncated to unsigned type [-Woverflow]
./-1u.cpp:15:28: warning: narrowing conversion of '-1' from 'int' to 'unsigned int' inside { } is ill-formed in C++11 [-Wnarrowing]
./-1u.cpp:15:28: warning: narrowing conversion of '18446744073709551615ull' from 'long long unsigned int' to 'unsigned int' inside { } is ill-formed in C++11 [-Wnarrowing]
./-1u.cpp:15:28: warning: large integer implicitly truncated to unsigned type [-Woverflow]
sh-3.2$ /mips/proj/performance/import/gcc/gcc-4.7.2.pkg/installdir/bin/g++ -Wall ./-1u.cpp
./-1u.cpp:7:20: warning: large integer implicitly truncated to unsigned type [-Woverflow]
./-1u.cpp:15:35: warning: narrowing conversion of '-1' from 'int' to 'unsigned int' inside { } is ill-formed in C++11 [-Wnarrowing]
./-1u.cpp:15:35: warning: narrowing conversion of '18446744073709551615ull' from 'long long unsigned int' to 'unsigned int' inside { } is ill-formed in C++11 [-Wnarrowing]
./-1u.cpp:15:35: warning: large integer implicitly truncated to unsigned type [-Woverflow]
Doesn't occur in an earlier compiler:
sh-3.2$ /usr/bin/g++ -Wall ./-1u.cpp
./-1u.cpp:7: warning: large integer implicitly truncated to unsigned type
./-1u.cpp:15: warning: large integer implicitly truncated to unsigned type
/usr/bin/g++ --version
g++ (GCC) 4.1.2 20080704 (Red Hat 4.1.2-51)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
A slightly more user-friendly version of #Ali's answer:
#include <type_traits>
struct all_ones_type {
template <typename T,
typename = typename std::enable_if<std::is_unsigned<T>::value>::type>
constexpr operator T () const
{ return static_cast<T>(-1); }
} const all_ones;
#include <iostream>
struct X {
unsigned short a;
unsigned long b;
unsigned long long c;
};
int main() {
X x = { all_ones, all_ones, all_ones };
std::cout << x.a << "\n"
<< x.b << "\n"
<< x.c << std::endl;
}
Depending on what you want to happen on an attempted conversion to signed type, you could change the enable_if to allow all integral types, or add another overload with a nice static_assert.
How about this? It only works for unsigned types but the question specifically says unsigned. (See rubenvb's comments below.)
#include <cinttypes>
#include <iomanip>
#include <iostream>
#include <limits>
#include <type_traits>
template <typename T>
T all_bits_one() {
static_assert(std::is_unsigned<T>::value, "the type must be unsigned");
return std::numeric_limits<T>::max();
}
struct Ui {
typedef unsigned int the_type;
the_type ufield;
};
struct ULL {
typedef unsigned long long the_type;
the_type ufield;
};
struct U64 {
typedef uint64_t the_type;
the_type ufield;
};
int main() {
using namespace std;
Ui ui = { all_bits_one< Ui::the_type>() };
ULL ull = { all_bits_one<ULL::the_type>() };
U64 u64 = { all_bits_one<U64::the_type>() };
cout << hex;
cout << "unsigned int: " << ui.ufield << endl;
cout << "unsigned long long: " << ull.ufield << endl;
cout << "unsigned int 64: " << u64.ufield << endl;
//all_bits_one<int>(); // causes compile-time error if uncommented
return 0;
}
The user doesn't have to know the exact type of the_type or the number of bits it is represented on.
There is some code duplication which could be removed but that would require a better understanding of your code and the problem you are dealing with.
I guess you simplified your code before posting. As it stands, your structs make no sense to me, a simple typedef would suffice.
Why not provide the mask along with the type?
C:
struct U { unsigned ufield; };
#define U_MASK (-1U)
// somewhere far away
U u = {U_MASK};
C++:
struct U { unsigned ufield; static constexpr unsigned MASK = -1; };
// somewhere far away
U u = {U::MASK};
All fancy template code aside, here's some fancy C++11 code:
struct Foo
{
unsigned a;
unsigned long b;
unsigned long long c;
};
Foo foo = { decltype(Foo::a)(-1), decltype(Foo::b)(-1), decltype(Foo::c)(-1) };
which is error-prone, but functional.
The best solution is still to use a (typed) enum (class) for this.
Another way
// C++03 and C++11
Ueg u = { (Ueg().ufield - 1) };
// C99 and C11 (works only inside of functions)
Ueg u = { (Ueg){0}.ufield - 1 };
Inspired by Ali, but using Template Argument Deduction.
T all_bits_one(T& dummy) { return ~(T()); }
unsigned long u = all_bits_one(u);
Maybe what you want by initializing an uint to -1 is that all bits are set to '1'?
In that case:
typedef uint MyUIntType;
MyUIntType mID = (MyUIntType)~0;
~ applies a "1's complement" to 0 which effectively flips all its bits.
The result will be the biggest value that your uint type can hold, which is useful in cases where 0 is a meaningful value and variables need to be initialized to "something else".