My question is, within C++, is the following code defined? Some of it? And if it is, what's it supposed to do in these four scenarios?
word << 100;
word >> 100;
word << -100;
word >> -100;
word is a uint32_t
(This is for a bottleneck in a 3d lighting renderer. One of the more minor improvements in the inner most loop I wanna make is eliminating needless conditional forks. One of those forks is checking to see if a left shift should be done on several 32 bit words as part of a hamming weight count. If the left shift accepts absurd values, the checks don't need done at all)
In the C++0X draft N3290, §5.8:
The behavior is undefined if the right operand is negative,
or greater than or equal to the length in bits of the promoted left
operand.
Note: the above paragraph is identical in the C++03 standard.
So the last two are undefined. The others, I believe depend on whether word is signed or not, if word is at least 101bits long. If word is "smaller" than 101bits, the above applies and the behavior is undefined.
Here are the next two sections of that paragraph in C++0X (these do differ in C++03):
The value of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are zero-filled. If E1 has an unsigned
type, the value of the result is E1 × 2E2 , reduced modulo one more than the maximum value representable
in the result type. Otherwise, if E1 has a signed type and non-negative value, and E1 × 2E2 is representable
in the result type, then that is the resulting value; otherwise, the behavior is undefined.
The value of E1 >> E2 is E1 right-shifted E2 bit positions. If E1 has an unsigned type or if E1 has a signed
type and a non-negative value, the value of the result is the integral part of the quotient of E1/2E2 . If E1
has a signed type and a negative value, the resulting value is implementation-defined.
The C standard doesn't say what should happen when shift count is negative or greater than (or even equal) to the precision of the variable.
The reason is that the C standard didn't want to impose a behavior that would require extra code to be emitted in case of parametric shift. Since different CPUs do different things the standard says that anything can happen.
With x86 hardware the shift operator only uses last 5 bits of the shift counter to decide the shift amount (this can be seen by reading the CPU reference manual) so this is what most probably will happen with any C or C++ compiler on that platform.
See also this answer for a similar question.
Related
In c++20, signed integers are now defined to use two's complement,
see http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r3.html
This is a welcome change, however one of the bullet-points caught my eye:
Change Left-shift on signed integer types produces the same
results as left-shift on the corresponding unsigned integer type.
This seem like a strange change. Will this not shift away the sign bit?
The C++17 wording for signed left shifts (E1 << E2) was:
Otherwise, if E1 has a signed type and non-negative value, and E1×2E2 is representable in the corresponding unsigned type of the result type, then that value, converted to the result type, is the resulting value; otherwise, the behavior is undefined.
Note that it speaks of being representable in "the corresponding unsigned type". So if you have a 32-bit signed integer whose value is the 0x7FFFFFFF, and you left-shift it by 1, the resulting shift is representable in a 32-bit unsigned integer (0xFFFFFFFE). But then this unsigned value gets converted into the result type. And converting an unsigned integer whose value is too big for the corresponding signed type is implementation-defined.
Overall, in C++17, left-shifting into the sign bit could happen through implementation-defined behavior, and even then only if you don't shift beyond the unsigned result type's size. Going past that is explicitly UB.
The C++20 wording, for both signed and unsigned integers, is:
The value of E1 << E2 is the unique value congruent to E1×2E2 modulo 2N, where N is the width of the type of the result.
Integer congruence modulo a number basically means cutting off the bits beyond the modulo number. The "width" of an integer is explicitly defined as:
The range of representable values for a signed integer type is −2N−1 to 2N−1−1 (inclusive), where N is called the width of the type.
This means that for a 32-bit signed integer, the width is 31. So the modulous of the result of a shift is 31 bits, which cuts off the sign bit, explicitly preventing shifting into it.
So in C++20, we have a harder guarantee; implementations can never do a signed left-shift into the sign bit. This is different from C++17 only in the sense that implementation variance/UB has been explicitly defined to not happen.
So left shift wasn't defined to shift into the sign bit in C++17, and is defined not to do so in C++20.
What exactly that quote means probably refers to the fact that left shift on a negative number is now valid, shifting is always well-defined no matter how much shifting you do, and the wording for the signed/unsigned shifting is overall the same.
Yes, the left shifting signed integer behavior changed with C++20.
With C++17, left-shifting a positive signed integer into the sign bit invokes implementation defined behavior.1 Example:
int i = INT_MAX;
int j = i << 1; // implementation defined behavior with std < C++20
C++20 changed this to defined behavior because it mandates two's complement representation for signed integers.2,3
With C++17, shifting a negative signed integer invokes undefined behavior.1 Example:
int i = -1;
int j = i << 1; // undefined behavior with std < C++20
In C++20, this changed as well and this operation now also invokes defined behavior.3
This seem like a strange change. Will this not shift away the sign bit?
Yes, a signed left shift shifts away the sign bit. Example:
int i = 1 << (sizeof(int)*8-1); // C++20: defined behavior, set most significant bit
int j = i << 1; // C++20: defined behavior, set to 0
The main reason for specifying something as undefined or implementation defined behavior is to allow for efficient implementations on different hardware.
Nowadays, since all CPUs implement two's complement it's natural that the C++ standard mandates it. And if you mandate two's complement it's only consequential that you make the above operations defined behavior because this is also how left shift behaves in all two's complement instruction set architectures (ISAs).
IOW, leaving it implementation defined and undefined wouldn't buy you anything.
Or, if you liked the previous undefined behavior why would you care if it gets changed to defined behavior? You can still avoid this operation as before. You wouldn't have to change your code.
1
The value of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are zero-filled. If E1 has an unsigned
type, the value of the result is E1 × 2**E2, reduced modulo one more than the maximum value representable in
the result type. Otherwise, if E1 has a signed type and non-negative value, and E1 × 2**E2 is representable
in the corresponding unsigned type of the result type, then that value, converted to the result type, is the
resulting value; otherwise, the behavior is undefined.
(C++17 final working draft, Section 8.8 Shift operators [expr.shift], Paragraph 2, page 132 - emphasis mine)
2
[..] For each value x of a signed integer type, the value of the
corresponding unsigned integer type congruent to x modulo 2 N has the same value of corresponding bits in
its value representation. 41) This is also known as two’s complement representation. [..]
(C++20 latest working draft, Section 6.8.1 Fundamental types [basic.fundamental], Paragraph 3, page 66)
3
The value of E1 << E2 is the unique value congruent to E1 × 2**E2 modulo 2**N, where N is the width of the
type of the result. [Note: E1 is left-shifted E2 bit positions; vacated bits are zero-filled. — end note]
(C++20 latest working draft, Section 7.6.7 Shift operators [expr.shift], Paragraph 2, page 129, link mine)
I'm reading C++ Primer and I'm slightly confused by a few comments which talk about how Bitwise operators deal with signed types. I'll quote:
Quote #1
(When talking about Bitwise operators) "If the operand is signed and
its value is negative, then the way that the “sign bit” is handled in
a number of the bitwise operations is machine dependent. Moreover,
doing a left shift that changes the value of the sign bit is
undefined"
Quote #2
(When talking about the rightshift operator) "If that operand is
unsigned, then the operator inserts 0-valued bits on the left; if it
is a signed type, the result is implementation defined—either copies
of the sign bit or 0-valued bits are inserted on the left."
The bitwise operators promote small integers (such as char) to signed ints. Isn't there an issue with this promotion to signed ints when bitwise operators often gives undefined or implementation-defined behaviour on signed operator types? Why wouldn't the standard promote char to unsigned int?
Edit: Here is the question I took out, but I've placed it back for context with some answers below.
An exercise later asks
"What is the value of ~'q' << 6 on a machine with 32-bit ints and 8 bit chars, that uses Latin-1 character set in which 'q' has the bit pattern 01110001?"
Well, 'q' is a character literal and would be promoted to int, giving
~'q' == ~0000000 00000000 00000000 01110001 == 11111111 11111111 11111111 10001110
The next step is to apply a left shift operator to the bits above, but as quote #1 mentions
"doing a left shift that changes the value of the sign bit is
undefined"
well I don't exactly know which bit is the sign bit, but surely the answer is undefined?
You're quite correct -- the expression ~'q' << 6 is undefined behavior according to the standard. Its even worse than you state, as the ~ operator is defined as computing "The one's complement" of the value, which is meaningless for a signed (2s-complement) integer -- the term "one's complement" only really means anything for an unsigned integer.
When doing bitwise operations, if you want strictly well-defined (according to the standard) results, you generally have to ensure that the values being operated on are unsigned. You can do that either with explicit casts, or by using explicitly unsigned constants (U-suffix) in binary operations. Doing a binary operation with a signed and unsigned int is done as unsigned (the signed value is converted to unsigned).
C and C++ are subtley different with the integer promotions, so you need to be careful here -- C++ will convert a smaller-than-int unsigned value to int (signed) before comparing with the other operand to see what should be done, while C will compare operands first.
It might be simplest to read the exact text of the Standard, instead of a summary like in Primer Plus. (The summary has to leave out detail by virtue of being a summary!)
The relevant portions are:
[expr.shift]
The shift operators << and >> group left-to-right.
The operands shall be of integral or unscoped enumeration type and integral promotions are performed. The type of the result is that of the promoted left operand. The behavior is undefined if the right operand is negative, or greater than or equal to the length in bits of the promoted left operand.
The value of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are zero-filled. If E1 has an unsigned type, the value of the result is E1 × 2E2 , reduced modulo one more than the maximum value representable in the result type. Otherwise, if E1 has a signed type and non-negative value, and E1 × 2E2 is representable in the corresponding unsigned type of the result type, then that value, converted to the result type, is the resulting value; otherwise, the behavior is undefined.
[expr.unary.op]/10
The operand of ˜ shall have integral or unscoped enumeration type; the result is the one’s complement of its operand. Integral promotions are performed. The type of the result is the type of the promoted operand.
Note that neither of these performs the usual arithmetic conversions (which is the conversion to a common type that is done by most of the binary operators).
The integral promotions:
[conv.prom]/1
A prvalue of an integer type other than bool, char16_t, char32_t, or wchar_t whose integer conversion rank is less than the rank of int can be converted to a prvalue of type int if int can represent all the values of the source type; otherwise, the source prvalue can be converted to a prvalue of type unsigned int.
(There are other entries for the types in the "other than" list, I have omitted them here but you can look it up in a Standard draft).
The thing to remmeber about the integer promotions is that they are value-preserving , if you have a char of value -30, then after promotion it will be an int of value -30. You don't need to think about things like "sign extension".
Your initial analysis of ~'q' is correct, and the result has type int (because int can represent all the values of char on normal systems).
It turns out that any int whose most significant bit is set represents a negative value (there are rules about this in another part of the standard that I haven't quoted here), so ~'q' is a negative int.
Looking at [expr.shift]/2 we see that this means left-shifting it causes undefined behaviour (it's not covered by any of the earlier cases in that paragraph).
Of course, by editing the question, my answer is now partly answering a different question than the one posed, so here goes an attempt to answer the "new" question:
The promotion rules (what gets converted to what) are well defined in the standard. The type char may be either signed or unsigned - in some compilers you can even give a flag to the compiler to say "I want unsigned char type" or "I want signed char type" - but most compilers just define char as either signed or unsigned.
A constant, such as 6 is signed by default. When an operation, such as 'q' << 6 is written in the code, the compiler will convert any smaller type to any larger type [or if you do any arithmetic in general, char is converted to int], so 'q' becomes the integer value of 'q'. If you want to avoid that, you should use 6u, or an explicit cast, such as static_cast<unsigned>('q') << 6 - that way, you are ensured that the operand is converted to unsigned, rather than signed.
The operations are undefined because different hardware behaves differently, and there are architectures with "strange" numbering systems, which means that the standards committee has to choose between "ruling out/making operations extremely inefficient" or "defining the standard in a way that isn't very clear". In a few architectures, overflowing integers may also be a trap, and if you shift such that you change the sign on the number, that typically counts as an overflow - and since trapping typically means "your code no longer runs", that would not be what your average programmer expects -> falls under the umbrella of "undefined behaviour". Most processors don't, and nothing really bad will happen if you do that.
Old answer:
So the solution to avoid this is to always cast your signed values (including char) to unsigned before shifting them (or accept that your code may not work on another compiler, the same compiler with different options, or the next release of the same compiler).
It is also worth noting that the resulting value is "nearly always what you expect" (in that the compiler/processor will just perform the left or right shift on the value, on right shifts using the sign bit to shift down), it's just undefined or implementation defined because SOME machine architectures may not have hardware to "do this right", and C compilers still need to work on those systems.
The sign bit is the highest bit in a twos-complement, and you are not changing that by shifting that number:
11111111 11111111 11111111 10001110 << 6 =
111111 11111111 11111111 11100011 10000000
^^^^^^--- goes away.
result=11111111 11111111 11100011 10000000
Or as a hex number: 0xffffe380.
Is it legal to do the following in C11, C++11 and C++14?
static_assert(((-4) >> 1) == -2, "my code assumes sign-extending right shift");
or the C equivalent:
_Static_assert(((-4) >> 1) == -2, "my code assumes sign-extending right shift");
I don't know the rules for constant-expressions regarding whether you can use implementation-defined operations like the above.
I'm aware that the opposite, signed shift left of negative numbers, is undefined regardless of machine type.
Yes. The C++11 standard says in [expr.shift]/3:
The value of E1 >> E2 is E1 right-shifted E2 bit positions. If
E1 has an unsigned type or if E1 has a signed type and a non-negative
value, the value of the result is the integral part of the quotient of
E1/2^E2. If E1 has a signed type and a negative value, the
resulting value is implementation-defined.
And nowhere in [expr.const]/2 it is said that such a shift, or expressions with implementation-defined values in general, are not constant expressions.
You will thus get a constant expression that has an implementation-defined value.
This is legal, insofaras it doesn't cause undefined behaviour.
The behaviour of right-shift of negative values is implementation-defined. The C and C++ standards do not guarantee it to be either arithmetic or logical; although so far as I know there has never been a CPU that didn't pick one or the other.
On my compiler, the following pseudo code (values replaced with binary):
sint32 word = (10000000 00000000 00000000 00000000);
word >>= 16;
produces a word with a bitfield that looks like this:
(11111111 11111111 10000000 00000000)
Can I rely on this behaviour for all platforms and C++ compilers?
From the following link:
INT34-C. Do not shift an expression by a negative number of bits or by greater than or equal to the number of bits that exist in the operand
Noncompliant Code Example (Right Shift)
The result of E1 >> E2 is E1 right-shifted E2 bit positions. If E1 has an unsigned type or if E1 has a signed type and a nonnegative value, the value of the result is the integral part of the quotient of E1 / 2E2. If E1 has a signed type and a negative value, the resulting value is implementation defined and can be either an arithmetic (signed) shift:
Or a logical (unsigned) shift:
This noncompliant code example fails to test whether the right operand is greater than or equal to the width of the promoted left operand, allowing undefined behavior.
unsigned int ui1;
unsigned int ui2;
unsigned int uresult;
/* Initialize ui1 and ui2 */
uresult = ui1 >> ui2;
Making assumptions about whether a right shift is implemented as an arithmetic (signed) shift or a logical (unsigned) shift can also lead to vulnerabilities. See recommendation INT13-C. Use bitwise operators only on unsigned operands.
From the latest C++20 draft:
Right-shift on signed integral types is an arithmetic right shift, which performs sign-extension.
No, you can't rely on this behaviour. Right shifting of negative quantities (which I assume your example is dealing with) is implementation defined.
In C++, no. It is implementation and/or platform dependent.
In some other languages, yes. In Java, for example, the >> operator is precisely defined to always fill using the left most bit (thereby preserving sign). The >>> operator fills using 0s. So if you want reliable behavior, one possible option would be to change to a different language. (Although obviously, this may not be an option depending on your circumstances.)
AFAIK integers may be represented as sign-magnitude in C++, in which case sign extension would fill with 0s. So you can't rely on this.
C++03 standard tells us that the result of applying the bitwise shift operators to signed types can be UB and Impl. defined for negative values. My question is following: why for operator << it has undefined behaviour, while for operator >> it is just implementation defined ? Is there a strict reason why the result of << couldn't be implementation defined also ?
Thanks in advance.
According to 5.8/2 (admittedly in C++ 98 which is all I have access to):
The value of E1 << E2 is E1
(interpreted as a bit pattern)
left shifted E2 bit positions; vacated
bits are zero filled. If E1 has an
unsigned type, the value of the result
is E1 multiplied by the quantity 2
raised to the power E2, reduced modulo
ULONG_MAX+1 if E1 has type unsigned
long, UINT_MAX+1 otherwise.
From this it looks to me like it's perfectly well defined for left shift. What's not defined is the representation of signed values (such as twos-complement) used so the numeric value of the result is implementation defined for negative values.
This is in contrast to right-shifting where the vacated bits may be zero or one filled depending on the representation of signed values.