Consider the following struct definition:
#define SIZE ... // it's a positive multiple of sizeof(Foo*)
struct Foo {
Foo* ptr;
char padding[SIZE - sizeof(Foo*)];
};
Given that SIZE is a positive multiple of the pointer size (sizeof(Foo*)), is it guaranteed by the standard that sizeof(Foo) == SIZE?
If it is not guaranteed, as a practical matter, are there any platforms in common use that provide a counter-example (where the equality doesn't hold)?
Yes, I'm aware of alignas...
There is no guarantee about padding.
C++ Standard (working draft n4741) 6.7(4) Types
4 The object representation of an object of type T is the sequence of N
unsigned char objects taken up by the object of type T, where N equals
sizeof(T). The value representation of an object is the set of bits that
hold the value of type T. Bits in the object representation that are not
part of the value representation are padding bits. For trivially copyable
types, the value representation is a set of bits in the object
representation that determines a value, which is one discrete element of
an implementation-defined set of values. (41)
(41) The intent is that the memory model of C++ is compatible with that
of ISO/IEC 9899 Programming Language C.
C++ Standard (working draft n4741) 8.5.2.3(2) Sizeof
When applied to a reference or a reference type, the result is the size
of the referenced type. When applied to a class, the result is the
number of bytes in an object of that class including any padding required
for placing objects of that type in an array. The result of applying
sizeof to a potentially-overlapping subobject is the size of the type,
not the size of the subobject.78 When applied to an array, the result is
the total number of bytes in the array. This implies that the size of an
array of n elements is n times the size of an element.
I can point to no example, off-hand, where it would not hold, but based on the standard's memory model compatibility with "ISO/IEC 9899 Programming Language C", there can be no guarantees given regarding padding -- it is implementation defined.
Related
Cppreference's page on std::memset states:
std::memset
// Defined in header <cstring>
void* memset( void* dest, int ch, std::size_t count );
[...] If the object is a potentially-overlapping subobject or is not
TriviallyCopyable (e.g., scalar, C-compatible struct, or an array of trivially copyable type), the behavior is undefined.
What rules in the standard supports this claim?
(Self-answered as I think I found the complete answer during the progress of posting the question)
[cstring.syn] Covers that we need to turn to the C standard library header string.h for the meaning of std::memset:
namespace std {
// ...
void* memset(void* s, int c, size_t n);
// ...
}
/1 The contents and meaning of the header are the same as
the C standard library header <string.h>. [...]
E.g. the C11 draft via N1570 specifies that the C standard library memset copies a byte (unsigned char) into the first n bytes of the destination object [emphasis mine]:
7.24.6.1 The memset function
/1 Synopsis
#include <string.h>
void *memset(void *s, int c, size_t n);
/2 Description
The memset function copies the value of c (converted to an unsigned
char) into each of the first n characters of the object pointed to by
s.
/3 Returns
The memset function returns the value of s.
Since this is the C standard "object" does not mean the same things as in C++; Section 3.15p1:
object region of data storage in the execution environment, the contents of which can represent values
With this in mind, we go back to the C++ standard and [basic.types]/4, which tells us that the object representation of a (C++) object is the sequence of underlying unsigned char objects, and that for trivially copyable types, particularly, the value representation of the object is part of the object representation:
The object representation of an object of type T is the sequence of N unsigned char objects taken up by the object of type T, where N equals sizeof(T). [...] For trivially copyable types, the value representation is a set of bits in the object representation that determines a value, which is one discrete element of an implementation-defined set of values42.
with the footnote 42 clarifying the intent of this memory model representation allowing compatibility when interfacing with the C:
The intent is that the memory model of C++ is compatible with that of ISO/IEC 9899 Programming Language C.
[basic.types]/3 expands on the effect of byte-wise copy of the underlying bytes from a source object to destination object (both of trivially copyable type), with an additional restriction that neither the source nor the destination object of such a byte-copy operation shall be a potentially-overlapping subobject:
For any trivially copyable type T, if two pointers to T point to distinct T objects obj1 and obj2, where neither obj1 nor obj2 is a potentially-overlapping subobject, if the underlying bytes ([intro.memory]) making up obj1 are copied into obj2,41 obj2 shall subsequently hold the same value as obj1.
From this this is holds that std::memset may be used to set all (n == sizeof(T)) or a sub-set of (n < sizeof(T)) of the unsigned char objects in the underlying object representation of an object whose type is trivially copyable and where the object is not a potentially-overlapping subobject. The resulting value representation of the object is a discrete element of an implementation-defined set of values that the particular object represents. Note however that reading these values is not necessarily well-defined, one notorious example being using std::memset to set all the bits of a floating point to zero in a non-IEEE754-adhering implementation where "all bits zero" could represent a trap. Another thing to consider when using std::memset is that whilst it may create formally well-defined objects (in the standardese sense) their resulting values may violate class invariants.
Here is the code:
unsigned int a; // a is indeterminate
unsigned long long b = 1; // b is initialized to 1
std::memcpy(&a, &b, sizeof(unsigned int));
unsigned int c = a; // Is this not undefined behavior? (Implementation-defined behavior?)
Is a guaranteed by the standard to be a determinate value where we access it to initialize c? Cppreference says:
void* memcpy( void* dest, const void* src, std::size_t count );
Copies count bytes from the object pointed to by src to the object pointed to by dest. Both objects are reinterpreted as arrays of unsigned char.
But I don't see anywhere in cppreference that says if an indeterminate value is "copied to" like this, it becomes determinate.
From the standard, it seems it's analogous to this:
unsigned int a; // a is indeterminate
unsigned long long b = 1; // b is initialized to 1
auto* a_ptr = reinterpret_cast<unsigned char*>(&a);
auto* b_ptr = reinterpret_cast<unsigned char*>(&b);
a_ptr[0] = b_ptr[0];
a_ptr[1] = b_ptr[1];
a_ptr[2] = b_ptr[2];
a_ptr[3] = b_ptr[3];
unsigned int c = a; // Is this undefined behavior? (Implementation defined behavior?)
It seems like the standard leaves room for this to be allowed, because the type aliasing rules allow for the object a to be accessed as an unsigned char this way. But I can't find something that says this makes a no longer indeterminate.
Is this not undefined behavior
It's UB, because you're copying into the wrong type. [basic.types]2 and 3 permit byte copying, but only between objects of the same type. You copied from a long long into an int. That has nothing to do with the value being indeterminate. Even though you're only copying sizeof(int) bytes, the fact that you're not copying from an actual int means that you don't get the protection of those rules.
If you were copying into the value of the same type, then [basic.types]3 says that it's equivalent to simply assigning them. That is, a " shall subsequently hold the same value as" b.
TL;DR: It's implementation-defined whether there will be undefined behavior or not. Proof-style, with lines of code numbered:
unsigned int a;
The variable a is assumed to have automatic storage duration. Its lifetime begins (6.6.3/1). Since it is not a class, its lifetime begins with default initialization, in which no other initialization is performed (9.3/7.3).
unsigned long long b = 1ull;
The variable b is assumed to have automatic storage duration. Its lifetime begins (6.6.3/1). Since it is not a class, its lifetime begins with copy-initialization (9.3/15).
std::memcpy(&a, &b, sizeof(unsigned int));
Per 16.2/2, std::memcpy should have the same semantics and preconditions as the C standard library's memcpy. In the C standard 7.21.2.1, assuming sizeof(unsigned int) == 4, 4 characters are copied from the object pointed to by &b into the object pointed to by &a. (These two points are what is missing from other answers.)
At this point, the sizes of unsigned int, unsigned long long, their representations (e.g. endianness), and the size of a character are all implementation defined (to my understanding, see 6.7.1/4 and its note saying that ISO C 5.2.4.2.1 applies). I will assume that the implementation is little-endian, unsigned int is 32 bits, unsigned long long is 64 bits, and a character is 8 bits.
Now that I have said what the implementation is, I know that a has a value-representation for an unsigned int of 1u. Nothing, so far, has been undefined behavior.
unsigned int c = a;
Now we access a. Then, 6.7/4 says that
For trivially copyable types, the value representation is a set of bits in the object representation that determines a value, which is one discrete element of an implementation-defined set of values.
I know now that the value of a is determined by the implementation-defined value bits in a, which I know hold the value-representation for 1u. The value of a is then 1u.
Then like (2), the variable c is copy-initialized to 1u.
We made use of implementation-defined values to find what happens. It is possible that the implementation-defined value of 1ull is not one of the implementation-defined set of values for unsigned int. In that case, accessing a will be undefined behavior, because the standard doesn't say what happens when you access a variable with a value-representation that is invalid.
AFAIK, we can take advantage of the fact that most implementations define an unsigned int where any possible bit pattern is a valid value-representation. Therefore, there will be no undefined behavior.
Note: I updated this answer since by exploring the issue further in some of the comments has reveled cases where it would be implementation defined or even undefined in a case I did not consider originally (specifically in C++17 as well).
I believe that this is either implementation defined behavior in some cases and undefined in others (as another answer came to conclude for similar reasons). In a sense it's implementation defined if it's undefined behavior or implementation defined, so I am not sure if it being undefined in general takes precedence in such a classification.
Since std::memcpy works entirely on the object representation of the types in question (by aliasing the pointers given to unsigned char as is specified by 6.10/8.8 [basic.lval]). If the bits within the bytes in question of the unsigned long long are guaranteed to be something specific then you can manipulate them however you wish or write them into the object representation of any other type. The destination type will then use the bits to form its value based on its value representation (whatever that may be) as is defined in 6.9/4 [basic.types]:
The object representation of an object of type T is the sequence of N
unsigned char objects taken up by the object of type T, where N equals
sizeof(T). The value representation of an object is the set of bits
that hold the value of type T. For trivially copyable types, the value
representation is a set of bits in the object representation that
determines a value, which is one discrete element of an
implementation-defined set of values.
And that:
The intent is that the memory model of C++ is compatible with that of
ISO/IEC 9899 Programming Language C.
Knowing this, all that matters now is what the object representation of the integer types in question are. According to 6.9.1/7 [basic.fundemental]:
Types bool, char, char16_t, char32_t, wchar_t, and the signed and
unsigned integer types are collectively called integral types. A
synonym for integral type is integer type. The representations of
integral types shall define values by use of a pure binary numeration
system. [Example: This International Standard permits two’s
complement, ones’ complement and signed magnitude representations for
integral types. — end example ]
A footnote does clarify the definition of "binary numeration system" however:
A positional representation for integers that uses the binary digits 0
and 1, in which the values represented by successive bits are
additive, begin with 1, and are multiplied by successive integral
power of 2, except perhaps for the bit with the highest position.
(Adapted from the American National Dictionary for Information
Processing Systems.)
We also know that unsigned integers have the same value representation as signed integers, just under a modulus according to 6.9.1/4 [basic.fundemental]:
Unsigned integers shall obey the laws of arithmetic modulo 2^n where n
is the number of bits in the value representation of that particular
size of integer.
While this does not say exactly what the value representation may be, based on the specified definition of a binary numeration system, successive bits are to be additive powers of two as expected (rather than allowing the bits to be in any given order), with the exception of the maybe present sign bit. Additionally since signed and unsigned value representations this means an unsigned integer will stored as an increasing binary sequence up until 2^(n-1) (past then depending on how signed number are handled things are implementation defined).
There are still some other considerations however, such as endianness and how many padding bits may be present due to sizeof(T) only measuring the size of the object representation rather than the value representation (as stated before). Since in C++17 there is no standard way (I think) to check for endianness, this is the main factor that would leave this to be implementation defined in what the outcome would be. As for padding bits, while they may be present (but not specified where they will be from what I can tell other than the implication that they will not interrupt the contiguous sequence of bits forming the value representation of a integer), writing to them can prove potentially problematic. Since the intent of the C++ memory model is based on the C99 standard's memory model in a "comparable" way, a footnote from 6.2.6.2 (which is referenced in the C++20 standard as a note to remind that it's based on that) can be taken which say as follows:
Some combinations of padding bits might generate trap representations,
for example, if one padding bit is a parity bit. Regardless, no
arithmetic operation on valid values can generate a trap
representation other than as part of an exceptional condition such as
an overflow, and this cannot occur with unsigned types. All other
combinations of padding bits are alternative object representations of
the value specified by the value bits.
This implies that writing directly to padding bits incorrectly could potentially generate a trap representation from what I can tell.
This shows that in some cases depending on if padding bits are present and endianness, the result can be influenced in an implementation-defined manner. If some combination of padding bits is also a trap representation, this may become undefined behavior.
While not possible in C++17, in C++20 one can use std::endian in conjunction with std::has_unique_object_representations<T> (which was present in C++17) or some math with CHAR_BIT, UINT_MAX/ULLONG_MAX and the sizeof those types to ensure the expected endianness is correct as well as the absence of padding bits, allowing this to actually produce the expected result in a defined manner given what was previously established with how integers are said to be stored. Of course C++20 also further refines this and specifies that integer are to be stored in two's complement alone eliminating further implementation-specific issues.
On my system both ptrdiff_t and size_t are 64-bit.
I would like to clarify two things:
I believe that no array could be as large as size_t due to address space restrictions. Is this true?
If yes, then, is there a guarantee that ptrdiff_t will be able to hold the result of subtraction of any pointers within the max-sized array?
No, there is no such guarantee. See, for example, here: https://en.cppreference.com/w/cpp/types/ptrdiff_t
If an array is so large (greater than PTRDIFF_MAX elements, but less
than SIZE_MAX bytes), that the difference between two pointers may not
be representable as std::ptrdiff_t, the result of subtracting two such
pointers is undefined.
Most implementations artificially restrict the maximum array size to make sure that difference between two pointers pointing into the same array fits into ptrdiff_t. So, it is more than likely that on your platform the maximum allowed array size is about SIZE_MAX / 2 (try it). This is not an "address space restriction", it is just a restriction internally enforced by your implementation. Under this restriction, legal pointer subtraction ("legal" = two pointers into the same array) will not overflow.
The language specification does not require that though. Implementations are not required to restrict their array size in that way, meaning that language specification allows seemingly legal pointer subtractions to overflow and produce undefined behavior. But most implementations prefer to defend against this by restricting their array sizes.
See the "three options" here for more details: Why is the maximum size of an array "too large"?
From [support.types.layout]/3
The type size_t is an implementation-defined unsigned integer type that is large enough to contain the size in bytes of any object.
So you are guaranteed that size_t can hold the size of the largest array you can have.
ptrdiff_t unfortunately is not so guaranteed. From [support.types.layout]/2
The type ptrdiff_t is an implementation-defined signed integer type that can hold the difference of two subscripts in an array object, as described in 8.7.
Which is okay-ish but then we have [expr.add]/5
When two pointers to elements of the same array object are subtracted, the type of the result is an implementation-defined signed integral type; this type shall be the same type that is defined as std::ptrdiff_t in the header (21.2). If the expressions P and Q point to, respectively, elements x[i] and x[j] of the same array object x, the expression P - Q has the value i − j; otherwise, the behavior is undefined. [ Note: If the value i − j is not in the range of representable values of type std::ptrdiff_t, the behavior is undefined. —end note ]
Which states that ptrdiff_t may not be large enough.
At cppref, I see a weird type trait checker: std::has_unique_object_representations
From its description, I cannot imagine any type T that make std::has_unique_object_representations<T>::value is false.
Is there any counter-example?
Understanding the purpose of this trait requires understanding the distinction between an objects "value representation" and its "object representation". From the standard:
The object representation of an object of type T is the sequence of N unsigned char objects taken up by the object of type T, where N equals sizeof(T). The value representation of an object is the set of bits that hold the value of type T . For trivially copyable types, the value representation is a set of bits in the object representation that determines a value, which is one discrete element of an implementation-defined set of values.
So, object representation is the total storage of an object. But consider the following object:
struct T
{
char c;
int i;
};
On many systems, sizeof(T) will be 8. Why? Because int has to be 4-byte aligned, so the compiler inserts 3 bytes of padding between c and i. Since those three bytes are part of the object's storage (based on sizeof(T)), they are part of the object's object representation.
But those three bytes are not part of its value representation. If you modified those bytes, they would not affect the values of c or i, or anything else about them.
If you wrote an operator== overload for T, changes to those bytes would not affect its results. Which also means that if you did write an operator== overload, it could not be implemented like this:
bool operator==(const T &lhs, const T &rhs)
{
return std::memcmp(&lhs, &rhs, sizeof(T)) == 0;
}
Why not? Because two Ts can have different values for those padding bytes, but still have the same value of c and i. And thus they have the same value representation and thus should be considered equivalent.
has_unique_object_representations is true when T's object representation and its value representation exactly overlay one another (and when T is trivially copyable). So, when would you care about this?
You can write a generalized hashing function that works on any trivially copyable type T by reading its value representation as an array of bytes and hashing them. Well, you could do that, but only if the type doesn't have padding bytes. And that's what has_unique_object_representations tells you: that all of the bytes in the object representation matter.
Also, note that float types, won't necessarily have this value be true, since binary equality and floating-point equality aren't the same thing in IEEE-754. So types which contain floats will also not necessarily have this be true. Indeed, implementations that use one's-complement signed integers, or signed integers with trap representations, will also not have this be true for such types.
I've been poring over the draft standard and can't seem to find what I'm looking for.
If I have a standard-layout type
struct T {
unsigned handle;
};
Then I know that reinterpret_cast<unsigned*>(&t) == &t.handle for some T t;
The goal is to create some vector<T> v and pass &v[0] to a C function that expects a pointer to an array of unsigned integers.
So, does the standard define sizeof(T) == sizeof(unsigned) and does that imply that an array of T would have the same layout as an array of unsigned?
While this question addresses a very similar topic, I'm asking about the specific case where both the data member and the class are standard layout, and the data member is a fundamental type.
I've read some paragraphs that seem to hint that maybe it might be true, but nothing that hits the nail on the head. For example:
§ 9.2.17
Two standard-layout struct (Clause 9) types are layout-compatible if
they have the same number of non-static data members and corresponding
non-static data members (in declaration order) have layout-compatible
types
This isn't quite what I'm looking for, I don't think.
You essentially are asking, given:
struct T {
U handle;
};
whether it's guaranteed that sizeof(T) == sizeof(U). No, it is not.
Section 9.2/17 of the ISO C++03 standard says:
A pointer to a POD-struct object, suitably converted using a
reinterpret_cast, points to its initial member (or if that member is a
bit-field, then to the unit in which it resides) and vice versa.
Suppose you have an array of struct T. The vice versa part means that the address of any of the T::handle members must also be a valid address of a struct T. Now, suppose that these members are of type char and that your claim is true. This would mean that struct T would be allowed to have an unaligned address, which seems rather unlikely. The standard usually tries to not tie the hands of implementations in such a way. For your claim to be true, the standard would have to require that struct T be allowed to have unaligned addresses. And it would have to be allowed for all structures, because struct T could be a forward-declared, opaque type.
Furthermore, section 9.2/17 goes on to state:
[Note: There might therefore be unnamed padding within a POD-struct object, but not at its beginning, as necessary to achieve appropriate alignment.]
Which, taken a different way, means that there is no guarantee that there will never be padding.
I am used to Borland environments and for them:
T is a struct in your case so sizeof(T) is size of struct
that depends on #pragma pack and align setting of your compiler
so sometimes it can be greater than sizeof(unsigned) !!!
for the same reason if you have 4Byte struct (uint32) and 16Byte allign
struct T { uint32 u; };
then T a[100] is not the same as uint32 a[100];
because T is uint32 + 12 Byte empty space !!!
RETRACTION: The argument is erroneous. The proof of Lemma 2 relies on a hidden premise that the alignment of an aggregate type is determined strictly by the alignments of its member types. As Dyp points out in the commentary, that premise is not supported by the standard. It is therefore admissible for struct { Foo f } to have a more strict alignment requirement that Foo.
I'll play devil's advocate here, since no one else seems to be willing. I will argue that standard C++ (I'll refer to N3797 herein) guarantees that sizeof(T) == sizeof(U) when T is a standard layout class (9/7) with default alignment having a single default-aligned non-static data member U, e.g,
struct T { // or class, or union
U u;
};
It's well-established that:
sizeof(T) >= sizeof(U)
offsetof(T, u) == 0 (9.2/19)
U must be a standard layout type for T to be a standard layout class
u has a representation consisting of exactly sizeof(U) contiguous bytes of memory (1.8/5)
Together these facts imply that the first sizeof(U) bytes of the representation of T are occupied by the representation of u. If sizeof(T) > sizeof(U), the excess bytes must then be tail padding: unused padding bytes inserted after the representation of u in T.
The argument is, in short:
The standard details the circumstances in which an implementation may add padding to a standard-layout class,
None of those cirumstances applies in this particular instance, and hence
A conforming implementation may not add padding.
Potential Sources of Padding
Under what circumstances does the standard allow an implementation to add such padding to the representation of a standard layout class? When it's necessary for alignment: per 3.11/1, "An alignment is an implementation-defined integer value representing the number of bytes between successive addresses at which a given object can be allocated." There are two mentions of adding padding, both for alignment reasons:
5.3.3 Sizeof [expr.sizeof]/2 states "When applied to a reference or a reference type, the result is the size of the referenced type. When applied
to a class, the result is the number of bytes in an object of that class including any padding required for placing objects of that type in an array. The size of a most derived class shall be greater than zero (1.8). The result of applying sizeof to a base class subobject is the size of the base class type.77 When applied to an array, the result is the total number of bytes in the array. This implies that the size of an array of n elements is n times the size of an element."
9.2 Class members [class.mem]/13 states "Implementation alignment requirements might cause two adjacent members not to be allocated immediately after each other; so might requirements for space for managing virtual functions (10.3) and virtual base classes (10.1)."
(Notably the C++ standard does not contain a blanket statement allowing implementations to insert padding in structures as in the C standards, e.g., N1570 (C11-ish) §6.7.2.1/15 "There may be unnamed padding within a structure object, but not at its beginning." and /17 "There may be unnamed padding at the end of a structure or union.")
Clearly the text of 9.2 doesn't apply to our problem, since (a) T has only one member and thus no "adjacent members", and (b) T is standard layout and hence has no virtual functions or virtual base classes (per 9/7). Demonstrating that 5.3.3/2 doesn't allow padding in our problem is more challenging.
Some Prerequisites
Lemma 1: For any type W with default alignment, alignof(W) divides sizeof(W): By 5.3.3/2, the size of an array of n elements of type W is exactly n times sizeof(W) (i.e., there is no "external" padding between array elements). The addresses of consecutive array elements are then sizeof(W) bytes apart. By the definition of alignment, it must then be that alignof(W) divides sizeof(W).
Lemma 2: The alignment alignof(W) of a default-aligned standard layout class W with only default-aligned data members is the least common multiple LCM(W) of the alignments of the data members (or 1 if there are none): Given an address at which an object of W can be allocated, the address LCM(W) bytes away must also be appropriately aligned: the difference between the addresses of member subobjects would also be LCM(W) bytes, and the alignment of each such member subobject divides LCM(W). Per the definition of alignment in 3.11/1, we have that alignof(W) divides LCM(W). Any whole number of bytes n < LCM(W) must not be divisible by the alignment of some member v of W, so an address that is only n bytes away from an address at which an object of W can be allocated is consequently not appropriately aligned for an object of W, i.e., alignof(W) >= LCM(W). Given that alignof(W) divides LCM(W) and alignof(W) >= LCM(W), we have alignof(W) == LCM(W).
Conclusion
Application of this lemma to the original problem has the immediate consequence that alignof(T) == alignof(U). So how much padding might be "required for placing objects of that type in an array"? None. Since alignof(T) == alignof(U) by the second lemma, and alignof(U) divides sizeof(U) by the first, it must be that alignof(T) divides sizeof(U) so zero bytes of padding are required to place objects of type T in an array.
Since all possible sources of padding bytes have been eliminated, an implementation may not add padding to T and we have sizeof(T) == sizeof(U) as required.