Imagine a struct made up of 32-bit, 16-bit, and 8-bit member values. Where the ordering of member values is such that each member is on it's natural boundary.
struct Foo
{
uint32_t a;
uint16_t b;
uint8_t c;
uint8_t d;
uint32_t e;
};
Member alignment and padding rules are documented for Visual C++. sizeof(Foo) on VC++ the above struct is predictably "12".
Now, I'm pretty sure the rule is that no assumption should be made about padding and alignment, but in practice, do other compilers on other operating systems make similar guarantees?
If not, is there an equivalent of "#pragma pack(1)" on GCC?
In practice, on any system where the uintXX_t types exist, you will get the desired alignment with no padding. Don't throw in ugly gcc-isms to try to guarantee it.
Edit: To elaborate on why it may be harmful to use attribute packed or aligned, it may cause the whole struct to be misaligned when used as a member of a larger struct or on the stack. This will definitely hurt performance and, on non-x86 machines, will generate much larger code. It also means it's invalid to take a pointer to any member of the struct, since code that accesses the value through a pointer will not be aware that it could be misaligned and thus could fault.
As for why it's unnecessary, keep in mind that attribute is specific to gcc and gcc-workalike compilers. The C standard does not leave alignment undefined or unspecified. It's implementation-defined which means the implementation is required to further specify and document how it behaves. gcc's behavior is, and always has been, to align each struct member on the next boundary of its natural alignment (the same alignment it would have when used outside of a struct, which is necessarily a number that evenly divides the size of the type). Since attribute is a gcc feature, if you use it you're already assuming a gcc-like compiler, but then by assumption you have the alignment you want already.
In general you are correct that it's not a safe assumption, although you will often get the packing you expect on many systems. You may want to use the packed attribute on your types when you use gcc.
E.g.
struct __attribute__((packed)) Blah { /* ... */ };
On systems that actually offer those types, it is highly likely to work. On, say, a 36-bit system those types would not be available in the first place.
GCC provides an attribute
__attribute__ ((packed))
With similar effect.
Related
A colleague of mine is working on C++ code that works with binary data arrays a lot. In certain places, he has code like
char *bytes = ...
T *p = (T*) bytes;
T v = p[i]; // UB
Here, T can be sometimes short or int (assume 16 and 32 bit respectively).
Now, unlike my colleague, I belong to the "no UB if at all possible" camp, while he is more along the lines of "if it works, it's OK". I am having a hard time trying to convince him otherwise.
Given that:
bytes really come from somewhere outside this compilation unit, being read from some binary file.
It's safe to assume that array really contains integers in the native endianness.
In practice, given mainstream C++ compilers like MSVC 2017 and gcc 4.8, and Intel x64 hardware, is such a thing really safe? I know it wouldn't be if T was, say, float (got bitten by it in the past).
char* can alias other entities without breaking strict aliasing rule.
Your code would be UB only if originally p + i wasn't a T originally.
char* byte = (char*) floats;
int *p = (int*) bytes;
int v = p[i]; // UB
but
char* byte = (char*) floats;
float *p = (float*) bytes;
float v = p[i]; // OK
If origin of byte is "unknown", compiler cannot benefit of UB for optimization and should assume we are in valid case and generate code according.
But how do you guaranty it is unknown ? Even outside the TU, something like Link-Time Optimization might allow to provide the hidden information.
Type-punned pointers are safe if one uses a construct which is recognized by the particular compiler one is using [i.e. any compiler that is configured support quality semantics if one is using straightforward constructs; neither gcc nor clang support quality semantics qualifies with optimizations are enabled, however, unless one uses -fno-strict-aliasing]. The authors of C89 were certainly aware that many applications required the use of various type-punning constructs beyond those mandated by the Standard, but thought the question of which constructs to recognize was best left as a quality-of-implementation issue. Given something like:
struct s1 { int objectClass; };
struct s2 { int objectClass; double x,y; };
struct s3 { int objectClass; char someData[32]; };
int getObjectClass(void *p) { return ((struct s1*)p)->objectClass; }
I think the authors of the Standard would have intended that the function be usable to read field objectClass of any of those structures [that is pretty much the whole purpose of the Common Initial Sequence rule] but there would be many ways by which compilers might achieve that. Some might recognize function calls as barriers to type-based aliasing analysis, while others might treat pointer casts in such a fashion. Most programs that use type punning would do several things that compilers might interpret as indications to be cautious with optimizations, so there was no particular need for a compiler to recognize any particular one of them. Further, since the authors of the Standard made no effort to forbid implementations that are "conforming" but are of such low-quality implementations as to be useless, there was no need to forbid compilers that somehow managed not to see any of the indications that storage might be used in interesting ways.
Unfortunately, for whatever reason, there hasn't been any effort by compiler vendors to find easy ways of recognizing common type-punning situations without needlessly impairing optimizations. While handling most cases would be fairly easy if compiler writers hadn't adopted designs that filter out the clearest and most useful evidence before applying optimization logic, both the designs of gcc and clang--and the mentalities of their maintainers--have evolved to oppose such a concept.
As far as I'm concerned, there is no reason why any "quality" implementation should have any trouble recognizing type punning in situations where all operations upon a byte of storage using a pointer converted to a pointer-to-PODS, or anything derived from that pointer, occur before the first time any of the following occurs:
That byte is accessed in conflicting fashion via means not derived from that pointer.
A pointer or reference is formed which will be used sometime in future to access that byte in conflicting fashion, or derive another that will.
Execution enters a function which will do one of the above before it exits.
Execution reaches the start of a bona fide loop [not, e.g. a do{...}while(0);] which will do one of the above before it exits.
A decently-designed compiler should have no problem recognizing those cases while still performing the vast majority of useful optimizations. Further, recognizing aliasing in such cases would be simpler and easier than trying to recognize it only in the cases mandated by the Standard. For those reasons, compilers that can't handle at least the above cases should be viewed as falling in the category of implementations that are of such low quality that the authors of the Standard didn't particularly want to allow, but saw no reason to forbid. Unfortunately, neither gcc nor clang offer any options to behave reasonably except by requiring that they disable type-based aliasing altogether. Unfortunately, the authors of gcc and clang would rather deride as "broken" any code needing features beyond what the Standard requires, than attempt a useful blend of optimization and semantics.
Incidentally, neither gcc nor clang should be relied upon to properly handle any situation in which storage that has been used as one type is later used as another, even when the Standard would require them to do so. Given something like:
union { struct s1 v1; struct s2 v2} unionArr[100];
void test(int i)
{
int test = unionArr[i].v2.objectClass;
unionArr[i].v1.objectClass = test;
}
Both clang and gcc will treat it as a no-op even if it is executed between code which writes unionArr[i].v2.objectClass and code which happens to reads member v1.objectClass of the same union object, thus causing them to ignore the possibility that the write to unionArr[i].v2.objectClass might affect v1.objectClass.
There has been already question about this topic (notably How to get address of some struct member in array of structures).
My question is the following :
when we use struct to describe a hardware device, so each structure member will correspond to some registers of the hardware device - how can we be sure that each member of the structure is mapped correctly on each register of the hardware ?
The ABI of the compiler dictates the alignment of the members, the user can also makes some mistake - and the only way to be sure that the mapping is done correctly is to check at run time.
The map file (at least for GNU ld) does not provide any clue about the placement of structure members.
Would there be a way to know at compiler or link time where each structure members are located ?
You can use offsetof along with, in C++, static_assert.
For example, the entirely arbitrary
#include <cstddef>
struct iom { // off,len
uint32_t rx; // +0,4
uint32_t tx; // +4,4
uint64_t clk; // +8,8
uint16_t irq; // +16,2
};
static_assert(offsetof(iom,rx)==0);
static_assert(offsetof(iom,tx)==4);
static_assert(offsetof(iom,clk)==8);
static_assert(offsetof(iom,irq)==16);
If the static_assert fails, for example because your compiler aligns the members to 64-bit boundaries, you need a compiler specific way to alter the alignment and padding. Eg, with gcc,
} __attribute__((packed));
at the end of the struct definition.
NB. I've answered for C++17.
C++11 or 14, or C11 require an error message as the second argument to static_assert, although you can wrap the whole thing in a macro to compose a nice string for you.
The offsetof macro works in C as well.
I'm building a C++ library which uses many functions and struct's defined in a C library. To avoid porting any code to C++, I add the typical conditional preprocessing to the C header files. For example,
//my_struct.h of the C library
#include <complex.h>
#ifdef __cplusplus
extern "C" {
#endif
typedef struct {
double d1,d2,d3;
#ifdef __cplusplus
std::complex<double> z1,z2,z3;
std::complex<double> *pz;
#else
double complex z1,z2,z3;
double complex *pz;
#endif
int i,j,k;
} my_struct;
//Memory allocating + initialization function
my_struct *
alloc_my_struct(double);
#ifdef __cplusplus
}
#endif
The implementation of alloc_my_struct() is compiled in C. It simply allocates memory via malloc() and initializes members of my_struct.
Now when I do the following in my C++ code,
#include "my_struct.h"
...
my_struct *const ms = alloc_my_struct(2.);
I notice that *ms always have the expected memory layout, i.e., any access such as ms->z1 evaluates to the expected value. I find this really cool considering that (correct me if I'm wrong) the memory layout of my_struct during allocation is decided by the C compiler (in my case gcc -std=c11), while during access by the C++ compiler (in my case g++ -std=c++11).
My question is : Is this compatibility standardized? If not, is there any way around it?
NOTE : I don't have enough knowledge to argue against alignment, padding, and other implementation-defined specifics. But it is noteworthy that the GNU scientific library, which is C-compiled, is implementing the same approach (although their structs do not involve C99 complex numbers) for use in C++. On the other hand, I've done sufficient research to conclude that C++11 guarantees layout compatibility between C99 double complex and std::complex<double>.
C and C++ do share memory layout rules. In both languages structs are placed in memory in the same way. And even if C++ did want to do things a little differently, placing the struct inside extern "C" {} guarantees C layout.
But what your code is doing relies on C++ std::complex and C99 complex to be the same.
So see:
https://gcc.gnu.org/ml/libstdc++/2007-02/msg00161.html
C Complex Numbers in C++?
Your program has undefined behaviour: your definitions of my_struct are not lexically identical.
You're gambling that alignment, padding and various other things will not change between the two compilers, which is bad enough… but since this is UB anything could happen even if it were true!
It may not always be identical!
In this case looks like sizeof(std::complex<double>) is identical to sizeof(double complex).
Also pay attention to the fact that the compilers may (or may not) add padding to the structs to make them aligned to a specific value, based on the optimization configuration. And the padding may not always be identical resulting in different structure sizes (between C and c++).
Links to related posts:
C/C++ Struct memory layout equivalency
I would add compiler-specific attributes to "pack" the fields,
thereby guaranteeing all the ints are adjacent and compact. This is
less about C vs. C++ and more about the fact that you are likely using
two "different" compilers when compiling in the two languages, even if
those compilers come from a single vendor.
Adding a constructor will not change the layout (though it will make
the class non-POD), but adding access specifiers like private between
the two fields may change the layout (in practice, not only in
theory).
C struct memory layout?
In C, the compiler is allowed to dictate some alignment for every
primitive type. Typically the alignment is the size of the type. But
it's entirely implementation-specific.
Padding bytes are introduced so every object is properly aligned.
Reordering is not allowed.
Possibly every remotely modern compiler implements #pragma pack which
allows control over padding and leaves it to the programmer to comply
with the ABI. (It is strictly nonstandard, though.)
From C99 §6.7.2.1:
12 Each non-bit-field member of a structure or union object is aligned
in an implementation- defined manner appropriate to its type.
13 Within a structure object, the non-bit-field members and the units
in which bit-fields reside have addresses that increase in the order
in which they are declared. A pointer to a structure object, suitably
converted, points to its initial member (or if that member is a
bit-field, then to the unit in which it resides), and vice versa.
There may be unnamed padding within a structure object, but not at its
beginning.
In general, C and C++ have compatible struct layouts, because the layout is dictated by the platform's ABI rules, not just by the language, and (for most implementations) C and C++ follow the same ABI rules for type sizes, data layout, calling conventions etc.
C++11 even defined a new term, standard-layout, which means the type will have a compatible layout to a similar type in C. That means it can't use virtual functions, private data members, multiple inheritance (and a few other things). A C++ standard-layout type should have the same layout as an equivalent C type.
As noted in other answers, your specific code is not safe in general because std::complex<double> and complex double are not equivalent types, and there is no guarantee that they are layout-compatible. However GCC's C++ standard library ensures it will work because std::complex<double> and std::complex<float> are implemented in terms of the underlying C types. Instead of containing two double, GCC's std::complex<double> has a single member of type __complex__ double, which the compiler implements identically to the equivalent C type.
GCC does this specifically to support code like yours, because it's a reasonable thing to want to do.
So combining GCC's special efforts for std::complex with the standard-layout rules and the platform ABI, means that your code will work with that implementation.
This is not necessarily portable to other C++ implementations.
Also note that by malloc() a struct with C++ object (std::complex<double>) you skipped the ctor and this is also UB - even if you expect the ctor is empty or just zero the value and harmless to be skipped, you can't complain if this breaks. So your program work is by pure luck.
I would like to avoid to fall into the XY trap so here is the original problem:
We have a small program which creates a shared memory segment on the PC. This program creates it by reading its structure from its header file (bunch of individual and nested struct definition). Basically just a .h and a .cpp file. This program will be compiled by g++.
We would like to create another program, a shared memory viewer, which displays the layout of this memory in a tree view. For that, we have to parse the previously mentioned header file and computing the offsets to read/manipulate the content of the specific part of the shared memory. We do not want to write a parser if it is not necessary especially because the header file contains additional declarations and definitions too. This program will be compiled by the same version of g++ as the previous program.
Originally, we wanted to use gccxml in the second program to parse the header file but it is based on 4.2 gcc and is cannot parse the included header files which contain C++11 code. Another idea is to use libclang to get the structure of that header file. libclang contains size information too, but I do not know if the size of the types and padding/alignment is the same in case of g++ and clang.
My question is: can you assume that the size of the C++ types and the padding/alignment of the structs will be the same when you compile the code with clang and g++? The environment (PC, OS) is the same. I am afraid we cannot, because the C++ standard does not specify the exact sizes of the types.
Do you know another solution to the original problem?
Short answer: Since clang has as a goal to "be compatible with gcc" (for both C and C++), I would say that you can expect it to generate same offsets and sizes for the same code.
Long answer:
Assuming you are using only basic types (int, short, double, char and pointers to those types), and we're restricting to gcc and clang (and their C++ versions), keeping to the same OS and same bitness (32- or 64-bit on "both sides"), then subject to actual bugs in the compiler, it should have the same structure layout.
Of course, that is a long list of restrictions, and of course the "subject to actual bugs" is a never-ending concern in these cases.
You can make your case a bit easier if you use defined size types, such as uint32_t rather than int - conversely, if you put a class member in the structure, that has virtual members, you'd be seriously in trouble - but that doesn't work very well with shared memory anyway, as it's not guaranteed to be at the same place in different applications.
Be wary of STL functionality - you may not get the same C++ library for the two compilers (you may, or may not, depending on how you installed it).
I would double check, by adding some code to print the offset and size of important members (and run with both compilers, of course) - don't forget to do this for the members deep inside some struct, since it could well be that the overall size of a struct could be identical and the content could be at different offsets.
(As others have said, I have seen projects where some code is generated with a script that prints the offsets of the struct members, and this is used as input for other programs in the project)
Actually, in this particular case, you should be fine.
The memory layout of data-structures is part of the ABI (Application Binary Interface), and gcc and clang both follow the Itanium ABI on x86 (and x86_64). Therefore, baring bugs, and provided they both compile for x86 or x86_64, they should end up with binary compatible types.
In the general case, you would typically cheat:
Use packed data structure: struct X { ... } __attribute__((packed)) __attribute__((aligned (8))); and you completely control the structure memory layout
As mentioned by Alf, have one compiler spew the offset of each member and use that to feed the generation of structures for the second compiler
Other ?
Size of data types vary from platform to platform. Instead of hardcoding, use sizeof operator to find out appropriate size applicable for the target platform, for example,
sizeof(int)
sizeof(char)
sizeof(double)
etc.
If you use fixed width integer types (http://en.cppreference.com/w/cpp/types/integer) in a C-style struct and arrange members in decreasing order of size (i.e. largest members first), it should be pretty safe.
I think I understand your issue. This is what Chrome does
COMPILE_ASSERT(sizeof(double) == 8, Double_size_not_8);
It assumes the sizes will match but checks just to make sure.
COMPILE_ASSERT is a macro. You can find the definition here but the short version is it's just what it says. An assert that happens at compile time.
If the sizes did not match then one way to deal with it is to define your header in bytes only. Instead of for example
struct SomeBinaryFileHeader {
int version;
int width;
int height;
};
You might do this
struct SomeBinaryFileHeaderReadWriteVersion {
uint_8 version_0;
uint_8 version_1;
uint_8 version_2;
uint_8 version_3;
uint_8 width_0;
uint_8 width_1;
uint_8 width_2;
uint_8 width_3;
uint_8 height_0;
uint_8 height_1;
uint_8 height_2;
uint_8 height_3;
}
Etc. and then convert from one to the other which will even work across endianness
While reading about the function InterlockedIncrement I saw the remark that the variable passed must be aligned on a 32-bit boundary. Normally I have seen the code which uses the InterlockedIncrement like this:
class A
{
public:
A();
void f();
private:
volatile long m_count;
};
A::A() : m_count(0)
{
}
void A::f()
{
::InterlockedIncrement(&m_count);
}
Does the above code work properly in multi-processor systems or should I take some more care for this?
It depends on your compiler settings. However, by default, anything eight bytes and under will be aligned on a natural boundary. Thus an "int" we be aligned on a 32-bit boundary.
Also, the "#pragma pack" directive can be used to change alignment inside a compile unit.
I would like to add that the answer assumes Microsoft C/C++ compiler. Packing rules might differ from compiler to compiler. But in general, I would assume that most C/C++ compilers for Windows use the same packing defaults just to make working with Microsoft SDK headers a bit easier.
The code looks fine (variables will be properly aligned unless you specifically do something to break that - usually involving casting or 'packed' structures).
Yes, this will work fine. Compilers usually do align unless instructed otherwise.
Strictly speaking, it really depends on your usage of A - for instance, if you pack an "A" object within a shell ITEMIDLIST, or a struct with a bad "pragma pack" the data may not be properly aligned.