C++ Data Member Alignment and Array Packing - c++

During a code review I've come across some code that defines a simple structure as follows:
class foo {
unsigned char a;
unsigned char b;
unsigned char c;
}
Elsewhere, an array of these objects is defined:
foo listOfFoos[SOME_NUM];
Later, the structures are raw-copied into a buffer:
memcpy(pBuff,listOfFoos,3*SOME_NUM);
This code relies on the assumptions that: a.) The size of foo is 3, and no padding is applied, and b.) An array of these objects is packed with no padding between them.
I've tried it with GNU on two platforms (RedHat 64b, Solaris 9), and it worked on both.
Are the assumptions above valid? If not, under what conditions (e.g. change in OS/compiler) might they fail?

It would definitely be safer to do:
sizeof(foo) * SOME_NUM

An array of objects is required to be contiguous, so there's never padding between the objects, though padding can be added to the end of an object (producing nearly the same effect).
Given that you're working with char's, the assumptions are probably right more often than not, but the C++ standard certainly doesn't guarantee it. A different compiler, or even just a change in the flags passed to your current compiler could result in padding being inserted between the elements of the struct or following the last element of the struct, or both.

If you copy your array like this you should use
memcpy(pBuff,listOfFoos,sizeof(listOfFoos));
This will always work as long as you allocated pBuff to the same size.
This way you are making no assumptions on padding and alignment at all.
Most compilers align a struct or class to the required alignment of the largest type included. In your case of chars that means no alignment and padding, but if you add a short for example your class would be 6 bytes large with one byte of padding added between the last char and your short.

I think the reason that this works because all of the fields in the structure are char which align one. If there is at least one field that does not align 1, the alignment of the structure/class will not be 1 (the alignment will depends on the field order and alignment).
Let see some example:
#include <stdio.h>
#include <stddef.h>
typedef struct {
unsigned char a;
unsigned char b;
unsigned char c;
} Foo;
typedef struct {
unsigned short i;
unsigned char a;
unsigned char b;
unsigned char c;
} Bar;
typedef struct { Foo F[5]; } F_B;
typedef struct { Bar B[5]; } B_F;
#define ALIGNMENT_OF(t) offsetof( struct { char x; t test; }, test )
int main(void) {
printf("Foo:: Size: %d; Alignment: %d\n", sizeof(Foo), ALIGNMENT_OF(Foo));
printf("Bar:: Size: %d; Alignment: %d\n", sizeof(Bar), ALIGNMENT_OF(Bar));
printf("F_B:: Size: %d; Alignment: %d\n", sizeof(F_B), ALIGNMENT_OF(F_B));
printf("B_F:: Size: %d; Alignment: %d\n", sizeof(B_F), ALIGNMENT_OF(B_F));
}
When executed, the result is:
Foo:: Size: 3; Alignment: 1
Bar:: Size: 6; Alignment: 2
F_B:: Size: 15; Alignment: 1
B_F:: Size: 30; Alignment: 2
You can see that Bar and F_B has alignment 2 so that its field i will be properly aligned. You can also see that Size of Bar is 6 and not 5. Similarly, the size of B_F (5 of Bar) is 30 and not 25.
So, if you is a hard code instead of sizeof(...), you will get a problem here.
Hope this helps.

It all comes down to memory alignment. Typical 32-bit machines read or write 4 bytes of memory per attempt. This structure is safe from problems because it falls under that 4 bytes easily with no confusing padding issues.
Now if the structure was as such:
class foo {
unsigned char a;
unsigned char b;
unsigned char c;
unsigned int i;
unsigned int j;
}
Your coworkers logic would probably lead to
memcpy(pBuff,listOfFoos,11*SOME_NUM);
(3 char's = 3 bytes, 2 ints = 2*4 bytes, so 3 + 8)
Unfortunately, due to padding the structure actually takes up 12 bytes. This is because you cannot fit three char's and an int into that 4 byte word, and so there's one byte of padded space there which pushes the int into it's own word. This becomes more and more of a problem the more diverse the data types become.

For situations where stuff like this is used, and I can't avoid it, I try to make the compilation break when the presumptions no longer hold. I use something like the following (or Boost.StaticAssert if the situation allows):
static_assert(sizeof(foo) <= 3);
// Macro for "static-assert" (only usefull on compile-time constant expressions)
#define static_assert(exp) static_assert_II(exp, __LINE__)
// Macro used by static_assert macro (don't use directly)
#define static_assert_II(exp, line) static_assert_III(exp, line)
// Macro used by static_assert macro (don't use directly)
#define static_assert_III(exp, line) enum static_assertion##line{static_assert_line_##line = 1/(exp)}

I would've been safe and replaced the magic number 3 with a sizeof(foo) I reckon.
My guess is that code optimised for future processor architectures will probably introduce some form of padding.
And trying to track down that sort of bug is a real pain!

As others have said, using sizeof(foo) is a safer bet. Some compilers (especially esoteric ones in the embedded world) will add a 4-byte header to classes. Others can do funky memory-alignment tricks, depending upon your compiler settings.
For a mainstream platform, you're probably alright, but its not a guarantee.

There might still be a problem with sizeof() when you are passing the data between two computers. On one of them the code might compile with padding and in the other without, in which case sizeof() would give different results. If the array data is passed from one computer to the other it will be misinterpreted because the array elements will not be found where expected.
One solution is to make sure that #pragma pack(1) is used whenever possible, but that may not be enough for the arrays. Best is to foresee the problem and use padding to a multiple of 8 bytes per array element.

Related

what does colon used in struct def mean in C++? [duplicate]

Is bitfield a C concept or C++?
Can it be used only within a structure? What are the other places we can use them?
AFAIK, bitfields are special structure variables that occupy the memory only for specified no. of bits. It is useful in saving memory and nothing else. Am I correct?
I coded a small program to understand the usage of bitfields - But, I think it is not working as expected. I expect the size of the below structure to be 1+4+2 = 7 bytes (considering the size of unsigned int is 4 bytes on my machine), But to my surprise it turns out to be 12 bytes (4+4+4). Can anyone let me know why?
#include <stdio.h>
struct s{
unsigned int a:1;
unsigned int b;
unsigned int c:2;
};
int main()
{
printf("sizeof struct s = %d bytes \n",sizeof(struct s));
return 0;
}
OUTPUT:
sizeof struct s = 12 bytes
Because a and c are not contiguous, they each reserve a full int's worth of memory space. If you move a and c together, the size of the struct becomes 8 bytes.
Moreover, you are telling the compiler that you want a to occupy only 1 bit, not 1 byte. So even though a and c next to each other should occupy only 3 bits total (still under a single byte), the combination of a and c still become word-aligned in memory on your 32-bit machine, hence occupying a full 4 bytes in addition to the int b.
Similarly, you would find that
struct s{
unsigned int b;
short s1;
short s2;
};
occupies 8 bytes, while
struct s{
short s1;
unsigned int b;
short s2;
};
occupies 12 bytes because in the latter case, the two shorts each sit in their own 32-bit alignment.
1) They originated in C, but are part of C++ too, unfortunately.
2) Yes, or within a class in C++.
3) As well as saving memory, they can be used for some forms of bit twiddling. However, both memory saving and twiddling are inherently implementation dependent - if you want to write portable software, avoid bit fields.
Its C.
Your comiler has rounded the memory allocation to 12 bytes for alignment purposes. Most computer memory syubsystems can't handle byte addressing.
Your program is working exactly as I'd expect. The compiler allocates adjacent bitfields into the same memory word, but yours are separated by a non-bitfield.
Move the bitfields next to each other and you'll probably get 8, which is the size of two ints on your machine. The bitfields would be packed into one int. This is compiler specific, however.
Bitfields are useful for saving space, but not much else.
Bitfields are widely used in firmware to map different fields in registers. This save a lot of manual bitwise operations which would have been necessary to read / write fields without it.
One disadvantage is you can't take address of bitfields.

why size of structure does not change when use 24 bit integer

I am trying to port embed code in windows platform.
I have come across below problem I am posting a sample code here.
here even after I use int24 size remains 12 bytes in windows why ?
struct INT24
{
INT32 data : 24;
};
struct myStruct
{
INT32 a;
INT32 b;
INT24 c;
};
int _tmain(int argc, _TCHAR* argv[])
{
unsigned char myArr[11] = { 0x00,0x00,0x00,0x00, 0x00,0x00,0x00,0x00, 0xFF,0xFF,0xFF };
myStruct *p = (myStruct*)myArr;
cout << sizeof(*p);
}
There are two reasons, each of which would be enough by themselves.
Presumably the size of INT32 is 4 bytes. The size of INT24 is also 4 bytes, because it contains a INT32 bit field. Since, myStruct contains 3 members of size 4, its size must therefore be at least 12.
Presumably the alignment requirement of INT32 is 4. So, even if the size of INT24 were 3, the size of myStruct would still have to be 12, because it must have at least the alignment requirement of INT32 and therefore the size of myStruct must be padded to the nearest multiple of 4.
any way or workaround ?
This is implementation specific, but the following hack may work for some compilers/cpu combinations. See the manual of your compiler for the syntax for similar feature, and the manual for your target cpu whether it supports unaligned memory access. Also do realize that unaligned memory access does have a performance penalty.
#pragma pack(push, 1)
struct INT24
{
INT32 data : 24;
};
#pragma pack(pop)
#pragma pack(push, 1)
struct myStruct
{
INT32 a;
INT32 b;
INT24 c;
};
#pragma pack(pop)
Packing a bit field might not work the same in all compilers. Be sure to check how yours behaves.
I think that a standard compliant way would be to store char arrays of sizes 3 and 4, and whenever you need to read or write one of the integer, you'd have to std::memcpy the value. That would be a bit burdensome to implement and possibly also slower than the #pragma pack hack.
Sadly for you, the compiler in optimising the code for a particular architecture reserves the right to pad out the structure by inserting spaces between members and even at the end of the structure.
Using a bit field does not reduce the size of the struct; you still get the whole of the "fielded" type in the struct.
The standard guarantees that the address of the first member of a struct is the same as the address of the struct, unless it's a polymorphic type.
But all is not lost: you can rely on the fact that an array of char will always be contiguous and contain no packing.
If CHAR_BIT is defined as 8 on your system (it probably is), you can model an array of 24 bit types on an array of char. If it's not 8 then even this approach will not work: I'd then suggest resorting to inline assembly.

Why are the values returned by sizeof() compiler dependent?

struct A
{
char c;
double d;
} a;
In mingw32-gcc.exe: sizeof a = 16
In gcc 4.6.3(ubuntu): sizeof a = 12
Why they are different? I think it should be 16, does gcc4.6.3 do some optimizations?
Compilers might perform data structure alignment for a target architecture if needed. It might done purely to improve runtime performance of the application, or in some cases is required by the processor (i.e. the program will not work if data is not aligned).
For example, most (but not all) SSE2 instructions require data to aligned on 16-byte boundary. To put it simply, everything in computer memory has an address. Let's say we have a simple array of doubles, like this:
double data[256];
In order to use SSE2 instructions that require 16-byte alignment, one must make sure that address of &data[0] is multiple of 16.
The alignment requirements differ from one architecture to another. On x86_64, it is recommended that all structures larger than 16 bytes align on 16-byte boundaries. In general, for the best performance, align data as follows:
Align 8-bit data at any address
Align 16-bit data to be contained within an aligned four-byte word
Align 32-bit data so that its base address is a multiple of four
Align 64-bit data so that its base address is a multiple of eight
Align 80-bit data so that its base address is a multiple of sixteen
Align 128-bit data so that its base address is a multiple of sixteen
Interestingly enough, most x86_64 CPUs would work with both aligned and non-aligned data. However, if the data is not aligned properly, CPU executes code significantly slower.
When compiler takes this into consideration, it may align members of the structure implicitly and that would affect its size. For example, let's say we have a structure like this:
struct A {
char a;
int b;
};
Assuming x86_64, the size of int is 32-bit or 4 bytes. Therefore, it is recommended to always make address of b a multiple of 4. But because a field size is only 1 byte, this won't be possible. Therefore, compiler would add 3 bytes of padding in between a and b implicitly:
struct A {
char a;
char __pad0[3]; /* This would be added by compiler,
without any field names - __pad0 is for
demonstration purposes */
int b;
};
How compiler does it depends not only on compiler and architecture, but on compiler settings (flags) you pass to the compiler. This behavior can also be affected using special language constructs. For example, one can ask the compiler to not perform any padding with packed attribute like this:
struct A {
char a;
int b;
} __attribute__((packed));
In your case, mingw32-gcc.exe has simply added 7 bytes between c and d to align d on 8 byte boundary. Whereas gcc 4.6.3 on Ubuntu has added only 3 to align d on 4 byte boundary.
Unless you are performing some optimizations, trying to use special extended instruction set, or have specific requirements for your data structures, I'd recommend you do not depend on specific compiler behavior and always assume that not only your structure might get padded, it might get padded differently between architectures, compilers and/or different compiler versions. Otherwise you'd need to semi-manually ensure data alignment and structure sizes using compiler attributes and settings, and make sure it all works across all compilers and platforms you are targeting using unit tests or maybe even static assertions.
For more information, please check out:
Data Alignment article on Wikipedia
Data Alignment when Migrating to 64-Bit IntelĀ® Architecture
GCC Variable Attributes
Hope it helps. Good Luck!
How to minimize padding:
It is always good to have all your struct members properly aligned and at the same time keep your structure size reasonable. Consider these 2 struct variants with members rearanged (from now on assume sizeof char, short, int, long, long long to be 1, 2, 4, 4, 8 respectively):
struct A
{
char a;
short b;
char c;
int d;
};
struct B
{
char a;
char c;
short b;
int d;
};
Both structures are supposed to keep the same data but while sizeof(struct A) will be 12 bytes, sizeof(struct B) will be 8 due to well-though-out member order which eliminated implicit padding:
struct A
{
char a;
char __pad0[1]; // implicit compiler padding
short b;
char c;
char __pad1[3]; // implicit compiler padding
int d;
};
struct B // no implicit padding
{
char a;
char c;
short b;
int d;
};
Rearranging struct members may be error prone with increase of member count. To make it less error prone - put longest at the beginning and shortest at the end:
struct B // no implicit padding
{
int d;
short b;
char a;
char c;
};
Implicit padding at the end of stuct:
Depending on your compiler, settings, platform etc used you may notice that compiler adds padding not only before struct members but also at the end (ie. after the last member). Below structure:
struct abcd
{
long long a;
char b;
};
may occupy 12 or 16 bytes (worst compilers will allow it to be 9 bytes). This padding may be easily overlooked but is very important if your structure will be array alement. It will ensure your a member in subsequent array cells/elements will be properly aligned too.
Final and random thoughts:
It will never hurt (and may actually save) you if - when working with structs - you follow these advices:
Do not rely on compiler to interleave your struct members with proper padding.
Make sure your struct (if outside array) is aligned to boundary required by its longest member.
Make sure you arrange your struct members so that longest are placed first and last member is shortest.
Make sure you explicitly padd your struct (if needed) so that if you create array of structs, every structure member has proper alignment.
Make sure that arrays of your structs are properly aligned too as although your struct may require 8 byte alignment, your compiler may align your array at 4 byte boundary.
The values returned by sizeof for structs are not mandated by any C standard. It's up to the compiler and machine architecture.
For example, it can be optimal to align data members on 4 byte boundaries: in which case the effective packed size of char c will be 4 bytes.

size of struct-array

If I have a struct A defined as:
struct A {
char* c;
float f;
int i;
};
and an array
A col[5];
then why is
sizeof(*(col+0))
16?
On your platform, 16 bytes are required to hold that structure, the structure being of type A.
You should keep in mind that *(col+0) is identical to col[0] so it's only one of the structure, not the entire array of them. If you wanted the size of the array, you would use sizeof(col).
Possibly because:
you are on a 64-bit platform and char* takes 8 bytes while int and float take 4 bytes,
you are on a 32-bit platform and char* takes 4 bytes but your compiler decided that the array would be faster if it dropped 4 bytes of padding there. Padding can be controlled on most compilers by #pragma pack(push,1) and #pragma pack(pop) respectively.
If you want to be sure, you can use offsetof (on GCC) or create an object and examine the addresses of its member fields to inspect which fields got actually padded and how much.
For starters, your original declaration was incorrect (this has now been fixed in a question edit). A is the name of the type; to declare an array named col, you want
A col[5];
not
col A[5];
sizeof(*(col+0)) is the same as sizeof col[0], which is the same as sizeof (A).
It's 16 because that's the size of that structure, for the compiler and system you're using (you haven't mentioned what it is).
I take it from the question that you were expecting something different, but you didn't say so.
Compilers may insert padding bytes between members, or after the last member, to ensure that each member is aligned properly. I find 16 bytes to be an unsurprising size for that structure on a 64-bit system -- and in this particular case, it's probably that no padding is even required.
And in case you weren't aware, sizeof yields a result in bytes, where a byte is usually (but not always) 8 bits.
Your problem is most likely that your processor platform uses 8-byte alignment on floats. So, your char* will take 4 (assuming you're on a 32-bit system) since it's a pointer which is an address. Your float will take 8, and your int will take another 4 which totals 16 bytes.
Compilers will often make certain types align on certain byte boundaries in order to speed up computation on the hardware platform in use.
For example, if you did:
struct x {
char y;
int z;
};
Your system would (probably) say the size of x was 8, padding the char out to an int inside the structure.
You can add pragmas (implementation dependent) to stop this:
#pragma pack(1)
struct x {
char y;
int z;
};
#pragma pack(0)
which would make the size of this equal to 5.
Edit: There seem to be two parts to this question. "Why is sizeof(A) equal to 16?" On balance, I see now that this is probably the question that was intended. Instead I am answering the second part, i.e. "Why is sizeof(*(col+0)) == sizeof(A)?"
col is an array. col + 0 is meaningless for arrays, so the compiler must convert col to a pointer first. Then col is effectively just an A*. Adding zero to a pointer changes nothing. Finally, you dereference it with * and are left with a simple A of size 16.
In short, sizeof(A) == sizeof(*(col+0))
PS: I have not addressed the question "Why does that one element of the array take up 16 bytes?" Others have answered that well.
On a modern x86-64 processor, char* is 8 bytes, float is 4 bytes, int is 4 bytes. So the sizes of the members added together is 16. What else would you be expecting? Did someone tell you a pointer is 4 bytes? Because that's only true for x86-32.

C++ struct containing unsigned char and int bug

Ok i have a struct in my C++ program that is like this:
struct thestruct
{
unsigned char var1;
unsigned char var2;
unsigned char var3[2];
unsigned char var4;
unsigned char var5[8];
int var6;
unsigned char var7[4];
};
When i use this struct, 3 random bytes get added before the "var6", if i delete "var5" it's still before "var6" so i know it's always before the "var6".
But if i remove the "var6" then the 3 extra bytes are gone.
If i only use a struct with a int in it, there is no extra bytes.
So there seem to be a conflict between the unsigned char and the int, how can i fix that?
The compiler is probably using its default alignment option, where members of size x are aligned on a memory boundary evenly divisible by x.
Depending on your compiler, you can affect this behaviour using a #pragma directive, for example:
#pragma pack(1)
will turn off the default alignment in Visual C++:
Specifies the value, in bytes, to be used for packing. The default value for n is 8. Valid values are 1, 2, 4, 8, and 16. The alignment of a member will be on a boundary that is either a multiple of n or a multiple of the size of the member, whichever is smaller.
Note that for low-level CPU performance reasons, it is usually best to try to align your data members so that they fall on an aligned boundary. Some CPU architectures require alignment, while others (such as Intel x86) tolerate misalignment with a decrease in performance (sometimes quite significantly).
Your data structure being aligned so that your int falls on word boundries, which for your target might be 32 or 64 bits.
You can reorganize your struct like so so that this won't happen:
struct thestruct
{
int var6;
unsigned char var1;
unsigned char var2;
unsigned char var3[2];
unsigned char var4;
unsigned char var5[8];
unsigned char var7[4];
};
Are you talking about padding bytes? That's not a bug. As allowed by the C++ standard, the compiler is adding padding to keep the members aligned. This is required for some architectures, and will greatly improve performance for others.
You're having a byte alignment problem. The compiler is adding padding to align the bytes. See this wikipedia article.
Read up on data structure alignment. Essentially, depending on the compiler and compile options, you'll get alignment onto different powers-of-2.
To avoid it, move multi-byte items (int or pointers) before single-byte (signed or unsigned char) items -- although it might still be there after your last item.
While rearranging the order you declare data members inside your struct is fine, it should be emphasized that overriding the default alignment by using #pragmas and such is a bad idea unless you know exactly what you're doing. Depending on your compiler and architecture, attempting to access unaligned data, particularly by storing the address in a pointer and later trying to dereference it, can easily give the dreaded Bus Error or other undefined behavior.