Unexpected behaviour using bit-fields and unions - c++

I was experimenting with bit-fields and unions and created this:
union REG{
struct{
char posX: 7;
char posY: 7;
unsigned char dir: 2;
};
unsigned short reg;
};
And when I run sizeof( short ), I get 2, but when I run sizeof( REG ), I get 4. That's weird to me because when I sum the bits I get 7+7+2=16, which is the size in bits of a 2 byte datatype.
I'm currently using the Dev-C++ editor with compiler TDM-GCC 9.9.2 64-bit Debug.
This is my first question, so please tell me if you need more information... Thanks in advance!
Edit: Upon further experimentation I realized the size is the same (2 bytes) when I set the size of posX and posY to 6 bits. But that still puzzles because the sum is 14 bits which is less than 2 bytes...
Edit 2: Thanks to AviBerger I realized that replacing the char/unsigned char datatype with short/unsigned short the result of '''sizeof( REG )''' turns into 2. But I still can't figure out "Why does this happen?"

From the spec we have
An implementation may allocate any addressable storage unit large enough to hold a bit-
field. If enough space remains, a bit-field that immediately follows another bit-field in a
structure shall be packed into adjacent bits of the same unit. If insufficient space remains,
whether a bit-field that does not fit is put into the next unit or overlaps adjacent units is
implementation-defined. The order of allocation of bit-fields within a unit (high-order to
low-order or low-order to high-order) is implementation-defined. The alignment of the
addressable storage unit is unspecified.
So the actual behavior depends on what size allocation unit the compiler chooses for the bitfields, and whether it allows fields to span mulitple allocation units. This choice is implementation defined, but a common implementation is to use the declared type of the bit field as the allocation unit, and not allow crossing allocation unit boundaries. So when you use (unisgned) char, it uses an 8-bit allocation unit. This means that no two of the bitfields can be combined into a single byte (7+7 > 8 and 7+2 > 8), so it ends up taking 3 allocation units (bytes), which then rounds up to 4 for alignment when combined with a short in the union.
When you change the bitfield size to 6, now the second and third bitfields can fit in a byte (6+2 = 8), so it only takes two allocation units.
When you change the bitfield type to short it uses a 16-bit allocation unit, so all 3 bitfields can fit in one allocation unit.

There are several points of finesse when working with struct and union. The most common is that fields are generously padded to be aligned to the CPU's word size.
struct {
char c1;
char c2;
} s1;
seems like it should be a two byte structure, but surprisingly often sizeof (s1) will be not 2, but 4—or even 8. This was the case even in the 1980s with 16-bit machines.
This is because C and C++ compilers will align each char element of a structure to a two byte or four byte boundary. I have yet to see structure elements aligned to 8 byte boundary, but we haven't got 64 bit architecture being that needy—yet.
The solution is to invoke a compilation option to "pack structures". This can either be done on the compiler command line or including a suitable #pragma option before the structure declaration:
#pragma pack(1) // this particular syntax is valid for many compilers
struct {
char c11;
char c12;
} s2;
#pragma pack(4)

Taken from the standard (n4835):
11.4.9 Bit-fields [class.bit]
1 [...] Allocation of bit-fields within a class object is implementation-defined. Alignment of bit-fields is implementation-defined. Bit-fields are packed into some addressable allocation unit. [Note: Bit-fields straddle allocation units on some machines and not on others. Bit-fields are assigned right-to-left on some machines, left-to-right on others. —end note]
As you see the size and alignment are implementation defined. So you might get the expected behaviour on other compilers/platforms, but on your compiler/platform you get different results than you expect.

Related

Struct Bit Packing and LSB / MSB ambiguity C++

I had to write a c++ code for the following packet header:
Original image link, PNG version of the above JPEG.
Here is the struct code I wrote for the above Packet Format. I want to know if the uint8_t or the uint16_t bit fields are correct
struct TelemetryTransferFramePrimaryHeader
{
//-- 6 Ocets Long --//
//-- Master Channel ID (2 octets)--//
uint16_t TransferFrameVersionNumber : 2;
uint16_t SpacecraftID : 10;
uint16_t VirtualChannelID : 3;
uint16_t OCFFlag : 1;
//-----------------//
uint8_t MasterChannelFrameCount;
uint8_t VirtualChannelFrameCount;
//-- Transfer Frame Data Field Status (2 octets) --//
uint16_t TransferFrameSecondaryHeaderFlag : 1;
uint16_t SyncFlag : 1;
uint16_t PacketOrderFlag : 1;
uint16_t SegmentLengthID : 2;
uint16_t FirstHeaderPointer : 11;
//-----------------//
};
How do I ensure that that the LSB -> MSB is preserved in the struct ?
I keep getting confused, and I've tried reading up but it ends up confusing me even more.
PS: I am using a 32bit processor.
Exactly how bits are mapped when using bit fields is implementation-specific. So it's very hard to say for sure if you did it right, we'd need to know the exact CPU and compiler (and compiler version, of course).
In short; don't do this. Bit fields are not very usable for things like this.
Do it manually instead, by declaring the words as needed and setting the bits inside them.
IMHO anyone trying to construct a struct in this way is in a state of sin.
The C99 Standard, for example, says:
An implementation may allocate any addressable storage unit large enough to hold a bitfield. If enough space remains, a bit-field that immediately follows another bit-field in a structure shall be packed into adjacent bits of the same unit. If insufficient space remains, whether a bit-field that does not fit is put into the next unit or overlaps adjacent units is implementation-defined. The order of allocation of bit-fields within a unit (high-order to low-order or low-order to high-order) is implementation-defined. The alignment of the addressable storage unit is unspecified.
Even if you could predict that your compiler would construct bit-fields in units of (say) uint32_t, and the fields were arranged first field LS bits... you still have endian-ness to deal with !
So... as unwind says... do it by hand !
I agree that you should not do this. However STMicroelectronics uses bitfields to access the bits of its Cortex-M3/M4 microcontroller registers. So any compiler vendor that wants its users to be able to use the STMicroelectronics Cortex-M3/M4 libraries needs to support the allocation of bitfields starting at the least significant bit. In my compiler this is the default, but it is also optional so I could reverse it if I wanted to.

Why favour data structure alignment?

The type of each member of the structure usually has a default alignment i.e.each structure member is aligned on a pre-determined boundary.
For this reason the padding is performed in the following wiki example:
struct MixedData
{
char Data1;
short Data2;
int Data3;
char Data4;
};
struct MixedData /* After compilation in 32-bit x86 machine */
{
char Data1; /* 1 byte */
/* 1 byte for the following 'short' to be aligned on a 2 byte boundary
assuming that the address where structure begins is an even number */
char Padding1[1];
short Data2; /* 2 bytes */
int Data3; /* 4 bytes - largest structure member */
char Data4; /* 1 byte */
char Padding2[3]; /* 3 bytes to make total size of the structure 12 bytes */
};
What is the (practical) reason that alignment should be preserved?
Unaligned reads and writes usually require the CPU to fetch the two adjacent words from memory (instead of just one) and to apply some additional bitwise arithmetic in order to perform the designated operation properly.
Some architectures, like x86 will allow it at a performance cost. Other architectures (most notably ARM), will either raise an exception (usually resulting in a SIGBUS signal for a user process) or even "round" the address to the closest boundary which could result in some very nasty bugs.
On many architectures, aligned reads and writes from and to main memory are much faster than their unaligned counterparts.
Usually, structure are aligned on a processor dependent alignment to make accessing them as quick as possible using the processors 'natural' register size.
For 32-bit processors, it's 4 bytes (or 32-bits), for 64-bit processors it's 8 bytes.
Some (non x86) processors will generate a fault if you try to access (say) an int if it's not aligned on the correct boundary.
Communication between different devices is a practical reason to preserve alignment. With default alignment, this structure would be 24 bytes long, whereas on a 64-bit processor, it would be 48 bytes, and non of the items except the first would be in the same place.
It is normally possible to change the structure padding with a compiler/pragma directive which may negate the need for the manual padding specified in your example, but that's typically different for each compiler.

Structures in C

I got a structure like this:
struct bar {
char x;
char *y;
};
I can assume that on a 32 bit system, that padding for char will make it 4 bytes total, and a pointer in 32 bit is 4, so the total size will be 8 right?
I know it's all implementation specific, but I think if it's within 1-4, it should be padded to 4, within 5-8 to 8 and 9-16 within 16, is this right? it seems to work.
Would I be right to say that the struct will be 12 bytes in a x64 arch, because pointers are 8 bytes? Or what do you think it should be?
I can assume that on a 32 bit system,
that padding for char will make it 4
bytes total, and a pointer in 32 bit
is 4, so the total size will be 8
right?
It's not safe to assume that, but that will often be the case, yes. For x86, fields are usually 32-bit aligned. The reason for this is to increase the system's performance at the cost of memory usage (see here).
Would I be right to say that the
struct will be 12 bytes in a x64 arch,
because pointers are 8 bytes? Or what
do you think it should be?
Similarly, for x64, fields are usually 64-bit/8-byte aligned, so sizeof(bar) would be 16.
As Anders points out, however, all this goes flying out the window once you start playing with alignment via /Zp, the pack directive, or whatever else your compiler supports.
Its a compiler switch, you can't assume anything. If you assume you may get into trouble.
For instance in Visual Studio you can decide using pragma pack(1) that you want it directly on the byte boundary.
You can't assume anything in general. Every platform decides its own padding rules.
That said, any architecture that uses "natural" alignment, where operands are padded to their own size (necessary and sufficient to avoid straddling naturally-aligned pages, cachelines, etc), will make bar twice the pointer size.
So, given natural alignment rules and nothing more, 8 bytes on 32-bit, 16 bytes on 64-bit.
$9.2/12-
Nonstatic data members of a
(non-union) class declared without an
intervening access-specifier are
allocated so that later members have
higher addresses within a class
object. The order of allocation of
nonstatic data members separated by an
access-specifier is unspecified
(11.1). Implementation alignment
requirements might cause two adjacent
members not to be allocated
immediately after each other; so might
requirements for space for managing
virtual functions (10.3) and virtual
base classes (10.1).
So, it is highly implementation specific as you already mentioned.
Not quite.
Padding depends on the alignment requirement of the next member. The natural alignment of built-in data types is their size.
There is no padding before char members since their alignment requirement is 1 (assuming char is 1 byte).
For example, if a char (again assume it is one byte) is followed by a short, which, say, is 2 bytes, there may be up to 1 byte of padding because a short must be 2-byte aligned. If a char is followed by double of the size of 8, there may be up to 7 bytes of padding because a double is 8-byte aligned. On the other hand, if a short is followed by a double, the may be up to 6 bytes of padding.
And the size of a structure is a multiple of the alignment of a member with the largest alignment requirement, so there may be tail padding. In the following structure, for instance,
struct baz {
double d;
char c;
};
the member with the largest alignment requirement is d, it's alignment requirement is 8, Which gives sizeof(baz) == 2 * alignof(double). There is 7 bytes of tail padding after member c.
gcc and other modern compilers support __alignof() operator. There is also a portable version in boost.
As others have mentioned, the behaviour can't be relied upon between platforms. However, if you still need to do this, then one thing you can use is BOOST_STATIC_ASSERT() to ensure that if the assumptions are violated then you find out at compile time, eg
#include <boost/static_assert.hpp>
#if ARCH==x86 // or whatever the platform specific #define is
BOOST_STATIC_ASSERT(sizeof(bar)==8);
#elif ARCH==x64
BOOST_STATIC_ASSERT(sizeof(bar)==16);
#else ...
If alignof() is available you could also use that to test your assumption.

C++ struct size: 2+4+2+2+4 = 16 [duplicate]

This question already has answers here:
Closed 13 years ago.
Possible Duplicate:
Why isn’t sizeof for a struct equal to the sum of sizeof of each member?
Why is the sizeof(); of this structure 16 bytes? I'm compiling in g++.
struct bitmapfileheader {
unsigned short bfType;
unsigned int bfSize;
unsigned short bfReserved1;
unsigned short bfReserved2;
unsigned int bfOffBits;
};
It's because the 4 byte ints are aligned to a 4 byte boundry, so there are 2 bytes of padding after bfType.
Alignment.
Likely on your platform ints have to be 4byte aligned and shorts are 2byte aligned.
+0 -1 : bfType
+2 -3 : <padding>
+4 -7: bfSize
+8 -9: bfReserve1
+10 -11: bfReserve2
+12 -15: bfOffBits
-------------
16 bytes
Alignment is good because unaligned structures require extra work for many architectures.
The individual fields in a structure need to be aligned appropriately. The compiler will pad additional space in the structure in order to satisfy alignment requirements.
If you don't want this, you can use the UNALIGNED macro.
I think your compiler uses 4-byte allignment for the fields.
This issue comes because of a concept known as alignment. In many cases, it is desirable to have a number placed at an address that is a multiple of the size of the number in bytes (up to some maximum, often the pointer size of the platform). A variable so placed is said to be aligned to a n-byte boundary, where n is the number. The exact effects of this depend on the processor. Many processors perform math faster if the data is properly aligned. Some are even incapable of performing operations (sometimes even load operations) on unsuitably-aligned data - in order to work on such data, it has to be loaded into two registers and then a series of bit shifts and masks need to be performed to get a usable value, and then it needs to be put back. Think of it like storing half of the int in each of two buckets and needing to put them together to use it, rather than simply storing the whole int in one bucket.
In your case, the initial bfType likely needs to be aligned to a 2-byte boundary, while bfSize likely needs to be aligned to a 4-byte boundary. The compiler has to accomodate this by aligning the entire struct to 4 bytes, and leaving 2 unused bytes between bfType and bfSize.
When compiling on the same system, however, the padding is probably going to be consistent, possibly depending on compiler options and the specific ABI used (generally, you're safe on the same platform unless you are trying to make things incompatible). You can freely make another struct with the same first 5 members, and they will take up 16 bytes of the other struct, in the exact same positions.
If you really need to avoid this behavior, you will have to check your compiler documentation. Most compilers offer an attribute or keyword to declare a variable as having no alignment, and another one to indicate that a struct should have no padding. But these are rarely necessary in the general course of things.
U can pragma pack the structure to avoid padding
ISO C++03, 9.2[class.mem]/12:
Nonstatic data members of a (non-union) class declared without an intervening access-specifier are allocated so that later members have higher addresses within a class object. The order of allocation of nonstatic data members separated by an access-specifier is unspecified (11.1). Implementation alignment requirements might cause two adjacent members not to be allocated immediately after each other; so might requirements for space for managing virtual functions (10.3) and virtual base classes (10.1).
because of the way memory is allocated, there will be padding after a short
This is due to alignment - the compiler has to do some padding.

C/C++: Force Bit Field Order and Alignment

I read that the order of bit fields within a struct is platform specific. What about if I use different compiler-specific packing options, will this guarantee data is stored in the proper order as they are written? For example:
struct Message
{
unsigned int version : 3;
unsigned int type : 1;
unsigned int id : 5;
unsigned int data : 6;
} __attribute__ ((__packed__));
On an Intel processor with the GCC compiler, the fields were laid out in memory as they are shown. Message.version was the first 3 bits in the buffer, and Message.type followed. If I find equivalent struct packing options for various compilers, will this be cross-platform?
No, it will not be fully-portable. Packing options for structs are extensions, and are themselves not fully portable. In addition to that, C99 §6.7.2.1, paragraph 10 says: "The order of allocation of bit-fields within a unit (high-order to low-order or low-order to high-order) is implementation-defined."
Even a single compiler might lay the bit field out differently depending on the endianness of the target platform, for example.
Bit fields vary widely from compiler to compiler, sorry.
With GCC, big endian machines lay out the bits big end first and little endian machines lay out the bits little end first.
K&R says "Adjacent [bit-]field members of structures are packed into implementation-dependent storage units in an implementation-dependent direction. When a field following another field will not fit ... it may be split between units or the unit may be padded. An unnamed field of width 0 forces this padding..."
Therefore, if you need machine independent binary layout you must do it yourself.
This last statement also applies to non-bitfields due to padding -- however all compilers seem to have some way of forcing byte packing of a structure, as I see you already discovered for GCC.
Bitfields should be avoided - they aren't very portable between compilers even for the same platform. from the C99 standard 6.7.2.1/10 - "Structure and union specifiers" (there's similar wording in the C90 standard):
An implementation may allocate any addressable storage unit large enough to hold a bitfield. If enough space remains, a bit-field that immediately follows another bit-field in a structure shall be packed into adjacent bits of the same unit. If insufficient space remains, whether a bit-field that does not fit is put into the next unit or overlaps adjacent units is implementation-defined. The order of allocation of bit-fields within a unit (high-order to low-order or low-order to high-order) is implementation-defined. The alignment of the addressable storage unit is unspecified.
You cannot guarantee whether a bit field will 'span' an int boundary or not and you can't specify whether a bitfield starts at the low-end of the int or the high end of the int (this is independant of whether the processor is big-endian or little-endian).
Prefer bitmasks. Use inlines (or even macros) to set, clear and test the bits.
endianness are talking about byte orders not bit orders. Nowadays , it is 99% sure that bit orders are fixed. However, when using bitfields, endianness should be taken in count. See the example below.
#include <stdio.h>
typedef struct tagT{
int a:4;
int b:4;
int c:8;
int d:16;
}T;
int main()
{
char data[]={0x12,0x34,0x56,0x78};
T *t = (T*)data;
printf("a =0x%x\n" ,t->a);
printf("b =0x%x\n" ,t->b);
printf("c =0x%x\n" ,t->c);
printf("d =0x%x\n" ,t->d);
return 0;
}
//- big endian : mips24k-linux-gcc (GCC) 4.2.3 - big endian
a =0x1
b =0x2
c =0x34
d =0x5678
1 2 3 4 5 6 7 8
\_/ \_/ \_____/ \_____________/
a b c d
// - little endian : gcc (Ubuntu 4.3.2-1ubuntu11) 4.3.2
a =0x2
b =0x1
c =0x34
d =0x7856
7 8 5 6 3 4 1 2
\_____________/ \_____/ \_/ \_/
d c b a
Most of the time, probably, but don't bet the farm on it, because if you're wrong, you'll lose big.
If you really, really need to have identical binary information, you'll need to create bitfields with bitmasks - e.g. you use an unsigned short (16 bit) for Message, and then make things like versionMask = 0xE000 to represent the three topmost bits.
There's a similar problem with alignment within structs. For instance, Sparc, PowerPC, and 680x0 CPUs are all big-endian, and the common default for Sparc and PowerPC compilers is to align struct members on 4-byte boundaries. However, one compiler I used for 680x0 only aligned on 2-byte boundaries - and there was no option to change the alignment!
So for some structs, the sizes on Sparc and PowerPC are identical, but smaller on 680x0, and some of the members are in different memory offsets within the struct.
This was a problem with one project I worked on, because a server process running on Sparc would query a client and find out it was big-endian, and assume it could just squirt binary structs out on the network and the client could cope. And that worked fine on PowerPC clients, and crashed big-time on 680x0 clients. I didn't write the code, and it took quite a while to find the problem. But it was easy to fix once I did.
Thanks #BenVoigt for your very useful comment starting
No, they were created to save memory.
Linux source does use a bit field to match to an external structure: /usr/include/linux/ip.h has this code for the first byte of an IP datagram
struct iphdr {
#if defined(__LITTLE_ENDIAN_BITFIELD)
__u8 ihl:4,
version:4;
#elif defined (__BIG_ENDIAN_BITFIELD)
__u8 version:4,
ihl:4;
#else
#error "Please fix <asm/byteorder.h>"
#endif
However in light of your comment I'm giving up trying to get this to work for the multi-byte bit field frag_off.
Of course the best answer is to use a class which reads/writes bit fields as a stream. Using the C bit field structure is just not guaranteed. Not to mention it is considered unprofessional/lazy/stupid to use this in real world coding.