I'm trying to align my structure and make it as small as possible using bit fields. I have to send this data back to a client, which will examine the fields to set a few data members.
The size of the structure is indeed the same, but when I set members it does not work at all.
Here's some example code:
#pragma pack(push, 1)
struct PW_INFO
{
char hash[16]; //Does not matter
uint32_t number; //Does not matter
uint32_t salt_id : 30; //Position: 0 bits
uint32_t enc_level : 7; //Position: 30 bits
uint32_t delta : 27; //Position: 37 bits
}; //Total size: 28 bytes
#pragma pack(pop)
void int64shrl(uint64_t& base, uint32_t to_shift, uint32_t position)
{
uint64_t res = static_cast<uint64_t>(to_shift);
res = Int64ShllMod32(res, position);
base |= res;
}
int32_t main()
{
std::cout << "Size of PW_INFO: " << sizeof(PW_INFO) << "\n"; //Returns 28 as expected (16 + sizeof(uint32_t) + 8)
PW_INFO pw = { "abc123", 0, 0, 0, 0 };
pw.enc_level = 105;
uint64_t base{ 0 };
&base; //debug purposes
int64shrl(base, 103, 30);
return 0;
}
Here's where it gets weird: setting the "salt_id" field (which is 30 bits into the bitfield) will yield the following result in memory:
0x003FFB8C 61 62 63 31 32 33 00 00 abc123..
0x003FFB94 00 00 00 00 00 00 00 00 ........
0x003FFB9C 00 00 00 00 00 00 00 00 ........
0x003FFBA4 69 00 00 00 i...
(Only the last 8 bytes are of concern since they represent the bit field.)
But, Int64ShllMod32 returns a correct result (the remote client undersands it perfectly):
0x003FFB7C 00 00 00 c0 19 00 00 00 ...À....
I'm guessing it has to do with alignment, if so how would I completely get rid of it? It seems even if the size is correct, it will try to align it (1 byte boundary as the #pragma directive suggests).
More information:
I use Visual Studio 2015 and its compiler.
I am not trying to write those in a different format, the reason I'm asking this is that I do NOT want to use my own format. They are reading from 64 bit bitfields everywhere, I don't have access to the source code but I see a lot of calls to Int64ShrlMod32 (from what I read, this is what the compiler produces when dealing with 8 byte structures).
The actual bitfield starts at "salt_id". 30 + 7 + 27 = 64 bits, I hope it is clearer now.
Related
For some reason I have a struct that needs to keep track of 56 bits of information ordered as 4 packs of 12 bits and 2 packs of 4 bits. This comes out to 7 bytes of information total.
I tried a bit field like so
struct foo {
uint16_t R : 12;
uint16_t G : 12;
uint16_t B : 12;
uint16_t A : 12;
uint8_t X : 4;
uint8_t Y : 4;
};
and was surprised to see sizeof(foo) evaluate to 10 on my machine (a linux x86_64 box) with g++ version 12.1. I tried reordering the fields like so
struct foo2 {
uint8_t X : 4;
uint16_t R : 12;
uint16_t G : 12;
uint16_t B : 12;
uint16_t A : 12;
uint8_t Y : 4;
};
and was surprised that the size now 8 bytes, which is what I originally expected. It's the same size as the structure I expected the first solution to effectively produce:
struct baseline {
uint16_t first;
uint16_t second;
uint16_t third;
uint8_t single;
};
I am aware of size and alignment and structure packing, but I am really stumped as to why the first ordering adds 2 extra bytes. There is no reason to add more than one byte of padding since the 56 bits I requested can be contained exactly by 7 bytes.
Minimal Working Example Try it on Wandbox
What am I missing?
PS: none of this changes if we change uint8_t to uint16_t
If we create an instance of struct foo, zero it out, set all bits in a field, and print the bytes, and do this for each field, we see the following:
R: ff 0f 00 00 00 00 00 00 00 00
G: 00 00 ff 0f 00 00 00 00 00 00
B: 00 00 00 00 ff 0f 00 00 00 00
A: 00 00 00 00 00 00 ff 0f 00 00
X: 00 00 00 00 00 00 00 f0 00 00
Y: 00 00 00 00 00 00 00 00 0f 00
So what appears to be happening is that each 12 bit field is starting in a new 16 bit storage unit. Then the first 4 bit field fills out the remaining bits in the prior 16 bit unit, then the last field takes up 4 bits in the last unit. This occupies 9 bites And since the largest field, in this case a bitfield storage unit, is 2 bytes wide, one byte of padding is added at the end.
So it appears that is 12 bit field, which has a 16 bit base type, is kept within a single 16 bit storage unit instead of being split between multiple storage units.
If we do the same for the modified struct:
X: 0f 00 00 00 00 00 00 00
R: f0 ff 00 00 00 00 00 00
G: 00 00 ff 0f 00 00 00 00
B: 00 00 00 00 ff 0f 00 00
A: 00 00 00 00 00 00 ff 0f
Y: 00 00 00 00 00 00 00 f0
We see that X takes up 4 bits of the first 16 bit storage unit, then R takes up the remaining 12 bits. The rest of the fields fill out as before. This results in 8 bytes being used, and so requires no additional padding.
While the exact details of the ordering of bitfields is implementation defined, the C standard does set a few rules.
From section 6.7.2.1p11:
An implementation may allocate any addressable storage unit large
enough to hold a bit- field. If enough space remains, a bit-field that
immediately follows another bit-field in a structure shall be packed
into adjacent bits of the same unit. If insufficient space remains,
whether a bit-field that does not fit is put into the next unit or
overlaps adjacent units is implementation-defined. The order of
allocation of bit-fields within a unit (high-order to low-order or
low-order to high-order) is implementation-defined. The alignment of
the addressable storage unit is unspecified.
And 6.7.2.1p15:
Within a structure object, the non-bit-field members and the units in
which bit-fields reside have addresses that increase in the order in
which they are declared.
Environment:
Windows 10 Home 10.0.16299
MS Visual Studio Community 2017 Version 15.6.6
Compile Settings: 64 bit - Multi-Byte Char Set
Since a few days, I'm facing a very annoying issue of memory leaks. I tried a lot but the problems still remains.
Suddently, I got the following messages on application exit :
{7196} normal block at 0x000001FD42D0EA90, 66 bytes long.
Data: <` f ) ) > 60 C6 CC 66 F7 7F 00 00 29 00 00 00 29 00 00 00
f:\dd\vctools\vc7libs\ship\atlmfc\src\mfc\oleinit.cpp(84) : {749} client block at 0x000001FD40F735F0, subtype c0, 104 bytes long.
f:\dd\vctools\vc7libs\ship\atlmfc\src\mfc\dumpcont.cpp(23) : atlTraceGeneral - a CCmdTarget object at $000001FD40F735F0, 104 bytes long
f:\dd\vctools\vc7libs\ship\atlmfc\src\mfc\plex.cpp(29) : {747} normal block at 0x000001FD40F78750, 248 bytes long. Data: < > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f:\dd\vctools\vc7libs\ship\atlmfc\src\mfc\occmgr.cpp(794) : {746} normal block at 0x000001FD40F5F1D0, 8 bytes long.
Data: < f > F0 BB A3 66 F7 7F 00 00
and also I got the following messages on application exit (I got 12 in fact)
{348} normal block at 0x00000236C88FE4A0, 69 bytes long.
Data: <` > , , > 60 C6 3E BE F6 7F 00 00 2C 00 00 00 2C 00 00 00
So, when I add the following line.....
_CrtSetBreakAlloc(348);
The compiler breaks with the message The xxxx.exe has triggered a breakpoint.
Always a the same place: the line
'pData = (CStringData*)malloc( nTotalSize );' in CAfxStringMgr::Allocate
CStringData* CAfxStringMgr::Allocate( int nChars, int nCharSize ) throw()
{
size_t nTotalSize;
CStringData* pData;
size_t nDataBytes;
ASSERT(nCharSize > 0);
if(nChars < 0)
{
ASSERT(FALSE);
return NULL;
}
nDataBytes = (nChars+1)*nCharSize;
nTotalSize = sizeof( CStringData )+nDataBytes;
pData = (CStringData*)malloc( nTotalSize );
if (pData == NULL)
return NULL;
pData->pStringMgr = this;
pData->nRefs = 1;
pData->nAllocLength = nChars;
pData->nDataLength = 0;
return pData;
}
The problem arised with function that passed CString as reference parameter. Like this one:
int CFoo::GetData(CString strMainData, CString& strData, int nPos, int nLen)
{
strData = strMainData.Mid(nPos + 2, nLen);
return ( nPos + 2 + nLen);
}
Any clue why I got this ? I'm working on that app since several weeks. I never experienced this issue. I modified the function to use char* as parameter and the, a copy to the CSring in the calling function but the problem remains. This behaviour appeared two weeks ago (but I didn't notice it directly). Could it be a compiler settings ?
Consider the following POD struct:
struct MessageWithArray {
uint32_t raw;
uint32_t myArray[10];
//MessageWithArray() : raw(0), myArray{ 10,20,30,40,50,60,70,80,90,100 } { };
};
Running the following:
#include <type_traits>
#include <iostream>
#include <fstream>
#include <string>
struct MessageWithArray {
uint32_t raw;
uint32_t myArray[10];
//MessageWithArray() : raw(0), myArray{ 10,20,30,40,50,60,70,80,90,100 } { };
};
//https://stackoverflow.com/questions/46108877/exact-definition-of-as-bytes-function
template <class T>
char* as_bytes(T& x) {
return &reinterpret_cast<char&>(x);
// or:
// return reinterpret_cast<char*>(std::addressof(x));
}
int main() {
MessageWithArray msg = { 0, {0,1,2,3,4,5,6,7,8,9} };
std::cout << "Size of MessageWithArray struct: " << sizeof(msg) << std::endl;
std::cout << "Is a POD? " << std::is_pod<MessageWithArray>() << std::endl;
std::ofstream buffer("message.txt");
buffer.write(as_bytes(msg), sizeof(msg));
return 0;
}
Gives the following output:
Size of MessageWithArray struct: 44
Is a POD? 1
A hex dump of the "message.txt" file looks like this:
00 00 00 00 00 00 00 00 01 00 00 00 02 00 00 00
03 00 00 00 04 00 00 00 05 00 00 00 06 00 00 00
07 00 00 00 08 00 00 00 09 00 00 00
Now if I uncomment the constructor (so that MessageWithArray has a zero-argument constructor), MessageWithArray becomes a non-POD struct. Then I use the constructor to initialize instead. This results in the following changes in the code:
....
struct MessageWithArray {
.....
MessageWithArray() : raw(0), myArray{ 10,20,30,40,50,60,70,80,90,100 }{ };
};
....
int main(){
MessageWithArray msg;
....
}
Running this code, I get:
Size of MessageWithArray struct: 44
Is a POD? 0
A hex dump of the "message.txt" file looks like this:
00 00 00 00 0D 0A 00 00 00 14 00 00 00 1E 00 00
00 28 00 00 00 32 00 00 00 3C 00 00 00 46 00 00
00 50 00 00 00 5A 00 00 00 64 00 00 00
Now, I'm not so interested in the actual hex values, what I'm curious about is why there is one more byte in the non-POD struct dump compared to the POD struct dump, when sizeof() declares they are the same number of bytes? Is it possible that, because the constructor makes the struct non-POD, that something hidden has been added to the struct? sizeof() should be an accurate compile-time check, correct? Is something possibly avoiding being measured by sizeof()?
Specifications: I am running this in an empty project in Visual Studio 2017 version 15.7.5, Microsoft Visual C++ 2017, on a Windows 10 machine.
Intel Core i7-4600M CPU
64-bit Operating System, x64-based processor
EDIT: I decided to initialize the struct to avoid Undefined Behaviour, and because the question is still valid with the initialization. Initializing it to a value without 10 preserves the behaviour I observed initially, because the data the array had never contained any 10s (even if it was garbage, and random).
It has nothing to do with POD-ness.
Your ofstream is opened in text mode (rather than binary mode). On windows it means that \n gets converted to \r\n.
In the second case there happened to be one 0x0A (\n) byte in the struct, that became 0x0D 0x0A (\r\n). That's why you see an extra byte.
Also, using uninitialized variables in the first case leads to undefined behaviour, which is this case didn't manifest itself.
Other answer explains the problem with writing binary data into stream opened in text mode, however this code is fundamentally wrong. There is no need to dump anything, the proper way to check sizes of those structures and verify that they are equal would be to use static_assert:
struct MessageWithArray {
uint32_t raw;
uint32_t myArray[10];
};
struct NonPodMessageWithArray {
uint32_t raw;
uint32_t myArray[10];
NonPodMessageWithArray() : raw(0), myArray{ 10,20,30,40,50,60,70,80,90,100 } {}
};
static_assert(sizeof(MessageWithArray) == sizeof(NonPodMessageWithArray));
online compiler
Following hexdump shows some data made by device i have on my hands. It stores year, month, day, hour, minute, seconds, and lenght in weird way for me (4 bytes marks for single digit in reverse order).
de 07 00 00 01 00 00 00 16 00 00 00 10 00 00 00
24 00 00 00 1d 00 00 00 15 00 00 00 X X X X
For example:
Year is marked as "000007de" aka 0x07de (=2014). Now; problem i am having is how to properly handle this in c/c++. (first 4 bytes)
How do i read those 4 bytes with "reverse" order to make proper hexadecimal for handling afterwards with like ints/longs?
If you read the value as int on the same architecture it has been generated with then you don't need to do anything, as this is the natural format for your system.
You only need to do something about this if you want to read it on a different architecture, with a different binary format.
So you can read it simply with
int32_t n;
fread(&n, sizeof int32_t, 1, FILE);
Of course the file has to be opened in binary mode and you need a 32 bit int.
If you read it in the reverse order, you can then change the endianness with something like:
uint32_t before = 0xde070000;
uint32_t after = ((before<<24) & 0xff000000) |
((before<<8) & 0xff0000) |
((before>>8) & 0xff00) |
((before>>24) & 0xff);
Edit: as pointed out in comments, this is only defined for unsigned 32-bits conversions.
I am reading in a binary file (in c++). And the header is something like this (printed in hexadecimal)
43 27 41 1A 00 00 00 00 23 00 00 00 00 00 00 00 04 63 68 72 31 FFFFFFB4 01 00 00 04 63 68 72 32 FFFFFFEE FFFFFFB7
when printed out using:
std::cout << hex << (int)mem[c];
Is there an efficient way to store 23 which is the 9th byte(?) into an integer without using stringstream? Or is stringstream the best way?
Something like
int n= mem[8]
I want to store 23 in n not 35.
You did store 23 in n. You only see 35 because you are outputting it with a routine that converts it to decimal for display. If you could look at the binary data inside the computer, you would see that it is in fact a hex 23.
You will get the same result as if you did:
int n=0x23;
(What you might think you want is impossible. What number should be stored in n for 1E? The only corresponding number is 31, which is what you are getting.)
Do you mean you want to treat the value as binary-coded decimal? In that case, you could convert it using something like:
unsigned char bcd = mem[8];
unsigned char ones = bcd % 16;
unsigned char tens = bcd / 16;
if (ones > 9 || tens > 9) {
// handle error
}
int n = 10*tens + ones;