An object foo is written to a new file on platform 1 as:
write( file, &myFoo, sizeof(struct foo) );
...and then read on platform 2 using:
read(file, &myFoo, filesize(file) );
The foo object has the following definition:
struct foo
{
char a;
int b;
long c;
char* d;
};
What kind of issues might arise when loading foo on platform 2?
Every kind of issue!
We don't know if char, int, long or char* are the same size on different platforms.
And what happened to the stuff d pointed to?
There might also be padding between the members, which could differ between platforms. Big endian and little ending systems would store the bytes of integers and pointers in different order. If you are really unlucky, there might be a middle endian system as well.
When you do this you need to watch out for:
Data type sizes (char is the only one you can trust)
Alignment / padding
Endianness
Pointing to invalid memory
Floating point representation
ASCII vs EBCDIC ? (yeah, seriously ?)
Probably others
i think, you have to use that pack pragma to ensure there are no padding.
otherwise char will have 4 bytes in size depending on the default padding method.
char* this address pointer can have 32bits on 32bit machine but 64bits on 64bit machine.
So store a pointer directly out is nonsense.
The last one is endian.
Related
I'm trying to cast a struct into a char vector.
I wanna send my struct casted in std::vector throw a UDP socket and cast it back on the other side. Here is my struct whith the PACK attribute.
#define PACK( __Declaration__ ) __pragma( pack(push, 1) ) __Declaration__ __pragma( pack(pop) )
PACK(struct Inputs
{
uint8_t structureHeader;
int16_t x;
int16_t y;
Key inputs[8];
});
Here is test code:
auto const ptr = reinterpret_cast<char*>(&in);
std::vector<char> buffer(ptr, ptr + sizeof in);
//send and receive via udp
Inputs* my_struct = reinterpret_cast<Inputs*>(&buffer[0]);
The issue is:
All works fine except my uint8_t or int8_t.
I don't know why but whenever and wherever I put a 1Bytes value in the struct,
when I cast it back the value is not readable (but the others are)
I tried to put only 16bits values and it works just fine even with the
maximum values so all bits are ok.
I think this is something with the alignment of the bytes in the memory but i can't figure out how to make it work.
Thank you.
I'm trying to cast a struct into a char vector.
You cannot cast an arbitrary object to a vector. You can cast your object to an array of char and then copy that array into a vector (which is actually what your code is doing).
auto const ptr = reinterpret_cast<char*>(&in);
std::vector<char> buffer(ptr, ptr + sizeof in);
That second line defines a new vector and initializes it by copying the bytes that represent your object into it. This is reasonable, but it's distinct from what you said you were trying to do.
I think this is something with the alignment of the bytes in the memory
This is good intuition. If you hadn't told the compiler to pack the struct, it would have inserted padding bytes to ensure each field starts at its natural alignment. The fact that the operation isn't reversible suggests that somehow the receiving end isn't packed exactly the same way. Are you sure the receiving program has exactly the same packing directive and struct layout?
On x86, you can get by with unaligned data, but you may pay a large performance cost whenever you access an unaligned member variable. With the packing set to one, and the first field being odd-sized, you've guaranteed that the next fields will be unaligned. I'd urge you to reconsider this. Design the struct so that all the fields fall at their natural alignment boundaries and that you don't need to adjust the packing. This may make your struct a little bigger, but it will avoid all the alignment and performance problems.
If you want to omit the padding bytes in your wire format, you'll have to copy the relevant fields byte by byte into the wire format and then copy them back out on the receiving end.
An aside regarding:
#define PACK( __Declaration__ ) __pragma( pack(push, 1) ) __Declaration__ __pragma( pack(pop) )
Identifiers that begin with underscore and a capital letter or with two underscores are reserved for "the implementation," so you probably shouldn't use __Declaration__ as the macro's parameter name. ("The implementation" refers to the compiler, the standard library, and any other runtime bits the compiler requires.)
1
vector class has dynamically allocated memory and uses pointers inside. So you can't send the vector (but you can send the underlying array)
2
SFML has a great class for doing this called sf::packet. It's free, open source, and cross-platform.
I was recently working on a personal cross platform socket library for use in other personal projects and I eventually quit it for SFML. There's just TOO much to test, I was spending all my time testing to make sure stuff worked and not getting any work done on the actual projects I wanted to do.
3
memcpy is your best friend. It is designed to be portable, and you can use that to your advantage.
You can use it to debug. memcpy the thing you want to see into a char array and check that it matches what you expect.
4
To save yourself from having to do tons of robustness testing, limit yourself to only chars, 32-bit integers, and 64-bit doubles. You're using different compilers? struct packing is compiler and architecture dependent. If you have to use a packed struct, you need to guarantee that the packing is working as expected on all platforms you will be using, and that all platforms have the same endianness. Obviously, that's what you're having trouble with and I'm sorry I can't help you more with that. I would I would recommend regular serializing and would definitely avoid struct packing if I was trying to make portable sockets.
If you can make those guarantees that I mentioned, sending is really easy on LINUX.
// POSIX
void send(int fd, Inputs& input)
{
int error = sendto(fd, &input, sizeof(input), ..., ..., ...);
...
}
winsock2 uses a char* instead of a void* :(
void send(int fd, Inputs& input)
{
char buf[sizeof(input)];
memcpy(buf, &input, sizeof(input));
int error = sendto(fd, buf, sizeof(input), ..., ..., ...);
...
}
Did you tried the most simple approach of:
unsigned char *pBuff = (unsigned char*)∈
for (unsigned int i = 0; i < sizeof(Inputs); i++) {
vecBuffer.push_back(*pBuff);
pBuff++;
}
This would work for both, pack and non pack, since you will iterate the sizeof.
Is bitfield a C concept or C++?
Can it be used only within a structure? What are the other places we can use them?
AFAIK, bitfields are special structure variables that occupy the memory only for specified no. of bits. It is useful in saving memory and nothing else. Am I correct?
I coded a small program to understand the usage of bitfields - But, I think it is not working as expected. I expect the size of the below structure to be 1+4+2 = 7 bytes (considering the size of unsigned int is 4 bytes on my machine), But to my surprise it turns out to be 12 bytes (4+4+4). Can anyone let me know why?
#include <stdio.h>
struct s{
unsigned int a:1;
unsigned int b;
unsigned int c:2;
};
int main()
{
printf("sizeof struct s = %d bytes \n",sizeof(struct s));
return 0;
}
OUTPUT:
sizeof struct s = 12 bytes
Because a and c are not contiguous, they each reserve a full int's worth of memory space. If you move a and c together, the size of the struct becomes 8 bytes.
Moreover, you are telling the compiler that you want a to occupy only 1 bit, not 1 byte. So even though a and c next to each other should occupy only 3 bits total (still under a single byte), the combination of a and c still become word-aligned in memory on your 32-bit machine, hence occupying a full 4 bytes in addition to the int b.
Similarly, you would find that
struct s{
unsigned int b;
short s1;
short s2;
};
occupies 8 bytes, while
struct s{
short s1;
unsigned int b;
short s2;
};
occupies 12 bytes because in the latter case, the two shorts each sit in their own 32-bit alignment.
1) They originated in C, but are part of C++ too, unfortunately.
2) Yes, or within a class in C++.
3) As well as saving memory, they can be used for some forms of bit twiddling. However, both memory saving and twiddling are inherently implementation dependent - if you want to write portable software, avoid bit fields.
Its C.
Your comiler has rounded the memory allocation to 12 bytes for alignment purposes. Most computer memory syubsystems can't handle byte addressing.
Your program is working exactly as I'd expect. The compiler allocates adjacent bitfields into the same memory word, but yours are separated by a non-bitfield.
Move the bitfields next to each other and you'll probably get 8, which is the size of two ints on your machine. The bitfields would be packed into one int. This is compiler specific, however.
Bitfields are useful for saving space, but not much else.
Bitfields are widely used in firmware to map different fields in registers. This save a lot of manual bitwise operations which would have been necessary to read / write fields without it.
One disadvantage is you can't take address of bitfields.
struct A
{
char c;
double d;
} a;
In mingw32-gcc.exe: sizeof a = 16
In gcc 4.6.3(ubuntu): sizeof a = 12
Why they are different? I think it should be 16, does gcc4.6.3 do some optimizations?
Compilers might perform data structure alignment for a target architecture if needed. It might done purely to improve runtime performance of the application, or in some cases is required by the processor (i.e. the program will not work if data is not aligned).
For example, most (but not all) SSE2 instructions require data to aligned on 16-byte boundary. To put it simply, everything in computer memory has an address. Let's say we have a simple array of doubles, like this:
double data[256];
In order to use SSE2 instructions that require 16-byte alignment, one must make sure that address of &data[0] is multiple of 16.
The alignment requirements differ from one architecture to another. On x86_64, it is recommended that all structures larger than 16 bytes align on 16-byte boundaries. In general, for the best performance, align data as follows:
Align 8-bit data at any address
Align 16-bit data to be contained within an aligned four-byte word
Align 32-bit data so that its base address is a multiple of four
Align 64-bit data so that its base address is a multiple of eight
Align 80-bit data so that its base address is a multiple of sixteen
Align 128-bit data so that its base address is a multiple of sixteen
Interestingly enough, most x86_64 CPUs would work with both aligned and non-aligned data. However, if the data is not aligned properly, CPU executes code significantly slower.
When compiler takes this into consideration, it may align members of the structure implicitly and that would affect its size. For example, let's say we have a structure like this:
struct A {
char a;
int b;
};
Assuming x86_64, the size of int is 32-bit or 4 bytes. Therefore, it is recommended to always make address of b a multiple of 4. But because a field size is only 1 byte, this won't be possible. Therefore, compiler would add 3 bytes of padding in between a and b implicitly:
struct A {
char a;
char __pad0[3]; /* This would be added by compiler,
without any field names - __pad0 is for
demonstration purposes */
int b;
};
How compiler does it depends not only on compiler and architecture, but on compiler settings (flags) you pass to the compiler. This behavior can also be affected using special language constructs. For example, one can ask the compiler to not perform any padding with packed attribute like this:
struct A {
char a;
int b;
} __attribute__((packed));
In your case, mingw32-gcc.exe has simply added 7 bytes between c and d to align d on 8 byte boundary. Whereas gcc 4.6.3 on Ubuntu has added only 3 to align d on 4 byte boundary.
Unless you are performing some optimizations, trying to use special extended instruction set, or have specific requirements for your data structures, I'd recommend you do not depend on specific compiler behavior and always assume that not only your structure might get padded, it might get padded differently between architectures, compilers and/or different compiler versions. Otherwise you'd need to semi-manually ensure data alignment and structure sizes using compiler attributes and settings, and make sure it all works across all compilers and platforms you are targeting using unit tests or maybe even static assertions.
For more information, please check out:
Data Alignment article on Wikipedia
Data Alignment when Migrating to 64-Bit Intel® Architecture
GCC Variable Attributes
Hope it helps. Good Luck!
How to minimize padding:
It is always good to have all your struct members properly aligned and at the same time keep your structure size reasonable. Consider these 2 struct variants with members rearanged (from now on assume sizeof char, short, int, long, long long to be 1, 2, 4, 4, 8 respectively):
struct A
{
char a;
short b;
char c;
int d;
};
struct B
{
char a;
char c;
short b;
int d;
};
Both structures are supposed to keep the same data but while sizeof(struct A) will be 12 bytes, sizeof(struct B) will be 8 due to well-though-out member order which eliminated implicit padding:
struct A
{
char a;
char __pad0[1]; // implicit compiler padding
short b;
char c;
char __pad1[3]; // implicit compiler padding
int d;
};
struct B // no implicit padding
{
char a;
char c;
short b;
int d;
};
Rearranging struct members may be error prone with increase of member count. To make it less error prone - put longest at the beginning and shortest at the end:
struct B // no implicit padding
{
int d;
short b;
char a;
char c;
};
Implicit padding at the end of stuct:
Depending on your compiler, settings, platform etc used you may notice that compiler adds padding not only before struct members but also at the end (ie. after the last member). Below structure:
struct abcd
{
long long a;
char b;
};
may occupy 12 or 16 bytes (worst compilers will allow it to be 9 bytes). This padding may be easily overlooked but is very important if your structure will be array alement. It will ensure your a member in subsequent array cells/elements will be properly aligned too.
Final and random thoughts:
It will never hurt (and may actually save) you if - when working with structs - you follow these advices:
Do not rely on compiler to interleave your struct members with proper padding.
Make sure your struct (if outside array) is aligned to boundary required by its longest member.
Make sure you arrange your struct members so that longest are placed first and last member is shortest.
Make sure you explicitly padd your struct (if needed) so that if you create array of structs, every structure member has proper alignment.
Make sure that arrays of your structs are properly aligned too as although your struct may require 8 byte alignment, your compiler may align your array at 4 byte boundary.
The values returned by sizeof for structs are not mandated by any C standard. It's up to the compiler and machine architecture.
For example, it can be optimal to align data members on 4 byte boundaries: in which case the effective packed size of char c will be 4 bytes.
Ok i have a struct in my C++ program that is like this:
struct thestruct
{
unsigned char var1;
unsigned char var2;
unsigned char var3[2];
unsigned char var4;
unsigned char var5[8];
int var6;
unsigned char var7[4];
};
When i use this struct, 3 random bytes get added before the "var6", if i delete "var5" it's still before "var6" so i know it's always before the "var6".
But if i remove the "var6" then the 3 extra bytes are gone.
If i only use a struct with a int in it, there is no extra bytes.
So there seem to be a conflict between the unsigned char and the int, how can i fix that?
The compiler is probably using its default alignment option, where members of size x are aligned on a memory boundary evenly divisible by x.
Depending on your compiler, you can affect this behaviour using a #pragma directive, for example:
#pragma pack(1)
will turn off the default alignment in Visual C++:
Specifies the value, in bytes, to be used for packing. The default value for n is 8. Valid values are 1, 2, 4, 8, and 16. The alignment of a member will be on a boundary that is either a multiple of n or a multiple of the size of the member, whichever is smaller.
Note that for low-level CPU performance reasons, it is usually best to try to align your data members so that they fall on an aligned boundary. Some CPU architectures require alignment, while others (such as Intel x86) tolerate misalignment with a decrease in performance (sometimes quite significantly).
Your data structure being aligned so that your int falls on word boundries, which for your target might be 32 or 64 bits.
You can reorganize your struct like so so that this won't happen:
struct thestruct
{
int var6;
unsigned char var1;
unsigned char var2;
unsigned char var3[2];
unsigned char var4;
unsigned char var5[8];
unsigned char var7[4];
};
Are you talking about padding bytes? That's not a bug. As allowed by the C++ standard, the compiler is adding padding to keep the members aligned. This is required for some architectures, and will greatly improve performance for others.
You're having a byte alignment problem. The compiler is adding padding to align the bytes. See this wikipedia article.
Read up on data structure alignment. Essentially, depending on the compiler and compile options, you'll get alignment onto different powers-of-2.
To avoid it, move multi-byte items (int or pointers) before single-byte (signed or unsigned char) items -- although it might still be there after your last item.
While rearranging the order you declare data members inside your struct is fine, it should be emphasized that overriding the default alignment by using #pragmas and such is a bad idea unless you know exactly what you're doing. Depending on your compiler and architecture, attempting to access unaligned data, particularly by storing the address in a pointer and later trying to dereference it, can easily give the dreaded Bus Error or other undefined behavior.
I've got this DLL I made. It's injected to another process. Inside the other process,
I do a search from it's memory space with the following function:
void MyDump(const void *m, unsigned int n)
{
const char *p = reinterpret_cast(m);
for (unsigned int i = 0; i < n; ++i) {
// Do something with p[i]...
}
}
Now my question. If the target process uses a data structure, let's say
struct S
{
unsigned char a;
unsigned char b;
unsigned char c;
};
Is it always presented the same way in the process' memory? I mean, if S.a = 2 (which always follows b = 3, c = 4), is the structure presented in a continuous row in the process' memory space, like
Offset
---------------------
0x0000 | 0x02 0x03 0x04
Or can those variables be in a different places there, like
Offset
---------------------
0x0000 | 0x00 0x02 0x00
0x03fc | 0x00 0x03 0x04
If the latter one, how to reconstruct the data-structure from various points from the memory?
Many thanks in advance,
nhaa123
If your victim is written in C or C++, and the datatypes used are truly that simple, then you'll always find them as a single block of bytes in memory.
But as soon as you have C++ types like std::string that observation no longer holds. For starters, the exact layout will differ between C++ compilers, and even different versions of the same compiler. The bytes of a std::string will likely not be in a contiguous array, but sometimes they are. If they're split in two, finding the second half probably will not help you in finding the first half.
Not throw in more complicated environments like a JIT'ting JVM running a Java app. The types you encounter in memory are very very complex; one could write a book about decoding them.
The order of member will always be the same and the structure will occupy a contiguous memory block.
Depending on a compiler padding might be added between members but it still will be the same if the program is recompiled with the same compiler and the same settings. If padding is added and you are unaware of it you can't detect it reliably at runtime - all the information the compiler had is lost to that moment and you are left to just analyze the patterns and guess.
It depends on the alignment of the structure.
If you have something like this:
struct A
{
int16_t a;
char b;
int32_t c;
char d;
}
then by default on 32bit platform( I dont know if that is true for 64bit ), the offset of c is 4 as there is one byte padded after b, and after d there are 3 more bytess padded at the end (if I remember correctly).
It will be different if the structure has a specified alignment.
Now my question. If the target process uses a data structure [...] is it always presented the same way in the process' memory? I mean, if S.a = 2 (which always follows b = 3, c = 4), is the structure presented in a continuous row in the process' memory space?
Yes, however it will often be padded to align members in ways you may not expect. Thus, simply recreating the data structure in order to interface with it via code injection.
I would highly recommend using ReClassEx or ReClass.NET, two open-source programs created specifically for reconstructing data structures from memory and generating useable C++ code! Check out a screenshot: