Related
Let's say I have the following
struct MyType { long a, b, c; char buffer[remainder] }
I wanted to do something like
char buffer[4096 - offsetof(MyType, buffer)]
But it appears that it's illegal
You can do:
struct ABC {long a,b,c; }
struct MyType : ABC {char buffer[4096-sizeof(ABC)];};
static_assert(sizeof(MyType)==4096,"!");
Your problem stems from trying to use the not-yet-fully-defined MyType type while defining it. You could do this with a union:
#include <iostream>
struct MyType {
union {
struct { long a, b, c; } data;
char buffer[4096];
};
};
static_assert(sizeof(MyType) == 4096, "MyType size should be exactly 4K");
int main() {
MyType x;
x.data.a = 42;
std::cout << sizeof(x) << " " << x.data.a << "\n";
return 0;
}
The output (on my system):
4096 42
Because it's a union, the type actually holds the a/b/c tuple and buffer area in an overlapped region of memory, big enough to hold the larger of the two. So, unless your long variable are really wide, that will be the 4K buffer area :-)
In any case, that size requirement is checked by the static_assert.
That may be less than ideal as buffer takes up the entire 4K. If instead you want to ensure that buffer is only the rest of the structure (after the long variables), you can use the following:
struct MyType {
long a, b, c;
char buffer[4096 - 3 * sizeof(long)];
};
and ensure that you use x.something rather than x.data.something when accessing the a, b, or c variables.
This solves your problem by using the size of three longs (these are fully defined) instead of the size of something not yet defined. It's still a good idea to keep the static_assert to ensure overall size is what you wanted.
Technically, the compiler has total control over padding and layout. A union/struct combo combined with a static_assert sanity check might be enough for government work, but std::aligned_storage is also there to give you memory blocks that are safe to put objects in.
struct MyType {
long a, b, c;
};
using MyTypeStorage = std::aligned_storage<4096, std::alignment_of<MyType>::value>::type;
/* ... */
MyTypeStorage myTypeStorage;
MyType* x = new (&myTypeStorage) MyType {};
https://godbolt.org/z/87e7Tc
I've come to work on an ongoing project where some unions are defined as follows:
/* header.h */
typedef union my_union_t {
float data[4];
struct {
float varA;
float varB;
float varC;
float varD;
};
} my_union;
If I understand well, unions are for saving space, so sizeof(my_union_t) = MAX of the variables in it. What are the advantages of using the statement above instead of this one:
typedef struct my_struct {
float varA;
float varB;
float varC;
float varD;
};
Won't be the space allocated for both of them the same?
And how can I initialize varA,varB... from my_union?
Unions are often used when implementing a variant like object (a type field and a union of data types), or in implementing serialisation.
The way you are using a union is a recipe for disaster.
You are assuming the the struct in the union is packing the floats with no gaps between then!
The standard guarantees that float data[4]; is contiguous, but not the structure elements. The only other thing you know is that the address of varA; is the same as the address of data[0].
Never use a union in this way.
As for your question: "And how can I initialize varA,varB... from my_union?". The answer is, access the structure members in the normal long-winded way not via the data[] array.
Union are not mostly for saving space, but to implement sum types (for that, you'll put the union in some struct or class having also a discriminating field which would keep the run-time tag). Also, I suggest you to use a recent standard of C++, at least C++11 since it has better support of unions (e.g. permits more easily union of objects and their construction or initialization).
The advantage of using your union is to be able to index the n-th floating point (with 0 <= n <= 3) as u.data[n]
To assign a union field in some variable declared my_union u; just code e.g. u.varB = 3.14; which in your case has the same effect as u.data[1] = 3.14;
A good example of well deserved union is a mutable object which can hold either an int or a string (you could not use derived classes in that case):
class IntOrString {
bool isint;
union {
int num; // when isint is true
str::string str; // when isint is false
};
public:
IntOrString(int n=0) : isint(true), num(n) {};
IntOrString(std::string s) : isint(false), str(s) {};
IntOrString(const IntOrString& o): isint(o.isint)
{ if (isint) num = o.num; else str = o.str); };
IntOrString(IntOrString&&p) : isint(p.isint)
{ if (isint) num = std::move (p.num);
else str = std::move (p.str); };
~IntOrString() { if (isint) num=0; else str->~std::string(); };
void set (int n)
{ if (!isint) str->~std::string(); isint=true; num=n; };
void set (std::string s) { str = s; isint=false; };
bool is_int() const { return isint; };
int as_int() const { return (isint?num:0; };
const std::string as_string() const { return (isint?"":str;};
};
Notice the explicit calls of destructor of str field. Notice also that you can safely use IntOrString in a standard container (std::vector<IntOrString>)
See also std::optional in future versions of C++ (which conceptually is a tagged union with void)
BTW, in Ocaml, you simply code:
type intorstring = Integer of int | String of string;;
and you'll use pattern matching. If you wanted to make that mutable, you'll need to make a record or a reference of it.
You'll better use union-s in a C++ idiomatic way (see this for general advices).
I think the best way to understand unions is to just to give 2 common practical examples.
The first example is working with images. Imagine you have and RGB image that is arranged in a long buffer.
What most people would do, is represent the buffer as a char* and then loop it by 3's to get the R,G,B.
What you could do instead, is make a little union, and use that to loop over the image buffer:
union RGB
{
char raw[3];
struct
{
char R;
char G;
char B;
} colors;
}
RGB* pixel = buffer[0];
///pixel.colors.R == The red color in the first pixel.
Another very useful use for unions is using registers and bitfields.
Lets say you have a 32 bit value, that represents some HW register, or something.
Sometimes, to save space, you can split the 32 bits into bit fields, but you also want the whole representation of that register as a 32 bit type.
This obviously saves bit shift calculation that a lot of programmers use for no reason at all.
union MySpecialRegister
{
uint32_t register;
struct
{
unsigned int firstField : 5;
unsigned int somethingInTheMiddle : 25;
unsigned int lastField : 6;
} data;
}
// Now you can read the raw register into the register field
// then you can read the fields using the inner data struct
The advantage is that with a union you can access the same memory in two different ways.
In your example the union contains four floats. You can access those floats as varA, varB... which might be more descriptive names or you can access the same variables as an array data[0], data[1]... which might be more useful in loops.
With a union you can also use the same memory for different kinds of data, you might find that useful for things like writing a function to tell you if you are on a big endian or little endian CPU.
No, it is not for saving space. It is for ability to represent some binary data as various data types.
for example
#include <iostream>
#include <stdint.h>
union Foo{
int x;
struct y
{
unsigned char b0, b1, b2, b3;
};
char z[sizeof(int)];
};
int main()
{
Foo bar;
bar.x = 100;
std::cout << std::hex; // to show number in hexadec repr;
for(size_t i = 0; i < sizeof(int); i++)
{
std::cout << "0x" << (int)bar.z[i] << " "; // int is just to show values as numbers, not a characters
}
return 0;
}
output: 0x64 0x0 0x0 0x0 The same values are stored in struct bar.y, but not in array but in sturcture members. Its because my machine have a little endiannes. If it were big, than the output would be reversed: 0x0 0x0 0x0 0x64
You can achieve the same using reinterpret_cast:
#include <iostream>
#include <stdint.h>
int main()
{
int x = 100;
char * xBytes = reinterpret_cast<char*>(&x);
std::cout << std::hex; // to show number in hexadec repr;
for (size_t i = 0; i < sizeof(int); i++)
{
std::cout << "0x" << (int)xBytes[i] << " "; // (int) is just to show values as numbers, not a characters
}
return 0;
}
its usefull, for example, when you need to read some binary file, that was written on a machine with different endianess than yours. You can just access values as bytearray and swap those bytes as you wish.
Also, it is usefull when you have to deal with bit fields, but its a whole different story :)
First of all: Avoid unions where the access goes to the same memory but to different types!
Unions did not save space at all. The only define multiple names on the same memory area! And you can only store one of the elements in one time in a union.
if you have
union X
{
int x;
char y[4];
};
you can store an int OR 4 chars but not both! The general problem is, that nobody knows which data is actually stored in a union. If you store a int and read the chars, the compiler will not check that and also there is no runtime check. A solution is often to provide an additional data element in a struct to a union which contains the actual stored data type as an enum.
struct Y
{
enum { IS_CHAR, IS_INT } tinfo;
union
{
int x;
char y[4];
};
}
But in c++ you always should use classes or structs which can derive from a maybe empty parent class like this:
class Base
{
};
class Int_Type: public Base
{
...
int x;
};
class Char_Type: public Base
{
...
char y[4];
};
So you can device pointers to base which actually can hold a Int or a Char Type for you. With virtual functions you can access the members in a object oriented way of programming.
As mentioned already from Basile's answer, a useful case can be the access via different names to the same type.
union X
{
struct data
{
float a;
float b;
};
float arr[2];
};
which allows different access ways to the same data with the same type. Using different types which are stored in the same memory should be avoided at all!
I would like to take the memory generated from my struct and push it into a byte array (char array) as well as the other way around (push the byte array back into a struct). It would be even better if I could skip the string generation step and go directly to writing memory into the EEPROM. (Do not worry about the eeprom bit, I can handle that by reading & writing individual bytes)
// These are just example structs (I will be using B)
typedef struct {int a,b,c;} A;
typedef struct {A q,w,e;} B;
#define OFFSET 0 // For now
void write(B input)
{
for (int i=0;i<sizeof(B);i++)
{
eepromWrite(i+OFFSET,memof(input,i));
}
}
B read()
{
B temp;
for (int i=0;i<sizeof(B);i++)
{
setmemof(temp,i,eepromRead(i+OFFSET));
}
return temp;
}
This example I wrote is not supposed to compile, it was meant to explain my ideas in a platform independent environment.
PLEASE NOTE: memof and setmemof do not exist. This is what I am asking for though my question. An alternative answer would be to use a char array as an intermediate step.
Assuming your structures contain objects and not pointers, you can do this with a simple cast:
save_b(B b) {
unsigned char b_data[sizeof(B)];
memcpy(b_data, (unsigned char *) &b, sizeof(B));
save_bytes(b_data, sizeof(B));
}
Actually, you shouldn't need to copy from the structure into a char array. I was just hoping to make the idea clear.
Be sure to look into #pragma pack, with determines how the elements in the stuctures are aligned. Any alignment greater than one byte may increase the size unnecessarily.
I must write array of struct Data to hard disk:
struct Data {
char cmember;
/* padding bytes */
int imember;
};
AFAIK, most of compilers will add some padding bytes between cmember and imember members of Data, but I want save to file only actual data (without paddings).
I have next code for saving Datas array (in buffer instead of file for simplification):
bool saveData(Data* data, int dataLen, char* targetBuff, int buffLen)
{
int actualLen = sizeof(char) + sizeof(int); // this code force us to know internal
// representation of Data structure
int actualTotalLen = dataLen * actualLen;
if(actualTotalLen > buffLen) {
return false;
}
for(int i = 0; i < dataLen; i++) {
memcpy(targetBuff, &data[i].cmember, sizeof(char));
targetBuff += sizeof(char);
memcpy(targetBuff, &data[i].imember, sizeof(int));
targetBuff += sizeof(int);
}
return true;
}
As you can see, I calculate actual size of Data struct with the code: int actualLen = sizeof(char) + sizeof(int). Is there any alternative to this ? (something like int actualLen = actualSizeof(Data))
P.S. this is synthetic example, but I think you understand idea of my question...
Just save each member of the struct one at a time. If you overload << to write a variable to a file, you can have
myfile << mystruct.member1 << mystruct.member2;
Then you could even overload << to take an entire struct, and do that inside the struct's operator<<, so in the end you have:
myfile << mystruct;
Resulting in save code that looks like:
myfile << count;
for (int i = 0; i < count; ++i)
myFile << data[i];
IMO all that fiddling about with memory addresses and memcpy is too much of a headache when you could do it this way. This general technique is called serialization - hit google for more, it's a well-developed area.
You will have to pack your structure.
The way to do that changes depending on the compiler you are using.
For visual c++:
#pragma pack(push)
#pragma pack(1)
struct PackedStruct {
/* members */
};
#pragma pack(pop)
This will tell the compiler to not pad members in the structure and restore the pack parameter to its initial value. Be aware that this will affect performance. If this struicture is used in critical code, you might want to copy the unpacked structure into a packed structure.
Also, resist temptations to use the command line parameter that totally disable padding, this will greatly affect performance.
IIUC, you are trying to copy the values of the structure members rather than the structure as a whole and store it to disk. Your approach looks good to me. I do not agree with those suggesting #pragma pack -- since they will help you get a packed structure at runtime.
Few notes:
sizeof(char) == 1, always, by definition
use the offsetof() macro
do not try to instantiate a Data object directly from this targetBuff (i.e. via casting) -- this is when you get into alignment issues and trip. Instead, copy the members out as you did while writing the buffer and you should not have issues
There is not an easy solution to this problem. You can usually create separate structures and tell the compiler to pack them tightly, something like:
/* GNU has attributes */
struct PackedData {
char cmember;
int imember;
} __attribute__((packed));
or:
/* MSVC has headers and #pragmas */
#include <pshpack1.h>
struct PackedData {
char cmember;
int imember;
};
#include <poppack.h>
Then you have to write code that transforms your unpacked structures into packed structures and vice-versa. If you are using C++, you can create template helper functions that are predicated on the structure type and then specialize them:
template <typename T>
std::ostream& encode_to_stream(std::ostream& os, T const& object) {
return os.write((char const*)&object, sizeof(object));
}
template <typename T>
std::istream& decode_from_stream(std::istream& is, T& object) {
return is.read((char*)&object, sizeof(object));
}
template<>
std::ostream& encode_to_stream<Data>(std::ostream& os, Data const& object) {
encode_to_stream<char>(os, object.cmember);
encode_to_stream<int>(os, object.imember);
return os;
}
template <>
std::istream& decode_from_stream<Data>(std::istream& is, Data& object) {
decode_from_stream<char>(is, object.cmember);
decode_from_stream<int>(is, object.imember);
return is;
}
The bonus is that the defaults will read and write POD objects including the padding. You can specialize as necessary to optimize your storage. However, you probably want to consider endianess, versioning, and other binary storage issues as well. It might be prudent to simply write an archival class that wraps your storage and provides methods for serialization and deserialization of primitives and then an open ended method that you can specialize as needed:
class Archive {
protected:
typedef unsigned char byte;
void writeBytes(byte const* byte_ptr, std::size_t byte_size) {
m_fstream.write((char const*)byte_ptr, byte_size);
}
public:
template <typename T>
void writePOD(T const& pod) {
writeBytes((byte const*)&pod, sizeof(pod));
}
// Users are required to specialize this to use it. If it is used
// for a type that it is not specialized for, a link error will occur.
template <typename T> void serializeObject(T const& obj);
};
template<>
void Archive::serializeObject<Data>(Data const& obj) {
writePOD(cmember);
writePOD(imember);
}
This is the approach that I have always ended up at after a bunch of perturbations in between. It is nicely extensible without requiring inheritance and gives you the flexibility to change your underlying data storage format as needed. You can even specialize writePOD to do different things for different underlying data types like ensuring that multibyte integers are written in network order or whatnot.
Don't know if this will help you, but I'm in the habit of ordering the members of the structs that I intend to write to files (or send over networks) so they have as little padding as possible. This is done my putting the members with the widest datatypes and most strict alignment first:
• pointers first
•double
•long long
•long
•float
•int
•short
•char
• bitfields last
Any padding added by the compiler will come at the end of the struct data.
In other words, you could simplify your problem by eliminating the padding (if possible) by reordering the struct members:
struct Data
{
int imember;
char cmember;
/* padding bytes here */
};
Obviously this won't solve your problem if you can't reorder the struct members (because it's used by a third-party API or because you need the initial members to have specific datatypes).
I would say that you are actually looking for serialization.
There are a number of framework for serialization, but I personally prefer Google Protocol Buffers over Boost.Serialization and other approaches.
Protocol Buffers has versioning and binary/human readable output.
If you are concerned about size, you always have the possibility of compressing the data. There are lightning fast compression algorithm like LZW which offer a good ratio speed/compression for example.
Look into the #pragma pack macro for your compiler. Some compilers use #pragma options align=packed or something similar.
As you can see, I calculate actual size of Data struct with the code: int actualLen = sizeof(char) + sizeof(int). Is there any alternative to this ?
No, not in standard C++.
Your compiler might provide a compiler-specific option, though. Packed structs as shown by Graeme and Coincoin might do.
If you don't want to use pragma pack, try to manually re-order the variables,
like
struct Data {
int imember;
char cmember;
};
You said #Coincoin that can not pack. If you just need size for some reason, here is dirty solution
#define STRUCT_ELEMENTS char cmember;/* padding bytes */ int imember;
typedef struct
{
STRUCT_ELEMENTS
}paddedData;
#pragma pack(push)
#pragma pack(1)
typedef struct
{
STRUCT_ELEMENTS
}packedData;
#pragma pop
now you have size of both;
sizeof(packedData);
sizeof(paddedData);
Only reason that I can think of why you can not pack is linking this to other program. In that case you will need to pack your structure and then unpeck when working whit external program.
No, there is no way within the language proper to get this information. One way to approach a solution is to define your data classes indirectly, using some feature of the language - it could be as old-fashioned as macros and the preprocessor, or as new-fangled as tuple templates. You need something which lets you iterate over the class members systematically.
Here's a macro based approach:
#undef Data_MEMBERS
#define Data_MEMBERS(Data_OP) \
Data_OP(c, char) \
Data_OP(i, int)
#undef Data_CLASS_DEFINITION
#define Data_CLASS_DEFINITION(name, type) \
type name##member;
struct Data {
Data_MEMBERS(Data_CLASS_DEFINITION)
};
#define Data_SERIAL_SIZER(name, type) \
sizeof(type) +
#define Data_Serial_Size \
(Data_MEMBERS(Data_SERIAL_SIZER) 0)
And so forth.
If you can rewrite the struct definition, you could try to use field specifiers to get rid of the holes, like so:
struct Data {
char cmember : 1;
int imember : 4;
};
Sadly, this does not guarantee that it still won't place imember 4 bytes after the start of cmember. But many compilers will get the idea and do it anyway.
Other alternatives:
Reorder your members by size (largest first). This is an old embedded world trick to minimize holes.
Use Ada instead.
The code
type Data is record
cmember : character;
imember : integer;
end record;
for Data use record
cmember at 0 range 0..7;
imemeber at 1 range 0..31;
end record;
Does exactly what you want.
I get a transmission over the network that's an array of chars/bytes. It contains a header and some data. I'd like to map the header onto a struct. Here's an example:
#pragma pack(1)
struct Header
{
unsigned short bodyLength;
int msgID;
unsigned short someOtherValue;
unsigned short protocolVersion;
};
int main()
{
boost::array<char, 128> msgBuffer;
Header header;
for(int x = 0; x < sizeof(Header); x++)
msgBuffer[x] = 0x01; // assign some values
memcpy(&header, msgBuffer.data(), sizeof(Header));
system("PAUSE");
return 0;
}
Will this always work assuming the structure never contains any variable length fields? Is there a platform independent / idiomatic way of doing this?
Note:
I have seen quite a few libraries on the internet that let you serialize/deserialize, but I get the impression that they can only deserialize something if it has ben previously serialized with the same library. Well, I have no control over the format of the transmission. I'm definitely going to get a byte/char array where all the values just follow upon each other.
Just plain copying is very likely to break, at least if the data can come from a different architecture (or even just compiler) than what you are on. This is for reasons of:
Endianness
Structure packing
That second link is GCC-specific, but this applies to all compilers.
I recommend reading the fields byte-by-byte, and assembling larger field (ints, etc) from those bytes. This gives you control of endianness and padding.
Some processors require that certain types are properly aligned. They will not accept the specified packing and generate a hardware trap.
And even on common x86 packed structures can cause the code to run more slowly.
Also you will have to take care when working with different endianness platforms.
By the way, if you want a simple and platform-independent communication mechanism with bindings to many programming languages, then have a look at YAMI.
The #pragma pack(1) directive should work on most compilers but you can check by working out how big your data structure should be (10 in your case if my maths is correct) and using printf("%d", sizeof(Header)); to check that the packing is being done.
As others have said you still need to be wary of Endianness if you're going between architectures.
I strongly disagree with the idea of reading byte by byte. If you take care of the structure packing in the struct declaration, you can copy into the struct without a problem. For the endiannes problem again reading byte by byte solves the problem but does not give you a generic solution. That method is very lame. I have done something like this before for a similar job and it worked allright without a glitch.
Think about this. I have a structure, I also have a corresponding definition of that structure. You may construct this by hand but I have had written a parser for this and used it for other things as well.
For example, the definition of the structure you gave above is "s i s s". ( s = short , i = int ) Then I give the struct address , this definition and structure packing option of this struct to a special function that deals with the endiannes thing and voila it is done.
SwitchEndianToBig(&header, "s i s s", 4); // 4 = structure packing option
Tell me if I'm wrong, but AFAIK, doing it that way will guarantee you that the data is correct - assuming the types have the same size on your different platforms :
#include <array>
#include <algorithm>
//#pragma pack(1) // not needed
struct Header
{
unsigned short bodyLength;
int msgID;
unsigned short someOtherValue;
unsigned short protocolVersion;
float testFloat;
Header() : bodyLength(42), msgID(34), someOtherValue(66), protocolVersion(69), testFloat( 3.14f ) {}
};
int main()
{
std::tr1::array<char, 128> msgBuffer;
Header header;
const char* rawData = reinterpret_cast< const char* >( &header );
std::copy( rawData, rawData + sizeof(Header), msgBuffer.data()); // assuming msgBuffer is always big enough
system("PAUSE");
return 0;
}
If the types are different on your targeted plateforms, you have to uses aliases (typedef) for each type to be sure the size of each used type is the same.
I know who I'm communicating with, so I don't really have to worry about endianness. But I like to stay away from compiler specific commands anyway.
So how about this:
const int kHeaderSizeInBytes = 6;
struct Header
{
unsigned short bodyLength;
unsigned short msgID;
unsigned short protocolVersion;
unsigned short convertUnsignedShort(char inputArray[sizeof(unsigned short)])
{return (((unsigned char) (inputArray[0])) << 8) + (unsigned char)(inputArray[1]);}
void operator<<(char inputArray[kHeaderSizeInBytes])
{
bodyLength = convertUnsignedShort(inputArray);
msgID = convertUnsignedShort(inputArray + sizeof(bodyLength));
protocolVersion = convertUnsignedShort(inputArray + sizeof(bodyLength) + sizeof(msgID));
}
};
int main()
{
boost::array<char, 128> msgBuffer;
Header header;
for(int x = 0; x < kHeaderSizeInBytes; x++)
msgBuffer[x] = x;
header << msgBuffer.data();
system("PAUSE");
return 0;
}
Gets rid of the pragma, but it isn't as general purpose as I'd like. Every time you add a field to the header you have to modify the << function. Can you iterate over struct fields somehow, get the type of the field and call the corresponding function?