Related
Here's a little puzzle I couldn't find a good answer for:
Given a struct with bitfields, such as
struct A {
unsigned foo:13;
unsigned bar:19;
};
Is there a (portable) way in C++ to get the correct mask for one of the bitfields, preferably as a compile-time constant function or template?
Something like this:
constinit unsigned mask = getmask<A::bar>(); // mask should be 0xFFFFE000
In theory, at runtime, I could crudely do:
unsigned getmask_bar() {
union AA {
unsigned mask;
A fields;
} aa{};
aa.fields.bar -= 1;
return aa.mask;
}
That could even be wrapped in a macro (yuck!) to make it "generic".
But I guess you can readily see the various deficiencies of this method.
Is there a nicer, generic C++ way of doing it? Or even a not-so-nice way? Is there something useful coming up for the next C++ standard(s)? Reflection?
Edit: Let me add that I am trying to find a way of making bitfield manipulation more flexible, so that it is up to the programmer to modify multiple fields at the same time using masking. I am after terse notation, so that things can be expressed concisely without lots of boilerplate. Think working with hardware registers in I/O drivers as a use case.
Unfortunately, there is no better way - in fact, there is no way to extract individual adjacent bit fields from a struct by inspecting its memory directly in C++.
From Cppreference:
The following properties of bit-fields are implementation-defined:
The value that results from assigning or initializing a signed bit-field with a value out of range, or from incrementing a signed
bit-field past its range.
Everything about the actual allocation details of bit-fields within the class object
For example, on some platforms, bit-fields don't straddle bytes, on others they do
Also, on some platforms, bit-fields are packed left-to-right, on others right-to-left
Your compiler might give you stronger guarantees; however, if you do rely on the behavior of a specific compiler, you can't expect your code to work with a different compiler/architecture pair. GCC doesn't even document their bit field packing, as far as I can tell, and it differs from one architecture to the next. So your code might work on a specific version of GCC on x86-64 but break on literally everything else, including other versions of the same compiler.
If you really want to be able to extract bitfields from a random structure in a generic way, your best bet is to pass a function pointer around (instead of a mask); that way, the function can access the field in a safe way and return the value to its caller (or set a value instead).
Something like this:
template<typename T>
auto extractThatBitField(const void *ptr) {
return static_cast<const T *>(ptr)->m_thatBitField;
}
auto *extractor1 = &extractThatBitField<Type1>;
auto *extractor2 = &extractThatBitField<Type2>;
/* ... */
Now, if you have a pair of {pointer, extractor}, you can get the value of the bitfield safely. (Of course, the extractor function has to match the type of the object behind that pointer.) It's not much overhead compared to having a {pointer, mask} pair instead; the function pointer is maybe 4 bytes larger than the mask on a 64-bit machine (if at all). The extractor function itself will just be a memory load, some bit twiddling, and a return instruction. It'll still be super fast.
This is portable and supported by the C++ standard, unlike inspecting the bits of a bitfield directly.
Alternatively, C++ allows casting between standard-layout structs that have common initial members. (Though keep in mind that this falls apart as soon as inheritance or private/protected members get involved! The first solution, above, works for all those cases as well.)
struct Common {
int m_a : 13;
int m_b : 19;
int : 0; //Needed to ensure the bit fields end on a byte boundary
};
struct Type1 {
int m_a : 13;
int m_b : 19;
int : 0;
Whatever m_whatever;
};
struct Type2 {
int m_a : 13;
int m_b : 19;
int : 0;
Something m_something;
};
int getFieldA(const void *ptr) {
//We still can't do type punning directly due
//to weirdness in various compilers' aliasing resolution.
//std::memcpy is the official way to do type punning.
//This won't compile to an actual memcpy call.
Common tmp;
std::memcpy(&tmp, ptr, sizeof(Common));
return tmp.m_a;
}
See also: Can memcpy be used for type punning?
While there might be an XY issue that led me here, I am curious about the limits of the new bit_cast in C++20, and this seems a good illustration of it.
Note that there is nothing inherently wrong with std::memcpy and it does not cause undefined behavior. It is also enough of an idiom to generally be optimized out to be the equivalent of a cast. The problem I have is that the intent of its usage is not as clear as it could be and it seems more C than C++.
How could you use bit_cast to do the same thing (without any undefined behavior) as the following:
#include <cstring>
void ptr2array(short (&x)[], int offset, const void * ptr)
{
std::memcpy(&(x[offset]), ptr, sizeof(void *));
}
void array2ptr(void * & ptr, int offset, short (&x)[])
{
std::memcpy(ptr, &(x[offset]), sizeof(void *));
}
Note that the following is fundamentally a no-op (tested here: https://godbolt.org/z/ce9Ecx8TG)
void * tst(void * orig)
{
short x[101];
int offset = 67;
void * ret=orig;
ptr2array(x, offset, ret);
array2ptr(ret, offset, x);
return ret;
}
Keep in mind that memcpy works fine with the misalignment (using offset).
Also requiring an array of shorts, as opposed to char, avoids some of the leniencies with char, but keeps the alignment issues.
bit_cast is for doing conversions between objects. It creates a new object whose binary data comes from some other object. It doesn't work on language arrays because, while they are objects, they don't work like objects. You can't pass them around the same way you can other objects. They decay to pointers, you can't return arrays, etc.
You can bit_cast with std::array and similar types, but even those are potentially a problem. std::array<char, sizeof(void*)> for example is not required to have the size of a void*. And bit_cast only works if the source and destination objects have the same size.
bit_cast also returns the object it creates. Which means you cannot (easily) bit_cast into existing storage. Nor is it easy to do a bit_cast from a range of bytes.
If you're trying to copy bits of data into storage... just do that. bit_cast isn't here to replace every use of memcpy.
I was just going through all the possible Undefined Behaviours in this thread, and one of them is
The result of assigning to partially overlapping objects
I wondered if anyone could give me a definition of what "partially overlapping objects" are and an example in code of how that could possibly be created?
As pointed out in other answers, a union is the most obvious way to arrange this.
This is an even clearer example of how partially overlapping objects might arise with the built in assignment operator. This example would not otherwise exhibit UB if it were not for the partially overlapping object restrictions.
union Y {
int n;
short s;
};
void test() {
Y y;
y.s = 3; // s is the active member of the union
y.n = y.s; // Although it is valid to read .s and then write to .x
// changing the active member of the union, .n and .s are
// not of the same type and partially overlap
}
You can get potential partial overlap even with objects of the same type. Consider this example in the case where short is strictly larger than char on an implementation that adds no padding to X.
struct X {
char c;
short n;
};
union Y {
X x;
short s;
};
void test() {
Y y;
y.s = 3; // s is the active member of the union
y.x.n = y.s; // Although it is valid to read .s and then write to .x
// changing the active member of the union, it may be
// that .s and .x.n partially overlap, hence UB.
}
A union is a good example for that.
You can create a structure of memory with overlapping members.
for example (from MSDN):
union DATATYPE // Declare union type
{
char ch;
int i;
long l;
float f;
double d;
} var1;
now if you use assign the char member all other member are undefined. That's because they are at the same memory block, and you've only set an actual value to a part of it:
DATATYPE blah;
blah.ch = 4;
If you will then try to access blah.i or blah.d or blah.f they will have an undefined value. (because only first byte, which is a char, had its value set)
This refers to the problem of pointer aliasing, which is forbidden in C++ to give compilers an easier time optimizing. A good explanation of the problem can be found in this thread
May be he mean a strict aliasing rule? Object in memory should not overlap with object of other type.
"Strict aliasing is an assumption, made by the C (or C++) compiler, that dereferencing pointers to objects of different types will never refer to the same memory location (i.e. alias each other.)"
The canonical example is using memcpy:
char *s = malloc(100);
int i;
for(i=0; i != 100;++i) s[i] = i; /* just populate it with some data */
char *t = s + 10; /* s and t may overlap since s[10+i] = t[i] */
memcpy(t, s, 20); /* if you are copying at least 10 bytes, there is overlap and the behavior is undefined */
The reason why memcpy is undefined behavior is because there's no required algorithm for performing the copy. In this circumstance, memmove was introduced as a safe alternative.
Let's we have a simple structure (POD).
struct xyz
{
float x, y, z;
};
May I assume that following code is OK? May I assume there is no any gaps? What the standard says? Is it true for PODs? Is it true for classes?
xyz v;
float* p = &v.x;
p[0] = 1.0f;
p[1] = 2.0f; // Is it ok?
p[2] = 3.0f; // Is it ok?
The answer here is a bit tricky. The C++ standard says that POD data types will have C layout compatability guarantees (Reference). According to section 9.2 of the C spec the members of a struct will be laid out in sequential order if
There is no accessibility modifier difference
No alignment issues with the data type
So yes this solution will work as long as the type float has a compatible alignment on the current platform (it's the platform word size). So this should work for 32 bit processors but my guess is that it would fail for 64 bit ones. Essentially anywhere that sizeof(void*) is different than sizeof(float)
This is not guaranteed by the standard, and will not work on many systems. The reasons are:
The compiler may align struct members as appropriate for the target platform, which may mean 32-bit alignment, 64-bit alignment, or anything else.
The size of the float might be 32 bits, or 64 bits. There's no guarantee that it's the same as the struct member alignment.
This means that p[1] might be at the same location as xyz.y, or it might overlap partially, or not at all.
No, it is not OK to do so except for the first field.
From the C++ standards:
9.2 Class members
A pointer to a POD-struct object,
suitably converted using a
reinterpret_cast, points to its
initial member (or if that member is a
bit-field, then to the unit in which
it resides) and vice versa. [Note:
There might therefore be unnamed
padding within a POD-struct object,
but not at its beginning, as necessary
to achieve appropriate alignment.
Depends on the hardware. The standard explicitly allows POD classes to have unspecified and unpredictable padding. I noted this on the C++ Wikipedia page and grabbed the footnote with the spec reference for you.
^ a b ISO/IEC (2003). ISO/IEC 14882:2003(E): Programming Languages - C++ ยง9.2 Class members [class.mem] para. 17
In practical terms, however, on common hardware and compilers it will be fine.
When in doubt, change the data structure to suit the application:
struct xyz
{
float p[3];
};
For readability you may want to consider:
struct xyz
{
enum { x_index = 0, y_index, z_index, MAX_FLOATS};
float p[MAX_FLOATS];
float X(void) const {return p[x_index];}
float X(const float& new_x) {p[x_index] = new_x;}
float Y(void) const {return p[y_index];}
float Y(const float& new_y) {p[y_index] = new_y;}
float Z(void) const {return p[z_index];}
float Z(const float& new_z) {p[z_index] = new_z;}
};
Perhaps even add some more encapsulation:
struct Functor
{
virtual void operator()(const float& f) = 0;
};
struct xyz
{
void for_each(Functor& ftor)
{
ftor(p[0]);
ftor(p[1]);
ftor(p[2]);
return;
}
private:
float p[3];
}
In general, if a data structure needs to be treated in two or more different ways, perhaps the data structure needs to be redesigned; or the code.
The standard requires that the order of arrangement in memory match the order of definition, but allows arbitrary padding between them. If you have an access specifier (public:, private: or protected:) between members, even the guarantee about order is lost.
Edit: in the specific case of all three members being of the same primitive type (i.e. not themselves structs or anything like that) you stand a pretty fair chance -- for primitive types, the object's size and alignment requirements are often the same, so it works out.
OTOH, this is only by accident, and tends to be more of a weakness than a strength; the code is wrong, so ideally it would fail immediately instead of appearing to work, right up to the day that you're giving a demo for the owner of the company that's going to be your most important customer, at which time it will (of course) fail in the most heinous possible fashion...
No, you may not assume that there are no gaps. You may check for you architecture, and if there aren't and you don't care about portability, it will be OK.
But imagine a 64-bit architecture with 32-bit floats. The compiler may align the struct's floats on 64-bit boundaries, and your
p[1]
will give you junk, and
p[2]
will give you what you think your getting from
p[1]
&c.
However, you compiler may give you some way to pack the structure. It still wouldn't be "standard"---the standard provides no such thing, and different compilers provide very incompatible ways of doing this--- but it is likely to be more portable.
Lets take a look at Doom III source code:
class idVec4 {
public:
float x;
float y;
float z;
float w;
...
const float * ToFloatPtr( void ) const;
float * ToFloatPtr( void );
...
}
ID_INLINE const float *idVec4::ToFloatPtr( void ) const {
return &x;
}
ID_INLINE float *idVec4::ToFloatPtr( void ) {
return &x;
}
It works on many systems.
Your code is OK (so long as it only ever handles data generated in the same environment). The structure will be laid out in memory as declared if it is POD. However, in general, there is a gotcha you need to be aware of: the compiler will insert padding into the structure to ensure each member's alignment requirements are obeyed.
Had your example been
struct xyz
{
float x;
bool y;
float z;
};
then z would have began 8 bytes into the structure and sizeof(xyz) would have been 12 as floats are (usually) 4 byte aligned.
Similarly, in the case
struct xyz
{
float x;
bool y;
};
sizeof(xyz) == 8, to ensure ((xyz*)ptr)+1 returns a pointer that obeys x's alignment requirements.
Since alignment requirements / type sizes may vary between compilers / platforms, such code is not in general portable.
As others have pointed out the alignment is not guaranteed by the spec. Many say it is hardware dependent, but actually it is also compiler dependent. Hardware may support many different formats. I remember that the PPC compiler support pragmas for how to "pack" the data. You could pack it on 'native' boundaries or force it to 32 bit boundaries, etc.
It would be nice to understand what you are trying to do. If you are trying to 'parse' input data, you are better off with a real parser. If you are going to serialize, then write a real serializer. If you are trying to twiddle bits such as for a driver, then the device spec should give you a specific memory map to write to. Then you can write your POD structure, specify the correct alignment pragmas (if supported) and move on.
structure packing (eg #pragma pack in MSVC) http://msdn.microsoft.com/en-us/library/aa273913%28v=vs.60%29.aspx
variable alignment
(eg __declspec(align( in MSVC) http://msdn.microsoft.com/en-us/library/83ythb65.aspx
are two factors that can wreck your assumptions. floats are usually 4 bytes wide, so it's rare to misalign such large variables. But it's still easy to break your code.
This issue is most visible when binary reading header struct with shorts (like BMP or TGA) - forgetting pack 1 causes a disaster.
I assume you want a struct to keep your coordinates accessed as members (.x, .y and .z) but you still want them to be accessed, let's say, an OpenGL way (as if it was an array).
You can try implementing the [] operator of the struct so it can be accessed as an array. Something like:
struct xyz
{
float x, y, z;
float& operator[] (unsigned int i)
{
switch (i)
{
case 0:
return x;
break;
case 1:
return y;
break;
case 2:
return z;
break;
default:
throw std::exception
break;
}
}
};
I have a testing struct definition as follows:
struct test{
int a, b, c;
bool d, e;
int f;
long g, h;
};
And somewhere I use it this way:
test* t = new test; // create the testing struct
int* ptr = (int*) t;
ptr[2] = 15; // directly manipulate the third word
cout << t->c; // look if it really affected the third integer
This works correctly on my Windows - it prints 15 as expected, but is it safe? Can I be really sure the variable is on the spot in memory I want it to be - expecially in case of such combined structs (for example f is on my compiler the fifth word, but it is a sixth variable)?
If not, is there any other way to manipulate struct members directly without actually having struct->member construct in the code?
It looks like you are asking two questions
Is it safe to treat &test as a 3 length int arrray?
It's probably best to avoid this. This may be a defined action in the C++ standard but even if it is, it's unlikely that everyone you work with will understand what you are doing here. I believe this is not supported if you read the standard because of the potential to pad structs but I am not sure.
Is there a better way to access a member without it's name?
Yes. Try using the offsetof macro/operator. This will provide the memory offset of a particular member within a structure and will allow you to correctly position a point to that member.
size_t offset = offsetof(mystruct,c);
int* pointerToC = (int*)((char*)&someTest + offset);
Another way though would be to just take the address of c directly
int* pointerToC = &(someTest->c);
No you can't be sure. The compiler is free to introduce padding between structure members.
To add to JaredPar's answer, another option in C++ only (not in plain C) is to create a pointer-to-member object:
struct test
{
int a, b, c;
bool d, e;
int f;
long g, h;
};
int main(void)
{
test t1, t2;
int test::*p; // declare p as pointing to an int member of test
p = &test::c; // p now points to 'c', but it's not associating with an object
t1->*p = 3; // sets t1.c to 3
t2->*p = 4; // sets t2.c to 4
p = &test::f;
t1->*p = 5; // sets t1.f to 5
t2->*p = 6; // sets t2.f to 6
}
You are probably looking for the offsetof macro. This will get you the byte offset of the member. You can then manipulate the member at that offset. Note though, this macro is implementation specific. Include stddef.h to get it to work.
It's probably not safe and is 100% un-readable; thus making that kind of code unacceptable in real life production code.
use set methods and boost::bind for create functor which will change this variable.
Aside from padding/alignment issues other answers have brought up, your code violates strict aliasing rules, which means it may break for optimized builds (not sure how MSVC does this, but GCC -O3 will break on this type of behavior). Essentially, because test *t and int *ptr are of different types, the compiler may assume they point to different parts of memory, and it may reorder operations.
Consider this minor modification:
test* t = new test;
int* ptr = (int*) t;
t->c = 13;
ptr[2] = 15;
cout << t->c;
The output at the end could be either 13 or 15, depending on the order of operations the compiler uses.
According to paragraph 9.2.17 of the standard, it is in fact legal to cast a pointer-to-struct to a pointer to its first member, providing the struct is POD:
A pointer to a POD-struct object,
suitably converted using a
reinterpret_cast, points to its
initial member (or if that member is a
bit-field, then to the unit in which
it resides) and vice versa. [Note:
There might therefore be unnamed
padding within a POD-struct object,
but not at its beginning, as necessary
to achieve appropriate alignment. ]
However, the standard makes no guarantees about the layout of structs -- even POD structs -- other than that the address of a later member will be greater than the address of an earlier member, provided there are no access specifiers (private:, protected: or public:) between them. So, treating the initial part of your struct test as an array of 3 integers is technically undefined behaviour.