I'd like to process data provided by an external library.
The lib holds the data and provides access to it like this:
const uint8_t* data;
std::pair<const uint8_t*, const uint8_t*> getvalue() const {
return std::make_pair(data + offset, data + length);
}
I know that the current data contains two uint16_t numbers, but I need to change their endianness.
So altogether the data is 4 bytes long and contains this numbers:
66 4 0 0
So I'd like to get two uint16_t numbers with 1090 and 0 value respectively.
I can do basic arithmetic and in one place change the endianness:
pair<const uint8_t*, const uint8_t*> dataPtrs = library.value();
vector<uint8_t> data(dataPtrs.first, dataPtrs.second);
uint16_t first = data[1] <<8 + data[0]
uint16_t second = data[3]<<8 + data[2]
However I'd like to do something more elegant (the vector is replaceable if there is better way for getting the uint16_ts).
How can I better create uint16_t from uint8_t*? I'd avoid memcpy if possible, and use something more modern/safe.
Boost has some nice header-only endian library which can work, but it needs an uint16_t input.
For going further, Boost also provides data types for changing endianness, so I could create a struct:
struct datatype {
big_int16_buf_t data1;
big_int16_buf_t data2;
}
Is it possible to safely (paddings, platform-dependency, etc) cast a valid, 4 bytes long uint8_t* to datatype? Maybe with something like this union?
typedef union {
uint8_t u8[4];
datatype correct_data;
} mydata;
Maybe with something like this union?
No. Type punning with unions is not well defined in C++.
This would work assuming big_int16_buf_t and therefore datatype is trivially copiable:
datatype d{};
std::memcpy(&d, data, sizeof d);
uint16_t first = data[1] <<8 + data[0]
uint16_t second = data[3]<<8 + data[2]
However I'd like to do something more elegant
This is actually (subjectively, in my opinion) quite an elegant way because it works the same way on all systems. This reads the data as little endian, whether the CPU is little, big or some other endian. This is well portable.
However I'd like to do something more elegant (the vector is replaceable if there is better way for getting the uint16_ts).
The vector seems entirely pointless. You could just as well use:
const std::uint8_t* data = dataPtrs.first;
How can I better create uint16_t from uint8_t*?
If you are certain that the data sitting behind the uint8_t pointer is truly a uint16_t, C++ allows: auto u16 = *static_cast<uint16_t const*>(data); Otherwise, this is UB.
Given a big endian value, transforming this into little endian can be done with the ntohs function (under linux, other OSes have similar functions).
But beware, if the pointer you hold points to two individual uint8_t values, you mustn't convert them by pointer-cast. In that case, you have to manually specify which value goes where (conceivably with a function template). This will be the most portable solution, and in all likelihood the compiler will create efficient code out of the shifts and ors.
Related
There is a valid uint8_t* buffer with four uint8_t bytes.
I know the buffer contains two uint16_t numbers in big-endian format, and I want to extract them.
I can create the required data manually, while taking care of the correct endianness:
const std::uint8_t* data = ...
uint16_t first = (data[1]<<8) + data[0];
uint16_t second = (data[3]<<8) + data[2];
I have been told (thanks to #eerorika):
this works the same way on all systems. This reads the data as little
endian, whether the CPU is little, big or some other endian. This is
well portable.
And this works as intended.
Now let's consider another solution with Boost's endian library:
struct datatype {
boost::endian::little_int16_buf_t first;
boost::endian::little_int16_buf_t second;
} d;
memcpy(&d, data, sizeof d);
This solution also works, and my question is: is this in any way worse in terms of portability, platform- and CPU-dependency than the first?
If I compile and run this on a non little-endian architecture, will it produce the same values as on a little-endian one?
I have a
typedef struct {
uint32_t Thread: HTHREAD_BUS_WIDTH;
uint32_t Member: 3;
uint32_t Proxy:3;
// Other members, fill out 32 bits
} MyStruct;
that I must transfer from one system to another as an item of
a buffer comprising 32-bit words.
What is the best way to serialize the struct, and on the other side,
to deserialize it? "best" means here safe casting, and no unneeded copying.
For one direction of casting, I have found (as member function)
int &ToInt() {
return *reinterpret_cast<int *>(this);}
Is there similar valid casting in the other way round, i.e. from integer to MyStruct; the best would be as a member function?
How can I define which bit means which field? (It may even the case,
that the deserialization happens in another program, in another language, in little/big endian systems?
How can I define which bit means which field?
You cannot. You have no control over the layout of bitfields.
"best" means here safe casting, and no unneeded copying.
There is no portable safe cast that could avoid copying.
A portable way to serialise bitfields is to manually shift into an integer, in the desired order. For example:
MyStruct value = something;
uint32_t out = 0;
out |= value.Thread;
out << HTHREAD_BUS_WIDTH;
out |= value.Member;
out << 3;
out |= value.Proxy;
In the shown example, the least significant bits contain the field Proxy while the other fields are adjacent in more significant bits.
Of course, in order to serialise this generated integer correctly, just like serialising any integer, you must take endianness into consideration. Serialisation of an integer can be portably implemented by repeatedly shifting the integer, and copying the bytes in order of significance into an array.
If you need to read from other system which might have different endianess you cannot rely on a portable bitfield. A solution is to "expanse" your structure so that each field is serialyzed as a 32 bit value in the "transport" buffer. A safe implementation could be something like:
typedef struct {
uint32_t Thread: HTHREAD_BUS_WIDTH;
uint32_t Member: 3;
uint32_t Proxy:3;
// Other members, fill out 32 bits
std::vector<uint32_t > to_buffer() const;
} MyStruct;
Implementation of to_buffer():
std::vector<uint32_t > MyStruct::to_buffer() const
{
std::vector<uint32_t> buffer;
buffer.push_back((uint32_t )(Thread);
buffer.push_back((uint32_t )(Member);
buffer.push_back((uint32_t )(Proxy);
// push other members
return buffer;
}
then on the reception side you can do the "buffer" to struct.
If you do not want to expanse the fields that do not use 32 bits you can always implement you own packing function by shifting and masking bits eg:
uint32_t menber_and_procy = (Member << 3) | proxy; // and so one for other members.
It is much more error prone.
From my own experience, if communication bandwith is not an issue, relying on "text like" content is a better choice (no endianess issues and very easy to debug).
I'd like to switch the "endianness" of float and double values, it works OK, by doing something like:
float const v{1.f};
swap(reinterpret_cast<::std::uint32_t const&>(v));
Does there exist a better way to do the swap, without a cast?
EDIT: swap() is a C++ wrapper for gcc's built-in functions, I did not include it here.
uint16_t __builtin_bswap16 (uint16_t x)
uint32_t __builtin_bswap32 (uint32_t x)
uint64_t __builtin_bswap64 (uint64_t x)
Swapping of endianess is needed for a some data formats, like CBOR.
While it is good practice to try and avoid casts, it's uses like this that are the reason casts exist. An endian swap is a raw data operation so in order to do it you must strip away the typing information, I would say that if it is not endian correct to begin with, then it is not a float and should never have been typed that way while in the wrong endianess state.
Simply swapping the byte order is sufficient. memcpy the float into a char val[4], create a dummy char rvrse[4], then set;
rvrse[3] = val[0];
rvrse[2] = val[1];
...
Then memcpy rvrse back into the original float. The best way is to write a function similar to ntohl but use templates to make it work for all types.
When am dealing with the unsigned data types, i have a requirement to have a generic container that would accommodate uint8,uint16,uint32 and uint64 types.
Having a void* and casting the above data types helped me, still replacing them instead with vector<uint8_t> made the code look more clean.
void* test = (uint32_t) 100;
vs
pushing the same thing bytewise onto a vector test;
which would be a better and cleaner solution.
PS: i cant use boost, due to restrictions in module.
You can use a union to store and access the data. This will allow all members of the union to occupy the same memory and you can access the appropriate data type without the need for casting.
union UintData
{
uint8_t v8;
uint16_t v16;
uint32_t v32;
uint64_t v64;
};
std::vector<UintData> data;
UintData test;
test.v32 = 0xffffffff;
data.push_back(data);
When accessing the data make sure you only access the currently active member (the last member set) otherwise the behavior is undefined.
union could be solution, but I wonder what is the size of the vector of union after pushing uint8_t, uint16_t, uint32_t and uint64_t...?
I'm fun of simple solution so I would try
std::vector<uint64_t>
this case you can insert all values.
Would the following be the most efficient way to get an int16 (short) value from a byte array?
inline __int16* ReadINT16(unsigned char* ByteArray,__int32 Offset){
return (__int16*)&ByteArray[Offset];
};
If the byte array contains a dump of the bytes in the same endian format as the machine, this code is being called on. Alternatives are welcome.
It depends on what you mean by "efficient", but note that in some architectures this method will fail if Offset is odd, since the resulting 16 bit int will be misaligned and you will get an exception when you subsequently try to access it. You should only use this method if you can guarantee that Offset is even, e.g.
inline int16_t ReadINT16(uint8_t *ByteArray, int32_t Offset){
assert((Offset & 1) == 0); // Offset must be multiple of 2
return *(int16_t*)&ByteArray[Offset];
};
Note also that I've changed this slightly so that it returns a 16 bit value directly, since returning a pointer and then subsequently de-referencing it will most likely less "efficient" than just returning a 16 bit value directly. I've also switched to standard Posix types for integers - I recommend you do the same.
I'm surprised no one has suggested this yet for a solution that is both alignment safe and correct across all architectures. (well, any architecture where there are 8 bits to a byte).
inline int16_t ReadINT16(uint8_t *ByteArray, int32_t Offset)
{
int16_t result;
memcpy(&result, ByteArray+Offset, sizeof(int16_t));
return result;
};
And I suppose the overhead of memcpy could be avoided:
inline int16_t ReadINT16(uint8_t *ByteArray, int32_t Offset)
{
int16_t result;
uint8_t* ptr1=(uint8_t*)&result;
uint8_t* ptr2 = ptr1+1;
*ptr1 = *ByteArray;
*ptr2 = *(ByteArray+1);
return result;
};
I believe alignment issues don't generate exceptions on x86. And if I recall, Windows (when it ran on Dec Alpha and others) would trap the alignment exception and fix it up (at a modest perf hit). And I do remember learning the hard way that Sparc on SunOS just flat out crashes when you have an alignment issue.
inline __int16* ReadINT16(unsigned char* ByteArray,__int32 Offset)
{
return (__int16*)&ByteArray[Offset];
};
Unfortunately this has undefined behavour in C++, because you are accessing storage using two different types which is not allowed under the strict aliasing rules. You can access the storage of a type using a char*, but not the other way around.
From previous questions I asked, the only safe way really is to use memcpy to copy the bytes into an int and then use that. (Which will likely be optimised to the same code you'd hope anyway, so just looks horribly inefficient).
Your code will probably work, and most people seem to do this... But the point is that you can't go crying to your compiler vendor when one day it generates code that doesn't do what you'd hope.
I see no problem with this, that's exactly what I'd do. As long as the byte array is safe to access and you make sure that the offset is correct (shorts are 2 bytes so you may want to make sure that they can't do odd offsets or something like that)