Is there anyway to BASICALLY do the following:
#include <boost/asio.hpp>
struct testStruct{
int x;
int y;
};
int main(){
struct testStruct t;
boost::asio::buffer b;
b = boost::asio::buffer(t);
return 0;
}
Where it seems to fail is passing 't' into the buffer, 'b'.
Use the scatter operation of more than a single buffer:
#include <boost/asio.hpp>
#include <vector>
struct testStruct{
int x;
int y;
};
int
main()
{
struct testStruct t;
t.x = 5;
t.y = 7;
std::vector<boost::asio::const_buffer> buffers;
buffers.push_back( boost::asio::buffer(&t.x, sizeof(t.x) ) );
buffers.push_back( boost::asio::buffer(&t.y, sizeof(t.y) ) );
boost::asio::io_service io_service;
boost::asio::ip::tcp::socket socket( io_service ); // note not connected!
std::size_t length = boost::asio::write( socket, buffers );
return 0;
}
Note you'll need to use a corresponding gather on the receiving side. This gets very tedious with anything more than the contrived example you have presented. Which is why I suggested using a more robust serialization mechanism in your previous question.
Just Use Boost.Serialization
You can get a demo from the http://www.boost.org/doc/libs/1_47_0/doc/html/boost_asio/examples.html
When you want to send an Object, It is better for you to Serialize it First.
There are a few things you need to be careful of.
1. Padding
The layout of your struct is implementation-specific. It's entirely possible for there to be placeholder bytes between the x and y members of your struct on the server, and none on the client.
To work around this, you should serialize your structures member by member into a character buffer, and deserialize them on the client in the same manner.
You could write some utility code to help you with this, here's a starting point:
class packet_writer
{
public:
template <typename iter> void write(iter begin, iter end)
{
buffer_.insert(buffer_.end(), begin, end);
}
template <typename T> void write(T data)
{
int8_t* begin = reinterpret_cast<int8_t*>(&data);
write(begin, begin + sizeof(data));
}
const std::vector<int8_t>& buffer() const
{
return buffer_;
}
private:
std::vector<int8_t> buffer_;
};
2. Endianness
Depending on architecture, or in some cases even depending on the current CPU mode (some POWER CPUs support endianness switching), the bytes of your members may be reversed. You have to detect the endianness of the host architecture, and swap the bytes to a predefined order for use in your protocol.
Related
I'm packing messages to be send to another device. I'm using C++11. Usually the payload is a single ASCII encoded value, but this one command has a bit more parameters. I'm trying to pack the payload in a single array of chars. I'm looking for a clean solution, and I thought that the following (example) solution should work:
Message foo(Parameters parameters) {
struct __attribute__((__packed__)) ParameterPayload {
std::array<char,2> a;
std::array<char,2> b;
std::array<char,2> c;
std::array<char,4> d;
}; // Actual struct has way more parameters
ParameterPayload paramPayload;
paramPayload.a = bar<2,10>(parameters.a);
paramPayload.b = bar<2,10>(parameters.b);
paramPayload.c = bar<2,10>(parameters.c);
paramPayload.d = bar<4,16>(parameters.d);
// This will not work, but I want something like this to work
auto payload = reinterpreted_cast<std::array<char, sizeof(ParameterPayload)>(paramPayload);
return baz<sizeof(ParameterPayload)>(payload);
}
template<size_t size, int base>
std::array<char, size> bar(int input> {
// ASCII encoding with a base (2, 10 or 16)
}
template<size_t payloadSize>
Message baz(std::array<char, payloadSize> payload) {
// Some packing computation
}
This is a rough example, but I think it will get the message across. How do I cast the struct to a single std:array<char, N>? Is it even possible? I'm trying not to do multiple std::copies instead, because that will cost more resources and will worsen readability.
another solution I looked into is using
const char* const payload = reinterpret_cast<const char* const>(¶mPayload);
and going from there... I could do a single copy after that, but I'd like to avoid it.
That casting is pedantically UB (and even we don't have guaranty of exact size of std::array).
I suggest to change to something like:
Message foo(Parameters parameters) {
std::array<char, 2+2+2+4> payload; // Actual struct has way more parameters
int offset = 0;
bar<2, 10>(parameters.a, payload.data() + offset, offset);
bar<2, 10>(parameters.b, payload.data() + offset, offset);
bar<2, 10>(parameters.c, payload.data() + offset, offset);
bar<4, 16>(parameters.d, payload.data() + offset, offset);
return baz(payload);
}
template<size_t size, int base>
void bar(int input, char* out, int& offset) /* std::span<char, size> out (in C++20) */
{
offset += size;
// ASCII encoding with a base (2, 10 or 16)
}
template<size_t payloadSize>
Message baz(std::array<char, payloadSize> payload) {
// Some packing computation
}
I need to process data that is given to me as a char buffer where the actual structure of the data depends on the values of some of its fields.
More specifically, consider the following header file:
struct IncomingMsgStruct
{
MsgHdrStruct msgHdr;
char msgData[MSG_DATA_MAX_SIZE]; // Can hold any of several structures
};
struct RelevantMessageData
{
DateTimeStruct dateTime;
CommonDataStruct commonData;
MsgBodyUnion msgBody;
};
struct DateTimeStruct { /* ... */ };
struct CommonDataStruct
{
char name[NAME_MAX_SIZE + 1];
MsgTypeEnum msgType;
// more elements here
};
union MsgBodyUnion
{
MsgBodyType1Struct msgBodyType1;
MsgBodyType2Struct msgBodyType2;
// ...
MsgBodyTypeNStruct msgBodyTypeN;
};
struct MsgBodyType1Struct { /* ... */ };
struct MsgBodyType2Struct { /* ... */ };
// ...
struct MsgBodyTypeNStruct { /* ... */ };
The structures contain data members (some of which are also structures) and member functions for initialization, conversion to string, etc. There are no constructors, destructors, virtual functions, or inheritance.
Please note that this is in the context of a legacy code that I have no control over. The header and the definitions in it are used by other components, and some of them can change with time.
The data is made available to me as a buffer of characters, so my processing function will look like:
ResultType processRelevantMessage(char const* inBuffer);
It is guaranteed that inBuffer contains a MsgStruct structure, and that its msgData member holds a RelevantMessageData structure. Correct alignment and endianness are also guaranteed as the data originated from the corresponding structures on the same platform.
For simplicity, let's assume that I am only interested in the case where msgType equals to a specific value, so only the members of, say MsgBodyType2Struct, will need to be accessed (and an error returned otherwise). I can generalize it to handle several types later.
My understanding is that a naive implementation using reinterpret_cast can run afoul of the C++ strict aliasing rules.
My question is:
How can I do it in standard-compliant C++ without invoking undefined behaviour, without changing or duplicating the definitions, and without extra copying or allocations?
Or, if that is not possible, how can I do it in GCC (possibly using flags such as -fno-strict-aliasing etc.)?
EDIT:
Since the data comes from the same platform, there should be no endianness concerns.
As mentioned above, I prefer to avoid copying.
Upon further reading, it seems to me that placement-new should be safe. So is the following implementation compliant?
ResultType processRelevantMessageType2(char const* in)
{
MsgStruct const* pMsgStruct = new (in) MsgStruct;
RelevantMessageData const* pRelevantMessageData = new (pMsgStruct->msgData) RelevantMessageData;
// Assume we're only interested in the MsgBodyType2Struct case
if (pRelevantMessageData->commonData.msgType == MSG_TYPE_2) {
MsgBodyType2Struct const& msgBodyType2Struct = pRelevantMessageData->msgBody.MsgBodyType2Struct;
// Can access the fields of msgBodyType2Struct here?
// ...
}
// ...
}
My understanding is that a naive implementation using reinterpret_cast can run afoul of the C++ strict aliasing rules.
Indeed. Also, consider that an array of bytes might start at an arbitrary address in memory, whereas a struct typically has some alignment restrictions that need to be satisfied. The safest way to deal with this is to create a new object of the desired type, and use std::memcpy() to copy the bytes from the buffer into the object:
ResultType processRelevantMessage(char const* inBuffer) {
MsgHdrStruct hdr;
std::memcpy(&hdr, inbuffer, sizeof hdr);
...
RelevantStruct data;
std::memcpy(&data, inbuffer + sizeof hdr, sizeof data);
...
}
The above is well-defined C++ code, you can use hdr and data afterwards without problems (as long as those are POD types that don't contain any pointers).
I suggest using a serialization library or write operator<< and operator>> overloads for those structs. You could use the functions htonl and ntohl which are available on some platforms or write a support class to stream numeric values yourself.
Such a class could look like this:
#include <bit>
#include <algorithm>
#include <cstring>
#include <iostream>
#include <iterator>
#include <limits>
#include <type_traits>
template<class T>
struct tfnet { // to/from net (or file)
static_assert(std::endian::native == std::endian::little ||
std::endian::native == std::endian::big); // endianess must be known
static_assert(std::numeric_limits<double>::is_iec559); // only support IEEE754
static_assert(std::is_arithmetic_v<T>); // only for arithmetic types
tfnet(T& v) : val(&v) {} // store a pointer to the value to be streamed
// write a value to a stream
friend std::ostream& operator<<(std::ostream& os, const tfnet& n) {
if constexpr(std::endian::native == std::endian::little) {
// reverse byte order to be in network byte order
char buf[sizeof(T)];
std::memcpy(buf, n.val, sizeof buf);
std::reverse(std::begin(buf), std::end(buf));
os.write(buf, sizeof buf);
} else {
// already in network byte order
os.write(n.val, sizeof(T));
}
return os;
}
// read a value from a stream
friend std::istream& operator>>(std::istream& is, const tfnet& n) {
char buf[sizeof(T)];
if(is.read(buf, sizeof buf)) {
if constexpr(std::endian::native == std::endian::little) {
// reverse byte order to be in network byte order
std::reverse(std::begin(buf), std::end(buf));
}
std::memcpy(n.val, buf, sizeof buf);
}
return is;
}
T* val;
};
Now, if you have a set of structs:
#include <cstdint>
struct data {
std::uint16_t x = 10;
std::uint32_t y = 20;
std::uint64_t z = 30;
};
struct compound {
data x;
int y = 40;
};
You can add the streaming operators for them:
std::ostream& operator<<(std::ostream& os, const data& d) {
return os << tfnet{d.x} << tfnet{d.y} << tfnet{d.z};
}
std::istream& operator>>(std::istream& is, data& d) {
return is >> tfnet{d.x} >> tfnet{d.y} >> tfnet{d.z};
}
std::ostream& operator<<(std::ostream& os, const compound& d) {
return os << d.x << tfnet{d.y}; // using data's operator<< for d.x
}
std::istream& operator>>(std::istream& is, compound& d) {
return is >> d.x >> tfnet{d.y}; // using data's operator>> for d.x
}
And reading/writing the structs:
#include <sstream>
int main() {
std::stringstream ss;
compound x;
compound y{{0,0,0},0};
ss << x; // write to stream
ss >> y; // read from stream
}
Demo
If you can't use the streaming operators directly on the source streams, you can put the char buffer you do get in an istringstream and extract the data from that using the added operators.
I'm using union to fill some message fields in a char type message buffer. If the length of the message is constant, it works correctly. See the simplified code sample below.
The problem is, my message can have variable length. Specifically, the const N will be decided on runtime. Is there a way to keep using unions by dynamically allocating memory for buf?
I'm exploring smart pointers but haven't had any luck so far.
const int N = 4;
struct repeating_group_t {
uint8_t field1;
uint8_t field2;
}rpt_group;
struct message_t
{
union
{
char buf[2 + 2*N];
struct {
uint8_t header;
uint8_t block_len;
std::array<repeating_group_t, N> group;
};
};
};
int main()
{
message_t msg;
msg.header = 0x32;
msg.block_len = 8;
for (auto i = 0; i < N; i++)
{
msg.group[i].field1 = i;
msg.group[i].field2 = 10*i;
}
// msg.buf is correctly filled
return 0;
}
As said in the comments, use std::vector.
int main() {
// before C++17 use char
std::vector<std::byte> v.
v.push_back(0x32);
v.push_back(8);
for (auto i = 0; i < N; i++) {
v.push_back(i);
const uint16_t a = 10 * i;
// store uint16_t in big endian
v.push_back(a >> 16);
v.push_back(a & 0xff);
}
}
For custom datatypes, you could provide your own stream-like or container-like container and overload operator>> or another custom function of your choice for your datatypes.
struct Message{
std::vector<std::byte> v;
Message& push8(uint8_t t) { ... }
// push 16 bits little endian
Message& push16le(uint16_t t) { ... }
// push 16 bits big endian
Message& push16be(uint16_t t) { ... }
// etc
Message& push(const Repeating_group& t) {
v.push_back(t.field1);
v.push_back(t.field2);
return v;
}
// etc.
};
int main(){
Message v;
v.push8(0x32).push8(8);
for (...) {
v.push(Repeating_group(i, i * 10));
}
}
You can't have N evaluated at runtime because both c-array (your buf) and std::array have size information in its type.
Also - using union for (de)serialization is not a good practice - size of your structure will depend on alignment needed on given machine it is compiled for and so on... You could add packed attribute to overcome it, but you still have plenty of platform dependency problems here.
Regarding variable length - you'd need to write custom (de)serializer that will understand and store/read that size information to recreate that container on the other end.
Where do you want to pass these messages?
I want to use a POD struct as a hash key in a map, e.g.
struct A { int x; int y; };
std::unordered_map<A, int> my_map;
but I can't do this, since no hash function is auto-generatable for such structs.
Why does the C++ standard not require a default hash for a POD struct?
Why do compilers (specifically, GCC 4.x / 5.x) offer such a hash, even if the standard doesn't mandate one?
How can I generate a hash function, using a template, in a portable way, for all of my POD structures (I'm willing to make semantic assumptions if necessary)?
As from the documentation, a possible implementation in your case would be:
#include<functional>
#include<unordered_map>
struct A { int x; int y; };
namespace std
{
template<> struct hash<A>
{
using argument_type = A;
using result_type = std::size_t;
result_type operator()(argument_type const& a) const
{
result_type const h1 ( std::hash<int>()(a.x) );
result_type const h2 ( std::hash<int>()(a.y) );
return h1 ^ (h2 << 1);
}
};
}
int main() {
std::unordered_map<A, int> my_map;
}
The compiler us not allowed to generate such a specialization because of the standard that does not define anything like that (as already mentioned in the comments).
There is a method to generate hash for POD, like good old c style. Only for real POD with no any linked data on the outside of struct. There is no checking of this requirements in code so use it only when you know and can guarantee this. All fields must be initialized (for example by default constructor like this A(), B() etc).
#pragma pack(push) /* push current alignment to stack */
#pragma pack(1) /* set alignment to 1 byte boundary */
struct A { int x; int y; };
struct B { int x; char ch[8] };
#pragma pack(pop) /* restore original alignment from stack */
struct C { int x __attribute__((packed)); };
template<class T> class PodHash;
template<>
class PodHash<A> {
public:
size_t operator()(const A &a) const
{
// it is possible to write hash func here char by char without using std::string
const std::string str =
std::string( reinterpret_cast<const std::string::value_type*>( &a ), sizeof(A) );
return std::hash<std::string>()( str );
}
};
std::unordered_map< A, int, PodHash<A> > m_mapMyMapA;
std::unordered_map< B, int, PodHash<B> > m_mapMyMapB;
UPD:
Data structure must be defined in data packing section with value of one byte or with pack attribute for prevent padding bytes.
UPD:
But I need to warn that replace deafult packing will make data loading/storing from/to memory for some fields little slowly, to prevent this need to arrange structure data fields with granularity that corresponding your (or most popular) architecture.
I suggest that you can add by yourself additional unused fields not for using but for arrange fields in your data structure for best prformance of memory loading/storing. Example:
struct A
{
char x; // 1 byte
char padding1[3]; // 3 byte for the following 'int'
int y; // 4 bytes - largest structure member
short z; // 2 byte
char padding2[2]; // 2 bytes to make total size of the structure 12 bytes
};
#pragma pack is supported by, at least:
Microsoft compiler
GNU compiler (webarchive)
clang-llvm compiler (webarchive)
Embarcadero (Borland) compiler (webarchive)
Sun WorkShop Compiler (webarchive)
Intel compiler is compatible with GCC, CLANG and Microsoft compiler
More flexible way is to declare comparision class and use it as template param of std::unordered_map.
struct A { int x; int y; };
emplate<class T> class MyHash;
template<>
class MyHash<A> {
public:
size_t operator()(const A &a) const
{
result_type const h1 ( std::hash<int>()(a.x) );
result_type const h2 ( std::hash<int>()(a.y) );
return h1 ^ (h2 << 1);
}
};
std::unordered_map<CString,CString,MyHash> m_mapMyMap;
You may want another Hash for same objects. Flexibility appear with code like this:
std::unordered_map<CString,CString, *MyAnotherHas* > m_mapMyMap;
I want a simple variant with
minimal overhead
that can be passed to functions written in the C language
so I decided to use a std::vector like this
typedef std::vector<char> SimpleVariant;
SimpleVariant data;
(1) store a std::string
for(std::string::iterator it = str.begin(); it != str.end(); ++it)
{
data.push_back( *it );
}
data.push_back('\0');
(2) store a double
data.resize(64);
std::sprintf(&data[0], "%.*g", 20, val);
(3) get a string
std::string str = std::string( m_data.begin(), m_data.end() );
(4) get a double
double dbl = boost::lexical_cast<double>(&data[0]);
Is this a reasonable approach given my requirements? Is there lightweight variant that I can use instead of trying to reinvent the wheel?
I am aware of boost::variant and boost::any they are too heavyweight for my needs
that can be passed to functions written in the C language
No respectable variant in C++ is C-compatible, beyond passing a pointer to it as void* and providing functions that might manipulate it. C variants are guaranteed to be non-type-safe, whereas C++ variants have their types guaranteed. All you can do is store a type enumeration and a block of memory corresponding to the maximum size.
Also, boost::any isn't more heavyweight than a vector. You're just re-implementing it. In a less efficient way. And less safely.
Serializing to the textual representation of the data into dynamically allocated memory can hardly be considered minimal overhead. And then again, regarding the C functions, what exactly do you want to pass the variant to the C functions for, do you want just to pass the variant through? do you intend on using the data inside the C functions?
If you only want to pass the variant through C code (but not use it there), then consider using boost::variant or boost::any and passing void*, if you intend on using the data inside C, I would recommend to use a structure with a tag to identify the type and an array big enough to hold the data you want to store and then cast the values into that array.
You haven't really given much information about how the C interface to this works. So this suggestion may not be helpful. But let's assume you have a C interface that looks like this:
int GetAValue(void *data, int size); //Returns the type of data stored in the data pointer. Will not write past size.
void DoSomethingWithValue(int type, void *data, int size); //Gives data to C, to do something with. Does not modify it.
void ModifyValue(int type, void *data, int size); //Gives data to C, where it modifies it.
I would use a boost::variant as follows:
typedef boost::variant<int, double, std::string> MyVariant;
struct DoSomethingWithValueVisit : public boost::static_visitor<>
{
void operator()(int val)
{
DoSomethingWithValue(TYPE_INTEGER, &val, sizeof(int));
}
void operator()(double val)
{
DoSomethingWithValue(TYPE_DOUBLE, &val, sizeof(double));
}
void operator()(const std::string &val)
{
DoSomethingWithValue(TYPE_STRING, (void*)val.c_str(), val.size() + 1);
}
};
struct ModifyValueVisit : public boost::static_visitor<>
{
void operator()(int &val)
{
ModifyValue(TYPE_INTEGER, &val, sizeof(int));
}
void operator()(double &val)
{
ModifyValue(TYPE_DOUBLE, &val, sizeof(double));
}
void operator()(std::string &val)
{
char buffer[128];
strncpy(buffer, val.c_str(), 127);
ModifyValue(TYPE_STRING, buffer, 128);
val = buffer;
}
};
MyVariant GetAValueFromC()
{
char buffer[128];
int type = GetAValue(buffer, 128);
switch(type)
{
case TYPE_INTEGER:
return *reinterpret_cast<int*>(&buffer[0]);
case TYPE_DOUBLE:
return *reinterpret_cast<double*>(&buffer[0]);
case TYPE_STRING:
return std::string(buffer);
}
}
int main()
{
MyVariant value = GetAValueFromC();
//Non-modifying
boost::apply_visitor(DoSomethingWithValueVisit(), value);
//Modifying
boost::apply_visitor(ModifyValueVisit(), value);
}
Feel free to add more types to the variant as needed.