A Better Way To Build a Packet - Byte by Byte? - c++

This is related to my question asked here today on SO. Is there a better way to build a packet to send over serial rather than doing this:
unsigned char buff[255];
buff[0] = 0x02
buff[1] = 0x01
buff[2] = 0x03
WriteFile(.., buff,3, &dwBytesWrite,..);
Note: I have about twenty commands to send, so if there was a better way to send these bytes to the serial device in a more concise manner rather than having to specify each byte, it would be great. Each byte is hexadecimal, with the last byte being the checksum. I should clarify that I know I will have to specify each byte to build the commands, but is there a better way than having to specify each array position?

You can initialize static buffers like so:
const unsigned char command[] = {0x13, 0x37, 0xf0, 0x0d};
You could even use these to initialize non-const buffers and then replace only changing bytes by index.

Not sure what you're asking. If you ask about the problem of setting the byte one by one and messing up the data, usually this is doen with a packed struct with members having meaningful names. Like:
#pragma push(pack)
#pragma pack(1)
struct FooHeader {
uint someField;
byte someFlag;
dword someStatus;
};
#pragma pack(pop)
FooHeader hdr;
hdr.someField = 2;
hdr.someFlag = 3;
hdr.someStatus = 4;
WriteFile(..., sizeof(hdr), &hdr);

Is there a better way to build a packet than assembling it byte by byte?
Yes, but it will require some thought and some careful engineering. Many of the other answers tell you other mechanisms by which you can put together a sequence of bytes in C++. But I suggest you design an abstraction that represents a part of a packet:
class PacketField {
void add_to_packet(Packet p);
};
Then you can define various subclasses:
Add a single byte to the packet
Add a 16-bit integer in big-endian order. Another for little-endian. Other widths besides 16.
Add a string to the packet; code the string by inserting the length and then the bytes.
You also can define a higher-order version:
PacketField sequence(PacketField first, PacketField second);
Returns a field that consists of the two arguments in sequence. If you like operator overloading you could overload this as + or <<.
Your underlying Packet abstraction will just be an extensible sequence of bytes (dynamic array) with some kind of write method.
If you wind up programming a lot of network protocols, you'll find this sort of design pays off big time.
Edit: The point of the PacketField class is composability and reuse:
By composing packet fields you can create more complex packet fields. For example, you could define "add a TCP header" as a function from PacketFields to PacketFields.
With luck you build up a library of PacketFields that are specific to your application or protocol family or whatever. Then you reuse the fields in the library.
You can create subclasses of PacketField that take extra parameters.
It's quite possibly that you can do something equally nice without having to have this extra level of indirection; I'm recommending it because I've seen it used effectively in other applications. You are decoupling the knowledge of how to build a packet (which can be applied to any packet, any time) from the act of actually building a particular packet. Separating concerns like this can help reuse.

Yes, there is a better method. Have your classes read from and write to a packed buffer. You could even implement this as an interface. Templates would help to.
An example of writing:
template <typename Member_Type>
void Store_Value_In_Buffer(const Member_Type&, member,
unsigned char *& p_buffer)
{
*((Member_Type *)(p_buffer)) = member;
p_buffer += sizeof(Member_Type);
return;
}
struct My_Class
{
unsigned int datum;
void store_to_buffer(unsigned char *& p_buffer)
{
Store_Value_In_Buffer(datum, buffer);
return;
}
};
//...
unsigned char buffer[256];
unsigned char * p_buffer(buffer);
MyClass object;
object.datum = 5;
object.store_to_buffer(p_buffer);
std::cout.write(p_buffer, 256);
Part of the interface is also to query the objects for the size that they would occupy in the buffer, say a method size_in_buffer. This is left as an exercise for the reader. :-)

There is a much better way, which is using structs to set the structures. This is usually how network packets are built on a low level.
For example, say you have packets which have an id, length, flag byte, and data, you'd do something like this:
struct packet_header {
int id;
byte length;
byte flags;
};
byte my_packet[] = new byte[100];
packet_header *header = &my_packet;
header->id = 20;
header->length = 10; // This can be set automatically by a function, maybe?
// etc.
header++; // Header now points to the data section.
Do note that you're going to have to make sure that the structures are "packed", i.e. when you write byte length, it really takes up a byte. Usually, you'd achieve this using something like #pragma pack or similar (you'll have to read about your compiler's pragma settings).
Also, note that you should probably use functions to do common operations. For example, create a function which gets as input the size, data to send, and other information, and fills out the packet header and data for you. This way, you can perform calculations about the actual size you want to write in the length field, you can calculate the CRC inside the function, etc.
Edit: This is a C-centric way of doing things, which is the style of a lot of networking code. A more C++-centric (object oriented) approach could also work, but I'm less familiar with them.

const char *c = "\x02\x02\x03";

Related

C++ casting a struct to std::vector<char> memory alignment

I'm trying to cast a struct into a char vector.
I wanna send my struct casted in std::vector throw a UDP socket and cast it back on the other side. Here is my struct whith the PACK attribute.
#define PACK( __Declaration__ ) __pragma( pack(push, 1) ) __Declaration__ __pragma( pack(pop) )
PACK(struct Inputs
{
uint8_t structureHeader;
int16_t x;
int16_t y;
Key inputs[8];
});
Here is test code:
auto const ptr = reinterpret_cast<char*>(&in);
std::vector<char> buffer(ptr, ptr + sizeof in);
//send and receive via udp
Inputs* my_struct = reinterpret_cast<Inputs*>(&buffer[0]);
The issue is:
All works fine except my uint8_t or int8_t.
I don't know why but whenever and wherever I put a 1Bytes value in the struct,
when I cast it back the value is not readable (but the others are)
I tried to put only 16bits values and it works just fine even with the
maximum values so all bits are ok.
I think this is something with the alignment of the bytes in the memory but i can't figure out how to make it work.
Thank you.
I'm trying to cast a struct into a char vector.
You cannot cast an arbitrary object to a vector. You can cast your object to an array of char and then copy that array into a vector (which is actually what your code is doing).
auto const ptr = reinterpret_cast<char*>(&in);
std::vector<char> buffer(ptr, ptr + sizeof in);
That second line defines a new vector and initializes it by copying the bytes that represent your object into it. This is reasonable, but it's distinct from what you said you were trying to do.
I think this is something with the alignment of the bytes in the memory
This is good intuition. If you hadn't told the compiler to pack the struct, it would have inserted padding bytes to ensure each field starts at its natural alignment. The fact that the operation isn't reversible suggests that somehow the receiving end isn't packed exactly the same way. Are you sure the receiving program has exactly the same packing directive and struct layout?
On x86, you can get by with unaligned data, but you may pay a large performance cost whenever you access an unaligned member variable. With the packing set to one, and the first field being odd-sized, you've guaranteed that the next fields will be unaligned. I'd urge you to reconsider this. Design the struct so that all the fields fall at their natural alignment boundaries and that you don't need to adjust the packing. This may make your struct a little bigger, but it will avoid all the alignment and performance problems.
If you want to omit the padding bytes in your wire format, you'll have to copy the relevant fields byte by byte into the wire format and then copy them back out on the receiving end.
An aside regarding:
#define PACK( __Declaration__ ) __pragma( pack(push, 1) ) __Declaration__ __pragma( pack(pop) )
Identifiers that begin with underscore and a capital letter or with two underscores are reserved for "the implementation," so you probably shouldn't use __Declaration__ as the macro's parameter name. ("The implementation" refers to the compiler, the standard library, and any other runtime bits the compiler requires.)
1
vector class has dynamically allocated memory and uses pointers inside. So you can't send the vector (but you can send the underlying array)
2
SFML has a great class for doing this called sf::packet. It's free, open source, and cross-platform.
I was recently working on a personal cross platform socket library for use in other personal projects and I eventually quit it for SFML. There's just TOO much to test, I was spending all my time testing to make sure stuff worked and not getting any work done on the actual projects I wanted to do.
3
memcpy is your best friend. It is designed to be portable, and you can use that to your advantage.
You can use it to debug. memcpy the thing you want to see into a char array and check that it matches what you expect.
4
To save yourself from having to do tons of robustness testing, limit yourself to only chars, 32-bit integers, and 64-bit doubles. You're using different compilers? struct packing is compiler and architecture dependent. If you have to use a packed struct, you need to guarantee that the packing is working as expected on all platforms you will be using, and that all platforms have the same endianness. Obviously, that's what you're having trouble with and I'm sorry I can't help you more with that. I would I would recommend regular serializing and would definitely avoid struct packing if I was trying to make portable sockets.
If you can make those guarantees that I mentioned, sending is really easy on LINUX.
// POSIX
void send(int fd, Inputs& input)
{
int error = sendto(fd, &input, sizeof(input), ..., ..., ...);
...
}
winsock2 uses a char* instead of a void* :(
void send(int fd, Inputs& input)
{
char buf[sizeof(input)];
memcpy(buf, &input, sizeof(input));
int error = sendto(fd, buf, sizeof(input), ..., ..., ...);
...
}
Did you tried the most simple approach of:
unsigned char *pBuff = (unsigned char*)&in;
for (unsigned int i = 0; i < sizeof(Inputs); i++) {
vecBuffer.push_back(*pBuff);
pBuff++;
}
This would work for both, pack and non pack, since you will iterate the sizeof.

How to read and write data in 8 bit integers unit form by c++ file functions

Is it possible to store data in integer form from 0 to 255 rather than 8-bit characters.Although both are same thing, how can we do it, for example, with write() function?
Is it ok to directly cast any integer to char and vice versa? Does something like
{
int a[1]=213;
write((char*)a,1);
}
and
{
int a[1];
read((char*)a,1);
cout<<a;
}
work to get 213 from the same location in the file? It may work on that computer but is it portable, in other words, is it suitable for cross-platform projects in that way? If I create a file format for each game level(which will store objects' coordinates in the current level's file) using this principle, will it work on other computers/systems/platforms in order to have loaded same level?
The code you show would write the first (lowest-address) byte of a[0]'s object representation - which may or may not be the byte with the value 213. The particular object representation of an int is imeplementation defined.
The portable way of writing one byte with the value of 213 would be
unsigned char c = a[0];
write(&c, 1);
You have the right idea, but it could use a bit of refinement.
{
int intToWrite = 213;
unsigned char byteToWrite = 0;
if ( intToWrite > 255 || intToWrite < 0 )
{
doError();
return();
}
// since your range is 0-255, you really want the low order byte of the int.
// Just reading the 1st byte may or may not work for your architecture. I
// prefer to let the compiler handle the conversion via casting.
byteToWrite = (unsigned char) intToWrite;
write( &byteToWrite, sizeof(byteToWrite) );
// you can hard code the size, but I try to be in the habit of using sizeof
// since it is better when dealing with multibyte types
}
{
int a = 0;
unsigned char toRead = 0;
// just like the write, the byte ordering of the int will depend on your
// architecture. You could write code to explicitly handle this, but it's
// easier to let the compiler figure it out via implicit conversions
read( &toRead, sizeof(toRead) );
a = toRead;
cout<<a;
}
If you need to minimize space or otherwise can't afford the extra char sitting around, then it's definitely possible to read/write a particular location in your integer. However, it can need linking in new headers (e.g. using htons/ntons) or annoying (using platform #defines).
It will work, with some caveats:
Use reinterpret_cast<char*>(x) instead of (char*)x to be explicit that you’re performing a cast that’s ordinarily unsafe.
sizeof(int) varies between platforms, so you may wish to use a fixed-size integer type from <cstdint> such as int32_t.
Endianness can also differ between platforms, so you should be aware of the platform byte order and swap byte orders to a consistent format when writing the file. You can detect endianness at runtime and swap bytes manually, or use htonl and ntohl to convert between host and network (big-endian) byte order.
Also, as a practical matter, I recommend you prefer text-based formats—they’re less compact, but far easier to debug when things go wrong, since you can examine them in any text editor. If you determine that loading and parsing these files is too slow, then consider moving to a binary format.

c++ best way to compare byte array to struct

I need help. I have an unsigned char * and say I have a struct
struct{
int a=3;
char b='d';
double c=3.14;
char d='e';
} cmp;
unsigned char input[1000];
l= recv(sockfd,input , sizeof(cmp),0);
I want to compare cmp and input. What is the fastest way?
Thanks a lot in advance.
If the compiler guarantees that there are no gaps between fields in the struct (usually happen due to packing) or you can use a #pragna to cancel any such gaps, then you can compare by either:
memcmp(&cmp, input, sizeof(stuct ThesSruct));
Or, my preferred:
cmp == *(struct TheStruct *)input // provided the struct doesn't contain pointers.
But a much safer way would be to compare it on a field by field basis. And even more, prepare special functions for extracting ints, floats, etc.. from the raw input. For example, extracting an int at index n may be as simple as
*(int *)&input[n]
But it might be more complicated, like shifting chars at 8, 16, 24 bits.
In short, accessing the communication data must be done with the most robust way, checking every basic element and not assuming anything.
Give reinterpret_cast a try. This will allow you to arbitrarily cast the char * to a cmp *
http://msdn.microsoft.com/en-us/library/e0w9f63b.aspx
In the general case James Kantzes comment is correct, you can't compare like that. This is , among other things, due to byte padding.
However in the specific case with the following assumptions;
The sender is on the same cpu architecture as the receiver
The sender is using the same compiler and linker as the receiver
The applications are compiled with the same compiler/linker flags
...other things...you get the gist.
The sender is sending it straight from the struct.
cmp c{ ...set variables... };
send(sockfd, (char*)&c, sizeof(c));
So in short, this is a very brittle way of transporting structs and you shouldn't do it for anything except simple tests or quick hacks.

Read Bytes off a Serial Device (and make sense of them??)

I'm pulling my hair out trying to figure out how to read bytes off a serial device, check a checksum, and then convert them into something that I can actually read.
I have a device which "should" be sending me various messages, each started with the byte $83 and ended with the byte $84. The second to last byte is supposedly a checksum, generated by XORign all the other values together and comparing.
The actual values coming back should be alphanumeric, but I can't make heads or tail of the data. I'm newish to C++ - I'm sure that's not helping.
I've read several guides on serial programming, but I'm lost.
Can anyone help me, link me, or show me how to read bytes off a serial device, watch for $83 and $84, and then make sense of the data in between?
Here is the format of each message:
$FF byte Destination Address
$10 byte Message Length 16 Bytes
$37 byte Message Type
$00 byte Message subtype
BankAngle int -179 to +180
PitchAngle int -90 to +90
YawAngle int -179 to +180
Slip sint -50 to +50
GForce fps 0 to 6G
MISC byte Mode bits
Heading word 0 to 359
N/A not used
Voltage byte input voltage
This is all coming off an MGL SP-4 AHRS, and for ease of use I am targeting a Linux system, specifically Ubuntu. I am using the GCC compiler end the Eclipse CDT for development.
Where I'm lost
I can read the data into a buffer, but then I'm not versed enough in C++ to make sense of it after that, since it's not ASCII. I'm interested in learning what I need to know, but I don't know what I need to know.
I have a Perl / Java background.
Accomplishing this is going to be wholly dependent on the Operating System and platform that you target. Since the device you mention is mounted internally to an aircraft in the general use-case, I will assume you are not targeting a Windows platform, but more likely a Linux or embedded system. There are a number of resources available for performing serial I/O on such platforms (for example: the Serial Programming HOW-TO) that you should look at. Additionally, as suggested in the device's Installation Manual (available here about halfway down the page), you should "Consult the SP-4 OEM manual for message formats and message type selection." I suspect you will obtain the most relevant and useful information from that document. You may want to check if the manufacturer provides an API for your platform, as that would negate the need for you to implement the actual communication routine.
As far as making sense of the data, once you can read bytes from your serial interface, you can leverage structs and unions to make accessing your data more programmer-friendly. For the rough message outline you provided, something like this might be appropriate:
struct _message
{
uint8_t DestinationAddress;
uint8_t MessageLength;
uint8_t MessageType;
uint8_t MessageSubtype;
int32_t BankAngle; //assuming an int is 32 bits
int32_t PitchAngle;
int32_t YawAngle;
sint_t Slip; //not sure what a 'sint' is
fps_t GForce; //likewise 'fps'
uint8_t MISC;
uint16_t Heading; //assuming a word is 16 bits
uint8_t Unused[UNUSED_BYTES]; //however many there are
uintt_t Voltage;
}
struct myMessage
{
union
{
char raw[MAX_MESSAGE_SIZE]; //sizeof(largest possible message)
struct _message message;
}
}
This way, if you were to declare struct myMessage serialData;, you can read your message into serialData.raw, and then conveniently access its members (e.g. serialData.message.DestinationAddress).
Edit: In response to your edit, I'll provide an example of how to make sense of your data. This example supposes there is only one message type you have to worry about, but it can be easily extended to other types.
struct myMessage serialData;
memcpy(serialData.raw, serialDataBuffer, MAX_MESSAGE_SIZE); //copy data from your buffer
if(serialData.message.MessageType == SOME_MESSAGE_TYPE)
{
//you have usable data here.
printf("I am a SOME_MESSAGE!\n");
}
Now, supposing that these integral types are really only useful for data transmission, you need to translate these bits into "usable data". Say one of these fields is actually an encoded floating-point number. One common scheme is to select a bit-weight (sometimes also called resolution). I don't know if this is directly applicable to your device, or if it is what the real values are, but let's say for the sake of discussion, that the YawAngle field had a resolution of 0.00014 degrees/bit. To translate the value in your message (serialData.message.YawAngle) from its uint32_t value to a double, for example, you might do this:
double YawAngleValue = 0.00014 * serialData.message.YawAngle;
...and that's about it. The OEM manual should tell you how the data is encoded, and you should be able to work out how to decode it from there.
Now, let's say you've got two message types to handle. The one I've already shown you, and a theoretical CRITICAL_BITS message. To add that type using the scheme I've laid out, you would first define the CRITICAL_BITS structure (perhaps as follows):
struct _critical_bits
{
uint8_t DestinationAddress;
uint8_t MessageLength;
uint8_t MessageType;
uint8_t MessageSubtype;
uint32_t SomeCriticalData;
}
...and then add it to the struct myMessage definition like so:
struct myMessage
{
union
{
char raw[MAX_MESSAGE_SIZE]; //sizeof(largest possible message)
struct _message message;
struct _critical_bits critical_message;
}
}
...then you can access the SomeCriticalData just like the other fields.
if(serialData.message.MessageType == CRITICAL_MESSAGE_TYPE)
{
uint32_t critical_bits = serialData.critical_message.SomeCriticalData;
}
You can find a little more information on how this works by reading about structs. Bear in mind, that instances of the struct myMessage type will only ever contain one set of meaningful data at a time. Put more simply, if serialData contains CRITICAL_MESSAGE_TYPE data, then the data in serialData.critical_message is valid, but serialData.message is not --even though the language does not prevent you from accessing that data if you request it.
Edit: One more example; to calculate the checksum of a message, using the algorithm you've specified, you would probably want something like this (assuming you already know the message is completely within the buffer):
uint8_t calculate_checksum(struct myMessage *data)
{
uint8_t number_bytes = data->message.MessageLength;
uint8_t checksum = 0;
int i;
for(i=0; i<number_bytes; ++i)
{
//this performs a XOR with checksum and the byte
//in the message at offset i
checksum ^= data->raw[i];
}
return checksum;
}
You might need to adjust that function for bytes that aren't included, check to make sure that data != NULL, etc. but it should get you started.

Creating and sending data packets in C/C++

Let's say I want to send the following data to a socket using C or C++, all in one packet:
Headers
-------
Field 1: 2 byte hex
Field 2: 2 byte hex
Field 3: 4 byte hex
Data
----
Field1 : 2 byte hex
Field1 : 8 byte hex
What would the code typically look like to create and send the packet containing all this data?
Let's suppose that your program is already organized to have the header in one struct and the data in another struct. For example, you might have these data structures:
#include <stdint.h>
struct header {
uint16_t f1;
uint16_t f2;
uint32_t f3;
};
struct data {
uint16_t pf1;
uint64_t pf2;
};
Let's call this organization "host format". It really doesn't matter to me what the host format is, as long as it is useful to the rest of your program. Let's call the format that you will pass to the send() call "network format". (I chose these names to match the htons (host-to-network-short) and htonl (host-to-network-long) names.)
Here are some conversion functions that we might find handy. Each of these converts your host format structures to a network format buffer.
#include <arpa/inet.h>
#include <string.h>
void htonHeader(struct header h, char buffer[8]) {
uint16_t u16;
uint32_t u32;
u16 = htons(h.f1);
memcpy(buffer+0, &u16, 2);
u16 = htons(h.f2);
memcpy(buffer+2, &u16, 2);
u32 = htonl(h.f3);
memcpy(buffer+4, &u32, 4);
}
void htonData(struct data d, char buffer[10]) {
uint16_t u16;
uint32_t u32;
u16 = htons(d.pf1);
memcpy(buffer+0, &u16, 2);
u32 = htonl(d.pf2>>32);
memcpy(buffer+2, &u32, 4);
u32 = htonl(d.pf2);
memcpy(buffer+6, u32, 4);
}
void htonHeaderData(struct header h, struct data d, char buffer[18]) {
htonHeader(h, buffer+0);
htonData(d, buffer+8);
}
To send your data, do this:
...
char buffer[18];
htonHeaderData(myPacketHeader, myPacketData, buffer);
send(sockfd, buffer, 18, 0);
...
Again, you don't have to use the header and data structs that I defined. Just use whatever your program needs. The key is that you have a conversion function that writes all of the data, at well-defined offsets, in a well-defined byte order, to a buffer, and that you pass that buffer to the send() function.
On the other side of the network connection, you will need a program to interpret the data it receives. On that side, you need to write the corresponding functions (ntohHeader, etc). Those function will memcpy the bits out of a buffer and into a local variable, which it can pass to ntohs or ntohl. I'll leave those functions for you to write.
Well, typically it would look like it's preparing that packet structure into a memory buffer (making judicious calls the the htonl family of functions).
If would then use the send, sendto, sendmsg or write functions, hopefully with a lot of care taken with the length of the buffer and good error handling/reporting.
(Or one of the Win32 apis for the send, if that is the target plateforms.)
You'll find a good presentation about all this at Beej's Guide to Network Programming.
Specifially for the byte packing part (with endian consideration), look at the serialization topic. (There's way more detail in that section than what you need for plain fixed-size integer data types.
The code would look different depending on the OS's networking library (*nix uses Berkeley sockets, Windows uses Winsock, etc.). However, you could create a struct containing all the data you wanted to send in a packet, e.g.,
typedef struct
{
short field1;
short field2;
int field3;
} HeaderStruct;
typedef struct
{
short field1;
long long field2;
} PacketDataStruct;
assuming a 32-bit int size.
Edit:
As someone kindly reminded me in the comments, don't forget about converting to and from Network Order. Networking libraries will have functions to assist with this, such as ntohs, nothl, htons, and htonl.
One simple answer is that it would be sent in the format that the receiver expects. That begs the question a bit, though. Assuming the data is a fixed size as shown and the receiving end expects, then you could use a packed (1 byte alignment) structure and store the data in each field. The reason for using 1 byte alignment is that it is typically easier to make sure both ends are expecting the same data. Without 1 byte alignment, then the structure would possibly look different based on compiler options, 32-bit versus 64-bit architecture, etc.) And, typically, it is expected that you would send the values in network byte order if the hex values are integers. You can use functions such as htons and htonl (and possibly htobe64 if available) to convert them.
Assuming that the data is in the structure with the desired byte order, then the send call may be something like this:
ret = send( socket, &mystruct, sizeof( mystruct ), 0 );
That assumes that mystruct is declared as an instance of the structure as opposed to a pointer to the structure.