Let's say I want to send the following data to a socket using C or C++, all in one packet:
Headers
-------
Field 1: 2 byte hex
Field 2: 2 byte hex
Field 3: 4 byte hex
Data
----
Field1 : 2 byte hex
Field1 : 8 byte hex
What would the code typically look like to create and send the packet containing all this data?
Let's suppose that your program is already organized to have the header in one struct and the data in another struct. For example, you might have these data structures:
#include <stdint.h>
struct header {
uint16_t f1;
uint16_t f2;
uint32_t f3;
};
struct data {
uint16_t pf1;
uint64_t pf2;
};
Let's call this organization "host format". It really doesn't matter to me what the host format is, as long as it is useful to the rest of your program. Let's call the format that you will pass to the send() call "network format". (I chose these names to match the htons (host-to-network-short) and htonl (host-to-network-long) names.)
Here are some conversion functions that we might find handy. Each of these converts your host format structures to a network format buffer.
#include <arpa/inet.h>
#include <string.h>
void htonHeader(struct header h, char buffer[8]) {
uint16_t u16;
uint32_t u32;
u16 = htons(h.f1);
memcpy(buffer+0, &u16, 2);
u16 = htons(h.f2);
memcpy(buffer+2, &u16, 2);
u32 = htonl(h.f3);
memcpy(buffer+4, &u32, 4);
}
void htonData(struct data d, char buffer[10]) {
uint16_t u16;
uint32_t u32;
u16 = htons(d.pf1);
memcpy(buffer+0, &u16, 2);
u32 = htonl(d.pf2>>32);
memcpy(buffer+2, &u32, 4);
u32 = htonl(d.pf2);
memcpy(buffer+6, u32, 4);
}
void htonHeaderData(struct header h, struct data d, char buffer[18]) {
htonHeader(h, buffer+0);
htonData(d, buffer+8);
}
To send your data, do this:
...
char buffer[18];
htonHeaderData(myPacketHeader, myPacketData, buffer);
send(sockfd, buffer, 18, 0);
...
Again, you don't have to use the header and data structs that I defined. Just use whatever your program needs. The key is that you have a conversion function that writes all of the data, at well-defined offsets, in a well-defined byte order, to a buffer, and that you pass that buffer to the send() function.
On the other side of the network connection, you will need a program to interpret the data it receives. On that side, you need to write the corresponding functions (ntohHeader, etc). Those function will memcpy the bits out of a buffer and into a local variable, which it can pass to ntohs or ntohl. I'll leave those functions for you to write.
Well, typically it would look like it's preparing that packet structure into a memory buffer (making judicious calls the the htonl family of functions).
If would then use the send, sendto, sendmsg or write functions, hopefully with a lot of care taken with the length of the buffer and good error handling/reporting.
(Or one of the Win32 apis for the send, if that is the target plateforms.)
You'll find a good presentation about all this at Beej's Guide to Network Programming.
Specifially for the byte packing part (with endian consideration), look at the serialization topic. (There's way more detail in that section than what you need for plain fixed-size integer data types.
The code would look different depending on the OS's networking library (*nix uses Berkeley sockets, Windows uses Winsock, etc.). However, you could create a struct containing all the data you wanted to send in a packet, e.g.,
typedef struct
{
short field1;
short field2;
int field3;
} HeaderStruct;
typedef struct
{
short field1;
long long field2;
} PacketDataStruct;
assuming a 32-bit int size.
Edit:
As someone kindly reminded me in the comments, don't forget about converting to and from Network Order. Networking libraries will have functions to assist with this, such as ntohs, nothl, htons, and htonl.
One simple answer is that it would be sent in the format that the receiver expects. That begs the question a bit, though. Assuming the data is a fixed size as shown and the receiving end expects, then you could use a packed (1 byte alignment) structure and store the data in each field. The reason for using 1 byte alignment is that it is typically easier to make sure both ends are expecting the same data. Without 1 byte alignment, then the structure would possibly look different based on compiler options, 32-bit versus 64-bit architecture, etc.) And, typically, it is expected that you would send the values in network byte order if the hex values are integers. You can use functions such as htons and htonl (and possibly htobe64 if available) to convert them.
Assuming that the data is in the structure with the desired byte order, then the send call may be something like this:
ret = send( socket, &mystruct, sizeof( mystruct ), 0 );
That assumes that mystruct is declared as an instance of the structure as opposed to a pointer to the structure.
Related
I'm trying to write a small application that will "concatenate" a bunch of integer variables of various sizes (uint8, uint16, uint32) into a 128 byte message that will be sent via UDP socket.
On the receiving side, I would like to split the message back into the individual integers and store them for further processing. For now I am assuming that endianness will not be an issue.
Could I get some hints on how to concatenate the int variables and later extract them from that 128 byte message?
So, let's say you expect to recieve data in the following order:
int32_t header; int8_t opcode; int16_t args[32]; int32_t clients[2]; ...
This is just an example, parameters could be whatever they are in your actuai task.
You can wrap those parameters into struct or class. I'd prefer a struct here because it does not seem like you really need to create a ctor, access specifiers or any other fancy stuff class can provide. So, something like that:
#pragma pack(push, 1)
struct DataFromMyHardware {
int32_t header;
int8_t opcode;
int16_t args[32];
int32_t clients[2];
...
};
#pragma pack(pop)
pragmas here used to tell compiler to not optimize placement or alignment of variables in struct, so it'll be stored in memory as is.
This way, you can use this on sender:
DataFromMyHardware buffer;
buffer.header = 0xDEADBEEF;
buffer.opcode = 42;
...
send(socket, &buffer, sizeof(buffer), 0);
and on reciever:
DataFromMyHardware buffer;
recv(socket, (void*)&buffer, sizeof(buffer), 0);
Side note: it's very likely that your device uses network byte-order, so you probably want to use nhohl/ntohs on reciever and htonl/htons on sender.
You can create a character buffer prefix that will precede each integer which will define which type of integer to read so then you can read the correct size of the integer that is defined.
for ex: Precede each integer with a character representing the integer. The first byte will contain the character which you will interpret as an, it could be any ascii character that will represent the integer.
array == [byte][byte-8bit][byte][2 byte-16bit][byte][4 byte -32bit]...
UINT8 = 'a'
UINT16 = 'b'
UINT32 = 'c' or whatever code you want the ascii to be... I used a,b,c so its a readable character you can read in the debugger
You will then have to build your array to be sent with the prefix so you know what size to do your next read.
array = [a0b00c0000a0b00c0000] and so on, you can build the array using memcpy
You can then send the whole packet of 128 bytes
Make sure you read exactly 128 bytes, and then you can deconstruct, remember that when you read you have to check the amount that was read from the socket, and continue to read until you receive the correct amount. -- sometimes a read will not return the correct amount of bytes that you will expect.
When you receive the packet, you can deconstruct the packet using the headers, and depending on the header that you receive you can remove the correct integer and size, and the headers with the correct size should be deconstructed rather easily.
Also remember that UDP is lossy so you will possibly lose packets
Now also remember that if your constructed packet does not equal exactly 128 bytes every time you will need to add another byte which will be equivalent to a integer which will tell you exactly how many bytes were sent... 122..126..127 etc, and read that as the first header on the other side.
I am doing a Header for an UDP socket which have a restrictions using bytes.
| Packet ID (1 byte) | Packet Size (2 bytes) | Subpacket ID (1 Byte) | etc
I did an struct for store this kind of attributes like:
typedef struct WHEATHER_STRUCT
{
unsigned char packetID[1];
unsigned char packetSize[2];
unsigned char subPacketID[1];
unsigned char subPacketOffset[2];
...
} wheather_struct;
I initialized this struct using new and I updated the values. The question is about if I want to use only 2 bytes in Packet Size attribute. What of these two forms that I wrote below is the correct one?
*weather_struct->packetSize = '50';
or
*weather_struct->packetSize = 50;
If you can use C++11 and gcc (or clang) then I would do this:
typedef struct WHEATHER_STRUCT
{
uint8_t packetID;
uint16_t packetSize;
uint8_t subPacketID;
uint16_t subPacketOffset;
// ...
} __attribute__((packed)) wheather_struct;
If you can't use C++11 then you can use unsigned char and unsigned short instead.
If you're using Visual C then you can do:
#pragma pack (push, 1)
typedef struct ...
#pragma (pop)
Beware also byte ordering issues, depending on what architectures you need to support. You can use htons() and ntohs() to overcome this problem.
Live demo at Wandbox
Packing and unpacking data from IP packets is a problem as old as the internet itself (indeed, older).
Different machine architectures have different layouts for representing integers, which can cause problems when communicating between machines.
For this reason, the IP stack standardises on encoding integers in 'network byte order' (which basically means most significant byte first).
Standard functions exist to convert values in network byte order to native types and vice versa. I urge you to consider using these as your code will then be more portable.
Furthermore, it makes sense to abstract data representations from the program's point of view. c++ compilers can perform the conversions very efficiently.
Example:
#include <arpa/inet.h>
#include <cstring>
#include <cstdint>
typedef struct WEATHER_STRUCT
{
std::int8_t packetID;
std::uint16_t packetSize;
std::uint8_t subPacketID;
std::uint16_t subPacketOffset;
} weather_struct;
const std::int8_t* populate(weather_struct& target, const std::int8_t* source)
{
auto get16 = [&source]
{
std::uint16_t buf16;
std::memcpy(&buf16, source, 2);
source += 2;
return ntohs(buf16);
};
target.packetID = *source++;
target.packetSize = get16();
target.subPacketID = *source++;
target.subPacketOffset = get16();
return source;
}
uint8_t* serialise(uint8_t* target, weather_struct const& source)
{
auto write16 = [&target](std::uint16_t val)
{
val = ntohs(val);
std::memcpy(target, &val, 2);
target += 2;
};
*target++ = source.packetID;
write16(source.packetSize);
*target++ = source.subPacketID;
write16(source.subPacketOffset);
return target;
}
https://linux.die.net/man/3/htons
here's an link to a c++17 version of the above:
https://godbolt.org/z/oRASjI
A further note on conversion costs:
Data arriving into or leaving your program is an event that happens once per payload. Suffering a conversion cost here incurs a negligible penalty.
Once the data has arrived in your program, or before it leaves, it may be manipulated many times by your code.
Some processors architectures suffer huge performance penalties during data access if data is not aligned on natural word boundaries. This is why attributes such as packed exist - the compiler is doing all it can to avoid misaligned data. Using a packed attribute is tantamount to deliberately telling the compiler to produce very suboptimal code.
For this reason, I would recommend not using packed structures (e.g. __attribute__((packed)) etc) for data that will be referred to by program logic.
Compared to RAM, networks are many orders of magnitude slower. A minuscule performance hit (literally nanoseconds) at the point of encoding or decoding a network packet is inconsequential compared to the cost of actually transmitting it.
Packing structures can cause horrible performance issues in program code and often leads to portability headaches.
Neither is correct, you need to treat the two bytes as a single 16-bit number. You probably also need to take into account the different endianness of the network stream to your processor architecture (depending on the protocol, most are big endian).
The correct code would therefore be:
*((uint16_t*)weather_struct->packetSize) = htons(50);
It would be simpler if packetSize were uint16_t to start with:
weather_struct->packetSize = htons(50);
I'm using WinSock to send UDP packets to the server, I need to send the data in big endian. I'm not sure how to convert the byte order of my structure before sending.
I have a struct like this:
struct ConnectIn
{
std::int64_t ConnectionID = 0x41727101980;
std::int32_t Action = 0;
std::int32_t TransactionID;
ConnectIn(std::int32_t transactionID)
{
TransactionID = transactionID;
}
};
And at the moment I'm sending like this:
ConnectIn msg(123);
int len = sizeof(msg);
int bytesSent = sendto(s, (char*)&msg, len, 0, (SOCKADDR*)&dest, sizeof(address));
How can I convert the byte order of msg to big endian before sending?
If you're curious, the data I'm sending is for the Bit Torrent UDP tracker protocol.
If you want to do this manually then what you do is swap each member individually. You convert the members from the host computer's byte ordering to the network's byte ordering. On Win32 htonll() is for 64-bit integers and htonl() is for 32-bit integers:
#include <Winsock2.h>
ConnectIn msg(123);
msg.ConnectionID = htonll(msg.ConnectionID);
msg.Action = htonl(msg.Action);
msg.TransactionID= htonl(msg.TransactionID);
Then you might also want to send the members individually, to avoid relying on the host system's struct layout. The Windows ABI doesn't insert any padding in this struct, but perhaps for some other struct you use it does. So here's the basic idea:
char buf[sizeof msg.ConnectionID + sizeof msg.Action + sizeof msg.TransactionID];
char *bufi = buf;
std::memcpy(bufi, &msg.ConnectionID, sizeof msg.ConnectionID);
bufi += sizeof msg.ConnectionID;
std::memcpy(bufi, &msg.Action, sizeof msg.Action);
bufi += sizeof msg.Action;
std::memcpy(bufi, &msg.TransactionID, sizeof msg.TransactionID);
bufi += sizeof msg.TransactionID;
int len = sizeof buf;
int bytesSent = sendto(s, buf, len, 0, (SOCKADDR*)&dest, sizeof(address));
Then on the receiving side you use the appropriate ntoh*() functions for 64-bit and 32-bit types to convert from the network's byte ordering to the receiving host's byte ordering.
Yes, the Network Byte Order (NBO) is Big Endian and so you need to find a way to send that structure on the web.
What you're currently doing won't work: you're sending the whole struct but the receiver may have a different endianness, padding and so on.
The easiest options are:
Sending each field with a protocol-defined layout
Third part libraries which handle serialization: Google Protobuf is one of the most common ones.
For the first option, there're some functions which take care of that in the Winsock2 library. These are:
(WSA)ntoht (Network to Host t, where t can be short and unsigned)
(WSA)htont (Host to Network t, where t can be short and unsigned)
WSA functions are a little bit different and Windows-only.
The Network Programming Guide
Winsock Reference
One option is converting each of the numbers individually
For GCC:
int32_t __builtin_bswap32 (int32_t x)
int64_t __builtin_bswap64 (int64_t x)
For MSVC:
unsigned short _byteswap_ushort(unsigned short value);
unsigned long _byteswap_ulong(unsigned long value);
unsigned __int64 _byteswap_uint64(unsigned __int64 value);
I'm pulling my hair out trying to figure out how to read bytes off a serial device, check a checksum, and then convert them into something that I can actually read.
I have a device which "should" be sending me various messages, each started with the byte $83 and ended with the byte $84. The second to last byte is supposedly a checksum, generated by XORign all the other values together and comparing.
The actual values coming back should be alphanumeric, but I can't make heads or tail of the data. I'm newish to C++ - I'm sure that's not helping.
I've read several guides on serial programming, but I'm lost.
Can anyone help me, link me, or show me how to read bytes off a serial device, watch for $83 and $84, and then make sense of the data in between?
Here is the format of each message:
$FF byte Destination Address
$10 byte Message Length 16 Bytes
$37 byte Message Type
$00 byte Message subtype
BankAngle int -179 to +180
PitchAngle int -90 to +90
YawAngle int -179 to +180
Slip sint -50 to +50
GForce fps 0 to 6G
MISC byte Mode bits
Heading word 0 to 359
N/A not used
Voltage byte input voltage
This is all coming off an MGL SP-4 AHRS, and for ease of use I am targeting a Linux system, specifically Ubuntu. I am using the GCC compiler end the Eclipse CDT for development.
Where I'm lost
I can read the data into a buffer, but then I'm not versed enough in C++ to make sense of it after that, since it's not ASCII. I'm interested in learning what I need to know, but I don't know what I need to know.
I have a Perl / Java background.
Accomplishing this is going to be wholly dependent on the Operating System and platform that you target. Since the device you mention is mounted internally to an aircraft in the general use-case, I will assume you are not targeting a Windows platform, but more likely a Linux or embedded system. There are a number of resources available for performing serial I/O on such platforms (for example: the Serial Programming HOW-TO) that you should look at. Additionally, as suggested in the device's Installation Manual (available here about halfway down the page), you should "Consult the SP-4 OEM manual for message formats and message type selection." I suspect you will obtain the most relevant and useful information from that document. You may want to check if the manufacturer provides an API for your platform, as that would negate the need for you to implement the actual communication routine.
As far as making sense of the data, once you can read bytes from your serial interface, you can leverage structs and unions to make accessing your data more programmer-friendly. For the rough message outline you provided, something like this might be appropriate:
struct _message
{
uint8_t DestinationAddress;
uint8_t MessageLength;
uint8_t MessageType;
uint8_t MessageSubtype;
int32_t BankAngle; //assuming an int is 32 bits
int32_t PitchAngle;
int32_t YawAngle;
sint_t Slip; //not sure what a 'sint' is
fps_t GForce; //likewise 'fps'
uint8_t MISC;
uint16_t Heading; //assuming a word is 16 bits
uint8_t Unused[UNUSED_BYTES]; //however many there are
uintt_t Voltage;
}
struct myMessage
{
union
{
char raw[MAX_MESSAGE_SIZE]; //sizeof(largest possible message)
struct _message message;
}
}
This way, if you were to declare struct myMessage serialData;, you can read your message into serialData.raw, and then conveniently access its members (e.g. serialData.message.DestinationAddress).
Edit: In response to your edit, I'll provide an example of how to make sense of your data. This example supposes there is only one message type you have to worry about, but it can be easily extended to other types.
struct myMessage serialData;
memcpy(serialData.raw, serialDataBuffer, MAX_MESSAGE_SIZE); //copy data from your buffer
if(serialData.message.MessageType == SOME_MESSAGE_TYPE)
{
//you have usable data here.
printf("I am a SOME_MESSAGE!\n");
}
Now, supposing that these integral types are really only useful for data transmission, you need to translate these bits into "usable data". Say one of these fields is actually an encoded floating-point number. One common scheme is to select a bit-weight (sometimes also called resolution). I don't know if this is directly applicable to your device, or if it is what the real values are, but let's say for the sake of discussion, that the YawAngle field had a resolution of 0.00014 degrees/bit. To translate the value in your message (serialData.message.YawAngle) from its uint32_t value to a double, for example, you might do this:
double YawAngleValue = 0.00014 * serialData.message.YawAngle;
...and that's about it. The OEM manual should tell you how the data is encoded, and you should be able to work out how to decode it from there.
Now, let's say you've got two message types to handle. The one I've already shown you, and a theoretical CRITICAL_BITS message. To add that type using the scheme I've laid out, you would first define the CRITICAL_BITS structure (perhaps as follows):
struct _critical_bits
{
uint8_t DestinationAddress;
uint8_t MessageLength;
uint8_t MessageType;
uint8_t MessageSubtype;
uint32_t SomeCriticalData;
}
...and then add it to the struct myMessage definition like so:
struct myMessage
{
union
{
char raw[MAX_MESSAGE_SIZE]; //sizeof(largest possible message)
struct _message message;
struct _critical_bits critical_message;
}
}
...then you can access the SomeCriticalData just like the other fields.
if(serialData.message.MessageType == CRITICAL_MESSAGE_TYPE)
{
uint32_t critical_bits = serialData.critical_message.SomeCriticalData;
}
You can find a little more information on how this works by reading about structs. Bear in mind, that instances of the struct myMessage type will only ever contain one set of meaningful data at a time. Put more simply, if serialData contains CRITICAL_MESSAGE_TYPE data, then the data in serialData.critical_message is valid, but serialData.message is not --even though the language does not prevent you from accessing that data if you request it.
Edit: One more example; to calculate the checksum of a message, using the algorithm you've specified, you would probably want something like this (assuming you already know the message is completely within the buffer):
uint8_t calculate_checksum(struct myMessage *data)
{
uint8_t number_bytes = data->message.MessageLength;
uint8_t checksum = 0;
int i;
for(i=0; i<number_bytes; ++i)
{
//this performs a XOR with checksum and the byte
//in the message at offset i
checksum ^= data->raw[i];
}
return checksum;
}
You might need to adjust that function for bytes that aren't included, check to make sure that data != NULL, etc. but it should get you started.
This is related to my question asked here today on SO. Is there a better way to build a packet to send over serial rather than doing this:
unsigned char buff[255];
buff[0] = 0x02
buff[1] = 0x01
buff[2] = 0x03
WriteFile(.., buff,3, &dwBytesWrite,..);
Note: I have about twenty commands to send, so if there was a better way to send these bytes to the serial device in a more concise manner rather than having to specify each byte, it would be great. Each byte is hexadecimal, with the last byte being the checksum. I should clarify that I know I will have to specify each byte to build the commands, but is there a better way than having to specify each array position?
You can initialize static buffers like so:
const unsigned char command[] = {0x13, 0x37, 0xf0, 0x0d};
You could even use these to initialize non-const buffers and then replace only changing bytes by index.
Not sure what you're asking. If you ask about the problem of setting the byte one by one and messing up the data, usually this is doen with a packed struct with members having meaningful names. Like:
#pragma push(pack)
#pragma pack(1)
struct FooHeader {
uint someField;
byte someFlag;
dword someStatus;
};
#pragma pack(pop)
FooHeader hdr;
hdr.someField = 2;
hdr.someFlag = 3;
hdr.someStatus = 4;
WriteFile(..., sizeof(hdr), &hdr);
Is there a better way to build a packet than assembling it byte by byte?
Yes, but it will require some thought and some careful engineering. Many of the other answers tell you other mechanisms by which you can put together a sequence of bytes in C++. But I suggest you design an abstraction that represents a part of a packet:
class PacketField {
void add_to_packet(Packet p);
};
Then you can define various subclasses:
Add a single byte to the packet
Add a 16-bit integer in big-endian order. Another for little-endian. Other widths besides 16.
Add a string to the packet; code the string by inserting the length and then the bytes.
You also can define a higher-order version:
PacketField sequence(PacketField first, PacketField second);
Returns a field that consists of the two arguments in sequence. If you like operator overloading you could overload this as + or <<.
Your underlying Packet abstraction will just be an extensible sequence of bytes (dynamic array) with some kind of write method.
If you wind up programming a lot of network protocols, you'll find this sort of design pays off big time.
Edit: The point of the PacketField class is composability and reuse:
By composing packet fields you can create more complex packet fields. For example, you could define "add a TCP header" as a function from PacketFields to PacketFields.
With luck you build up a library of PacketFields that are specific to your application or protocol family or whatever. Then you reuse the fields in the library.
You can create subclasses of PacketField that take extra parameters.
It's quite possibly that you can do something equally nice without having to have this extra level of indirection; I'm recommending it because I've seen it used effectively in other applications. You are decoupling the knowledge of how to build a packet (which can be applied to any packet, any time) from the act of actually building a particular packet. Separating concerns like this can help reuse.
Yes, there is a better method. Have your classes read from and write to a packed buffer. You could even implement this as an interface. Templates would help to.
An example of writing:
template <typename Member_Type>
void Store_Value_In_Buffer(const Member_Type&, member,
unsigned char *& p_buffer)
{
*((Member_Type *)(p_buffer)) = member;
p_buffer += sizeof(Member_Type);
return;
}
struct My_Class
{
unsigned int datum;
void store_to_buffer(unsigned char *& p_buffer)
{
Store_Value_In_Buffer(datum, buffer);
return;
}
};
//...
unsigned char buffer[256];
unsigned char * p_buffer(buffer);
MyClass object;
object.datum = 5;
object.store_to_buffer(p_buffer);
std::cout.write(p_buffer, 256);
Part of the interface is also to query the objects for the size that they would occupy in the buffer, say a method size_in_buffer. This is left as an exercise for the reader. :-)
There is a much better way, which is using structs to set the structures. This is usually how network packets are built on a low level.
For example, say you have packets which have an id, length, flag byte, and data, you'd do something like this:
struct packet_header {
int id;
byte length;
byte flags;
};
byte my_packet[] = new byte[100];
packet_header *header = &my_packet;
header->id = 20;
header->length = 10; // This can be set automatically by a function, maybe?
// etc.
header++; // Header now points to the data section.
Do note that you're going to have to make sure that the structures are "packed", i.e. when you write byte length, it really takes up a byte. Usually, you'd achieve this using something like #pragma pack or similar (you'll have to read about your compiler's pragma settings).
Also, note that you should probably use functions to do common operations. For example, create a function which gets as input the size, data to send, and other information, and fills out the packet header and data for you. This way, you can perform calculations about the actual size you want to write in the length field, you can calculate the CRC inside the function, etc.
Edit: This is a C-centric way of doing things, which is the style of a lot of networking code. A more C++-centric (object oriented) approach could also work, but I'm less familiar with them.
const char *c = "\x02\x02\x03";