I want to send a struct sMessage byte by byte from one microcontroller to another:
typedef struct {
unsigned char mu8MessageId;
unsigned char mu8MessageSenderId;
unsigned char mu8MessageReceiverId;
unsigned short mu16MessageSizePayload;
unsigned short mu16MessageSizeTotal;
} sMessage;
sMessage msMessage;
Therefore I use a simple function
MSGSENDER_vSend(void* pvMessage, unsigned short u16Size)
which takes a void poiner and the size of the data to be sent. At the receiver I receive the sent data byte by byte, and reconstructs the sent data out of it. I have the same byte order on both devices.
Unfortunately I receive the bytes in a different order if I
call this function once with the struct msMessage and its size as parameters (byte order seems to be right)
MSGSENDER_vSend(&msMessage, sizeof(msMessage))
call this function for every member of the struct (byte order is not correct)
MSGSENDER_vSend(&msMessage.mu8MessageId, sizeof(msMessage.mu8MessageId))
MSGSENDER_vSend(&msMessage.mu8MessageSenderId, sizeof(msMessage.mu8MessageSenderId))
//...
Why are the bytes sent in a different order?
Depending on the compiler and the compile flags, there can be "gaps" between the members in the struct. If it optimizes for speed, there is a chance that it will try to align the members according to the largest member type.
In your case you are using 3 1-byte members and 2 2-byte members and may insert 1 byte of void between the last char and the first short so the short is aligned by two (since it's 2 bytes)
Anyway, you should never send (or store) raw structs since they can change if you recompile the application with different flags. The safest way is always to send the data members one by one, then you have full control of the order on the receiving side
Related
I am taking binary input from a file to a buffer vector then casting the pointer of that buffer to be my struct type.
The goal is for the data to populate the struct perfectly.
I know the size of all the various fields and the order they're going to come in.
As a result my struct needs to be tightly packed and be 42 bytes long.
My issue is that it is coming out at 44 bytes long when I test it.
Also, the first value lines up. After that, the data is incorrect.
Here's the struct:
#pragma pack(push, 1)
struct myStruct
{
uint8_t ID;
uint32_t size: 24;
uint16_t value;
char name[12];
char description[4];
char shoppingList[14];
char otherValue[6];
};
#pragma pack(pop)
Also, the first value lines up. After that, the data is incorrect.
uint32_t size: 24;
If you want to guarantee portably that this is three bytes with no padding before the next member, you're going to need to use a byte buffer and do the conversions yourself.
#pragma pack is an extension, and the packing of bitfield members is anyway implementation-defined.
FWIW both GCC and CLANG do seem to do what you want in this case, but unless it's defined by a platform ABI depending on this is still brittle.
I'm trying to write a small application that will "concatenate" a bunch of integer variables of various sizes (uint8, uint16, uint32) into a 128 byte message that will be sent via UDP socket.
On the receiving side, I would like to split the message back into the individual integers and store them for further processing. For now I am assuming that endianness will not be an issue.
Could I get some hints on how to concatenate the int variables and later extract them from that 128 byte message?
So, let's say you expect to recieve data in the following order:
int32_t header; int8_t opcode; int16_t args[32]; int32_t clients[2]; ...
This is just an example, parameters could be whatever they are in your actuai task.
You can wrap those parameters into struct or class. I'd prefer a struct here because it does not seem like you really need to create a ctor, access specifiers or any other fancy stuff class can provide. So, something like that:
#pragma pack(push, 1)
struct DataFromMyHardware {
int32_t header;
int8_t opcode;
int16_t args[32];
int32_t clients[2];
...
};
#pragma pack(pop)
pragmas here used to tell compiler to not optimize placement or alignment of variables in struct, so it'll be stored in memory as is.
This way, you can use this on sender:
DataFromMyHardware buffer;
buffer.header = 0xDEADBEEF;
buffer.opcode = 42;
...
send(socket, &buffer, sizeof(buffer), 0);
and on reciever:
DataFromMyHardware buffer;
recv(socket, (void*)&buffer, sizeof(buffer), 0);
Side note: it's very likely that your device uses network byte-order, so you probably want to use nhohl/ntohs on reciever and htonl/htons on sender.
You can create a character buffer prefix that will precede each integer which will define which type of integer to read so then you can read the correct size of the integer that is defined.
for ex: Precede each integer with a character representing the integer. The first byte will contain the character which you will interpret as an, it could be any ascii character that will represent the integer.
array == [byte][byte-8bit][byte][2 byte-16bit][byte][4 byte -32bit]...
UINT8 = 'a'
UINT16 = 'b'
UINT32 = 'c' or whatever code you want the ascii to be... I used a,b,c so its a readable character you can read in the debugger
You will then have to build your array to be sent with the prefix so you know what size to do your next read.
array = [a0b00c0000a0b00c0000] and so on, you can build the array using memcpy
You can then send the whole packet of 128 bytes
Make sure you read exactly 128 bytes, and then you can deconstruct, remember that when you read you have to check the amount that was read from the socket, and continue to read until you receive the correct amount. -- sometimes a read will not return the correct amount of bytes that you will expect.
When you receive the packet, you can deconstruct the packet using the headers, and depending on the header that you receive you can remove the correct integer and size, and the headers with the correct size should be deconstructed rather easily.
Also remember that UDP is lossy so you will possibly lose packets
Now also remember that if your constructed packet does not equal exactly 128 bytes every time you will need to add another byte which will be equivalent to a integer which will tell you exactly how many bytes were sent... 122..126..127 etc, and read that as the first header on the other side.
I'm writing a networking application using sockets in c++ so lets jump straight to the problem :
i'm storing my data as an array of int16_ts (the choice of int16_t being for consistency accross different platforms) , as we know each of these int16_ts would be two consecutive bytes in memory. i want to store each of those bytes in a char so that each int16 would be translated to exactly two bytes and eventually send the entire char* over the socket
please notice that i'm not looking for something such as std::to_string cause i want each int16_t to occupy exactly two bytes.
any help is appreciated !
You would need to copy each int16_t one at a time to the char * buffer, calling htons() on each one to translate the bytes into network byte order. Then on the receiving side, you would call ntohs() to convert back.
int send_array(int16_t *myarray, int len, int socket) {
char buf[1000];
int16_t *p;
int i;
p = (int16_t *)buf;
for (i=0;i<len;i++) {
p[i] = htons(myarray[i]);
}
return send(socket,buf,len*sizeof(int16_t),0);
}
C++ has a new type, char16_t, that's designed to hold UTF-16 characters. If you mean you want to send the 16-bit hints over the wire one byte at a time, convert each one to network byte order with htons(), store them in a new array of uint16_t, then send that array over the socket. If you want to address the array of shorts as an array of bytes, you can do that either through a union, with a pointer cast, or with a reference to an array. Example: char* const p=reinterpret_cast<char*>(&shorts[0]);.
Let's say I want to send the following data to a socket using C or C++, all in one packet:
Headers
-------
Field 1: 2 byte hex
Field 2: 2 byte hex
Field 3: 4 byte hex
Data
----
Field1 : 2 byte hex
Field1 : 8 byte hex
What would the code typically look like to create and send the packet containing all this data?
Let's suppose that your program is already organized to have the header in one struct and the data in another struct. For example, you might have these data structures:
#include <stdint.h>
struct header {
uint16_t f1;
uint16_t f2;
uint32_t f3;
};
struct data {
uint16_t pf1;
uint64_t pf2;
};
Let's call this organization "host format". It really doesn't matter to me what the host format is, as long as it is useful to the rest of your program. Let's call the format that you will pass to the send() call "network format". (I chose these names to match the htons (host-to-network-short) and htonl (host-to-network-long) names.)
Here are some conversion functions that we might find handy. Each of these converts your host format structures to a network format buffer.
#include <arpa/inet.h>
#include <string.h>
void htonHeader(struct header h, char buffer[8]) {
uint16_t u16;
uint32_t u32;
u16 = htons(h.f1);
memcpy(buffer+0, &u16, 2);
u16 = htons(h.f2);
memcpy(buffer+2, &u16, 2);
u32 = htonl(h.f3);
memcpy(buffer+4, &u32, 4);
}
void htonData(struct data d, char buffer[10]) {
uint16_t u16;
uint32_t u32;
u16 = htons(d.pf1);
memcpy(buffer+0, &u16, 2);
u32 = htonl(d.pf2>>32);
memcpy(buffer+2, &u32, 4);
u32 = htonl(d.pf2);
memcpy(buffer+6, u32, 4);
}
void htonHeaderData(struct header h, struct data d, char buffer[18]) {
htonHeader(h, buffer+0);
htonData(d, buffer+8);
}
To send your data, do this:
...
char buffer[18];
htonHeaderData(myPacketHeader, myPacketData, buffer);
send(sockfd, buffer, 18, 0);
...
Again, you don't have to use the header and data structs that I defined. Just use whatever your program needs. The key is that you have a conversion function that writes all of the data, at well-defined offsets, in a well-defined byte order, to a buffer, and that you pass that buffer to the send() function.
On the other side of the network connection, you will need a program to interpret the data it receives. On that side, you need to write the corresponding functions (ntohHeader, etc). Those function will memcpy the bits out of a buffer and into a local variable, which it can pass to ntohs or ntohl. I'll leave those functions for you to write.
Well, typically it would look like it's preparing that packet structure into a memory buffer (making judicious calls the the htonl family of functions).
If would then use the send, sendto, sendmsg or write functions, hopefully with a lot of care taken with the length of the buffer and good error handling/reporting.
(Or one of the Win32 apis for the send, if that is the target plateforms.)
You'll find a good presentation about all this at Beej's Guide to Network Programming.
Specifially for the byte packing part (with endian consideration), look at the serialization topic. (There's way more detail in that section than what you need for plain fixed-size integer data types.
The code would look different depending on the OS's networking library (*nix uses Berkeley sockets, Windows uses Winsock, etc.). However, you could create a struct containing all the data you wanted to send in a packet, e.g.,
typedef struct
{
short field1;
short field2;
int field3;
} HeaderStruct;
typedef struct
{
short field1;
long long field2;
} PacketDataStruct;
assuming a 32-bit int size.
Edit:
As someone kindly reminded me in the comments, don't forget about converting to and from Network Order. Networking libraries will have functions to assist with this, such as ntohs, nothl, htons, and htonl.
One simple answer is that it would be sent in the format that the receiver expects. That begs the question a bit, though. Assuming the data is a fixed size as shown and the receiving end expects, then you could use a packed (1 byte alignment) structure and store the data in each field. The reason for using 1 byte alignment is that it is typically easier to make sure both ends are expecting the same data. Without 1 byte alignment, then the structure would possibly look different based on compiler options, 32-bit versus 64-bit architecture, etc.) And, typically, it is expected that you would send the values in network byte order if the hex values are integers. You can use functions such as htons and htonl (and possibly htobe64 if available) to convert them.
Assuming that the data is in the structure with the desired byte order, then the send call may be something like this:
ret = send( socket, &mystruct, sizeof( mystruct ), 0 );
That assumes that mystruct is declared as an instance of the structure as opposed to a pointer to the structure.
I have an 8 byte message where the differing chunks of the message are mapped to datums of different types (int, bool, etc.), and they vary in bit sizes (an int value is 12 bits in the message, etc.). I want to pass only the bits a datum is concerned with, but I am not sure if there is a better way. My current thoughts is to make a bit array type with a vector back end and have a templated accessor to get the value contained within to the type specified. Although as I am typing this I am starting to think a great big union of all the possible types could be passed to each datum.
EDIT:
The messages contain varying types of data. For example, one message contains an 8-bit int and 5 1-bit bools, while another message contains a 16-bit Timestamped (my own class) and an 8-bit int.
Are the messages alway of the same format/order? Ie. 12bitsInt|8bitsChar|etc. If so a simple solution would be to set up appropriate bitmasks to grab each particular value. Ie. if the first 12 bits (low order) corresponded to an integer we could do:
__uint64 Message; // Obviously has data in it.
int IntPortion = Message & 0x00000111;
Which will copy the first 12 bits of the Message into the first 12 bits of your integer type. Set up appropriate bit masks for each chunk of the message and proceed. If the message format is not constant... well I would need you to elaborate maybe with an example message. Also the boost library has some nice bit manipulation classes:
Dynamic Bitset
Might be overkill if the format is constant though.
Have you looked at using a struct with explicit member sizes? For example, if you have a message where the field sizes are:
1st field is 12 bits
2nd field is 20 bits
3rd field is 4 bits
...
You could define a struct like:
typedef struct {
unsigned int field_1 : 12;
unsigned int field_2 : 20;
unsigned int field_3 : 4;
/* etc */
} message_t;
Assuming that you have the message in a simple char array, either copy the data into a message_t struct or cast it to a message_t* :-
char buffer[8] = /* however it's populated */
message_t* message_ptr = &buffer;
int field1 = message->field_1;
int field2 = message->field_2;