I am trying to send a set of three variables, a 64 bit integer and two 32 bit integers, using boost asio. I know how to send the data using boost asio but I am struggling to convert the three variables into something I can send using boost asio, any ideas?
The types I'm using for the variables are as follows:
boost::uint64_t
boost::uint32_t
boost::uint32_t
The purpose of this is to send the data as UDP Tracker Connect Request (Bittorrent Protocol), a description of which can be found here: http://www.bittorrent.org/beps/bep_0015.html#udp-tracker-protocol
Offset Size Name Value
0 64-bit integer connection_id 0x41727101980
8 32-bit integer action 0 // connect
12 32-bit integer transaction_id
16
Create a raw memory buffer. Use endian-aware copy functions to place the integers in the buffer. Send the buffer.
What endian does the bittorrent protocol use? It's big endian, so any solution here that relies on casting won't work on your typical consumer electronics these days, because these use little-endian format in memory. In creating your buffer to send, you therefore also have to swap the bytes.
Okay, you're trying to match an existing network protocol that has documented its expected byte offset and endianness for each field. This is one of the times where you want to use a raw buffer of uint8_t. Your code should look something like this:
// This is *not* necessarily the same as sizeof(struct containing 1 uint64_t
// and 2 uint32_t).
#define BT_CONNECT_REQUEST_WIRE_LEN 16
// ...
uint8_t send_buf[BT_CONNECT_REQUEST_WIRE_LEN];
cpu_to_be64(connection_id, &send_buf[ 0]);
cpu_to_be32(0 /*action=connect*/, &send_buf[ 8]);
cpu_to_be32(transaction_id, &send_buf[12]);
// transmit 'send_buf' using boost::asio
The cpu_to_be32 function should look like this:
void
cpu_to_be32(uint32_t n, uint8_t *dest)
{
dest[0] = uint8_t((n & 0xFF000000) >> 24);
dest[1] = uint8_t((n & 0x00FF0000) >> 16);
dest[2] = uint8_t((n & 0x0000FF00) >> 8);
dest[3] = uint8_t((n & 0x000000FF) >> 0);
}
The inverse (be32_to_cpu) and the analogue (cpu_to_be64) are left as exercises. You might also like to try your hand at writing template functions that deduce the appropriate size from their first argument, but personally I think having an explicit indication of the size in the function name makes this kind of code more self-documenting.
It's easy to convert a structure to array/vector/string which can be sent via boost::asio. For example:
struct Type
{
boost::uint64_t v1;
boost::uint32_t v2;
boost::uint32_t v3;
}
Type t;
std::string str( reinterpret_cast<char*> (&t), sizeof(t) );
I don't know architecture of your application, but it's also possible to create asio::buffer just from memory:
boost::asio::buffer( &t, sizeof(t) );
In this case you should be careful about lifetime of t.
Related
I'm trying to write a small application that will "concatenate" a bunch of integer variables of various sizes (uint8, uint16, uint32) into a 128 byte message that will be sent via UDP socket.
On the receiving side, I would like to split the message back into the individual integers and store them for further processing. For now I am assuming that endianness will not be an issue.
Could I get some hints on how to concatenate the int variables and later extract them from that 128 byte message?
So, let's say you expect to recieve data in the following order:
int32_t header; int8_t opcode; int16_t args[32]; int32_t clients[2]; ...
This is just an example, parameters could be whatever they are in your actuai task.
You can wrap those parameters into struct or class. I'd prefer a struct here because it does not seem like you really need to create a ctor, access specifiers or any other fancy stuff class can provide. So, something like that:
#pragma pack(push, 1)
struct DataFromMyHardware {
int32_t header;
int8_t opcode;
int16_t args[32];
int32_t clients[2];
...
};
#pragma pack(pop)
pragmas here used to tell compiler to not optimize placement or alignment of variables in struct, so it'll be stored in memory as is.
This way, you can use this on sender:
DataFromMyHardware buffer;
buffer.header = 0xDEADBEEF;
buffer.opcode = 42;
...
send(socket, &buffer, sizeof(buffer), 0);
and on reciever:
DataFromMyHardware buffer;
recv(socket, (void*)&buffer, sizeof(buffer), 0);
Side note: it's very likely that your device uses network byte-order, so you probably want to use nhohl/ntohs on reciever and htonl/htons on sender.
You can create a character buffer prefix that will precede each integer which will define which type of integer to read so then you can read the correct size of the integer that is defined.
for ex: Precede each integer with a character representing the integer. The first byte will contain the character which you will interpret as an, it could be any ascii character that will represent the integer.
array == [byte][byte-8bit][byte][2 byte-16bit][byte][4 byte -32bit]...
UINT8 = 'a'
UINT16 = 'b'
UINT32 = 'c' or whatever code you want the ascii to be... I used a,b,c so its a readable character you can read in the debugger
You will then have to build your array to be sent with the prefix so you know what size to do your next read.
array = [a0b00c0000a0b00c0000] and so on, you can build the array using memcpy
You can then send the whole packet of 128 bytes
Make sure you read exactly 128 bytes, and then you can deconstruct, remember that when you read you have to check the amount that was read from the socket, and continue to read until you receive the correct amount. -- sometimes a read will not return the correct amount of bytes that you will expect.
When you receive the packet, you can deconstruct the packet using the headers, and depending on the header that you receive you can remove the correct integer and size, and the headers with the correct size should be deconstructed rather easily.
Also remember that UDP is lossy so you will possibly lose packets
Now also remember that if your constructed packet does not equal exactly 128 bytes every time you will need to add another byte which will be equivalent to a integer which will tell you exactly how many bytes were sent... 122..126..127 etc, and read that as the first header on the other side.
I need to pack this pointer (which is 64 bits wide) into 4 WORD's, and then later in some other part of the code I need to extract (assemble) these words back into this pointer.
the code looks like this:
std::vector<WORD> vec;
vec.push_back( this ); // how?
later in the code;
pThis = vec.at(0); // how?
I did take a look at LOWORD/HIWORD and LOWBYTE/HIBYTE macros however I still have no idea how would I go about this.
If you ask why on earth would anyone need this, here is why:
I need to fill in creation data of DLGITEMTEMPLATEEX structure which takes WORD as last argument to specify size, data following this is where you put your data, my data is this pointer, and since I'm working on words (std::vector<WORD>) to fill the structure, the last data is 4 WORDS (64 bits) representing the pointer!
Any suggestion or sample code is welcome.
Well, the best way would be to define a struct that derive form DLGITEMTEMPLATEEX. That way, you can avoid to manually do the conversion. While it might not be defined from the standard, it will works on Windows. That kind of code if platform specific anyway!
struct MyTemplate : DLGITEMTEMPLATEEX
{
MyTemplate(MyPointerType *myVariable)
: extraCount(sizeof *this - sizeof(DLGITEMTEMPLATEEX))
, myVariable(myVariable)
{ }
MyPointerType *myVariable; // 64 if compiled for 64 bit target.
};
And when using the data, you do a static_cast to convert back to that structure.
You could add a few static_assert and assertion to validate that it works as intended.
You could use simple bitshifting to seperate the 64bit into 16bits.
#include <stdio.h>
int main(){
uint64_t bigword = 0xDEADBEEFADEADBED;
uint16_t fourth = bigword ;
uint16_t third = bigword >> 16;
uint16_t second = bigword >> 32;
uint16_t first = bigword >> 48;
printf("%llx %x %x %x %x\n",bigword,fourth,third,second,first);
return 0;
}
Then reverse the process when shifting the words back into the 64bit.
I am working on translating a system from python to c++. I need to be able to perform actions in c++ that are generally performed by using Python's struct.unpack (interpreting binary strings as numerical values). For integer values, I am able to get this to (sort of) work, using the data types in stdint.h:
struct.unpack("i", str) ==> *(int32_t*) str; //str is a char* containing the data
This works properly for little-endian binary strings, but fails on big-endian binary strings. Basically, I need an equivalent to using the > tag in struct.unpack:
struct.unpack(">i", str) ==> ???
Please note, if there is a better way to do this, I am all ears. However, I cannot use c++11, nor any 3rd party libraries other than Boost. I will also need to be able to interpret floats and doubles, as in struct.unpack(">f", str) and struct.unpack(">d", str), but I'll get to that when I solve this.
NOTE I should point out that the endianness of my machine is irrelevant in this case. I know that the bitstream I receive in my code will ALWAYS be big-endian, and that's why I need a solution that will always cover the big-endian case. The article pointed out by BoBTFish in the comments seems to offer a solution.
For 32 and 16-bit values:
This is exactly the problem you have for network data, which is big-endian. You can use the the ntohl to turn a 32-bit into host order, little-endian in your case.
The ntohl() function converts the unsigned integer netlong from network byte order to
host byte order.
int res = ntohl(*((int32_t) str)));
This will also take care of the case where your host is big-endian and won't do anything.
For 64-bit values
Non-standardly on linux/BSD you can take a look at 64 bit ntohl() in C++?, which points to htobe64
These functions convert the byte encoding of integer values from the byte order that
the current CPU (the "host") uses, to and from little-endian and big-endian byte
order.
For windows try: How do I convert between big-endian and little-endian values in C++?
Which points to _byteswap_uint64 and as well as a 16 and 32-bit solution and a gcc-specific __builtin_bswap(32/64) call.
Other Sizes
Most systems don't have values that aren't 16/32/64 bits long. At that point I might try to store it in a 64-bit value, shift it and they translate. I'd write some good tests. I suspectt is an uncommon situation and more details would help.
Unpack the string one byte at a time.
unsigned char *str;
unsigned int result;
result = *str++ << 24;
result |= *str++ << 16;
result |= *str++ << 8;
result |= *str++;
First, the cast you're doing:
char *str = ...;
int32_t i = *(int32_t*)str;
results in undefined behavior due to the strict aliasing rule (unless str is initialized with something like int32_t x; char *str = (char*)&x;). In practical terms that cast can result in an unaligned read which causes a bus error (a crash) on some platforms and slow performance on others.
Instead you should be doing something like:
int32_t i;
std::memcpy(&i, c, sizeof(i));
There are a number of functions for swapping bytes between the host's native byte ordering and a host independent ordering: ntoh*(), hton*(), where * is nothing, l, or s for the different types supported. Since different hosts may have different byte orderings then this may be what you want to use if the data you're reading uses a consistent serialized form on all platforms.
ntoh(i);
You can also manually move bytes around in str before copying it into the integer.
std::swap(str[0],str[3]);
std::swap(str[1],str[2]);
std::memcpy(&i,str,sizeof(i));
Or you can manually manipulate the integer's value using shifts and bitwise operators.
std::memcpy(&i,str,sizeof(i));
i = (i&0xFFFF0000)>>16 | (i&0x0000FFFF)<<16;
i = (i&0xFF00FF00)>>8 | (i&0x00FF00FF)<<8;
This falls in the realm of bit twiddling.
for (i=0;i<sizeof(struct foo);i++) dst[i] = src[i ^ mask];
where mask == (sizeof type -1) if the stored and native endianness differ.
With this technique one can convert a struct to bit masks:
struct foo {
byte a,b; // mask = 0,0
short e; // mask = 1,1
int g; // mask = 3,3,3,3,
double i; // mask = 7,7,7,7,7,7,7,7
} s; // notice that all units must be aligned according their native size
Again these masks can be encoded with two bits per symbol: (1<<n)-1, meaning that in 64-bit machines one can encode necessary masks of a 32 byte sized struct in a single constant (with 1,2,4 and 8 byte alignments).
unsigned int mask = 0xffffaa50; // or zero if the endianness matches
for (i=0;i<16;i++) {
dst[i]=src[i ^ ((1<<(mask & 3))-1]; mask>>=2;
}
If your as received values are truly strings, (char* or std::string) and you know their format information, sscanf(), and atoi(), well, really ato() will be your friends. They take well formatted strings and convert them per passed-in formats (kind of reverse printf).
Let's say I want to send the following data to a socket using C or C++, all in one packet:
Headers
-------
Field 1: 2 byte hex
Field 2: 2 byte hex
Field 3: 4 byte hex
Data
----
Field1 : 2 byte hex
Field1 : 8 byte hex
What would the code typically look like to create and send the packet containing all this data?
Let's suppose that your program is already organized to have the header in one struct and the data in another struct. For example, you might have these data structures:
#include <stdint.h>
struct header {
uint16_t f1;
uint16_t f2;
uint32_t f3;
};
struct data {
uint16_t pf1;
uint64_t pf2;
};
Let's call this organization "host format". It really doesn't matter to me what the host format is, as long as it is useful to the rest of your program. Let's call the format that you will pass to the send() call "network format". (I chose these names to match the htons (host-to-network-short) and htonl (host-to-network-long) names.)
Here are some conversion functions that we might find handy. Each of these converts your host format structures to a network format buffer.
#include <arpa/inet.h>
#include <string.h>
void htonHeader(struct header h, char buffer[8]) {
uint16_t u16;
uint32_t u32;
u16 = htons(h.f1);
memcpy(buffer+0, &u16, 2);
u16 = htons(h.f2);
memcpy(buffer+2, &u16, 2);
u32 = htonl(h.f3);
memcpy(buffer+4, &u32, 4);
}
void htonData(struct data d, char buffer[10]) {
uint16_t u16;
uint32_t u32;
u16 = htons(d.pf1);
memcpy(buffer+0, &u16, 2);
u32 = htonl(d.pf2>>32);
memcpy(buffer+2, &u32, 4);
u32 = htonl(d.pf2);
memcpy(buffer+6, u32, 4);
}
void htonHeaderData(struct header h, struct data d, char buffer[18]) {
htonHeader(h, buffer+0);
htonData(d, buffer+8);
}
To send your data, do this:
...
char buffer[18];
htonHeaderData(myPacketHeader, myPacketData, buffer);
send(sockfd, buffer, 18, 0);
...
Again, you don't have to use the header and data structs that I defined. Just use whatever your program needs. The key is that you have a conversion function that writes all of the data, at well-defined offsets, in a well-defined byte order, to a buffer, and that you pass that buffer to the send() function.
On the other side of the network connection, you will need a program to interpret the data it receives. On that side, you need to write the corresponding functions (ntohHeader, etc). Those function will memcpy the bits out of a buffer and into a local variable, which it can pass to ntohs or ntohl. I'll leave those functions for you to write.
Well, typically it would look like it's preparing that packet structure into a memory buffer (making judicious calls the the htonl family of functions).
If would then use the send, sendto, sendmsg or write functions, hopefully with a lot of care taken with the length of the buffer and good error handling/reporting.
(Or one of the Win32 apis for the send, if that is the target plateforms.)
You'll find a good presentation about all this at Beej's Guide to Network Programming.
Specifially for the byte packing part (with endian consideration), look at the serialization topic. (There's way more detail in that section than what you need for plain fixed-size integer data types.
The code would look different depending on the OS's networking library (*nix uses Berkeley sockets, Windows uses Winsock, etc.). However, you could create a struct containing all the data you wanted to send in a packet, e.g.,
typedef struct
{
short field1;
short field2;
int field3;
} HeaderStruct;
typedef struct
{
short field1;
long long field2;
} PacketDataStruct;
assuming a 32-bit int size.
Edit:
As someone kindly reminded me in the comments, don't forget about converting to and from Network Order. Networking libraries will have functions to assist with this, such as ntohs, nothl, htons, and htonl.
One simple answer is that it would be sent in the format that the receiver expects. That begs the question a bit, though. Assuming the data is a fixed size as shown and the receiving end expects, then you could use a packed (1 byte alignment) structure and store the data in each field. The reason for using 1 byte alignment is that it is typically easier to make sure both ends are expecting the same data. Without 1 byte alignment, then the structure would possibly look different based on compiler options, 32-bit versus 64-bit architecture, etc.) And, typically, it is expected that you would send the values in network byte order if the hex values are integers. You can use functions such as htons and htonl (and possibly htobe64 if available) to convert them.
Assuming that the data is in the structure with the desired byte order, then the send call may be something like this:
ret = send( socket, &mystruct, sizeof( mystruct ), 0 );
That assumes that mystruct is declared as an instance of the structure as opposed to a pointer to the structure.
This is related to my question asked here today on SO. Is there a better way to build a packet to send over serial rather than doing this:
unsigned char buff[255];
buff[0] = 0x02
buff[1] = 0x01
buff[2] = 0x03
WriteFile(.., buff,3, &dwBytesWrite,..);
Note: I have about twenty commands to send, so if there was a better way to send these bytes to the serial device in a more concise manner rather than having to specify each byte, it would be great. Each byte is hexadecimal, with the last byte being the checksum. I should clarify that I know I will have to specify each byte to build the commands, but is there a better way than having to specify each array position?
You can initialize static buffers like so:
const unsigned char command[] = {0x13, 0x37, 0xf0, 0x0d};
You could even use these to initialize non-const buffers and then replace only changing bytes by index.
Not sure what you're asking. If you ask about the problem of setting the byte one by one and messing up the data, usually this is doen with a packed struct with members having meaningful names. Like:
#pragma push(pack)
#pragma pack(1)
struct FooHeader {
uint someField;
byte someFlag;
dword someStatus;
};
#pragma pack(pop)
FooHeader hdr;
hdr.someField = 2;
hdr.someFlag = 3;
hdr.someStatus = 4;
WriteFile(..., sizeof(hdr), &hdr);
Is there a better way to build a packet than assembling it byte by byte?
Yes, but it will require some thought and some careful engineering. Many of the other answers tell you other mechanisms by which you can put together a sequence of bytes in C++. But I suggest you design an abstraction that represents a part of a packet:
class PacketField {
void add_to_packet(Packet p);
};
Then you can define various subclasses:
Add a single byte to the packet
Add a 16-bit integer in big-endian order. Another for little-endian. Other widths besides 16.
Add a string to the packet; code the string by inserting the length and then the bytes.
You also can define a higher-order version:
PacketField sequence(PacketField first, PacketField second);
Returns a field that consists of the two arguments in sequence. If you like operator overloading you could overload this as + or <<.
Your underlying Packet abstraction will just be an extensible sequence of bytes (dynamic array) with some kind of write method.
If you wind up programming a lot of network protocols, you'll find this sort of design pays off big time.
Edit: The point of the PacketField class is composability and reuse:
By composing packet fields you can create more complex packet fields. For example, you could define "add a TCP header" as a function from PacketFields to PacketFields.
With luck you build up a library of PacketFields that are specific to your application or protocol family or whatever. Then you reuse the fields in the library.
You can create subclasses of PacketField that take extra parameters.
It's quite possibly that you can do something equally nice without having to have this extra level of indirection; I'm recommending it because I've seen it used effectively in other applications. You are decoupling the knowledge of how to build a packet (which can be applied to any packet, any time) from the act of actually building a particular packet. Separating concerns like this can help reuse.
Yes, there is a better method. Have your classes read from and write to a packed buffer. You could even implement this as an interface. Templates would help to.
An example of writing:
template <typename Member_Type>
void Store_Value_In_Buffer(const Member_Type&, member,
unsigned char *& p_buffer)
{
*((Member_Type *)(p_buffer)) = member;
p_buffer += sizeof(Member_Type);
return;
}
struct My_Class
{
unsigned int datum;
void store_to_buffer(unsigned char *& p_buffer)
{
Store_Value_In_Buffer(datum, buffer);
return;
}
};
//...
unsigned char buffer[256];
unsigned char * p_buffer(buffer);
MyClass object;
object.datum = 5;
object.store_to_buffer(p_buffer);
std::cout.write(p_buffer, 256);
Part of the interface is also to query the objects for the size that they would occupy in the buffer, say a method size_in_buffer. This is left as an exercise for the reader. :-)
There is a much better way, which is using structs to set the structures. This is usually how network packets are built on a low level.
For example, say you have packets which have an id, length, flag byte, and data, you'd do something like this:
struct packet_header {
int id;
byte length;
byte flags;
};
byte my_packet[] = new byte[100];
packet_header *header = &my_packet;
header->id = 20;
header->length = 10; // This can be set automatically by a function, maybe?
// etc.
header++; // Header now points to the data section.
Do note that you're going to have to make sure that the structures are "packed", i.e. when you write byte length, it really takes up a byte. Usually, you'd achieve this using something like #pragma pack or similar (you'll have to read about your compiler's pragma settings).
Also, note that you should probably use functions to do common operations. For example, create a function which gets as input the size, data to send, and other information, and fills out the packet header and data for you. This way, you can perform calculations about the actual size you want to write in the length field, you can calculate the CRC inside the function, etc.
Edit: This is a C-centric way of doing things, which is the style of a lot of networking code. A more C++-centric (object oriented) approach could also work, but I'm less familiar with them.
const char *c = "\x02\x02\x03";