I'm using WinSock to send UDP packets to the server, I need to send the data in big endian. I'm not sure how to convert the byte order of my structure before sending.
I have a struct like this:
struct ConnectIn
{
std::int64_t ConnectionID = 0x41727101980;
std::int32_t Action = 0;
std::int32_t TransactionID;
ConnectIn(std::int32_t transactionID)
{
TransactionID = transactionID;
}
};
And at the moment I'm sending like this:
ConnectIn msg(123);
int len = sizeof(msg);
int bytesSent = sendto(s, (char*)&msg, len, 0, (SOCKADDR*)&dest, sizeof(address));
How can I convert the byte order of msg to big endian before sending?
If you're curious, the data I'm sending is for the Bit Torrent UDP tracker protocol.
If you want to do this manually then what you do is swap each member individually. You convert the members from the host computer's byte ordering to the network's byte ordering. On Win32 htonll() is for 64-bit integers and htonl() is for 32-bit integers:
#include <Winsock2.h>
ConnectIn msg(123);
msg.ConnectionID = htonll(msg.ConnectionID);
msg.Action = htonl(msg.Action);
msg.TransactionID= htonl(msg.TransactionID);
Then you might also want to send the members individually, to avoid relying on the host system's struct layout. The Windows ABI doesn't insert any padding in this struct, but perhaps for some other struct you use it does. So here's the basic idea:
char buf[sizeof msg.ConnectionID + sizeof msg.Action + sizeof msg.TransactionID];
char *bufi = buf;
std::memcpy(bufi, &msg.ConnectionID, sizeof msg.ConnectionID);
bufi += sizeof msg.ConnectionID;
std::memcpy(bufi, &msg.Action, sizeof msg.Action);
bufi += sizeof msg.Action;
std::memcpy(bufi, &msg.TransactionID, sizeof msg.TransactionID);
bufi += sizeof msg.TransactionID;
int len = sizeof buf;
int bytesSent = sendto(s, buf, len, 0, (SOCKADDR*)&dest, sizeof(address));
Then on the receiving side you use the appropriate ntoh*() functions for 64-bit and 32-bit types to convert from the network's byte ordering to the receiving host's byte ordering.
Yes, the Network Byte Order (NBO) is Big Endian and so you need to find a way to send that structure on the web.
What you're currently doing won't work: you're sending the whole struct but the receiver may have a different endianness, padding and so on.
The easiest options are:
Sending each field with a protocol-defined layout
Third part libraries which handle serialization: Google Protobuf is one of the most common ones.
For the first option, there're some functions which take care of that in the Winsock2 library. These are:
(WSA)ntoht (Network to Host t, where t can be short and unsigned)
(WSA)htont (Host to Network t, where t can be short and unsigned)
WSA functions are a little bit different and Windows-only.
The Network Programming Guide
Winsock Reference
One option is converting each of the numbers individually
For GCC:
int32_t __builtin_bswap32 (int32_t x)
int64_t __builtin_bswap64 (int64_t x)
For MSVC:
unsigned short _byteswap_ushort(unsigned short value);
unsigned long _byteswap_ulong(unsigned long value);
unsigned __int64 _byteswap_uint64(unsigned __int64 value);
Related
I am doing a Header for an UDP socket which have a restrictions using bytes.
| Packet ID (1 byte) | Packet Size (2 bytes) | Subpacket ID (1 Byte) | etc
I did an struct for store this kind of attributes like:
typedef struct WHEATHER_STRUCT
{
unsigned char packetID[1];
unsigned char packetSize[2];
unsigned char subPacketID[1];
unsigned char subPacketOffset[2];
...
} wheather_struct;
I initialized this struct using new and I updated the values. The question is about if I want to use only 2 bytes in Packet Size attribute. What of these two forms that I wrote below is the correct one?
*weather_struct->packetSize = '50';
or
*weather_struct->packetSize = 50;
If you can use C++11 and gcc (or clang) then I would do this:
typedef struct WHEATHER_STRUCT
{
uint8_t packetID;
uint16_t packetSize;
uint8_t subPacketID;
uint16_t subPacketOffset;
// ...
} __attribute__((packed)) wheather_struct;
If you can't use C++11 then you can use unsigned char and unsigned short instead.
If you're using Visual C then you can do:
#pragma pack (push, 1)
typedef struct ...
#pragma (pop)
Beware also byte ordering issues, depending on what architectures you need to support. You can use htons() and ntohs() to overcome this problem.
Live demo at Wandbox
Packing and unpacking data from IP packets is a problem as old as the internet itself (indeed, older).
Different machine architectures have different layouts for representing integers, which can cause problems when communicating between machines.
For this reason, the IP stack standardises on encoding integers in 'network byte order' (which basically means most significant byte first).
Standard functions exist to convert values in network byte order to native types and vice versa. I urge you to consider using these as your code will then be more portable.
Furthermore, it makes sense to abstract data representations from the program's point of view. c++ compilers can perform the conversions very efficiently.
Example:
#include <arpa/inet.h>
#include <cstring>
#include <cstdint>
typedef struct WEATHER_STRUCT
{
std::int8_t packetID;
std::uint16_t packetSize;
std::uint8_t subPacketID;
std::uint16_t subPacketOffset;
} weather_struct;
const std::int8_t* populate(weather_struct& target, const std::int8_t* source)
{
auto get16 = [&source]
{
std::uint16_t buf16;
std::memcpy(&buf16, source, 2);
source += 2;
return ntohs(buf16);
};
target.packetID = *source++;
target.packetSize = get16();
target.subPacketID = *source++;
target.subPacketOffset = get16();
return source;
}
uint8_t* serialise(uint8_t* target, weather_struct const& source)
{
auto write16 = [&target](std::uint16_t val)
{
val = ntohs(val);
std::memcpy(target, &val, 2);
target += 2;
};
*target++ = source.packetID;
write16(source.packetSize);
*target++ = source.subPacketID;
write16(source.subPacketOffset);
return target;
}
https://linux.die.net/man/3/htons
here's an link to a c++17 version of the above:
https://godbolt.org/z/oRASjI
A further note on conversion costs:
Data arriving into or leaving your program is an event that happens once per payload. Suffering a conversion cost here incurs a negligible penalty.
Once the data has arrived in your program, or before it leaves, it may be manipulated many times by your code.
Some processors architectures suffer huge performance penalties during data access if data is not aligned on natural word boundaries. This is why attributes such as packed exist - the compiler is doing all it can to avoid misaligned data. Using a packed attribute is tantamount to deliberately telling the compiler to produce very suboptimal code.
For this reason, I would recommend not using packed structures (e.g. __attribute__((packed)) etc) for data that will be referred to by program logic.
Compared to RAM, networks are many orders of magnitude slower. A minuscule performance hit (literally nanoseconds) at the point of encoding or decoding a network packet is inconsequential compared to the cost of actually transmitting it.
Packing structures can cause horrible performance issues in program code and often leads to portability headaches.
Neither is correct, you need to treat the two bytes as a single 16-bit number. You probably also need to take into account the different endianness of the network stream to your processor architecture (depending on the protocol, most are big endian).
The correct code would therefore be:
*((uint16_t*)weather_struct->packetSize) = htons(50);
It would be simpler if packetSize were uint16_t to start with:
weather_struct->packetSize = htons(50);
I'm writing a networking application using sockets in c++ so lets jump straight to the problem :
i'm storing my data as an array of int16_ts (the choice of int16_t being for consistency accross different platforms) , as we know each of these int16_ts would be two consecutive bytes in memory. i want to store each of those bytes in a char so that each int16 would be translated to exactly two bytes and eventually send the entire char* over the socket
please notice that i'm not looking for something such as std::to_string cause i want each int16_t to occupy exactly two bytes.
any help is appreciated !
You would need to copy each int16_t one at a time to the char * buffer, calling htons() on each one to translate the bytes into network byte order. Then on the receiving side, you would call ntohs() to convert back.
int send_array(int16_t *myarray, int len, int socket) {
char buf[1000];
int16_t *p;
int i;
p = (int16_t *)buf;
for (i=0;i<len;i++) {
p[i] = htons(myarray[i]);
}
return send(socket,buf,len*sizeof(int16_t),0);
}
C++ has a new type, char16_t, that's designed to hold UTF-16 characters. If you mean you want to send the 16-bit hints over the wire one byte at a time, convert each one to network byte order with htons(), store them in a new array of uint16_t, then send that array over the socket. If you want to address the array of shorts as an array of bytes, you can do that either through a union, with a pointer cast, or with a reference to an array. Example: char* const p=reinterpret_cast<char*>(&shorts[0]);.
I am trying to send a set of three variables, a 64 bit integer and two 32 bit integers, using boost asio. I know how to send the data using boost asio but I am struggling to convert the three variables into something I can send using boost asio, any ideas?
The types I'm using for the variables are as follows:
boost::uint64_t
boost::uint32_t
boost::uint32_t
The purpose of this is to send the data as UDP Tracker Connect Request (Bittorrent Protocol), a description of which can be found here: http://www.bittorrent.org/beps/bep_0015.html#udp-tracker-protocol
Offset Size Name Value
0 64-bit integer connection_id 0x41727101980
8 32-bit integer action 0 // connect
12 32-bit integer transaction_id
16
Create a raw memory buffer. Use endian-aware copy functions to place the integers in the buffer. Send the buffer.
What endian does the bittorrent protocol use? It's big endian, so any solution here that relies on casting won't work on your typical consumer electronics these days, because these use little-endian format in memory. In creating your buffer to send, you therefore also have to swap the bytes.
Okay, you're trying to match an existing network protocol that has documented its expected byte offset and endianness for each field. This is one of the times where you want to use a raw buffer of uint8_t. Your code should look something like this:
// This is *not* necessarily the same as sizeof(struct containing 1 uint64_t
// and 2 uint32_t).
#define BT_CONNECT_REQUEST_WIRE_LEN 16
// ...
uint8_t send_buf[BT_CONNECT_REQUEST_WIRE_LEN];
cpu_to_be64(connection_id, &send_buf[ 0]);
cpu_to_be32(0 /*action=connect*/, &send_buf[ 8]);
cpu_to_be32(transaction_id, &send_buf[12]);
// transmit 'send_buf' using boost::asio
The cpu_to_be32 function should look like this:
void
cpu_to_be32(uint32_t n, uint8_t *dest)
{
dest[0] = uint8_t((n & 0xFF000000) >> 24);
dest[1] = uint8_t((n & 0x00FF0000) >> 16);
dest[2] = uint8_t((n & 0x0000FF00) >> 8);
dest[3] = uint8_t((n & 0x000000FF) >> 0);
}
The inverse (be32_to_cpu) and the analogue (cpu_to_be64) are left as exercises. You might also like to try your hand at writing template functions that deduce the appropriate size from their first argument, but personally I think having an explicit indication of the size in the function name makes this kind of code more self-documenting.
It's easy to convert a structure to array/vector/string which can be sent via boost::asio. For example:
struct Type
{
boost::uint64_t v1;
boost::uint32_t v2;
boost::uint32_t v3;
}
Type t;
std::string str( reinterpret_cast<char*> (&t), sizeof(t) );
I don't know architecture of your application, but it's also possible to create asio::buffer just from memory:
boost::asio::buffer( &t, sizeof(t) );
In this case you should be careful about lifetime of t.
I am working on a ping tool and I am consistently getting an access violation around my sending buffer while calculating the ICMP checksum when using a packet size of 45041 or greater (including ICMP header). Any packet with size 45040 or below throws no error and properly transmits with correct checksum. Abbreviated code is below; the access violation occurs when dereferencing the buffer within the while loop in the checksum function on the first iteration.
typedef struct ICMPHeader
{
BYTE type; // ICMP packet type
BYTE code; // Type sub code
USHORT checksum;
USHORT id;
USHORT seq;
} ICMPHeader;
typedef struct echoRequest
{
ICMPHeader icmpHead;
char *data;
} EchoRequest;
// ...
EchoRequest *sendBuffer = new EchoRequest();
sendBuffer->data = new char[packetSize];
memset((void *)sendBuffer->data, 0xfa, packetSize);
sendBuffer->icmpHead.checksum = ipChecksum((USHORT *)sendBuffer,
packetSize + sizeof(sendBuffer->icmpHead));
// ...
// checksum function
USHORT ipChecksum(USHORT *buffer, unsigned long size)
{
unsigned long cksum = 0;
while (size > 1)
{
cksum += *buffer++;
size -= sizeof(USHORT);
}
if (size)
cksum += *(UCHAR *)buffer;
cksum = (cksum >> 16) + (cksum & 0xffff);
cksum += (cksum >> 16);
return (USHORT)(~cksum);
}
Any ideas as to why this is happening?
Exact error wording: Unhandled exception at 0x009C2582 in PingProject.exe: 0xC0000005: Access violation reading location 0x004D5000.
Using Visual Studio Professional 2012 with platform toolset v100 for .NET 4.0
Your ipChecksum function expects a pointer to the data it's supposed to checksum, not a pointer to a structure that contains a pointer to the data to checksum. So first it checksums icmpHead, which is good. But then it checksums the pointer to data, which makes no sense. And then it checksums off the end of the EchoRequest structure.
If you want this code to be interpreted as c++ by the readers you need to fix some things.
memset really?
use reinterpret_cast to convert one pointer type to another.
it's generally considered a much better practice to use size_t instead of unsigned long
use smart pointers instead.
use static_cast to convert ulong to ushort.
USHORT is not guaranteed to be 16 bits. Use a different type instead.
Edit: You're waaaay above MTU. Keep your packets under 1k bytes. IEEE 802.3 expects 1492 though this value might vary.
Let's say I want to send the following data to a socket using C or C++, all in one packet:
Headers
-------
Field 1: 2 byte hex
Field 2: 2 byte hex
Field 3: 4 byte hex
Data
----
Field1 : 2 byte hex
Field1 : 8 byte hex
What would the code typically look like to create and send the packet containing all this data?
Let's suppose that your program is already organized to have the header in one struct and the data in another struct. For example, you might have these data structures:
#include <stdint.h>
struct header {
uint16_t f1;
uint16_t f2;
uint32_t f3;
};
struct data {
uint16_t pf1;
uint64_t pf2;
};
Let's call this organization "host format". It really doesn't matter to me what the host format is, as long as it is useful to the rest of your program. Let's call the format that you will pass to the send() call "network format". (I chose these names to match the htons (host-to-network-short) and htonl (host-to-network-long) names.)
Here are some conversion functions that we might find handy. Each of these converts your host format structures to a network format buffer.
#include <arpa/inet.h>
#include <string.h>
void htonHeader(struct header h, char buffer[8]) {
uint16_t u16;
uint32_t u32;
u16 = htons(h.f1);
memcpy(buffer+0, &u16, 2);
u16 = htons(h.f2);
memcpy(buffer+2, &u16, 2);
u32 = htonl(h.f3);
memcpy(buffer+4, &u32, 4);
}
void htonData(struct data d, char buffer[10]) {
uint16_t u16;
uint32_t u32;
u16 = htons(d.pf1);
memcpy(buffer+0, &u16, 2);
u32 = htonl(d.pf2>>32);
memcpy(buffer+2, &u32, 4);
u32 = htonl(d.pf2);
memcpy(buffer+6, u32, 4);
}
void htonHeaderData(struct header h, struct data d, char buffer[18]) {
htonHeader(h, buffer+0);
htonData(d, buffer+8);
}
To send your data, do this:
...
char buffer[18];
htonHeaderData(myPacketHeader, myPacketData, buffer);
send(sockfd, buffer, 18, 0);
...
Again, you don't have to use the header and data structs that I defined. Just use whatever your program needs. The key is that you have a conversion function that writes all of the data, at well-defined offsets, in a well-defined byte order, to a buffer, and that you pass that buffer to the send() function.
On the other side of the network connection, you will need a program to interpret the data it receives. On that side, you need to write the corresponding functions (ntohHeader, etc). Those function will memcpy the bits out of a buffer and into a local variable, which it can pass to ntohs or ntohl. I'll leave those functions for you to write.
Well, typically it would look like it's preparing that packet structure into a memory buffer (making judicious calls the the htonl family of functions).
If would then use the send, sendto, sendmsg or write functions, hopefully with a lot of care taken with the length of the buffer and good error handling/reporting.
(Or one of the Win32 apis for the send, if that is the target plateforms.)
You'll find a good presentation about all this at Beej's Guide to Network Programming.
Specifially for the byte packing part (with endian consideration), look at the serialization topic. (There's way more detail in that section than what you need for plain fixed-size integer data types.
The code would look different depending on the OS's networking library (*nix uses Berkeley sockets, Windows uses Winsock, etc.). However, you could create a struct containing all the data you wanted to send in a packet, e.g.,
typedef struct
{
short field1;
short field2;
int field3;
} HeaderStruct;
typedef struct
{
short field1;
long long field2;
} PacketDataStruct;
assuming a 32-bit int size.
Edit:
As someone kindly reminded me in the comments, don't forget about converting to and from Network Order. Networking libraries will have functions to assist with this, such as ntohs, nothl, htons, and htonl.
One simple answer is that it would be sent in the format that the receiver expects. That begs the question a bit, though. Assuming the data is a fixed size as shown and the receiving end expects, then you could use a packed (1 byte alignment) structure and store the data in each field. The reason for using 1 byte alignment is that it is typically easier to make sure both ends are expecting the same data. Without 1 byte alignment, then the structure would possibly look different based on compiler options, 32-bit versus 64-bit architecture, etc.) And, typically, it is expected that you would send the values in network byte order if the hex values are integers. You can use functions such as htons and htonl (and possibly htobe64 if available) to convert them.
Assuming that the data is in the structure with the desired byte order, then the send call may be something like this:
ret = send( socket, &mystruct, sizeof( mystruct ), 0 );
That assumes that mystruct is declared as an instance of the structure as opposed to a pointer to the structure.