Read "varint" from linux sockets - c++

I need to read a VarInts from linux sockets in C/C++. Any library, idea or something?
I tried reading and casting char to bool[8] to try without success to read a VarInt...
Also, this is for compatibility with new Minecraft 1.7.2 communication protocol, so, the documentation of the protocol may also help.
Let me explain my project: I'm making a Minecraft server software to run in my VPS (because java is too slow...) and I got stuck with the protocol. One thread waits for the connections and when it has a new connection, it creates a new Client object and starts the Client thread that starts communicating with the client.
I think that there is no need to show code. In case I'm wrong, tell me and I'll edit with some code.

First off, note that varints are sent as actual bytes, not strings of the characters 1 and 0.
For an unsigned varint, I believe the following will decode it for you, assuming you've got the varint data in a buffer pointed to by data. This example function returns the number of bytes decoded in the reference argument int decoded_bytes.
uint64_t decode_unsigned_varint( const uint8_t *const data, int &decoded_bytes )
{
int i = 0;
uint64_t decoded_value = 0;
int shift_amount = 0;
do
{
decoded_value |= (uint64_t)(data[i] & 0x7F) << shift_amount;
shift_amount += 7;
} while ( (data[i++] & 0x80) != 0 );
decoded_bytes = i;
return decoded_value;
}
To decode a signed varint, you can use this second function that calls the first:
int64_t decode_signed_varint( const uint8_t *const data, int &decoded_bytes )
{
uint64_t unsigned_value = decode_unsigned_varint(data, decoded_bytes);
return (int64_t)( unsigned_value & 1 ? ~(unsigned_value >> 1)
: (unsigned_value >> 1) );
}
I believe both of these functions are correct. I did some basic testing with the code below to verify a couple datapoints from the Google page. The output is correct.
#include <stdint.h>
#include <iostream>
uint64_t decode_unsigned_varint( const uint8_t *const data, int &decoded_bytes )
{
int i = 0;
uint64_t decoded_value = 0;
int shift_amount = 0;
do
{
decoded_value |= (uint64_t)(data[i] & 0x7F) << shift_amount;
shift_amount += 7;
} while ( (data[i++] & 0x80) != 0 );
decoded_bytes = i;
return decoded_value;
}
int64_t decode_signed_varint( const uint8_t *const data, int &decoded_bytes )
{
uint64_t unsigned_value = decode_unsigned_varint(data, decoded_bytes);
return (int64_t)( unsigned_value & 1 ? ~(unsigned_value >> 1)
: (unsigned_value >> 1) );
}
uint8_t ex_p300[] = { 0xAC, 0x02 };
uint8_t ex_n1 [] = { 0x01 };
using namespace std;
int main()
{
int decoded_bytes_p300;
uint64_t p300;
p300 = decode_unsigned_varint( ex_p300, decoded_bytes_p300 );
int decoded_bytes_n1;
int64_t n1;
n1 = decode_signed_varint( ex_n1, decoded_bytes_n1 );
cout << "p300 = " << p300
<< " decoded_bytes_p300 = " << decoded_bytes_p300 << endl;
cout << "n1 = " << n1
<< " decoded_bytes_n1 = " << decoded_bytes_n1 << endl;
return 0;
}
To encode varints, you could use the following functions. Note that the buffer uint8_t *const data should have room for at least 10 bytes, as the largest varint is 10 bytes long.
#include
// Encode an unsigned 64-bit varint. Returns number of encoded bytes.
// 'buffer' must have room for up to 10 bytes.
int encode_unsigned_varint(uint8_t *const buffer, uint64_t value)
{
int encoded = 0;
do
{
uint8_t next_byte = value & 0x7F;
value >>= 7;
if (value)
next_byte |= 0x80;
buffer[encoded++] = next_byte;
} while (value);
return encoded;
}
// Encode a signed 64-bit varint. Works by first zig-zag transforming
// signed value into an unsigned value, and then reusing the unsigned
// encoder. 'buffer' must have room for up to 10 bytes.
int encode_signed_varint(uint8_t *const buffer, int64_t value)
{
uint64_t uvalue;
uvalue = uint64_t( value < 0 ? ~(value << 1) : (value << 1) );
return encode_unsigned_varint( buffer, uvalue );
}

Related

Data transfer over sockets[TCP] how to pack multiple integer in c/c++ and transfer the data with send() recv()?

I'm making a small client/server based game, on linux in c/c++ and I need to send the player turn to the server.
Here is my problem.
I want to send two integers to the server and sometimes it works perfectly, but sometimes the server receives both integer in the first recv() and its stuck.
I know that the best way is to package the messages.
The problem is I don't know how the syntax should look like.
In theory--> the player input would be like an int column = 4 and a second int row = 1 and I package the message as 4|1 or something like this. Then I send from client to server and encode it on the server.
An example would be great or maybe some advice how stuff like this is handled probably.
I'm still very new to socket programming.
Here is how my function looks like:
Client:
#define BUFFER 512
void send_turn_to_server(int sock, int row, int column)
{
// sends row to server from player turn
char char_row[BUFFER];
sprintf(char_row, "%d", row);
char *message_from_client = char_row;
int len, bytes_sent_row;
len = strlen(message_from_client);
if (sendall(sock, message_from_client, &len) == -1)
{
perror("sendall");
printf("We only sent %d bytes because of the error!\n", len);
}
char char_column[BUFFER];
int bytes_sent_column;
//sends column from player turn
//sprintf converts the int to char
sprintf(char_column, "%d", column);
char *column_from_client = char_column;
len = strlen(column_from_client);
if (sendall(sock, column_from_client, &len) == -1)
{
perror("sendall");
printf("We only sent %d bytes because of the error!\n", len);
}
cout << "send_turn_to_server_complete" << endl;
}
Here I use a function from Beej's Guide to Network Programming, so I can be sure the whole buffer is sent.
Client:
int sendall(int s, char *buf, int *len)
{
int total = 0; // how many bytes we've sent
int bytesleft = *len; // how many we have left to send
int n;
while (total < *len)
{
n = send(s, buf + total, bytesleft, 0);
if (n == -1)
{
break;
}
total += n;
bytesleft -= n;
}
*len = total; // return number actually sent here
return n == -1 ? -1 : 0; // return -1 on failure, 0 on success
}
Server:
int receive_player_turn(int sock, int &int_row, int &int_column)
{
int byte_count;
char buf[BUFFER];
byte_count = recv(sock, buf, sizeof buf, 0);
cout << "The row from player: " << buf << endl;
//The C library function int atoi(const char *str) converts the string argument str to an integer (type int).
int_row = atoi(buf);
//cleans the buffer
bzero(buf, sizeof(buf));
byte_count = recv(sock, buf, sizeof buf, 0);
cout << buf << endl;
cout << "The column from player: " << buf << endl;
//converts the char string to an int
int_column = atoi(buf);
cout << endl
<< "receive player turn worked" << endl
<< "players turn was in the row " << int_row << " and in the column " << int_column + 1 << endl;
return int_row, int_column;
}
output correct from server:
Player connected: SchleichsSalaticus
The row from player: 7
4
The column from player: 4
receive player turn worked
players turn was in the row 7 and in the column 5
7 4
output wrong from server:
Player connected: SchleichsSalaticus
The row from player: 74
The issue is that TCP is a continuous stream, with no concept of the start or end of a ”message” because it is not message-based.
Most times, people use a very simple ”framing protocol” whereby you always send a 4-byte header on every transfer which tells the recipient how many bytes to read, then you send that many bytes as your message.
Use htonl() to send the 4-byte header in network byte order then you will be interoperable. There is a very similar example here.
One possible solution could be defining a format for the message that client send to server. For example you could define a protocol as follow:
[4 bytes length of your message][2 bytes for first player][2 bytes for second one] and in server side you should at first in rcv function get 4 bytes and extract the length of the arrived message and based on the receiving length(L) call again the rcv function with size L after that you should parse received messaged and extract the turn of each players.
If all your messages are expected to be of same length, then you do not need a message header. Something like that given below should work fine. In general you should be prepared to receive less or more than your expected message, as well as for one message to be split across many receives.
Also, I would recommend one function that receives bytes making no assumption about what they mean, and another that interprets them. Then the first one can be applied more broadly.
Treat the following only as pseudo code. not tested.
// use a buffer length double of MESSAGE_LENGTH.
static int offset = 0; // not thread safe.
// loop to receive a message.
while(offset < MESSAGE_LENGTH) {
byte_count = recv(sock, &buf[offset], (sizeof(buf)-offset), 0);
if(byte_count > 0) {
offset += byte_count;
}
else {
// add error handling here. close socket.
break out of loop
}
}
// process buf here, but do not clear it.
// received message always starts at buf[0].
if(no receive error above) {
process_received_message(buf); //
}
// move part of next message (if any) to start of buffer.
if(offset > MESSAGE_LENGTH) {
// copy the start of next message to start of buffer.
// and remember the new offset to avoid overwriting them.
char* pSrc = &buf[MESSAGE_LENGTH];
char* pSrcEnd = &buf[offset];
char* pDest = buf;
while(pSrc < pSrcEnd){
*pDest++ = *pSrc++;
} //or memcpy.
offset -= MESSAGE_LENGTH;
}
else {
offset = 0;
}
On many hardware architectures, integers and other types have alignment requirements. The compiler normally takes care of this, but when in a buffer, unaligned accesses can be an issue. Furthermore, the server and the client might not use the same byte order.
Here is a set of inline helper functions you can use to pack and unpack integer types to/from a buffer:
/* SPDX-License-Identifier: CC0-1.0 */
#ifndef PACKING_H
#define PACKING_H
#include <stdint.h>
/* Packing and unpacking unsigned and signed integers in
little-endian byte order.
Works on all architectures and OSes when compiled
using a standards-conforming C implementation, C99 or later.
*/
static inline void pack_u8(unsigned char *dst, uint8_t val)
{
dst[0] = val & 255;
}
static inline void pack_u16(unsigned char *dst, uint16_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
}
static inline void pack_u24(unsigned char *dst, uint32_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
dst[2] = (val >> 16) & 255;
}
static inline void pack_u32(unsigned char *dst, uint32_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
dst[2] = (val >> 16) & 255;
dst[3] = (val >> 24) & 255;
}
static inline void pack_u40(unsigned char *dst, uint64_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
dst[2] = (val >> 16) & 255;
dst[3] = (val >> 24) & 255;
dst[4] = (val >> 32) & 255;
}
static inline void pack_u48(unsigned char *dst, uint64_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
dst[2] = (val >> 16) & 255;
dst[3] = (val >> 24) & 255;
dst[4] = (val >> 32) & 255;
dst[5] = (val >> 40) & 255;
}
static inline void pack_u56(unsigned char *dst, uint64_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
dst[2] = (val >> 16) & 255;
dst[3] = (val >> 24) & 255;
dst[4] = (val >> 32) & 255;
dst[5] = (val >> 40) & 255;
dst[6] = (val >> 48) & 255;
}
static inline void pack_u64(unsigned char *dst, uint64_t val)
{
dst[0] = val & 255;
dst[1] = (val >> 8) & 255;
dst[2] = (val >> 16) & 255;
dst[3] = (val >> 24) & 255;
dst[4] = (val >> 32) & 255;
dst[5] = (val >> 40) & 255;
dst[6] = (val >> 48) & 255;
dst[7] = (val >> 56) & 255;
}
static inline void pack_i8(unsigned char *dst, int8_t val)
{
pack_u8((uint8_t)val);
}
static inline void pack_i16(unsigned char *dst, int16_t val)
{
pack_u16((uint16_t)val);
}
static inline void pack_i24(unsigned char *dst, int32_t val)
{
pack_u24((uint32_t)val);
}
static inline void pack_i32(unsigned char *dst, int32_t val)
{
pack_u32((uint32_t)val);
}
static inline void pack_i40(unsigned char *dst, int64_t val)
{
pack_u40((uint64_t)val);
}
static inline void pack_i48(unsigned char *dst, int64_t val)
{
pack_u48((uint64_t)val);
}
static inline void pack_i56(unsigned char *dst, int64_t val)
{
pack_u56((uint64_t)val);
}
static inline void pack_i64(unsigned char *dst, int64_t val)
{
pack_u64((uint64_t)val);
}
static inline uint8_t unpack_u8(const unsigned char *src)
{
return (uint_fast8_t)(src[0] & 255);
}
static inline uint16_t unpack_u16(const unsigned char *src)
{
return (uint_fast16_t)(src[0] & 255)
| ((uint_fast16_t)(src[1] & 255) << 8);
}
static inline uint32_t unpack_u24(const unsigned char *src)
{
return (uint_fast32_t)(src[0] & 255)
| ((uint_fast32_t)(src[1] & 255) << 8)
| ((uint_fast32_t)(src[2] & 255) << 16);
}
static inline uint32_t unpack_u32(const unsigned char *src)
{
return (uint_fast32_t)(src[0] & 255)
| ((uint_fast32_t)(src[1] & 255) << 8)
| ((uint_fast32_t)(src[2] & 255) << 16)
| ((uint_fast32_t)(src[3] & 255) << 24);
}
static inline uint64_t unpack_u40(const unsigned char *src)
{
return (uint_fast64_t)(src[0] & 255)
| ((uint_fast64_t)(src[1] & 255) << 8)
| ((uint_fast64_t)(src[2] & 255) << 16)
| ((uint_fast64_t)(src[3] & 255) << 24)
| ((uint_fast64_t)(src[4] & 255) << 32);
}
static inline uint64_t unpack_u48(const unsigned char *src)
{
return (uint_fast64_t)(src[0] & 255)
| ((uint_fast64_t)(src[1] & 255) << 8)
| ((uint_fast64_t)(src[2] & 255) << 16)
| ((uint_fast64_t)(src[3] & 255) << 24)
| ((uint_fast64_t)(src[4] & 255) << 32)
| ((uint_fast64_t)(src[5] & 255) << 40);
}
static inline uint64_t unpack_u56(const unsigned char *src)
{
return (uint_fast64_t)(src[0] & 255)
| ((uint_fast64_t)(src[1] & 255) << 8)
| ((uint_fast64_t)(src[2] & 255) << 16)
| ((uint_fast64_t)(src[3] & 255) << 24)
| ((uint_fast64_t)(src[4] & 255) << 32)
| ((uint_fast64_t)(src[5] & 255) << 40)
| ((uint_fast64_t)(src[6] & 255) << 48);
}
static inline uint64_t unpack_u64(const unsigned char *src)
{
return (uint_fast64_t)(src[0] & 255)
| ((uint_fast64_t)(src[1] & 255) << 8)
| ((uint_fast64_t)(src[2] & 255) << 16)
| ((uint_fast64_t)(src[3] & 255) << 24)
| ((uint_fast64_t)(src[4] & 255) << 32)
| ((uint_fast64_t)(src[5] & 255) << 40)
| ((uint_fast64_t)(src[6] & 255) << 48)
| ((uint_fast64_t)(src[7] & 255) << 56);
}
static inline int8_t unpack_i8(const unsigned char *src)
{
return (int8_t)(src[0] & 255);
}
static inline int16_t unpack_i16(const unsigned char *src)
{
return (int16_t)unpack_u16(src);
}
static inline int32_t unpack_i24(const unsigned char *src)
{
uint_fast32_t u = unpack_u24(src);
/* Sign extend to 32 bits */
if (u & 0x800000)
u |= 0xFF000000;
return (int32_t)u;
}
static inline int32_t unpack_i32(const unsigned char *src)
{
return (int32_t)unpack_u32(src);
}
static inline int64_t unpack_i40(const unsigned char *src)
{
uint_fast64_t u = unpack_u40(src);
/* Sign extend to 64 bits */
if (u & UINT64_C(0x0000008000000000))
u |= UINT64_C(0xFFFFFF0000000000);
return (int64_t)u;
}
static inline int64_t unpack_i48(const unsigned char *src)
{
uint_fast64_t u = unpack_i48(src);
/* Sign extend to 64 bits */
if (u & UINT64_C(0x0000800000000000))
u |= UINT64_C(0xFFFF000000000000);
return (int64_t)u;
}
static inline int64_t unpack_i56(const unsigned char *src)
{
uint_fast64_t u = unpack_u56(src);
/* Sign extend to 64 bits */
if (u & UINT64_C(0x0080000000000000))
u |= UINT64_C(0xFF00000000000000);
return (int64_t)u;
}
static inline int64_t unpack_i64(const unsigned char *src)
{
return (int64_t)unpack_u64(src);
}
#endif /* PACKING_H */
When packed, these values are in two's complement little-endian byte order.
pack_uN() and unpack_uN() work with unsigned integers from 0 to 2N-1, inclusive.
pack_iN() and unpack_iN() work with signed integers from -2N-1 to 2N-1-1, inclusive.
Let's consider a simple binary protocol, where each message starts with two bytes: first one the total length of this message, and the second one identifying the type of the message.
This has the nice feature that if something odd happens, it is always possible to resynchronize by sending at least 256 zeroes. Each zero is an invalid length for the message, so they should just be skipped by the receiver. You probably won't need this, but it may come in handy someday.
To receive a message of this form, we can use the following function:
/* Receive a single message.
'fd' is the socket descriptor, and
'msg' is a buffer of at least 255 chars.
Returns -1 with errno set if an error occurs,
or the message type (0 to 255, inclusive) if success.
*/
int recv_message(const int fd, unsigned char *msg)
{
ssize_t n;
msg[0] = 0;
msg[1] = 0;
/* Loop to skip zero bytes. */
do {
do {
n = read(fd, msg, 1);
} while (n == -1 && errno == EINTR);
if (n == -1) {
/* Error; errno already set. */
return -1;
} else
if (n == 0) {
/* Other end closed the socket. */
errno = EPIPE;
return -1;
} else
if (n != 1) {
errno = EIO;
return -1;
}
} while (msg[0] == 0);
/* Read the rest of the message. */
{
unsigned char *const end = msg + msg[0];
unsigned char *ptr = msg + 1;
while (ptr < end) {
n = read(fd, ptr, (size_t)(end - ptr));
if (n > 0) {
ptr += n;
} else
if (n == 0) {
/* Other end closed socket */
errno = EPIPE;
return -1;
} else
if (n != -1) {
errno = EIO;
return -1;
} else
if (errno != EINTR) {
/* Error; errno already set */
return -1;
}
}
}
/* Success, return message type. */
return msg[1];
}
In your own code, you can use the above like this:
unsigned char buffer[256];
switch(receive_message(fd, buffer)) {
case -1:
if (errno == EPIPE) {
/* The other end closed the connection */
} else {
/* Other error; see strerror(errno). */
}
break or return or abort;
case 0: /* Exit/cancel game */
break or return or abort;
case 4: /* Coordinate message */
int x = unpack_i16(buffer + 2);
int y = unpack_i16(buffer + 4);
/* x,y is the coordinate pair; do something */
break;
default:
/* Ignore all other message types */
}
where I randomly chose 0 as the abort-game message type, and 4 as the coordinate message type.
Instead of scattering such statements here and there in your client, put it in a function. You could also consider using a finite-state machine to represent the game state.
To send messages, you can use a helper function like
/* Send one or more messages; does not verify contents.
Returns 0 if success, -1 with errno set if an error occurs.
*/
int send_message(const int fd, const void *msg, const size_t len)
{
const unsigned char *const end = (const unsigned char *)msg + len;
const unsigned char *ptr = (const unsigned char *)msg;
ssize_t n;
while (ptr < end) {
n = write(fd, ptr, (size_t)(end - ptr));
if (n > 0) {
ptr += n;
} else
if (n != -1) {
/* C library bug, should not occur */
errno = EIO;
return -1;
} else
if (errno != EINTR) {
/* Other error */
return -1;
}
}
return 0;
}
so that sending an abort game (type 0) message would be
int send_abort_message(const int fd)
{
unsigned char buffer[2] = { 1, 0 };
return send_message(fd, buffer, 2);
}
and sending a coordinate (type 4) message would be e.g.
int send_coordinates(const int fd, const int x, const int y)
{
unsigned char buffer[2 + 2 + 2];
buffer[0] = 6; /* Length in bytes/chars */
buffer[1] = 4; /* Type */
pack_i16(buffer + 2, x);
pack_i16(buffer + 4, y);
return send_message(fd, buffer, 6);
}
If the game is not turn-based, you won't want to block in the sends or receives, like the above functions do.
Nonblocking I/O is the way to go. Essentially, you'll have something like
static int server_fd = -1;
static size_t send_size = 0;
static unsigned char *send_data = NULL;
static size_t send_next = 0; /* First unsent byte */
static size_t send_ends = 0; /* End of buffered data */
static size_t recv_size = 0;
static unsigned char *recv_data = NULL;
static size_t recv_next = 0; /* Start of next message */
static size_t recv_ends = 0; /* End of buffered data */
and you set the server_fd nonblocking using e.g. fcntl(server_fd, F_SETFL, O_NONBLOCK);.
A communicator function will try to send and receive as much data as possible. It will return 1 if it sent anything, 2 if it received anything, 3 if both, 0 if neither, and -1 if an error occurred:
int communicate(void) {
int retval = 0;
ssize_t n;
while (send_next < send_ends) {
n = write(server_fd, send_data + send_next, send_ends - send_next);
if (n > 0) {
send_next += n;
retval |= 1;
} else
if (n != -1) {
/* errno already set */
return -1;
} else
if (errno == EAGAIN || errno == EWOULDBLOCK) {
/* Cannot send more without blocking */
break;
} else
if (errno != EINTR) {
/* Error, errno set */
return -1;
}
}
/* If send buffer became empty, reset it. */
if (send_next >= send_ends) {
send_next = 0;
send_ends = 0;
}
/* If receive buffer is empty, reset it. */
if (recv_next >= recv_ends) {
recv_next = 0;
recv_ends = 0;
}
/* Receive loop. */
while (1) {
/* Receive buffer full? */
if (recv_ends + 256 > recv_ends) {
/* First try to repack. */
if (recv_next > 0) {
memmove(recv_data, recv_data + recv_next, recv_ends - recv_next);
recv_ends -= recv_next;
recv_next = 0;
}
if (recv_ends + 256 > recv_ends) {
/* Allocate 16k more (256 messages!) */
size_t new_size = recv_size + 16384;
unsigned char *new_data;
new_data = realloc(recv_data, new_size);
if (!new_data) {
errno = ENOMEM;
return -1;
}
recv_data = new_data;
recv_size = new_size;
}
}
/* Try to receive incoming data. */
n = read(server_fd, recv_data + recv_ends, recv_size - recv_ends);
if (n > 0) {
recv_ends += n;
retval |= 2;
} else
if (n == 0) {
/* Other end closed the connection. */
errno = EPIPE;
return -1;
} else
if (n != -1) {
errno = EIO;
return -1;
} else
if (errno == EAGAIN || errno == EWOULDBLOCK) {
break;
} else
if (errno != EINTR) {
return -1;
}
}
return retval;
}
When there is nothing to do, and you want to wait for a short while (some milliseconds), but interrupt the wait whenever more I/O can be done, use
/* Wait for max 'ms' milliseconds for communication to occur.
Returns 1 if data received, 2 if sent, 3 if both, 0 if neither
(having waited for 'ms' milliseconds), or -1 if an error occurs.
*/
int communicate_wait(int ms)
{
struct pollfd fds[1];
int retval;
/* Zero timeout is "forever", and we don't want that. */
if (ms < 1)
ms = 1;
/* We try communicating right now. */
retval = communicate();
if (retval)
return retval;
/* Poll until I/O possible. */
fds[0].fd = server_fd;
if (send_ends > send_next)
fds[0].events = POLLIN | POLLOUT;
else
fds[0].events = POLLIN;
fds[0].revents = 0;
poll(fds, 1, ms);
/* We retry I/O now. */
return communicate();
}
To process messages received thus far, you use a loop:
while (recv_next < recv_ends && recv_next + recv_data[recv_next] <= recv_ends) {
if (recv_data[recv_next] == 0) {
recv_next++;
continue;
}
/* recv_data[recv_next+0] is the length of the message,
recv_data[recv_next+1] is the type of the message. */
switch (recv_data[recv_next + 1]) {
case 4: /* Coordinate message */
if (recv_data[recv_next] >= 6) {
int x = unpack_i16(recv_data + recv_next + 2);
int y = unpack_i16(recv_data + recv_next + 4);
/* Do something with x and y ... */
}
break;
/* Handle other message types ... */
}
recv_next += recv_data[recv_next];
}
Then you recalculate game state, update the display, communicate some more, and repeat.

Convert HEX to printable string/char

I'm using CNG to generate a hash.
Result of BCryptFinishHash call is MD5 of a input in hex form.
Example:
char *outHash = "\x02\x34\x75\01..."
I want to convert it to printable string: 02347501...
How can I do that?
To encode a byte array in hex and write the encoded data to a std::string, do this:
static inline char
hex_digit(unsigned int n)
{
if (n < 10) return '0' + n;
if (n < 16) return 'a' + (n - 10);
abort();
}
std::string
encode_bytes(const unsigned char *bytes, size_t len)
{
std::string rv;
rv.reserve(len * 2);
for (size_t i = 0; i < len; i++) {
rv.push_back(hex_digit((bytes[i] & 0xF0) >> 4));
rv.push_back(hex_digit((bytes[i] & 0x0F) >> 0));
}
return rv;
}
Note that you must know the length of the byte array. It is not safe to treat it as a NUL-terminated "C string", because binary data can contain internal zero bytes. To know the length of a hash generated by CNG, call BCryptGetProperty to get the BCRYPT_HASH_LENGTH property.
we can use CryptBinaryToString here with CRYPT_STRING_HEXASCII or CRYPT_STRING_HEX or CRYPT_STRING_HEXRAW or CRYPT_STRING_HEX | CRYPT_STRING_NOCRLF or CRYPT_STRING_HEXRAW | CRYPT_STRING_NOCRLF depen how you want format string. for example
void print(PUCHAR pbHash, ULONG cbHash, DWORD dwFlags = CRYPT_STRING_HEXRAW | CRYPT_STRING_NOCRLF)
{
ULONG cch = 0;
if (CryptBinaryToStringW(pbHash, cbHash, dwFlags, 0, &cch))
{
if (PWSTR sz = (PWSTR)_malloca(cch * sizeof(WCHAR)))
{
if (CryptBinaryToStringW(pbHash, cbHash, dwFlags, sz, &cch))
{
DbgPrint("%S\n", sz);
}
_freea(sz);
}
}
}
If you need an easy, one time solution, this is a useful tool:
https://codebeautify.org/hex-string-converter
However, if you're looking to do this within your code itself, I found this from an earlier thread (AKA, this is not my work but that of #KEINE LUST from here )
int main(void)
{
unsigned char readingreg[4];
readingreg[0] = 0x4a;
readingreg[1] = 0xaa;
readingreg[2] = 0xaa;
readingreg[3] = 0xa0;
char temp[4];
sprintf(temp, "%x", readingreg[0]);
printf("This is element 0: %s\n", temp);
return 0;
}
You can print it like this:
for(const char *wsk=outHash; *wsk; ++wsk){
printf("%02hhx", *wsk);
}
Edit based that cstring can have 0x00 numbers.
C
const char outHash[] = "\x02\x34\x75";
const int size = sizeof(outHash)/sizeof(char) - 1;
for(int i = 0; i < size; ++i){
printf("%02hhx", outHash [i]);
}
C++
std::string outHash = "\x02\x34\x75";
for(int i = 0; i < outHash.size(); ++i) {
printf("%02hhx", outHash [i]);
}
Loop over the characters and print the numerical value (in hex).
#include <iostream>
#include <iomanip>
int main()
{
char* outHash = "\x02\x34\x75\x01\x23\xff"; // Get from your Hash function.
int sizeOfHash = 6; // Use appropriate size for BCryptFinishHash()
// Set up the characteristics of the stream.
// setw(2): Each printed object will use a min width of 2
// setfill('0'): If the object is less than 2 char then fill the space with '0'
// hex: Print numbers in hex.
std::cout << std::setw(2) << std::setfill('0') << std::hex;
// Create a view of the object.
// Makes it simpler to loop over.
std::string_view view(outHash, sizeOfHash);
// Loop over the string.
for(unsigned char val: view) {
// Convert to `unsigned char` to make sure you don't print
// negative numbers. Then convert from there to `int` so that
// the `std::hex will kick in and convert to hex value.
std::cout << static_cast<int>(val);
}
std::cout << "\n";
}
I am working on C++ wrapper around Windows Crypto API & CNG which I am using in my projects. I plan to move all of it to github but for now it is just a work in progress, but you can find it useful for Crypto basics like HEX / Base64 encode / decode etc.
https://github.com/m4x1m1l14n/Crypto
You can use Crypto::Hex::Encode() method to achieve what you want.
#include <Crypto\Hex.hpp>
#include <Crypto\Random.hpp>
using namespace m4x1m1l14n;
char arr[] = { 0xaa, 0xbb, 0xcc, 0xdd, 0x99, 0x00 };
encoded = Crypto::Hex::Encode(arr, sizeof(arr));
/* encoded = "aabbccdd9900" */
Also you can use wrapper for MD5 which is located in Hash namespace, like this. (If you are not using large amount of data)
#include <Crypto\Hex.hpp>
#include <Crypto\Hash.hpp>
using namespace m4x1m1l14n;
encoded = Crypto::Hex::Encode(Crypto::Hash::MD5("Whatever you want to hash"));

C++ - MPEG TS - Parsing header - PID is messed up - Big endian 32 bits mask

I think I'm having a tunnel vision here so I need your help.
I am trying to parse a MPEG Transport-stream file and I'm stuck on the Header, on the wiki you will see that some 32bits BE MASK are provided in order to extract the data from the 4 bytes header. My code is taking into account endianess (I think) and reverses the bytes if it detects that you're running on a little endian. Then I cast the char* to an int and apply the mask, all the values look fine but the PID is messed up and I don't get why...
header definition
namespace ts {
#define SYNC_BYTE_MASK 0xff000000
#define TEI_MASK 0x800000
#define PAYLOAD_START_MASK 0x400000
#define PRIORITY_MASK 0x200000
#define PID_MASK 0x1fff00
#define SCRAMBLING_CTL_MASK 0xc0
#define ADAPTATION_FIELD_MASK 0x20
#define HAS_PAYLOAD_MASK 0x10
#define COUNTER_MASK 0xf
#define HEADER_BYTES 4
#define HEADER_BITS 8 * HEADER_BYTES
class Header {
public:
std::bitset<HEADER_BITS> *full;
unsigned char _syncByte;
bool _tei;
bool _payloadStart;
bool _priority;
int16_t _pid;
std::bitset<2> *_scramblingCtl;
bool _adaptationField;
bool _hasPayload;
int _counter;
Header(const char *, size_t);
~Header();
const std::string toString();
bool isValid();
};
}
Header values assignment
ts::Header::Header(const char *header, size_t n) {
uint32_t bytes = reverseLE(header, n);
// just for display
char t[4];
memcpy(t, header, 4);
std::cout << "Original: " << std::bitset<32>(*((uint32_t *)t)) << std::endl;
this->full = new std::bitset<HEADER_BITS>(bytes);
uint32_t tmp = bytes & SYNC_BYTE_MASK;
this->_syncByte = ((char *)&tmp)[n - 1];
this->_tei = bytes & TEI_MASK;
this->_payloadStart = bytes & PAYLOAD_START_MASK;
this->_priority = bytes & PRIORITY_MASK;
this->_pid = bytes & PID_MASK; // THIS ONE IS MESSED UP !!
this->_scramblingCtl = new std::bitset<2>(bytes & SCRAMBLING_CTL_MASK);
this->_adaptationField = bytes & ADAPTATION_FIELD_MASK;
this->_hasPayload = bytes & HAS_PAYLOAD_MASK;
this->_counter = bytes & COUNTER_MASK;
}
Functions to reverse
#include "utils.h"
int is_big_endian(void)
{
union {
uint32_t i;
char c[4];
} e = { 0x01000000 };
return e.c[0];
}
void swap(char *s, int a, int b) {
char tmp;
tmp = s[a];
s[a] = s[b];
s[b] = tmp;
}
// Converts string to int taking endianess into account
uint32_t reverseLE(const char *bits, size_t n) {
uint32_t ret = 0;
char *cp = (char *)malloc(n * sizeof(char));
memcpy(cp, bits, n);
if ( ! is_big_endian() ) {
for (int i = 0; i < n / 2; i++)
swap(cp, i, n - 1 - i);
}
ret = *((uint32_t *)cp);
free(cp);
return ret;
}
Here's an example of an header that should have a PID of 33
Original: 00010010001000010000000001000111
Binary: 01000111000000000010000100010010
Sync byte: G
TEI: 0
Payload start: 0
Priority: 0
PID: 8448 0010000100000000
Scrambling Ctl: 00
Adaptation field: 0
Has Payload: 1
Counter: 2
Somehow it gets reversed again, and I don't get why...
Ok so the problem was that the 13bits of the PID are located at str[1] and str[2] which means that after casting *((int *)str) and applying the mask there is still 8 trailing 0 bits from the last byte str[3].
Solution:
this->_pid = bytes & PID_MASK;
this->_pid >>= 8;
Thanks to #Wimmel.

Atmega8A uart spi eeprom

everyone, I want to write and store my string at spi eeprom, then read back from spi eeprom and display in terminal through uart. I already follow the step in [1]: http://ww1.microchip.com/downloads/en/DeviceDoc/21822E.pdf . But it seem that it can only display one letter. I don't know if the other letter is save in spi eeprom or not. I hope someone can help me.
I am using:
chip:Atmega8a
software:avr studio 5
terminal: Bray terminal.
#include <avr/io.h>
#include <util/delay.h>
void serial_init(void)
{
UBRRH = 0x00;
UBRRL = 95;
UCSRB = (1 << RXEN) | (1 << TXEN) | (1<<RXCIE);
UCSRC = (1<<URSEL)|(1<<USBS)|(3<<UCSZ0)|(1 << UCSZ1);
}
void SPI_MasterInit(void)
{
DDRB = 0b00101100;
DDR_SPI = (1<<DD_MOSI)|(1<<DD_SCK)|(1<<DD_SS);
SPCR = 0b01010000;
SPSR = 0b00000001;
}
char spi_transfer(volatile char data)
{
SPDR = data;
while(!(SPSR & (1<<SPIF)));
{
}
return SPDR;
}
void SPI_MasterTransmit(unsigned long data)
{
unsigned long address;
DDR_SPI &= ~(1<<DD_SS); //ss goes low
spi_transfer(WREN); //enable write operation
DDR_SPI |= (1<<DD_SS); //ss goes high
_delay_ms(10);
DDR_SPI &= ~(1<<DD_SS); //ss goes low
spi_transfer(WRITE); // write data to memory
spi_transfer(address>>8); // address MSB first
spi_transfer(address);
spi_transfer(data); // send lsb
DDR_SPI |= (1<<DD_SS); //ss goes high
}int resetEEPROM()
{
DDR_SPI &= ~(1<<DD_SS); // Select EEPROM
spi_transfer(WREN); // Send WRITE_ENABLE command
DDR_SPI |= (1<<DD_SS); // Release EEPROM
DDR_SPI &= ~(1<<DD_SS); // Select EEPROM again after WREN cmd
spi_transfer(WRDI); // send CHIP_ERASE command
DDR_SPI |= (1<<DD_SS); // Release EEPROM
return 0;
} // END eraseEEPROM()
unsigned long SPI_MasterReceive(unsigned long address) //terima data //read address
{
unsigned long data;
DDR_SPI &= ~(1<<DD_SS); //ss goes low
spi_transfer(READ); //enable write operation
spi_transfer(address>>8); // address MSB first
spi_transfer(address);
data = spi_transfer(0xff);
DDR_SPI |= (1<<DD_SS); //goes high
return data;
}
int main(void)
{
long int data;
unsigned long address;
serial_init();
SPI_MasterInit();
resetEEPROM();
data = Usart_Receive();
while (1)
{
if (Usart_Receive() == '.')
{
USART_Print("\r\nStore\r\n");
SPI_MasterTransmit(data); //store in spi eeprom
}
if (Usart_Receive() == '>')
{
USART_Print("\nout \r\n");
data = SPI_MasterReceive(address); //read data from the memory
Usart_Transmit(data);
}
}
return 0;
}
There is a way to write more than one byte to the EEPROM at once, but your code does not do that. Instead, you are writing one byte per write operation, and always at the same address. You are overwriting any previous bytes with each new one.
If you want to store more than one byte, you need to change the address as you write, or change the way you are writing to more than one byte at a time. (Note that you can only write multiple bytes if they are the same page of EEPROM memory.)
Perhaps a circular buffer?
Here are my Circular Buffer code. Based on this http://www.rn-wissen.de/index.php/UART_mit_avr-gcc
#include <avr/io.h>
#include <fifo.h>
#define FIFOBUF_SIZE 128
uint8_t fifobuf[FIFOBUF_SIZE];
fifo_t fifo;
ISR (USART_RXC_vect)
{
_inline_fifo_put(&fifo, UDR);
}
void serial_init(void)
{
cli();
UBRRH = 0x00;
UBRRL = 95;
UCSRB = (1 << RXCIE) | (1 << RXEN) | (1 << TXEN);
UCSRC = (1<<URSEL)|(1<<USBS)|(3<<UCSZ0);
sei();
}
void fifo_init (fifo_t *f, uint8_t * buffer, const uint8_t size)
{
f-> count = 0;
f-> pread = f-> pwrite = buffer;
f-> read2end = f-> write2end = f-> size = size;
}
static inline int Usart_Transmit (const uint8_t c)
{
PORTD= 0b00000100; //RTS Enable
while ((UCSRA & (1 << UDRE)) ==0) {};
UDR = c;
PORTD= 0b00000000; //RTS Enable
return 1;
}
int main(void)
{
unsigned long data;
unsigned long address;
fifo_init(&fifo, fifobuf, FIFOBUF_SIZE);
serial_init();
while (1)
{
SPI_MasterInit();
resetEEPROM();
SPI_MasterTransmit(Usart_Receive());
_delay_ms(100);
if (fifo.count > 0) //; fifo.count >8 ; fifo.count
{
Usart_Transmit(_inline_fifo_get(&fifo));
}
data = SPI_MasterReceive(address); //read data from the memory
_delay_ms(100);
Usart_Transmit(data);
}
return 0;
}
it Came out all of the letter, but not follow the sequence. Example like this " bfabeaabbfcabf ", I am only type " abcdef "
And can you show me how to set the EEPROM address in spi EEPROM. Like e.g. show me some link or example about this spi EEPROM address. I ask for your Kindness to help me about this because I have been about 2 months searching on the internet, there only few examples on how to handle spi EEPROM address. Mostly I just found about ATmega EEPROM, LTD. And all of Them are not give me a good result. Thank in advance for your time. :)

QT socket read losing bytes

I'm trying to work with length-preceded TCP messages using Qt. I have following method:
QByteArray con::read()
{
QByteArray s;
s = _pSocket->read(4);
if (s.length() == 4) {
int size = char_to_int32(s);
s = _pSocket->read(size);
}
return s;
}
Well, it does not work. Looks like I lose all data after reading first 4 bytes: the first read works fine, but read(size) returns nothing. Is there a way to solve this?
The char_to_int32 is:
int char_to_int32(QByteArray s)
{
int size = 0;
size |= (s.at(0) << 24);
size |= (s.at(1) << 16);
size |= (s.at(2) << 8);
size |= (s.at(3));
return size;
}
EDIT :
The sending function (plain C):
int send(int connfd, const unsigned char* message, unsigned int size) {
int c;
unsigned char* bytes = (unsigned char*) malloc(4 + size);
int32_to_char(size, bytes); // converts message size to 4 bytes
memcpy(bytes + 4, message, size);
c = write(connfd, bytes, 4 + size);
free(bytes);
if (c <= 0)
return -1;
else
return 0;
}
By the way, when I call _pSocket->readAll(), the entire packet is read, including 4-byte size and message itself.
EDIT :
void int32_to_char(uint32_t in, char* bytes) {
bytes[0] = (in >> 24) & 0xFF;
bytes[1] = (in >> 16) & 0xFF;
bytes[2] = (in >> 8) & 0xFF;
bytes[3] = in & 0xFF;
return;
}
As you are using the QByteArray QIODevice::read(qint64 maxSize) function, you may not be detecting errors correctly:
This function has no way of reporting errors; returning an empty QByteArray() can mean either that no data was currently available for reading, or that an error occurred.
Some things to try:
Use the qint64 QIODevice::read(char* data, qint64 maxSize) which reports errors:
If an error occurs ... this function returns -1.
Call QIODevice::errorString and QAbstractSocket::error to find out what is going wrong.
For bonus points, listen to the QAbstractSocket::error error signal.
If this is a new protocol you are creating, try using QDataStream for serialization, this automatically handles length prefixs and is platform independent. Your char_to_int32 will break if you mix platforms with different endienness, and may break between different OSs or compilers as int is not guaranteed to be 32 bits (it is defined as at least 16 bits).
If you can't use QDataStream, at least use the htons, ntohs ... functions.
Edit
Here is some example code showing hton/ntoh usage. Note that uint32_t and not int is used as it's guaranteed to be 32 bits. I've also used memcpy rather than pointer casts in the encode/decode to prevent aliasing and alignment problems (I've just done a cast in the test function for brevity).
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>
void encode(uint32_t in, char* out)
{
/* Host to Network long (32 bits) */
const uint32_t t = htonl(in);
memcpy(out, &t, sizeof(t));
}
uint32_t decode(char* in)
{
uint32_t t;
memcpy(&t, in, sizeof(t));
/* Network to Host long (32 bits) */
return ntohl(t);
}
void test(uint32_t v)
{
char buffer[4];
printf("Host Input: %08x\n", v);
encode(v, buffer);
printf("Network: %08x\n", *((uint32_t*)buffer));
printf("Host Output: %08x\n\n", decode(buffer));
}
int main(int argc, char** argv)
{
test(0);
test(1);
test(0x55);
test(0x55000000);
return 0;
}