I'm trying to send data with a fixed-length header that tells the server how many bytes of data it's going to have to have available to read before it reads it. I'm having trouble doing this, though. The maximum number of bytes of data I want to be able to send at once is 65536, so I'm sending a uint16_t type variable as the header of my data because the maximum number it can represent is 65536.
The problem is, a uint16_t takes up two bytes, but numbers less than 255 only require one byte. So I have this code on the client side:
uint16_t messageSize = clientSendBuf.size(); //clientSendBuf is the data I want to send
char *bytes((char*)&messageSize);
clientSendBuf.prepend(bytes);
client.write(clientSendBuf);
And on the server, I handle receiving messages like this:
char serverReceiveBuf[65536];
uint16_t messageSize;
client->read((char*)&messageSize, sizeof(uint16_t));
client->read(serverReceiveBuf, messageSize);
I'm going to change this around a bit later because it's not the best solution (particularly for when all of the data isn't available yet), but I want to get this fixed first. My problem is that when clientSendBuf.size() is too small (in my test case it was 16 bytes, I assume this happens for every value under 255) reading data with
client->read((char*)&messageSize, sizeof(uint16_t));
reads a second byte that isn't part of the header, giving and incorrect value for messageSize and crashing the server. If I replace sizeof(uint16_t) with 1, then the server reads the data fine as I'd expect, although then I have a messageSize maximum of 255, which is much lower than I want. How do I make it so that the messageSize prepended to clientSendBuf is always two bytes, even for numbers <255?
Your
clientSendBuf.prepend(bytes);
Should also be told that it needs to send 2 bytes; now it treats the bytes as a zero-terminated string, which accidently works since on your platform the second byte of 0x0010 is zero (using little-endian numbers: 0x16, 0x00).
The prepend(char*, int) method will do the trick:
// use this instead:
cliendSendBuf.prepend(bytes, sizeof(messageSize));
Related
I want to send a payload that has this structure:
uint8_t num_of_products;
//1 product
uint8_t action;
time_t unix_time;//uint64_t
uint32_t name_len;//up to 256
char* name;
uint8_t num_of_ips;
//1 ip out if num_of_ips
uint8_t ip_ver; //0 error , 1 ipv4 , 2 ipv6
char* IP;// ipv4 : 4 , ipv6 : 16
before sending the packet I aggregate products using memcpy into jumbo size mbuf
from tests I did name_len must go through hton in order to not look "inverted" in wireshark.
my question is ,what logic can I apply in order to get the byte order right for a custom structure with inner variables with unknown size
i.e what should go through hton what should be left as is
If you are aiming to have your message in network byte order (big endian), only your integer fields that take up more than one byte will need the htonl treamnet. In your case, that would be time_t unix_time and uint32_t name_len. Your strings and single-byte fields (such as num_of_products) won't need any specific conversion.
As some of the commenters in your question suggested - it's really up to you if you want to use a strict network byte order. Serializing your message to have a strict byte ordering is useful if you intend your code to run across different platforms.
Writing efficient byte packing code is annoyingly hard. You wind up writing a lot of code to just to save a few bytes of network bandwidth.
User jxh mentioned JSON as a possible encoding for your message. Not sure why he deleted his answer because it was on point. But in any case, a standard messaging format of either JSON or XML (or any ascii text schema) is 100x easier to observe in wireshark and when debugging.
I am working with a device (the wiimote) that takes commands through the DATA pipe, and only accepts command packets that are EXACTLY as long as the command itself. For example, it will accept:
0x11 0x10
but it will not accept:
0x11 0x10 0x00 0x00 0x00 ... etc.
This is a problem on windows, as WriteFile() on windows requires that the byte[] passed to it is at least as long as caps.OutputReportByteLength. On mac, where this limitation isn't present, my code works correctly. Here is the code from hid.c that causes this issue:
/* Make sure the right number of bytes are passed to WriteFile. Windows
expects the number of bytes which are in the _longest_ report (plus
one for the report number) bytes even if the data is a report
which is shorter than that. Windows gives us this value in
caps.OutputReportByteLength. If a user passes in fewer bytes than this,
create a temporary buffer which is the proper size. */
if (length >= dev->output_report_length) {
/* The user passed the right number of bytes. Use the buffer as-is. */
buf = (unsigned char *) data;
} else {
/* Create a temporary buffer and copy the user's data
into it, padding the rest with zeros. */
buf = (unsigned char *) malloc(dev->output_report_length);
memcpy(buf, data, length);
memset(buf + length, 0, dev->output_report_length - length);
length = dev->output_report_length;
}
res = WriteFile(dev->device_handle, buf, length, NULL, &ol);
Removing the above code, as mentioned in the comments, results in an error from WriteFile().
Is there any way that I can pass data to the device of arbitrary size? Thanks in advance for any assistance.
Solved. I used a solution similar to the guys over at Dolphin, a Wii emulator. Apparently, on the Microsoft bluetooth stack, WriteFile() doesn't work correctly, causing the Wiimote to return with an error. By using HidD_SetOutputReport() on the MS stack and WriteFile() on the BlueSoleil stack, I was able to successfully connect to the device (at least on my machine).
I haven't tested this on the BlueSoleil stack, but Dolphin is using this method so it is safe to say it works.
Here is a gist containing an ugly implementation of this fix:
https://gist.github.com/Flafla2/d261a156ea2e3e3c1e5c
I am using openssl to encrypt the string and i get null terminated string. I have now encrypted string and I want to send it over network with base64 encoding. I just need to send the encrypted data, how can I calculate the length of the string on the other side before decryption?
unsigned char *plaintext = (unsigned char*) "The quick brown fox jumps sover thes lazy dog";
unsigned char ciphertext[256] = {};
// After using openssl envelope_seal(), with EVP_aes_128_cbc()
strlen((const char*)ciphertext) // length is different each time due to null terminated binary string
sizeof(ciphertext) // lenght is equal to 256
int envelope_seal( ) // gives me the length of that string as 48.
Kindly help me to calculate the length of the ciphertext which is 48 but none of the method in my knowledge gives me the correct output.
AES is a block cipher. It has block size of 16 bytes - which means if you want to encrypt some data with it the length of the data in bytes must be multiple of 16 (if it is not you might need to use padding such as say PKCS7, more details).
Now after you encrypt a string with AES (say length of string is 32 bytes) - you can't use strlen anymore to get the length of the result, because the result, isn't a string anymore it is some byte array which represents the results of encryption. Actually you don't need to get the length anyway, it will be same size as plaintext - 32 bytes as we said in our case.
So I think you don't have issues with calculating length anymore - now if the other side should know length of the ciphertext you can send the length (say 32 in our case) in advance in packet. The other side should reconstruct the plain text now (and also remove padding bytes if one was used).
note: After you performed encryption and have the ciphertext you can apply base64 encoding to it and send it over, but you could as well send the byte array representing the ciphertext too.
In regard to comments, I will try to briefly highlight how this process goes. Say you have string char * str = "hello" - which is 5 bytes and you need to encrypt it. Like I said you can't encrypt it directly, you need to pad it to make multiple of 16. For this you can use PKCS7 padding (it is similar to PKCS5 but is for 16 bytes blocks). Now when you apply padding e.g., char * paddedText = PKCS7(str), you will end up with byte array which is 16 bytes.
Now, there is no more problem. You can encrypt this 16 bytes of plaintext. Result will also be 16 bytes cipher text.
Now if you want to decrypt this 16 bytes of cipher text, you can decrypt it directly (because it is multiple of 16). Now, due to the way PKCS7 padding works you will easily be able to tell from the result that only first 5 bytes were original string and you will remove 11 redundant bytes (how PKCS5 works, see in my link-PKCS7 is similar just for 16 byte block lengths), and obtain "hello".
Finally, I don't see where is the problem, when you send from client to server, you can just encode message length, e.g., 16 in packet, so that receiver knows 16 bytes represent cipher text. But again as I said, using this 16 bytes, the receiver, after decrypting will be able to determine that only first 5 bytes were original text (due to used padding PKCS7). So there is no need to send anything except 16; with help of PKCS padding scheme you will be able to tell that only first 5 bytes were plain text.
ps. After some discussions with OP it seems padding was not his main issue, because openssl method that was used seems to take care of that.
If the valid data in the array isn't terminated, then there's no way to tell its length by looking at the array.
If envelope_seal told you the length, then use that, passing it wherever the length is needed.
AES is a block cipher. Therefore the length of ciphertext will be the length of your plaintext modulo blocksize, rounded up to nearest blocksize.
I'm using an Arduino (duemilanove) with the official Ethernet shield to send data to the controller for controlling an LED matrix. I am trying to send some raw 32-bit unsigned int values (unix timestamps) to the controller by taking the 4 bytes in the 32-bit value on the desktop and sending it to the arduino as 4 consecutive bytes. However, whenever a byte value is larger than 127, the returned value by the ethernet client library is 63.
The following is a basic example of what I'm doing on the arduino side of things. Some things have been removed for neatness.
byte buffer[32];
memset(buffer, 0, 32);
int data;
int i=0;
data = client.read();
while(data != -1 && i < 32)
{
buffer[i++] = (byte)data;
data = client.read();
}
So, whenever the input byte is bigger than 127 the variable "data" will end up getting set to 63! At first I thought the problem was further down the line (buffer used to be char instead of byte) but when I print out "data" right after the read, it's still 63.
Any ideas what could be causing this? I know client.read() is supposed to output int and internally reads data from the socket as uint8_t which is a full byte and unsigned, so I should be able to at least go to 255...
EDIT: Right you are, Hans. Didn't realize that Encoding.ASCII.GetBytes only supported the first 7 bits and not all 8.
I'm more inclined to suspect the transmit side. Are you positive the transmit side is working correctly? Have you verified with a wireshark capture or some such?
63 is the ASCII code for ?. There's some relevance to the values, ASCII doesn't have character codes for values over 127. An ASCII encoder commonly replaces invalid codes like this with a question mark. Default behavior for the .NET Encoding.ASCII encoder for example.
It isn't exactly clear where that might happen. Definitely not in your snippet. Probably on the other end of the wire. Write bytes, not characters.
+1 for Hans Passant and Karl Bielefeldt.
Can you just send the data without encoding? How is the data being sent? TCP/UDP/IP/Ethernet definitely support sending binary data without restriction. If this isn't possible, perhaps converting the data to hex will solve the problem. Base64 will also work (better) but is considerably more work. For small amounts of data, hex is probably the easiest and fastest solution.
+1 again to Karl and Ben for mentioning wireshark. Invaluable for debugging network problems like this.
Our server is seemingly packet based. It is an adaptation from an old serial based system. It has been added, modified, re-built, etc over the years. Since TCP is a stream protocol and not a packet protocol, sometimes the packets get broken up. The ServerSocket is designed in such a way that when the Client sends data, part of the data contains the size of our message such as 55. Sometimes these packets are split into multiple pieces. They arrive in order but since we do not know how the messages will be split, our server sometimes does not know how to identify the split message.
So, having given you the background information. What is the best method to rebuild the packets as they come in if they are split? We are using C++ Builder 5 (yes I know, old IDE but this is all we can work with at the moment. ALOT of work to re-design in .NET or newer technology).
TCP guarantees that the data will arrive in the same order it was sent.
That beeing said, you can just append all the incoming data to a buffer. Then check if your buffer contains one or more packets, and remove them from the buffer, keeping all the remaining data into the buffer for future check.
This, of course, suppose that your packets have some header that indicates the size of the following data.
Lets consider packets have the following structure:
[LEN] X X X...
Where LEN is the size of the data and each X is an byte.
If you receive:
4 X X X
[--1--]
The packet is not complete, you can leave it in the buffer. Then, other data arrives, you just append it to the buffer:
4 X X X X 3 X X X
[---2---]
You then have 2 complete messages that you can easily parse.
If you do it, don't forget to send any length in a host-independant form (ntohs and ntohl can help).
This is often accomplished by prefixing messages with a one or two-byte length value which, like you said, gives the length of the remaining data. If I've understood you correctly, you're sending this as plain text (i.e., '5', '5') and this might get split up. Since you don't know the length of a decimal number, it's somewhat ambiguous. If you absolutely need to go with plain text, perhaps you could encode the length as a 16-bit hex value, i.e.:
00ff <255 bytes data>
000a <10 bytes data>
This way, the length of the size header is fixed to 4 bytes and can be used as a minimum read length when receiving on the socket.
Edit: Perhaps I misunderstood -- if reading the length value isn't a problem, deal with splits by concatenating incoming data to a string, byte buffer, or whatever until its length is equal to the value you read in the beginning. TCP will take care of the rest.
Take extra precautions to make sure that you can't get stuck in a blocking read state should the client not send a complete message. For example, say you receive the length header, and start a loop that keeps reading through blocking recv() calls until the buffer is filled. If a malicious client intentionally stops sending data, your server might be locked until the client either disconnects, or starts sending.
I would have a function called readBytes or something that takes a buffer and a length parameter and reads until that many bytes have been read. You'll need to capture the number of bytes actually read and if it's less than the number you're expecting, advance your buffer pointer and read the rest. Keep looping until you've read them all.
Then call this function once for the header (containing the length), assuming that the header is a fixed length. Once you have the length of the actual data, call this function again.