I am doing a synchronous read/write using boost-asio. The data is coming in binary format, without boundary, the length information is encoded in the packet format. So it is important to read in with specified size. Can ip::tcp::iostream do that? Can someone provide an example? Thanks.
Simple:
boost::asio::read(socket, buffers, boost::asio::transfer_exactly(your_fixed_size));
I work on a program wich send different data with different size. I use a fixed header of 8 byte to encode the size, then, I add the data :
enum { header_length = 8 }; //const header length
I get the size (m_outbound_data is a std::string == a serialized object)
//give header length
std::ostringstream header_stream
header_stream << std::setw(header_length) //set a field padding for header
<< std::hex //set next val to hexadecimal
<< m_data_out.m_outbound_data.size(); //write size in hexa
m_data_out.m_outbound_header = header_stream.str(); //m_outbound_head == size in hexa in a std::string
//m_outbound_header = [ 8 byte size ]
//m_outbound_data = [ serialized data ]
//write all data in the std::vector and send it
std::vector<boost::asio::const_buffer> buffer;
buffer.push_back(boost::asio::buffer(m_data_out.m_outbound_header));
buffer.push_back(boost::asio::buffer(m_data_out.m_outbound_data));
And for reading, you need to read in 2 time : 1st read 8 byte to get the size, then read the data in a vector and deserialize into object :
struct network_data_in {
char m_inbound_header[header_length]; //size of data to read
std::vector<char> m_inbound_data; // read data
};
I use this struct to get data, call read on the m_inbound_header to fill the buffer with size first, then, in the handle :
//get size of data
std::istringstream is(std::string(m_data_in.m_inbound_header, header_length));
std::size_t m_inbound_datasize = 0;
is >> std::hex >> m_inbound_datasize;
m_data_in.m_inbound_data.resize(m_inbound_datasize); //resize the vector
then call again read with the m_inbound_data on buffer, this result of reading exactly the data sent
In the second handle_read you juste have to deserialize the data :
//extract data
std::string archive_data (&(m_data_in.m_inbound_data[0]),m_data_in.m_inbound_data.size());
std::istringstream archive_stream(archive_data);
boost::archive::text_iarchive archive(archive_stream);
archive >> t; //deserialize
Hope that help you !
TCP is a stream-based protocol. This means that whatever you read is just a stream of bytes.
Let's consider an example: you have a message of a fixed size and you send it over TCP. How can the program at the other end read the entire message? there are two ways, one is to surround you message with control chracters (e.g. STX at start and ETX at end). At the start, the program would discard any chars before STX, then read any other chars into the message buffer until ETX is encountered.
Another way is to encode the message length in a fixed-size header (which apparently is your case). So the best thing you can do is figure out a way to read the message length, parse it and read the remaining bytes accordingly.
Related
We have a file which is present in a data store (S3) which contains data in the form of byte[] (uploaded using Java language).
Now when i download the file, the data i get is in the form of std::basic_streambuf (Ideally it should also be having bytes). Now i want to send this data to another API which takes uint8_t* as the input.
What is the way to do so? Is it making any sense to even do that?
I tried this:
// Assume streambuf is:
std::streambuf *buf;
std::stringstream ss;
ss << buf;
// Solution1
const std::string output1 = ss.str();
cout<<output1;
// This prints the whole data with some weird characters (i think weird characters are valid because data is in byte form). Upon converting output1 to uint8_t*, the final data contains only 20 characters/bytes.
// Solution2
uint8_t* finalString;
ss >> finalString;
cout<<finalString;
// This prints only initial 20 characters/bytes and the remaining part is skipped.
So with both Solution1 and Solution2, ultimate goal of getting uint8_t* of full data could not be achieved. What is the suggested way to do so?
You have to read your data out of the buffer (since the buffer itself can be streaming the data in as it's available). One possible implementation is something like this:
vector<uint8_t> bytes;
do {
bytes.push_back(buf->sgetc());
} while(buf->snextc() != EOF);
// your data is in bytes.data() of type uint8_t*
Of course if you know the number of bytes from the beginning instead of having to read the buffer to find out, simply pre-allocate the vector beforehand.
I got a client and a server application which will send each other data by using the Asio (Standalone) library. Both applications consists of two (logical) parts:
A high level part: dealing with complex objects e.g. users, permissions,...
A low level part: sending data over network between client and server
Let's assume the complex objects are already serialized by using Protocoll Buffers and the low level part of the application receives the data as std::string from the high level part. I would like to use this function from Protocoll Buffers for this job:
bool SerializeToString(string* output) const;: serializes the message
and stores the bytes in the given string. Note that the bytes are
binary, not text; we only use the string class as a convenient
container.
And say I transfer this data with async_write on the client side:
size_t dataLength = strlen(data);
//writes a certain number of bytes of data to a stream.
asio::async_write(mSocket,
asio::buffer(data, dataLength),
std::bind(&Client::writeCallback, this,
std::placeholders::_1,
std::placeholders::_2));
How can I read this data on the server side? I don't know how much data I will have to read. Therefore this will not work (length is unknown):
asio::async_read(mSocket,
asio::buffer(mResponse, length),
std::bind(&Server::readCallback, this,
std::placeholders::_1,
std::placeholders::_2));
What is the best way to solve this problem? I could think of two solutions:
Append a 'special' character at the end of data and read until I reach this 'end of data signal'. The problem is, what if this character appears in data somehow? I don't know how Protocoll Buffers serializes my data.
Send a binary string with size_of_data + data instead of data. But I don't know how to serialize the size in an platform independent way, add it to the binary data and extract it again.
Edit: Maybe I could use this:
uint64_t length = strlen(data);
uint64_t nwlength = htonl(length);
uint8_t len[8];
len[0] = nwlength >> 56;
len[1] = nwlength >> 48;
len[2] = nwlength >> 40;
len[3] = nwlength >> 32;
len[4] = nwlength >> 24;
len[5] = nwlength >> 16;
len[6] = nwlength >> 8;
len[7] = nwlength >> 0;
std::string test(len);
mRequest = data;
mRequest.insert(0, test);
and send mRequest to the server? Any traps or caveats with this code?
How could I read the length on server side and the content afterwards?
Maybe like this:
void Server::readHeader(){
asio::async_read(mSocket,
asio::buffer(header, HEADER_LENGTH),
std::bind(&Server::readHeaderCallback, this,
std::placeholders::_1,
std::placeholders::_2),
asio::transfer_exactly(HEADER_LENGTH));
}
void Server::readHeaderCallback(const asio::error_code& error,
size_t bytes_transferred){
if(!error && decodeHeader(header, mResponseLength)){
//reading header finished, now read the content
readContent();
}
else{
if(error) std::cout << "Read failed: " << error.message() << "\n";
else std::cout << "decodeHeader failed \n";
}
}
void Server::readContent(){
asio::async_read(mSocket,
asio::buffer(mResponse, mResponseLength),
std::bind(&Server::readContentCallback, this,
std::placeholders::_1,
std::placeholders::_2),
asio::transfer_exactly(mResponseLength));
}
void Server::readContentCallback(const asio::error_code& error,
size_t bytes_transferred){
if (!error){
//handle content
}
else{
//#todo remove this cout
std::cout << "Read failed: " << error.message() << "\n";
}
}
Please note that I try to use transfer_exactly. Will this work?
When sending variable length messages over a stream-based protocol, there are generally three solutions to indicate message boundaries:
Use a delimiter to specify message boundaries. The async_read_until() operations provide a convenient way to read variable length delimited messages. When using a delimiter, one needs to consider the potential of a delimiter collision, where the delimiter appears within the contents of a message, but does not indicate a boundary. There are various techniques to handle delimiter collisions, such as escape characters or escape sequences.
Use a fixed-length header with a variable-length body protocol. The header will provide meta-information about the message, such as the length of the body. The official Asio chat example demonstrates one way to handle fixed-length header and variable-length body protocols.
If binary data is being sent, then one will need to consider handling byte-ordering. The hton() and ntoh() family of functions can help with byte-ordering. For example, consider a protocol that defines the field as two bytes in network-byte-order (big-endian) and a client reads the field as a uint16_t. If the value 10 is sent, and a little-endian machine reads it without converting from network-order to local-order, then the client will read the value as 2560. The Asio chat example avoids handling endianness by encoding the body length to string instead of a binary form.
Use the connection's end-of-file to indicate the end of a message. While this makes sending and receiving messages easy, it limits the sender to only one message per connection. To send an additional message, one would need to established another connection.
A few observations about the code:
The Protocol Buffers' SerializeToString() function serializes a message to a binary form. One should avoid using text based functions, such as strlen(), on the serialized string. For instance, strlen() may incorrectly determine the length, as it will treat the first byte with a value of 0 as the terminating null byte, even if that byte is part of the encoded value.
When providing an explicitly sized buffer to an operation via asio::buffer(buffer, n), the default completion condition of transfer_all will function the same as transfer_exactly(n). As such, the duplicate use of variables can be removed:
asio::async_read(mSocket,
asio::buffer(header, HEADER_LENGTH),
std::bind(&Server::readHeaderCallback, this,
std::placeholders::_1,
std::placeholders::_2));
The htonl() overloads support uint16_t and uint_32t, not uint64_t.
Asio supports scatter/gather operations, allowing a receive operation to scatter-read into multiple buffers, and transmit operations can gather-write from multiple buffers. As such, one does not necessarily need to have both the fixed-length header and message-body contained with a single buffer.
std::string body_buffer;
body.SerializeToString(&body_buffer);
std::string header_buffer = encode_header(body_buffer.size());
// Use "gather-write" to send both the header and data in a
// single write operation.
std::vector<boost::asio::const_buffer> buffers;
buffers.push_back(boost::asio::buffer(header_buffer));
buffers.push_back(boost::asio::buffer(body_buffer));
boost::asio::write(socket_, buffers);
client must call
socket.shutdown(asio::ip::tcp::socket::shutdown_both);
socket.close();
on the server size read until EOF detected
std::string reveive_complete_message(tcp::socket& sock)
{
std::string json_msg;
asio::error_code error;
char buf[255];
while (1)
{
//read some data up to buffer size
size_t len = sock.read_some(asio::buffer(buf), error);
//store the received buffer and increment the total return message
std::string str(buf, len);
json_msg += str;
if (error == asio::error::eof)
{
//EOF received, the connection was closed by client
break;
}
else if (error)
{
throw asio::system_error(error);
}
}
return json_msg;
}
I need to parse a text file stream after downloading and without saving the text file to harddisk or sdcard using tftp protocol. I've the data in payload. Please help.
struct pbuf {
struct pbuf *next;
void *payload;
u16_t tot_len;
u16_t len;
u8_t type;
u8_t flags;
u16_t ref;
};
The following code assigns the opcode and filename. But how to get the contents of the file?
/* Extract the opcode from a TFTP message in a buffer */
tftp_opcode tftp_decode_op(char *buf)
{
return (tftp_opcode)(buf[1]);
}
void tftp_extract_filename(char *fname, char *buf)
{
strcpy(fname, buf + 2);
}
In the TFTP protocol, you first get a write request packet (opcode WRQ), and then the data in separate data packets (opcode DATA). You will need to do something whenever a data packet is received in order to process the contents of the file.
I'd recommend reading up a bit on the TFTP protocol if you haven't done that yet. An overview is at http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol#Protocol_walkthrough and the spec (which is very readable) is at https://www.rfc-editor.org/rfc/rfc1350 .
If your text file not more then some KB then why you not use static buffer to store content of it.In TFTP protocol you must know at which (in source point of view ) point the packet received.So you get packet buffer and size of the packet.So simply copy that received packet buffer in above static buffer and increase offest of static buffer and so on....So at last you will get whole text file content in static buffer.
I am taking input from a file in binary mode using C++; I read the data into unsigned ints, process them, and write them to another file. The problem is that sometimes, at the end of the file, there might be a little bit of data left that isn't large enough to fit into an int; in this case, I want to pad the end of the file with 0s and record how much padding was needed, until the data is large enough to fill an unsigned int.
Here is how I am reading from the file:
std::ifstream fin;
fin.open('filename.whatever', std::ios::in | std::ios::binary);
if(fin) {
unsigned int m;
while(fin >> m) {
//processing the data and writing to another file here
}
//TODO: read the remaining data and pad it here prior to processing
} else {
//output to error stream and exit with failure condition
}
The TODO in the code is where I'm having trouble. After the file input finishes and the loop exits, I need to read in the remaining data at the end of the file that was too small to fill an unsigned int. I need to then pad the end of that data with 0's in binary, recording enough about how much padding was done to be able to un-pad the data in the future.
How is this done, and is this already done automatically by C++?
NOTE: I cannot read the data into anything but an unsigned int, as I am processing the data as if it were an unsigned integer for encryption purposes.
EDIT: It was suggested that I simply read what remains into an array of chars. Am I correct in assuming that this will read in ALL remaining data from the file? It is important to note that I want this to work on any file that C++ can open for input and/or output in binary mode. Thanks for pointing out that I failed to include the detail of opening the file in binary mode.
EDIT: The files my code operates on are not created by anything I have written; they could be audio, video, or text. My goal is to make my code format-agnostic, so I can make no assumptions about the amount of data within a file.
EDIT: ok, so based on constructive comments, this is something of the approach I am seeing, documented in comments where the operations would take place:
std::ifstream fin;
fin.open('filename.whatever', std::ios::in | std::ios::binary);
if(fin) {
unsigned int m;
while(fin >> m) {
//processing the data and writing to another file here
}
//1: declare Char array
//2: fill it with what remains in the file
//3: fill the rest of it until it's the same size as an unsigned int
} else {
//output to error stream and exit with failure condition
}
The question, at this point, is this: is this truly format-agnostic? In other words, are bytes used to measure file size as discrete units, or can a file be, say, 11.25 bytes in size? I should know this, I know, but I've got to ask it anyway.
Are bytes used to measure file size as discrete units, or can a file be, say, 11.25 bytes in size?
No data type can be less than a byte, and your file is represented as an array of char meaning each character is one byte. Thus it is impossible to not get a whole number measure in bytes.
Here is step one, two, and three as per your post:
while (fin >> m)
{
// ...
}
std::ostringstream buffer;
buffer << fin.rdbuf();
std::string contents = buffer.str();
// fill with 0s
std::fill(contents.begin(), contents.end(), '0');
I have a trouble, my server application sends packet 8 bytes length - AABBCC1122334455 but my application receives this packet in two parts AABBCC1122 and 334455, via "recv" function, how can i fix that?
Thanks!
To sum up a liitle bit:
TCP connection doesn't operate with packets or messages on the application level, you're dealing with stream of bytes. From this point of view it's similar to writing and reading from a file.
Both send and recv can send and receive less data than provided in the argument. You have to deal with it correctly (usually by applying proper loop around the call).
As you're dealing with streams, you have to find the way to convert it to meaningful data in your application. In other words, you have to design serialisation protocol.
From what you've already mentioned, you most probably want to send some kind of messages (well, it's usually what people do). The key thing is to discover the boundaries of messages properly. If your messages are of fixed size, you simply grab the same amount of data from the stream and translate it to your message; otherwise, you need a different approach:
If you can come up with a character which cannot exist in your message, it could be your delimiter. You can then read the stream until you reach the character and it'll be your message. If you transfer ASCII characters (strings) you can use zero as a separator.
If you transfer binary data (raw integers etc.), all characters can appear in your message, so nothing can act as a delimiter. Probably the most common approach in this case is to use fixed-size prefix containing size of your message. Size of this extra field depends on the max size of your message (you will be probably safe with 4 bytes, but if you know what is the maximum size, you can use lower values). Then your packet would look like SSSS|PPPPPPPPP... (stream of bytes), where S is the additional size field and P is your payload (the real message in your application, number of P bytes is determined by value of S). You know every packet starts with 4 special bytes (S bytes), so you can read them as an 32-bit integer. Once you know the size of the encapsulated message, you read all the P bytes. After you're done with one packet, you're ready to read another one from the socket.
Good news though, you can come up with something completely different. All you need to know is how to deserialise your message from a stream of bytes and how send/recv behave. Good luck!
EDIT:
Example of function receiving arbitrary number of bytes into array:
bool recv_full(int sock, char *buffer, size_t size)
{
size_t received = 0;
while (received < size)
{
ssize_t r = recv(sock, buffer + received, size - received, 0);
if (r <= 0) break;
received += r;
}
return received == size;
}
And example of receiving packet with 2-byte prefix defining size of payload (size of payload is then limited to 65kB):
uint16_t msgSize = 0;
char msg[0xffff];
if (recv_full(sock, reinterpret_cast<char *>(&msgSize), sizeof(msgSize)) &&
recv_full(sock, msg, msgSize))
{
// Got the message in msg array
}
else
{
// Something bad happened to the connection
}
That's just how recv() works on most platforms. You have to check the number of bytes you receive and continue calling it in a loop until you get the number that you need.
You "fix" that by reading from TCP socket in a loop until you get enough bytes to make sense to your application.
my server application sends packet 8 bytes length
Not really. Your server sends 8 individual bytes, not a packet 8 bytes long. TCP data is sent over a byte stream, not a packet stream. TCP neither respects nor maintains any "packet" boundary that you might have in mind.
If you know that your data is provided in quanta of N bytes, then call recv in a loop:
std::vector<char> read_packet(int N) {
std::vector buffer(N);
int total = 0, count;
while ( total < N && (count = recv(sock_fd, &buffer[N], N-total, 0)) > 0 )
total += count;
return buffer;
}
std::vector<char> packet = read_packet(8);
If your packet is variable length, try sending it before the data itself:
int read_int() {
std::vector<char> buffer = read_packet(sizeof (int));
int result;
memcpy((void*)&result, (void*)&buffer[0], sizeof(int));
return result;
}
int length = read_int();
std::vector<char> data = read_buffer(length);