I have a server thats sending out data records as strings of varying length(for eg, 79,80,81,82)
I want to be able to receive exactly one record at a time.I've delimited records with a (r) but because I dont know howmany bytes I have to receive, It sometimes merges records and makes it difficult for me to process.
I have two ideas for you:
Use XML for the protocol. This way you know exactly when each message ends.
Send in the header of each "packet" the packet size, this way you know how much to read from the socket for this specific packet.
Edit:
Look at this dummy code for (2)
int buffer_size;
char* buffer;
read( socket, &buffer_size, sizeof(buffer_size));
buffer = (char*) malloc(packet_size);
read( socket, buffer, buffer_size );
// do something
free( buffer) ;
EDIT:
I recommend looking at the comments here, as they note that the contect might not be ready by a simple "read()", you need to keep "read()"ing, until you get the correct buffer size.
Also - you might not need to read the size. Basically you need to look for the ending top level tag of the XML. This can be done by parsing the whole XML, or parlty parsing the XML you get from the stream untill you have 0 nodes "open".
You should delimit with null byte. Show us your code, and we may be able to help you.
Stream sockets do not natively support an idea of a "record" - the abstraction they provide is that of a continuous stream.
You must implement a layer on top of them to provide "records". It sounds like you are already part way there, with the end-of-record delimiter. The pseudo-code to complete it is:
create empty buffer;
forever {
recv data and append to buffer;
while (buffer contains end-of-record marker) {
remove first record from buffer and process it;
move remaining data to beginning of buffer;
}
}
Are you sending your data as a stream?
You can send it as a structure which is easier to parse and retrieve the data from.
struct Message
{
int dataSize;
char data[256];
};
Related
I want to implement a client for a sensor that sends data over tcp and uses the following protocol:
the message-header starts with the byte-sequence 0xAFFEC0CC2 of type uint32
the header in total is 24 Bytes long (including the start sequence) and contains the size in bytes of the message-body as a uint32
the message-body is sent directly after the header and not terminated by a demimiter
Currently, I got the following code (assume a connected socket exists)
typedef unsigned char byte;
boost::system::error_code error;
boost::asio::streambuf buf;
std::string magic_word_s = {static_cast<char>(0xAF), static_cast<char>(0xFE),
static_cast<char>(0xC0), static_cast<char>(0xC2)};
ssize_t n = boost::asio::read_until(socket_, buf, magic_word_s, error);
if(error)
std::cerr << boost::system::system_error(error).what() << std::endl;
buf.consume(n);
n = boost::asio::read(socket_, buf, boost::asio::transfer_exactly(20);
const byte * p = boost::asio::buffer_cast<const byte>(buf.data());
uint32_t size_of_body = *((byte*)p);
unfortunately the documentation for read_until remarks:
After a successful read_until operation, the streambuf may contain additional data beyond the delimiter. An application will typically leave that data in the streambuf for a subsequent read_until operation to examine.
which means that I loose synchronization with the described protocol.
Is there an elegant way to solve this?
Well... as it says... you just "leave" it in the object, or temporary store it in another, and handle the whole message (below called 'packet') if it is complete.
I have a similar approach in one of my projects. I'll explain a little how I did it, that should give you a rough idea how you can handle the packets correctly.
In my Read-Handler (-callback) I keep checking if the packet is complete. The meta-data information (header for you) is temporary stored in a map associated with the remote-partner (map<RemoteAddress, InfoStructure>).
For example it can look like this:
4 byte identifier
4 byte message-length
n byte message
Handle incoming data, check if identifier + message-length are received already, continue to check if message-data is completed with received data.
Leave rest of the packet in the temporary buffer, erase old data.
Continue with handling when next packet arrives or check if received data completes next packet already...
This approach may sound a little slow, but I get even with SSL 10MB/s+ on a slow machine.
Without SSL much higher transfer-rates are possible.
With this approach, you may also take a look into read_some or its asynchronous version.
I got simple server-client app.
The idia is:
Client send to server string with sql-request.
Server process the request and send back on client answer with data formated in csv style string.
Everything is already adjusted accept one thing. The problem is to read answer on client cause I don't know the size of recieved string.
Code is bellow:
Server:
std::string answer;
answer = sql_read(*msg); //get the string with data from request
clientSock->write_some(buffer(answer, answer.size())); //send it back on client
Client:
std::string answer;
bytesRead = sock->read_some(boost::asio::buffer(readBuf, inputSize)); //read the answer, but how do I know correct size of string?
string_ptr msg(new std::string(readBuf, bytesRead));
answer = *msg;
How to read the resulting string without knowing its size?
Or if my aproach (with sending data in string) is radically wrong how to do it in correct way?
You must handle this yourself in the protocol the client and server use to speak to each other.
One way is to use a deliminator and use boost::asio::read_until(). Example:
boost::asio::streambuf buf;
boost::asio::read_until (sock, buf, "\r\n"); // or '\0' if zero terminated
Check the Boost.Asio documentation.
Another way is to send the size in an fixed length header before sending the variable length part.
You won't know the size of the string in advance. Where you do the read_some you will need to do a loop to read all the data from the socket. You need to check the number of bytes read and the size of the buffer you allocated. If they are equal then you filled up your read buffer and you need to do another read to check for any remaining data.
I'm using boost 1.53 and serialization to transfer an array of 520 floats over TCP/IP. I put a debug code printout to see the amount of data to be send : it's about 5 K. No problem for me here, but this value somehow depends on the actual data to be serialized. It could be 5400, 5500 and so on.
The question is : what is the right way to receive such data block? For the moment I use read_some() call. But as I've figured out it doesn't guarantee that the whole serialized block of data will be read out. Am I wrong?
How to ensure that there will be a complete archive at RX side? Is there any exception to be thrown when it is not possible to deserialize a chunk of data?
as far as tcpip packet can be received to a number of smaller packets so I'd recommend to add some additional data to tcpip
something like this:
serialize you data to stream
get size of stream
send to tcpip buffer starting with size of stream and then data from the stream
receiver reads size and then reads the rest of the packet.
after you received the full packet - call deserialization
Yes. read_some is potentially a no-op on conforming implementations[1].
Instead do a loop using read() and gcount(), like:
std::istream& is = gotten_from_somewhere_or_a_parameter();
std::vector<byte> v(256);
std::streamsize bytes_read;
do
{
is.read(v.data(),v.size());
bytes_read = stream.gcount ();
// do something with the bytes read
} while(bytes_read);
[1] Notably, gcc's standard library implementation seems to always return something for std::filebuf but on MSVC, the first call will simply always return 0 bytes read :)
Now the server (implemented with java) will send some stream data to me, my code is like below:
connect(socket, SIGNAL(readyRead()), this, SLOT(read_from_server()));
in the read_from_server():
{
while (socket->bytesAvailable())
{
QString temp = socket->readAll();
}
}
but I find that even the server sent me a string with only several characters, the data is truncated, and my function is called twice, thus temp is the never complete data that I want.
If server send me a longer string, my function may be called three or more times, making me diffficult to know at which time the data is transfered completely.
So anyone can tell me how to completely receive the data easily, without so many steps of bothering? I'm sorry if this is duplicated with some questions else, I couldn't get their answers work for me. Many thanks!
What you're seeing is normal for client-server communication. Data is sent in packets and the readyRead signal is informing your program that there is data available, but has no concept of what or how much data there is, so you have to handle this.
To read the data correctly, you will need a buffer, as mentioned by #ratchetfreak, to append the bytes as they're read from the stream. It is important that you know the format of the data being sent, in order to know when you have a complete message. I have previously used at least two methods to do this: -
1) Ensure that sent messages begin with the size, in bytes, of the message being sent. On receiving data, you start by reading the size and keep appending to your buffer until it totals the size to expect.
2) Send all data in a known format, such as JSON or XML, which can be checked for the end of the message. For example, in the case of JSON, all packets will begin with an opening brace '{' and end with a closing brace '}', so you could count braces and match up the data, or use QJsonDocument::fromRawData to verify that the data is complete.
Having used both of these methods, I recommend using the first; include the size of a message that is being sent.
you can use a buffer field to hold the unfinished data temporarily and handle packets as they complete:
{
while (socket->bytesAvailable())
{
buffer.append(socket->readAll());
int packetSize = getPacketSize(buffer);
while(packetSize>0)
{
handlePacket(buffer.left(packetSize);
buffer.remove(0,packetSize);
packetSize = getPacketSize(buffer);
}
}
}
If all of the data has not yet arrived then your while loop will exit prematurely. You need to use a message format that will let the receiving code determine when the complete message has been received. For example, the message could begin with a length element, or if you are dealing with text the message could end with some character used as a terminator.
Problem is that during tcp data transfer data are send in undefined chunks. If you are trying to read defined block size you have to know in advance expected chunk size ore have a way to determinate when your block ends (something like zero terminated c-string).
Check if this answer doesn't help you (there is a trick to wait for expected data block).
I have a TCP client connecting to my server which is sending raw data packets. How, using Boost.Asio, can I get the "whole" packet every time (asynchronously, of course)? Assume these packets can be any size up to the full size of my memory.
Basically, I want to avoid creating a statically sized buffer.
Typically when you build a custom protocol on the top of TCP/IP you use a simple message format where first 4 bytes is an unsigned integer containing the message length and the rest is the message data. If you have such a protocol then the reception loop is as simple as below (not sure what is ASIO notation, so it's just an idea)
for(;;) {
uint_32_t len = 0u;
read(socket, &len, 4); // may need multiple reads in non-blocking mode
len = ntohl(len);
assert (len < my_max_len);
char* buf = new char[len];
read(socket, buf, len); // may need multiple reads in non-blocking mode
...
}
typically, when you do async IO, your protocol should support it.
one easy way is to prefix a byte array with it's length at the logical level, and have the reading code buffer up until it has a full buffer ready for parsing.
if you don't do it, you will end up with this logic scattered all over the place (think about reading a null terminated string, and what it means if you just get a part of it every time select/poll returns).
TCP doesn't operate with packets. It provides you one contiguous stream. You can ask for the next N bytes, or for all the data received so far, but there is no "packet" boundary, no way to distinguish what is or is not a packet.