Boost async_read doesn't give me an "end of frame" flag - c++

I'm still working on some kind of client for communication with an IP Camera. Now I have the following issue:
I send a request to the camera ( a RTSP DESCRIBEin particular ). Now I get it's answer which looks like this:
RTSP/1.0 200 OK
CSeq: 2
Date: Thu, Jan 01 1970 00:31:41 GMT
Content-Base: rtsp://192.168.0.42/mpeg4?mode=Live&stream=-1&buffer=0&seek=0&fps=100& metainfo=/
Content-Type: application/sdp
Content-Length: 517
This is the header of the answer, followed by a so called Session Description which has the size shown in the field Content-Length. Actually I don't care much for the Session Description , I'm just interested in the Content-Base field. But still, since there is some communication following on the same socket, I need to get rid of all the data.
For receiving'm using the async_read calls from boost::asio.
My code looks ( simplified ) like this:
CommandReadBuffer::CallbackFromAsyncWrite()
{
boost::asio::async_read_until(*m_Socket, m_ReceiveBuffer,"\r\n\r\n",
boost::bind(&CommandReadBuffer::handle_rtsp_describe, this->shared_from_this(),
boost::asio::placeholders::error,boost::asio::placeholders::bytes_transferred));
}
This one reads at least the header ( shown above ) since its terminated by a blank line. As usual for async_write it just reads some more of the data, but nevermind. Now to the next callback function:
void CommandReadBuffer::handle_rtsp_describe(const boost::system::error_code& err,size_t bytesTransferred)
{
std::istream response_stream(&m_ReceiveBuffer);
std::string header;
// Just dump the data on the console
while (std::getline(response_stream, header))
{
// Normally I would search here for the desired content-base field
std::cout << header << "\n";
}
boost::asio::async_read(*m_Socket, m_ReceiveBuffer, boost::asio::transfer_at_least(1),
boost::bind(&CommandReadBuffer::handle_rtsp_setup, this->shared_from_this(),
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
Now this works fine as well, if I print out the number of received bytes it's always 215.
Now we go on to the critical callback:
void CommandReadBuffer::handle_rtsp_setup(const boost::system::error_code& err, size_t bytesTransferred)
{
std::cout << "Error: " << err.message() << "\n";
if (!err)
{
// Write all of the data that has been read so far.
std::cout << &m_ReceiveBuffer;
// Continue reading remaining data until EOF.
m_DeadlineTimer->async_wait(boost::bind(&CommandReadBuffer::handleTimeout, this->shared_from_this(),boost::asio::placeholders::error));
boost::asio::async_read(*m_Socket, m_ReceiveBuffer, boost::asio::transfer_at_least(1),
boost::bind(&CommandReadBuffer::handle_rtsp_setup, this->shared_from_this(),
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
else if (err != boost::asio::error::eof)
{
std::cout << "Error: " << err.message() << "\n";
}
else
{
std::cout << "End of Frame " << err.message() << "\n";
}
}
This part reads 220 Bytes. If I look at console Output from this call and compare it with the actualy payload of the frame ( as seen in Wireshark ) I can see that all data has been received. Now I would actually assume that async_read would set me the eof error. But instead the return code of error is success and so it calls async_read again. This time there is no data to be received and it never calls the callback function ( since there will be no more incoming data ).
Now I actually don't know how I could determine that all data has been sent. Actually I would expect the error flag to be set.
Now this is very similar to the implementation of the Boost Example for an Async HTTP client. Also it is done the same way in the Example Boost Async HTTP Client. I implemented this in another call and there it actually works.
Now in my opinion it should make no difference for the async_read call wether it is HTTP or RTSP - end of frame is end of frame, if there is no more data to read.
I'm also aware that according to the boost documentation I am using
void async_read(
AsyncReadStream & s,
basic_streambuf< Allocator > & b,
CompletionCondition completion_condition,
ReadHandler handler);
which means the function will continue until
The supplied buffer is full (that is, it has reached maximum size).
The completion_condition function object returns 0.
So if there is no more data to read, it just continues.
But I also tried the overloaded function without the CompletionCondition parameter, which should return when an error occurs ( EOF !!! ) - But this just won't callback either...
Any suggestions? I just don't get what I'm doing wrong...

I have written an RTSP client and server library using boost asio and can offer the following advice:
The RTSP message syntax is generic: there is no need for different DESCRIBE and SETUP handlers. In general
write an RTSP request
to read the response do a boost::asio::async_read_until("\r\n\r\n")
then check for the Content-Length header
if content_length > 0 do a boost::asio::transfer_at_least(content_length)
Further, why are you expecting an EOF? The connection is still open: the server is waiting for either another SETUP or a PLAY request and typically won't close the connection until the RTSP TCP connection has been timed out, which has a default value of 60 seconds according to RFC2326.
If in your application, you have completed interaction with the RTSP server, close the connection after you have read the response.

Related

boost aiso icmp socket: is having async_receive_from got called repeatedly sufficient?

when reading the official boost icmp aync example https://www.boost.org/doc/libs/1_78_0/doc/html/boost_asio/example/cpp03/icmp/ping.cpp:
void start_receive()
{
// Discard any data already in the buffer.
reply_buffer_.consume(reply_buffer_.size());
// Wait for a reply. We prepare the buffer to receive up to 64KB.
socket_.async_receive(reply_buffer_.prepare(65536),
boost::bind(&pinger::handle_receive, this, _2));
}
void handle_receive(std::size_t length)
{
// The actual number of bytes received is committed to the buffer so that we
// can extract it using a std::istream object.
reply_buffer_.commit(length);
// Decode the reply packet.
std::istream is(&reply_buffer_);
ipv4_header ipv4_hdr;
icmp_header icmp_hdr;
is >> ipv4_hdr >> icmp_hdr;
// We can receive all ICMP packets received by the host, so we need to
// filter out only the echo replies that match the our identifier and
// expected sequence number.
if (is && icmp_hdr.type() == icmp_header::echo_reply && icmp_hdr.identifier() == get_identifier() && icmp_hdr.sequence_number() == sequence_number_)
{
// If this is the first reply, interrupt the five second timeout.
if (num_replies_++ == 0)
timer_.cancel();
// Print out some information about the reply packet.
chrono::steady_clock::time_point now = chrono::steady_clock::now();
chrono::steady_clock::duration elapsed = now - time_sent_;
std::cout << length - ipv4_hdr.header_length()
<< " bytes from " << ipv4_hdr.source_address()
<< ": icmp_seq=" << icmp_hdr.sequence_number()
<< ", ttl=" << ipv4_hdr.time_to_live()
<< ", time="
<< chrono::duration_cast<chrono::milliseconds>(elapsed).count()
<< std::endl;
}
start_receive();
}
I've got some questions:
what if in handle_receive,I do some time-consuming work, say, acquire some lock just before the start_recieve got called again, will the demo example miss some icmp packets? I have this question because after some experiments, I found handle_receive got called only once instead of repeatedly after async handler registered, so you have to register the handler everytime handle_receive got fired
what can i do to avoid this situation?
what if in handle_receive,I do some time-consuming work, [...], will the demo example miss some icmp packets?
Yes. It will add latency. If the latency exceeds 5s, the request will be timed out.
say, acquire some lock just before the start_recieve got called again
That's a problem. The lock will not guard any async operations anyways. You have a flaw in your logic there. It's hard to tell without seeing the code or knowing what you'd use the lock for.
what can i do to avoid this situation?
Instead of locking, use a strand.
Instead of doing time-consuming work in a completion handler, do it on a different thread.

BOOST::ASIO - UDP - endpoint gets overwritten

I am trying to implement some keep-alive service in UDP using BOOST::ASIO, these are the general steps:
Sending keep-alives to 2 processes on the same machine, they are listening on the same ip with a different port.
Loop to send async_send_to to both, and the callback is a function that calls async_receive_from with a callback F().
Both refer to the same endpoint and data buffers.
while loop with io_service.run_one() inside.
The processes reply immediately.
The issue is that sporadically I either get the 2 differing ports when I check the endpoints' ports (the wanted case) F() runs, or, I get twice the same port.
It seems as the endpoint buffer (and probably the data) is getting overwritten by the later packet.
I was thinking the since I'm using run_one() the packets should be processed one by one and there will be no overwriting.
Initial send -
void GetInstancesHeartbeat(udp::endpoint &sender_endpoint)
{
int instanceIndex = 0;
for (; instanceIndex <= amountOfInstances ; instanceIndex++)
{
udp::endpoint endpoint = udp::endpoint(IP, Port+ instanceIndex);
m_instancesSocket->async_send_to(
boost::asio::buffer((char*)&(message),
sizeof(message)),endpoint,
boost::bind(&ClusterManager::handle_send_to_instance,
this, boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred,
sender_endpoint));
}
}
Then the handler -
void handle_send_to_instance(const boost::system::error_code& error, size_t
bytes_recvd, udp::endpoint &sender_endpoint)
{
m_instancesSocket->async_receive_from(
boost::asio::buffer(m_dataBuffer, m_maxLength), m_endpoint,
boost::bind(&ClusterManager::handle_receive_from_instance, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred,
sender_endpoint));
}
While loop -
while(true){
io_service.run_one();
}
And the handle receive where the port results twice the same -
void handle_receive_from_instance(const boost::system::error_code& error, size_t
bytes_recvd, udp::endpoint&sender_endpoint)
{
if (!error && bytes_recvd > 0)
{
int instancePort = m_endpoint.port();
} else {
//PRINT ERROR
}
}
The actual operations are asynchronous, so there's no telling when the endpoint reference gets written to. That's the nature of asynchronous calls.
So, what you need to have is an endpoint receiving variable per asynchronous call (you might store it per instance index).
There are a number of other really suspicious bits:
what's the type of message? For most types you'd write just boost::asio::buffer(message) (which deals with T [], std::vector<T>, array<T> etc). This works when T is char or any POD type.
If message is actually a struct of some type, consider using a single-element array to avoid having to dangerous casting:
Live On Coliru
POD message[1] = {pod};
s.async_send_to(boost::asio::buffer(message), udp::endpoint{{}, 6767}, [](boost::system::error_code ec, size_t transferred) {
std::cout << "Transferred: " << transferred << " (" << ec.message() << ")\n";
});
(Sends 12 bytes on a typical system).
Whatever you do, don't write the unsafe C-style cast (Why use static_cast<int>(x) instead of (int)x?).
You have while(true) { io.run_one(); } which is an infinite loop. A better way to write it would be: while(io.run_one()) {}
However, that would basically be the same as io.run();, but less correctly and less efficiently (see https://www.boost.org/doc/libs/1_68_0/boost/asio/detail/impl/scheduler.ipp line 138), so why not use it?

TCP client in Boost asio

Im building a TCP client using Boost::asio Libs. My program has a write() thread that sends a command to the server
write(*_socket,boost::asio::buffer("sspi l1\n\n",sizeof("sspi l1\n\n")));
Then a read thread is started that reads from the buffer all the time, as there can be messages broadcasted from the server due to any other client
void TCP_IP_Connection::readTCP()
{
size_t l=0;
this->len=0;
boost::system::error_code error;
try
{//loop reading all values from router
while(1)
{
//wait for reply??
l=_socket->read_some(boost::asio::buffer(this->reply,sizeof(this->reply)),error);
if(error)
throw boost::system::system_error(error);
if(l>0)
{
this->dataProcess(l);
}
else
boost::this_thread::sleep(boost::posix_time::milliseconds(5000));
_io.run();
if(error==boost::asio::error::eof) //connection closed by router
std::cout<<"connection closed by router";
_io.reset();
}
}
catch (std::exception& e)
{
std::cerr << e.what() << std::endl;
}
}
This thread runs al time in a while(1) loop and is supposed to sleep when the received data length is less than zero. It reads all the data and calls the data parser function. After that the write thread is used to send another command, with read thread running. But instead of the required response the server sends back
? ""
ERROR: Unknown command
I tried using the wireshark. I can see the command being send properly. What can be mistake I'm doing here?
sizeof("sspi l1\n\n") returns 10, but I can only count 9 characters in that string.
Try this instead:
const std::string cmd("sspi l1\n\n");
write(*_socket,boost::asio::buffer(cmd, cmd.length()));
Or when you have it as a string it is enough to do
const std::string cmd("sspi l1\n\n");
write(*_socket,boost::asio::buffer(cmd));
The second argument specifies a maximum length of the string to use. But since it is a constant string, the second argument is not strictly necessary.

boost::asio async server design

Currently I'm using design when server reads first 4 bytes of stream then read N bytes after header decoding.
But I found that time between first async_read and second read is 3-4 ms. I just printed in console timestamp from callbacks for measuring. I sent 10 bytes of data in total. Why it takes so much time to read?
I running it in debug mode but I think that 1 connection for debug is
not so much to have a 3 ms delay between reads from socket. Maybe I need
another approach to cut TCP stream on "packets"?
UPDATE: I post some code here
void parseHeader(const boost::system::error_code& error)
{
cout<<"[parseHeader] "<<lib::GET_SERVER_TIME()<<endl;
if (error) {
close();
return;
}
GenTCPmsg::header result = msg.parseHeader();
if (result.error == GenTCPmsg::parse_error::__NO_ERROR__) {
msg.setDataLength(result.size);
boost::asio::async_read(*socket,
boost::asio::buffer(msg.data(), result.size),
(*_strand).wrap(
boost::bind(&ConnectionInterface::parsePacket, shared_from_this(), boost::asio::placeholders::error)));
} else {
close();
}
}
void parsePacket(const boost::system::error_code& error)
{
cout<<"[parsePacket] "<<lib::GET_SERVER_TIME()<<endl;
if (error) {
close();
return;
}
protocol->parsePacket(msg);
msg.flush();
boost::asio::async_read(*socket,
boost::asio::buffer(msg.data(), config::HEADER_SIZE),
(*_strand).wrap(
boost::bind(&ConnectionInterface::parseHeader, shared_from_this(), boost::asio::placeholders::error)));
}
As you see unix timestamps differ in 3-4 ms. I want to understand why so many time elapse between parseHeader and parsePacket. This is not a client problem, summary data is 10 bytes, but i cant sent much much more, delay is exactly between calls. I'm using flash client version 11. What i do is just send ByteArray through opened socket. I don't sure that delays on client. I send all 10 bytes at once. How can i debug where actual delay is?
There are far too many unknowns to identify the root cause of the delay from the posted code. Nevertheless, there are a few approaches and considerations that can be taken to help to identify the problem:
Enable handler tracking for Boost.Asio 1.47+. Simply define BOOST_ASIO_ENABLE_HANDLER_TRACKING and Boost.Asio will write debug output, including timestamps, to the standard error stream. These timestamps can be used to help filter out delays introduced by application code (parseHeader(), parsePacket(), etc.).
Verify that byte-ordering is being handled properly. For example, if the protocol defines the header's size field as two bytes in network-byte-order and the server is handling the field as a raw short, then upon receiving a message that has a body size of 10:
A big-endian machine will call async_read reading 10 bytes. The read operation should complete quickly as the socket already has the 10 byte body available for reading.
A little-endian machine will call async_read reading 2560 bytes. The read operation will likely remain outstanding, as far more bytes are trying to be read than is intended.
Use tracing tools such as strace, ltrace, etc.
Modify Boost.Asio, adding timestamps throughout the callstack. Boost.Asio is shipped as a header-file only library. Thus, users may modify it to provide as much verbosity as desired. While not the cleanest or easiest of approaches, adding a print statement with timestamps throughout the callstack may help provide visibility into timing.
Try duplicating the behavior in a short, simple, self contained example. Start with the simplest of examples to determine if the delay is systamtic. Then, iteratively expand upon the example so that it becomes closer to the real-code with each iteration.
Here is a simple example from which I started:
#include <iostream>
#include <boost/array.hpp>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/enable_shared_from_this.hpp>
#include <boost/make_shared.hpp>
#include <boost/shared_ptr.hpp>
class tcp_server
: public boost::enable_shared_from_this< tcp_server >
{
private:
enum
{
header_size = 4,
data_size = 10,
buffer_size = 1024,
max_stamp = 50
};
typedef boost::asio::ip::tcp tcp;
public:
typedef boost::array< boost::posix_time::ptime, max_stamp > time_stamps;
public:
tcp_server( boost::asio::io_service& service,
unsigned short port )
: strand_( service ),
acceptor_( service, tcp::endpoint( tcp::v4(), port ) ),
socket_( service ),
index_( 0 )
{}
/// #brief Returns collection of timestamps.
time_stamps& stamps()
{
return stamps_;
}
/// #brief Start the server.
void start()
{
acceptor_.async_accept(
socket_,
boost::bind( &tcp_server::handle_accept, this,
boost::asio::placeholders::error ) );
}
private:
/// #brief Accept connection.
void handle_accept( const boost::system::error_code& error )
{
if ( error )
{
std::cout << error.message() << std::endl;
return;
}
read_header();
}
/// #brief Read header.
void read_header()
{
boost::asio::async_read(
socket_,
boost::asio::buffer( buffer_, header_size ),
boost::bind( &tcp_server::handle_read_header, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred ) );
}
/// #brief Handle reading header.
void
handle_read_header( const boost::system::error_code& error,
std::size_t bytes_transferred )
{
if ( error )
{
std::cout << error.message() << std::endl;
return;
}
// If no more stamps can be recorded, then stop the async-chain so
// that io_service::run can return.
if ( !record_stamp() ) return;
// Read data.
boost::asio::async_read(
socket_,
boost::asio::buffer( buffer_, data_size ),
boost::bind( &tcp_server::handle_read_data, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred ) );
}
/// #brief Handle reading data.
void handle_read_data( const boost::system::error_code& error,
std::size_t bytes_transferred )
{
if ( error )
{
std::cout << error.message() << std::endl;
return;
}
// If no more stamps can be recorded, then stop the async-chain so
// that io_service::run can return.
if ( !record_stamp() ) return;
// Start reading header again.
read_header();
}
/// #brief Record time stamp.
bool record_stamp()
{
stamps_[ index_++ ] = boost::posix_time::microsec_clock::local_time();
return index_ < max_stamp;
}
private:
boost::asio::io_service::strand strand_;
tcp::acceptor acceptor_;
tcp::socket socket_;
boost::array< char, buffer_size > buffer_;
time_stamps stamps_;
unsigned int index_;
};
int main()
{
boost::asio::io_service service;
// Create and start the server.
boost::shared_ptr< tcp_server > server =
boost::make_shared< tcp_server >( boost::ref(service ), 33333 );
server->start();
// Run. This will exit once enough time stamps have been sampled.
service.run();
// Iterate through the stamps.
tcp_server::time_stamps& stamps = server->stamps();
typedef tcp_server::time_stamps::iterator stamp_iterator;
using boost::posix_time::time_duration;
for ( stamp_iterator iterator = stamps.begin() + 1,
end = stamps.end();
iterator != end;
++iterator )
{
// Obtain the delta between the current stamp and the previous.
time_duration delta = *iterator - *(iterator - 1);
std::cout << "Delta: " << delta.total_milliseconds() << " ms"
<< std::endl;
}
// Calculate the total delta.
time_duration delta = *stamps.rbegin() - *stamps.begin();
std::cout << "Total"
<< "\n Start: " << *stamps.begin()
<< "\n End: " << *stamps.rbegin()
<< "\n Delta: " << delta.total_milliseconds() << " ms"
<< std::endl;
}
A few notes about the implementation:
There is only one thread (main) and one asynchronous chain read_header->handle_read_header->handle_read_data. This should minimize the amount of time a ready-to-run handler spends waiting for an available thread.
To focus on boost::asio::async_read, noise is minimized by:
Using a pre-allocated buffer.
Not using shared_from_this() or strand::wrap.
Recording the timestamps, and perform processing post-collection.
I compiled on CentOS 5.4 using gcc 4.4.0 and Boost 1.50. To drive the data, I opted to send 1000 bytes using netcat:
$ ./a.out > output &
[1] 18623
$ echo "$(for i in {0..1000}; do echo -n "0"; done)" | nc 127.0.0.1 33333
[1]+ Done ./a.out >output
$ tail output
Delta: 0 ms
Delta: 0 ms
Delta: 0 ms
Delta: 0 ms
Delta: 0 ms
Delta: 0 ms
Total
Start: 2012-Sep-10 21:22:45.585780
End: 2012-Sep-10 21:22:45.586716
Delta: 0 ms
Observing no delay, I expanded upon the example by modifying the boost::asio::async_read calls, replacing this with shared_from_this() and wrapping the ReadHandlerss with strand_.wrap(). I ran the updated example and still observed no delay. Unfortunately, that is as far as I could get based on the code posted in the question.
Consider expanding upon the example, adding in a piece from the real implementation with each iteration. For example:
Start with using the msg variable's type to control the buffer.
Next, send valid data, and introduce parseHeader() and parsePacket functions.
Finally, introduce the lib::GET_SERVER_TIME() print.
If the example code is as close as possible to the real code, and no delay is being observed with boost::asio::async_read, then the ReadHandlers may be ready-to-run in the real code, but they are waiting on synchronization (the strand) or a resource (a thread), resulting in a delay:
If the delay is the result of synchronization with the strand, then consider Robin's suggestion by reading a larger block of data to potentially reduce the amount of reads required per-message.
If the delay is the result of waiting for a thread, then consider having an additional thread call io_service::run().
One thing that makes Boost.Asio awesome is using the async feature to the fullest. Relying on a specific number of bytes read in one batch, possibly ditching some of what could already been read, isn't really what you should be doing.
Instead, look at the example for the webserver especially this: http://www.boost.org/doc/libs/1_51_0/doc/html/boost_asio/example/http/server/connection.cpp
A boost triboolean is used to either a) complete the request if all data is available in one batch, b) ditch it if it's available but not valid and c) just read more when the io_service chooses to if the request was incomplete. The connection object is shared with the handler through a shared pointer.
Why is this superior to most other methods? You can possibly save the time between reads already parsing the request. This is sadly not followed through in the example but idealy you'd thread the handler so it can work on the data already available while the rest is added to the buffer. The only time it's blocking is when the data is incomplete.
Hope this helps, can't shed any light on why there is a 3ms delay between reads though.

Simplest QT TCP client

I would like to connect to a listening server and transmit some data. I looked at the examples available but they seem to have extra functions that do not seem very helpful to me (i.e. connect, fortune, etc.). This is the code I have so far:
QTcpSocket t;
t.connectToHost("127.0.0.1", 9000);
Assuming the server is listening and robust, what do I need to implement to send a data variable with datatype QByteArray?
very simple with QTcpSocket. Begin as you did...
void MainWindow::connectTcp()
{
QByteArray data; // <-- fill with data
_pSocket = new QTcpSocket( this ); // <-- needs to be a member variable: QTcpSocket * _pSocket;
connect( _pSocket, SIGNAL(readyRead()), SLOT(readTcpData()) );
_pSocket->connectToHost("127.0.0.1", 9000);
if( _pSocket->waitForConnected() ) {
_pSocket->write( data );
}
}
void MainWindow::readTcpData()
{
QByteArray data = pSocket->readAll();
}
Be aware, though, that for reading from the TcpSocket you may receive the data in more than one transmission, ie. when the server send you the string "123456" you may receive "123" and "456". It is your responsibility to check whether the transmission is complete. Unfortunately, this almost always results in your class being stateful: the class has to remember what transmission it is expecting, whether it has started already and if it's complete. So far, I haven't figured out an elegant way around that.
In my case I was reading xml data, and sometimes I would not get all in one packet.
Here is an elegant solution. WaitForReadyRead could also have a time out in it and
then some extra error checking in case that timeout is reached. In my case I should never
receive an incomplete xml, but if it did happen this would lock the thread up indefinetly
without the timeout:
while(!xml.atEnd()) {
QXmlStreamReader::TokenType t = xml.readNext();
if(xml.error()) {
if(xml.error() == QXmlStreamReader::PrematureEndOfDocumentError) {
cout << "reading extra data" << endl;
sock->waitForReadyRead();
xml.addData(sock->readAll());
cout << "extra data successful" << endl;
continue;
} else {
break;
}
}
...