I am writing a program using boost asio to receive multicase messages from around 30 multicase ip in linux with c++. I am here to seek advances on how to minimize packet drop from my client side during runtime. I have already maximized the NIC receive buffer. I am using a 8 core cpu. I am also wondering will the NIC card create same number of buffer queue to equal to number of sockets in the program? Beside configure the NIC card, could I do something on the linux kernel? Since I believe kernel will do buffer copy from the NIC first before our program copy data from it, right?
template<typename msg, int id>
void data_service<msg, id>::on_rt_recv( char* p_raw_packet, int p_length, const boost::system::error_code& error )
{
if (!error)
{
//post to strand and wait to proceed
processing_strand_.post(boost::bind(&data_service::on_rt_recv_handler, this,
p_raw_packet,
p_length));
//continue to listen as soon as possible
auto new_buffer = get_new_buffer();
rt_socket_[p_line]->async_receive_from(boost::asio::buffer(new_buffer, BUFFER_SIZE_), rt_endpoint_,
boost::bind(&data_service::on_rt_recv, this,
new_buffer,
boost::asio::placeholders::bytes_transferred,
boost::asio::placeholders::error));
}
else if (error != boost::asio::error::operation_aborted)
{
memory_pool_.free((void*)p_raw_packet);
}
}
Packet loss issue was caused by hardware, including switch, and NIC cards. the packet rate is actually 2500 * 70 /sec because there were 70 udp sockets. Highly recommend a network monitoring tool call Wireshark which provide load of information regarding to your current network traffic.
Regarding to Demon solution, Boost asio under the hook use iocp in window, and epoll in unix.
No buffer size need to adjust as well.
Related
I am writing a server that accepts data from a device and processes it. Everything works fine unless there is an interruption in the network (i.e., if I unplug the Ethernet cable, then reconnect it). I'm using read_until() because the protocol that the device uses terminates the packet with a specific sequence of bytes. When the data stream is interrupted, read_until() blocks, as expected. However when the stream starts up again, it remains blocked. If I look at the data stream with Wireshark, the device continues transmitting and each packet is being ACK'ed by the network stack. But if I look at bytes_readable it is always 0. How can I detect the interruption and how to re-establish a connection to the data stream? Below is a code snippet and thanks in advance for any help you can offer. [Go easy on me, this is my first Stack Overflow question....and yes I did try to search for an answer.]
using boost::asio::ip::tcp;
boost::asio::io_service IOservice;
tcp::acceptor acceptor(IOservice, tcp::endpoint(tcp::v4(), listenPort));
tcp::socket socket(IOservice);
acceptor.accept(socket);
for (;;)
{
len = boost::asio::read_until(socket, sbuf, end);
// Process sbuf
// etc.
}
Remember, the client initiates a connection, so the only thing you need to achieve is to re-create the socket and start accepting again. I will keep the format of your snippet but I hope your real code is properly encapsulated.
using SocketType = boost::asio::ip::tcp::socket;
std::unique_ptr<SocketType> CreateSocketAndAccept(
boost::asio::io_service& io_service,
boost::asio::ip::tcp::acceptor& acceptor) {
auto socket = std::make_unique<boost::asio::ip::tcp::socket>(io_service);
boost::system::error_code ec;
acceptor.accept(*socket.get(), ec);
if (ec) {
//TODO: Add handler.
}
return socket;
}
...
auto socket = CreateSocketAndAccept(IOservice, acceptor);
for (;;) {
boost::system::error_code ec;
auto len = boost::asio::read_until(*socket.get(), sbuf, end, ec);
if (ec) // you could be more picky here of course,
// e.g. check against connection_reset, connection_aborted
socket = CreateSocketAndAccept(IOservice, acceptor);
...
}
Footnote: Should go without saying, socket needs to stay in scope.
Edit: Based on the comments bellow.
The listening socket itself does not know whether a client is silent or whether it got cut off. All operations, especially synchronous, should impose a time limit on completion. Consider setting SO_RCVTIMEO or SO_KEEPALIVE (per socket, or system wide, for more info How to use SO_KEEPALIVE option properly to detect that the client at the other end is down?).
Another option is to go async and implement a full fledged "shared" socket server (BOOST example page is a great start).
Either way, you might run into data consistency issues and be forced to deal with it, e.g. when the client detects an interrupted connection, it would resend the data. (or something more complex using higher level protocols)
If you want to stay synchronous, the way I've seen things handled is to destroy the socket when you detect an interruption. The blocking call should throw an exception that you can catch and then start accepting connections again.
for (;;)
{
try {
len = boost::asio::read_until(socket, sbuf, end);
// Process sbuf
// etc.
}
catch (const boost::system::system_error& e) {
// clean up. Start accepting new connections.
}
}
As Tom mentions in his answer, there is no difference between inactivity and ungraceful disconnection so you need an external mechanism to detect this.
If you're expecting continuous data transfer, maybe a timeout per connection on the server side is enough. A simple ping could also work. After accepting a connection, ping your client every X seconds and declare the connection dead if he doesn't answer.
I have a very simple wrapper for boost::asio sockets sending multicast messages:
// header
class MulticastSender
{
public:
/// Constructor
/// #param ip - The multicast address to broadcast on
/// #param port - The multicast port to broadcast on
MulticastSender(const String& ip, const UInt16 port);
/// Sends a multicast message
/// #param msg - The message to send
/// #param size - The size of the message (in bytes)
/// #return number of bytes sent
size_t send(const void* msg, const size_t size);
private:
boost::asio::io_service m_service;
boost::asio::ip::udp::endpoint m_endpoint;
boost::asio::ip::udp::socket m_socket;
};
// implementation
inline MulticastSender::MulticastSender(const String& ip, const UInt16 port) :
m_endpoint(boost::asio::ip::address_v4::from_string(ip), port),
m_socket(m_service, m_endpoint.protocol())
{
m_socket.set_option(boost::asio::socket_base::send_buffer_size(8 * 1024 * 1024));
m_socket.set_option(boost::asio::socket_base::broadcast(true));
m_socket.set_option(boost::asio::socket_base::reuse_address(true));
}
inline size_t MulticastSender::send(const void* msg, const size_t size)
{
try
{
return m_socket.send_to(boost::asio::buffer(msg, size), m_endpoint);
}
catch (const std::exception& e)
{
setError(e.what());
}
return 0;
}
// read and send a message
MulticastSender sender(ip, port);
while(readFile(&msg)) sender.send(&msg, sizeof(msg));
When compiled on Windows 7 using Visual Studio 2013, I get throughput of ~11 MB/s, on Ubuntu 14.04 ~100 MB/s. I added timers and was able to validate the send(...) method is the culprit.
I tried with and without antivirus enabled, and tried disabling a few other services with no luck. Some I cannot disable due to permissions on the computer, like the firewall.
I assume there is a service on Windows running that is interfering, or my implementation is missing something that is effecting the application on Windows and not Linux.
Any ideas on what might be cauing this would be appreciated
Is windows and ubuntu running on the same machine?
If not, it seems that your windows machine is limited by 100Mbit Ethernet, while the ubuntu machine seems to work with 1Gbit Ethernet.
(In case thats not the cause of the problem, i am sorry for posting an anwser instead of commenting. But i am not able to do so and your code is that simple and the data rates are so obvious [11*8MB/s ~ 100Mbit/s and 100MB/s ~ 800Mbit/s]. I just had to make that hint...)
If you data transfer if huge say more that 10 MB messages i would suggest you to use TCP instead of UPD/Multicast. TCP is a reliable protocol.
I read in a case where a stream of 300 byte packets was being sent over Ethernet (1500 byte MTU) and TCP was 50% faster than UDP. Because TCP will try and buffer the data and fill a full network segment thus making more efficient use of the available bandwidth but UDP puts the packet on the wire immediately thus congesting the network with lots of small packets. In windows i suggest you to use TCP over UDP/Multicast.
I'm using boost::asio::write() to write data from a buffer to a com-Port. It's a serial port with a baud rate 115200 which means (as far as my understanding goes) that I can write effectively 11520 byte/s or 11,52KB/s data to the socket.
Now I'm having a quite big chunk of data (10015 bytes) which i want to write. I think that this should take little less than a second to really write on the port. But boost::asio::write() returns already 300 microseconds after the call with the transferred bytes 10015. I think this is impossible with that baud rate?
So my question is what is it actually doing? Really writing it to the port, or just some other kind of buffer maybe, which later writes it to the port.
I'd like the write() to only return after all the bytes have really been written to the port.
EDIT with code example:
The problem is that i always run into the timeout for the future/promise because it takes alone more than 100ms to send the message, but I think the timer should only start after the last byte is sent. Because write() is supposed to block?
void serial::write(std::vector<uint8_t> message) {
//create new promise for the request
promise = new boost::promise<deque<uint8_t>>;
boost::unique_future<deque<uint8_t>> future = promise->get_future();
// --- Write message to serial port --- //
boost::asio::write(serial_,boost::asio::buffer(message));
//wait for data or timeout
if (future.wait_for(boost::chrono::milliseconds(100))==boost::future_status::timeout) {
cout << "ACK timeout!" << endl;
//delete pointer and set it to 0
delete promise;
promise=nullptr;
}
//delete pointer and set it to 0 after getting a message
delete promise;
promise=nullptr;
}
How can I achieve this?
Thanks!
In short, boost::asio::write() blocks until all data has been written to the stream; it does not block until all data has been transmitted. To wait until data has been transmitted, consider using tcdrain().
Each serial port has both a receive and transmit buffer within kernel space. This allows the kernel to buffer received data if a process cannot immediately read it from the serial port, and allows data written to a serial port to be buffered if the device cannot immediately transmit it. To block until the data has been transmitted, one could use tcdrain(serial_.native_handle()).
These kernel buffers allow for the write and read rates to exceed that of the transmit and receive rates. However, while the application may write data at a faster rate than the serial port can transmit, the kernel will transmit at the appropriate rates.
Problem
- I am working on a Streaming server & created a nonblocking socket using:
flag=fcntl(m_fd,F_GETFL);
flag|=O_NONBLOCK;
fcntl(m_fd,F_SETFL,flag);
Server then sends the Media file contents using code:
bool SendData(const char *pData,long nSize)
{
int fd=m_pSock->get_fd();
fd_set write_flag;
while(1)
{
FD_ZERO(&write_flag);
FD_SET(fd,&write_flag);
struct timeval tout;
tout.tv_sec=0;
tout.tv_usec=500000;
int res=select(fd+1,0,&write_flag,0,&tout);
if(-1==res)
{
print("select() failure\n");
return false;
}
if(1==res)
{
unsigned long sndLen=0;
if(!m_pSock->send(pData,nSize,&sndLen))
{
print(socket send() failure\n");
return false;
}
nSize-=sndLen;
if(!nSize)
return true; //everything is sent
}
}
}
Using above code, I am streaming a say 200sec audio file, which I expect that Server should stream it in 2-3secs using full n/w available bandwidth(Throttle off), but the problem is that Server is taking 199~200secs to stream full contents.
While debugging, I commented the
m_pSock->send()
section & tried to dump the file locally. It takes 1~2secs to dump the file.
Questions
- If I am using a NonBlocking TCP socket, why does send() taking so much time?
Since the data is always available, select() will return immediately (as we have seen while dumping the file). Does that mean send() is affected by the recv() on the client side?
Any inputs on this would be helpul. Client behavior is not in our scope.
Your client is probably doing some buffering to avoid network jitter, but it is likely still playing the audio file in real time. So, the file transfer rate is matched to the rate that the client is consuming the data. Since it is a 200 second audio file, it will take about 200 seconds to complete the transfer.
Because TCP output and input buffers are propably much smaller than the audio file, reading speed of the receiving application can slow down the sending speed.
When both the TCP output buffer of sender and the input buffer of receiver are both full, TCP stack of the sender is not able to receive any data from the sender. So sending will be blocked, until there is space.
If the receiver reads the TCP stream same speed as data is needed for playing. Then the transfer takes about 200 seconds. Or little bit less.
This can be avoided by using application layer buffering in the receiving end.
The problem could be that if the client side is using blocking TCP, plus is processing all the data on a single thread with no no buffer/queue etc right through to the "player" of the file, then your side being non-blocking will only speed things until you reach the point where the TCP/IP protocol stack buffers, NIC buffers etc are full. Then you will ultimately still only be able to send data as fast as the client side is consuming it. Remember TCP is a reliable, point-to-point protocol.
Where does your client code come from in your testing? Is it some sort of simple test client someone has written?
I want to verify the connection status before performing read/write operations.
Is there a way to make an isConnect() method?
I saw this, but it seems "ugly".
I have tested is_open() function as well, but it doesn't have the expected behavior.
TCP is meant to be robust in the face of a harsh network; even though TCP provides what looks like a persistent end-to-end connection, it's all just a lie, each packet is really just a unique, unreliable datagram.
The connections are really just virtual conduits created with a little state tracked at each end of the connection (Source and destination ports and addresses, and local socket). The network stack uses this state to know which process to give each incoming packet to and what state to put in the header of each outgoing packet.
Because of the underlying — inherently connectionless and unreliable — nature of the network, the stack will only report a severed connection when the remote end sends a FIN packet to close the connection, or if it doesn't receive an ACK response to a sent packet (after a timeout and a couple retries).
Because of the asynchronous nature of asio, the easiest way to be notified of a graceful disconnection is to have an outstanding async_read which will return error::eof immediately when the connection is closed. But this alone still leaves the possibility of other issues like half-open connections and network issues going undetected.
The most effectively way to work around unexpected connection interruption is to use some sort of keep-alive or ping. This occasional attempt to transfer data over the connection will allow expedient detection of an unintentionally severed connection.
The TCP protocol actually has a built-in keep-alive mechanism which can be configured in asio using asio::tcp::socket::keep_alive. The nice thing about TCP keep-alive is that it's transparent to the user-mode application, and only the peers interested in keep-alive need configure it. The downside is that you need OS level access/knowledge to configure the timeout parameters, they're unfortunately not exposed via a simple socket option and usually have default timeout values that are quite large (7200 seconds on Linux).
Probably the most common method of keep-alive is to implement it at the application layer, where the application has a special noop or ping message and does nothing but respond when tickled. This method gives you the most flexibility in implementing a keep-alive strategy.
TCP promises to watch for dropped packets -- retrying as appropriate -- to give you a reliable connection, for some definition of reliable. Of course TCP can't handle cases where the server crashes, or your Ethernet cable falls out or something similar occurs. Additionally, knowing that your TCP connection is up doesn't necessarily mean that a protocol that will go over the TCP connection is ready (eg., your HTTP webserver or your FTP server may be in some broken state).
If you know the protocol being sent over TCP then there is probably a way in that protocol to tell you if things are in good shape (for HTTP it would be a HEAD request)
If you are sure that the remote socket has not sent anything (e.g. because you haven't sent a request to it yet), then you can set your local socket to a non blocking mode and try to read one or more bytes from it.
Given that the server hasn't sent anything, you'll either get a asio::error::would_block or some other error. If former, your local socket has not yet detected a disconnection. If latter, your socket has been closed.
Here is an example code:
#include <iostream>
#include <boost/asio.hpp>
#include <boost/asio/spawn.hpp>
#include <boost/asio/steady_timer.hpp>
using namespace std;
using namespace boost;
using tcp = asio::ip::tcp;
template<class Duration>
void async_sleep(asio::io_service& ios, Duration d, asio::yield_context yield)
{
auto timer = asio::steady_timer(ios);
timer.expires_from_now(d);
timer.async_wait(yield);
}
int main()
{
asio::io_service ios;
tcp::acceptor acceptor(ios, tcp::endpoint(tcp::v4(), 0));
boost::asio::spawn(ios, [&](boost::asio::yield_context yield) {
tcp::socket s(ios);
acceptor.async_accept(s, yield);
// Keep the socket from going out of scope for 5 seconds.
async_sleep(ios, chrono::seconds(5), yield);
});
boost::asio::spawn(ios, [&](boost::asio::yield_context yield) {
tcp::socket s(ios);
s.async_connect(acceptor.local_endpoint(), yield);
// This is essential to make the `read_some` function not block.
s.non_blocking(true);
while (true) {
system::error_code ec;
char c;
// Unfortunately, this only works when the buffer has non
// zero size (tested on Ubuntu 16.04).
s.read_some(asio::mutable_buffer(&c, 1), ec);
if (ec && ec != asio::error::would_block) break;
cerr << "Socket is still connected" << endl;
async_sleep(ios, chrono::seconds(1), yield);
}
cerr << "Socket is closed" << endl;
});
ios.run();
}
And the output:
Socket is still connected
Socket is still connected
Socket is still connected
Socket is still connected
Socket is still connected
Socket is closed
Tested on:
Ubuntu: 16.04
Kernel: 4.15.0-36-generic
Boost: 1.67
Though, I don't know whether or not this behavior depends on any of those versions.
you can send a dummy byte on a socket and see if it will return an error.