TL;DR
I see that my sockets are in TIME_WAIT with the ss tool in Ubuntu 1804, but I can't find in the docs for boost sockets or on SO how to set the time delay to 0 such that the socket immediately closes (or better yet, set it to an arbitrarily small value for my application.
I am writing a socket application with Boost asio. Both the client and the server are using boost sockets. I see that, when the client sends a shutdown connection command: mysocket.shutdown( boost::asio::ip::tcp::socket::shutdown_both, error_code);, and my client C++ application closes down a 2 seconds later, I get TIME_WAIT on the output of ss -ap | grep application_port. I have been looking around SO and the internet looking for ways to set TIME_WAIT with Boost C++, but instead, I keep finding questions for why a TIME_WAIT happens.
Here are some:
https://stackoverflow.com/questions/14666740/receive-data-on-socket-in-time-wait-state
https://stackoverflow.com/questions/35006324/time-wait-with-boost-asio
https://stackoverflow.com/questions/47528798/closing-socket-cause-time-wait-pending-state
If I am interpreting the internet correctly (and the TCP protocol), the reason why TIME_WAIT happens is because the server connection is waiting for an ACK to allow for the socket conn to die, while the client-side socket has already died.
The question, again:
Is there a way to set the TIME_WAIT delay option locally for a C++ executable using Boost sockets? If so, how?
Changing the SO_LINGER option should help: here's the discussion about it:
TCP option SO_LINGER (zero) - when it's required
Boost has api for this reason so you can play around with changing linger to 0:
boost::asio::ip::tcp::socket socket(io_service);
boost::asio::socket_base::linger option(true, 30);
socket.set_option(option);
https://www.boost.org/doc/libs/1_50_0/doc/html/boost_asio/reference/socket_base/linger.html
Anyway, it's good to make sure if you really really have to do that. In case if you have a large number of sockets in TIME_WAIT it most probably means that the client site did not close connection gently. So if you can modify the client code, you can consider it as a first option. Here's nice explanation about how to gently finish the communication on TCP socket (it's not about boost but the logic is the same) Properly close a TCP socket
Most probably you are not closing the socket. The shutdown operation only informs that peer that there will be no more data to read. So most probably you have to:
- call shutdown
- make sure there's nothng more to read
- close the socket
Related
I'm trying to get the TCP header of a TCP connection in C++11. Reading through already existing StackOverflow questions (here, here, here and here) it seems like I have to open a RAW_SOCKET or to write a Linux Kernel Module (LKM) to have access to it.
From what I've understood, opening a raw socket means handling the whole TCP protocol (handshake, window size, etc...). Is there a way to obtain the TCP header and let the kernel manage the TCP protocol (either "by hand" or with some framework)?
I know I could use libpcap for capturing the packets, but this would mean for my application making somehow a match from the incoming packet in the TCP socket and the captured packet from libpcap. While this is a possible solution, it'd be a cumbersome one (and I wouldn't like to do that).
Any help is appreciated, thank you!
A "quick and dirty" approach might be using two connections, an external connection to the remote host and a pure internal one. Sure, this won't be the most efficient approach, but is easy (and fast) to implement (the core feature of QAD "solutions"...):
socket ext_raw ------- socket remote, TCP (likely, at least)
socket int_raw ---
| (loop back connection)
socket int_tcp ---
Any incoming messages at ext_raw and int_raw are just forwarded from one to the other (while incoming messages on ext_raw can be inspected for TCP headers), whereas all the normal TCP handling is done by the internal TCP socket. So in a way, you'll be tunneling the TCP connection through your two raw sockets...
I have few questions, all related to keep_alive.
What is the difference between basic_socket_acceptor::keep_alive and basic_stream_socket::keep_alive? When to use which?
Do we need to use any kind of keep_alive for ip::tcp::acceptor? It doesn't make sense to me, as there is no connection as such for acceptor but there is a keep_alive option for it as well, hence the confusion.
If keep_alive is set, then what is the behaviour of Boost Asio when it detects broken connection? How/when does it notify the user-code? Does it throw exception? If so, which exception? I don't see any such details in the documentation.
What is the difference between basic_socket_acceptor::keep_alive and
basic_stream_socket::keep_alive? When to use which?
Both are same. In the documentation, it appears under basic_socket_acceptor and basic_stream_socket because both are derived from socket_base in which keepalive option is actually visible (it's a typedef).
As per the example in the documentation, you will always use it like:
boost::asio::socket_base::keep_alive option(true);
socket.set_option(option);
Do we need to use any kind of keep_alive for ip::tcp::acceptor?
No you don't have to and you cannot. set_option anyways can be called only on a socket object (I believe only after socket is opened).
If keep_alive is set, then what is the behaviour of Boost Asio when it
detects broken connection?
This is dependent on the platform. On linux you would be getting broken pipe error or EPOLLERR/EPOLLHUP when keep_alive probe fails.
UPDATE (from my comment below):
This failure is not propagated to the user code. So, either probably you need to implement an application level ping or use the timeout socket option.
The basic_socket_acceptor::keep_alive and basic_stream_socket::keep_alive are the same. The documentation notes that they are both inherited from the socket_base class which defines the socket_base::keep_alive option.
basic_stream_socket::keep_alive
Inherited from socket_base.
Socket option to send keep-alives.
While keep-alive on the listening socket is not directly useful for the listening socket, on some systems the newly accepted socket inherits some socket options from the listening socket. The inherited socket options are generally options that would affect the TCP three-way handshake that must complete before accept() returns, such as SO_KEEPALIVE. Consequently, Asio supports setting the keep-alive option on an acceptor; however, Asio does not copy socket options to the new socket.
The keep-alive feature allows for write operations to be notified that a connection is broken as determined by the keep-alive mechanism1. Hence, when the keep-alive probe fails, the next Asio write operation on the socket will fail2, passing the error_code to the application in the same way other error codes are provided. One should consult the operating system's documentation to determine the expected error code from the write operation:
On Windows, WSASend() is documented as returning WSAENETRESET (boost::asio::error:: connection_reset)
On Linux, the error will vary based on how the keep-alive probe fails. If no responses occur, then ETIMEOUT (boost::asio::error::timed_out) will occur. If an ICMP error is returned in response to a keep-alive probe, then the relevant ICMP error will be returned instead. For example, one could observe EHOSTUNREACH (boost::asio::error::host_unreachable) being returned
1. See 4.2.3.6 on the Requirements for Internet Hosts—Communication Layers specified in RFC 1122
2. SO_KEEPALIVE notifies thread writing to the socket via a SIGPIPE signal, but Asio explicitly disables receiving SIGPIPE on write operations. Consequently, the underlying system call will return with the relevant error
This depends on the platform you are running on. On linux if you do exactly the following,
boost::asio::socket_base::keep_alive option(true);
socket.set_option(option);
then you are basically protected from minor network interruptions that can occur and cause a read or write error on the socket. If you set keep_alive to true on the socket pointer then there are couple of certain ways to detect an error on the socket you are reading/writing on:
Firstly, you can detect a socket error by implementing a ping pong mechanism of sending a health packet in intervals between the peers.
Or, you can also detect an error when you are returned with a boost::asio::error::eof error from the socket which basically means that the peer has closed the connection. Note that a read on the socket can still return an boost::asio::error::eof error if the connection is closed by the peer.
I have a server application written in C++. When a client connects, it creates a new thread for him. In that thread there is a BLOCKING reading from a socket. Because there is a possibility for a client to accidentally disconnect and left behind a thread still hanging on the read function, there is a thread that checks if the sockets are still alive by sending "heartbeat messages". The message consists of 1 character and is "ignored" by the client (it is not processed like other messages). The write looks like this:
write(fd, ";", 1);
It works fine, but is it really necessary to send a random character through the socket? I tried to send an empty message ("" with length 0), but it didn't work. Is there any better way to solve this socket checking?
Edit:
I'm using BSD sockets (TCP).
I'm assuming when you say, "socket, you mean a TCP network socket.
If that's true, then the TCP protocol gives you a keepalive option that you would need to ask the OS to use.
I think this StackOverflow answer gets at what you would need to do, assuming a BSDish socket library.
In my experience, using heartbeat messages on TCP (and checking for responses, e.g. NOP/NOP-ACK) is the easiest way to get reliable and timely indication of connectivity at the application layer. The network layer can do some interesting things but getting notification in your application can be tricky.
If you can switch to UDP, you'll have more control and flexibility at the application layer, and probably reduced traffic overall since you can customize the communications, but you'll need to handle reliability, packet ordering, etc. yourself.
You can set connection KEEPALIVE. You may have interests in this link: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
It is ok you create a thread for each new coming requests if it is only toy. In most of time, i use poll, that is non-blocking io, for performance improvement.
Recently I began working with the Boost::Asio library (C++). I'm looking for a way to send a TCP SYN message to a end destination. However I can't find any way of doing this, does somebody knows a way to accomplish it?
The TCP stack usually deals with this, not your code. If you just call boost::asio::ip::tcp::socket::connect() on an appropriately constructed instance, you will cause a TCP SYN packet to be sent, along with the rest of the TCP handshake and session handling.
Update:
If you want to implement TCP yourself you will need to deal with more than just a TCP SYN, otherwise you're just writing code to attack systems with half-open connections. You need a raw socket and you need to construct the contents of the packet yourself. If you are doing this you should be able to RTFM to find out more.
I'm having a problem with one of my C++ applications on Windows 2008x64 (same app runs just fine on Windows 2003x64).
After a crash or even sometimes after a regular shutdown/restart cycle it has a problem using a socket on port 82 it needs to receive commands.
Looking at netstat I see the socket is still in listening state more than 10 minutes after the application stopped (the process is definitely not running anymore).
TCP 0.0.0.0:82 LISTENING
I tried setting the socket option to REUSEADDR but as far as I know that only affects re-connecting to a port that's in TIME_WAIT state. Either way this change didn't seem to make any difference.
int doReuse = 1;
setsockopt(listenFd, SOL_SOCKET, SO_REUSEADDR,
(const char *)&doReuse, sizeof(doReuse));
Any ideas what I can do to solve or at least avoid this problem?
EDIT:
Did netstat -an but this is all I am getting:
TCP 0.0.0.0:82 0.0.0.0:0 LISTENING
For netstat -anb I get:
TCP 0.0.0.0:82 0.0.0.0:0 LISTENING
[System]
I'm aware of shutting down gracefully, but even if the app crashes for some reason I still need to be able to restart it. The application in question uses an in-house library that internally uses Windows Sockets API.
EDIT:
Apparently there is no solution for this problem, so for development I will go with a proxy / tool to work around it. Thanks for all the suggestions, much appreciated.
If this is only hurting you at debug time, use tcpview from the sysinternals folks to force the socket closed. I am assuming it works on your platform, but I am not sure.
If you're doing blocking operations on any sockets, do not use an indefinite timeout. This can cause weird behavior on a multiprocessor machine in my experience. I'm not sure what Windows server OS it was, but, it was one or two versions previous to 2003 Server.
Instead of an indefinite timeout, use a 30 to 60 second timeout and then just repeat the wait. This goes for overlapped IO and IOCompletion ports as well, if you're using them.
If this is an app you're shipping for others to use, good luck. Windows can be a pure bastard when using sockets...
I tried setting the socket option to
REUSEADDR but as far as I know that
only affects re-connecting to a port
that's in TIME_WAIT state.
That's not quite correct. It will let you re-use a port in TIME_WAIT state for any purpose, i.e. listen or connect. But I agree it won't help with this. I'm surprised by the comment about the OS taking 10 minutes to detect the crashed listener. It should clean up all resources as soon as the process ends, other than ports in the TIME_WAIT state.
The first thing to check is that it really is your application listening on that port. Use:
netstat -anb
to figure out which process is listenin on that port.
The second thing to check is that your are closing the socket gracefully when your application shuts down. If you're using a high-level socket API that shouldn't be too much of an issue (you are using a socket API, right?).
Finally, how is your application structured? Is it threaded? Does it launch other processes? How do you know that your application is really shut down?
Run
netstat -ano
This will give you the PID of the process that has the port open. Check that process from the task manager. Make sure you have "list processes from all users" is checked.
http://hea-www.harvard.edu/~fine/Tech/addrinuse.html is a great resource for "Bind: Address Already in Use" errors.
Some extracts:
TIME_WAIT is the state that typically ties up the port for several minutes after the process has completed. The length of the associated timeout varies on different operating systems, and may be dynamic on some operating systems, however typical values are in the range of one to four minutes.
Strategies for Avoidance
SO_REUSEADDR
This is the both the simplest and the most effective option for reducing the "address already in use" error.
Client Closes First
TIME_WAIT can be avoided if the remote end initiates the closure. So the server can avoid problems by letting the client close first.
Reduce Timeout
If (for whatever reason) neither of these options works for you, it may also be possible to shorten the timeout associated with TIME_WAIT.
After seeing https://superuser.com/a/453827/56937 I discovered that there was a WerFault process that was suspended.
It must have inherited the sockets from the non-existent process because killing it freed up my listening ports.