Socket programming, what about "CLOSE_WAIT", "FIN_WAIT_2" and "LISTENING"? - c++

I am writing a socket based C application which seems to behave in a very unstable way.
The code is standard socket handling on TCP port 6683, and I know it has worked before. Instead of mentioning the source code, I believe the most interesting are the results of the netstat -aon command:
When it worked fine, the results of the netstat -aon| grep 6683 command were:
TCP 127.0.0.1:6683 127.0.0.1:50888 CLOSE_WAIT 6128
TCP 127.0.0.1:50888 127.0.0.1:6683 FIN_WAIT_2 3764
When it does not work anymore, the results of the netstat -aon | grep 6683 command are:
TCP 127.0.0.1:6683 0.0.0.0:0 LISTENING 7800
Does anybody know the meaning of the mentioned "netstat" results, what this might mean for the socket handling and what can I do in order to return to the situation giving the first results?
Thanks

From Microsoft's Support Website:
FIN_WAIT_2 Indicates that the client just received acknowledgment of the first FIN signal from the server
LISTENING Indicates that the server is ready to accept a connection
CLOSE_WAIT Indicates that the server has received the first FIN signal from the client and the connection is in the process of being
closed
Based on the above, you know that in your first case the server has received the client's FIN, and the client has received the ACK from sending the FIN to the server.
However, in the second case the server is ready to accept a connection, which to me sounds like you haven't established your TCP connection yet.
Without really knowing what your C program is trying to do, it's hard to diagnose your issues here, but I would take a look at the documentation for netstat and go from there.

From the netstat documentation :
FIN_WAIT2
Connection is closed, and the socket is waiting for a shutdown from the remote end.
CLOSE_WAIT
The remote end has shut down, waiting for the socket to close.
The LISTENING state is just the server socket waiting for clients. This is a normal behavior for a listening server socket (which is not the same as the connection server socket).
You can see that the side with FIN_WAIT2 is closed and waiting for the other, but the side with CLOSE_WAIT, is currently closing, but not yet closed. Based on the LISTENING socket, the client is closed, and the currently closing side is the server. The server is probably waiting because there is data to read that was not yet read. It can't close the socket without data loss, which is unacceptable for TCP. The connection should close normally after all the data left on the server side is read.

Thanks for the fast responses. Meanwhile I have found what was going wrong:
My application was a server application, creating a client process, and setting up a TCP socket for communicating with that client process (which was done using the C commands:
snprintf(buf, 1024, "client_process.exe %s %d", szListenHost, iListenPort);
CreateProcess(NULL, buf, NULL, NULL, FALSE, dwCreationFlags,
NULL, NULL, &sin, &pin);
Sometimes this went fine, sometimes not, based on the place from where I launched my server process: when I launched it from the official directory, it was working fine. When I launched it from my development environment, it was not working.
The reason for this was very simple: in my development environment, the "client_process.exe" file was not present in the current directory.
I have now copied the "client_process.exe" into that directory and added an extra check:
int return_value = CreateProcess(NULL, buf, NULL, NULL, FALSE,
dwCreationFlags, NULL, NULL, &sin, &pin);
if (return_value == 0) {
printf("The client_process.exe has not been started successfully.\n");
word error_return_value = GetLastError();
printf("The corresponding error value is:[%d]\n", error_return_value);
if (error_return_value == 2) {
printf("The client_process.exe is not present in the current directory.\n");
}
}
Kind regards

Related

UDP port is not free after closesocket call (Windows)

I have two listening sockets (created by calls to socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDP)), at this moment they are maintained from one thread.
After creating, they have to be closed (closesocket(...)) and reopened again on the same ports. But bind(...) call returns error 10048 WSAEADDRINUSE on one of these sockets (the second one is opened successfully), and I see by using netstat that the UDP port stays open (closesocket(...) returned no error, SO_REUSEADDR always set to TRUE on all sockets). And this "closed" UDP port stays open as long as the 2nd socket is open (they have no relation, but the "closed" port is closing a second after the 2nd socket is closed).
Let's summarize:
Open sockets and bind them to ports 8888 and 9999.
Close 8888 socket, create new socket, bind it to port 8888 -> success.
Close 9999 socket, create new socket, and try to bind it to port 9999 -> error WSAEADDRINUSE.
Close 8888 socket -> success.
After about a second after #4, port 9999 is freed (by observing in external tool).
I have discovered something similar to my problem: https://stackoverflow.com/a/26129726/10101917, but in my case moving all socket operations to one thread does not solve the problem.
What is happening here?
I have found what caused this problem. The thing I have programmed is the DLL. I have discovered, that another DLL from this app is using QProcess class of Qt library version 5.7.1. I have checked sources of Qt and discovered that this class actually starts process with bInheritHandles set to TRUE. When I manually have reset this value to FALSE all issues were gone.
It is obvious that the issue was caused by the following: one of UDP socket handles was inherited by child process and that process didn't let socket handle to be closed, until that process stop.
Thanks to this comment for pointing to the solution.

Disable TIME_WAIT with boost sockets

TL;DR
I see that my sockets are in TIME_WAIT with the ss tool in Ubuntu 1804, but I can't find in the docs for boost sockets or on SO how to set the time delay to 0 such that the socket immediately closes (or better yet, set it to an arbitrarily small value for my application.
I am writing a socket application with Boost asio. Both the client and the server are using boost sockets. I see that, when the client sends a shutdown connection command: mysocket.shutdown( boost::asio::ip::tcp::socket::shutdown_both, error_code);, and my client C++ application closes down a 2 seconds later, I get TIME_WAIT on the output of ss -ap | grep application_port. I have been looking around SO and the internet looking for ways to set TIME_WAIT with Boost C++, but instead, I keep finding questions for why a TIME_WAIT happens.
Here are some:
https://stackoverflow.com/questions/14666740/receive-data-on-socket-in-time-wait-state
https://stackoverflow.com/questions/35006324/time-wait-with-boost-asio
https://stackoverflow.com/questions/47528798/closing-socket-cause-time-wait-pending-state
If I am interpreting the internet correctly (and the TCP protocol), the reason why TIME_WAIT happens is because the server connection is waiting for an ACK to allow for the socket conn to die, while the client-side socket has already died.
The question, again:
Is there a way to set the TIME_WAIT delay option locally for a C++ executable using Boost sockets? If so, how?
Changing the SO_LINGER option should help: here's the discussion about it:
TCP option SO_LINGER (zero) - when it's required
Boost has api for this reason so you can play around with changing linger to 0:
boost::asio::ip::tcp::socket socket(io_service);
boost::asio::socket_base::linger option(true, 30);
socket.set_option(option);
https://www.boost.org/doc/libs/1_50_0/doc/html/boost_asio/reference/socket_base/linger.html
Anyway, it's good to make sure if you really really have to do that. In case if you have a large number of sockets in TIME_WAIT it most probably means that the client site did not close connection gently. So if you can modify the client code, you can consider it as a first option. Here's nice explanation about how to gently finish the communication on TCP socket (it's not about boost but the logic is the same) Properly close a TCP socket
Most probably you are not closing the socket. The shutdown operation only informs that peer that there will be no more data to read. So most probably you have to:
- call shutdown
- make sure there's nothng more to read
- close the socket

When will a connected UDP socket be closed by the OS?

I have a UDP file descriptor in a C++ program running under Linux. I call connect() on it to connect it to a remote address and then read and write from that socket.
According to UNIX Network Programming, "Asynchronous errors are returned to the process for connected UDP sockets." I'm guessing that these asynchronous errors will cause the UDP socket to be closed by the OS, but the book isn't that clear. It also isn't clear what types of asynchronous errors are possible, though it's suggested that if the port on the remote machine is not open, the socket will be closed.
So my question is: Under what conditions will Linux close the UDP file descriptor?
Bad port number?
Bad IP address?
Any others?
connect() on an UDP socket just records the port number and IP address you pass in, so it'll only accept packets from that IP/port, and you can use the socket fd to send/write data without specifying the remote address for each send/write call.
Regarding this, async errors means if you send() something, and that send call results in an error occuring later (e.g. when the TCP/IP stack actually sends the packet, or an ICMP packet is later returned), a subsequent send will return that error. Such async errors are only returned on a "connected" UDP socket. (The linux udp(7) manpage suggest errors are returned whether the socket is connected or not, but testing shows this is not the cases at least when a sent UDP packet generates an ICMP error. It might be that send() errors are returned if you recv() on that socket, instead of subsequent send() calls produce an error )
The socket is not closed though, you'll have to close it yourself either by calling close() or exiting the program. e.g. if you connect() your UDP socket, and send to a port noone is listening to, an ICMP packet is normally returned and a subsequent send() call will fail with errno set to ECONNREFUSED. You can continue sending on that socket though, it doesn't get closed by the OS, and if someone starts listening on the port in the mean time the packets will get through.
UDP sockets are connectionless, so there is no real sense of "openness" state attached to them - this is unlike TCP sockets where a socket may be in any number of connection states as determined by the exchange of packets up to a given point.
The only sense in which UDP sockets can be opened and closed is in the sense that they are system level objects with some internal state and a file descriptor. Sockets are never automatically closed in the event of an error and will remain open indefinitely, unless their owning process terminates or calls close on them.
To address your other concern, if the destination port on the destination host is not opened, the sender of a UDP packet will never know.** UDP provides no means of receiver acknowledgement. The packet is routed and, if it arrives at the host, checked for correctness and either successfully received or discarded. There are a number of reasons why send might return an error code when writing to a UDP socket, but none of them have to do with the state of the receiving host.** I recommend consulting the sendto manpage for possible failure modes.
On the other hand, in the case of a TCP socket attempting to connect to an unopened port, the sender will never receive an acknowledgement of its initial connection request, and ultimately connect will fail. At this point it would be up to the sender to stop sending data over the socket (as this will only generate more errors), but even in this case however, the socket file descriptor is never automatically closed.
** See response by #Zuljin in the comments.
The OS won't close your socket just because an error has happened. If the other end disappears, you can continue to send messages to it (but may receive further errors).

C++ Winsock P2P

Scenario
Does anyone have any good examples of peer-to-peer (p2p) networking in C++ using Winsock?
It's a requirement I have for a client who specifically needs to use this technology (god knows why).
I need to determine whether this is feasible.
Any help would be greatly appreciated.
EDIT
And I would like to avoid using libraries so that I can understand the underlying source code and further my knoweldge.
Since I don't know what information you are looking for, I'll try to describe how to set up a socket program and what pitfalls I've run into.
To start with, *read the Winsock tutorial at MSDN. This is a basic program to connect, send a message and disconnect. It's great for getting a feel for socket programming.
With that, lets start:
Considerations:
Blocking or non-blocking
First off, you need to decide whether you want a blocking or non-blocking program. If you have a GUI you would need to use non-blocking or threading in order to not freeze the program. The way I did it was to use the blocking calls, but always calling select before calling the blocking functions (more on select later). This way I avoid threading and mutex's and whatnot but still use the basic accept, send and receive calls.
You cannot rely that your packets will arrive the way you send them!
You have no control over this either. This was the biggest issue I ran into, basically because the network card can decide what information to send and when to send it. The way I solved it was to make a networkPackageStruct, containing a size and data, where size is the total amound of data in that packet. Note that a message that you send can be split into 2 or more packets and can also be merged with another message you send.
Consider the following:
You send two messages
"Hello"
"World!"
When you send these two messages with the send function your recv function might not get them like this. It could look like this:
"Hel"
"loWorld!"
or perhaps
"HelloWorld!"
whatever the underlying network feels like.
Log (almost) everything!
Debugging a network program is hard because you don't have full control over it (since it's on two computers). If you run into a blocking operation you can't see it either. This could as well be called "Know your blocking code". When one side sends something, you don't know if it will arrive on the other side, so keep track of what is sent and what is received.
Pay attention to socket errors
Winsock functions return a lot of information. Know your WSAGetLastError() function. I'll won't keep it in the examples below, but note that they tend to return alot of information. Every time you get a SOCKET_ERROR or INVALID_SOCKET check the Winsock Error Messages to look it up.
Setting up the connection:
Since you don't want a server, all clients would need a listening socket to accept new connections. The easiest is:
SOCKET s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
sockaddr_in localAddress;
localAddress.sinfamily = AF_INET;
localAddress.sin_port = htons(10000); // or whatever port you'd like to listen to
localAddress.sin_addr.s_addr = INADDR_ANY;
INADDR_ANY is great - it makes your socket listen on all your IP addresses instead of just one IP address.
bind(s, (SOCKADDR*)&localAddress, sizeof(localAddress));
listen(s, SOMAXCONN);
Here comes the interesting part. bind and listen won't block but accept will. The trick is to use select to check if there is an incoming connection. So the above code is just to set the socket up. in your program loop you check for new data in socket.
Exchanging data
The way I solved it is was to use select alot. Basically you see if there are anything you need to respond to on any of your sockets. This is done with the FD_xxx functions.
// receiving data
fd_set mySet;
FD_ZERO(&mySet);
FD_SET(s, &mySet);
// loop all your sockets and add to the mySet like the call above
timeval zero = { 0, 0 };
int sel = select(0, &mySet, NULL, NULL, &zero);
if (FD_ISSET(s, &mySet)){
// you have a new caller
sockaddr_in remote;
SOCKET newSocket = accept(s, (SOCKADDR*)&remote, sizeof(remote));
}
// loop through your sockets and check if they have the FD_ISSET() set
In the newSocket you now have a new peer. So that was for receiving data. But note! send is also blocking! One of the head-scratching errors I got was that send blocked me. This was however also solved with select.
// sending data
// in: SOCKET sender
fd_set mySet;
FD_ZERO(&mySet);
FD_SET(sender, &mySet);
timeval zero = { 0, 0 };
int sel = select(0, NULL, mySet, NULL, &zero);
if (FD_ISSET(sender, &mySet)){
// ok to send data
}
Shutting down
Finally, there are two ways to shutdown. You either just disconnect by closing your program, or you call the shutdown function.
Calling shutdown will make your peer select trigger. recv will however not receive any data, but will instead return 0. I have not noticed any other case where recv returns 0, so it is (somewhat) safe to say that this can be considered a shutdown-code. calling shutdown is the nicest thing to do.
Shutting down the connection without calling shutdown just is cold-hearted, but of course works. You still need to handle the error even if you use shutdown, since it might not be your program that closes the connection. A good error code to remember is 10054 which is WSAECONNRESET: Connection reset by peer.
If you just want to implement a P2P application on Microsoft Windows, you can try with Windows Peer-to-Peer Networking
If you want to implement a new P2P protocol of your own, you can study the eMule protocol, and eMule source code. You could do further if you look into Shareaza source code, it do eMule/Guntella/Gnutella/BitTorrent.

WSAEventSelect model

Hey I'm using the WSAEventSelect for event notifications of sockets. So far everything is cool and working like a charm, but there is one problem.
The client is a .NET application and the server is written in Winsock C++. In the .NET application I'm using System.Net.Sockets.Socket class for TCP/IP. When I call the Socket.Shutdown() and Socket.Close() method, I receive the FD_CLOSE event in the server, which I'm pretty sure is fine. Okay the problem occurs when I check the iErrorCode of WSANETWORKEVENTS which I passed to WSAEnumNetworkEvents. I check it like this
if (listenerNetworkEvents.lNetworkEvents & FD_CLOSE)
{
if (listenerNetworkEvents.iErrorCode[FD_CLOSE_BIT] != 0)
{
// it comes here
// which means there is an error
// and the ERROR I got is
// WSAECONNABORTED
printf("FD_CLOSE failed with error %d\n",
listenerNetworkEvents.iErrorCode[FD_CLOSE_BIT]);
break;
}
closesocket(socketArray[Index]);
}
But it fails with the WSAECONNABORTED error. Why is that so?
EDIT: Btw, I'm running both the client and server on the same computer, is it because of that? And I received the FD_CLOSE event when I do this:
server.Shutdown(SocketShutdown.Both); // in .NET C#, client code
I'm guessing you're calling Shutdown() and then Close() immediately afterward. That will give the symptom you're seeing, because this is "slamming the connection shut". Shutdown() does initiate a graceful disconnect (TCP FIN), but immediately following it with Close() aborts that, sending a TCP RST packet to the remote peer. Your Shutdown(SocketShutdown.Both) call slams the connection shut, too, by the way.
The correct pattern is:
Call Shutdown() with the direction parameter set to "write", meaning we won't be sending any more data to the remote peer. This causes the stack to send the TCP FIN packet.
Go back to waiting for Winsock events. When the remote peer is also done writing, it will call Shutdown("write"), too, causing its stack to send your machine a TCP FIN packet, and for your application to get an FD_CLOSE event. While waiting, your code should be prepared to continue reading from the socket, because the remote peer might still be sending data.
(Please excuse the pseudo-C# above. I don't speak .NET, only C++.)
Both peers are expected to use this same shutdown pattern: each tells the other when it's done writing, and then waits to receive notification that the remote peer is done writing before it closes its socket.
The important thing to realize is that TCP is a bidirectional protocol: each side can send and receive independently of the other. Closing the socket to reading is not a nice thing to do. It's like having a conversation with another person but only talking and being unwilling to listen. The graceful shutdown protocol says, "I'm done talking now. I'm going to wait until you stop talking before I walk away."