I am creating a TCP connection from my linux program with boost.asio. I wonder how do I get the value of its congestion window (cwnd) from the program? The only way I know of is to parse /proc/net/tcp, but this does not feel right. I'd rather use a dedicated syscall to get this info.
A solution to a similar question (How to monitor cwnd and ssthresh values for a TCP connection?) suggests using TCP Probe, but it feels even less appealing.
So what is the best way to get the value of cwnd?
I did this with netlink and INET_DIAG-sockets based on this helpful example: https://github.com/kristrev/inet-diag-example
It turned out getsockopt() is able to return the same tcp_info when called with TCP_INFO option:
tcp_info tcpi = {};
socklen_t len = sizeof(tcp_info);
getsockopt(tcp_socket, SOL_TCP, TCP_INFO, &tcpi, &len);
tcpi.tcpi_snd_cwnd; // <-- CWND
Related
When conducting a stress test on some server code I wrote, I noticed that even though I am calling close() on the descriptor handle (and verifying the result for errors) that the descriptor is not released which eventually causes accept() to return an error "Too many open files".
Now I understand that this is because of the ulimit, what I don't understand is why I am hitting it if I call close() after each synchronous accept/read/send cycle?
I am validating that the descriptors are in fact there by running a watch with lsof:
ctsvr 9733 mike 1017u sock 0,7 0t0 3323579 can't identify protocol
ctsvr 9733 mike 1018u sock 0,7 0t0 3323581 can't identify protocol
...
And sure enough there are about 1000 or so of them. Further more, checking with netstat I can see that there are no hanging TCP states (no WAIT or STOPPED or anything).
If I simply do a single connect/send/recv from the client, I do notice that the socket does stay listed in lsof; so this is not even a load issue.
The server is running on an Ubuntu Linux 64-bit machine.
Any thoughts?
So using strace (thanks Gearoid), which I have no idea how I ever lived without, I noted I was in fact closing the descriptors.
However. And for the sake of posterity I lay bare my foolish mistake:
Socket::Socket() : impl(new Impl) {
impl->fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
....
}
Socket::ptr_t Socket::accept() {
auto r = ::accept(impl->fd, NULL, NULL);
...
ptr_t s(new Socket);
s->impl->fd = r;
return s;
}
As you can see, my constructor allocated a socket immediately, and then I replaced the descriptor with the one returned by accept - creating a leak. I had refactored the accept code from a standalone Acceptor class into the Socket class without changing this.
Using strace I could easily see socket() being run each time which lead to my light bulb moment.
Thanks all for the help!
Have you ever called perror() after close()?
I think the returned string will give you some help;
You are most probably hanging on a recv() or send() command. Consider setting a timeout using setsockopt .
I noticed a similar output on lsof when the socket was closed on the other end but my thread was keeping the socket open hanging on the recv() command waiting for data.
I have in my application a failure that arose which does not seem to be reproducible. I have a TCP socket connection which failed and the application tried to reconnect it. In the second call to connect() attempting to reconnect, I got an error result with errno == EADDRNOTAVAIL which the man page for connect() says means: "The specified address is not available from the local machine."
Looking at the call to connect(), the second argument appears to be the address to which the error is referring to, but as I understand it, this argument is the TCP socket address of the remote host, so I am confused about the man page referring to the local machine. Is it that this address to the remote TCP socket host is not available from my local machine? If so, why would this be? It had to have succeeded calling connect() the first time before the connection failed and it attempted to reconnect and got this error. The arguments to connect() were the same both times.
Would this error be a transient one which, if I had tried calling connect again might have gone away if I waited long enough? If not, how should I try to recover from this failure?
Check this link
http://www.toptip.ca/2010/02/linux-eaddrnotavail-address-not.html
EDIT: Yes I meant to add more but had to cut it there because of an emergency
Did you close the socket before attempting to reconnect? Closing will tell the system that the socketpair (ip/port) is now free.
Here are additional items too look at:
If the local port is already connected to the given remote IP and port (i.e., there's already an identical socketpair), you'll receive this error (see bug link below).
Binding a socket address which isn't the local one will produce this error. if the IP addresses of a machine are 127.0.0.1 and 1.2.3.4, and you're trying to bind to 1.2.3.5 you are going to get this error.
EADDRNOTAVAIL: The specified address is unavailable on the remote machine or the address field of the name structure is all zeroes.
Link with a bug similar to yours (answer is close to the bottom)
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4294599
It seems that your socket is basically stuck in one of the TCP internal states and that adding a delay for reconnection might solve your problem as they seem to have done in that bug report.
This can also happen if an invalid port is given, like 0.
If you are unwilling to change the number of temporary ports available (as suggested by David), or you need more connections than the theoretical maximum, there are two other methods to reduce the number of ports in use. However, they are to various degrees violations of the TCP standard, so they should be used with care.
The first is to turn on SO_LINGER with a zero-second timeout, forcing the TCP stack to send a RST packet and flush the connection state. There is one subtlety, however: you should call shutdown on the socket file descriptor before you close, so that you have a chance to send a FIN packet before the RST packet. So the code will look something like:
shutdown(fd, SHUT_RDWR);
struct linger linger;
linger.l_onoff = 1;
linger.l_linger = 0;
// todo: test for error
setsockopt(fd, SOL_SOCKET, SO_LINGER,
(char *) &linger, sizeof(linger));
close(fd);
The server should only see a premature connection reset if the FIN packet gets reordered with the RST packet.
See TCP option SO_LINGER (zero) - when it's required for more details. (Experimentally, it doesn't seem to matter where you set setsockopt.)
The second is to use SO_REUSEADDR and an explicit bind (even if you're the client), which will allow Linux to reuse temporary ports when you run, before they are done waiting. Note that you must use bind with INADDR_ANY and port 0, otherwise SO_REUSEADDR is not respected. Your code will look something like:
int opts = 1;
// todo: test for error
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR,
(char *) &opts, sizeof(int));
struct sockaddr_in listen_addr;
listen_addr.sin_family = AF_INET;
listen_addr.sin_port = 0;
listen_addr.sin_addr.s_addr = INADDR_ANY;
// todo: test for error
bind(fd, (struct sockaddr *) &listen_addr, sizeof(listen_addr));
// todo: test for addr
// saddr is the struct sockaddr_in you're connecting to
connect(fd, (struct sockaddr *) &saddr, sizeof(saddr));
This option is less good because you'll still saturate the internal kernel data structures for TCP connections as per netstat -an | grep -e tcp -e udp | wc -l. However, you won't start reusing ports until this happens.
I got this issue. I got it resolve by enabling tcp timestamp.
Root cause:
After connection close, Connections will go in TIME_WAIT state for some
time.
During this state if any new connections comes with same IP and PORT,
if SO_REUSEADDR is not provided during socket creation then socket bind()
will fail with error EADDRINUSE.
But even though after providing SO_REUSEADDR also sockect connect() may
fail with error EADDRNOTAVAIL if tcp timestamp is not enable on both side.
Solution:
Please enable tcp timestamp on both side client and server.
echo 1 > /proc/sys/net/ipv4/tcp_timestamps
Reason to enable tcp_timestamp:
When we enable tcp_tw_reuse, sockets in TIME_WAIT state can be used before they expire, and the kernel will try to make sure that there is no collision regarding TCP sequence numbers. If we enable tcp_timestamps, it will make sure that those collisions cannot happen. However, we need TCP timestamps to be enabled on both ends. See the definition of tcp_twsk_unique for the gory details.
reference:
https://serverfault.com/questions/342741/what-are-the-ramifications-of-setting-tcp-tw-recycle-reuse-to-1
Another thing to check is that the interface is up. I got confused by this one recently while using network namespaces, since it seems creating a new network namespace produces an entirely independent loopback interface but doesn't bring it up (at least, with Debian wheezy's versions of things). This escaped me for a while since one doesn't typically think of loopback as ever being down.
How do I receive layer 2 packets in POSIXy C++? The packets only have src and dst MAC address, type/length, and custom formatted data. They're not TCP or UDP or IP or IGMP or ARP or whatever - they're a home-brewed format given unto me by the Hardware guys.
My socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW) never returns from its recvfrom().
I can send fine, I just can't receive no matter what options I fling at the network stack.
(Platform is VxWorks, but I can translate POSIX or Linux or whatever...)
receive code (current incarnation):
int s;
if ((s = socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW)) < 0) {
printf("socket create error.");
return -1;
}
struct ifreq _ifr;
strncpy(_ifr.ifr_name, "lltemac0", strlen("lltemac0"));
ioctl(s, IP_SIOCGIFINDEX, &_ifr);
struct sockaddr_ll _sockAttrib;
memset(&_sockAttrib, 0, sizeof(_sockAttrib));
_sockAttrib.sll_len = sizeof(_sockAttrib);
_sockAttrib.sll_family = AF_PACKET;
_sockAttrib.sll_protocol = IFT_ETHER;
_sockAttrib.sll_ifindex = _ifr.ifr_ifindex;
_sockAttrib.sll_hatype = 0xFFFF;
_sockAttrib.sll_pkttype = PACKET_HOST;
_sockAttrib.sll_halen = 6;
_sockAttrib.sll_addr[0] = 0x00;
_sockAttrib.sll_addr[1] = 0x02;
_sockAttrib.sll_addr[2] = 0x03;
_sockAttrib.sll_addr[3] = 0x12;
_sockAttrib.sll_addr[4] = 0x34;
_sockAttrib.sll_addr[5] = 0x56;
int _sockAttribLen = sizeof(_sockAttrib);
char packet[64];
memset(packet, 0, sizeof(packet));
if (recvfrom(s, (char *)packet, sizeof(packet), 0,
(struct sockaddr *)&_sockAttrib, &_sockAttribLen) < 0)
{
printf("packet receive error.");
}
// code never reaches here
I think the way to do this is to write your own Network Service that binds to the MUX layer in the VxWorks network stack. This is reasonably well documented in the VxWorks Network Programmer's Guide and something I have done a number of times.
A custom Network Service can be configured to see all layer 2 packets received on a network interface using the MUX_PROTO_SNARF service type, which is how Wind River's own WDB protocol works, or packets with a specific protocol type.
It is also possible to add a socket interface to your custom Network Service by writing a custom socket back-end that sits between the Network Service and the socket API. This is not required if you are happy to do the application processing in the Network Service.
You haven't said which version of VxWorks you are using but I think the above holds for VxWorks 5.5.x and 6.x
Have you tried setting the socket protocol to htons(ETH_P_ALL) as prescribed in packet(7)? What you're doing doesn't have much to do with IP (although IPPROTO_RAW may be some wildcard value, dunno)
I think this is going to be a bit tougher problem to solve than you expect. Given that it's not IP at all (or apparently any other protocol anything will recognize), I don't think you'll be able to solve your problem(s) entirely with user-level code. On Linux, I think you'd need to write your own device agnostic interface driver (probably using NAPI). Getting it to work under VxWorks will almost certainly be non-trivial (more like a complete rewrite from the ground-up than what most people would think of as a port).
Have you tried confirming via Wireshark that a packet has actually been sent from the other end?
Also, for debugging, ask your hardware guys if they have a debug pin (you can attach to a logic analyzer) that they can assert when it receives a packet. Just to make sure that the hardware is getting the packets fine.
First you need to specify the protocol as ETH_P_ALL so that your interface gets all the packet. Set your socket to be on promiscuous mode. Then bind your RAW socket to an interface before you perform a receive.
I'm hoping someone can explain a situation -- any situation -- in which a non-blocking Windows socket would not immediately return after using send() on it. On other platforms, my code works as intended, so it appears to be a Windows-specific issue.
The way I can tell it isn't returning is quite simple: I cout a message immediately before and after the send() function, and upon running, the console only shows the first message. I can't seem to reproduce it in a simplified example, so I apologize for the lack of code. I will try to provide more specifics if necessary.
Edit:
The specific code where the socket is created and set as non-blocking is the following. I'm assuming that there is indeed no way a non-blocking socket would behave this way, and that something else is causing it to stop before the second message is shown. Thank you for your help anyway...I will try my best to get to the bottom of it, but right now I'm quite mystified.
SOCKET new_sock = accept(sock, (struct sockaddr*)&addr, &addr_size);
#ifdef _WIN32
unsigned long ul;
ioctlsocket(new_sock, FIONBIO, &ul);
#else
fcntl(new_sock,F_SETFL,O_NONBLOCK);
#endif
setsockopt(new_sock, SOL_SOCKET, SO_LINGER, (unsigned char *)&lg, sizeof(lg));
setsockopt(new_sock, SOL_SOCKET, SO_OOBINLINE, (unsigned char *)&opt, sizeof(int));
if this is the actual code, you havn't initialized ul . Do:
unsigned long ul = 1;
ioctlsocket(new_sock, FIONBIO, &ul);
If your "cout" message display does not contain an std::endl, the call might not be blocking without you noticing !
Send will block on filling the ip stack buffers on a connected socket. If you have a connected socket and you start throwing data at it faster then the ip stack can roll it out the wire with the attendant ack nak paddywacks then the send will block.
I have not seen that properly set non blocking sends will block though. They will error with wouldblock. So I would look to how you are setting non blocking io. Are you sure you are setting non blocking on the resultant socket not the base socket... ie the socket returned from the accept.
I am writing an XMLRPC client in c++ that is intended to talk to a python XMLRPC server.
Unfortunately, at this time, the python XMLRPC server is only capable of fielding one request on a connection, then it shuts down, I discovered this thanks to mhawke's response to my previous query about a related subject
Because of this, I have to create a new socket connection to my python server every time I want to make an XMLRPC request. This means the creation and deletion of a lot of sockets. Everything works fine, until I approach ~4000 requests. At this point I get socket error 10048, Socket in use.
I've tried sleeping the thread to let winsock fix its file descriptors, a trick that worked when a python client of mine had an identical issue, to no avail.
I've tried the following
int err = setsockopt(s_,SOL_SOCKET,SO_REUSEADDR,(char*)TRUE,sizeof(BOOL));
with no success.
I'm using winsock 2.0, so WSADATA::iMaxSockets shouldn't come into play, and either way, I checked and its set to 0 (I assume that means infinity)
4000 requests doesn't seem like an outlandish number of requests to make during the run of an application. Is there some way to use SO_KEEPALIVE on the client side while the server continually closes and reopens?
Am I totally missing something?
The problem is being caused by sockets hanging around in the TIME_WAIT state which is entered once you close the client's socket. By default the socket will remain in this state for 4 minutes before it is available for reuse. Your client (possibly helped by other processes) is consuming them all within a 4 minute period. See this answer for a good explanation and a possible non-code solution.
Windows dynamically allocates port numbers in the range 1024-5000 (3977 ports) when you do not explicitly bind the socket address. This Python code demonstrates the problem:
import socket
sockets = []
while True:
s = socket.socket()
s.connect(('some_host', 80))
sockets.append(s.getsockname())
s.close()
print len(sockets)
sockets.sort()
print "Lowest port: ", sockets[0][1], " Highest port: ", sockets[-1][1]
# on Windows you should see something like this...
3960
Lowest port: 1025 Highest port: 5000
If you try to run this immeditaely again, it should fail very quickly since all dynamic ports are in the TIME_WAIT state.
There are a few ways around this:
Manage your own port assignments and
use bind() to explicitly bind your
client socket to a specific port
that you increment each time your
create a socket. You'll still have
to handle the case where a port is
already in use, but you will not be
limited to dynamic ports. e.g.
port = 5000
while True:
s = socket.socket()
s.bind(('your_host', port))
s.connect(('some_host', 80))
s.close()
port += 1
Fiddle with the SO_LINGER socket
option. I have found that this
sometimes works in Windows (although
not exactly sure why):
s.setsockopt(socket.SOL_SOCKET,
socket.SO_LINGER, 1)
I don't know if this will help in
your particular application,
however, it is possible to send
multiple XMLRPC requests over the
same connection using the
multicall method. Basically
this allows you to accumulate
several requests and then send them
all at once. You will not get any
responses until you actually send
the accumulated requests, so you can
essentially think of this as batch
processing - does this fit in with
your application design?
Update:
I tossed this into the code and it seems to be working now.
if(::connect(s_, (sockaddr *) &addr, sizeof(sockaddr)))
{
int err = WSAGetLastError();
if(err == 10048) //if socket in user error, force kill and reopen socket
{
closesocket(s_);
WSACleanup();
WSADATA info;
WSAStartup(MAKEWORD(2,0), &info);
s_ = socket(AF_INET,SOCK_STREAM,0);
setsockopt(s_,SOL_SOCKET,SO_REUSEADDR,(char*)&x,sizeof(BOOL));
}
}
Basically, if you encounter the 10048 error (socket in use), you can simply close the socket, call cleanup, and restart WSA, the reset the socket and its sockopt
(the last sockopt may not be necessary)
i must have been missing the WSACleanup/WSAStartup calls before, because closesocket() and socket() were definitely being called
this error only occurs once every 4000ish calls.
I am curious as to why this may be, even though this seems to fix it.
If anyone has any input on the subject i would be very curious to hear it
Do you close the sockets after using it?