I am consistently seeing a 3+ second delay when opening a connection to a Unidata server via OpenAsync and the ADO.NET provider by Rocket.
What can I do to diagnose this delay?
I use jdbc and odbc connections all the time, and three seconds just ain't right.
You can start by breaking things down into client, network, server.
On the client, is it the same if you use an IP address vs hostname? If different, then DNS delay. Do you have any antivirus or other software that might be delaying connections? Can you turn that off, or do a test from a freshly configured machine?
For network, can you ping the unidata server in millseconds? Are there firewalls in the way? Can you install wireshark and filter packets by destination? Is there a delay when you ssh? Is the server configured with LDAP/AD, or local authentication - could be connecting to another server to authenticate?
On the server, you can turn on uniobjects logging, then restart unirpcd. Do you have something in your LOGIN/UOLOGIN that is hanging up the remote connection?
$cat $UDTHOME/serverdebug
udcs 9 /tmp/udcs.log
I have a server application (unimrcpserver.exe) that is answering requests from client processes. This server process listens to several ports.
with netstat -a command I get the following lines for my process.
TCP 192.168.10.65:2544 MERTB-PC:0 LISTENING
TCP 192.168.10.65:2554 MERTB-PC:0 LISTENING
TCP 192.168.10.65:9060 MERTB-PC:0 LISTENING
(netstat output is long I only put relevant lines here)
Normally when the system works I make requests to the server from these ports and each of them works fine.
When I was doing stress tests I saw a situation where the system no longer responded the my requests that I make through the port 2554.
netstat -a still gives me the above lines so the server is somehow still listening to this port. When I run telnet on the same machine it gives an error :
telnet 192.168.10.65 2554
Connecting To 192.168.10.65...Could not open connection to the host, on port 2554: Connect failed
I also wrote a simple program with c++ to get the exact error message that the system generates to a connect() request. This time I get the following error:
No connection could be made because the target machine actively refused it
Additional info: Everything is on the same Windows machine. Firewall is disabled. This situation occurred only once when I am doing stress tests that sends multiple requests at the same time. Before the situation occurred the system handled to around 13000 requests, which took around half an hour.
So the question is : How can this situation occur? The port is being reported as "LISTENING" with netstat but I cannot connect to it. If it can be caused by a programming error what kind of an error can cause this kind of behavior?
A new connection can be "actively refused" under several conditions:
there is no LISTENING socket on the IP:Port being connected to.
there is a LISTENING socket, but its backlog of pending connections is full, so it cannot accept a new connection at that moment.
A firewall is blocking it. Though the firewall is more likely to use a different error, if it sends an error at all.
Since there is a LISTENING socket, #2 is the most likely/common case. If so, it means the server app is not accepting clients from its backlog fast enough, if at all.
A client cannot differentiate between these conditions. All it can do is detect the connect failure - WSAECONNREFUSED or ECONNREFUSED, depending on platform - and try again later.
So the question is : How can this situation occur? The port is being reported >>as "LISTENING" with netstat but I cannot connect to it. If it can be caused >>by a programming error what kind of an error can cause this kind of behavior?
Yes,It could be caused by a programming error on the server. I have seen it happening when the server's listening thread is deadlocked. The socket's state is "listening" but if the listening thread has some global state and is blocked on other threads waiting on a mutex to be released you will encounter this.
Also, like others here stated if the CPU is loaded due to your stress test and that might cause the server to refuse connections since the threads might be busy processing and the listening thread never got a chance to accept the connection.
I have client server application that works with Firebird server. Everytime when clients connect to the server they(client apps) don't check if there is a network connection to the server so at this time my application sometimes freezes when the server computer is switched off or service has stopped, so first of all I need to check connection if remote host is switched on or at some port anything listening....
Before establishing the connection I need to check it and make sure server and service is running using Indy components.
Any ideas? also I can use IcmpClient to ping remote host and then establish connection but which is the most optimal way ?
If you just want to check if the server computer can be reached, you could do a "ping" to check that. However, if you want to check if a specific TCP port is open, then the only way to find that out is to actually do a proper connect, which leads to the "freezing" program while the connection times out if there is no-one listening on that port.
TCP standard has "simultaneous open" feature.
The implication of the feature, client trying to connect to local port, when the port is from ephemeral range, can occasionally connect to itself (see here).
So client think it's connected to server, while it actually connected to itself. From other side, server can not open its server port, since it's occupied/stolen by client.
I'm using RHEL 5.3 and my clients constantly tries to connect to local server.
Eventually client connects to itself.
I want to prevent the situation. I see two possible solutions to the problem:
Don't use ephemeral ports for server ports.
Agree ephemeral port range and configure it on your machines (see ephemeral range)
Check connect() as somebody propose here.
What do you thinks?
How do you handle the issue?
P.S. 1
Except of the solution, which I obviously looking for,
I'd like you to share your real life experience with the problem.
When I found the cause of the problem, I was "astonished" on my work place people are not familiar with it. Polling server by connecting it periodically is IMHO common practice,
so how it's that the problem is not commonly known.
When I stumbled into this I was flabbergasted. I could figure out that the outgoing
port number accidentally matches the incoming port number, but not why the TCP
handshake (SYN SYN-ACK ACK) would succeed (ask yourself: who is sending the ACK if
there is nobody doing a listen() and accept()???)
Both Linux and FreeBSD show this behavior.
Anyway, one solution is to stay out of the high range of port numbers for servers.
I noticed that Darwin side-steps this issue by not allowing the outgoing port
to be the same as the destination port. They must have been bitten by this as well...
An easy way to show this effect is as follows:
while true
do
telnet 127.0.0.1 50000
done
And wait for a minute or so and you will be chatting with yourself...
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
hello?
hello?
Anyway, it makes good job interview material.
Bind the client socket to port 0 (system assigns), check the system assigned port, if it matches the local server port you already know the server is down and and can skip connect().
For server you need to bind() socket to port. Once addr:port pair had socket bound, it will no longer be used for implicit binding in connect().
No problem, no trouble.
Note that this solution is theoretical and I have not tested it on my own. I've not experienced it before (or did not realize) and hopefully I won't experience it anymore.
I'm assuming that you cannot edit neither the client source code nor the server source. Additionally I'm assuming the real problem is the server which cannot start.
Launch the server with a starter application. If the target port that the server will bind is being used by any process, create an RST (reset packet) by using raw sockets.
The post below briefly describes what an RST packet is (taken from http://forum.soft32.com/linux/killing-socket-connection-cmdline-ftopict473059.html)
You have to look at a "raw socket" packet generator.
And you have to be superuser.
You probably need a network sniffer as well.
http://en.wikipedia.org/wiki/Raw_socket
http://kerneltrap.org/node/3072 - TCP RST attacks
http://search.cpan.org/dist/Net-RawIP/lib/Net/RawIP.pm - a Perl module
http://mixter.void.ru/rawip.html - raw IP in C
In the C version, you want a TH_RST packet.
RST is designed to handle the following case.
A and B establish a connection.
B reboots, and forgets about this.
A sends a packet to B to port X from port Y.
B sends a RST packet back, saying "what are you talking about? I don't
have a connection with you. Please close this connection down."
So you have to know/fake the IP address of B, and know both ports X
and Y. One of the ports will be the well known port number. The other
you have to find out. I thnk you also need to know the sequence
number.
Typically people do this with a sniffer. You could use a switch with a
packet mirroring function, or run a sniffer on either host A or B.
As a note, Comcast did this to disable P2P traffic.
http://www.eff.org/wp/packet-forgery-isps-report-comcast-affair
In our case we don't need to use a sniffer since we know the information below:
So you have to know/fake the IP address of B, and know both ports X
and Y
X = Y and B's IP address is localhost
Tutorial on http://mixter.void.ru/rawip.html describes how to use Raw Sockets.
NOTE that any other process on the system might also steal our target port from ephemeral pool. (e.g. Mozilla Firefox) This solution will not work on this type of connections since X != Y B's IP address is not localhost but something like 192.168.1.43 on eth0. In this case you might use netstat to retrieve X, Y and B's IP address and then create a RST packet accordingly.
Hmm, that is an odd problem. If you have a client / server on the same machine and it will always be on the same machine perhaps shared memory or a Unix domain socket or some other form of IPC is a better choice.
Other options would be to run the server on a fixed port and the client on a fixed source port. Say, the server runs on 5000 and the client runs on 5001. You do have the issue of binding to either of these if something else is bound to them.
You could run the server on an even port number and force the client to an odd port number. Pick a random number in the ephemeral range, OR it with 1, and then call bind() with that. If bind() fails with EADDRINUSE then pick a different odd port number and try again.
This option isn't actually implemented in most TCPs. Do you have an actual problem?
That's an interesting issue! If you're mostly concerned that your server is running, you could always implement a heartbeat mechanism in the server itself to report status to another process. Or you could write a script to check and see if your server process is running.
If you're concerned more about the actual connection to the server being available, I'd suggest moving your client to a different machine. This way you can verify that your server at least has some network connectivity.
In my opinion, this is a bug in the TCP spec; listening sockets shouldn't be able to send unsolicited SYNs, and receiving a SYN (rather than a SYN+ACK) after you've sent one should be illegal and result in a reset, which would quickly let the client close the unluckily-chosen local port. But nobody asked for my opinion ;)
As you say, the obvious answer is not to listen in the ephemeral port range. Another solution, if you know you'll be connecting to a local machine, is to design your protocol so that the server sends the first message, and have a short timeout on the client side for receiving that message.
The actual problem you are having seems to be that while the server is down, something else can use the ephemeral port you expect for your server as the source port for an outgoing connection. The detail of how that happens is separate to the actual problem, and it can happen in ways other than the way you describe.
The solution to that problem is to set SO_REUSEADDR on the socket. That will let you create a server on a port that has a current outgoing connection.
If you really care about that port number, you can use operating specific methods to stop it being allocated as an ephemeral port.
How do I detect if a remote client is running Remote Desktop Protocol? and it is also accepting remote desktop connections ??
Like Open an port to detect HTTP and send request, receive request headers and see in request headers information about HTTP so I will know the person is running HTTP weather if he changed the port e.g: running HTTP 6551.
Attempt and make a connection with something that is RDP-connection aware (RDP is not HTTP). Of course, failing to establish an initial handshake is not proof that a connection can not be established. It could be blocked by a firewall, listening on another port, etc.
The MS-RDPBCGR specification, page 16 talks about connecting which in turn defers to X.224, go figure.
It'd likely just be easiest to use Wireshark and observe in-the-wild behavior to develop a minimal detection case. I suspect only the very initial portion of the handshake needs to be generated/replayed in order to "decide" that it's a listening RDP server.
(Or, perhaps use an existing RDP client which has this "test connect" functionality or the ability to be scripted.)
A fast way is to pen a shell and type
telnet IPADDRESS 3389
If you get a connection, chances are good that an RDP server is on the other side. RDP can run on any port, but TCP Port 3389 is set per default.
Windows 7 requires some extra steps to enable the telnet Client.
You could do netstat -a in the command line and see if the default port for remote desktop connection is listening, ie. TCP:3389 but thats only if the client hasn't changed the ports for MSTSC