Arduino hangs during multiple http requests on Linux but not on Windows - c++

i have a problem when i send http requests from the arduino to linux mysql server. It randomly hangs after some requests (~150) but on windows it runs smoothly.
Because of this i am thinking that the problem is not on the arduino code but somewhere else.
The linux server runs on a raspberry pi (Raspbian).
Any suggestions?
The arduino code HERE
if(!getPage(server,serverPort)) Serial.print(F("Fail "));
byte getPage(IPAddress ipBuf,int thisPort)
{
int inChar;
Serial.print(F("connecting..."));
if(client.connect(ipBuf,thisPort))
{
Serial.println(F("connected"));
strcpy(outBuf,"GET /write3.php?value0=");
itoa(value0,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value1=");
itoa(dht_humidity,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value2=");
itoa(temperature,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value3=");
itoa(pressure,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value4=");
itoa(altitude,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value5=");
itoa(gust,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value6=");
itoa(dir,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value7=");
itoa(rain,tBuf,10);
strcat(outBuf,tBuf);
strcat(outBuf,"&value8=");
itoa(knots,tBuf,10);
strcat(outBuf,tBuf);
client.write(outBuf);
client.println(" HTTP/1.1");
client.println("Host: 192.168.1.3");
client.println("Connection: close");
client.println();
EDIT: Just hanged at DEBUG___3
Serial.println("DEBUG___2");
client.write(outBuf);
Serial.println("DEBUG___3");
client.println(" HTTP/1.1");
Serial.println("DEBUG___4");
client.println("Host: 192.168.1.3");
Serial.println("DEBUG___5");
client.println("Connection: close");
client.println();
I captured the network traffic with wireshark when it hanged:
(Large pics: hanged http request ---- successfull http request)
Here is the http request that hanged
And here is a successful request:
Any ideas guys? Still stuck there !!!

Presuming you are certain the outBuf is not overflowed (it is smaller than maximum possible request string size and you are using strcat (source of all evil))
The server may timeout in the long time between when the connection is opened, and when you actually send any bytes. Preconstruct the outBuf so that it is ready to go when the connection opens.
strncat(outBuf,...,127);
... request is fully constructed
if(client.connect(ipBuf,thisPort)) {
client.write(outBuf);
client.println(" HTTP/1.1");
client.println("Host: 192.168.1.3");
...
To the server, the connection that is SYN'd but not used is a resource to be recovered. The Arduino will be so slow, it will look like an idle connection. Also a SYN flood is an old denial of service vector that most servers will protect against.
A difference in the timeout values would explain why the Linux and Win based server act different. You could confirm this by running Wireshark on the traffic. If the server is timing out on the Arduino, you will see this sequence:
the SYN handshake proceed between server and Arduino
a little time passes
a RST from the server indicating the connection is dead
the Arduino sends the request string to the now dead connection
the Arduino program hangs at the client.write() because the connection state is out of
sync
the client will probably be retrying - you should see some retransmissions

Related

Cesanta Mongoose - problems when connecting to localhost

I'm having issues building an HTTP server using the Cesanta Mongoose web server library. The issue that I'm having occurs when I have an HTTP server built to listen on port 8080, and a client sending an HTTP request to localhost:8080. The problem is that the server processes the request fine and sends back a response, but the client only processes and prints the response after I kill the server process. Basically Mongoose works where you create connections which take an event handler function, ev_handler(). This event handler function is called whenever an
"event" occurs, such as the receiving of a request or a reply. On the server side, the event handler function is called fine when it receives a request from the client on 8080. However, the client-side event handler function is not called when the response sends the reply, but is called only after the server process is killed. I suspected that this may have something to do with the fact that the connection is on localhost, and I was right - this issue does not occur when the client sends requests to addresses other than localhost. The event handler function is called fine. Here is the ev_handler function on the client-side for reference:
static void ev_handler(struct mg_connection *c, int ev, void *p) {
if (ev == MG_EV_HTTP_REPLY) {
struct http_message *hm = (struct http_message *)p;
c->flags |= MG_F_CLOSE_IMMEDIATELY;
fwrite(hm->message.p, 1, (int)hm->message.len, stdout);
putchar('\n');
exit_flag = 1;
} else if (ev == MG_EV_CLOSE) {
exit_flag = 1;
};
}
Is this a common issue when trying to establish a connection on localhost with a server on the same computer?
The cause of such behavior is the fact that client connection does not fire an event until all data is read. How client knows the all data is read? There are 3 possibilities:
Server has sent Content-Length: XXX header and client has read XXX bytes of the message body, so it knows it received everything.
Server has sent Transfer-Encoding: chunked header, and sent all data chunks followed by an empty chunk. When client receives an empty chunk, it knows it received everything.
Server set neither Content-Lenth, nor Transfer-Encoding. Client does not know in this case what is the size of the body, and it keeps reading until server closes the connection.
What you see is (3). Solution: set Content-Length in your server code.

Error reconnecting boost beast (asio) websocket and http connection after disconnect

I am creating a client application that connects to a server using a an ssl Websocket connection and an ssl Http (Keep-Alive) connection and I am using boost::beast package to do the same. So as to detect a dead connection i have implemented a simple ping-pong mechanism. These all work fine, but an issue comes up when handling the ping-pong failure. The issue is as follows:
For testing my code i connected to the remote server, sent few messages and then turned off my wifi. As expected after a certain period it detected that it did not receive any message from the server and it tried to do an async_shutdown for the http connection and an async_close for the websocket connection. First thing i noticed was that both these calls block their respective strands until the wifi is back up.
And after the wifi is up, the application tries to reset the stream before reconnect:
void HttpKeepAliveConnection::recreateSocket()
{
_receivedPongForLastPing = true;
_sslContext.reset(new boost::asio::ssl::context({boost::asio::ssl::context::sslv23_client}));
_stream.reset(new HttpStream(_ioContext, *_sslContext));
}
And reset ws variable for websocket:
void WebsocketConnection::recreateSocket()
{
_receivedPongForLastPing = true;
_sslContext.reset(new boost::asio::ssl::context({boost::asio::ssl::context::sslv23_client}));
_ws.reset(new WebSocket(_ioContext, *_sslContext));
}
Unfortunately it fails at either on_connect or on_ssl_handshake. Following are my logs:
156 AsioConnectionBase.cpp:53 (2018-08-06 15:34:38.458536) [0x00007ffff601e700] : Started connect sequence. Connection Name: HttpKeepAliveConn
157 AsioConnectionBase.cpp:122 (2018-08-06 15:34:38.459802) [0x00007ffff481b700] : Failed establishing connection to destination. Connection failed. Connection Name: HttpKeepAliveConn. Host: xxxxxxxxx. Port: 443. Error: Operation canceled
158 APIManager.cpp:175 (2018-08-06 15:34:38.459886) [0x00007ffff481b700] : Received error callback from connection. Restarting connection in a sec. Connection Name: HttpKeepAliveConn
159 AsioConnectionBase.cpp:53 (2018-08-06 15:34:39.460009) [0x00007ffff481b700] : Started connect sequence. Connection Name: HttpKeepAliveConn
160 HttpKeepAliveConnection.cpp:32 (2018-08-06 15:34:39.460515) [0x00007ffff481b700] : Failed ssl handshake. Connection failed.Connection Name: HttpKeepAliveConn. Host: xxxxxxxxx. Port: 443. Error: Bad file descriptor
161 APIManager.cpp:175 (2018-08-06 15:34:39.460674) [0x00007ffff481b700] : Received error callback from connection. Restarting connection in a sec. Connection Name: HttpKeepAliveConn
So I have 2 questions:
How do we close a connection if internet is down and a proper tcp close is not possible.
Before reconnecting what are the variables in boost::beast (or for that matter boost::asio as boost::beast is built on top of asio) that needs to be reset
Have been stuck trying to debug this for couple of hours. Any help is appreciated
EDIT
So I figured out where I went wrong. Both Alan Birtles and Vinnie Falco were right. The way to close a dead ssl connection after your ping timer has expired (and none of the handlers have returned yet) is
In your timer handler
_stream->lowest_layer().close();
For websocket
_ws->lowest_layer().close();
Wait for one of your handlers (typically read handler) to return with error (typically boost::asio::error::operation_aborted error). From there, queue the start of the next reconnect. (Do not queue the reconnect immediately after step 1, it will result in memory issues that I faced. I know this is asio 101, but is easy to forget)
For resetting socket, all that is required is for the stream to be reset
_stream.reset(new HttpStream(_ioContext, _sslContext));
For websocket
_ws.reset(new WebSocket(_ioContext, _sslContext));
I don't think asio::ssl::stream can be used again after being closed.
How do we close a connection if internet is down and a proper tcp close is not possible.
Simply allow the socket or stream object to be destroyed.

After Successfull TLS handshake the server closes with error SSL routines:SSL3_GET_RECORD:wrong version number

We are using openssl 1.0.2k for our TLS related functionalities.
In one of our deployment the client is able to complete the TLS handshakes using TLSv1.2 and was able to send application data towards server.After some requests the TLS connections closed from the server side with the below error
"error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number"
TLS handshake steps:
1. Client hello
2. Server Hello
3. Certificate,Certificate Request, Server hello done
4. Certificate,Client Key Exchange,Change Cipher spec,Encrypted handshake message
5. Change Cipher spec,Encrypted handshake message
6. Application data exchanges between client and server
7. Encrypted Alert(server to client)
8. Encrypted Alert( client to server
The error logs from server side says "error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number"
Can you please let us know the cause for this issue. If the ssl version is mismatching then the handshake phase should not succeed right?
But in our case handshake is successful and after some application data transfer our server is failing with this error.
If the ssl version is mismatching then the handshake phase should not succeed right?
No. Any TLS packet have header, and header has TLS version inside:
(
byte - record_type
byte[2] - version
byte[2] - length
) header
byte[length] - encrypted or raw data
Header is always in raw, it is never encrypted. Even if during handshake client sent TLS 1.2 version in all TLS packets, he can send another version after handshake is finished. Or someone in between can modify network traffic. In this case OpenSSL throws described error.
In my case, I was using OpenSSL for client functionality.
I was calling SSL_set_connect_state after SSL_connect. It should be called before.
SSL_set_connect_state (for client only) cleans up all the state!
snippet:
void SSL_set_connect_state(SSL *s)
{
s->server = 0;
s->shutdown = 0;
ossl_statem_clear(s);
s->handshake_func = s->method->ssl_connect;
clear_ciphers(s);
}
In my case:
1) Client <-> Server handshake succeeded.
2) SSL_write from client side (client sending message to server) lead to exact same error as mentioned in question (on server side)
I looked at pkt dump on server side.
read from 0x2651570 [0x2656c63] (5 bytes => 5 (0x5)) .
0000 - 16 03 01 01 e2 .....
ERROR
139688140752544:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong >version number:s3_pkt.c:337:
1) 5 Bytes read in the above snipped is the size of SSL record. Server received data, and it attempted reading SSL record.
2) 1'st byte of the record is the SSL record type In this case ===> x16 => '22'
This itself is wrong, as far as server is concerned, handshake was successful and it was expecting application data. Instead it received data with SSL record for handshake, hence it was throwing the error.
A correct snippet of application data is as follows: 'x17' ==> 23
read from 0x2664f80 [0x2656c63] (5 bytes => 5 (0x5)) .
0000 - 17 03 03 00 1c
Since SSL_set_connect_state was called after connecting, client state was lost and SSL_write will attempt handshake if handshake wasnt performed before (client thought so as its state was lost!)
More data on these SSL records can be found here:
https://www.ibm.com/support/knowledgecenter/SSB23S_1.1.0.12/gtps7/s5rcd.html

Python socket.recv Closing Socket Prematurely

I have a web proxy that starts a TCP listener socket that accepts connections from clients. The listener accepts connections via:
clientConnection, clientAddress = listenerSocket.accept()
and then a new thread handles the client connection from there.
To mock a client connection, I am using telnet to connect to the proxy and issue commands. The proxy needs to receive data from telnet and I need to make sure that I receive all of it. To achieve this, I am doing the following:
while True:
requestBytes = clientConnection.recv(1024)
if not requestBytes:
break
requestBuffer += requestBytes
The proxy then decodes the bytes and does some things with them that takes a little bit of time, and then has to send a response back to the same client. However, when using the above code the connection with clientConnection gets closed long before I can process the bytes and respond.
Here's what I don't understand, when I use the following instead:
while True:
requestBytes = clientConnection.recv(1024)
requestBuffer += requestBytes
break
It works just fine and the clientConnection remains intact. This obviously has a problem if I receive more than 1024 bytes, but the clientConnection does not get closed.
More specifically, the error occurs after I have a response to send to the client and call:
clientConnection.sendall(response)
clientConnection.shutdown(1)
clientConnection.close()
The line clientConnection.shutdown(1) throws the error:
[Errno 107] Transport endpoint is not connected
which is confusing because somehow it was able to still call sendall on the previous line. Note that I did not actually receive anything on the client side.
I am sure that the connection is not getting closed elsewhere in the code. What exactly is happening here and what is the best way to do something like recvall and keep the clientConnection open?

boost asio async_read header connection closes too early

Providing a MCVE is going to be hard, the scenario is the following:
a server written in c++ with boost asio offers some services
a client written in c++ with boost asio requests services
There are custom headers and most communication is done using multipart/form.
However, in the case where the server returns a 401 for an unauthorized access,
the client receives a broken pipe (system error 32).
AFAIK this happens when the server connection closes too early.
So, running into gdb, I can see that the problem is indeed the transition from the async_write which sends the request, to the async_read_until which reads the first line of the HTTP Header:
The connect routine sends the request from the client to the server:
boost::asio::async_write(*socket_.get(),
request_,
boost::bind(&asio_handler<http_socket>::write_request,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
And the write_request callback, checks if the request was sent OK, and then reads the first line (until the first newline):
template <class T>
void asio_handler<T>::write_request(const boost::system::error_code & err,
const std::size_t bytes)
{
if (!err) {
// read until first newline
boost::asio::async_read_until(*socket_,
buffer_,
"\r\n",
boost::bind(&asio_handler::read_status_line,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else {
end(err);
}
}
The problem is that the end(err) is always called with a broken pipe (error code 32). Meaning, as far as I understand, that the server closed the connection. The server indeed closes the connection, but only after it has sent a message HTTP/1.1 401 Unauthorized.
using curl with the appropriate request, we do get the actual message/error before the server closes the connection
using our client written in C++/boost asio we only get the broken pipe and no data
only when the server leaves the connection open, do we get to the point of reading the error (401) but that defeats the purpose, since now the connection is left open.
I would really appreciate any hints or tips. I understand that without the code its hard to help, so I can add more source at any time.
EDIT:
If I do not check for errors between writing the request, and reading the server reply, then I do get the actual HTTP 401 error. However this seems counter-intuitive, and I am not sure why this happens or if it is supposed to happen.
The observed behavior is allowed per the HTTP specification.
A client or server may close the socket at anytime. The server can provide a response and close the connection before the client has finished transmitting the request. When writing the body, it is recommended that clients monitor the socket for an error or close notification. From the RFC 7230, HTTP/1.1: Message Syntax and Routing Section 6.5. Failures and Timeouts:
6.5. Failures and Timeouts
A client, server, or proxy MAY close the transport connection at any time. [...]
A client sending a message body SHOULD monitor the network connection for an error response while it is transmitting the request. If the client sees a response that indicates the server does not wish to receive the message body and is closing the connection, the client SHOULD immediately cease transmitting the body and close its side of the connection.
On a graceful connection closure, the server will send a response to the client before closing the underlying socket:
6.6. Tear-down
A server that sends a "close" connection option MUST initiate a close of the connection [...] after it sends the response containing "close". [...]
Given the above behaviors, there are three possible scenarios. The async_write() operation completes with:
success, indicating the request was written in full. The client may or may not have received the HTTP Response yet
an error, indicating the request was not written in full. If there is data available to be read on the socket, then it may contain the HTTP Response sent by the server before the connection terminated. The HTTP connection may have terminated gracefully
an error, indicating the request was not written in full. If there is no data available to be read on the socket, then the HTTP connection was not terminated gracefully
Consider either:
initiating the async_read() operation if the async_write() is successful or there is data available to be read
void write_request(
const boost::system::error_code & error,
const std::size_t bytes_transferred)
{
// The server may close the connection before the HTTP Request finished
// writing. In that case, the HTTP Response will be available on the
// socket. Only stop the call chain if an error occurred and no data is
// available.
if (error && !socket_->available())
{
return;
}
boost::asio::async_read_until(*socket_, buffer_, "\r\n", ...);
}
per the RFC recommendation, initiate the async_read() operation at the same time as the async_write(). If the server indicates the HTTP connection is closing, then the client would shutdown its send side of the socket. The additional state handling may not warrant the extra complexity