Connection reset by peer error while using celery stats() - python-2.7

I'm trying to get stats for my celery Que (rabbitmq). I'm using celery.app.control.Inspect().stats() API. I'm doing this on a web server, I can get the stats only one time. If I refresh the page I'm getting "[Errno 104] Connection reset by peer" Error. how can I deal with this.
/init.py
celtasks = Celery(app.name,"rabbit mq url")
/helpers.py
get_stats():
stats = celtasks.control.Inspect().stats()
return stats
whenever there is a request "get_stats" function is hit. It is only working for the first request after this, it says connection reset by peer error.
If I go by connection has been reset and try to create the connection again, I get error
updated /helpers.py
get_stats():
celtasks = Celery(app.name,"rabbit mq url")
stats = celtasks.control.Inspect().stats()
return stats
Rabbitmq logs
=WARNING REPORT==== 10-Jul-2017::14:11:54 ===
closing AMQP connection <0.29185.6> (10.246.170.70:48618 -> 10.24.83.115:5672):
connection_closed_abruptly
=WARNING REPORT==== 10-Jul-2017::14:11:54 ===
closing AMQP connection <0.29197.6> (10.246.170.70:48620 -> 10.24.83.115:5672):
connection_closed_abruptly
"rabbit#oser000300.log-20170625" 9054L, 361662C

AT most times , CONNECTION RESET BY PEER is because the server close the connection itself, however the client does not know . When client want to communicate to sever through this broke connection, it receive this ERROR. In your case , maybe the hang time (time interval between two stats()) is too long, and server think this connection is useless and close it .

Related

issue from bsrn data retrieval via ftp

I was able to retrive data from (BSRN) after receiving username and password! After a week, it led to the issue shown below! Any suggestions on how to solve it?
data, metadata = pvlib.iotools.get_bsrn(
start=pd.Timestamp(2020,2,1), end=pd.Timestamp(2020,2,2),
station='BOS', username='redacted', password='redacted')
data
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

How to gracefully handle auto disconnect of Daphne websockets

Daphne has a parameter --websocket_timeout link. As mentioned in the doc,
--websocket_timeout WEBSOCKET_TIMEOUT
Maximum time to allow a websocket to be connected. -1 for infinite.
The socket is disconnected and no further communication can be done. However, the client does not receives a disconnect event, hence cant handle it gracefully. How does my client get to know whether the socket is disconnected or not?? I don't want to keep (at client) a timer nor want to keep rechecking it.
This is how I deploy my app
daphne -b 0.0.0.0 -p 8000 --websocket_timeout 1800 app.asgi:application
The socket gets auto-disconnected after every 30 mins, but the client never gets to know about this.
Whats the right way to go about it,.??
Update
Trying to send an event before the connection is closed. I'm over-riding my websocket_disconnect handler that sends the json before disconnecting. However, it does not send the event.
class Consumer(AsyncJsonWebsocketConsumer):
async def websocket_disconnect(self, message):
"""Over-riding."""
print('Inside websocket_disconnect consumer')
await self.send_json(
"event": "disconnecting..."
)
await super().websocket_disconnect(message)
I'm not sure it's a problem that needs a solution. The client has a certainty that after X minutes of inactivity it will get disconnected, where X is determined by the server. It has no certainty it won't happen before that. So you need connectivity handling code regardless.
While it seems dirty to keep an idling connection around, I can't imagine it costing a lot of resources.
Your premise that the client doesn't get to know about it is wrong. When you register the onclose handler, the client receives a disconnect event and can act accordingly.

Error reconnecting boost beast (asio) websocket and http connection after disconnect

I am creating a client application that connects to a server using a an ssl Websocket connection and an ssl Http (Keep-Alive) connection and I am using boost::beast package to do the same. So as to detect a dead connection i have implemented a simple ping-pong mechanism. These all work fine, but an issue comes up when handling the ping-pong failure. The issue is as follows:
For testing my code i connected to the remote server, sent few messages and then turned off my wifi. As expected after a certain period it detected that it did not receive any message from the server and it tried to do an async_shutdown for the http connection and an async_close for the websocket connection. First thing i noticed was that both these calls block their respective strands until the wifi is back up.
And after the wifi is up, the application tries to reset the stream before reconnect:
void HttpKeepAliveConnection::recreateSocket()
{
_receivedPongForLastPing = true;
_sslContext.reset(new boost::asio::ssl::context({boost::asio::ssl::context::sslv23_client}));
_stream.reset(new HttpStream(_ioContext, *_sslContext));
}
And reset ws variable for websocket:
void WebsocketConnection::recreateSocket()
{
_receivedPongForLastPing = true;
_sslContext.reset(new boost::asio::ssl::context({boost::asio::ssl::context::sslv23_client}));
_ws.reset(new WebSocket(_ioContext, *_sslContext));
}
Unfortunately it fails at either on_connect or on_ssl_handshake. Following are my logs:
156 AsioConnectionBase.cpp:53 (2018-08-06 15:34:38.458536) [0x00007ffff601e700] : Started connect sequence. Connection Name: HttpKeepAliveConn
157 AsioConnectionBase.cpp:122 (2018-08-06 15:34:38.459802) [0x00007ffff481b700] : Failed establishing connection to destination. Connection failed. Connection Name: HttpKeepAliveConn. Host: xxxxxxxxx. Port: 443. Error: Operation canceled
158 APIManager.cpp:175 (2018-08-06 15:34:38.459886) [0x00007ffff481b700] : Received error callback from connection. Restarting connection in a sec. Connection Name: HttpKeepAliveConn
159 AsioConnectionBase.cpp:53 (2018-08-06 15:34:39.460009) [0x00007ffff481b700] : Started connect sequence. Connection Name: HttpKeepAliveConn
160 HttpKeepAliveConnection.cpp:32 (2018-08-06 15:34:39.460515) [0x00007ffff481b700] : Failed ssl handshake. Connection failed.Connection Name: HttpKeepAliveConn. Host: xxxxxxxxx. Port: 443. Error: Bad file descriptor
161 APIManager.cpp:175 (2018-08-06 15:34:39.460674) [0x00007ffff481b700] : Received error callback from connection. Restarting connection in a sec. Connection Name: HttpKeepAliveConn
So I have 2 questions:
How do we close a connection if internet is down and a proper tcp close is not possible.
Before reconnecting what are the variables in boost::beast (or for that matter boost::asio as boost::beast is built on top of asio) that needs to be reset
Have been stuck trying to debug this for couple of hours. Any help is appreciated
EDIT
So I figured out where I went wrong. Both Alan Birtles and Vinnie Falco were right. The way to close a dead ssl connection after your ping timer has expired (and none of the handlers have returned yet) is
In your timer handler
_stream->lowest_layer().close();
For websocket
_ws->lowest_layer().close();
Wait for one of your handlers (typically read handler) to return with error (typically boost::asio::error::operation_aborted error). From there, queue the start of the next reconnect. (Do not queue the reconnect immediately after step 1, it will result in memory issues that I faced. I know this is asio 101, but is easy to forget)
For resetting socket, all that is required is for the stream to be reset
_stream.reset(new HttpStream(_ioContext, _sslContext));
For websocket
_ws.reset(new WebSocket(_ioContext, _sslContext));
I don't think asio::ssl::stream can be used again after being closed.
How do we close a connection if internet is down and a proper tcp close is not possible.
Simply allow the socket or stream object to be destroyed.

Gremlin remote command fails with timeout error: Host did not respond in a timely fashion

I connected to a remote gremlin server via gremlin groovy shell. Connection succeeded. But for any remote command I try to execute it gives timeout error. Even for command :> 1+1
gremlin> :remote connect tinkerpop.server conf/senthil.yaml
==>Connected - 10.40.40.65/10.40.40.65:50080
gremlin> :> 1+1
Host did not respond in a timely fashion - check the server status and submit again.
Display stack trace? [yN]
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException: Host did not respond in a timely fashion - check the server status and submit again.
at org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:120)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)
at org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)
at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:101)
at org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
This is my conf file: remote.yaml
hosts: [10.40.40.65]
port: 50080
serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
I'm using dynamodb + titan.
You might not have a truly successful connection. The console (and underlying driver) is optimistic in that it really doesn't fail a connection until a request is sent as it expects the server may come online "later". I would go back to investigating if the server is running, if you have the right IP, if the host property is not set to something like "localhost" if you are connecting remotely, if the port is open, that you are using a compatible version of TinkerPop, etc.

How do I set timeout for TIdHTTPProxyServer (not connection timout)

I am using TIdHTTPProxyServer and now I want to terminate connection when it is success to connect to the target HTTP server but receive no response for a long time(i.g. 3 mins)
Currently I find no related property or event about it. And even if the client terminate the connection before the proxy server receive the response from the HTTP server. OnException Event will not be fired until the proxy server receive the response. (That is, if the proxy server still receive no response from HTTP Server, I even do not know the client has already terminate the connection...)
Any help will be appreciated.
Thanks!
Willy
Indy uses infinite timeouts by default. To do what you are asking for, you need to set the ReadTimeout property of the outbound connection to the target server. You can access that connection via the TIdHTTPProxyServerContext.OutboundClient property. Use the OnHTTPBeforeCommand event, which is triggered just before the OutboundClient connects to the target server, eg:
#include "IdTCPClient.hpp"
void __fastcall TForm1::IdHTTPProxyServer1HTTPBeforeCommand(TIdHTTPProxyServerContext *AContext)
{
static_cast<TIdTCPClient*>(AContext->OutboundClient)->ReadTimeout = ...;
}