GCP components update causes "ConnectionRefusedError: [Errno 111] Connection refused" errors - google-cloud-platform

My GCP deployment was working fine until I did "gcloud components update" two weeks ago. I understand that the command updates the libraries, and there was an option to revert back to the previous version at the end of the update, but I do not understand why there are so many connection-refused errors with the new update. On the client side, I see "502 Bad Gateway errors". These errors are not consistent. I see them occurring 12 times in the last 48 minutes.
2018-07-31 09:13:19.000 MST
Traceback (most recent call last): File "/env/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection raise err File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
Expand all | Collapse all
{
insertId: "crhycg4civkq14" 
labels: {…} 
logName: "projects/eps-server/logs/appengine.googleapis.com%2Fstderr" 
receiveTimestamp: "2018-07-31T16:13:24.966198879Z" 
resource: {…} 
textPayload: "Traceback (most recent call last):
File "/env/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
raise err
File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
" 
timestamp: "2018-07-31T16:13:19Z" 
}
2018-07-31 09:13:19.000 MST
AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fdd9f316b00>" raised exception!
Expand all | Collapse all
{
insertId: "crhycg4civkq14" 
labels: {…} 
logName: "projects/eps-server/logs/appengine.googleapis.com%2Fstderr" 
receiveTimestamp: "2018-07-31T16:13:24.966198879Z" 
resource: {…} 
textPayload: "AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fdd9f316b00>" raised exception!
" 
timestamp: "2018-07-31T16:13:19Z" 
}

Related

Why cant I access my hosted Flask webApp via EC2 (AWS)

I successfully created a Flask app locally (Ubuntu) and wanted to host this app via amazon EC2. I copied all the files to aws and run the application via
(testVenv38) ubuntu#ip-xxx-yy-39-70:~/virtuelleUmgebungen/testVenv38/flaskTest$ python3 application.py
* Serving Flask app "flaskTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: xxx-yyy-030
Afterwards I was expecting to access my application from any machine via the following link (yellow) followed by the "5000" from definition in my application.py
app.run(host='0.0.0.0', port=5000, debug=True)
(yellow link is https://3.69.xyz.abc/5000), but I get no response. How can I make my app accessible for everybody?
Here is the whole log. But thats the same I get when I run it locally, so this does not indicate the problem I guess.
(testVenv38) ubuntu#ip-xxxx:~/virtuelleUmgebungen/testVenv38/flasklogin-tutorial-master$ python3 application.py
* Serving Flask app "flask_login_tutorial" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: xxx
failed to send traces to Datadog Agent at http://localhost:8126
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 356, in _send_payload
response = self._put(payload, headers)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 332, in _put
conn.request("PUT", self._endpoint, data, headers)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 947, in send
self.connect()
File "/usr/lib/python3.8/http/client.py", line 918, in connect
self.sock = self._create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 458, in flush_queue
self._retry_upload(self._send_payload, encoded, n_traces)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 369, in iter
six.raise_from(retry_exc, fut.exception())
File "<string>", line 3, in raise_from
tenacity.RetryError: RetryError[<Future at 0x7f04d4bdfd60 state=finished raised ConnectionRefusedError>]
failed to send traces to Datadog Agent at http://localhost:8126
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 356, in _send_payload
response = self._put(payload, headers)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 332, in _put
conn.request("PUT", self._endpoint, data, headers)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 947, in send
self.connect()
File "/usr/lib/python3.8/http/client.py", line 918, in connect
self.sock = self._create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/ddtrace/internal/writer.py", line 458, in flush_queue
self._retry_upload(self._send_payload, encoded, n_traces)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/ubuntu/virtuelleUmgebungen/testVenv38/lib/python3.8/site-packages/tenacity/__init__.py", line 369, in iter
six.raise_from(retry_exc, fut.exception())
File "<string>", line 3, in raise_from
tenacity.RetryError: RetryError[<Future at 0x7f4c81faf490 state=finished raised ConnectionRefusedError>]
Edit2:
The opened ports (other than my own ip for ssh access) are the following:
Portbereich Protokoll Quelle Sicherheitsgruppen
Alle Alle sg-XXX7113746eda1a92 default
22 TCP 0.0.0.0/0 default
Edit3:
I added the following entry to incoming traffic:
0 - 65535 TCP 0.0.0.0/0 default
Still does not work.

How to debug: ConnectionResetError: [Errno 54] Connection reset by peer

I am now trying to debug this error in django for ages. Is someone able to help me? The reason for this error is that I am doing a successful ajax call, however, then at the end the error is thrown. Please see:
Exception happened during processing of request from ('127.0.0.1', 59734)
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 720, in __init__
self.handle()
File "4_Code/Floq_Django/venv/lib/python3.8/site-packages/django/core/servers/basehttp.py", line 174, in handle
self.handle_one_request()
File "Floq_Django/venv/lib/python3.8/site-packages/django/core/servers/basehttp.py", line 182, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
Thank you so much.

Django Connection reset by peer 0.0.0.0:8000

System check identified no issues (0 silenced).
January 30, 2020 - 23:18:31
Django version 3.0.2, using settings 'GroverLeagues.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
[30/Jan/2020 23:18:36] "GET / HTTP/1.1" 302 0
----------------------------------------
Exception happened during processing of request from ('<IP redacted>', 53076)
Traceback (most recent call last):
File "/usr/lib/python3.6/socketserver.py", line 654, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
self.handle()
File "/home/lakepry/groverenv/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 174, in handle
self.handle_one_request()
File "/home/lakepry/groverenv/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 182, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
I'm deploying my first django application following Digitalocean's tutorial on deploying with postgresql database and it won't let me access my webpage in a browser. In the browser it says, "This site can't be reached". Port 8000 is allowed through the firewall as well. Any ideas on what to do?
edit: Here is the link to the tutorial https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04

AWS lambda function in VPC unable to put item in Dynamodb table

This is the error I keep getting
[DEBUG] 2018-11-09T16:56:03.543Z eaa6c2df-e43f-11e8-ae27-cf0081fe803c ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
raise err
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
conn = self._new_conn()
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 139, in _new_conn
(self.host, self.timeout))
botocore.vendored.requests.packages.urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/adapters.py", line 370, in send
timeout=timeout
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/retry.py", line 271, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
botocore.vendored.requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dynamodb.eu-west-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/runtime/botocore/endpoint.py", line 222, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/var/runtime/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/var/runtime/botocore/vendored/requests/adapters.py", line 419, in send
raise ConnectTimeout(e, request=request)
botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='dynamodb.eu-west-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)'))
The crazy part is my code worked just fine yesterday. And now all of a sudden my lambda function seems not to be able to access the dynamodb. They are connected via a VPC endpoint and it worked fine yesterday. This has happened to me before and when I just left it for the night it ended up working the next day. This is unacceptable behaviour though and I need to find out what is causing it.
Any ideas?
I couldn't get this to work simply using the new vpc endpoint feature so I went for the old way of doing things which is using a nat gateway. Here's a good link on how to do it:
https://gist.github.com/reggi/dc5f2620b7b4f515e68e46255ac042a7

Django runserver error

When using the development server, I get this error:
Validating models...
0 errors found
Django version 1.3 beta 1, using settings 'book_scouts.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 283
, in run
self.finish_response()
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 323
, in finish_response
self.write(data)
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 402
, in write
self.send_headers()
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 467
, in send_headers
self._write(str(self.headers))
File "C:\Python26\lib\socket.py", line 318, in write
self.flush()
File "C:\Python26\lib\socket.py", line 297, in flush
self._sock.sendall(buffer(data, write_offset, buffer_size))
error: [Errno 10053] An established connection was aborted by the software in yo
ur host machine
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 2426)
Traceback (most recent call last):
File "C:\Python26\lib\SocketServer.py", line 283, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python26\lib\SocketServer.py", line 309, in process_request
self.finish_request(request, client_address)
File "C:\Python26\lib\SocketServer.py", line 322, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 569
, in __init__
BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
File "C:\Python26\lib\SocketServer.py", line 618, in __init__
self.finish()
File "C:\Python26\lib\SocketServer.py", line 661, in finish
self.wfile.flush()
File "C:\Python26\lib\socket.py", line 297, in flush
self._sock.sendall(buffer(data, write_offset, buffer_size))
error: [Errno 10053] An established connection was aborted by the software in yo
ur host machine
I can't pin it down to any single source. It seems to happen randomly, afaik. It causes the page I'm testing to load up really long, and the page just loads fine.
Using Django 1.3 Beta 1
UPDATE: I'm using Opera/Firefox for testing now, and the problem hasn't come up since. I'm afraid to try again in Chrome.
I got the same thing and asked on django-users google group. Nobody had any idea what's wrong.
It seems only I get this when I refresh the page from browser and not when I click on links inside the page.
Also just as a heads up: I tried on Python 2.6 and 2.7 both stock version (form python.org website) and ActiveState one. Same thing happens randomly.
LE: I'm getting this on the stable version of Django.