This is the error I keep getting
[DEBUG] 2018-11-09T16:56:03.543Z eaa6c2df-e43f-11e8-ae27-cf0081fe803c ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
raise err
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
conn = self._new_conn()
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 139, in _new_conn
(self.host, self.timeout))
botocore.vendored.requests.packages.urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/adapters.py", line 370, in send
timeout=timeout
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/retry.py", line 271, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
botocore.vendored.requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dynamodb.eu-west-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/runtime/botocore/endpoint.py", line 222, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/var/runtime/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/var/runtime/botocore/vendored/requests/adapters.py", line 419, in send
raise ConnectTimeout(e, request=request)
botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='dynamodb.eu-west-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)'))
The crazy part is my code worked just fine yesterday. And now all of a sudden my lambda function seems not to be able to access the dynamodb. They are connected via a VPC endpoint and it worked fine yesterday. This has happened to me before and when I just left it for the night it ended up working the next day. This is unacceptable behaviour though and I need to find out what is causing it.
Any ideas?
I couldn't get this to work simply using the new vpc endpoint feature so I went for the old way of doing things which is using a nat gateway. Here's a good link on how to do it:
https://gist.github.com/reggi/dc5f2620b7b4f515e68e46255ac042a7
Related
I ran the following python sample code on this page (you need a private key to run this): https://cloud.google.com/speech-to-text/docs/quickstart-client-libraries
# Imports the Google Cloud client library
from google.cloud import speech
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key.json'
# Instantiates a client
client = speech.SpeechClient()
# The name of the audio file to transcribe
gcs_uri = "gs://cloud-samples-data/speech/brooklyn_bridge.raw"
audio = speech.RecognitionAudio(uri=gcs_uri)
config = speech.RecognitionConfig(
encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
language_code="en-US",
)
# Detects speech in the audio file
response = client.recognize(config=config, audio=audio)
for result in response.results:
print("Transcript: {}".format(result.alternatives[0].transcript))
This results in the following error:
Traceback (most recent call last):
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\grpc\_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\grpc\_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"#1614347442.192000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4090,"referenced_errors":[{"created":"#1614347442.192000000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\retry.py", line 184, in retry_target
return target()
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/98274/Desktop/carla_ppo/test_connection.py", line 22, in <module>
response = client.recognize(config=config, audio=audio)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\cloud\speech_v1\services\speech\client.py", line 334, in recognize
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\gapic_v1\method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "C:\Users\98274\anaconda3\envs\carla\lib\site-packages\google\api_core\retry.py", line 206, in retry_target
last_exc,
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 120.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x0000026B56606A68>, config {
encoding: LINEAR16
sample_rate_hertz: 16000
language_code: "en-US"
}
audio {
uri: "gs://cloud-samples-data/speech/brooklyn_bridge.raw"
}
, metadata=[('x-goog-api-client', 'gl-python/3.7.9 grpc/1.31.0 gax/1.23.0 gapic/2.0.1')]), last exception: 503 failed to connect to all addresses
What's the problem? Does it related to internet issue because I am in China? How can I solve it?
I am posting the answer as CommunityWiki to further contribuite to the community.
As discussed in the comment section, the code provided in the documentation is running smoothly. However, you have to make sure that you set your credentials using GOOGLE_APPLICATION_CREDENTIALS so your request is authenticated. Also,the client libraries have to be properly installed.
In #antaressgzz case's the request to the API was being blocked by a firewall rule. Thus, it is a good practice to check these rules, changing them if applicable.
We have a Glue Python Shell Pipeline Job trying to invoke AWS AppFlow through boto3 APIs. The challenge we have is that we need to design our solution without the need for internet access within Glue
Previously we had challenges in upgrading the boto3 APIs in AWS Glue Shell to the latest version without internet. But then we have found the solution and documented in the following link. In our architecture, we are using AWS Python Shell as our lightweight Datapipeline Engine leveraging boto3 APIs
Git Glue Boto3 Bug & Solution
The following Appflow API python code is working perfectly fine in our local Jupyter Notebooks, as AWS App flow API is invoked over the internet.
##Extra code as per above link to update boto3 version as well, which is omitted
import boto3
print(boto3.__version__)
client=boto3.client('appflow')
response=client.describe_flow(
flowName='sf_non_pii'
)
But while we use the same code in AWS Glue, it is failing as it is not able to access the internet.
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://appflow.us-east-2.amazonaws.com/describe-flow"
Please find the logs below
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 386, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
% (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f5f8802c0f0>, 'Connection to appflow.us-east-2.amazonaws.com timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 123, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-a3cgmro3/ref_appflow.py", line 44, in <module>
File "/tmp/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/tmp/botocore/client.py", line 663, in _make_api_call
operation_model, request_dict, request_context)
File "/tmp/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/tmp/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/tmp/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/tmp/botocore/endpoint.py", line 256, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/tmp/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/tmp/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/tmp/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/tmp/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/tmp/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/tmp/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/tmp/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/tmp/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/tmp/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/tmp/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/tmp/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/tmp/botocore/httpsession.py", line 287, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://appflow.us-east-2.amazonaws.com/describe-flow"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 142, in <module>
raise e_type(e_value).with_traceback(new_stack)
TypeError: __init__() takes 1 positional argument but 2 were given
We had faced similar issues for Redshift, AWS Athena, SSM, etc. But for these services, we have VPC endpoints available. Creating the endpoints enabled our glue job to connect privately within the VPC. Please refer to the link below for supported products
AWS services that you can use with AWS PrivateLink
But since Appflow is a newer product, we do not have endpoint, even though some unclear documentation regarding a throwaway create and delete VPC endpoint as below
Private Amazon AppFlow flows
Please let us know how we can invoke boto3 APIs for Appflow without going through the internet.
Update: I have found that celery is even working fine with other api calls, just not the google api, so it makes me think somehow google api and celery are not compatible or some setting needs to be changed?
I have celery setup fine in that it works for simple tasks like a hello world (calling hello_world.delay() in django views there is no problem).
I created another more complicated function that involves making many calls to a an api to retrieve data to update the database. Just calling this function without celery (update_comments()) there is no problem, and the database gets updated(takes around 15 seconds or so...that is why I want to use celery). Currently the function makes dozens of calls to the api to update the model.
The problem is once I add .delay() to make it a celery task. I get the below errors (maxretry ssl error). I know I have celery setup correctly for basic tasks as it works fine for a basic hello world function, but not this more complicated function with many api calls. I can't figure out why there is this disparity or if there is any setting I am supposed to change because I have an api call. Currently just running on dev.
Tasks
#task()
def update_comments():
for channel in Channel.objects.all():
search_url='https://www.googleapis.com/youtube/v3/commentThreads'
params1={
'part': 'replies,snippet',
'allThreadsRelatedToChannelId': channel.channel,
'searchTerms': '#vi',
'maxResults': 100,
'order': 'time',
'key' : settings.YOUTUBE_API_DATA_KEY
}
params2={
'part': 'replies,snippet',
'allThreadsRelatedToChannelId': channel.channel,
'searchTerms': '#faq',
'maxResults': 100,
'order': 'time',
'key' : settings.YOUTUBE_API_DATA_KEY
}
params3={
'part': 'replies,snippet',
'allThreadsRelatedToChannelId': channel.channel,
'searchTerms': '#save',
'maxResults': 100,
'order': 'time',
'key' : settings.YOUTUBE_API_DATA_KEY
}
r1=requests.get(search_url, params=params1)
r2=requests.get(search_url, params=params2)
r3=requests.get(search_url, params=params3)
rj1=r1.json()
rj2=r2.json()
rj3=r3.json()
a=1
b=1
c=1
for x in range(0, len(rj1['items'])):
if a ==1:
reply_count=rj1['items'][x]['snippet']['totalReplyCount']
for y in range (0, reply_count):
if rj1['items'][x]['replies']['comments'][y]['snippet']['channelId']==channel.channel:
if Comments.objects.filter(question=rj1['items'][x]['snippet']['topLevelComment']['snippet']['textOriginal'], answer=rj1['items'][x]['replies']['comments'][y]['snippet']['textOriginal'], channel=channel, key="#vi").exists():
a=2
break
else:
#get out of for loop
new_comment=Comments(question=rj1['items'][x]['snippet']['topLevelComment']['snippet']['textOriginal'], answer=rj1['items'][x]['replies']['comments'][y]['snippet']['textOriginal'], channel=channel, key="#vi")
new_comment.save()
break
else:
continue
break
else:
break
for x in range(0, len(rj2['items'])):
if b == 1:
reply_count=rj2['items'][x]['snippet']['totalReplyCount']
for y in range (0, reply_count):
if rj2['items'][x]['replies']['comments'][y]['snippet']['channelId']==channel.channel:
if Comments.objects.filter(question=rj2['items'][x]['snippet']['topLevelComment']['snippet']['textOriginal'], answer=rj2['items'][x]['replies']['comments'][y]['snippet']['textOriginal'], channel=channel, key="#faq").exists():
b=2
break
else:
#get out of for loop
new_comment=Comments(question=rj2['items'][x]['snippet']['topLevelComment']['snippet']['textOriginal'], answer=rj2['items'][x]['replies']['comments'][y]['snippet']['textOriginal'], channel=channel, key="#faq")
new_comment.save()
break
else:
continue
break
else:
break
for x in range(0, len(rj3['items'])):
if c == 1:
reply_count=rj3['items'][x]['snippet']['totalReplyCount']
for y in range (0, reply_count):
if rj3['items'][x]['replies']['comments'][y]['snippet']['channelId']==channel.channel:
if Comments.objects.filter(question=rj3['items'][x]['snippet']['topLevelComment']['snippet']['textOriginal'], answer=rj3['items'][x]['replies']['comments'][y]['snippet']['textOriginal'], channel=channel, key="#save").exists():
c=2
break
else:
#get out of for loop
new_comment=Comments(question=rj3['items'][x]['snippet']['topLevelComment']['snippet']['textOriginal'], answer=rj3['items'][x]['replies']['comments'][y]['snippet']['textOriginal'], channel=channel, key="#save")
new_comment.save()
break
else:
continue
break
else:
break
Views
def tes(request):
update_comments.delay()
Settings
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0' BROKER_TRANSPORT =
'redis' CELERY_RESULT_BACKEND = 'django-db'
ERROR
ERROR/ForkPoolWorker-4] Task todo.tasks.update_comments[d368d4eb-d149-4461-a829-e8372cb9a6a5] raised unexpected: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /youtube/v3/commentThreads?part=replies%2Csnippet&allThreadsRelatedToChannelId=UC_EyncGJh2QuQHhcfuDWL6g&searchTerms=%23vi&maxResults=100&order=time&key=xx (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x109380cd0>: Failed to establish a new connection: [Errno 60] Operation timed out'))"))
Traceback (most recent call last):
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connection.py", line 334, in connect
conn = self._new_conn()
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x109380cd0>: Failed to establish a new connection: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /youtube/v3/commentThreads?part=replies%2Csnippet&allThreadsRelatedToChannelId=UC_EyncGJh2QuQHhcfuDWL6g&searchTerms=%23vi&maxResults=100&order=time&key=xx (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x109380cd0>: Failed to establish a new connection: [Errno 60] Operation timed out'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "/Users/xx/Desktop/trydjango/src/todo/tasks.py", line 49, in update_comments
r1=requests.get(search_url, params=params1)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/Users/xx/Desktop/trydjango/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /youtube/v3/commentThreads?part=replies%2Csnippet&allThreadsRelatedToChannelId=UC_EyncGJh2QuQHhcfuDWL6g&searchTerms=%23vi&maxResults=100&order=time&key=xx (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x109380cd0>: Failed to establish a new connection: [Errno 60] Operation timed out'))
My GCP deployment was working fine until I did "gcloud components update" two weeks ago. I understand that the command updates the libraries, and there was an option to revert back to the previous version at the end of the update, but I do not understand why there are so many connection-refused errors with the new update. On the client side, I see "502 Bad Gateway errors". These errors are not consistent. I see them occurring 12 times in the last 48 minutes.
2018-07-31 09:13:19.000 MST
Traceback (most recent call last): File "/env/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection raise err File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
Expand all | Collapse all
{
insertId: "crhycg4civkq14"
labels: {…}
logName: "projects/eps-server/logs/appengine.googleapis.com%2Fstderr"
receiveTimestamp: "2018-07-31T16:13:24.966198879Z"
resource: {…}
textPayload: "Traceback (most recent call last):
File "/env/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
raise err
File "/env/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
"
timestamp: "2018-07-31T16:13:19Z"
}
2018-07-31 09:13:19.000 MST
AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fdd9f316b00>" raised exception!
Expand all | Collapse all
{
insertId: "crhycg4civkq14"
labels: {…}
logName: "projects/eps-server/logs/appengine.googleapis.com%2Fstderr"
receiveTimestamp: "2018-07-31T16:13:24.966198879Z"
resource: {…}
textPayload: "AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fdd9f316b00>" raised exception!
"
timestamp: "2018-07-31T16:13:19Z"
}
I have a django celery task that is only partly executing.
I start up the app and the connection looks good:
INFO/MainProcess] Connected to
redis://elasticache.cache.amazonaws.com:6379/0 [2018-02-17
23:27:24,314: INFO/MainProcess] mingle: searching for neighbors
[2018-02-17 23:27:25,339: INFO/MainProcess] mingle: all alone
[2018-02-17 23:27:25,604: INFO/MainProcess]
worker1#test_vmstracker_com ready.
I initiate the process and the task is received an executed:
[2018-02-17 23:27:49,810: INFO/MainProcess] Received task:
tracking.tasks.escalate[92f54d48202] ETA:[2018-02-18
07:27:59.797380+00:00] [2018-02-17 23:27:49,830: INFO/MainProcess]
Received task: tracking.tasks.escalate[09a0aebef72b] ETA:[2018-02-18
07:28:19.809712+00:00] [2018-02-17 23:28:00,205:
WARNING/ForkPoolWorker-7] -my app is working-
Then I start getting errors and it doesn't finish the task where my app sends an email
[2018-02-17 23:28:00,214: ERROR/ForkPoolWorker-7] Connection to Redis
lost: Retry (0/20) now. [2018-02-17 23:28:00,220:
ERROR/ForkPoolWorker-7] Connection to Redis lost: Retry (1/2
Does anyone know why only have executes and then the connection is lost?
Here is the full stacktrace:
[2018-02-17 23:28:19,382: WARNING/ForkPoolWorker-7] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: ConnectionError("Error while reading from socket: ('Connection closed by server.',)",):
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2879, in execute
return execute(conn, stack, raise_on_error)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
self.parse_response(connection, '_')
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
self, connection, command_name, **options)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
response = self._buffer.readline()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
self._read_from_socket()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 434, in trace_task
uuid, retval, task_request, publish_result,
File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 652, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 213, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 203, in ensure
**retry_policy)
File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 222, in _set
pipe.execute()
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2894, in execute
return execute(conn, stack, raise_on_error)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
self.parse_response(connection, '_')
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
self, connection, command_name, **options)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
response = self._buffer.readline()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
self._read_from_socket()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
exc, exc_info.traceback)))
The problem was that my app server and broker instances were too small. I'm using ec2. As soon as I upgraded to the large hardware, the problem went away. Either the ec2 instance or elasticache instance was too small in terms of CPU, memory, and network.