Celery workers stop working with amazon sqs - django

I am using celery 3.1.9 with sqs. I have worker common_w running as daemon. It works with common queue on sqs.
A worker unexpectedly stop processing tasks. No exceptions and errors.
Last logs with option -l DEBUG:
[2014-09-03 21:01:14,766: DEBUG/MainProcess] Method: GET
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Path: /684818426251/dev_common_w_ip-10-84-163-209-celery-pidbox
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Data:
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Headers: {}
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Host: eu-west-1.queue.amazonaws.com
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Port: 443
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Params: {'Action': 'ReceiveMessage', 'Version': '2012-11-05', 'MaxNumberOfMessages': 10}
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Token: None
[2014-09-03 21:01:14,767: DEBUG/MainProcess] CanonicalRequest:
GET
/684818426251/dev_common_w_ip-10-84-163-209-celery-pidbox
Action=ReceiveMessage&MaxNumberOfMessages=10&Version=2012-11-05
host:eu-west-1.queue.amazonaws.com
x-amz-date:20140903T170114Z
host;x-amz-date
e3b0c44298fc1c149afbf4c899sdfasf32wefwef49b934ca495991b7852b855
[2014-09-03 21:01:14,768: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20140903T170114Z
20140903/eu-west-1/sqs/aws4_request
9a9761b49ba9a06e469bwkfj48u83yghkhejwejlr8fce8eb078ac8c4c9ffd9e
[2014-09-03 21:01:14,768: DEBUG/MainProcess] Signature:
2de3c082bc6f01f5d5ecd66b6r89283ryuu8j8rrdaf0c40eba6cc0ceb62df6e
[2014-09-03 21:01:14,824: DEBUG/MainProcess] <?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult/><ResponseMetadata><RequestId>4712119a-38ca-51b4-bfd1-5d1f8fu8uc4</RequestId></ResponseMetadata></ReceiveMessageResponse>
[2014-09-03 21:01:14,824: INFO/MainProcess] Received task: skazka.sender.tasks.wait_action_worker[f84c52fe-8748-4c81-b718-f23f23fasdgbg34g]
[2014-09-03 21:01:14,824: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x21666e0> (args:('skazka.sender.tasks.wait_action_worker', 'f84c52fe-8748-4c81-b718-f23f23fasdgbg34g', (1967L,), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (1967L,), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'common', u'exchange': u'common'}, 'expires': None, u'hostname': 'common_w#ip-10-84-163-209', 'task': 'skazka.sender.tasks.wait_action_worker', 'callbacks': None, u'correlation_id': u'f84c52fe-8748-4c81-b718-f23f23fasdgbg34g', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'07a91182-23a6-3afb-b3cc-70a2fa3fw333', 'id': 'f84c52fe-8748-4c81-b718-f23f23fasdgbg34g', u'headers': {}}) kwargs:{})
UPDATE:
strace says:
futex(0x7ffff79b3e00, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
enter code here
Then I set in the config:
CELERYD_FORCE_EXECV = True
Until it works fine...

Related

why cant i see prints and logs from lambda in cloudwatch , the function is invoked repeatedly?

lambda backend function is invoked every 20 sec from s3 static website - GET request goes via api gateway calls lambda, it tries to get item from dynamodb table and should return it. It returns 500 error- smth wrong on server side.
Enough time(9sec) is given for the lambda to finish executing code - lambda's own time out should not be an issue.
Lambda has a role attached that allows get_item from dynamodb table - IAM is not an issue.
However, when troubleshooting i only see logs in cloudwatch until it tries to get_item. I put lots of print and logs, but it does not go past that line, I even nested try..except - it does not catch any errors. I dont see how to detect whats' wrong. I set logging level to debug - it prints some stuff.
import logging
import boto3
import sys
logging.getLogger().setLevel(logging.DEBUG)
def lambda_handler(event, context):
logging.info('doing retrieving from table votes')
try:
logging.info('********************* TRYING retrieving from table votes')
# dynamodb = boto3.client('dynamodb')
table = boto3.resource('dynamodb', region_name='us-east-1').Table('Votes')
print(table)
logging.info('event')
logging.info(event)
print(type(table))
logging.info(table)
logging.info(type(table))
# logging.error(table)
# logging.error(type(table))
#************************
# I dont see the result of count nowhere in the logs of cloud watch
try:
count = table.get_item(Key={'voter':{'S': 'count'}})
except Exception as e:
logging.info('catching it here - if you see it then something wrong with get_item count')
logging.info('********************BAD******************')
logging.error('********************BAD******************')
e = sys.exc_info()[0]
exception_type = e.__class__.__name__
exception_message = str(e)
logging.error('--------------------------------')
logging.error(exception_message)
logging.error(exception_type)
################## BELOW DONT GET PRINTED AT ALL in cloud watch logs
logging.info('count')
logging.info(count)
print('****************COUNT*********************')
print(count)
print('----------------------------------------------------')
logging.info(count)
a = count["Item"]["a"]
b = count["Item"]["b"]
logging.info('count [Item]')
logging.info(count["Item"])
logging.info('------------------------------------')
logging.info('ok retrieve from table votes')
logging.info('a is ' + a)
logging.info('b is ' + b)
logging.info('************************************ success! a: ' + a + ' and b: ' + b)
return {'statusCode': 200, 'body': '{"a": ' + a + ', "b": ' + b + '}'}
except Exception as e:
logging.info('********************BAD******************')
e = sys.exc_info()[0]
exception_type = e.__class__.__name__
exception_message = str(e)
logging.error('--------------------------------')
logging.error(exception_message)
logging.error(exception_type)
logging.error('---------------------------------------------')
return {'statusCode': 500, 'body': '{"status": "error getting from table votes"}'}
the full logs of 1 request id from start to finish. For some reason it never prints the result of "count = get_item)
START RequestId: 55a5e428-c8d6-4914-908c-20ccec1153dd Version: $LATEST
2023-01-11T21:29:08.805+05:00
[INFO] 2023-01-11T16:29:08.805Z 55a5e428-c8d6-4914-908c-20ccec1153dd doing retrieving from table votes
2023-01-11T21:29:08.805+05:00
[INFO] 2023-01-11T16:29:08.805Z 55a5e428-c8d6-4914-908c-20ccec1153dd ********************* TRYING retrieving from table votes
2023-01-11T21:29:08.806+05:00
[DEBUG] 2023-01-11T16:29:08.806Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-01-11T21:29:08.824+05:00
[DEBUG] 2023-01-11T16:29:08.824Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-call.apigateway to before-call.api-gateway
2023-01-11T21:29:08.825+05:00
[DEBUG] 2023-01-11T16:29:08.825Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-01-11T21:29:08.826+05:00
[DEBUG] 2023-01-11T16:29:08.826Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-01-11T21:29:08.827+05:00
[DEBUG] 2023-01-11T16:29:08.827Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-01-11T21:29:08.827+05:00
[DEBUG] 2023-01-11T16:29:08.827Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-01-11T21:29:08.884+05:00
[DEBUG] 2023-01-11T16:29:08.884Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.924Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.925Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.925Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.925Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2023-01-11T21:29:09.044+05:00
[DEBUG] 2023-01-11T16:29:09.044Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/boto3/data/dynamodb/2012-08-10/resources-1.json
2023-01-11T21:29:09.047+05:00
[DEBUG] 2023-01-11T16:29:09.047Z 55a5e428-c8d6-4914-908c-20ccec1153dd IMDS ENDPOINT: http://169.254.169.254/
2023-01-11T21:29:09.105+05:00
[DEBUG] 2023-01-11T16:29:09.105Z 55a5e428-c8d6-4914-908c-20ccec1153dd Looking for credentials via: env
2023-01-11T21:29:09.105+05:00
[INFO] 2023-01-11T16:29:09.105Z 55a5e428-c8d6-4914-908c-20ccec1153dd Found credentials in environment variables.
2023-01-11T21:29:09.106+05:00
[DEBUG] 2023-01-11T16:29:09.106Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/botocore/data/endpoints.json
2023-01-11T21:29:09.266+05:00
[DEBUG] 2023-01-11T16:29:09.266Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa39bdec160>
2023-01-11T21:29:09.365+05:00
[DEBUG] 2023-01-11T16:29:09.364Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/botocore/data/dynamodb/2012-08-10/service-2.json
2023-01-11T21:29:09.427+05:00
[DEBUG] 2023-01-11T16:29:09.426Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-client-class.dynamodb: calling handler <function add_generate_presigned_url at 0x7fa39be91c10>
2023-01-11T21:29:09.485+05:00
[DEBUG] 2023-01-11T16:29:09.484Z 55a5e428-c8d6-4914-908c-20ccec1153dd Setting dynamodb timeout as (60, 60)
2023-01-11T21:29:09.485+05:00
[DEBUG] 2023-01-11T16:29:09.485Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/botocore/data/_retry.json
2023-01-11T21:29:09.486+05:00
[DEBUG] 2023-01-11T16:29:09.486Z 55a5e428-c8d6-4914-908c-20ccec1153dd Registering retry handlers for service: dynamodb
2023-01-11T21:29:09.487+05:00
[DEBUG] 2023-01-11T16:29:09.486Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading dynamodb:dynamodb
2023-01-11T21:29:09.487+05:00
[DEBUG] 2023-01-11T16:29:09.487Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-resource-class.dynamodb.ServiceResource: calling handler <function lazy_call.<locals>._handler at 0x7fa39bd7aca0>
2023-01-11T21:29:09.546+05:00
[DEBUG] 2023-01-11T16:29:09.546Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading dynamodb:Table
2023-01-11T21:29:09.547+05:00
[DEBUG] 2023-01-11T16:29:09.547Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-resource-class.dynamodb.Table: calling handler <function lazy_call.<locals>._handler at 0x7fa39bd7ad30>
2023-01-11T21:29:09.584+05:00
[DEBUG] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-resource-class.dynamodb.Table: calling handler <function lazy_call.<locals>._handler at 0x7fa39bd7aca0>
2023-01-11T21:29:09.584+05:00
dynamodb.Table(name='Votes')
2023-01-11T21:29:09.584+05:00
[INFO] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd event
2023-01-11T21:29:09.584+05:00
[INFO] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd {'version': '2.0', 'routeKey': 'GET /results', 'rawPath': '/results', 'rawQueryString': '', 'headers': {'accept': 'application/json','user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36', 'x-amzn-trace-id': 'Root=1-63bee3d4-679c4d201abf991d1f331f33', 'x-forwarded-for': '164.40.37.179', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https'}, 'requestContext': {'accountId': '025416187662', 'apiId': '5y7dfynd34', 'domainName': '5y7dfynd34.execute-api.us-east-1.amazonaws.com', 'domainPrefix': '5y7dfynd34', 'http': {'method': 'GET', 'path': '/results', 'protocol': 'HTTP/1.1', 'sourceIp': '164.40.37.179', 'userAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'}, 'requestId': 'eliJRin9oAMEc9Q=', 'routeKey': 'GET /results', 'stage': '$default', 'time': '11/Jan/2023:16:29:08 +0000', 'timeEpoch': 1673454548760}, 'isBase64Encoded': False}
2023-01-11T21:29:09.584+05:00
<class 'boto3.resources.factory.dynamodb.Table'>
2023-01-11T21:29:09.585+05:00
[INFO] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd dynamodb.Table(name='Votes')
2023-01-11T21:29:09.585+05:00
[INFO] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd <class 'boto3.resources.factory.dynamodb.Table'>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Calling dynamodb:get_item with {'TableName': 'Votes', 'Key': {'voter': {'S': 'count'}}}
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event provide-client-params.dynamodb.GetItem: calling handler <function _dynamodb_params at 0x7fa39b870ca0>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <bound method TransformationInjector.inject_condition_expressions of <boto3.dynamodb.transform.TransformationInjector object at 0x7fa39b852730>>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <bound method TransformationInjector.inject_attribute_value_input of <boto3.dynamodb.transform.TransformationInjector object at 0x7fa39b852730>>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <function generate_idempotent_uuid at 0x7fa39be0d3a0>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <function block_endpoint_discovery_required_operations at 0x7fa39be32d30>
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-call.dynamodb.GetItem: calling handler <function inject_api_version_header_if_needed at 0x7fa39be11c10>
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Making request for OperationModel(name=GetItem) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'DynamoDB_20120810.GetItem', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.20.32 Python/3.9.13 Linux/4.14.255-296-236.539.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.23.32 Resource'}, 'body': b'{"TableName": "Votes", "Key": {"voter": {"M": {"S": {"S": "count"}}}}}', 'url': 'https://dynamodb.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa39b897c40>, 'has_streaming_input': False, 'auth_type': None}}
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event request-created.dynamodb.GetItem: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa39b897a90>>
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event choose-signer.dynamodb.GetItem: calling handler <function set_operation_specific_signer at 0x7fa39be0d280>
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd Calculating signature using v4 auth.
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.0
host:dynamodb.us-east-1.amazonaws.com
x-amz-date:20230111T162909Z
x-amz-security- x-amz-target:DynamoDB_20120810.GetItem
content-type;host;x-amz-date;x-amz-security-token;x-amz-target
6dd016d6033694be300988a73dded6cba15ade0cf920e8bafb56369e3719c397
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd CanonicalRequest: POST / content-type:application/x-amz-json-1.0 host:dynamodb.us-east-1.amazonaws.com x-amz-date:20230111T162909Z x-amz-target:DynamoDB_20120810.GetItem content-type;host;x-amz-date;x-amz-security-token;x-amz-target 6dd016d6033694be300988a73dded6cba15ade0cf920e8bafb56369e3719c397
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd StringToSign:
AWS4-HMAC-SHA256
20230111T162909Z
20230111/us-east-1/dynamodb/aws4_request
33bbba9cdeb906cc5b3ddc600b02d47f0a73e019d5f3efa0627ea82e05e86eee
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd StringToSign: AWS4-HMAC-SHA256 20230111T162909Z 20230111/us-east-1/dynamodb/aws4_request 33bbba9cdeb906cc5b3ddc600b02d47f0a73e019d5f3efa0627ea82e05e86eee
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd Signature: f36e8c5a9c7d47f1ef41c1ecce566a988f7243f8f95bf7f7c43b951a87e488eb
2023-01-11T21:29:09.644+05:00
[DEBUG] 2023-01-11T16:29:09.643Z 55a5e428-c8d6-4914-908c-20ccec1153dd Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://dynamodb.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'DynamoDB_20120810.GetItem', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.20.32 Python/3.9.13 Linux/4.14.255-296-236.539.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.23.32 Resource', 'X-Amz-Date': b'20230111T162909Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAQL2XMH4HN6PEJXPA/20230111/us-east-1/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=f36e8c5a9c7d47f1ef41c1ecce566a988f7243f8f95bf7f7c43b951a87e488eb', 'Content-Length': '70'}>
[DEBUG] 2023-01-11T16:29:09.644Z 55a5e428-c8d6-4914-908c-20ccec1153dd Certificate path: /var/runtime/botocore/cacert.pem
2023-01-11T21:29:09.644+05:00
[DEBUG] 2023-01-11T16:29:09.644Z 55a5e428-c8d6-4914-908c-20ccec1153dd Starting new HTTPS connection (1): dynamodb.us-east-1.amazonaws.com:443
2023-01-11T21:29:17.816+05:00
2023-01-11T16:29:17.815Z 55a5e428-c8d6-4914-908c-20ccec1153dd Task timed out after 9.01 seconds
2023-01-11T21:29:17.816+05:00
END RequestId: 55a5e428-c8d6-4914-908c-20ccec1153dd

It looks like the issue is that the table.get_item(Key={'voter':{'S': 'count'}}) call is raising an exception, but your exception handling code is not catching it. One possible cause for this could be that the table.get_item method is returning a botocore.exception.ClientError object, rather than a Python Exception object. You can modify your except block to catch botocore.exception.ClientError and add some log information like request_id and error message in the catch block.
Another way to detect the error is to enable enhanced monitoring for your lambda function, this will give more detailed metrics on the function performance and error.
One thing you could try is to use sys.exc_info() to get the current exception inside the catch block, and print out the exception's class and message. That way you can see exactly what type of exception is being raised and what the error message is.
You could also try to call get_item separately in a local environment to see if it raises an exception, this will help you to figure out if the issue is with your code or the IAM policy or resource configurations.

My lambda function results in a timeout when I call multiple AWS services using it for multiple times

I am moving my Flask API(s) to Lambda + API Gateway. It requires me to connect with Cognito and RDS in the same lambda function. It works well a couple of times. I provide email and password as request body. Lambda takes these credentials, logs me in to Cognito using boto3 library and returns access token. If authentication is successful, I proceed to get the extra user details using the same email. I get all of them, package them in a dictionary and return from Lambda via API gateway.
This works well for 5-10 times (varies everytime it works) and then stops working (starts to timeout) for an unexplainable amount of time (hours to a day) and then starts to work again for the same code. I do not know why this is happening. I can share more info if you want. But if any of you has experienced throttling(??) of this kind which doesn't really show up as throttled lambda reports but rather lambda stopping to work with other AWS services.
I have tried checking whether all lambda operations cease to stop or not and I found that they don't. Only the ones that involve talking to other aws services with boto3. The weird thing is that everything works seamlessly when boto3 is called from inside an EC2. It timesout after a few successful calls from inside a lambda.
Logs:
4th last line takes all the time (establishing connection). I have tried with both admin_initiate_auth and initiate_auth. Both fail in similar fashion (continuous timeout after a few successful attempts)
2022-01-12T00:50:54.991+09:00 2022-01-11 15:50:54,990 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2022-01-12T00:50:54.991+09:00 [DEBUG] 2022-01-11T15:50:54.991Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2022-01-12T00:50:54.993+09:00 2022-01-11 15:50:54,991 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2022-01-12T00:50:54.993+09:00 [DEBUG] 2022-01-11T15:50:54.993Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2022-01-12T00:50:54.994+09:00 2022-01-11 15:50:54,993 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2022-01-12T00:50:55.011+09:00 [DEBUG] 2022-01-11T15:50:54.994Z a809239e-e99a-47e1-8e72-eb27645a0798 Looking for credentials via: env
2022-01-12T00:50:55.011+09:00 2022-01-11 15:50:54,994 botocore.credentials [DEBUG] Looking for credentials via: env
2022-01-12T00:50:55.011+09:00 [DEBUG] 2022-01-11T15:50:55.11Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2022-01-12T00:50:55.011+09:00 2022-01-11 15:50:55,011 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2022-01-12T00:50:55.014+09:00 [INFO] 2022-01-11T15:50:55.11Z a809239e-e99a-47e1-8e72-eb27645a0798 Found credentials in environment variables.
2022-01-12T00:50:55.014+09:00 2022-01-11 15:50:55,011 botocore.credentials [INFO] Found credentials in environment variables.
2022-01-12T00:50:55.014+09:00 [DEBUG] 2022-01-11T15:50:55.14Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2022-01-12T00:50:55.014+09:00 2022-01-11 15:50:55,014 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2022-01-12T00:50:55.015+09:00 [DEBUG] 2022-01-11T15:50:55.15Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2022-01-12T00:50:55.015+09:00 2022-01-11 15:50:55,015 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2022-01-12T00:50:55.015+09:00 [DEBUG] 2022-01-11T15:50:55.15Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2022-01-12T00:50:55.015+09:00 2022-01-11 15:50:55,015 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2022-01-12T00:50:55.015+09:00 2022-01-11 15:50:55,015 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2022-01-12T00:50:55.015+09:00 [DEBUG] 2022-01-11T15:50:55.15Z a809239e-e99a-47e1-8e72-eb27645a0798 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2022-01-12T00:50:55.030+09:00 [DEBUG] 2022-01-11T15:50:55.30Z a809239e-e99a-47e1-8e72-eb27645a0798 Loading JSON file: /var/runtime/botocore/data/endpoints.json
2022-01-12T00:50:55.030+09:00 2022-01-11 15:50:55,030 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/endpoints.json
2022-01-12T00:50:55.056+09:00 [DEBUG] 2022-01-11T15:50:55.56Z a809239e-e99a-47e1-8e72-eb27645a0798 Loading JSON file: /var/runtime/botocore/data/endpoints.json
2022-01-12T00:50:55.070+09:00 2022-01-11 15:50:55,056 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/endpoints.json
2022-01-12T00:50:55.097+09:00 [DEBUG] 2022-01-11T15:50:55.96Z a809239e-e99a-47e1-8e72-eb27645a0798 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f955cea00d0>
2022-01-12T00:50:55.110+09:00 2022-01-11 15:50:55,096 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f955cea00d0>
2022-01-12T00:50:55.110+09:00 [DEBUG] 2022-01-11T15:50:55.110Z a809239e-e99a-47e1-8e72-eb27645a0798 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f955cea00d0>
2022-01-12T00:50:55.110+09:00 2022-01-11 15:50:55,110 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f955cea00d0>
2022-01-12T00:50:55.315+09:00 [DEBUG] 2022-01-11T15:50:55.315Z a809239e-e99a-47e1-8e72-eb27645a0798 Loading JSON file: /var/runtime/botocore/data/codeguruprofiler/2019-07-18/service-2.json
2022-01-12T00:50:55.315+09:00 2022-01-11 15:50:55,315 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/codeguruprofiler/2019-07-18/service-2.json
2022-01-12T00:50:55.330+09:00 [DEBUG] 2022-01-11T15:50:55.330Z a809239e-e99a-47e1-8e72-eb27645a0798 Event creating-client-class.codeguruprofiler: calling handler <function add_generate_presigned_url at 0x7f955cf3ec10>
2022-01-12T00:50:55.332+09:00 2022-01-11 15:50:55,330 botocore.hooks [DEBUG] Event creating-client-class.codeguruprofiler: calling handler <function add_generate_presigned_url at 0x7f955cf3ec10>
2022-01-12T00:50:55.332+09:00 [DEBUG] 2022-01-11T15:50:55.331Z a809239e-e99a-47e1-8e72-eb27645a0798 Loading JSON file: /var/runtime/botocore/data/cognito-idp/2016-04-18/service-2.json
2022-01-12T00:50:55.333+09:00 [DEBUG] 2022-01-11T15:50:55.332Z a809239e-e99a-47e1-8e72-eb27645a0798 Creating a regex based endpoint for codeguru-profiler, ap-northeast-1
2022-01-12T00:50:55.333+09:00 2022-01-11 15:50:55,331 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/cognito-idp/2016-04-18/service-2.json
2022-01-12T00:50:55.357+09:00 2022-01-11 15:50:55,332 botocore.regions [DEBUG] Creating a regex based endpoint for codeguru-profiler, ap-northeast-1
2022-01-12T00:50:55.371+09:00 [DEBUG] 2022-01-11T15:50:55.371Z a809239e-e99a-47e1-8e72-eb27645a0798 Setting codeguru-profiler timeout as (60, 60)
2022-01-12T00:50:55.372+09:00 2022-01-11 15:50:55,371 botocore.endpoint [DEBUG] Setting codeguru-profiler timeout as (60, 60)
2022-01-12T00:50:55.391+09:00 [DEBUG] 2022-01-11T15:50:55.391Z a809239e-e99a-47e1-8e72-eb27645a0798 Event before-parameter-build.codeguruprofiler.ConfigureAgent: calling handler <function generate_idempotent_uuid at 0x7f955cead550>
2022-01-12T00:50:55.392+09:00 2022-01-11 15:50:55,391 botocore.hooks [DEBUG] Event before-parameter-build.codeguruprofiler.ConfigureAgent: calling handler <function generate_idempotent_uuid at 0x7f955cead550>
2022-01-12T00:50:55.393+09:00 [DEBUG] 2022-01-11T15:50:55.392Z a809239e-e99a-47e1-8e72-eb27645a0798 Event creating-client-class.cognito-identity-provider: calling handler <function add_generate_presigned_url at 0x7f955cf3ec10>
2022-01-12T00:50:55.393+09:00 2022-01-11 15:50:55,392 botocore.hooks [DEBUG] Event creating-client-class.cognito-identity-provider: calling handler <function add_generate_presigned_url at 0x7f955cf3ec10>
2022-01-12T00:50:55.410+09:00 [DEBUG] 2022-01-11T15:50:55.393Z a809239e-e99a-47e1-8e72-eb27645a0798 Event before-call.codeguruprofiler.ConfigureAgent: calling handler <function inject_api_version_header_if_needed at 0x7f955ceb3dc0>
2022-01-12T00:50:55.411+09:00 2022-01-11 15:50:55,393 botocore.hooks [DEBUG] Event before-call.codeguruprofiler.ConfigureAgent: calling handler <function inject_api_version_header_if_needed at 0x7f955ceb3dc0>
2022-01-12T00:50:55.411+09:00 [DEBUG] 2022-01-11T15:50:55.411Z a809239e-e99a-47e1-8e72-eb27645a0798 Setting cognito-idp timeout as (60, 60)
2022-01-12T00:50:55.411+09:00 2022-01-11 15:50:55,411 botocore.endpoint [DEBUG] Setting cognito-idp timeout as (60, 60)
2022-01-12T00:50:55.412+09:00 [DEBUG] 2022-01-11T15:50:55.411Z a809239e-e99a-47e1-8e72-eb27645a0798 Making request for OperationModel(name=ConfigureAgent) with params: {'url_path': '/profilingGroups/aws-lambda-iws_authenticate/configureAgent', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55'}, 'body': b'{"fleetInstanceId": "arn:aws:lambda:ap-northeast-1:836281999550:function:iws_authenticate", "metadata": {"ComputePlatform": "AWSLambda", "LambdaFunctionArn": "arn:aws:lambda:ap-northeast-1:836281999550:function:iws_authenticate", "AgentId": "458edeb0-c867-48b5-9e62-551272da93b0", "LambdaMemoryLimitInMB": "512", "ExecutionEnvironment": "AWS_Lambda_python3.9", "AwsRequestId": "a809239e-e99a-47e1-8e72-eb27645a0798", "LambdaRemainingTimeInMilliseconds": "4500"}}', 'url': 'https://codeguru-profiler.ap-northeast-1.amazonaws.com/profilingGroups/aws-lambda-iws_authenticate/configureAgent', 'context': {'client_region': 'ap-northeast-1', 'client_config': <botocore.config.Config object at 0x7f955c9f7fd0>, 'has_streaming_input': False, 'auth_type': None}}
2022-01-12T00:50:55.412+09:00 2022-01-11 15:50:55,411 botocore.endpoint [DEBUG] Making request for OperationModel(name=ConfigureAgent) with params: {'url_path': '/profilingGroups/aws-lambda-iws_authenticate/configureAgent', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55'}, 'body': b'{"fleetInstanceId": "arn:aws:lambda:ap-northeast-1:836281999550:function:iws_authenticate", "metadata": {"ComputePlatform": "AWSLambda", "LambdaFunctionArn": "arn:aws:lambda:ap-northeast-1:836281999550:function:iws_authenticate", "AgentId": "458edeb0-c867-48b5-9e62-551272da93b0", "LambdaMemoryLimitInMB": "512", "ExecutionEnvironment": "AWS_Lambda_python3.9", "AwsRequestId": "a809239e-e99a-47e1-8e72-eb27645a0798", "LambdaRemainingTimeInMilliseconds": "4500"}}', 'url': 'https://codeguru-profiler.ap-northeast-1.amazonaws.com/profilingGroups/aws-lambda-iws_authenticate/configureAgent', 'context': {'client_region': 'ap-northeast-1', 'client_config': <botocore.config.Config object at 0x7f955c9f7fd0>, 'has_streaming_input': False, 'auth_type': None}}
2022-01-12T00:50:55.412+09:00 [DEBUG] 2022-01-11T15:50:55.412Z a809239e-e99a-47e1-8e72-eb27645a0798 Event request-created.codeguruprofiler.ConfigureAgent: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f955c9f7fa0>>
2022-01-12T00:50:55.413+09:00 2022-01-11 15:50:55,412 botocore.hooks [DEBUG] Event request-created.codeguruprofiler.ConfigureAgent: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f955c9f7fa0>>
2022-01-12T00:50:55.413+09:00 [DEBUG] 2022-01-11T15:50:55.413Z a809239e-e99a-47e1-8e72-eb27645a0798 Event choose-signer.codeguruprofiler.ConfigureAgent: calling handler <function set_operation_specific_signer at 0x7f955cead430>
2022-01-12T00:50:55.413+09:00 2022-01-11 15:50:55,413 botocore.hooks [DEBUG] Event choose-signer.codeguruprofiler.ConfigureAgent: calling handler <function set_operation_specific_signer at 0x7f955cead430>
2022-01-12T00:50:55.413+09:00 [DEBUG] 2022-01-11T15:50:55.413Z a809239e-e99a-47e1-8e72-eb27645a0798 Loading JSON file: /var/runtime/botocore/data/_retry.json
2022-01-12T00:50:55.414+09:00 2022-01-11 15:50:55,413 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/_retry.json
2022-01-12T00:50:55.414+09:00 [DEBUG] 2022-01-11T15:50:55.414Z a809239e-e99a-47e1-8e72-eb27645a0798 Calculating signature using v4 auth.
2022-01-12T00:50:55.415+09:00 2022-01-11 15:50:55,414 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-01-12T00:50:55.415+09:00 [DEBUG] 2022-01-11T15:50:55.414Z a809239e-e99a-47e1-8e72-eb27645a0798 Registering retry handlers for service: cognito-idp
2022-01-12T00:50:55.415+09:00 2022-01-11 15:50:55,414 botocore.client [DEBUG] Registering retry handlers for service: cognito-idp
2022-01-12T00:50:55.415+09:00 [DEBUG] 2022-01-11T15:50:55.415Z a809239e-e99a-47e1-8e72-eb27645a0798 CanonicalRequest: POST /profilingGroups/aws-lambda-iws_authenticate/configureAgent host:codeguru-profiler.ap-northeast-1.amazonaws.com x-amz-date:20220111T155055Z x-amz-security-token:IQoJb3JpZ2luX2VjEBgaDmFwLW5vcnRoZWFzdC0xIkgwRgIhAOTuf6AUHEuBTrhZpkvV90wWbm5kmKc3aEKlD2fYxl4xAiEAzzlmQcYnW2dwcJ+Xyc+sEAv9eTyE4Y/J3y6RSxmlIs0qlQIIMRABGgw4MzYyODE5OTk1NTAiDLC/IUPGIzyq3dt51yryATMQDjN+w/W7dcykQk9P6Ky0WT6h6cRXTGjsBB+CIv69H6dPDJ/0UESiiup73b4HvZiTmTk7Yt+q3ILS58VrkWpvargPXoIlER3uLuVt7aFHJCmkmp6jHKhFt/yamgSs8K2w7O14jexFB0FBG2+UGYCaS5PiqmuSdu9uzoGPv47uYNdV45I7FV4bBXlALqSn6X5cYwBw6kvCBFarXo6fDGCIroPGtQoJm7I7PYhUvwLuMicggwNcK+dd+CGDjuOHYzm/EFsoSffEjcABaPBo2VTZGITsMenZ9qgo9ej2+Wz27sHHKuuTFYNYe4XMSDNEcOswMN7O9o4GOpkBAfJaBdyNLS2hnjegVlrtQcPSBEMISziKhnRgnrlZSwMtKL6SBTWBwQNDsbxIybXHWuIkssoJH1rbq53Rp6Q/6h+5jgRUpULt4DGFqrH7a36wFEXR//FiEIg6qhIzmemoOpFEeYKTQx10imqLMLDk2nfv3n/3FwJLJr10/nempPNj3T79mmK5Y8FlT8sebkqREcLLsUuXYua6 host;x-amz-date;x-amz-security-token 6597adf75fd24a25ec215f3148f1e86f3e04dfd2c2a3cb64fa57d24b7fe70192
2022-01-12T00:50:55.416+09:00 2022-01-11 15:50:55,415 botocore.auth [DEBUG] CanonicalRequest:
2022-01-12T00:50:55.416+09:00 POST
2022-01-12T00:50:55.416+09:00 /profilingGroups/aws-lambda-iws_authenticate/configureAgent
2022-01-12T00:50:55.416+09:00 host:codeguru-profiler.ap-northeast-1.amazonaws.com
2022-01-12T00:50:55.416+09:00 x-amz-date:20220111T155055Z
2022-01-12T00:50:55.416+09:00 x-amz-security-token:IQoJb3JpZ2luX2VjEBgaDmFwLW5vcnRoZWFzdC0xIkgwRgIhAOTuf6AUHEuBTrhZpkvV90wWbm5kmKc3aEKlD2fYxl4xAiEAzzlmQcYnW2dwcJ+Xyc+sEAv9eTyE4Y/J3y6RSxmlIs0qlQIIMRABGgw4MzYyODE5OTk1NTAiDLC/IUPGIzyq3dt51yryATMQDjN+w/W7dcykQk9P6Ky0WT6h6cRXTGjsBB+CIv69H6dPDJ/0UESiiup73b4HvZiTmTk7Yt+q3ILS58VrkWpvargPXoIlER3uLuVt7aFHJCmkmp6jHKhFt/yamgSs8K2w7O14jexFB0FBG2+UGYCaS5PiqmuSdu9uzoGPv47uYNdV45I7FV4bBXlALqSn6X5cYwBw6kvCBFarXo6fDGCIroPGtQoJm7I7PYhUvwLuMicggwNcK+dd+CGDjuOHYzm/EFsoSffEjcABaPBo2VTZGITsMenZ9qgo9ej2+Wz27sHHKuuTFYNYe4XMSDNEcOswMN7O9o4GOpkBAfJaBdyNLS2hnjegVlrtQcPSBEMISziKhnRgnrlZSwMtKL6SBTWBwQNDsbxIybXHWuIkssoJH1rbq53Rp6Q/6h+5jgRUpULt4DGFqrH7a36wFEXR//FiEIg6qhIzmemoOpFEeYKTQx10imqLMLDk2nfv3n/3FwJLJr10/nempPNj3T79mmK5Y8FlT8sebkqREcLLsUuXYua6
2022-01-12T00:50:55.416+09:00 host;x-amz-date;x-amz-security-token
2022-01-12T00:50:55.416+09:00 6597adf75fd24a25ec215f3148f1e86f3e04dfd2c2a3cb64fa57d24b7fe70192
2022-01-12T00:50:55.416+09:00 [DEBUG] 2022-01-11T15:50:55.416Z a809239e-e99a-47e1-8e72-eb27645a0798 StringToSign: AWS4-HMAC-SHA256 20220111T155055Z 20220111/ap-northeast-1/codeguru-profiler/aws4_request a6d46b3081b0c8cc0c9922f6871b8158b81cb79db2cd29d39138e844858bc0a1
2022-01-12T00:50:55.416+09:00 2022-01-11 15:50:55,416 botocore.auth [DEBUG] StringToSign:
2022-01-12T00:50:55.416+09:00 AWS4-HMAC-SHA256
2022-01-12T00:50:55.416+09:00 20220111T155055Z
2022-01-12T00:50:55.416+09:00 20220111/ap-northeast-1/codeguru-profiler/aws4_request
2022-01-12T00:50:55.416+09:00 a6d46b3081b0c8cc0c9922f6871b8158b81cb79db2cd29d39138e844858bc0a1
2022-01-12T00:50:55.416+09:00 [DEBUG] 2022-01-11T15:50:55.416Z a809239e-e99a-47e1-8e72-eb27645a0798 Signature: 1f2b56c6850d65dd451cc554ce801f2460994150aa8ef89b2846876c3e8abf10
2022-01-12T00:50:55.430+09:00 2022-01-11 15:50:55,416 botocore.auth [DEBUG] Signature:
2022-01-12T00:50:55.430+09:00 1f2b56c6850d65dd451cc554ce801f2460994150aa8ef89b2846876c3e8abf10
2022-01-12T00:50:55.430+09:00 [DEBUG] 2022-01-11T15:50:55.416Z a809239e-e99a-47e1-8e72-eb27645a0798 Event before-parameter-build.cognito-identity-provider.InitiateAuth: calling handler <function generate_idempotent_uuid at 0x7f955cead550>
2022-01-12T00:50:55.430+09:00 2022-01-11 15:50:55,416 botocore.hooks [DEBUG] Event before-parameter-build.cognito-identity-provider.InitiateAuth: calling handler <function generate_idempotent_uuid at 0x7f955cead550>
2022-01-12T00:50:55.431+09:00 [DEBUG] 2022-01-11T15:50:55.430Z a809239e-e99a-47e1-8e72-eb27645a0798 Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://codeguru-profiler.ap-northeast-1.amazonaws.com/profilingGroups/aws-lambda-iws_authenticate/configureAgent, headers={'User-Agent': b'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55', 'X-Amz-Date': b'20220111T155055Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEBgaDmFwLW5vcnRoZWFzdC0xIkgwRgIhAOTuf6AUHEuBTrhZpkvV90wWbm5kmKc3aEKlD2fYxl4xAiEAzzlmQcYnW2dwcJ+Xyc+sEAv9eTyE4Y/J3y6RSxmlIs0qlQIIMRABGgw4MzYyODE5OTk1NTAiDLC/IUPGIzyq3dt51yryATMQDjN+w/W7dcykQk9P6Ky0WT6h6cRXTGjsBB+CIv69H6dPDJ/0UESiiup73b4HvZiTmTk7Yt+q3ILS58VrkWpvargPXoIlER3uLuVt7aFHJCmkmp6jHKhFt/yamgSs8K2w7O14jexFB0FBG2+UGYCaS5PiqmuSdu9uzoGPv47uYNdV45I7FV4bBXlALqSn6X5cYwBw6kvCBFarXo6fDGCIroPGtQoJm7I7PYhUvwLuMicggwNcK+dd+CGDjuOHYzm/EFsoSffEjcABaPBo2VTZGITsMenZ9qgo9ej2+Wz27sHHKuuTFYNYe4XMSDNEcOswMN7O9o4GOpkBAfJaBdyNLS2hnjegVlrtQcPSBEMISziKhnRgnrlZSwMtKL6SBTWBwQNDsbxIybXHWuIkssoJH1rbq53Rp6Q/6h+5jgRUpULt4DGFqrH7a36wFEXR//FiEIg6qhIzmemoOpFEeYKTQx10imqLMLDk2nfv3n/3FwJLJr10/nempPNj3T79mmK5Y8FlT8sebkqREcLLsUuXYua6', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA4FNSKUC7MXWAR2GV/20220111/ap-northeast-1/codeguru-profiler/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=1f2b56c6850d65dd451cc554ce801f2460994150aa8ef89b2846876c3e8abf10', 'Content-Length': '462'}>
2022-01-12T00:50:55.431+09:00 2022-01-11 15:50:55,430 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://codeguru-profiler.ap-northeast-1.amazonaws.com/profilingGroups/aws-lambda-iws_authenticate/configureAgent, headers={'User-Agent': b'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55', 'X-Amz-Date': b'20220111T155055Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEBgaDmFwLW5vcnRoZWFzdC0xIkgwRgIhAOTuf6AUHEuBTrhZpkvV90wWbm5kmKc3aEKlD2fYxl4xAiEAzzlmQcYnW2dwcJ+Xyc+sEAv9eTyE4Y/J3y6RSxmlIs0qlQIIMRABGgw4MzYyODE5OTk1NTAiDLC/IUPGIzyq3dt51yryATMQDjN+w/W7dcykQk9P6Ky0WT6h6cRXTGjsBB+CIv69H6dPDJ/0UESiiup73b4HvZiTmTk7Yt+q3ILS58VrkWpvargPXoIlER3uLuVt7aFHJCmkmp6jHKhFt/yamgSs8K2w7O14jexFB0FBG2+UGYCaS5PiqmuSdu9uzoGPv47uYNdV45I7FV4bBXlALqSn6X5cYwBw6kvCBFarXo6fDGCIroPGtQoJm7I7PYhUvwLuMicggwNcK+dd+CGDjuOHYzm/EFsoSffEjcABaPBo2VTZGITsMenZ9qgo9ej2+Wz27sHHKuuTFYNYe4XMSDNEcOswMN7O9o4GOpkBAfJaBdyNLS2hnjegVlrtQcPSBEMISziKhnRgnrlZSwMtKL6SBTWBwQNDsbxIybXHWuIkssoJH1rbq53Rp6Q/6h+5jgRUpULt4DGFqrH7a36wFEXR//FiEIg6qhIzmemoOpFEeYKTQx10imqLMLDk2nfv3n/3FwJLJr10/nempPNj3T79mmK5Y8FlT8sebkqREcLLsUuXYua6', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA4FNSKUC7MXWAR2GV/20220111/ap-northeast-1/codeguru-profiler/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=1f2b56c6850d65dd451cc554ce801f2460994150aa8ef89b2846876c3e8abf10', 'Content-Length': '462'}>
2022-01-12T00:50:55.432+09:00 2022-01-11 15:50:55,431 botocore.hooks [DEBUG] Event before-call.cognito-identity-provider.InitiateAuth: calling handler <function inject_api_version_header_if_needed at 0x7f955ceb3dc0>
2022-01-12T00:50:55.432+09:00 [DEBUG] 2022-01-11T15:50:55.431Z a809239e-e99a-47e1-8e72-eb27645a0798 Event before-call.cognito-identity-provider.InitiateAuth: calling handler <function inject_api_version_header_if_needed at 0x7f955ceb3dc0>
2022-01-12T00:50:55.432+09:00 [DEBUG] 2022-01-11T15:50:55.432Z a809239e-e99a-47e1-8e72-eb27645a0798 Certificate path: /var/runtime/botocore/cacert.pem
2022-01-12T00:50:55.432+09:00 2022-01-11 15:50:55,432 botocore.httpsession [DEBUG] Certificate path: /var/runtime/botocore/cacert.pem
2022-01-12T00:50:55.432+09:00 [DEBUG] 2022-01-11T15:50:55.432Z a809239e-e99a-47e1-8e72-eb27645a0798 Making request for OperationModel(name=InitiateAuth) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AWSCognitoIdentityProviderService.InitiateAuth', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55'}, 'body': b'{"AuthFlow": "USER_PASSWORD_AUTH", "AuthParameters": {"USERNAME": "opesupo1#test.com", "PASSWORD": "password"}, "ClientId": "6h3if11rdlftv78gmboefijgk8"}', 'url': 'https://cognito-idp.ap-northeast-1.amazonaws.com/', 'context': {'client_region': 'ap-northeast-1', 'client_config': <botocore.config.Config object at 0x7f955cdb50d0>, 'has_streaming_input': False, 'auth_type': 'none'}}
2022-01-12T00:50:55.433+09:00 2022-01-11 15:50:55,432 botocore.endpoint [DEBUG] Making request for OperationModel(name=InitiateAuth) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AWSCognitoIdentityProviderService.InitiateAuth', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55'}, 'body': b'{"AuthFlow": "USER_PASSWORD_AUTH", "AuthParameters": {"USERNAME": "opesupo1#test.com", "PASSWORD": "password"}, "ClientId": "6h3if11rdlftv78gmboefijgk8"}', 'url': 'https://cognito-idp.ap-northeast-1.amazonaws.com/', 'context': {'client_region': 'ap-northeast-1', 'client_config': <botocore.config.Config object at 0x7f955cdb50d0>, 'has_streaming_input': False, 'auth_type': 'none'}}
2022-01-12T00:50:55.433+09:00 [DEBUG] 2022-01-11T15:50:55.432Z a809239e-e99a-47e1-8e72-eb27645a0798 Starting new HTTPS connection (1): codeguru-profiler.ap-northeast-1.amazonaws.com:443
2022-01-12T00:50:55.433+09:00 2022-01-11 15:50:55,432 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): codeguru-profiler.ap-northeast-1.amazonaws.com:443
2022-01-12T00:50:55.433+09:00 [DEBUG] 2022-01-11T15:50:55.433Z a809239e-e99a-47e1-8e72-eb27645a0798 Event request-created.cognito-identity-provider.InitiateAuth: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f955cdb51c0>>
2022-01-12T00:50:55.433+09:00 2022-01-11 15:50:55,433 botocore.hooks [DEBUG] Event request-created.cognito-identity-provider.InitiateAuth: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f955cdb51c0>>
2022-01-12T00:50:55.434+09:00 [DEBUG] 2022-01-11T15:50:55.434Z a809239e-e99a-47e1-8e72-eb27645a0798 Event choose-signer.cognito-identity-provider.InitiateAuth: calling handler <function set_operation_specific_signer at 0x7f955cead430>
2022-01-12T00:50:55.434+09:00 2022-01-11 15:50:55,434 botocore.hooks [DEBUG] Event choose-signer.cognito-identity-provider.InitiateAuth: calling handler <function set_operation_specific_signer at 0x7f955cead430>
2022-01-12T00:50:55.434+09:00 [DEBUG] 2022-01-11T15:50:55.434Z a809239e-e99a-47e1-8e72-eb27645a0798 Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://cognito-idp.ap-northeast-1.amazonaws.com/, headers={'X-Amz-Target': b'AWSCognitoIdentityProviderService.InitiateAuth', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55', 'Content-Length': '153'}>
2022-01-12T00:50:55.434+09:00 2022-01-11 15:50:55,434 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://cognito-idp.ap-northeast-1.amazonaws.com/, headers={'X-Amz-Target': b'AWSCognitoIdentityProviderService.InitiateAuth', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.18.55 Python/3.9.8 Linux/4.14.252-207.481.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.21.55', 'Content-Length': '153'}>
2022-01-12T00:50:55.435+09:00 [DEBUG] 2022-01-11T15:50:55.435Z a809239e-e99a-47e1-8e72-eb27645a0798 Certificate path: /var/runtime/botocore/cacert.pem
2022-01-12T00:50:55.435+09:00 2022-01-11 15:50:55,435 botocore.httpsession [DEBUG] Certificate path: /var/runtime/botocore/cacert.pem
2022-01-12T00:50:55.436+09:00 2022-01-11 15:50:55,436 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): cognito-idp.ap-northeast-1.amazonaws.com:443
2022-01-12T00:50:55.436+09:00 [DEBUG] 2022-01-11T15:50:55.436Z a809239e-e99a-47e1-8e72-eb27645a0798 Starting new HTTPS connection (1): cognito-idp.ap-northeast-1.amazonaws.com:443
2022-01-12T00:51:00.398+09:00 END RequestId: a809239e-e99a-47e1-8e72-eb27645a0798
2022-01-12T00:51:00.398+09:00 REPORT RequestId: a809239e-e99a-47e1-8e72-eb27645a0798 Duration: 5506.60 ms Billed Duration: 5000 ms Memory Size: 512 MB Max Memory Used: 93 MB Init Duration: 490.72 ms XRAY TraceId: 1-61dda75e-232fd59e22db005744701ec9 SegmentId: 5ad1e3777e572f65 Sampled: true
2022-01-12T00:51:00.398+09:00 2022-01-11T15:51:00.398Z a809239e-e99a-47e1-8e72-eb27645a0798 Task timed out after 5.51 seconds
Please help me.

Celery quits unexpectedly on startup on AWS Elastic Beanstalk

I'm having troubles starting a Celery worker on Elastic Beanstalk instance. After a couple of seconds, it just quits unexpectedly with no error. The instance should have enough RAM. I'm attaching the output by worker with log level debug (sensitive information replaced by **). Any guidance would be super helpful. Thanks.
(venv) [ec2-user#ip-** app]$ celery -A app worker -l debug
[12/Mar/2018 10:18:29] DEBUG [raven.contrib.django.client.DjangoClient:265] Configuring Raven for host: <raven.conf.remote.RemoteConfig object at 0x7f556309e940>
-------------- celery#ip-** v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.9.75-25.55.amzn1.x86_64-x86_64-with-glibc2.3.4 2018-03-12 10:18:29
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: app:0x7f556998edd8
- ** ---------- .> transport: sqs://**:**#localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. common.tasks.send_templated_email
. orders.tasks.import_orders_from_all_companies
[2018-03-12 10:18:29,783: DEBUG/MainProcess] Setting config variable for region to 'eu-central-1'
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable config_file from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable credentials_file from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable data_path from defaults.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable region from instance vars with value 'eu-central-1'.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable ca_bundle from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading variable api_versions from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/endpoints.json
[2018-03-12 10:18:29,790: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,790: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f55633fe048>
[2018-03-12 10:18:29,793: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2018-03-12 10:18:29,796: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f5563448d90>
[2018-03-12 10:18:29,797: DEBUG/MainProcess] The s3 config key is not a dictionary type, ignoring its value of: None
[2018-03-12 10:18:29,802: DEBUG/MainProcess] Setting sqs timeout as (60, 60)
[2018-03-12 10:18:29,802: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/_retry.json
[2018-03-12 10:18:29,803: DEBUG/MainProcess] Registering retry handlers for service: sqs
[2018-03-12 10:18:29,803: DEBUG/MainProcess] Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Making request for OperationModel(name=ListQueues) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'ListQueues', 'Version': '2012-11-05', 'QueueNamePrefix': ''}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5562069f28>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5562069cf8>>
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,805: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,805: DEBUG/MainProcess] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z
content-type;host;x-amz-date
**
[2018-03-12 10:18:29,805: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,806: DEBUG/MainProcess] Signature:
**
[2018-03-12 10:18:29,806: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,807: INFO/MainProcess] Starting new HTTPS connection (1): eu-central-1.queue.amazonaws.com
[2018-03-12 10:18:29,839: DEBUG/MainProcess] "POST / HTTP/1.1" 200 409
[2018-03-12 10:18:29,840: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '409', 'connection': 'keep-alive', 'x-amzn-requestid': '**'}
[2018-03-12 10:18:29,840: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult><QueueUrl>**</QueueUrl><QueueUrl>**</QueueUrl></ListQueuesResult><ResponseMetadata><RequestId>**</RequestId></ResponseMetadata></ListQueuesResponse>'
[2018-03-12 10:18:29,841: DEBUG/MainProcess] Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f556201c390>
[2018-03-12 10:18:29,841: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,841: INFO/MainProcess] Connected to sqs://**:**#localhost//
[2018-03-12 10:18:29,850: DEBUG/MainProcess] Setting config variable for region to 'eu-central-1'
[2018-03-12 10:18:29,850: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable config_file from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable credentials_file from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable data_path from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable region from instance vars with value 'eu-central-1'.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable ca_bundle from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable api_versions from defaults.
[2018-03-12 10:18:29,853: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/endpoints.json
[2018-03-12 10:18:29,857: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,857: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f55633fe048>
[2018-03-12 10:18:29,861: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2018-03-12 10:18:29,863: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f5563448d90>
[2018-03-12 10:18:29,863: DEBUG/MainProcess] The s3 config key is not a dictionary type, ignoring its value of: None
[2018-03-12 10:18:29,865: DEBUG/MainProcess] Setting sqs timeout as (60, 60)
[2018-03-12 10:18:29,865: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/_retry.json
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Registering retry handlers for service: sqs
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Making request for OperationModel(name=ListQueues) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'ListQueues', 'Version': '2012-11-05', 'QueueNamePrefix': ''}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5561acfbe0>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5561acfb38>>
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,868: DEBUG/MainProcess] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z
content-type;host;x-amz-date
**
[2018-03-12 10:18:29,868: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,868: DEBUG/MainProcess] Signature:
**
[2018-03-12 10:18:29,869: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,869: INFO/MainProcess] Starting new HTTPS connection (1): eu-central-1.queue.amazonaws.com
[2018-03-12 10:18:29,895: DEBUG/MainProcess] "POST / HTTP/1.1" 200 409
[2018-03-12 10:18:29,895: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '409', 'connection': 'keep-alive', 'x-amzn-requestid': '5b132733-22e0-5567-8762-74136ac526ec'}
[2018-03-12 10:18:29,896: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult><QueueUrl>**</QueueUrl><QueueUrl>**</QueueUrl></ListQueuesResult><ResponseMetadata><RequestId>5b132733-22e0-5567-8762-74136ac526ec</RequestId></ResponseMetadata></ListQueuesResponse>'
[2018-03-12 10:18:29,896: DEBUG/MainProcess] Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f5561a763c8>
[2018-03-12 10:18:29,896: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,899: DEBUG/MainProcess] Event before-parameter-build.sqs.GetQueueAttributes: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Making request for OperationModel(name=GetQueueAttributes) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'GetQueueAttributes', 'Version': '2012-11-05', 'QueueUrl': '**', 'AttributeName.1': 'ApproximateNumberOfMessages'}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5562069f28>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Event request-created.sqs.GetQueueAttributes: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5562069cf8>>
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Event choose-signer.sqs.GetQueueAttributes: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,901: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,901: DEBUG/MainProcess] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z
content-type;host;x-amz-date
**
[2018-03-12 10:18:29,901: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,901: DEBUG/MainProcess] Signature:
9fb0d1ad68b5d25bf148cc11857b1e1083418557229ca2c47e8b525b54880b74
[2018-03-12 10:18:29,902: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,910: DEBUG/MainProcess] "POST / HTTP/1.1" 200 357
[2018-03-12 10:18:29,911: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '357', 'connection': 'keep-alive', 'x-amzn-requestid': '9aa38f1d-25e5-576d-a9a9-dc3d6dc029a0'}
[2018-03-12 10:18:29,911: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><GetQueueAttributesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><GetQueueAttributesResult><Attribute><Name>ApproximateNumberOfMessages</Name><Value>0</Value></Attribute></GetQueueAttributesResult><ResponseMetadata><RequestId>**</RequestId></ResponseMetadata></GetQueueAttributesResponse>'
[2018-03-12 10:18:29,912: DEBUG/MainProcess] Event needs-retry.sqs.GetQueueAttributes: calling handler <botocore.retryhandler.RetryHandler object at 0x7f556201c390>
[2018-03-12 10:18:29,912: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,921: DEBUG/MainProcess] Canceling task consumer...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] Canceling task consumer...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] Closing consumer channel...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

I have solved the issue. Amazon instance required PyCurl and some additional packages to connect properly with SQS.
I suggest that you already have a script file to run celery daemon(run_supervised_celeryd.sh)
Here is my EB config file:
packages:
yum:
libjpeg-turbo-devel: []
libpng-devel: []
libcurl-devel: []
container_commands:
01_migrate:
command: "django-admin.py migrate --noinput"
leader_only: true
02_collectstatic:
command: "python manage.py collectstatic --noinput"
03_pycurl:
command: 'source /opt/python/run/venv/bin/activate && pip3 install /usr/local/share/pycurl-7.43.0.tar.gz --global-option="--with-nss" --upgrade'
04_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
files:
"/usr/local/share/pycurl-7.43.0.tar.gz" :
mode: "000644"
owner: root
group: root
source: https://pypi.python.org/packages/source/p/pycurl/pycurl-7.43.0.tar.gz
add env variable PYCURL_SSL_LIBRARY="nss"
All listed settings solve the issue

Monitoring tasks not working with django-celery, rabbitmq on virtualenv

I trying to make a work django with django-cellery, I created my project on virtualenv and I used template https://github.com/xenith/django-base-template and followings:
Django==1.6.5
celery==3.1.11
django-celery==3.1.10
My celery settings in settings/local.py
import djcelery
djcelery.setup_loader()
BROKER_URL = 'amqp://django:pas****#10.0.1.17:5672/myvhost'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend'
CELERY_ALWAYS_EAGER = True
I have one Periodic Task "Update_All_Feeds" when I start "celery beat", it seems that everything working fine, task is executed every 10sec.
python manage.py celery beat
celery beat v3.1.11 (Cipater) is starting.
__ - ... __ - _
Configuration ->
. broker -> amqp://django#10.0.1.17:5672/myvhost
. loader -> djcelery.loaders.DjangoLoader
. scheduler -> djcelery.schedulers.DatabaseScheduler
. logfile -> [stderr]#%INFO
. maxinterval -> now (0s)
[2014-05-31 20:53:16,544: INFO/MainProcess] beat: Starting...
[2014-05-31 20:53:16,544: INFO/MainProcess] Writing entries...
[2014-05-31 20:53:16,669: INFO/MainProcess] Scheduler: Sending due task Update_All_ Feeds (update_all_feeds)
[2014-05-31 20:53:19,031: WARNING/MainProcess] /home/phosting/python/django/polskifeed/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:903: RuntimeWarning: DateTimeField FeedItem.pub_date received a naive datetime (2014-05-31 19:21:49) while time zone support is active.
RuntimeWarning)
[2014-05-31 20:53:19,081: INFO/MainProcess] Writing entries...
[2014-05-31 20:53:26,675: INFO/MainProcess] Scheduler: Sending due task Update_All_ Feeds (update_all_feeds)
[2014-05-31 20:53:36,682: INFO/MainProcess] Scheduler: Sending due task Update_All_ Feeds (update_all_feeds)
[2014-05-31 20:53:46,688: INFO/MainProcess] Scheduler: Sending due task Update_All_ Feeds (update_all_feeds)
[2014-05-31 20:53:56,695: INFO/MainProcess] Scheduler: Sending due task Update_All_ Feeds (update_all_feeds)
But starting this with celeryd is not doing anything
python manage.py celeryd -l DEBUG
-------------- celery#czterykaty v3.1.11 (Cipater)
---- **** -----
--- * *** * -- Linux-2.6.32-bpo.5-xen-amd64-x86_64-with-debian-7.0
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: default:0x13bb510 (djcelery.loaders.DjangoLoader)
- ** ---------- .> transport: amqp://django#10.0.1.17:5672/myvhost
- ** ---------- .> results: djcelery.backends.database:DatabaseBackend
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. update_all_feeds
. update_feeds
[2014-05-31 20:58:55,353: DEBUG/MainProcess] | Worker: Starting Hub
[2014-05-31 20:58:55,354: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:55,354: DEBUG/MainProcess] | Worker: Starting Pool
[2014-05-31 20:58:55,672: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:55,675: DEBUG/MainProcess] | Worker: Starting Consumer
[2014-05-31 20:58:55,676: DEBUG/MainProcess] | Consumer: Starting Connection
[2014-05-31 20:58:55,739: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit#czterykaty.luser.nl', u'platform': u'Erlang/OTP', u'version': u'3.3.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2014-05-31 20:58:55,741: DEBUG/MainProcess] Open OK!
[2014-05-31 20:58:55,744: INFO/MainProcess] Connected to amqp://django#10.0.1.17:5672/myvhost
[2014-05-31 20:58:55,744: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:55,744: DEBUG/MainProcess] | Consumer: Starting Events
[2014-05-31 20:58:55,791: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit#czterykaty.luser.nl', u'platform': u'Erlang/OTP', u'version': u'3.3.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2014-05-31 20:58:55,795: DEBUG/MainProcess] Open OK!
[2014-05-31 20:58:55,797: DEBUG/MainProcess] using channel_id: 1
[2014-05-31 20:58:55,800: DEBUG/MainProcess] Channel open
[2014-05-31 20:58:55,802: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:55,802: DEBUG/MainProcess] | Consumer: Starting Mingle
[2014-05-31 20:58:55,803: INFO/MainProcess] mingle: searching for neighbors
[2014-05-31 20:58:55,805: DEBUG/MainProcess] using channel_id: 1
[2014-05-31 20:58:55,807: DEBUG/MainProcess] Channel open
[2014-05-31 20:58:57,266: INFO/MainProcess] mingle: all alone
[2014-05-31 20:58:57,266: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:57,267: DEBUG/MainProcess] | Consumer: Starting Gossip
[2014-05-31 20:58:57,268: DEBUG/MainProcess] using channel_id: 2
[2014-05-31 20:58:57,270: DEBUG/MainProcess] Channel open
[2014-05-31 20:58:57,282: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:57,282: DEBUG/MainProcess] | Consumer: Starting Heart
[2014-05-31 20:58:57,285: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:57,286: DEBUG/MainProcess] | Consumer: Starting Tasks
[2014-05-31 20:58:57,299: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:57,299: DEBUG/MainProcess] | Consumer: Starting Control
[2014-05-31 20:58:57,300: DEBUG/MainProcess] using channel_id: 3
[2014-05-31 20:58:57,303: DEBUG/MainProcess] Channel open
[2014-05-31 20:58:57,311: DEBUG/MainProcess] ^-- substep ok
[2014-05-31 20:58:57,311: DEBUG/MainProcess] | Consumer: Starting event loop
[2014-05-31 20:58:57,315: WARNING/MainProcess] celery#czterykaty ready.
[2014-05-31 20:58:57,316: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2014-05-31 20:58:57,317: DEBUG/MainProcess] basic.qos: prefetch_count->16
Periodic tasks are setup trough djcelery/admin interface, and also tasks list are empty This is my first experience with celery, and djano-celery, so I not sure what is wrong.

I managed this working.
I adjusted my settings in settings/local.py with followings:
import djcelery
djcelery.setup_loader()
BROKER_URL = 'amqp://django:django123#10.0.1.17:5672/myvhost'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
Started celeryd with flag -E and -B
python manage.py celeryd -l INFO -E -B
And for monitoring events, was necessary to start celerycam
python manage.py celerycam

Sending emails using django-SES (Amazon SES)

I've been trying to configure the django-SES services to send outgoing emails and am not sure what is wrong here. Am receving a strange error message,
settings.py
# Email Configuration using Amazon SES Services
EMAIL_BACKEND = 'django_ses.SESBackend'
# These are optional -- if they're set as environment variables they won't
# need to be set here as well
AWS_SES_ACCESS_KEY_ID = 'xxxxxxx'
AWS_SES_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxx'
# Additionally, you can specify an optional region, like so:
AWS_SES_REGION_NAME = 'us-east-1'
AWS_SES_REGION_ENDPOINT = 'email-smtp.us-east-1.amazonaws.com'
In my design, am inserting all emails into a table and then using celery task to go through all pending emails and firing them.
here is my tasks.py
#task(name='common_lib.send_notification', ignore_result=True)
#transaction.commit_manually
def fire_pending_email():
try:
Notification = get_model('common_lib', 'Notification')
NotificationEmail = get_model('common_lib', 'NotificationEmail')
pending_notifications=Notification.objects.values_list('id', flat=True).filter(status=Notification.STATUS_PENDING)
for email in NotificationEmail.objects.filter(notification__in=pending_notifications):
msg = EmailMultiAlternatives(email.subject, email.text_body, 'noreply#xx.com.xx', [email.send_to, ])
if email.html_body:
msg.attach_alternative(email.html_body, "text/html")
msg.send()
transaction.commit()
return 'Successful'
except Exception as e:
transaction.rollback()
logging.error(str(e))
finally:
pass
yet in the celery debug console am seeing the following error
[2012-11-13 11:45:28,061: INFO/MainProcess] Got task from broker: common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384]
[2012-11-13 11:45:28,069: DEBUG/MainProcess] Mediator: Running callback for task: common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384]
[2012-11-13 11:45:28,069: DEBUG/MainProcess] TaskPool: Apply <function trace_task_ret at 0x9f38a3c> (args:('common_lib.send_notification', '4dc71dee-fc7c-4ddc-a02c-4097c73e4384', [], {}, {'retries': 0, 'is_eager': False, 'task': 'common_lib.send_notification', 'group': None, 'eta': None, 'delivery_info': {'priority': None, 'routing_key': u'celery', 'exchange': u'celery'}, 'args': [], 'expires': None, 'callbacks': None, 'errbacks': None, 'hostname': 'ubuntu', 'kwargs': {}, 'id': '4dc71dee-fc7c-4ddc-a02c-4097c73e4384', 'utc': True}) kwargs:{})
[2012-11-13 11:45:28,077: DEBUG/MainProcess] Task accepted: common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384] pid:8256
[2012-11-13 11:45:28,097: DEBUG/MainProcess] (0.001) SELECT `common_lib_notification_email`.`id`, `common_lib_notification_email`.`notification_id`, `common_lib_notification_email`.`send_to`, `common_lib_notification_email`.`template`, `common_lib_notification_email`.`subject`, `common_lib_notification_email`.`html_body`, `common_lib_notification_email`.`text_body` FROM `common_lib_notification_email` WHERE `common_lib_notification_email`.`notification_id` IN (SELECT U0.`id` FROM `common_lib_notification` U0 WHERE U0.`status` = 'P' ); args=(u'P',)
[2012-11-13 11:45:28,103: DEBUG/MainProcess] Method: POST
[2012-11-13 11:45:28,107: DEBUG/MainProcess] Path: /
[2012-11-13 11:45:28,107: DEBUG/MainProcess] Data: Action=GetSendQuota
[2012-11-13 11:45:28,107: DEBUG/MainProcess] Headers: {'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'}
[2012-11-13 11:45:28,109: DEBUG/MainProcess] Host: email-smtp.us-east-1.amazonaws.com
[2012-11-13 11:45:28,109: DEBUG/MainProcess] establishing HTTPS connection: host=email-smtp.us-east-1.amazonaws.com, kwargs={}
[2012-11-13 11:45:28,109: DEBUG/MainProcess] Token: None
[2012-11-13 11:45:28,702: DEBUG/MainProcess] wrapping ssl socket; CA certificate file=/home/mo/projects/garageenv/local/lib/python2.7/site-packages/boto/cacerts/cacerts.txt
[2012-11-13 11:45:29,385: DEBUG/MainProcess] validating server certificate: hostname=email-smtp.us-east-1.amazonaws.com, certificate hosts=[u'email-smtp.us-east-1.amazonaws.com']
[2012-11-13 11:45:39,618: ERROR/MainProcess] <unknown>:1:0: syntax error
[2012-11-13 11:45:39,619: INFO/MainProcess] Task common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384] succeeded in 11.5491399765s: None
UPDATE
when I changed the setting to
AWS_SES_REGION_ENDPOINT = 'email.us-east-1.amazonaws.com'
I got a different error, as below
[2012-11-13 13:24:05,907: DEBUG/MainProcess] Method: POST
[2012-11-13 13:24:05,916: DEBUG/MainProcess] Path: /
[2012-11-13 13:24:05,917: DEBUG/MainProcess] Data: Action=GetSendQuota
[2012-11-13 13:24:05,917: DEBUG/MainProcess] Headers: {'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'}
[2012-11-13 13:24:05,918: DEBUG/MainProcess] Host: email.us-east-1.amazonaws.com
[2012-11-13 13:24:05,918: DEBUG/MainProcess] establishing HTTPS connection: host=email.us-east-1.amazonaws.com, kwargs={}
[2012-11-13 13:24:05,919: DEBUG/MainProcess] Token: None
[2012-11-13 13:24:06,511: DEBUG/MainProcess] wrapping ssl socket; CA certificate file=/home/mo/projects/garageenv/local/lib/python2.7/site-packages/boto/cacerts/cacerts.txt
[2012-11-13 13:24:06,952: DEBUG/MainProcess] validating server certificate: hostname=email.us-east-1.amazonaws.com, certificate hosts=['email.us-east-1.amazonaws.com', 'email.amazonaws.com']
[2012-11-13 13:24:07,177: ERROR/MainProcess] 403 Forbidden
[2012-11-13 13:24:07,178: ERROR/MainProcess] <ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestId>41c15592-2d7c-11e2-a590-f33d1568f3ea</RequestId>
</ErrorResponse>
[2012-11-13 13:24:07,180: ERROR/MainProcess] BotoServerError: 403 Forbidden
<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestId>41c15592-2d7c-11e2-a590-f33d1568f3ea</RequestId>
</ErrorResponse>
[2012-11-13 13:24:07,184: INFO/MainProcess] Task common_lib.send_notification[3b6a049e-d5cb-45f4-842b-633d816a132e] succeeded in 1.31089687347s: None

Can you try using this
AWS_SES_REGION_ENDPOINT = 'email.us-east-1.amazonaws.com'
And not the smtp server setting on AWS's dashboard?
(You used AWS_SES_REGION_ENDPOINT = 'email-smtp.us-east-1.amazonaws.com' as mentioned above)
Once you have updated this, you got a new error as updated in your question. This confirms that you now have the correct AWS_SES_REGION_ENDPOINT setting set.
The reason you are getting this new error is most likely because you are confusing the access keys and giving amazon a wrong set of credentials - see detailed comments here - https://github.com/boto/boto/issues/476#issuecomment-7679158
Follow the solution prescribed in the comment and you should be fine, I think.