I've been trying to configure the django-SES services to send outgoing emails and am not sure what is wrong here. Am receving a strange error message,
settings.py
# Email Configuration using Amazon SES Services
EMAIL_BACKEND = 'django_ses.SESBackend'
# These are optional -- if they're set as environment variables they won't
# need to be set here as well
AWS_SES_ACCESS_KEY_ID = 'xxxxxxx'
AWS_SES_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxx'
# Additionally, you can specify an optional region, like so:
AWS_SES_REGION_NAME = 'us-east-1'
AWS_SES_REGION_ENDPOINT = 'email-smtp.us-east-1.amazonaws.com'
In my design, am inserting all emails into a table and then using celery task to go through all pending emails and firing them.
here is my tasks.py
#task(name='common_lib.send_notification', ignore_result=True)
#transaction.commit_manually
def fire_pending_email():
try:
Notification = get_model('common_lib', 'Notification')
NotificationEmail = get_model('common_lib', 'NotificationEmail')
pending_notifications=Notification.objects.values_list('id', flat=True).filter(status=Notification.STATUS_PENDING)
for email in NotificationEmail.objects.filter(notification__in=pending_notifications):
msg = EmailMultiAlternatives(email.subject, email.text_body, 'noreply#xx.com.xx', [email.send_to, ])
if email.html_body:
msg.attach_alternative(email.html_body, "text/html")
msg.send()
transaction.commit()
return 'Successful'
except Exception as e:
transaction.rollback()
logging.error(str(e))
finally:
pass
yet in the celery debug console am seeing the following error
[2012-11-13 11:45:28,061: INFO/MainProcess] Got task from broker: common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384]
[2012-11-13 11:45:28,069: DEBUG/MainProcess] Mediator: Running callback for task: common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384]
[2012-11-13 11:45:28,069: DEBUG/MainProcess] TaskPool: Apply <function trace_task_ret at 0x9f38a3c> (args:('common_lib.send_notification', '4dc71dee-fc7c-4ddc-a02c-4097c73e4384', [], {}, {'retries': 0, 'is_eager': False, 'task': 'common_lib.send_notification', 'group': None, 'eta': None, 'delivery_info': {'priority': None, 'routing_key': u'celery', 'exchange': u'celery'}, 'args': [], 'expires': None, 'callbacks': None, 'errbacks': None, 'hostname': 'ubuntu', 'kwargs': {}, 'id': '4dc71dee-fc7c-4ddc-a02c-4097c73e4384', 'utc': True}) kwargs:{})
[2012-11-13 11:45:28,077: DEBUG/MainProcess] Task accepted: common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384] pid:8256
[2012-11-13 11:45:28,097: DEBUG/MainProcess] (0.001) SELECT `common_lib_notification_email`.`id`, `common_lib_notification_email`.`notification_id`, `common_lib_notification_email`.`send_to`, `common_lib_notification_email`.`template`, `common_lib_notification_email`.`subject`, `common_lib_notification_email`.`html_body`, `common_lib_notification_email`.`text_body` FROM `common_lib_notification_email` WHERE `common_lib_notification_email`.`notification_id` IN (SELECT U0.`id` FROM `common_lib_notification` U0 WHERE U0.`status` = 'P' ); args=(u'P',)
[2012-11-13 11:45:28,103: DEBUG/MainProcess] Method: POST
[2012-11-13 11:45:28,107: DEBUG/MainProcess] Path: /
[2012-11-13 11:45:28,107: DEBUG/MainProcess] Data: Action=GetSendQuota
[2012-11-13 11:45:28,107: DEBUG/MainProcess] Headers: {'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'}
[2012-11-13 11:45:28,109: DEBUG/MainProcess] Host: email-smtp.us-east-1.amazonaws.com
[2012-11-13 11:45:28,109: DEBUG/MainProcess] establishing HTTPS connection: host=email-smtp.us-east-1.amazonaws.com, kwargs={}
[2012-11-13 11:45:28,109: DEBUG/MainProcess] Token: None
[2012-11-13 11:45:28,702: DEBUG/MainProcess] wrapping ssl socket; CA certificate file=/home/mo/projects/garageenv/local/lib/python2.7/site-packages/boto/cacerts/cacerts.txt
[2012-11-13 11:45:29,385: DEBUG/MainProcess] validating server certificate: hostname=email-smtp.us-east-1.amazonaws.com, certificate hosts=[u'email-smtp.us-east-1.amazonaws.com']
[2012-11-13 11:45:39,618: ERROR/MainProcess] <unknown>:1:0: syntax error
[2012-11-13 11:45:39,619: INFO/MainProcess] Task common_lib.send_notification[4dc71dee-fc7c-4ddc-a02c-4097c73e4384] succeeded in 11.5491399765s: None
UPDATE
when I changed the setting to
AWS_SES_REGION_ENDPOINT = 'email.us-east-1.amazonaws.com'
I got a different error, as below
[2012-11-13 13:24:05,907: DEBUG/MainProcess] Method: POST
[2012-11-13 13:24:05,916: DEBUG/MainProcess] Path: /
[2012-11-13 13:24:05,917: DEBUG/MainProcess] Data: Action=GetSendQuota
[2012-11-13 13:24:05,917: DEBUG/MainProcess] Headers: {'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'}
[2012-11-13 13:24:05,918: DEBUG/MainProcess] Host: email.us-east-1.amazonaws.com
[2012-11-13 13:24:05,918: DEBUG/MainProcess] establishing HTTPS connection: host=email.us-east-1.amazonaws.com, kwargs={}
[2012-11-13 13:24:05,919: DEBUG/MainProcess] Token: None
[2012-11-13 13:24:06,511: DEBUG/MainProcess] wrapping ssl socket; CA certificate file=/home/mo/projects/garageenv/local/lib/python2.7/site-packages/boto/cacerts/cacerts.txt
[2012-11-13 13:24:06,952: DEBUG/MainProcess] validating server certificate: hostname=email.us-east-1.amazonaws.com, certificate hosts=['email.us-east-1.amazonaws.com', 'email.amazonaws.com']
[2012-11-13 13:24:07,177: ERROR/MainProcess] 403 Forbidden
[2012-11-13 13:24:07,178: ERROR/MainProcess] <ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestId>41c15592-2d7c-11e2-a590-f33d1568f3ea</RequestId>
</ErrorResponse>
[2012-11-13 13:24:07,180: ERROR/MainProcess] BotoServerError: 403 Forbidden
<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestId>41c15592-2d7c-11e2-a590-f33d1568f3ea</RequestId>
</ErrorResponse>
[2012-11-13 13:24:07,184: INFO/MainProcess] Task common_lib.send_notification[3b6a049e-d5cb-45f4-842b-633d816a132e] succeeded in 1.31089687347s: None
Can you try using this
AWS_SES_REGION_ENDPOINT = 'email.us-east-1.amazonaws.com'
And not the smtp server setting on AWS's dashboard?
(You used AWS_SES_REGION_ENDPOINT = 'email-smtp.us-east-1.amazonaws.com' as mentioned above)
Once you have updated this, you got a new error as updated in your question. This confirms that you now have the correct AWS_SES_REGION_ENDPOINT setting set.
The reason you are getting this new error is most likely because you are confusing the access keys and giving amazon a wrong set of credentials - see detailed comments here - https://github.com/boto/boto/issues/476#issuecomment-7679158
Follow the solution prescribed in the comment and you should be fine, I think.
Related
lambda backend function is invoked every 20 sec from s3 static website - GET request goes via api gateway calls lambda, it tries to get item from dynamodb table and should return it. It returns 500 error- smth wrong on server side.
Enough time(9sec) is given for the lambda to finish executing code - lambda's own time out should not be an issue.
Lambda has a role attached that allows get_item from dynamodb table - IAM is not an issue.
However, when troubleshooting i only see logs in cloudwatch until it tries to get_item. I put lots of print and logs, but it does not go past that line, I even nested try..except - it does not catch any errors. I dont see how to detect whats' wrong. I set logging level to debug - it prints some stuff.
import logging
import boto3
import sys
logging.getLogger().setLevel(logging.DEBUG)
def lambda_handler(event, context):
logging.info('doing retrieving from table votes')
try:
logging.info('********************* TRYING retrieving from table votes')
# dynamodb = boto3.client('dynamodb')
table = boto3.resource('dynamodb', region_name='us-east-1').Table('Votes')
print(table)
logging.info('event')
logging.info(event)
print(type(table))
logging.info(table)
logging.info(type(table))
# logging.error(table)
# logging.error(type(table))
#************************
# I dont see the result of count nowhere in the logs of cloud watch
try:
count = table.get_item(Key={'voter':{'S': 'count'}})
except Exception as e:
logging.info('catching it here - if you see it then something wrong with get_item count')
logging.info('********************BAD******************')
logging.error('********************BAD******************')
e = sys.exc_info()[0]
exception_type = e.__class__.__name__
exception_message = str(e)
logging.error('--------------------------------')
logging.error(exception_message)
logging.error(exception_type)
################## BELOW DONT GET PRINTED AT ALL in cloud watch logs
logging.info('count')
logging.info(count)
print('****************COUNT*********************')
print(count)
print('----------------------------------------------------')
logging.info(count)
a = count["Item"]["a"]
b = count["Item"]["b"]
logging.info('count [Item]')
logging.info(count["Item"])
logging.info('------------------------------------')
logging.info('ok retrieve from table votes')
logging.info('a is ' + a)
logging.info('b is ' + b)
logging.info('************************************ success! a: ' + a + ' and b: ' + b)
return {'statusCode': 200, 'body': '{"a": ' + a + ', "b": ' + b + '}'}
except Exception as e:
logging.info('********************BAD******************')
e = sys.exc_info()[0]
exception_type = e.__class__.__name__
exception_message = str(e)
logging.error('--------------------------------')
logging.error(exception_message)
logging.error(exception_type)
logging.error('---------------------------------------------')
return {'statusCode': 500, 'body': '{"status": "error getting from table votes"}'}
the full logs of 1 request id from start to finish. For some reason it never prints the result of "count = get_item)
START RequestId: 55a5e428-c8d6-4914-908c-20ccec1153dd Version: $LATEST
2023-01-11T21:29:08.805+05:00
[INFO] 2023-01-11T16:29:08.805Z 55a5e428-c8d6-4914-908c-20ccec1153dd doing retrieving from table votes
2023-01-11T21:29:08.805+05:00
[INFO] 2023-01-11T16:29:08.805Z 55a5e428-c8d6-4914-908c-20ccec1153dd ********************* TRYING retrieving from table votes
2023-01-11T21:29:08.806+05:00
[DEBUG] 2023-01-11T16:29:08.806Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-01-11T21:29:08.824+05:00
[DEBUG] 2023-01-11T16:29:08.824Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-call.apigateway to before-call.api-gateway
2023-01-11T21:29:08.825+05:00
[DEBUG] 2023-01-11T16:29:08.825Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-01-11T21:29:08.826+05:00
[DEBUG] 2023-01-11T16:29:08.826Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-01-11T21:29:08.827+05:00
[DEBUG] 2023-01-11T16:29:08.827Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-01-11T21:29:08.827+05:00
[DEBUG] 2023-01-11T16:29:08.827Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-01-11T21:29:08.884+05:00
[DEBUG] 2023-01-11T16:29:08.884Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.924Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.925Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.925Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-01-11T21:29:08.925+05:00
[DEBUG] 2023-01-11T16:29:08.925Z 55a5e428-c8d6-4914-908c-20ccec1153dd Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2023-01-11T21:29:09.044+05:00
[DEBUG] 2023-01-11T16:29:09.044Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/boto3/data/dynamodb/2012-08-10/resources-1.json
2023-01-11T21:29:09.047+05:00
[DEBUG] 2023-01-11T16:29:09.047Z 55a5e428-c8d6-4914-908c-20ccec1153dd IMDS ENDPOINT: http://169.254.169.254/
2023-01-11T21:29:09.105+05:00
[DEBUG] 2023-01-11T16:29:09.105Z 55a5e428-c8d6-4914-908c-20ccec1153dd Looking for credentials via: env
2023-01-11T21:29:09.105+05:00
[INFO] 2023-01-11T16:29:09.105Z 55a5e428-c8d6-4914-908c-20ccec1153dd Found credentials in environment variables.
2023-01-11T21:29:09.106+05:00
[DEBUG] 2023-01-11T16:29:09.106Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/botocore/data/endpoints.json
2023-01-11T21:29:09.266+05:00
[DEBUG] 2023-01-11T16:29:09.266Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa39bdec160>
2023-01-11T21:29:09.365+05:00
[DEBUG] 2023-01-11T16:29:09.364Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/botocore/data/dynamodb/2012-08-10/service-2.json
2023-01-11T21:29:09.427+05:00
[DEBUG] 2023-01-11T16:29:09.426Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-client-class.dynamodb: calling handler <function add_generate_presigned_url at 0x7fa39be91c10>
2023-01-11T21:29:09.485+05:00
[DEBUG] 2023-01-11T16:29:09.484Z 55a5e428-c8d6-4914-908c-20ccec1153dd Setting dynamodb timeout as (60, 60)
2023-01-11T21:29:09.485+05:00
[DEBUG] 2023-01-11T16:29:09.485Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading JSON file: /var/runtime/botocore/data/_retry.json
2023-01-11T21:29:09.486+05:00
[DEBUG] 2023-01-11T16:29:09.486Z 55a5e428-c8d6-4914-908c-20ccec1153dd Registering retry handlers for service: dynamodb
2023-01-11T21:29:09.487+05:00
[DEBUG] 2023-01-11T16:29:09.486Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading dynamodb:dynamodb
2023-01-11T21:29:09.487+05:00
[DEBUG] 2023-01-11T16:29:09.487Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-resource-class.dynamodb.ServiceResource: calling handler <function lazy_call.<locals>._handler at 0x7fa39bd7aca0>
2023-01-11T21:29:09.546+05:00
[DEBUG] 2023-01-11T16:29:09.546Z 55a5e428-c8d6-4914-908c-20ccec1153dd Loading dynamodb:Table
2023-01-11T21:29:09.547+05:00
[DEBUG] 2023-01-11T16:29:09.547Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-resource-class.dynamodb.Table: calling handler <function lazy_call.<locals>._handler at 0x7fa39bd7ad30>
2023-01-11T21:29:09.584+05:00
[DEBUG] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event creating-resource-class.dynamodb.Table: calling handler <function lazy_call.<locals>._handler at 0x7fa39bd7aca0>
2023-01-11T21:29:09.584+05:00
dynamodb.Table(name='Votes')
2023-01-11T21:29:09.584+05:00
[INFO] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd event
2023-01-11T21:29:09.584+05:00
[INFO] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd {'version': '2.0', 'routeKey': 'GET /results', 'rawPath': '/results', 'rawQueryString': '', 'headers': {'accept': 'application/json','user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36', 'x-amzn-trace-id': 'Root=1-63bee3d4-679c4d201abf991d1f331f33', 'x-forwarded-for': '164.40.37.179', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https'}, 'requestContext': {'accountId': '025416187662', 'apiId': '5y7dfynd34', 'domainName': '5y7dfynd34.execute-api.us-east-1.amazonaws.com', 'domainPrefix': '5y7dfynd34', 'http': {'method': 'GET', 'path': '/results', 'protocol': 'HTTP/1.1', 'sourceIp': '164.40.37.179', 'userAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'}, 'requestId': 'eliJRin9oAMEc9Q=', 'routeKey': 'GET /results', 'stage': '$default', 'time': '11/Jan/2023:16:29:08 +0000', 'timeEpoch': 1673454548760}, 'isBase64Encoded': False}
2023-01-11T21:29:09.584+05:00
<class 'boto3.resources.factory.dynamodb.Table'>
2023-01-11T21:29:09.585+05:00
[INFO] 2023-01-11T16:29:09.584Z 55a5e428-c8d6-4914-908c-20ccec1153dd dynamodb.Table(name='Votes')
2023-01-11T21:29:09.585+05:00
[INFO] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd <class 'boto3.resources.factory.dynamodb.Table'>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Calling dynamodb:get_item with {'TableName': 'Votes', 'Key': {'voter': {'S': 'count'}}}
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event provide-client-params.dynamodb.GetItem: calling handler <function _dynamodb_params at 0x7fa39b870ca0>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <bound method TransformationInjector.inject_condition_expressions of <boto3.dynamodb.transform.TransformationInjector object at 0x7fa39b852730>>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <bound method TransformationInjector.inject_attribute_value_input of <boto3.dynamodb.transform.TransformationInjector object at 0x7fa39b852730>>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <function generate_idempotent_uuid at 0x7fa39be0d3a0>
2023-01-11T21:29:09.585+05:00
[DEBUG] 2023-01-11T16:29:09.585Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-parameter-build.dynamodb.GetItem: calling handler <function block_endpoint_discovery_required_operations at 0x7fa39be32d30>
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event before-call.dynamodb.GetItem: calling handler <function inject_api_version_header_if_needed at 0x7fa39be11c10>
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Making request for OperationModel(name=GetItem) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'DynamoDB_20120810.GetItem', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.20.32 Python/3.9.13 Linux/4.14.255-296-236.539.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.23.32 Resource'}, 'body': b'{"TableName": "Votes", "Key": {"voter": {"M": {"S": {"S": "count"}}}}}', 'url': 'https://dynamodb.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa39b897c40>, 'has_streaming_input': False, 'auth_type': None}}
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event request-created.dynamodb.GetItem: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa39b897a90>>
2023-01-11T21:29:09.586+05:00
[DEBUG] 2023-01-11T16:29:09.586Z 55a5e428-c8d6-4914-908c-20ccec1153dd Event choose-signer.dynamodb.GetItem: calling handler <function set_operation_specific_signer at 0x7fa39be0d280>
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd Calculating signature using v4 auth.
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.0
host:dynamodb.us-east-1.amazonaws.com
x-amz-date:20230111T162909Z
x-amz-security- x-amz-target:DynamoDB_20120810.GetItem
content-type;host;x-amz-date;x-amz-security-token;x-amz-target
6dd016d6033694be300988a73dded6cba15ade0cf920e8bafb56369e3719c397
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd CanonicalRequest: POST / content-type:application/x-amz-json-1.0 host:dynamodb.us-east-1.amazonaws.com x-amz-date:20230111T162909Z x-amz-target:DynamoDB_20120810.GetItem content-type;host;x-amz-date;x-amz-security-token;x-amz-target 6dd016d6033694be300988a73dded6cba15ade0cf920e8bafb56369e3719c397
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd StringToSign:
AWS4-HMAC-SHA256
20230111T162909Z
20230111/us-east-1/dynamodb/aws4_request
33bbba9cdeb906cc5b3ddc600b02d47f0a73e019d5f3efa0627ea82e05e86eee
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd StringToSign: AWS4-HMAC-SHA256 20230111T162909Z 20230111/us-east-1/dynamodb/aws4_request 33bbba9cdeb906cc5b3ddc600b02d47f0a73e019d5f3efa0627ea82e05e86eee
2023-01-11T21:29:09.587+05:00
[DEBUG] 2023-01-11T16:29:09.587Z 55a5e428-c8d6-4914-908c-20ccec1153dd Signature: f36e8c5a9c7d47f1ef41c1ecce566a988f7243f8f95bf7f7c43b951a87e488eb
2023-01-11T21:29:09.644+05:00
[DEBUG] 2023-01-11T16:29:09.643Z 55a5e428-c8d6-4914-908c-20ccec1153dd Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://dynamodb.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'DynamoDB_20120810.GetItem', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.20.32 Python/3.9.13 Linux/4.14.255-296-236.539.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.23.32 Resource', 'X-Amz-Date': b'20230111T162909Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAQL2XMH4HN6PEJXPA/20230111/us-east-1/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=f36e8c5a9c7d47f1ef41c1ecce566a988f7243f8f95bf7f7c43b951a87e488eb', 'Content-Length': '70'}>
[DEBUG] 2023-01-11T16:29:09.644Z 55a5e428-c8d6-4914-908c-20ccec1153dd Certificate path: /var/runtime/botocore/cacert.pem
2023-01-11T21:29:09.644+05:00
[DEBUG] 2023-01-11T16:29:09.644Z 55a5e428-c8d6-4914-908c-20ccec1153dd Starting new HTTPS connection (1): dynamodb.us-east-1.amazonaws.com:443
2023-01-11T21:29:17.816+05:00
2023-01-11T16:29:17.815Z 55a5e428-c8d6-4914-908c-20ccec1153dd Task timed out after 9.01 seconds
2023-01-11T21:29:17.816+05:00
END RequestId: 55a5e428-c8d6-4914-908c-20ccec1153dd
It looks like the issue is that the table.get_item(Key={'voter':{'S': 'count'}}) call is raising an exception, but your exception handling code is not catching it. One possible cause for this could be that the table.get_item method is returning a botocore.exception.ClientError object, rather than a Python Exception object. You can modify your except block to catch botocore.exception.ClientError and add some log information like request_id and error message in the catch block.
Another way to detect the error is to enable enhanced monitoring for your lambda function, this will give more detailed metrics on the function performance and error.
One thing you could try is to use sys.exc_info() to get the current exception inside the catch block, and print out the exception's class and message. That way you can see exactly what type of exception is being raised and what the error message is.
You could also try to call get_item separately in a local environment to see if it raises an exception, this will help you to figure out if the issue is with your code or the IAM policy or resource configurations.
This might be a duplicate but none of the previous answers match my conditions.
I installed gsutil as part of the google-cloud-sdk following https://cloud.google.com/sdk/docs/install. I could configure gcloud properly without errors.
Every time I try to use gsutil, like for example with gsutil -D ls, I get
INFO 0518 14:52:16.412453 base_api.py] Body: (none)
INFO 0518 14:52:16.412517 transport.py] Attempting refresh to obtain initial access_token
DEBUG 0518 14:52:16.412719 multiprocess_file_storage.py] Read credential file
DEBUG 0518 14:52:16.412842 multiprocess_file_storage.py] Read credential file
INFO 0518 14:52:16.412883 reauth_creds.py] Refreshing access_token
INFO 0518 14:53:16.546304 retry_util.py] Retrying request, attempt #1...
DEBUG 0518 14:53:16.546867 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 0518 14:53:16.547127 http_wrapper.py] Retrying request to url https://storage.googleapis.com/storage/v1/b?alt=blablabla after exception timed out
and more and more of those retries.
I see some users here that experienced the same, for instance this points out to a WAN Blocking setting enabled, that is not my case. Here the OP says that it was a human error regarding proxy settings, but I don't have any
➜ set | grep -i proxy
The same proxy thing seems to have solved it for another OP
In the same question another user says that it might be due to a conflicting ~/.boto config file, so I deleted it and tried, with the same results.
I tried reinstalling google SDK several times with the same result.
I tried configuring gsutil as a standalone application setting gcloud config set pass_credentials_to_gsutil false and running gsutil config. Again, without luck
This user seems to be experiencing my same problem, but he ends up saying that his solution was to restart the shell (exec -l $SHELL) and quitting/reopening the command line and keep trying until it works...
So my question is, does anyone know a reliable non-proxy related way to solve this retrying issue in gsutil?
EDIT 1:
The output of curl -v https://storage.googleapis.com/storage/v1/b is
* Trying 2800:3f0:4002:800::2010:443...
* TCP_NODELAY set
* Trying 172.217.30.240:443...
* TCP_NODELAY set
* Connected to storage.googleapis.com (172.217.30.240) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.storage.googleapis.com
* start date: Apr 13 10:15:35 2021 GMT
* expire date: Jul 6 10:15:34 2021 GMT
* subjectAltName: host "storage.googleapis.com" matched cert's "*.googleapis.com"
* issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564e279b8e10)
> GET /storage/v1/b HTTP/2
> Host: storage.googleapis.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 400
< x-guploader-uploadid: ABg5-Ux5VPhppIWB7G_da1ydkOJWv1BqXepMdpyJDPZ3zbTSRwPoqE44IqaPQMzLvWbSOab0bePewJXiwBXPpus9JDs
< content-type: application/json; charset=UTF-8
< date: Tue, 18 May 2021 19:41:52 GMT
< vary: Origin
< vary: X-Origin
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< expires: Mon, 01 Jan 1990 00:00:00 GMT
< pragma: no-cache
< content-length: 297
< server: UploadServer
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
{
"error": {
"code": 400,
"message": "Required parameter: project",
"errors": [
{
"message": "Required parameter: project",
"domain": "global",
"reason": "required",
"locationType": "parameter",
"location": "project"
}
]
}
}
* Connection #0 to host storage.googleapis.com left intact
EDIT 2:
The complete output of gsutil -D ls is:
➜ gsutil -D ls
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 4.62
checksum: fe14a00285d4702ed626050d0f9ae955 (OK)
boto version: 2.49.0
python version: 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
OS: Linux 5.8.0-50-generic
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /home/username/.config/gcloud/legacy_credentials/username#mail.com/.boto
gsutil path: /home/username/google-cloud-sdk/bin/gsutil
compiled crcmod: False
installed via package manager: False
editable install: False
Command being run: /home/username/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=default_project -o Credentials:use_client_certificate=False -D ls
config_file_list: ['/home/username/.config/gcloud/legacy_credentials/username#mail.com/.boto']
config: [('working_dir', '/mnt/pyami'), ('debug', '0'), ('https_validate_certificates', 'true'), ('working_dir', '/mnt/pyami'), ('debug', '0'), ('default_project_id', 'default_project')]
DEBUG 0518 17:46:39.910250 multiprocess_file_storage.py] Read credential file
DEBUG 0518 17:46:39.910444 multiprocess_file_storage.py] Read credential file
INFO 0518 17:46:39.910933 base_api.py] Calling method storage.buckets.list with StorageBucketsListRequest: <StorageBucketsListRequest
maxResults: 1000
project: 'default_project'
projection: ProjectionValueValuesEnum(noAcl, 1)>
INFO 0518 17:46:39.911321 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b?alt=json&fields=nextPageToken%2Citems%2Fid&maxResults=1000&project=default_project&projection=noAcl
INFO 0518 17:46:39.911495 base_api.py] Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools Python/3.8.5 gsutil/4.62 (linux) analytics/disabled '
'interactive/True command/ls google-cloud-sdk/341.0.0'}
INFO 0518 17:46:39.911611 base_api.py] Body: (none)
INFO 0518 17:46:39.911687 transport.py] Attempting refresh to obtain initial access_token
DEBUG 0518 17:46:39.911900 multiprocess_file_storage.py] Read credential file
DEBUG 0518 17:46:39.912035 multiprocess_file_storage.py] Read credential file
INFO 0518 17:46:39.912081 reauth_creds.py] Refreshing access_token
INFO 0518 17:47:40.014159 retry_util.py] Retrying request, attempt #1...
DEBUG 0518 17:47:40.014368 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 0518 17:47:40.014440 http_wrapper.py] Retrying request to url https://storage.googleapis.com/storage/v1/b?alt=json&fields=nextPageToken%2Citems%2Fid&maxResults=1000&project=default_project&projection=noAcl after exception timed out
INFO 0518 17:47:41.531516 transport.py] Attempting refresh to obtain initial access_token
DEBUG 0518 17:47:41.532971 multiprocess_file_storage.py] Read credential file
DEBUG 0518 17:47:41.533422 multiprocess_file_storage.py] Read credential file
INFO 0518 17:47:41.533568 reauth_creds.py] Refreshing access_token
INFO 0518 17:48:41.590354 retry_util.py] Retrying request, attempt #2...
DEBUG 0518 17:48:41.590671 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 0518 17:48:41.590815 http_wrapper.py] Retrying request to url https://storage.googleapis.com/storage/v1/b?alt=json&fields=nextPageToken%2Citems%2Fid&maxResults=1000&project=default_project&projection=noAcl after exception timed out
INFO 0518 17:48:46.107598 transport.py] Attempting refresh to obtain initial access_token
DEBUG 0518 17:48:46.108518 multiprocess_file_storage.py] Read credential file
DEBUG 0518 17:48:46.108928 multiprocess_file_storage.py] Read credential file
INFO 0518 17:48:46.109037 reauth_creds.py] Refreshing access_token
^CDEBUG: Exception stack trace:
NoneType: None
DEBUG: Caught CTRL-C (signal 2) - Exception stack trace:
File "/home/username/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
gsutil.RunMain()
File "/home/username/google-cloud-sdk/platform/gsutil/gsutil.py", line 122, in RunMain
sys.exit(gslib.__main__.main())
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 435, in main
return _RunNamedCommandAndHandleExceptions(
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 631, in _RunNamedCommandAndHandleExceptions
return command_runner.RunNamedCommand(command_name,
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 568, in RunCommand
for blr in self.WildcardIterator(
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 484, in IterBuckets
for blr in self._ExpandBucketWildcards(bucket_fields=bucket_fields):
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 400, in _ExpandBucketWildcards
for bucket in self.gsutil_api.ListBuckets(
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 703, in ListBuckets
bucket_list = self.api_client.buckets.List(apitools_request,
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 362, in List
return self._RunMethod(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 734, in _RunMethod
http_response = http_wrapper.MakeRequest(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 348, in MakeRequest
return _MakeRequestNoRetry(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 397, in _MakeRequestNoRetry
info, content = http.request(
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/transport.py", line 159, in new_request
credentials._refresh(orig_request_method)
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/client.py", line 761, in _refresh
self._do_refresh_request(http)
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth_creds.py", line 112, in _do_refresh_request
self._update(*reauth.refresh_access_token(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 267, in refresh_access_token
response, content = _reauth_client.refresh_grant(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/_reauth_client.py", line 147, in refresh_grant
response, content = http_request(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth_creds.py", line 105, in http_request
response, content = transport.request(
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1985, in request
(response, content) = self._request(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1650, in _request
(response, content) = self._conn_request(
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1557, in _conn_request
conn.connect()
File "/home/username/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1324, in connect
sock.connect((self.host, self.port))
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/sig_handling.py", line 92, in _SignalHandler
_final_signal_handlers[signal_num](signal_num, cur_stack_frame)
File "/home/username/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 519, in _HandleControlC
stack_trace = ''.join(traceback.format_list(traceback.extract_stack()))
I have the same exact error, and I found out that its due to my machine resolving hosts into ipv6 which causes timeouts, so in ubuntu I just have to disable ipv6 and restart, after that YAY! it works now I can gsutil.
for some other to encounter same issue try doing this. :)
After giving up on this I decided to reinstall one last time the whole google-cloud-sdk suite, but this time using the snap version. Installing it via snap solved the issue for me. I think this points to some issue with my environment that was bypassed thanks to the snap containerization.
So no clear answer here, but if anyone is experiencing the same problem giving a chance to snap may solve the issue as it did for me
I'm developing a django project for use in America, specifically the New York timezone and the system is hosted on AWS, with SES sending email.
The email backend is using django-anymail which is a simple wrapper for SES and the system uses send_mail from django core.
To support this I've opted for the following Django settings;
EMAIL_BACKEND = "anymail.backends.amazon_ses.EmailBackend"
LANGUAGE_CODE = 'en'
TIME_ZONE = 'America/New_York'
USE_I18N = False
USE_L10N = True
USE_TZ = True
ANYMAIL = {
"AMAZON_SES_CLIENT_PARAMS": {
"region_name": AWS_SES_REGION_NAME,
},
}
With the above settings django calls tzset() on startup which modifies the system timezone. This then means the timestamp used by botocore to sign the requests for SES is not UTC, because the following error is received from message sending;
An error occurred (ExpiredToken) when calling the SendRawEmail operation: The security token included in the request is expired
Emails are sent successfully by changing settings to TIME_ZONE = 'UTC'.
I can only assume that the requests are being signed in UTC -4 which then hit AWS which is in UTC.
How can django run in a specific timezone, but boto operate with UTC timestamps?
The system is running in a docker container (pre-production);
docker compose 3.4 (unix host)
python 2.7
django 1.11
django-anymail 3.0
LocaleMiddleware is loaded
I'm not able to reproduce the error you're seeing with the settings you've described, but I can show you what is working correctly for me with extra logging, and you could compare that to your failing case to try to see what's different.
I ran this code in the Django shell (python manage.py shell) just for convenience, but you could put it in a debugging view or anywhere else that works for you.
Our working theory is that boto is using the wrong time zone to calculate timestamps for signing the API request, so let's enable some detailed boto3 logging that covers that area:
import boto3
boto3.set_stream_logger('botocore.auth') # log the signature logic
boto3.set_stream_logger('botocore.endpoint') # log the API request
# boto3.set_stream_logger('botocore.parsers') # log the API response (if you want)
Now try to send a message:
from django.core.mail import send_mail
send_mail("Test", "testing", None, ['success#simulator.amazonses.com'])
You should see log output that looks something like this:
2019-03-19 20:48:32,321 botocore.endpoint [DEBUG] Setting email timeout as (60, 60)
2019-03-19 20:48:32,580 botocore.endpoint [DEBUG] Making request for OperationModel(name=SendRawEmail) with params: {'body': {'Action': u'SendRawEmail', 'Version': u'2010-12-01', 'RawMessage.Data': [base64 message omitted]'}, 'url': u'https://email.us-east-1.amazonaws.com/', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.9.117 Python/2.7.15 Darwin/18.2.0 Botocore/1.12.117 django-anymail/3.0-amazon-ses'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x10dadd1d0>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2019-03-19 20:48:32,581 botocore.auth [DEBUG] Calculating signature using v4 auth.
2019-03-19 20:48:32,581 botocore.auth [DEBUG] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:email.us-east-1.amazonaws.com
x-amz-date:20190320T064832Z
content-type;host;x-amz-date
[redacted]
2019-03-19 20:48:32,582 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20190320T064832Z
20190320/us-east-1/ses/aws4_request
[redacted]
2019-03-19 20:48:32,582 botocore.auth [DEBUG] Signature:
[redacted]
2019-03-19 20:48:32,582 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://email.us-east-1.amazonaws.com/, headers={'Content-Length': '437', 'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'Authorization': 'AWS4-HMAC-SHA256 Credential=[key id redacted]/20190320/us-east-1/ses/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=[redacted]', 'X-Amz-Date': '20190320T064832Z', 'User-Agent': 'Boto3/1.9.117 Python/2.7.15 Darwin/18.2.0 Botocore/1.12.117 django-anymail/3.0-amazon-ses'}>
The important parts here are the dates:
2019-03-19 20:48:32,581 botocore.auth [DEBUG] CanonicalRequest:
...
x-amz-date:20190320T064832Z
2019-03-19 20:48:32,582 botocore.auth [DEBUG] StringToSign:
...
20190320T064832Z
20190320/...
2019-03-19 20:48:32,582 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest ...
headers={
'Authorization': '.../20190320/...',
'X-Amz-Date': '20190320T064832Z', ...}>
Notice the signature calculations are all based on the UTC date (2019-03-20)—not the current local date in my Django timezone (2019-03-19).
So it looks like boto3 does use UTC for the signature calculations, despite the Django/environment time zone. And indeed, the send works for me without error.
So the question is, what's different when you see the problem?
What is the x-amz-date in the CanonicalRequest?
Is that, in fact, the actual UTC datetime when you send the message?
(If not, the clock in your Docker container might be way off.)
Does that same date appear again correctly in the StringToSign, both as a full timestamp and a truncated date?
And does it appear again in the AWSPreparedRequest headers, both Authorization and X-Amz-Date? (If you see a Date header instead of X-Amz-Date, that would also be interesting.)
Hope that helps you either get a little closer to a solution, or at least figure out what detail is essential to reproducing the problem.
I'm having troubles starting a Celery worker on Elastic Beanstalk instance. After a couple of seconds, it just quits unexpectedly with no error. The instance should have enough RAM. I'm attaching the output by worker with log level debug (sensitive information replaced by **). Any guidance would be super helpful. Thanks.
(venv) [ec2-user#ip-** app]$ celery -A app worker -l debug
[12/Mar/2018 10:18:29] DEBUG [raven.contrib.django.client.DjangoClient:265] Configuring Raven for host: <raven.conf.remote.RemoteConfig object at 0x7f556309e940>
-------------- celery#ip-** v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.9.75-25.55.amzn1.x86_64-x86_64-with-glibc2.3.4 2018-03-12 10:18:29
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: app:0x7f556998edd8
- ** ---------- .> transport: sqs://**:**#localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. common.tasks.send_templated_email
. orders.tasks.import_orders_from_all_companies
[2018-03-12 10:18:29,783: DEBUG/MainProcess] Setting config variable for region to 'eu-central-1'
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable config_file from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable credentials_file from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable data_path from defaults.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable region from instance vars with value 'eu-central-1'.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable ca_bundle from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading variable api_versions from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/endpoints.json
[2018-03-12 10:18:29,790: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,790: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f55633fe048>
[2018-03-12 10:18:29,793: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2018-03-12 10:18:29,796: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f5563448d90>
[2018-03-12 10:18:29,797: DEBUG/MainProcess] The s3 config key is not a dictionary type, ignoring its value of: None
[2018-03-12 10:18:29,802: DEBUG/MainProcess] Setting sqs timeout as (60, 60)
[2018-03-12 10:18:29,802: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/_retry.json
[2018-03-12 10:18:29,803: DEBUG/MainProcess] Registering retry handlers for service: sqs
[2018-03-12 10:18:29,803: DEBUG/MainProcess] Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Making request for OperationModel(name=ListQueues) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'ListQueues', 'Version': '2012-11-05', 'QueueNamePrefix': ''}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5562069f28>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5562069cf8>>
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,805: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,805: DEBUG/MainProcess] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z
content-type;host;x-amz-date
**
[2018-03-12 10:18:29,805: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,806: DEBUG/MainProcess] Signature:
**
[2018-03-12 10:18:29,806: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,807: INFO/MainProcess] Starting new HTTPS connection (1): eu-central-1.queue.amazonaws.com
[2018-03-12 10:18:29,839: DEBUG/MainProcess] "POST / HTTP/1.1" 200 409
[2018-03-12 10:18:29,840: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '409', 'connection': 'keep-alive', 'x-amzn-requestid': '**'}
[2018-03-12 10:18:29,840: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult><QueueUrl>**</QueueUrl><QueueUrl>**</QueueUrl></ListQueuesResult><ResponseMetadata><RequestId>**</RequestId></ResponseMetadata></ListQueuesResponse>'
[2018-03-12 10:18:29,841: DEBUG/MainProcess] Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f556201c390>
[2018-03-12 10:18:29,841: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,841: INFO/MainProcess] Connected to sqs://**:**#localhost//
[2018-03-12 10:18:29,850: DEBUG/MainProcess] Setting config variable for region to 'eu-central-1'
[2018-03-12 10:18:29,850: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable config_file from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable credentials_file from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable data_path from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable region from instance vars with value 'eu-central-1'.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable ca_bundle from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable api_versions from defaults.
[2018-03-12 10:18:29,853: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/endpoints.json
[2018-03-12 10:18:29,857: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,857: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f55633fe048>
[2018-03-12 10:18:29,861: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2018-03-12 10:18:29,863: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f5563448d90>
[2018-03-12 10:18:29,863: DEBUG/MainProcess] The s3 config key is not a dictionary type, ignoring its value of: None
[2018-03-12 10:18:29,865: DEBUG/MainProcess] Setting sqs timeout as (60, 60)
[2018-03-12 10:18:29,865: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/_retry.json
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Registering retry handlers for service: sqs
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Making request for OperationModel(name=ListQueues) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'ListQueues', 'Version': '2012-11-05', 'QueueNamePrefix': ''}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5561acfbe0>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5561acfb38>>
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,868: DEBUG/MainProcess] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z
content-type;host;x-amz-date
**
[2018-03-12 10:18:29,868: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,868: DEBUG/MainProcess] Signature:
**
[2018-03-12 10:18:29,869: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,869: INFO/MainProcess] Starting new HTTPS connection (1): eu-central-1.queue.amazonaws.com
[2018-03-12 10:18:29,895: DEBUG/MainProcess] "POST / HTTP/1.1" 200 409
[2018-03-12 10:18:29,895: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '409', 'connection': 'keep-alive', 'x-amzn-requestid': '5b132733-22e0-5567-8762-74136ac526ec'}
[2018-03-12 10:18:29,896: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult><QueueUrl>**</QueueUrl><QueueUrl>**</QueueUrl></ListQueuesResult><ResponseMetadata><RequestId>5b132733-22e0-5567-8762-74136ac526ec</RequestId></ResponseMetadata></ListQueuesResponse>'
[2018-03-12 10:18:29,896: DEBUG/MainProcess] Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f5561a763c8>
[2018-03-12 10:18:29,896: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,899: DEBUG/MainProcess] Event before-parameter-build.sqs.GetQueueAttributes: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Making request for OperationModel(name=GetQueueAttributes) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'GetQueueAttributes', 'Version': '2012-11-05', 'QueueUrl': '**', 'AttributeName.1': 'ApproximateNumberOfMessages'}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5562069f28>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Event request-created.sqs.GetQueueAttributes: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5562069cf8>>
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Event choose-signer.sqs.GetQueueAttributes: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,901: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,901: DEBUG/MainProcess] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z
content-type;host;x-amz-date
**
[2018-03-12 10:18:29,901: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,901: DEBUG/MainProcess] Signature:
9fb0d1ad68b5d25bf148cc11857b1e1083418557229ca2c47e8b525b54880b74
[2018-03-12 10:18:29,902: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,910: DEBUG/MainProcess] "POST / HTTP/1.1" 200 357
[2018-03-12 10:18:29,911: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '357', 'connection': 'keep-alive', 'x-amzn-requestid': '9aa38f1d-25e5-576d-a9a9-dc3d6dc029a0'}
[2018-03-12 10:18:29,911: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><GetQueueAttributesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><GetQueueAttributesResult><Attribute><Name>ApproximateNumberOfMessages</Name><Value>0</Value></Attribute></GetQueueAttributesResult><ResponseMetadata><RequestId>**</RequestId></ResponseMetadata></GetQueueAttributesResponse>'
[2018-03-12 10:18:29,912: DEBUG/MainProcess] Event needs-retry.sqs.GetQueueAttributes: calling handler <botocore.retryhandler.RetryHandler object at 0x7f556201c390>
[2018-03-12 10:18:29,912: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,921: DEBUG/MainProcess] Canceling task consumer...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] Canceling task consumer...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] Closing consumer channel...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
I have solved the issue. Amazon instance required PyCurl and some additional packages to connect properly with SQS.
I suggest that you already have a script file to run celery daemon(run_supervised_celeryd.sh)
Here is my EB config file:
packages:
yum:
libjpeg-turbo-devel: []
libpng-devel: []
libcurl-devel: []
container_commands:
01_migrate:
command: "django-admin.py migrate --noinput"
leader_only: true
02_collectstatic:
command: "python manage.py collectstatic --noinput"
03_pycurl:
command: 'source /opt/python/run/venv/bin/activate && pip3 install /usr/local/share/pycurl-7.43.0.tar.gz --global-option="--with-nss" --upgrade'
04_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
files:
"/usr/local/share/pycurl-7.43.0.tar.gz" :
mode: "000644"
owner: root
group: root
source: https://pypi.python.org/packages/source/p/pycurl/pycurl-7.43.0.tar.gz
add env variable PYCURL_SSL_LIBRARY="nss"
All listed settings solve the issue
I am using celery 3.1.9 with sqs. I have worker common_w running as daemon. It works with common queue on sqs.
A worker unexpectedly stop processing tasks. No exceptions and errors.
Last logs with option -l DEBUG:
[2014-09-03 21:01:14,766: DEBUG/MainProcess] Method: GET
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Path: /684818426251/dev_common_w_ip-10-84-163-209-celery-pidbox
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Data:
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Headers: {}
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Host: eu-west-1.queue.amazonaws.com
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Port: 443
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Params: {'Action': 'ReceiveMessage', 'Version': '2012-11-05', 'MaxNumberOfMessages': 10}
[2014-09-03 21:01:14,767: DEBUG/MainProcess] Token: None
[2014-09-03 21:01:14,767: DEBUG/MainProcess] CanonicalRequest:
GET
/684818426251/dev_common_w_ip-10-84-163-209-celery-pidbox
Action=ReceiveMessage&MaxNumberOfMessages=10&Version=2012-11-05
host:eu-west-1.queue.amazonaws.com
x-amz-date:20140903T170114Z
host;x-amz-date
e3b0c44298fc1c149afbf4c899sdfasf32wefwef49b934ca495991b7852b855
[2014-09-03 21:01:14,768: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20140903T170114Z
20140903/eu-west-1/sqs/aws4_request
9a9761b49ba9a06e469bwkfj48u83yghkhejwejlr8fce8eb078ac8c4c9ffd9e
[2014-09-03 21:01:14,768: DEBUG/MainProcess] Signature:
2de3c082bc6f01f5d5ecd66b6r89283ryuu8j8rrdaf0c40eba6cc0ceb62df6e
[2014-09-03 21:01:14,824: DEBUG/MainProcess] <?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult/><ResponseMetadata><RequestId>4712119a-38ca-51b4-bfd1-5d1f8fu8uc4</RequestId></ResponseMetadata></ReceiveMessageResponse>
[2014-09-03 21:01:14,824: INFO/MainProcess] Received task: skazka.sender.tasks.wait_action_worker[f84c52fe-8748-4c81-b718-f23f23fasdgbg34g]
[2014-09-03 21:01:14,824: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x21666e0> (args:('skazka.sender.tasks.wait_action_worker', 'f84c52fe-8748-4c81-b718-f23f23fasdgbg34g', (1967L,), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (1967L,), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'common', u'exchange': u'common'}, 'expires': None, u'hostname': 'common_w#ip-10-84-163-209', 'task': 'skazka.sender.tasks.wait_action_worker', 'callbacks': None, u'correlation_id': u'f84c52fe-8748-4c81-b718-f23f23fasdgbg34g', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'07a91182-23a6-3afb-b3cc-70a2fa3fw333', 'id': 'f84c52fe-8748-4c81-b718-f23f23fasdgbg34g', u'headers': {}}) kwargs:{})
UPDATE:
strace says:
futex(0x7ffff79b3e00, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
enter code here
Then I set in the config:
CELERYD_FORCE_EXECV = True
Until it works fine...