Amazon SES with Django not in UTC timezone - django

Question

I'm developing a django project for use in America, specifically the New York timezone and the system is hosted on AWS, with SES sending email.
The email backend is using django-anymail which is a simple wrapper for SES and the system uses send_mail from django core.
To support this I've opted for the following Django settings;
EMAIL_BACKEND = "anymail.backends.amazon_ses.EmailBackend"
LANGUAGE_CODE = 'en'
TIME_ZONE = 'America/New_York'
USE_I18N = False
USE_L10N = True
USE_TZ = True
ANYMAIL = {
"AMAZON_SES_CLIENT_PARAMS": {
"region_name": AWS_SES_REGION_NAME,
},
}
With the above settings django calls tzset() on startup which modifies the system timezone. This then means the timestamp used by botocore to sign the requests for SES is not UTC, because the following error is received from message sending;
An error occurred (ExpiredToken) when calling the SendRawEmail operation: The security token included in the request is expired
Emails are sent successfully by changing settings to TIME_ZONE = 'UTC'.
I can only assume that the requests are being signed in UTC -4 which then hit AWS which is in UTC.
How can django run in a specific timezone, but boto operate with UTC timestamps?
The system is running in a docker container (pre-production);
docker compose 3.4 (unix host)
python 2.7
django 1.11
django-anymail 3.0
LocaleMiddleware is loaded

Answer 1

I'm not able to reproduce the error you're seeing with the settings you've described, but I can show you what is working correctly for me with extra logging, and you could compare that to your failing case to try to see what's different.
I ran this code in the Django shell (python manage.py shell) just for convenience, but you could put it in a debugging view or anywhere else that works for you.
Our working theory is that boto is using the wrong time zone to calculate timestamps for signing the API request, so let's enable some detailed boto3 logging that covers that area:
import boto3
boto3.set_stream_logger('botocore.auth') # log the signature logic
boto3.set_stream_logger('botocore.endpoint') # log the API request
# boto3.set_stream_logger('botocore.parsers') # log the API response (if you want)
Now try to send a message:
from django.core.mail import send_mail
send_mail("Test", "testing", None, ['success#simulator.amazonses.com'])
You should see log output that looks something like this:
2019-03-19 20:48:32,321 botocore.endpoint [DEBUG] Setting email timeout as (60, 60)
2019-03-19 20:48:32,580 botocore.endpoint [DEBUG] Making request for OperationModel(name=SendRawEmail) with params: {'body': {'Action': u'SendRawEmail', 'Version': u'2010-12-01', 'RawMessage.Data': [base64 message omitted]'}, 'url': u'https://email.us-east-1.amazonaws.com/', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.9.117 Python/2.7.15 Darwin/18.2.0 Botocore/1.12.117 django-anymail/3.0-amazon-ses'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x10dadd1d0>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2019-03-19 20:48:32,581 botocore.auth [DEBUG] Calculating signature using v4 auth.
2019-03-19 20:48:32,581 botocore.auth [DEBUG] CanonicalRequest:
POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:email.us-east-1.amazonaws.com
x-amz-date:20190320T064832Z
content-type;host;x-amz-date
[redacted]
2019-03-19 20:48:32,582 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20190320T064832Z
20190320/us-east-1/ses/aws4_request
[redacted]
2019-03-19 20:48:32,582 botocore.auth [DEBUG] Signature:
[redacted]
2019-03-19 20:48:32,582 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://email.us-east-1.amazonaws.com/, headers={'Content-Length': '437', 'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'Authorization': 'AWS4-HMAC-SHA256 Credential=[key id redacted]/20190320/us-east-1/ses/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=[redacted]', 'X-Amz-Date': '20190320T064832Z', 'User-Agent': 'Boto3/1.9.117 Python/2.7.15 Darwin/18.2.0 Botocore/1.12.117 django-anymail/3.0-amazon-ses'}>
The important parts here are the dates:
2019-03-19 20:48:32,581 botocore.auth [DEBUG] CanonicalRequest:
...
x-amz-date:20190320T064832Z
2019-03-19 20:48:32,582 botocore.auth [DEBUG] StringToSign:
...
20190320T064832Z
20190320/...
2019-03-19 20:48:32,582 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest ...
headers={
'Authorization': '.../20190320/...',
'X-Amz-Date': '20190320T064832Z', ...}>
Notice the signature calculations are all based on the UTC date (2019-03-20)—not the current local date in my Django timezone (2019-03-19).
So it looks like boto3 does use UTC for the signature calculations, despite the Django/environment time zone. And indeed, the send works for me without error.
So the question is, what's different when you see the problem?
What is the x-amz-date in the CanonicalRequest?
Is that, in fact, the actual UTC datetime when you send the message?
(If not, the clock in your Docker container might be way off.)
Does that same date appear again correctly in the StringToSign, both as a full timestamp and a truncated date?
And does it appear again in the AWSPreparedRequest headers, both Authorization and X-Amz-Date? (If you see a Date header instead of X-Amz-Date, that would also be interesting.)
Hope that helps you either get a little closer to a solution, or at least figure out what detail is essential to reproducing the problem.