I have an GET based API gateway set up pointing to a Lambda with Lambda Proxy integration enabled
The API has AWS IAM as the auth method.
On my local, I have AWS Auth setup with temp session token
The following works without issue
curl -s GET "https://<ID>.execute-api.us-west-2.amazonaws.com/dev" \
--header "x-amz-security-token: ${SESSION_TOKEN}" \
--user $ACCESS_KEY:$SECRET_KEY \
--aws-sigv4 "aws:amz:us-west-2:execute-api" | jq .
But when I add query params to the url, it fails
curl -s GET "https://<ID>.execute-api.us-west-2.amazonaws.com/dev?a=${v1}&b=${v2}" \
--header "x-amz-security-token: ${SESSION_TOKEN}" \
--user $ACCESS_KEY:$SECRET_KEY \
--aws-sigv4 "aws:amz:us-west-2:execute-api" | jq .
This is the response that I get is
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/dev\nb=def&a=abc\nhost:<ID>.execute-api.us-west-2.amazonaws.com\nx-amz-date:20230104T112344Z\n\nhost;x-amz-date\<date-token>'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20230104T112344Z\n20230104/us-west-2/execute-api/aws4_request\<token>'\n"
}
Looks like I need to add the query params part to the signature part. How do I do that ? Or is there something else that I'm missing ?
I am unable to successfully acquire an id token/access token from my AWS cognito user pool when I supply an auth code. I have written a shell script (see below), and receive invalid_grant back from the server.
I have encoded the base64 Authorization Basic header for client_id:client_secret generated with python as:
import base64
encode='my_client_id_string:my_client_secret_string'
base64.b64encode(encode)
#!/usr/bin/env sh
curl --location --request POST 'https://<domain>.auth.us-east-2.amazoncognito.com/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic <base64 encode string client_id:client_secret>' \
--data-urlencode 'grant_type=authorization_code' \
--data-urlencode 'client_id=<client_id from app settings' \
--data-urlencode 'code=<code received from redirect url to my localhost app endpoint>' \
--data-urlencode 'redirect_uri=http://localhost:8000/my_redirect'
Any ideas?
Solved it!
The problem was caused by an invalid client id. I had supplied a typo for the client id value!
I am trying to create a bash script to upload files to my s3 bucket. I am having difficulty generating the correct signature.
I get the following error message:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
Here is my script:
Thanks for your help!
#!/usr/bin/env bash
#upload to S3 bucket
sourceFilePath="$1"
#file path at S3
folderPathAtS3="packages";
#S3 bucket region
region="eu-central-1"
#S3 bucket name
bucket="my-bucket-name";
#S3 HTTP Resource URL for your file
resource="/${bucket}/${folderPathAtS3}";
#set content type
contentType="gzip";
#get date as RFC 7231 format
dateValue="$(date +'%a, %d %b %Y %H:%M:%S %z')"
acl="x-amz-acl:private"
#String to generate signature
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${acl}\n${resource}";
#S3 key
s3Key="my-key";
#S3 secret
s3Secret="my-secret-code";
#Generate signature, Amazon re-calculates the signature and compares if it matches the one that was contained in your request. That way the secret access key never needs to be transmitted over the network.
signature=$(echo -en "${stringToSign}" | openssl sha1 -hmac ${s3Secret} -binary | base64);
#Curl to make PUT request.
curl -L -X PUT -T "${sourceFilePath}" \
-H "Host: ${bucket}.${region}.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "$acl" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://s3.amazonaws.com/${bucket}/${folderPathAtS3}
Your signature seems fine, but your request is wrong and consequently does not match.
-H "Host: ${bucket}.${region}.amazonaws.com" \ is incorrect.
The correct value is ${bucket}.s3 ${region}.amazonaws.com. You're overlooking the s3. in the hostname... but even if correct, this is still invalidj because your URL https://s3.amazonaws.com/${bucket}/... also includes the bucket, which means your bucket name is being implicitly added to the beginning of the object key because it appears twice.
Additionally, https://s3.amazonaws.com is us-east-1. To connect to the correct region, your URL needs to be one of these variants:
https://${region}.s3.amazonaws.com/${bucket}/${folderPathAtS3}
https://${bucket}.${region}.s3.amazonaws.com/${folderPathAtS3}
https://${bucket}.s3.amazonaws.com/${folderPathAtS3}
Use one of these formats, and eliminate -H "Host: ..." because it will then be redundant.
The last of the 3 URL formats will only start to work after the bucket is more than a few minutes or hours old. S3 creates these automatically but it takes some time.
We have an Amazon SES setup that works well and sends thousands of emails a day via SMTP. Trying to follow a best practice of "rotating" access keys we went to
https://console.aws.amazon.com/iam/home and creating a new access key for the exact same user which is used to send emails. The new key is supposedly active but when trying to email with the access keys, we keep getting
535 Authentication Credentials Invalid
Switching to the old access keys works well and emails are sent. Tried a couple of times to delete the new access keys and create others. Same machine, same software. We have proper copy+paste skills to ensure we're using the same ID/Password provided in the CSV coming from Amazon. Here the dialog from Amazon:
So what's going on? Is there a time limit till the new key becomes active? Is there some other hidden limitation somewhere?
You are confusing the SMTP credentials with access_key and secret. They are different.
access_key/secret --> Use in SDK and CLI
SMTP credentials --> Use to configure SES SMTP
You are creating a new access_key/secret and using it as SMTP credentials
Instead you create a new SMTP credentials and use it
Key rotation is different from SMTP credential rotation
No need to create a new user
It is likely you are using the SMTP credentials that does not change even if you generate another set of access_key/secret. In your case it looks like you are using the SMTP server and not the SDK. So generating a new set of access_key/secret has no effect on SMTP credentials.
If you want to create a new set of SMTP credentials, go to AWS SES dashboard and create SMTP credentials.
For more information: Obtaining Your Amazon SES SMTP Credentials
Yes, there's a hidden limitation in the way AWS approaches the SMTP password for SES. And they are using a very confusing way of handling these credentials.
The answer from helloV is on the right track, but it's not entirely correct. Both AWS and his answer tell us that Access_key/Secret_key and SES SMTP credentials are different things, but:
If you create fresh SES SMTP credentials, it creates a new IAM User with an Access Key/Secret Key pair
The Access Key Id is the same as the username for SMTP
If you delete or disable this key, you lose your SMTP access. So they are clearly very related.
The password for SMTP is derived from the Secret Key
It turns out that a new access_key/secret_key pair on an existing IAM user, can be used for SMTP, and therefore keys can be rotated without creating new users.
AWS converts the Secret Access Key to generate the SMTP password, as they explain in this documentation page:
The following pseudocode shows the algorithm that converts an AWS Secret Access Key to an Amazon SES SMTP password.
key = AWS Secret Access Key;
message = "SendRawEmail";
versionInBytes = 0x02;
signatureInBytes = HmacSha256(message, key);
signatureAndVer = Concatenate(versionInBytes, signatureInBytes);
smtpPassword = Base64(signatureAndVer);
So using the Secret Access key, the SMTP password can be generated
With bash and openssl installed, the following command will output the password for use in SMTP:
(echo -en "\x02"; echo -n 'SendRawEmail' \
| openssl dgst -sha256 -hmac $AWS_SECRET_ACCESS_KEY -binary) \
| openssl enc -base64
Just replace $AWS_SECRET_ACCESS_KEY with your key, or set the variable beforehand
Since, both Secret Key and SMTP Password are in different format, you need to convert Secret Key to SMTP Password using algorithm provided by AWS.
You can find it here:
https://aws.amazon.com/premiumsupport/knowledge-center/ses-rotate-smtp-access-keys/
Here is a working piece of code for transformning your secret key to smtp password using bash :
#!/usr/bin/env bash
# Convert AWS Secret Access Key to an Amazon SES SMTP password
# using the following pseudocode:
#
# date = "11111111";
# service = "ses";
# terminal = "aws4_request";
# message = "SendRawEmail";
# version = 0x04;
#
# kDate = HmacSha256(date, "AWS4" + key);
# kRegion = HmacSha256(region, kDate);
# kService = HmacSha256(service, kRegion);
# kTerminal = HmacSha256(terminal, kService);
# kMessage = HmacSha256(message, kTerminal);
# signatureAndVersion = Concatenate(version, kMessage);
# smtpPassword = Base64(signatureAndVersion);
#
# Usage:
# chmod u+x aws-ses-smtp-password.sh
# ./aws-ses-smtp-password.sh secret-key-here
# See: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html
#
if [ "$#" -ne 1 ]; then
echo "Usage: ./aws-ses-smtp-password.sh secret-key-here"
exit 1
fi
KEY="${1}"
DATE="11111111"
REGION="eu-west-1"
SERVICE="ses"
TERMINAL="aws4_request"
MESSAGE="SendRawEmail"
VERSION="4"
VERSION_IN_BYTES=$(printf \\$(printf '%03o' "${VERSION}"));
#SIGNATURE_IN_BYTES=$(echo -n "${MESSAGE}" | openssl dgst -sha256 -hmac "${KEY}" -binary);
SIGNATURE_IN_BYTES=$(echo -n "${DATE}" | openssl dgst -sha256 -mac HMAC -macopt "key:AWS4${KEY}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${REGION}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${SERVICE}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${TERMINAL}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${MESSAGE}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" -binary | sed 's/^.* //');
SIGNATURE_AND_VERSION="${VERSION_IN_BYTES}${SIGNATURE_IN_BYTES}"
SMTP_PASSWORD=$(echo -n "${SIGNATURE_AND_VERSION}" | base64);
echo "${SMTP_PASSWORD}"
Curl successfully uploads the file to S3 using a signed url:
curl -v -k -X PUT \
-H "x-amz-server-side-encryption: AES256" \
-H "Content-Type: application/pdf" \
-T "__tests__/resources/test.pdf" \
"http://mybucket.s3.amazonaws.com/test.pdf?AWSAccessKeyId=IDKEY&Expires=1489458783&Signature=SIGNATURE
I've tried replicating this in Grails using the REST client plugin:
String url = "http://mybucket.s3.amazonaws.com/test.pdf?AWSAccessKeyId=IDKEY&Expires=1489458783&Signature=SIGNATURE"
RestResponse resp = rest.put(url){
header "x-amz-server-side-encryption", "AES256"
header "Content-Type", "application/pdf"
body pdf
}
But Amazon rejects the upload, saying the arguments are incorrect...probably due to the pdf being sent as a "body" parameter. Any ideas?
Instead of using a rest client to upload it would be simpler to use the AWS Java SDK in your Grails app.
See an example here of using a pre-signed url to upload http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObjectJavaSDK.html