Private API gateway with IAM authentication not liking my security token - amazon-web-services

Question

I have a private API gateway with a / endpoint and a /foo with IAM auth enabled.
I created a policy which I attached to my instance's role :
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"execute-api:Invoke"
],
"Resource": [
"arn:aws:execute-api:*:*:*"
]
}
]
}
I have some code to do the AWS signature stuff and also used Postman to create a snippet with the same key/secret/session token. Both give the same result on /foo. It always says :
{"message":"The security token included in the request is invalid"}
I had a concern that the docs do not say you can attach the policy to a role only a user or group.
https://aws.amazon.com/premiumsupport/knowledge-center/iam-authentication-api-gateway/
That whole page doesn't mention roles once... Can I attach a policy to a role to use it with IAM auth'ed API gateway??
The / endpoint returns me a 200 response, and my API GW resource policy denies/allows access to *. If I can get to /, I can get to /foo. (And if I disable the IAM auth, I can get /foo)
The VPC endpoint allows * on *.
In the execution logs I see nothing for the failed attempts.
The attempts to / log : API Key authorized because method 'GET /' does not require API Key. Request will not contribute to throttle or quota limits
And I can see the X-Amz-Security-Token in the payload.
But requests to /foo don't appear there, only the access logs. And I've added some fields but nothing that sheds any light on the problem.
Anything I'm forgetting?? And any ideas why isn't it working?
Here is my signing python, there may be a bug, but it is getting the same error as Postman, which makes me think not! Replace the host/endpoint and path to your own. I added a few print debug lines to show the intermediate steps because I did get some errors about the canonical URL being wrong to start with.
#!/usr/bin/python3
# This is based on the AWS General Reference
# Signing AWS API Requests top available at
# https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
# To use it :
# pip3 install requests
# To use it on instance with instance role :
# sudo yum -y -q install jq
# export AWS_ACCESS_KEY_ID=$(curl 169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance | jq -r .AccessKeyId)
# export AWS_SECRET_ACCESS_KEY=$(curl 169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance | jq -r .SecretAccessKey)
# export AWS_SESSION_TOKEN=$(curl 169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance | jq -r .Token)
# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac
import requests # pip install requests
# ************* REQUEST VALUES *************
method = 'GET'
service = 'execute-api'
host = 'xxx.execute-api.eu-west-2.amazonaws.com'
region = 'eu-west-2'
endpoint = 'https://xxx.execute-api.eu-west-2.amazonaws.com'
path='/stage/foo/'
request_parameters = ''
# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning
# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
session_token = os.environ.get('AWS_SESSION_TOKEN')
if access_key is None or secret_key is None:
print('No access key is available.')
sys.exit()
# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope
# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
# Step 1 is to define the verb (GET, POST, etc.)--already done.
# Step 2: Create canonical URI--the part of the URI from domain to query
# string (use '/' if no path)
canonical_uri = path
# Step 3: Create the canonical query string. In this example (a GET request),
# request parameters are in the query string. Query string values must
# be URL-encoded (space=%20). The parameters must be sorted by name.
# For this example, the query string is pre-formatted in the request_parameters variable.
canonical_querystring = request_parameters
# Step 4: Create the canonical headers and signed headers. Header names
# must be trimmed and lowercase, and sorted in code point order from
# low to high. Note that there is a trailing \n.
canonical_headers = 'host:' + host + '\n' + 'x-amz-date:' + amzdate + '\n' 'x-amz-security-token:' + session_token + '\n'
# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers lists those that you want to be included in the
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = 'host;x-amz-date;x-amz-security-token'
# Step 6: Create payload hash (hash of the request body content). For GET
# requests, the payload is an empty string ("").
payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()
# Step 7: Combine elements to create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
print ("CANONICAL REQUEST : " + canonical_request)
print ()
# ************* TASK 2: CREATE THE STRING TO SIGN*************
# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + '\n' + amzdate + '\n' + credential_scope + '\n' + hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
print ("STRING TO SIGN : " + string_to_sign )
# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, datestamp, region, service)
# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()
# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# The signing information can be either in a query string value or in
# a header named Authorization. This code shows how to use a header.
# Create authorization header and add to request headers
authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' + 'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
# The request can include any headers, but MUST include "host", "x-amz-date",
# and (for this scenario) "Authorization". "host" and "x-amz-date" must
# be included in the canonical_headers and signed_headers, as noted
# earlier. Order here is not significant.
# Python note: The 'host' header is added automatically by the Python 'requests' library.
headers = {'x-amz-date':amzdate, 'Authorization':authorization_header, 'X-Amz-Security-Token':session_token}
# ************* SEND THE REQUEST *************
request_url = endpoint + path + canonical_querystring
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
print('Headers = ' + str(headers))
r = requests.get(request_url, headers=headers)
print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)

Answer 1

Aha, I found an answer. I was pulling the credentials from the wrong endpoint. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html shows "identity-credentials/ec2/security-credentials/ec2-instance" are "Internal use only". There is an iam/security-credentials/{role} that works a LOT better!!