Goal: Generate Signed-URL Using OAuth2.0 Access Token
The examples and source codes I find for signing Google Cloud Storage blobs all require service account credentials file (the private key to be specific). For instance:
https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#storage-signed-url-get-object-python
However, since I follow the authorization flow discussed here, I only have OAuth2.0 access token (and I do NOT have the credentials file and private key of a service account with access to GCS bucket/object). Hence, I was wondering how I can sign blobs using OAuth2.0 access tokens.
The Code Used:
I use the following to sign blob:
# First, get access token:
service_account = "<email address of a service account>"
access_token = build(
serviceName='iamcredentials',
version='v1',
http=http
).projects().serviceAccounts().generateAccessToken(
name="projects/{}/serviceAccounts/{}".format(
"-",
service_account),
body=body
).execute()["accessToken"]
credentials = AccessTokenCredentials(access_token, "MyAgent/1.0", None)
# Second, use the access token to sign a blob
url = "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{}:signBlob".format(service_account)
encoded = base64.b64encode(blob)
sign_blob_request_body = {"payload": encoded}
response = requests.post(url,
data=json.dumps(sign_blob_request_body),
headers={
'Content-Type': 'application/json',
'Authorization': 'Bearer {}'.format(credentials.access_token)})
signature = response.json()["signedBlob"]
# Third, use the signature to create signed URL:
encoded_signature = base64.b64encode(signature)
signed_url = "https://storage.googleapis.com/<BUCKET>/<OBJECT>?" \
"GoogleAccessId={}&" \
"Expires={}&" \
"Signature={}".format(service_account,
expiration,
encoded_signature)
The Error Message Received:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
</Message>
<StringToSign>GET 1561832204 /<BUCKET>/<OBJECT></StringToSign>
</Error>
In case you do NOT want to use API secret key, follow procedure described in this sample that is using iamcredentials.signBlob() API signing URL 'remotely' for a service account with no need to distribute API secret key.
Signature string (that has to be signed) has this format:
signature_string = ('{verb}\n'
'{content_md5}\n'
'{content_type}\n'
'{expiration}\n'
'{resource}')
Related
This probably makes no sense.
I am using passport.js to authenticate users in the app using a JWT token.
This is how the token is generated:
jwt.sign(
payload,
keys.SecretKey,
{ expiresIn: 3600 * 24 * 356 },
(error, token) => {
res.json({
success: true,
token: "Bearer " + token,
});
}
);
And then in each http request, I add the token in the Authorization header.
I'm trying to limit access to AWS S3 images only to logged-in users.
One way to do this is to generated signed url each time a user tries to access an image like I am trying to do here which is not perfect (at least so far).
So I wonder, if I can do something, maybe use an S3 policy that would allow me to use the JWT token to give access to authenticated users access to S3 resources.
I am trying to make a JWT call to storage API using the example listed here with some changes as below -
def generate_jwt():
"""Generates a signed JSON Web Token using a Google API Service Account."""
now = int(time.time())
sa_email = os.environ["FUNCTION_IDENTITY"]
expiry_length = 3600
# build payload
payload = {
'iat': now,
# expires after 'expiry_length' seconds.
"exp": now + expiry_length,
# iss must match 'issuer' in the security configuration in your
# swagger spec (e.g. service account email). It can be any string.
'iss': sa_email,
# aud must be either your Endpoints service name, or match the value
# specified as the 'x-google-audience' in the OpenAPI document.
'aud': "https://storage.googleapis.com",
# sub and email should match the service account's email address
'sub': sa_email,
'email': sa_email
}
# sign with keyfile
sa_keyfile="cred.json"
signer = google.auth.crypt.RSASigner.from_service_account_file(sa_keyfile)
jwt = google.auth.jwt.encode(signer, payload)
return jwt
and calliing it here
def make_jwt_request(signed_jwt, url="https://storage.googleapis.com/storage/v1/b/BUCKET_NAME"):
"""Makes an authorized request to the endpoint"""
headers = {
'Authorization': 'Bearer {}'.format(signed_jwt.decode('utf-8')),
'content-type': 'application/json',
"Host": "www.googleapis.com",
}
response = requests.get(url, headers=headers)
print(response.status_code, response.content)
response.raise_for_status()
but getting error as Couldn't parse the specified URI. Illegal URI.
I dont understand why it is a illegal URI. I tried with https://googleapis.com/storage/b/BUCKETNMAE but still same error. could not find anything on SO or google docs about this. any idea what wrong am I doing here ?
Google Cloud Storage does not accept a Signed JWT for authorization. Once you create the Signed JWT you must exchange the JWT for an Access Token.
Refer to my answer here or my article for a complete example in Python.
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
I'm trying to use the JSON API for Google Cloud Storage to retrieve a file from Google Cloud Storage. I am not allowed to use the SDKs. Is it possible to create a JWT from a ServiceAccount.json file and use the JWT to access files from Google Cloud Storage? I have a script in node.js that generates a JWT from the service account, but i'm not sure if the audience is right
const jwt = require('jsonwebtoken');
const serviceAccount = require('./serviceAccount.json');
const issuedAt = Math.floor(Date.now() / 1000);
const TOKEN_DURATION_IN_SECONDS = 3600;
let params = {
'iss': serviceAccount.client_email,
'sub': serviceAccount.client_email,
'aud': serviceAccount.project_id,
'iat': issuedAt,
'exp': issuedAt + TOKEN_DURATION_IN_SECONDS,
};
let options = {
algorithm: 'RS256',
header: {
'kid': serviceAccount.private_key_id,
'typ': 'JWT',
'alg': 'RS256',
},
};
let token = jwt.sign(params, serviceAccount.private_key, options);
console.log(token);
I then use that JWT to call the Google Cloud Storage JSON API:
https://www.googleapis.com/storage/v1/b/test
Using the header: Authorization Bearer {token}
That simply resulted in a Invalid Credentials response.
A few questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen examples where it's a url and also where it's the projectId. Neither work for me.
One of the JSON API examples said the Authorization token should be an oauth token. Can I use a JWT instead or do I need to make a call using the JWT to get an access token?
Is my bucket path correct? Is the base folder for the bucket path your projectId? Should my path be /{projectId}/test. I've tried both and neither work.
Recap
This is an IoT project and I need embedded devices to download files from Google Cloud Storage. I need to create a web portal to upload files to (using Firebase Functions) and pass to the device either a bucket path or a private/signed URL that. The bottom line being I need to access a Google Cloud Storage bucket using a service account key. If there is an embedded SDK - great, but I couldn't find one for C. My only thought was to use the JSON API. If there is a way I can sign a URL which can only be accessed using a service account - that works too.
Thanks!
Yes, you can create your own Signed JWT from a service account Json (or P12) file and exchange the JWT for an Access Token that you then use as Authorization: Bearer TOKEN
I have written a number of articles on how to use Json and P12 credentials.
Google Cloud – Creating OAuth Access Tokens for REST API Calls
For your questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen
examples where it's a url and also where it's the projectId. Neither
work for me.
Set aud to "https://www.googleapis.com/oauth2/v4/token"
One of the JSON API examples said the Authorization token should be an
oauth token. Can I use a JWT instead or do I need to make a call using
the JWT to get an access token?
Some APIs accept signed JWTs, others expect an OAuth Access Token. It is just easier to always obtain the OAuth Access Token. In my example code below, I show you how.
Is my bucket path correct? Is the base folder for the bucket path your
projectId? Should my path be /{projectId}/test. I've tried both and
neither work.
Your url shold look like this (Python string building example)
url = "https://www.googleapis.com/storage/v1/b?project=" + project
Below I show you how to call two services (GCE and GCS). Most Google APIs will follow similar styles for building the REST API urls.
From the code in your question, you are missing the last step in the OAuth process. You need to exchange your Signed JWT for an Access Token.
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
Here is a complete Python 3.x example that will list GCE instances. Below this code are changes to display GCS Buckets.
'''
This program lists lists the Google Compute Engine Instances in one zone
'''
import time
import json
import jwt
import requests
import httplib2
# Project ID for this request.
project = 'development-123456'
# The name of the zone for this request.
zone = 'us-west1-a'
# Service Account Credentials, Json format
json_filename = 'service-account.json'
# Permissions to request for Access Token
scopes = "https://www.googleapis.com/auth/cloud-platform"
# Set how long this token will be valid in seconds
expires_in = 3600 # Expires in 1 hour
def load_json_credentials(filename):
''' Load the Google Service Account Credentials from Json file '''
with open(filename, 'r') as f:
data = f.read()
return json.loads(data)
def load_private_key(json_cred):
''' Return the private key from the json credentials '''
return json_cred['private_key']
def create_signed_jwt(pkey, pkey_id, email, scope):
'''
Create a Signed JWT from a service account Json credentials file
This Signed JWT will later be exchanged for an Access Token
'''
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# Note: this token expires and cannot be refreshed. The token must be recreated
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256",
"typ": "JWT" # Google uses SHA256withRSA
}
# JWT Payload
payload = {
"iss": email, # Issuer claim
"sub": email, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scope # Permissions
}
# Encode the headers and payload and sign creating a Signed JWT (JWS)
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
return sig
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
def gce_list_instances(accessToken):
'''
This functions lists the Google Compute Engine Instances in one zone
'''
# Endpoint that we will call
url = "https://www.googleapis.com/compute/v1/projects/" + project + "/zones/" + zone + "/instances"
# One of the headers is "Authorization: Bearer $TOKEN"
headers = {
"Host": "www.googleapis.com",
"Authorization": "Bearer " + accessToken,
"Content-Type": "application/json"
}
h = httplib2.Http()
resp, content = h.request(uri=url, method="GET", headers=headers)
status = int(resp.status)
if status < 200 or status >= 300:
print('Error: HTTP Request failed')
return
j = json.loads(content.decode('utf-8').replace('\n', ''))
print('Compute instances in zone', zone)
print('------------------------------------------------------------')
for item in j['items']:
print(item['name'])
if __name__ == '__main__':
cred = load_json_credentials(json_filename)
private_key = load_private_key(cred)
s_jwt = create_signed_jwt(
private_key,
cred['private_key_id'],
cred['client_email'],
scopes)
token, err = exchangeJwtForAccessToken(s_jwt)
if token is None:
print('Error:', err)
exit(1)
gce_list_instances(token)
To display GCS Buckets instead, modify the code:
# Create the HTTP url for the Google Storage REST API
url = "https://www.googleapis.com/storage/v1/b?project=" + project
resp, content = h.request(uri=url, method="GET", headers=headers)
s = content.decode('utf-8').replace('\n', '')
j = json.loads(s)
print('')
print('Buckets')
print('----------------------------------------')
for item in j['items']:
print(item['name'])
I found this [Service account authorization without OAuth].(https://developers.google.com/identity/protocols/oauth2/service-account#jwt-auth
You can avoid having to make a network request to Google's authorization server before making an API call.
Available APIs are listed in https://github.com/googleapis/googleapis.
It looks like Google Cloud Storage api is not yet published as per the comments in the repository.
Were you able to use the cloud storage API with JWT?
I'm trying to invoke my API Gateway with authenticated users with the REST API. For this I'm using: Cognito UserPool + Cognito Identity Pool + API Gateway + AWS_IAM Authorization + Cognito Credentials. From what I've gathered I need to sign my request with (temporary credentials). Based on this thread I want to sign my request with the following keys:
{
SecretKey: '',
AccesKeyId: '',
SessionKey: ''
}
If I use an associated user from my IAM console and use the corresponding SecretKey + AccesKeyID everything works fine. However, I want to use the Unauthenticated and Authenticated roles from my Identity Pools to apply IAM policies based on authenticated or unauthenticated users.
FYI: I can call the authenticated functions from this part of the documentation.
I'm building a React-Native app, and because of that I want to keep the native SDK to a minimum and I'm only using AWSCognitoIdentityProvider part. For the user handling.
I trying to receive the correct keys using this Objective-C code:
[[self.credentialsProvider credentials] continueWithBlock:^id(AWSTask *task) {
if (task.error) {
NSLog(#"Error: %#", task.error);
}
else {
AWSCredentials *response = task.result;
NSString *accessKey = response.accessKey;
NSString *secretKey = response.secretKey;
NSString *sessionKey = response.sessionKey;
NSDictionary *responseData = #{
#"AccessKey" : accessKey,
#"SecretKey" : secretKey,
#"SessionKey": sessionKey
};
}
return nil;
}];
The rest I've setup using the relevant docs.
I (wrongly?) tried to sign my requests with the
AccessKey, SecretKey, SessionKey retrieved from the CredentialsProvider above.
{
SecretKey: credentials.SecretKey,
AccesKeyId: credentials.AccessKey,
SessionKey: credentials.SessionKey
}
The signing fails with the following error:
{ message: 'The security token included in the request is invalid.' }
So the question I have is: which keys should I use to sign my requests for authenticated users so that I can apply the attached IAM policies from my Cognito Setup?
Thanks for any help :)
Like Michael - sqlbot, point out, it should be SessionToken instead of SessionKey. I found a better instruction on how to get credentials from Cognito.
I am trying to access a simple AWS IOT REST service but I have not been able to do so successfully yet. Here is what I did.
I created an iam user in my aws and downloaded the access key and secret key
Logged into AWS IOT with that user and created a "thing"
From the thing's property I found the REST URL for the shadow
Used Postman with the new "aws signature" feature and provided it with the access key, secret key, region (us-east-1) and service name (iot)
Tried to "GET" the endpoint and this is what I got -
{
"message": "Credential should be scoped to correct service. ",
"traceId": "be056198-d202-455f-ab85-805defd1260d"
}
I thought there is something wrong with postman so I tried using aws-sdk-sample example of connecting to S3 and changed it to connect to the IOT URL.
Here is my program snippet (Java)
String awsAccessKey = "fasfasfasdfsdafs";
String awsSecretKey = "asdfasdfasfasdfasdfasdf/asdfsdafsd/fsdafasdf";
URL endpointUrl = null;
String regionName = "us-east-1";
try {
endpointUrl = new URL("https://dasfsdfasdf.iot.us-east-1.amazonaws.com/things/SOMETHING/shadow");
}catch (Exception e){
e.printStackTrace();
}
Map<String, String> headers = new HashMap<String, String>();
headers.put("x-amz-content-sha256", AWSSignerBase.EMPTY_BODY_SHA256);
AWSSignerForAuthorizationHeader signer = new AWSSignerForAuthorizationHeader(
endpointUrl, "GET", "iot", regionName);
String authorization = signer.computeSignature(headers,
null, // no query parameters
AWSSignerBase.EMPTY_BODY_SHA256,
awsAccessKey,
awsSecretKey);
// place the computed signature into a formatted 'Authorization' header
// and call S3
headers.put("Authorization", authorization);
String response = HttpUtils.invokeHttpRequest(endpointUrl, "GET", headers, null);
System.out.println("--------- Response content ---------");
System.out.println(response);
System.out.println("------------------------------------");
This gives me the same error -
--------- Request headers ---------
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=fasfasfasdfsdafs/20160212/us-east-1/iot/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3b2194051a8dde8fe617219c78c2a79b77ec92338028e9e917a74e8307f4e914
x-amz-date: 20160212T182525Z
Host: dasfsdfasdf.iot.us-east-1.amazonaws.com
--------- Response content ---------
{"message":"Credential should be scoped to correct service. ","traceId":"cd3e0d96-82fa-4da5-a4e1-b736af6c5e34"}
------------------------------------
Can someone tell me what I am doing wrong please? AWS documentation does not have much information on this error. Please help
Sign your request with iotdata instead if iot
example:
AWSSignerForAuthorizationHeader signer = new AWSSignerForAuthorizationHeader(
endpointUrl, "GET", "iotdata", regionName);
In your 4th step, don't fill anything for Service Name. Postman will default the value with execute-api.
Hope this works!
Its basically due to Service name is not given correctly you can use service Name = 'iotdata' instead of iot.
If you user Key management then Service Name would be kms.
For EC2 Service Name would be ec2 etc.
Use the AWS IoT SDK for Node.js instead. Download the IoT Console generated private key and client cert as well as the CA Root cert from here. Start with the scripts in the examples directory.