Google Cloud Storage JSON API with JWT Token - google-cloud-platform

Question

I'm trying to use the JSON API for Google Cloud Storage to retrieve a file from Google Cloud Storage. I am not allowed to use the SDKs. Is it possible to create a JWT from a ServiceAccount.json file and use the JWT to access files from Google Cloud Storage? I have a script in node.js that generates a JWT from the service account, but i'm not sure if the audience is right
const jwt = require('jsonwebtoken');
const serviceAccount = require('./serviceAccount.json');
const issuedAt = Math.floor(Date.now() / 1000);
const TOKEN_DURATION_IN_SECONDS = 3600;
let params = {
'iss': serviceAccount.client_email,
'sub': serviceAccount.client_email,
'aud': serviceAccount.project_id,
'iat': issuedAt,
'exp': issuedAt + TOKEN_DURATION_IN_SECONDS,
};
let options = {
algorithm: 'RS256',
header: {
'kid': serviceAccount.private_key_id,
'typ': 'JWT',
'alg': 'RS256',
},
};
let token = jwt.sign(params, serviceAccount.private_key, options);
console.log(token);
I then use that JWT to call the Google Cloud Storage JSON API:
https://www.googleapis.com/storage/v1/b/test
Using the header: Authorization Bearer {token}
That simply resulted in a Invalid Credentials response.
A few questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen examples where it's a url and also where it's the projectId. Neither work for me.
One of the JSON API examples said the Authorization token should be an oauth token. Can I use a JWT instead or do I need to make a call using the JWT to get an access token?
Is my bucket path correct? Is the base folder for the bucket path your projectId? Should my path be /{projectId}/test. I've tried both and neither work.
Recap
This is an IoT project and I need embedded devices to download files from Google Cloud Storage. I need to create a web portal to upload files to (using Firebase Functions) and pass to the device either a bucket path or a private/signed URL that. The bottom line being I need to access a Google Cloud Storage bucket using a service account key. If there is an embedded SDK - great, but I couldn't find one for C. My only thought was to use the JSON API. If there is a way I can sign a URL which can only be accessed using a service account - that works too.
Thanks!

Answer 1

Yes, you can create your own Signed JWT from a service account Json (or P12) file and exchange the JWT for an Access Token that you then use as Authorization: Bearer TOKEN
I have written a number of articles on how to use Json and P12 credentials.
Google Cloud – Creating OAuth Access Tokens for REST API Calls
For your questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen
examples where it's a url and also where it's the projectId. Neither
work for me.
Set aud to "https://www.googleapis.com/oauth2/v4/token"
One of the JSON API examples said the Authorization token should be an
oauth token. Can I use a JWT instead or do I need to make a call using
the JWT to get an access token?
Some APIs accept signed JWTs, others expect an OAuth Access Token. It is just easier to always obtain the OAuth Access Token. In my example code below, I show you how.
Is my bucket path correct? Is the base folder for the bucket path your
projectId? Should my path be /{projectId}/test. I've tried both and
neither work.
Your url shold look like this (Python string building example)
url = "https://www.googleapis.com/storage/v1/b?project=" + project
Below I show you how to call two services (GCE and GCS). Most Google APIs will follow similar styles for building the REST API urls.
From the code in your question, you are missing the last step in the OAuth process. You need to exchange your Signed JWT for an Access Token.
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
Here is a complete Python 3.x example that will list GCE instances. Below this code are changes to display GCS Buckets.
'''
This program lists lists the Google Compute Engine Instances in one zone
'''
import time
import json
import jwt
import requests
import httplib2
# Project ID for this request.
project = 'development-123456'
# The name of the zone for this request.
zone = 'us-west1-a'
# Service Account Credentials, Json format
json_filename = 'service-account.json'
# Permissions to request for Access Token
scopes = "https://www.googleapis.com/auth/cloud-platform"
# Set how long this token will be valid in seconds
expires_in = 3600 # Expires in 1 hour
def load_json_credentials(filename):
''' Load the Google Service Account Credentials from Json file '''
with open(filename, 'r') as f:
data = f.read()
return json.loads(data)
def load_private_key(json_cred):
''' Return the private key from the json credentials '''
return json_cred['private_key']
def create_signed_jwt(pkey, pkey_id, email, scope):
'''
Create a Signed JWT from a service account Json credentials file
This Signed JWT will later be exchanged for an Access Token
'''
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# Note: this token expires and cannot be refreshed. The token must be recreated
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256",
"typ": "JWT" # Google uses SHA256withRSA
}
# JWT Payload
payload = {
"iss": email, # Issuer claim
"sub": email, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scope # Permissions
}
# Encode the headers and payload and sign creating a Signed JWT (JWS)
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
return sig
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
def gce_list_instances(accessToken):
'''
This functions lists the Google Compute Engine Instances in one zone
'''
# Endpoint that we will call
url = "https://www.googleapis.com/compute/v1/projects/" + project + "/zones/" + zone + "/instances"
# One of the headers is "Authorization: Bearer $TOKEN"
headers = {
"Host": "www.googleapis.com",
"Authorization": "Bearer " + accessToken,
"Content-Type": "application/json"
}
h = httplib2.Http()
resp, content = h.request(uri=url, method="GET", headers=headers)
status = int(resp.status)
if status < 200 or status >= 300:
print('Error: HTTP Request failed')
return
j = json.loads(content.decode('utf-8').replace('\n', ''))
print('Compute instances in zone', zone)
print('------------------------------------------------------------')
for item in j['items']:
print(item['name'])
if __name__ == '__main__':
cred = load_json_credentials(json_filename)
private_key = load_private_key(cred)
s_jwt = create_signed_jwt(
private_key,
cred['private_key_id'],
cred['client_email'],
scopes)
token, err = exchangeJwtForAccessToken(s_jwt)
if token is None:
print('Error:', err)
exit(1)
gce_list_instances(token)
To display GCS Buckets instead, modify the code:
# Create the HTTP url for the Google Storage REST API
url = "https://www.googleapis.com/storage/v1/b?project=" + project
resp, content = h.request(uri=url, method="GET", headers=headers)
s = content.decode('utf-8').replace('\n', '')
j = json.loads(s)
print('')
print('Buckets')
print('----------------------------------------')
for item in j['items']:
print(item['name'])

Answer 2

I found this [Service account authorization without OAuth].(https://developers.google.com/identity/protocols/oauth2/service-account#jwt-auth
You can avoid having to make a network request to Google's authorization server before making an API call.
Available APIs are listed in https://github.com/googleapis/googleapis.
It looks like Google Cloud Storage api is not yet published as per the comments in the repository.
Were you able to use the cloud storage API with JWT?