I'm trying to generate a JWT for a given service account serviceA from a Google/Firebase Cloud function. Service account serviceB is running the function.
I got it working by using the account keys from JSON.
Given that the CF is running within Google Cloud, I want to leverage compute metadata to not having to store the private key with the functions.
I've been trying to access the metadata server for serviceA while serviceB is executing the CF. I deliberately don't want serviceA to run the CF.
The code
const request = require('request-promise');
const serviceAccountEmail = 'serviceA#<projectA>.iam.gserviceaccount.com';
const metadataServerTokenURL = `http://metadata/computeMetadata/v1/instance/service-accounts/${serviceAccountEmail}/identity?audience=<audience>`;
const tokenRequestOptions = {
uri: metadataServerTokenURL,
headers: {
'Metadata-Flavor': 'Google'
}
};
const token = await request(tokenRequestOptions);
The error
I'm currently getting a 404 not found error for the email provided
I guess it's
a) not possible what I'm trying to do, or
b) I'm missing some IAM permissions for serviceA
You can do this with the metadata server because they can only generate ID Token for the service account loaded with your instance (in this case the serviceB).
You can use another API for this: Service Account Credentials API, especially the generateIdToken method
In your case, you can do something like this (in python here)
import google.auth
from google.auth.transport.requests import AuthorizedSession
import json
# IAP audience is the ClientID of IAP-App-Engine-app in
# the API->credentials page
# Cloud Function and Cloud Run need the base URL of the service
audience = 'YOUR AUDIENCE'
# #1 Get the default credential to generate the access token
credentials, project_id = google.auth.default(
scopes='https://www.googleapis.com/auth/cloud-platform')
# #2 To use the current service account email
service_account_email = credentials.service_account_email
# Don't work with user account, so define manually the email
# service_account_email = 'MY SERVICE ACCOUNT EMAIL'
# #3 prepare the call the the service account credentials API
sa_credentials_url = f'https://iamcredentials.googleapis.com/' \
f'v1/projects/-/serviceAccounts/' \
f'{service_account_email}:generateIdToken'
headers = {'Content-Type': 'application/json'}
# Create an AuthorizedSession that includes
# automatically the access_token based on your credentials
authed_session = AuthorizedSession(credentials)
# Define the audience in the request body
# add the parameter "'includeEmail':true" for IAP access
body = json.dumps({'audience': audience})
# Make the call
token_response = authed_session.request('POST',sa_credentials_url,
data=body, headers=headers)
jwt = token_response.json()
id_token = jwt['token']
I wrote an article on this, this week
I adopted #guillaume blaquiere's solution to Typescript:
import { GaxiosOptions, Headers } from 'gaxios';
import { GoogleAuth } from 'google-auth-library';
interface TokenRequestResponse {
token: string
}
const service_account_email = 'MY SERVICE ACCOUNT EMAIL'
const audience: string = 'MY AUDIENCE';
const sa_credentials_url = `https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${service_account_email}:generateIdToken`
const headers: Headers = { 'Content-Type': 'application/json' }
const body = {
'audience': audience
};
const options: GaxiosOptions = {
method: 'POST',
url: sa_credentials_url,
body: JSON.stringify(body),
headers: headers,
};
const auth = new GoogleAuth({
scopes: 'https://www.googleapis.com/auth/cloud-platform'
});
const client = await auth.getClient();
const tokenResponse = await client.request(options);
const tokenRequestResponse = tokenResponse.data as TokenRequestResponse;
const token = tokenRequestResponse.token;
Related
in my scenario, I would like to trigger an Google Cloud Function based on HTTP endpoint during a Google Cloud Build. The HTTP request is done using a step with a python:3.7-slim container.
Based on this and this examples from the documentation, I wanted to use the following code:
REGION = 'us-central1'
PROJECT_ID = 'name-of-project'
RECEIVING_FUNCTION = 'my-cloud-function'
function_url = f'https://{REGION}-{PROJECT_ID}.cloudfunctions.net/{RECEIVING_FUNCTION}'
metadata_server_url = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience='
token_full_url = metadata_server_url + function_url
token_headers = {'Metadata-Flavor': 'Google'}
token_response = requests.get(token_full_url, headers=token_headers)
jwt = token_response.text
print(jwt)
r = requests.post(url=function_url, headers=function_headers, json=payload)
Surprisingly, the code fails because jwt is Not Found (according to the print statement).
I already tested the code and IAM settings by hard coding a valid identity token and also tested the exact same fetching mechanism on a test VM inside the same project.
The problem seems to be that the meta data fetching some is not working inside cloud build.
Am I missing something?
Thank you for any help!
The solution is to use a new IAM api to generate an ID_TOKEN, on a service account with an access token, if the requester (this one who generate the access token) has the role Service Account Token Creator on the service account (or widely on the project).
This first example use direct API calls
- name: gcr.io/cloud-builders/gcloud
entrypoint: "bash"
args:
- "-c"
- |
curl -X POST -H "content-type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-d '{"audience": "YOUR AUDIENCE"}' \
"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/YOUR SERVICE ACCOUNT:generateIdToken"
# Use Cloud Build Service Account
# service_account_email=$(gcloud config get-value account)
And here the Python code version
- name: python:3.7
entrypoint: "bash"
args:
- "-c"
- |
pip3 install google-auth requests
python3 extract-token.py
And extract-token.py content the following code
REGION = 'us-central1'
PROJECT_ID = 'name-of-project'
RECEIVING_FUNCTION = 'my-cloud-function'
function_url = f'https://{REGION}-{PROJECT_ID}.cloudfunctions.net/{RECEIVING_FUNCTION}'
import google.auth
credentials, project_id = google.auth.default(scopes='https://www.googleapis.com/auth/cloud-platform')
# To use the Cloud Build service account email
service_account_email = credentials.service_account_email
#service_account_email = "YOUR OWN SERVICE ACCOUNT"
metadata_server_url = f'https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{service_account_email}:generateIdToken'
token_headers = {'content-type': 'application/json'}
from google.auth.transport.requests import AuthorizedSession
authed_session = AuthorizedSession(credentials)
import json
body = json.dumps({'audience': function_url})
token_response = authed_session.request('POST',metadata_server_url, data=body, headers=token_headers)
jwt = token_response.json()
print(jwt['token'])
Don't hesitate if you need more details.
I think I will write an article on this on Medium, if you want I name you, let me know
The best here is to create a Feature Request (FR) in the Public Issue Tracker. There is a difference between filing an issue and a FR. The FR gives visibility to the Engineering team of the real needs; according to the number of users which are being affected by that, they prioritize the development of them. I suggest also to create a guthub repo so they can easily replicate it and make reference to the issues aforementioned.
On the other hand as a workaround, you can create a topic in Pub/Sub to receive build notifications:
gcloud pubsub topics create cloud-builds
Each time you submit a build, a message will be pushed to the topic, then you can create a PubSub Cloud Function and call your HTTP CF from there.
I used this example from github, mentioned in the docs Authenticating Function to function
const {get} = require('axios');
// TODO(developer): set these values
const REGION = 'us-central1';
const PROJECT_ID = 'YOUR PROJECTID';
const RECEIVING_FUNCTION = 'FUNCTION TO TRIGGER';
// Constants for setting up metadata server request
// See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
const functionURL = `https://${REGION}-${PROJECT_ID}.cloudfunctions.net/${RECEIVING_FUNCTION}`;
const metadataServerURL =
'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
const tokenUrl = metadataServerURL + functionURL;
exports.helloPubSub = async (event, context) => {
// Fetch the token
const message = event.data
? Buffer.from(event.data, 'base64').toString()
: 'Hello, World';
const tokenResponse = await get(tokenUrl, {
headers: {
'Metadata-Flavor': 'Google',
},
});
const token = tokenResponse.data;
// Provide the token in the request to the receiving function
try {
const functionResponse = await get(functionURL, {
headers: {Authorization: `bearer ${token}`},
});
console.log(message);
} catch (err) {
console.error(err);
}
};
Finally, when the ClouBuild submits, your PubSub CF will be triggered and you can call your CF inside it.
What i want to do: To call a google function from my server/machine & limit it usage with a (simple) authentication.
What i use: Node.js, google-auth-library library for authentication.
What have I done/tried:
1) Created a project in Google Cloud Functions
2) Created a simple google function
exports.helloWorld = (req, res) => {
let message = req.query.message || req.body.message || 'Hello World!';
res.status(200).send(message);
};
3) Set my custom service account
4) Enabled api:
- Cloud Functions API
- IAM Service Account Credentials API
- Cloud Run API
- Compute Engine API
- IAM Service Account Credentials API
5) Given to my server account necessary authorization (project owner, cloud function admin, IAM project admin... (need more?)
6) Generated key from my service account and saved it in json format
NB: with allUser permission (without authorization required), i can call my endpoint without problem
7) From my project i tried to auth my function in this way
const { JWT } = require('google-auth-library');
const fetch = require('node-fetch');
const keys = require('./service-account-keys.json');
async function callFunction(text) {
const url = `https://europe-west1-myFunction.cloudfunctions.net/test`;
const client = new JWT({
email: keys.client_email,
keyFile: keys,
key: keys.private_key,
scopes: [
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/iam',
],
});
const res = await client.request({ url });
const tokenInfo = await client.getTokenInfo(client.credentials.access_token);
try {
const response = await fetch(url, {
method: 'GET',
headers: {
Authorization: `Bearer ${client.credentials.access_token}`,
},
});
if (response.status !== 200) {
console.log(response);
return {};
}
return response.json();
} catch (e) {
console.error(e);
}
}
âšī¸ if i try to pass at client.request() url without name of function (https://europe-west1-myFunction.cloudfunctions.net), i not received error, but when use the JWT token obtained in fetch call, i received the same error.
RESULT:
Error:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>401 Unauthorized</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Unauthorized</h1>
<h2>Your client does not have permission to the requested URL <code>/test1</code>.</h2>
<h2></h2>
</body></html>
â How do I call a google function with any protection to prevent anyone from using it? (I don't need specific security, just that random users don't use it)
Thanks in advance for any help
When you call a private function (or a private Cloud Run) you have to use a google signed identity token.
In your code, you use an access token
headers: {
Authorization: `Bearer ${client.credentials.access_token}`,
},
Access token work when you have to request Google Cloud API, not your services
And the google signed is important, because you can easily generate a self signed identity token with the google auth lib, but it won't work
You have code sample here and I wrote a tool in Go if you want to have a try on it
** EDIT **
I worked on an example, and, even if I never liked Javascript, I have to admit that I'm jealous!! It's so simple in Node!!
Here my working example
const {GoogleAuth} = require('google-auth-library');
async function main() {
// Define your URL, here with Cloud Run but the security is exactly the same with Cloud Functions (same underlying infrastructure)
const url = "https://go111-vqg64v3fcq-uc.a.run.app"
// Here I use the default credential, not an explicit key like you
const auth = new GoogleAuth();
//Example with the key file, not recommended on GCP environment.
//const auth = new GoogleAuth({keyFilename:"/path/to/key.json"})
//Create your client with an Identity token.
const client = await auth.getIdTokenClient(url);
const res = await client.request({url});
console.log(res.data);
}
main().catch(console.error);
Note: Only Service account can generate and identity token with audience. If you are in your local computer, don't use your user account with the default credential mode.
My express server has a credentials.json containing credentials for a google service account. These credentials are used to get a jwt from google, and that jwt is used by my server to update google sheets owned by the service account.
var jwt_client = null;
// load credentials form a local file
fs.readFile('./private/credentials.json', (err, content) => {
if (err) return console.log('Error loading client secret file:', err);
// Authorize a client with credentials, then call the Google Sheets API.
authorize(JSON.parse(content));
});
// get JWT
function authorize(credentials) {
const {client_email, private_key} = credentials;
jwt_client = new google.auth.JWT(client_email, null, private_key, SCOPES);
}
var sheets = google.sheets({version: 'v4', auth: jwt_client });
// at this point i can call google api and make authorized requests
The issue is that I'm trying to move from node/express to npm serverless/aws. I'm using the same code but getting 403 - forbidden.
errors:
[ { message: 'The request is missing a valid API key.',
domain: 'global',
reason: 'forbidden' } ] }
Research has pointed me to many things including: AWS Cognito, storing credentials in environment variables, custom authorizers in API gateway. All of these seem viable to me but I am new to AWS so any advice on which direction to take would be greatly appreciated.
it is late, but may help someone else. Here is my working code.
const {google} = require('googleapis');
const KEY = require('./keys');
const _ = require('lodash');
const sheets = google.sheets('v4');
const jwtClient = new google.auth.JWT(
KEY.client_email,
null,
KEY.private_key,
[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/spreadsheets'
],
null
);
async function getGoogleSheetData() {
await jwtClient.authorize();
const request = {
// The ID of the spreadsheet to retrieve data from.
spreadsheetId: 'put your id here',
// The A1 notation of the values to retrieve.
range: 'put your range here', // TODO: Update placeholder value.
auth: jwtClient,
};
return await sheets.spreadsheets.values.get(request)
}
And then call it in the lambda handler. There is one thing I don't like is storing key.json as file in the project root. Will try to find some better place to save.
I wanted to access GCP storage bucket from outside. So I used following steps which google has provided.
Created the service account
Generated the jwt token using the private key that provided for service account.
When I called the above API to get access token by providing jwt token it gives following error.
{
"error": "invalid_scope",
"error_description": "Empty or missing scope not allowed."
}
Thanks in advance!
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
This is the Java code I used to generate the JWT
long now = System.currentTimeMillis();
try {
GoogleCredential credential = GoogleCredential.fromStream(new FileInputStream("service.json"));
PrivateKey privateKey = credential.getServiceAccountPrivateKey();
String privateKeyId = credential.getServiceAccountPrivateKeyId();
Algorithm algorithm = Algorithm.RSA256(null, (RSAPrivateKey) privateKey);
String signedJwt = JWT.create()
.withKeyId(privateKeyId)
.withIssuer("***********#************-******.iam.gserviceaccount.com")
.withSubject("***********#************-******.iam.gserviceaccount.com")
.withAudience("https://www.googleapis.com/oauth2/v4/token")
.withIssuedAt(new Date(now))
.withExpiresAt(new Date(now + 3600 * 1000L))
.sign(algorithm);
System.out.println(signedJwt);
} catch(Exception e) {
System.out.println(e);
}
I figure out the issue. It was with the payload that I passed to generate the JWT token.
Below I attched the python code which I used to genarate jwt token.
I got the reference from https://www.jhanley.com/google-cloud-creating-oauth-access-tokens-for-rest-api-calls/ below python code
import jwt
import time
# Permissions to request for Access Token
scopes = "https://www.googleapis.com/auth/devstorage.read_write"
# private key id
pkey_id = ""
# private key
pkey = ""
serviceid = ""
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
# Set how long this token will be valid in seconds
expires_in = 3600 # Expires in 1 hour
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# JWT Payload
payload = {
"iss": serviceid, # Issuer claim
"sub": serviceid, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scopes # Permissions
}
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256",
"typ": "JWT" # Google uses SHA256withRSA
}
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
print(sig)
I'm trying to use the JSON API for Google Cloud Storage to retrieve a file from Google Cloud Storage. I am not allowed to use the SDKs. Is it possible to create a JWT from a ServiceAccount.json file and use the JWT to access files from Google Cloud Storage? I have a script in node.js that generates a JWT from the service account, but i'm not sure if the audience is right
const jwt = require('jsonwebtoken');
const serviceAccount = require('./serviceAccount.json');
const issuedAt = Math.floor(Date.now() / 1000);
const TOKEN_DURATION_IN_SECONDS = 3600;
let params = {
'iss': serviceAccount.client_email,
'sub': serviceAccount.client_email,
'aud': serviceAccount.project_id,
'iat': issuedAt,
'exp': issuedAt + TOKEN_DURATION_IN_SECONDS,
};
let options = {
algorithm: 'RS256',
header: {
'kid': serviceAccount.private_key_id,
'typ': 'JWT',
'alg': 'RS256',
},
};
let token = jwt.sign(params, serviceAccount.private_key, options);
console.log(token);
I then use that JWT to call the Google Cloud Storage JSON API:
https://www.googleapis.com/storage/v1/b/test
Using the header: Authorization Bearer {token}
That simply resulted in a Invalid Credentials response.
A few questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen examples where it's a url and also where it's the projectId. Neither work for me.
One of the JSON API examples said the Authorization token should be an oauth token. Can I use a JWT instead or do I need to make a call using the JWT to get an access token?
Is my bucket path correct? Is the base folder for the bucket path your projectId? Should my path be /{projectId}/test. I've tried both and neither work.
Recap
This is an IoT project and I need embedded devices to download files from Google Cloud Storage. I need to create a web portal to upload files to (using Firebase Functions) and pass to the device either a bucket path or a private/signed URL that. The bottom line being I need to access a Google Cloud Storage bucket using a service account key. If there is an embedded SDK - great, but I couldn't find one for C. My only thought was to use the JSON API. If there is a way I can sign a URL which can only be accessed using a service account - that works too.
Thanks!
Yes, you can create your own Signed JWT from a service account Json (or P12) file and exchange the JWT for an Access Token that you then use as Authorization: Bearer TOKEN
I have written a number of articles on how to use Json and P12 credentials.
Google Cloud â Creating OAuth Access Tokens for REST API Calls
For your questions:
I'm not sure what the 'aud' should be when creating the JWT. I've seen
examples where it's a url and also where it's the projectId. Neither
work for me.
Set aud to "https://www.googleapis.com/oauth2/v4/token"
One of the JSON API examples said the Authorization token should be an
oauth token. Can I use a JWT instead or do I need to make a call using
the JWT to get an access token?
Some APIs accept signed JWTs, others expect an OAuth Access Token. It is just easier to always obtain the OAuth Access Token. In my example code below, I show you how.
Is my bucket path correct? Is the base folder for the bucket path your
projectId? Should my path be /{projectId}/test. I've tried both and
neither work.
Your url shold look like this (Python string building example)
url = "https://www.googleapis.com/storage/v1/b?project=" + project
Below I show you how to call two services (GCE and GCS). Most Google APIs will follow similar styles for building the REST API urls.
From the code in your question, you are missing the last step in the OAuth process. You need to exchange your Signed JWT for an Access Token.
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
Here is a complete Python 3.x example that will list GCE instances. Below this code are changes to display GCS Buckets.
'''
This program lists lists the Google Compute Engine Instances in one zone
'''
import time
import json
import jwt
import requests
import httplib2
# Project ID for this request.
project = 'development-123456'
# The name of the zone for this request.
zone = 'us-west1-a'
# Service Account Credentials, Json format
json_filename = 'service-account.json'
# Permissions to request for Access Token
scopes = "https://www.googleapis.com/auth/cloud-platform"
# Set how long this token will be valid in seconds
expires_in = 3600 # Expires in 1 hour
def load_json_credentials(filename):
''' Load the Google Service Account Credentials from Json file '''
with open(filename, 'r') as f:
data = f.read()
return json.loads(data)
def load_private_key(json_cred):
''' Return the private key from the json credentials '''
return json_cred['private_key']
def create_signed_jwt(pkey, pkey_id, email, scope):
'''
Create a Signed JWT from a service account Json credentials file
This Signed JWT will later be exchanged for an Access Token
'''
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# Note: this token expires and cannot be refreshed. The token must be recreated
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256",
"typ": "JWT" # Google uses SHA256withRSA
}
# JWT Payload
payload = {
"iss": email, # Issuer claim
"sub": email, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scope # Permissions
}
# Encode the headers and payload and sign creating a Signed JWT (JWS)
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
return sig
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
def gce_list_instances(accessToken):
'''
This functions lists the Google Compute Engine Instances in one zone
'''
# Endpoint that we will call
url = "https://www.googleapis.com/compute/v1/projects/" + project + "/zones/" + zone + "/instances"
# One of the headers is "Authorization: Bearer $TOKEN"
headers = {
"Host": "www.googleapis.com",
"Authorization": "Bearer " + accessToken,
"Content-Type": "application/json"
}
h = httplib2.Http()
resp, content = h.request(uri=url, method="GET", headers=headers)
status = int(resp.status)
if status < 200 or status >= 300:
print('Error: HTTP Request failed')
return
j = json.loads(content.decode('utf-8').replace('\n', ''))
print('Compute instances in zone', zone)
print('------------------------------------------------------------')
for item in j['items']:
print(item['name'])
if __name__ == '__main__':
cred = load_json_credentials(json_filename)
private_key = load_private_key(cred)
s_jwt = create_signed_jwt(
private_key,
cred['private_key_id'],
cred['client_email'],
scopes)
token, err = exchangeJwtForAccessToken(s_jwt)
if token is None:
print('Error:', err)
exit(1)
gce_list_instances(token)
To display GCS Buckets instead, modify the code:
# Create the HTTP url for the Google Storage REST API
url = "https://www.googleapis.com/storage/v1/b?project=" + project
resp, content = h.request(uri=url, method="GET", headers=headers)
s = content.decode('utf-8').replace('\n', '')
j = json.loads(s)
print('')
print('Buckets')
print('----------------------------------------')
for item in j['items']:
print(item['name'])
I found this [Service account authorization without OAuth].(https://developers.google.com/identity/protocols/oauth2/service-account#jwt-auth
You can avoid having to make a network request to Google's authorization server before making an API call.
Available APIs are listed in https://github.com/googleapis/googleapis.
It looks like Google Cloud Storage api is not yet published as per the comments in the repository.
Were you able to use the cloud storage API with JWT?