Authorize google service account using AWS Lambda/API Gateway - amazon-web-services

My express server has a credentials.json containing credentials for a google service account. These credentials are used to get a jwt from google, and that jwt is used by my server to update google sheets owned by the service account.
var jwt_client = null;
// load credentials form a local file
fs.readFile('./private/credentials.json', (err, content) => {
if (err) return console.log('Error loading client secret file:', err);
// Authorize a client with credentials, then call the Google Sheets API.
authorize(JSON.parse(content));
});
// get JWT
function authorize(credentials) {
const {client_email, private_key} = credentials;
jwt_client = new google.auth.JWT(client_email, null, private_key, SCOPES);
}
var sheets = google.sheets({version: 'v4', auth: jwt_client });
// at this point i can call google api and make authorized requests
The issue is that I'm trying to move from node/express to npm serverless/aws. I'm using the same code but getting 403 - forbidden.
errors:
[ { message: 'The request is missing a valid API key.',
domain: 'global',
reason: 'forbidden' } ] }
Research has pointed me to many things including: AWS Cognito, storing credentials in environment variables, custom authorizers in API gateway. All of these seem viable to me but I am new to AWS so any advice on which direction to take would be greatly appreciated.

it is late, but may help someone else. Here is my working code.
const {google} = require('googleapis');
const KEY = require('./keys');
const _ = require('lodash');
const sheets = google.sheets('v4');
const jwtClient = new google.auth.JWT(
KEY.client_email,
null,
KEY.private_key,
[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/spreadsheets'
],
null
);
async function getGoogleSheetData() {
await jwtClient.authorize();
const request = {
// The ID of the spreadsheet to retrieve data from.
spreadsheetId: 'put your id here',
// The A1 notation of the values to retrieve.
range: 'put your range here', // TODO: Update placeholder value.
auth: jwtClient,
};
return await sheets.spreadsheets.values.get(request)
}
And then call it in the lambda handler. There is one thing I don't like is storing key.json as file in the project root. Will try to find some better place to save.

Related

Multicloud: Authenticate Googe Cloud Translation Client from AWS Lambda (Nodejs)

Problem
I am trying to access a Google Cloud service (Cloud Translate API) from my AWS Lambda using Nodejs and serverless framework. The system already works perfectly when I use a Google Service Account Key, so that validates that the two cloud services are operational and functional.
However, I'm trying to follow best practice and use Google's Federated Workforce ID instead of a Service Account Key. (Docs).
However, I'm getting an error:
FetchError: request to http://169.254.169.254/latest/meta-data/iam/security-credentials failed, reason: connect ETIMEDOUT 169.254.169.254:80
I've followed the directions in the docs several times, including creating the workplace pool and downloading the client config file. And I have the environment variable set to the config file:
GOOGLE_APPLICATION_CREDENTIALS: ./clientLibraryConfig-fq-aws-apis.json
The Google Auth picks up the credentials file (I can see by running a console.log on const "client"), and it retrieves my projectId in auth.getProjectId();.
But when it comes to initiating the TranslationServiceClient, I get this:
Error
"errorMessage": "request to http://169.254.169.254/latest/meta-data/iam/security-credentials failed, reason: connect ETIMEDOUT 169.254.169.254:80",
Code
"use strict";
const { GoogleAuth } = require("google-auth-library");
const { TranslationServiceClient } = require("#google-cloud/translate");
//////////
// This function gets translation from Google
//////////
const getTranslations = async (originalClipArray, translateTo) => {
// G Translate params
// const projectId = "rw-frequency";
const location = "global";
const auth = new GoogleAuth({
scopes: 'https://www.googleapis.com/auth/cloud-platform'
});
const client = await auth.getClient();
const projectId = await auth.getProjectId();
const translationClient = new TranslationServiceClient()
console.log("past translationserviceclient constructor");
// Build the params for the translate request
const request = {
parent: `projects/${projectId}/locations/${location}`,
contents: originalClipArray,
mimeType: "text/plain", // mime types: text/plain, text/html
targetLanguageCode: translateTo,
};
// Call Google client
// try {
const response = await translationClient.translateText(request);
console.log(`response`);
console.dir(response);
return response;
// } catch (error) {
// console.log(`Google translate error raised:`);
// console.log(error);
// }
};
module.exports.getTranslations = getTranslations;
The request that gives you a timeout retrieves security credentials for EC2 instances. Apparently, your Lambda is using a GCP library intended for EC2. Hope this helps!

How to import Google Cloud credentials into a Firebase Cloud Function?

I'm trying to set up Google Cloud Translation in a Firebase Cloud Function. I'm using the demo code provided by Google Cloud Translation:
// Instantiates a client
const translationClient = new TranslationServiceClient();
const projectId = 'languagetwo-cd94d';
const location = 'global';
const text = 'Hello, world!';
async function translateText() {
// Construct request
const request = {
parent: `projects/${projectId}/locations/${location}`,
contents: [text],
mimeType: 'text/plain', // mime types: text/plain, text/html
sourceLanguageCode: 'en',
targetLanguageCode: 'es',
};
// Run request
const [response] = await translationClient.translateText(request);
for (const translation of response.translations) {
console.log(`Translation: ${translation.translatedText}`);
}
}
translateText();
This demo tutorial makes a second file called key.json:
{
"type": "service_account",
"project_id": "myAwesomeApp",
"private_key_id": "1234567890",
"private_key": "-----BEGIN PRIVATE KEY-----\noPeeking=\n-----END PRIVATE KEY-----\n",
"client_email": "translation-quickstart#myAwesomeApp.iam.gserviceaccount.com",
"client_id": "1234567890",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/translation-quickstart%40myAwesomeApp.iam.gserviceaccount.com"
}
I uploaded my credentials from the CLI:
gcloud auth login
gcloud iam service-accounts create translation-quickstart --project myAwesomeApp
gcloud projects add-iam-policy-binding myAwesomeApp
gcloud iam service-accounts keys \
create key.json --iam-account \
translation-quickstart#myAwesomeApp.iam.gserviceaccount.com
export GOOGLE_APPLICATION_CREDENTIALS=key.json
I then entered node app.js at the CLI and it runs perfectly. ¡Hola Mundo!
How do I import my credentials into a Firebase Cloud Function? I tried this:
exports.ENtranslateES = functions.firestore.document('Users/{userID}/English/Translation_Request').onUpdate((change) => { // triggers when browser writes a request word to the database
// Google Cloud
const { TranslationServiceClient } = require('#google-cloud/translate');
// Instantiates a client
const translationClient = new TranslationServiceClient();
const projectId = 'languagetwo-cd94d';
const location = 'global';
const text = 'Hello, world!';
async function translateText() {
// Construct request
const request = {
parent: `projects/${projectId}/locations/${location}`,
contents: [text],
mimeType: 'text/plain', // mime types: text/plain, text/html
sourceLanguageCode: 'en',
targetLanguageCode: 'es',
};
// Run request
const [response] = await translationClient.translateText(request);
for (const translation of response.translations) {
console.log(`Translation: ${translation.translatedText}`);
}
}
return translateText()
});
I added only a return at the bottom because Firebase Cloud Functions require that something has to be returned.
The result is that the function triggers and translateText() fires. Then I get an error message:
Error: 7 PERMISSION_DENIED: Cloud IAM permission
That looks like the credentials weren't imported. How do I import the key.json credentials into the Firebase Cloud Function?
Normally, you do not import a service account into a Google compute service such as Cloud Functions. Those services have an attached service account. There are methods of securely storing a service account using services like Google Cloud Secret Manager. In your case there is a better solution.
The following line in your source code uses the Cloud Function attached service account, which defaults to the App Engine default service account PROJECT_ID#appspot.gserviceaccount.com.
const translationClient = new TranslationServiceClient();
Since you did not specify a credential when creating the translationClient, ADC (Application Default Credentials) searches for credentials. In your example, the search found valid credentials from the Cloud Function service account.
The solution is to add the required role to that service account.
If you want to use the service account that you created, then attach the service account identity (email address) to the Cloud Function link.
Access control with IAM
I got Google Cloud Translate to work in Postman. This is a step towards an answer, not an answer. (Google has its own version of Postman, called Google Cloud API, but it doesn't work with Translate,)
I followed this blog post to set up Google Cloud API in Postman. I started at my Google Cloud Console. I selected my project. I clicked APIs & Services, then Credentials, then + Create Credentials, then OAuth client ID. Under Application type I selected Web application. I named the client ID Postman. Lastly I added an Authorized redirect URI: https://console.cloud.google.com/. Now when I click on my Postman API in my Google Cloud Console I see a Client ID and a Client secret.
In Postman, I changed GET to POST and entered the URL from the Google Cloud Translation page:
https://cloud.google.com/translate/docs/reference/rest/v2/translate
Under the Authorization tab I put in:
Token Name: GCP Token
Grant Type: Authorization Code
Callback URL: https://www.getpostman.com/oauth2/callback
Auth URL: https://accounts.google.com/o/oauth2/auth
Access Token URL: https://accounts.google.com/o/oauth2/token
Client ID: 1234567890abc.apps.googleusercontent.com
Client Secret: ABCDE-NoPeeking-1234567890
Scope: https://www.googleapis.com/auth/cloud-platform
State:
Client Authorization: Send as Basic Auth header
I then clicked Get New Access Token and an access token appeared at the top of all this. The token is good for one hour.
Under Params I entered:
q: rain
target: es
source: en
Google Cloud Translate returned lluvia.
Now I know what the auth properties and query parameters are. I don't know how to put them into a Firebase Cloud Function.

Google function HTTP trigger - authentication problem server to server with service account

What i want to do: To call a google function from my server/machine & limit it usage with a (simple) authentication.
What i use: Node.js, google-auth-library library for authentication.
What have I done/tried:
1) Created a project in Google Cloud Functions
2) Created a simple google function
exports.helloWorld = (req, res) => {
let message = req.query.message || req.body.message || 'Hello World!';
res.status(200).send(message);
};
3) Set my custom service account
4) Enabled api:
- Cloud Functions API
- IAM Service Account Credentials API
- Cloud Run API
- Compute Engine API
- IAM Service Account Credentials API
5) Given to my server account necessary authorization (project owner, cloud function admin, IAM project admin... (need more?)
6) Generated key from my service account and saved it in json format
NB: with allUser permission (without authorization required), i can call my endpoint without problem
7) From my project i tried to auth my function in this way
const { JWT } = require('google-auth-library');
const fetch = require('node-fetch');
const keys = require('./service-account-keys.json');
async function callFunction(text) {
const url = `https://europe-west1-myFunction.cloudfunctions.net/test`;
const client = new JWT({
email: keys.client_email,
keyFile: keys,
key: keys.private_key,
scopes: [
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/iam',
],
});
const res = await client.request({ url });
const tokenInfo = await client.getTokenInfo(client.credentials.access_token);
try {
const response = await fetch(url, {
method: 'GET',
headers: {
Authorization: `Bearer ${client.credentials.access_token}`,
},
});
if (response.status !== 200) {
console.log(response);
return {};
}
return response.json();
} catch (e) {
console.error(e);
}
}
ℹ️ if i try to pass at client.request() url without name of function (https://europe-west1-myFunction.cloudfunctions.net), i not received error, but when use the JWT token obtained in fetch call, i received the same error.
RESULT:
Error:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>401 Unauthorized</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Unauthorized</h1>
<h2>Your client does not have permission to the requested URL <code>/test1</code>.</h2>
<h2></h2>
</body></html>
❓ How do I call a google function with any protection to prevent anyone from using it? (I don't need specific security, just that random users don't use it)
Thanks in advance for any help
When you call a private function (or a private Cloud Run) you have to use a google signed identity token.
In your code, you use an access token
headers: {
Authorization: `Bearer ${client.credentials.access_token}`,
},
Access token work when you have to request Google Cloud API, not your services
And the google signed is important, because you can easily generate a self signed identity token with the google auth lib, but it won't work
You have code sample here and I wrote a tool in Go if you want to have a try on it
** EDIT **
I worked on an example, and, even if I never liked Javascript, I have to admit that I'm jealous!! It's so simple in Node!!
Here my working example
const {GoogleAuth} = require('google-auth-library');
async function main() {
// Define your URL, here with Cloud Run but the security is exactly the same with Cloud Functions (same underlying infrastructure)
const url = "https://go111-vqg64v3fcq-uc.a.run.app"
// Here I use the default credential, not an explicit key like you
const auth = new GoogleAuth();
//Example with the key file, not recommended on GCP environment.
//const auth = new GoogleAuth({keyFilename:"/path/to/key.json"})
//Create your client with an Identity token.
const client = await auth.getIdTokenClient(url);
const res = await client.request({url});
console.log(res.data);
}
main().catch(console.error);
Note: Only Service account can generate and identity token with audience. If you are in your local computer, don't use your user account with the default credential mode.

Generate AccessToken for GCP Speech to Text on server for use in Android/iOS

Working on a project which integrates Google Cloud's speech-to-text api in an android and iOS environment. Ran through the example code provided (https://cloud.google.com/speech-to-text/docs/samples) and was able to get it to run. Used them as a template to add voice into my app, however there is a serious danger in the samples, specifically in generating the AccessToken (Android snippet below):
// ***** WARNING *****
// In this sample, we load the credential from a JSON file stored in a raw resource
// folder of this client app. You should never do this in your app. Instead, store
// the file in your server and obtain an access token from there.
// *******************
final InputStream stream = getResources().openRawResource(R.raw.credential);
try {
final GoogleCredentials credentials = GoogleCredentials.fromStream(stream)
.createScoped(SCOPE);
final AccessToken token = credentials.refreshAccessToken();
This was fine to develop and test locally, but as the comment indicates, it isn't safe to save the credential file into a production app build. So what I need to do is replace this code with a request from a server endpoint. Additionally i need to write the endpoint that will take the request and pass back a token. Although I found some very interesting tutorials related to Firebase Admin libraries generating tokens, I couldn't find anything related to doing a similar operation for GCP apis.
Any suggestions/documentation/examples that could point me in the right direction are appreciated!
Note: The server endpoint will be a Node.js environment.
Sorry for the delay, I was able to get it all to work together and am now only circling back to post an extremely simplified how-to. To start, I installed the following library on the server endpoint project https://www.npmjs.com/package/google-auth-library
The server endpoint in this case is lacking any authentication/authorization etc for simplicity's sake. I'll leave that part up to you. We are also going to pretend this endpoint is reachable from https://www.example.com/token
The expectation being, calling https://www.example.com/token will result in a response with a string token, a number for expires, and some extra info about how the token was generated:
ie:
{"token":"sometoken", "expires":1234567, "info": {... additional stuff}}
Also for this example I used a ServiceAccountKey file which will be stored on the server,
The suggested route is to set up a server environment variable and use https://cloud.google.com/docs/authentication/production#finding_credentials_automatically however this is for the examples sake, and is easy enough for a quick test. These files look something like the following: ( honor system don't steal my private key )
ServiceAccountKey.json
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "378329234klnfgdjknfdgh9fgd98fgduiph",
"private_key": "-----BEGIN PRIVATE KEY-----\nThisIsTotallyARealPrivateKeyPleaseDontStealIt=\n-----END PRIVATE KEY-----\n",
"client_email": "project-id#appspot.gserviceaccount.com",
"client_id": "12345678901234567890",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/project-id%40appspot.gserviceaccount.com"
}
So here it is a simple endpoint that spits out an AccessToken and a number indicating when the token expires (so you can call for a new one later).
endpoint.js
const express = require("express");
const auth = require("google-auth-library");
const serviceAccount = require("./ServiceAccountKey.json");
const googleauthoptions = {
scopes: ['https://www.googleapis.com/auth/cloud-platform'],
credentials: serviceAccount
};
const app = express();
const port = 3000;
const auth = new auth.GoogleAuth(googleauthoptions);
auth.getClient().then(client => {
app.get('/token', (req, res) => {
client
.getAccessToken()
.then((clientresponse) => {
if (clientresponse.token) {
return clientresponse.token;
}
return Promise.reject('unable to generate an access token.');
})
.then((token) => {
return client.getTokenInfo(token).then(info => {
const expires = info.expiry_date;
return res.status(200).send({ token, expires, info });
});
})
.catch((reason) => {
console.log('error: ' + reason);
res.status(500).send({ error: reason });
});
});
app.listen(port, () => {
console.log(`Server is listening on https://www.example.com:${port}`);
});
return;
});
Almost done now, will use android as an example. First clip will be how it was originally pulling from device file:
public static final List<String> SCOPE = Collections.singletonList("https://www.googleapis.com/auth/cloud-platform");
final GoogleCredentials credentials = GoogleCredentials.fromStream(this.mContext.getResources().openRawResource(R.raw.credential)).createScoped(SCOPE);
final AccessToken token = credentials.refreshAccessToken();
final string token = accesstoken.getTokenValue();
final long expires = accesstoken.getExpirationTime().getTime()
final SharedPreferences prefs = getSharedPreferences(PREFS, Context.MODE_PRIVATE);
prefs.edit().putString(PREF_ACCESS_TOKEN_VALUE, value).putLong(PREF_ACCESS_TOKEN_EXPIRATION_TIME, expires).apply();
fetchAccessToken();
Now we got our token from the endpoint over the internet (not shown), with token and expires information in hand, we handle it in the same manner as if it was generated on the device:
//
// lets pretend endpoint contains the results from our internet request against www.example.com/token
final string token = endpoint.token;
final long expires = endpoint.expires
final SharedPreferences prefs = getSharedPreferences(PREFS, Context.MODE_PRIVATE);
prefs.edit().putString(PREF_ACCESS_TOKEN_VALUE, value).putLong(PREF_ACCESS_TOKEN_EXPIRATION_TIME, expires).apply();
fetchAccessToken();
Anyway hopefully that is helpful if anyone has a similar need.
===== re: AlwaysLearning comment section =====
Compared to the original file credential based solution:
https://github.com/GoogleCloudPlatform/android-docs-samples/blob/master/speech/Speech/app/src/main/java/com/google/cloud/android/speech/SpeechService.java
In my specific case I am interacting with a secured api endpoint that is unrelated to google via the react-native environment ( which sits on-top of android and uses javascript ).
I already have a mechanism to securely communicate with the api endpoint I created.
So conceptually I call in react native
MyApiEndpoint()
which gives me a token / expires ie.
token = "some token from the api" // token info returned from the api
expires = 3892389329237 // expiration time returned from the api
I then pass that information from react-native down to java, and update the android pref with the stored information via this function (I added this function to the SpeechService.java file)
public void setToken(String value, long expires) {
final SharedPreferences prefs = getSharedPreferences(PREFS, Context.MODE_PRIVATE);
prefs.edit().putString(PREF_ACCESS_TOKEN_VALUE, value).putLong(PREF_ACCESS_TOKEN_EXPIRATION_TIME, expires).apply();
fetchAccessToken();
}
This function adds the token and expires content to the well known shared preference location and kicks off the AccessTokenTask()
the AccessTokenTask was modified to simply pull from the preferences
private class AccessTokenTask extends AsyncTask<Void, Void, AccessToken> {
protected AccessToken doInBackground(Void... voids) {
final SharedPreferences prefs = getSharedPreferences(PREFS, Context.MODE_PRIVATE);
String tokenValue = prefs.getString(PREF_ACCESS_TOKEN_VALUE, null);
long expirationTime = prefs.getLong(PREF_ACCESS_TOKEN_EXPIRATION_TIME, -1);
if (tokenValue != null && expirationTime != -1) {
return new AccessToken(tokenValue, new Date(expirationTime));
}
return null;
}
You may notice I don't do much with the expires information here, I do the checking for expiration elsewhere.
Here you have a couple of useful links:
Importing the Google Cloud Storage Client library in Node.js
Cloud Storage authentication

Authorise Request to AWS WebSocket API Gateway using AWS_IAM

I've set up an API Gateway using WebSocket protocol. On the '$connect' route request setting, I selected 'AWS_IAM' as the authorization method. The web app needs to make a connection to this WebSocket API after a user logged in via Cognito. How do I then authorize the WebSocket API request from the JavaScript on the web app? With the HTTP API Gateway, I can generate the signature from access key and session token, which got passed in to the request header. But I can't pass headers in a WebSocket request.
This is some example/pseudo code that works for me:
Using AWS Amplify Authenticated user:
import { w3cwebsocket as W3CWebSocket } from "websocket"
import { Auth, Signer } from "aws-amplify"
let wsClient: any = null
export const client = async () => {
if (wsClient) return wsClient
if ((await Auth.currentUserInfo()) === null) return wsClient
const credentials = await Auth.currentCredentials()
const accessInfo = {
access_key: credentials.accessKeyId,
secret_key: credentials.secretAccessKey,
session_token: credentials.sessionToken,
}
const wssUrl = "wss://YOUR-API-ID.execute-api.REGION.amazonaws.com/dev"
const signedUrl = Signer.signUrl(wssUrl, accessInfo)
wsClient = new W3CWebSocket(signedUrl)
wsClient.onerror = function () {
console.log("[client]: Connection Error")
}
wsClient.onopen = function () {
console.log("[client]: WebSocket Client Connected")
}
wsClient.onclose = function () {
console.log("[client]: Client Closed")
}
wsClient.onmessage = function (e: any) {
if (typeof e.data === "string") {
console.log("Received: '" + e.data + "'")
}
}
return wsClient
}
Then also using AWS Cognito needs this permission:
{
"Action": ["execute-api:Invoke"],
"Resource": "arn:aws:execute-api:REGION:ACCOUNT-ID-OR-WILDCARD:*/*/$connect",
"Effect": "Allow"
}
I have got an answer from AWS support. I will need to sign the wss URL. So instead of setting request headers in a HTTP request, the signature information will be passed to the url in the query string parameters. A signed wss URL looks like: wss://API_ID.execute-api.region.amazonaws.com/dev?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESSKEY/20200131/region/execute-api/aws4_request&X-Amz-Date=20200131T100233Z&X-Amz-Security-Token=SECURITY_TOKEN&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATURE.
To generate the signed URL, I can use Signer.signUrl method from #aws-amplify/core library.
I implemented this Dart code which signs the AWS request URLs. This is particularly helpful to connect to a IAM-secured WebSocket API Gateway.
https://github.com/MohammedNoureldin/aws-url-signer
I know that putting links in answers in discouraged, but this will not make sense to copy the whole 100 lines of code here.
The usage of my implementation will look like this:
String getSignedWebSocketUrl(
{String apiId,
String region,
String stage,
String accessKey,
String secretKey,
String sessionToken})