I am trying to create projects programatically through the resource manager API from a google cloud function like so:
exports.createProjectAsync = async (projectId, projectName) => {
const scopes = "https://www.googleapis.com/auth/cloud-platform"
const url = `http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token?scopes=${scopes}`
const tokenResult = await fetch(url, {
headers: {
"Metadata-Flavor": "Google"
},
});
const tokenStatus = tokenResult.status;
functions.logger.log(`Call to get token has status ${tokenStatus}`);
const tokenData = await tokenResult.json()
functions.logger.log(`Call to get token has data ${JSON.stringify(tokenData)}`);
const accessToken = tokenData.access_token;
if (accessToken === null) {
throw new Error(`Failed to retrieve access token`);
}
const headers = {
'Content-Type': 'application/json',
'Authorization': `Bearer ${accessToken}`
};
const payload = {
"projectId": projectId,
"name": projectName,
"parent": {
"type": "folder",
"id": FOLDER_NUMBER
}
};
const projectsCreateUrl = `https://cloudresourcemanager.googleapis.com/v1/projects/`
const result = await fetch(projectsCreateUrl, {
method: 'POST',
headers: headers,
body: JSON.stringify(payload)
});
const status = result.status;
functions.logger.log(`Call to create project returned status ${status}`);
const data = await result.json()
functions.logger.log(`data: ${JSON.stringify(data)}`);
return data;
}
For testing purposes I've added the Organization Administrator role to the default service account. I cannot see the projects creator role in IAM:
When calling the API I get the following error:
{"error":{"code":403,"message":"The caller does not have permission","status":"PERMISSION_DENIED"}}
How can I successfully access this resource?
Although of course, it gives you the ability to modify its own permissions, as you can verify in the GCP documentation, the Organization Admin role does not allow to create a new project.
As you indicated, for that purpose the service account should be granted the Project Creator (roles/resourcemanager.projectCreator) role.
According to your screenshot, you are trying to grant this permission at the project level, but please, be aware that this role can only be granted at the organization and folder levels. This is the reason why the dropdown menu in the Google Cloud Web console is not providing you the Project Creator option.
If you have the necessary permissions over the folder or organization, try to assign that role at the corresponding level.
Related
Basically I'm trying to get a pre-signed URL for the putObject method in S3. I send this link back to my frontend, which then uses it to upload the file directly from the client.
Here's my code :
const AWS = require("aws-sdk");
const s3 = new AWS.S3({
accessKeyId: process.env.AWS_IAM_ACCESS,
secretAccessKey: process.env.AWS_IAM_SECRET,
region: 'ap-southeast-1',
});
const getPreSignedUrlS3 = async (data) => {
try {
//DO SOMETHING HERE TO GENERATE KEY
const options = {
Bucket: process.env.AWS_USER_CDN,
ContentType: data.type,
Key: key,
Expires: 5 * 60
};
return new Promise((resolve, reject) => {
s3.getSignedUrl(
"putObject", options,
(err, url) => {
if (err) {
reject(err);
}
else resolve({ url, key });
}
);
});
} catch (err) {
console.log(err)
return {
status: 500,
msg: 'Failed to sync with CDN, Please try again later!',
}
}
}
I'm getting the following error from the aws sdk : The security token included in the request is invalid.
Things I have tried :
Double check the permissions from my IAM user. Even made bucket access public for testing. My IAM user is given full s3 access policy.
Tried using my root user security key and access details. Still got the same error.
Regenerated new security credentials for my IAM user. I don't have any MFA turned on.
I'm following this documentation.
SDK Version : 2.756.0
I've been stuck on this for a while now. Any help is appreciated. Thank you.
Pre-signed URLs are created locally in the SDK so there's no need to use the asynchronous calls.
Instead, use a synchronous call to simplify your code, something like this:
const getPreSignedUrlS3 = (Bucket, Key, ContentType, Expires = 5 * 60) => {
const params = {
Bucket,
ContentType,
Key,
Expires
};
return s3.getSignedUrl("putObject", params);
}
I have this code:
import {v2beta3} from "#google-cloud/tasks";
const project = 'xxxxxxx'
const location = 'yyyyyyy'
const queue = 'zzzzzzzzz'
const client = new v2beta3.CloudTasksClient()
const parent = client.queuePath(project, location, queue)
const payload = {eventId: "fred"}
const convertedPayload = JSON.stringify(payload)
const body = Buffer.from(convertedPayload).toString('base64');
const task = {
httpRequest: {
httpMethod: "POST",
url: "https://webhook.site/9sssssssssss",
oidcToken: {
serviceAccountEmail: "aaaaaaaaaa#appspot.gserviceaccount.com",
},
headers: {
'Content-Type': 'application/json',
},
body,
},
};
(async function() {
try {
const [response] = await client.createTask({parent, task});
console.log(`Created task ${response.name}`);
} catch (error) {
console.log(error)
}
}());
When I run it from my laptop, it just works, which seems unauthenticated to me. Anyone can now enqueue a task on my queue.
What is the right way to authenticate to the GCP Cloud Tasks enqueue API?
As John Hanley pointed out in the comments, my local app was using Application Default Credentials to authenticate itself. When I switched to a different gcloud account by doing this:
gcloud auth application-default login
I get this error message when I try to run the code:
Error: 7 PERMISSION_DENIED: The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/yyyyyyy/locations/europe-west1/queues/default-xxxxxx" (or the resource may not exist).
I'm trying to use Google cloudtasks from Cloudflare workers.
This is a JS environment limited to web-workers standards minus some things that Cloudflare didn't implement.
Bottom line - I can't use Google's provided SDKs in that environment.
I'm trying to call the API using simple fetch, but always fail on the authentication part.
The discovery document says that
"parameters": {
...
"key": {
"description": "API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.",
"location": "query",
"type": "string"
}
}
So I tried calling the api with ?key=MY_API_KEY query param
Didn't work.
I also tried generating a token using Service Account downloaded json file with this library
Didn't work.
I tried following this guide to generate oauth access token which was what the error message told me that I need. But
running the command gcloud auth application-default print-access-token returned the error:
WARNING: Compute Engine Metadata server unavailable onattempt 1 of 3. Reason: timed out
WARNING: Compute Engine Metadata server unavailable onattempt 2 of 3. Reason: timed out
WARNING: Compute Engine Metadata server unavailable onattempt 3 of 3. Reason: [Errno 64] Host is down
WARNING: Authentication failed using Compute Engine authentication due to unavailable metadata server.
ERROR: (gcloud.auth.application-default.print-access-token) Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
The env variable above is set correctly to the service-account json file.
Even if it worked, I didn't understand how am I supposed to use it from my code, while it uses the cli tool gcloud
So my question is this - how can I access Google's cloud APIs from Cloudflare workers (web-workers javascript env.), specifically I'm interested in Cloudtasks, without using any CLI tool or Google SDK.
More specifically - how can I generate the required oauth2 access token?
Based on #john-hanley blog post I was able to make the following code work:
const fetch = require('node-fetch'); //in cloudflare workers env. this is not needed. 'fetch' is globally available
const jwt = require('jsonwebtoken');
const q = require('querystring');
async function main() {
const project = 'xxxx';
const location = 'us-central1';
const scopes = "https://www.googleapis.com/auth/cloud-platform"
const queue = 'queueName';
const parent = `projects/${project}/locations/${location}/queues/${queue}`
const url = `https://cloudtasks.googleapis.com/v2/${parent}/tasks`
const sjwt = await createSignedJwt(json.private_key, json.private_key_id, json.client_email, scopes);
const {token} = await exchangeJwtForAccessToken(sjwt)
const headers = { Authorization: `Bearer ${token}` }
const body = Buffer.from(JSON.stringify({"c":"b"})).toString('base64'); //Note this is not a string!
const task = { //in my case the task is HTTP request that google will send to my service outside GCP. You can create an appEngine task instead
httpRequest: {
"url": "https://where-google-should-send-the-task.com",
"httpMethod": "POST",
"headers": {
"Content-Type": "application/json; charset=UTF-8",
"Authorization": "only-if-you-need-it"
},
"body": body
}
};
const request = {
parent: parent,
task: task
};
const response = await fetch(url, {
method: "POST",
body: JSON.stringify(request),
headers
})
const res = await response.json()
console.log(res)
}
async function createSignedJwt (pkey, pkey_id, email, scope) {
const authUrl = "https://oauth2.googleapis.com/token"
const options = {
algorithm: "RS256",
keyid: pkey_id,
expiresIn: 3600,
audience: authUrl,
issuer: email
// header: { //this is not needed because it's jsonwebtoken's default behavior to add the correct typ when the payload is a json
// "typ": "JWT"
// }
}
const payload = {
"scope": scope
}
return jwt.sign(payload, pkey, options)
}
/**
* This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
*/
async function exchangeJwtForAccessToken(signedJwt) {
const authUrl = "https://oauth2.googleapis.com/token"
const params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signedJwt
}
const body = q.stringify(params);
const res = await fetch(authUrl, {
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded"
},
body
})
if (!res.ok) {
return {
error: "Could not fetch access token. " + await res.text()
}
}
const resJson = await res.json();
return {
token: resJson.access_token
}
}
// for convenience, I'm placing the JSON here. For production it should be stored in secret-manager or injected via environment variable
const json = {
"type": "service_account",
"project_id": "xxxx",
"private_key_id": "xxxx",
"private_key": "-----BEGIN PRIVATE KEY-----xxxx-----END PRIVATE KEY-----\n",
"client_email": "xxxx#xxxx.iam.gserviceaccount.com",
"client_id": "xxxx",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxx%40xxxx.iam.gserviceaccount.com"
}
main()
What I want to do?
I want to create REST API that returns data from my DynamoDB table which is being created by GraphQL model.
What I've done
Create GraphQL model
type Public #model {
id: ID!
name: String!
}
Create REST API with Lambda Function with access to my PublicTable
$ amplify add api
? Please select from one of the below mentioned services: REST
? Provide a friendly name for your resource to be used as a label for this category in the project: rest
? Provide a path (e.g., /book/{isbn}): /items
? Choose a Lambda source Create a new Lambda function
? Provide an AWS Lambda function name: listPublic
? Choose the runtime that you want to use: NodeJS
? Choose the function template that you want to use: Hello World
Available advanced settings:
- Resource access permissions
- Scheduled recurring invocation
- Lambda layers configuration
? Do you want to configure advanced settings? Yes
? Do you want to access other resources in this project from your Lambda function? Yes
? Select the category storage
? Storage has 8 resources in this project. Select the one you would like your Lambda to access Public:#model(appsync)
? Select the operations you want to permit for Public:#model(appsync) create, read, update, delete
You can access the following resource attributes as environment variables from your Lambda function
API_MYPROJECT_GRAPHQLAPIIDOUTPUT
API_MYPROJECT_PUBLICTABLE_ARN
API_MYPROJECT_PUBLICTABLE_NAME
ENV
REGION
? Do you want to invoke this function on a recurring schedule? No
? Do you want to configure Lambda layers for this function? No
? Do you want to edit the local lambda function now? No
Successfully added resource listPublic locally.
Next steps:
Check out sample function code generated in <project-dir>/amplify/backend/function/listPublic/src
"amplify function build" builds all of your functions currently in the project
"amplify mock function <functionName>" runs your function locally
"amplify push" builds all of your local backend resources and provisions them in the cloud
"amplify publish" builds all of your local backend and front-end resources (if you added hosting category) and provisions them in the cloud
Succesfully added the Lambda function locally
? Restrict API access No
? Do you want to add another path? No
Successfully added resource rest locally
Edit my Lambda function
/* Amplify Params - DO NOT EDIT
API_MYPROJECT_GRAPHQLAPIIDOUTPUT
API_MYPROJECT_PUBLICTABLE_ARN
API_MYPROJECT_PUBLICTABLE_NAME
ENV
REGION
Amplify Params - DO NOT EDIT */
const AWS = require("aws-sdk");
const region = process.env.REGION
AWS.config.update({ region });
const docClient = new AWS.DynamoDB.DocumentClient();
const params = {
TableName: "PublicTable"
}
async function listItems(){
try {
const data = await docClient.scan(params).promise()
return data
} catch (err) {
return err
}
}
exports.handler = async (event) => {
try {
const data = await listItems()
return { body: JSON.stringify(data) }
} catch (err) {
return { error: err }
}
};
Push my updates
$ amplify push
Open my REST API endpoint /items
{
"message": "User: arn:aws:sts::829736458236:assumed-role/myprojectLambdaRolef4f571b-dev/listPublic-dev is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:us-east-1:8297345848236:table/Public-ssrh52tnjvcdrp5h7evy3zdldsd-dev",
"code": "AccessDeniedException",
"time": "2021-04-21T21:21:32.778Z",
"requestId": "JOA5KO3GVS3QG7RQ2V824NGFVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 28.689093010346657
}
Problems
What I did wrong?
How do I access my table and why I didn't get it when I created it?
Why API_MYPROJECT_PUBLICTABLE_NAME and other constants are needed?
Decision
The problem turned out to be either the NodeJS version or the amplify-cli version. After updating amplify-cli and installing the node on the 14.16.0 version, everything worked.
I also changed the name of the table to what Amplify creates for us, although this code did not work before. The code became like this:
/* Amplify Params - DO NOT EDIT
API_MYPROJECT_GRAPHQLAPIIDOUTPUT
API_MYPROJECT_PUBLICTABLE_ARN
API_MYPROJECT_PUBLICTABLE_NAME
ENV
REGION
Amplify Params - DO NOT EDIT */
const AWS = require("aws-sdk");
const region = process.env.REGION
const tableName = process.env.API_MYPROJECT_PUBLICTABLE_NAME
AWS.config.update({ region });
const docClient = new AWS.DynamoDB.DocumentClient();
const params = {
TableName: tableName
}
async function listItems(){
try {
const data = await docClient.scan(params).promise()
return data
} catch (err) {
return err
}
}
exports.handler = async (event) => {
try {
const data = await listItems()
return { body: JSON.stringify(data) }
} catch (err) {
return { error: err }
}
};
I tried with the aws-sdk-react-native module:
https://github.com/awslabs/aws-sdk-react-native
The configuration took some time, but thanks to this links I could i.e. list the topics:
https://github.com/awslabs/aws-sdk-react-native/issues/35
https://github.com/awslabs/aws-sdk-react-native/blob/master/SNS/IntegrationTests/SNSTests.js
The test includes a sample how to subscribe to get emails but not how to get notifications in the app. I don't know how to get the platformEndpoint, the PlatformApplicationArn and the deviceToken.
endPoint = sns.createPlatformEndpoint({
PlatformApplicationArn: '{APPLICATION_ARN}',
Token: '{DEVICE_TOKEN}'
})
...
var subscribeRequest= {
"Protocol":"application",
"TopicArn":topicARN,
"Endpoint":endPoint
}
try{
await AWSSNS.Subscribe(subscribeRequest);
}catch(e){
console.error(e);
shouldResolve = false;
return shouldResolve;
}
Are there any samples for this?
I'm also looking for an authentication sample.
Would it be easier to use firebase?
Thanks
I have used GCM over SNS to send notifications. Here are the steps I went through assuming you already set up GCM and added required libraries from AWS React Native SDK:
First create a SNS app from AWS:
Then you need to create Federated Identity through Cognito service of AWS. This is required for sending your device token from mobile app to AWS SNS app. Choose Manage Federated Identities
Then create your pool, don't forget to check Enable Access to unauthenticated identities
When you create the pool you will need to create IAM Roles for unauthenticated and authenticated roles of that pool. AWS will help you create new roles for that but you need to go to IAM Roles menu and attach AmazonSNSFullAccess to created roles, otherwise from the mobile app you won't able to send device token.
After doing these steps you will able send your device token using Amazon's React Native SDK. I have written a helper class for sending token to AWS SNS as suggested here:
class AWSUtility {
constructor() {
const region = "us-west-1"; //change it with your region
const IDENTITY_POOL_ID = "pool id created from Federated Identities"
AWSCognitoCredentials.initWithOptions({region, identity_pool_id: IDENTITY_POOL_ID});
AWSSNS.initWithOptions({region});
}
addTokenToAWSSNS(token, snsEndpointARN) {
const applicationArn = "change with SNS application Amazon resource name";
return Promise.try(() => {
if (!snsEndpointARN) {
return this.createPlatformEndpoint(token, applicationArn);
} else {
return AWSSNS.GetEndpointAttributes({EndpointArn: snsEndpointARN})
.then((result) => {
const {Attributes = {}} = result;
const {Token, Enabled} = Attributes;
const updateNeeded = Token !== token || Enabled !== 'true';
if (updateNeeded) {
return this.updateEndpoint(token).then(() => result.EndpointArn);
}
return snsEndpointARN;
})
.catch(() => {
this.createPlatformEndpoint(token, applicationArn)
});
}
});
}
updateEndpoint(snsEndpointARN, token) {
//AWS is returning error saying that it requires 6 params to update endpoint, if anyone has any idea about it let me know please
return AWSSNS.SetEndpointAttributes({EndpointArn: snsEndpointARN, Attributes: {Token: token, Enabled: true}});
}
createPlatformEndpoint(token, applicationArn) {
return AWSSNS.CreatePlatformEndpoint({Token: token, PlatformApplicationArn: applicationArn})
.then(result => result.EndpointArn)
.catch((error = {}) => {
console.log(error);
});
}
}
export default new AWSUtility();