I'm new to Google APIs and I've been trying for days to use a service account to upload content to a Google Cloud Storage bucket. I'm able to accomplish this, but only with a temporary access token obtained from the Google API playground, and I need to be able to get new access tokens so the service account can always upload content.
I've been experimenting with the following, but I keep getting access denied, even though the account in question has 'owner' permissions.
curl -X POST / -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \ -H "Content-Type: application/json; charset=utf-8" \ -d #Documents/request.json \ https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/content-uploader#kol-hatorah-kulah.iam.gserviceaccount.com:generateAccessToken
response:
{ "error": { "code": 403, "message": "The caller does not have permission", "status": "PERMISSION_DENIED" } }
When I run gcloud config list I get the correct project, and the account is my work email, which is also in Google Cloud as an owner.
Thanks in advance!
DISCLAIMER - my solution works for Workload Identity Federation
related problems.
I've had hard time with this error, but finally found it!
For me it was wrong attribute mapping.
I was following some tutorial (which probably went outdated) and mapping was different than from official github action task documentation (here)
I had repository_owner and aud. Changed it for repository and...
It works!
To sum up my mapping looks like this:
"google.subject" = "assertion.sub"
"attribute.actor" = "assertion.actor"
"attribute.repository" = "assertion.repository"
So if you got here because of same tutorial... you've been served!
Your curl command is attempting to use a service account identity to generate an Access Token. The command is failing because you do not have permission.
Add the role roles/iam.serviceAccountTokenCreator to the identity running the command.
Related
Reverse-engineering the gcloud tool—with the suggested --log-http mostly—I was able to get auth tokens generated, and even selected the same scope (double checking at https://www.googleapis.com/oauth2/v1/tokeninfo):
http://[redacted]/callback?state=[redacted]&code=[redacted]&scope=email+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&authuser=0&prompt=consent
https://oauth2.googleapis.com:443/token?grant_type=authorization_code&code=[redacted]&redirect_uri=[redacted]/callback&client_id=[redacted]apps.googleusercontent.com&client_secret=[redacted]
Headers I try to hit the /zones/list endpoint with:
Content-Type: application/json
charset: utf-8
Authorization: Bearer [redacted]
X-Goog-User-Project: [redacted]
I've also tried putting access_token=[redacted] in my query string. But not matter what I do, I always get:
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}
Other endpoints like like /projects/list seem to work with this access token (in header). What am I doing wrong?
As I see from the error message, you got 401 http status code. It means that the access token you used does not give you access (is not authorised) to list zones. And here you need to know that you cannot compare zone.list and projects.list. In fact:
zone.list is part of compute engine API, whereas projects.list is part of resource-manager API
zone.list requires compute.zones.list permission that is part of Compute Engine roles, like Compute Viewer whereas projects.list requires resourcemanager.projects.list which you'll find with many roles even the Browser role which is the smallest role (I think) in terms of number of permission.
To summarise, typically you used an access token of an identity which has access to projects.list but not to zone.list. Example this identity has only Browser role.
Then, there this sentence that caught my attention on how you are getting the access token:
Reverse-engineering the gcloud tool
You need to know that getting an access token is quite easy.
Run gcloud auth login which obtains access credentials for your user account via a web-based authorization flow.
Run gcloud auth print-access-token which prints an access token for your user account, so his permissions.
So if your user account has a role with compute.zones.list and resourcemanager.projects.list permissions (for example Compute Viewer role has both), you will be able to call both endpoints successfully.
Finally, here is a curl example, listing zones, using the generated access token:
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" https://compute.googleapis.com/compute/v1/projects/<PROJECT_ID>/zones
Turns out my original solution was 100% correct. For some odd reason the string being saved in the access token field of my struct was different to the one gathered from the remote auth flow.
I'm following https://cloud.google.com/vision/docs/quickstart-cli
I've created a Google Cloud account, created a project, enabled Vision API, setup billing.
I now execute the cURL:
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-H "X-Goog-User-Project: dragon-ocr-324006" \
https://vision.googleapis.com/v1/images:annotate -d #request.json
I get the response:
{
"error": {
"code": 403,
"message": "Caller does not have required permission to use project dragon-ocr-324006. Grant the caller the Owner or Editor role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=dragon-ocr-324006 and then retry (propagation of new permission may take a few minutes).",
"status": "PERMISSION_DENIED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Google developer console IAM admin",
"url": "https://console.developers.google.com/iam-admin/iam/project?project=dragon-ocr-324006"
}
]
},
{
"#type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "USER_PROJECT_DENIED",
"domain": "googleapis.com",
"metadata": {
"consumer": "projects/dragon-ocr-324006",
"service": "vision.googleapis.com"
}
}
]
}
}
What does this mean? Who is the 'caller'?
When I do gcloud auth application-default login it lets me log in as, I guess, root user for my gcloud. And that must be the caller...?
So, I click that link, and get:
Permissions error... great! And now other pages give the same perms-error. So I have to repoint my browser to https://console.cloud.google.com/ and go in manually.
So, both root-user and project-user (if I got that right) have Owner permission.
So what is the problem.
Maybe my local machine doesn't have the updated profile for the project-user?
ok, so rm -rf ~/.config/gcloud and gcloud auth application-default login
Quick test: gcloud auth application-default print-access-token gives me an access token, great!
I rerun my crl.sh script and get the same problem.
Now here's the kicker. I've got another gcloud account I just created today, and if I run it on that one, it completes fine!
So what am I doing wrong on the first account?
Bear with me because you asked different questions :)
First:
Who is the 'caller'? When I do gcloud auth application-default login it lets me log in as, I guess, root user for my gcloud. And that must be the caller...?
It's normal that the error message is not referring a specific caller / identity. In fact, you are using an access token in your curl through gcloud auth application-default print-access-token. Access tokens are used to inform an API that the bearer of the token has been authorised to access the API. It doesn't hold any identity information.
Second:
That access token has been generated for you based on the credentials, you already setup as default credentials. You get these credentials in 2 ways
you run export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-file.json" But I think you didn't use this since you didn't evoke any key file.
you run gcloud auth application-default login, you had to go through a web flow, and the creadentials are generated and stored under ~/.config/gcloud/application_default_credentials.json
Third:
So, both root-user and project-user (if I got that right) have Owner permission.
ok, so rm -rf ~/.config/gcloud and gcloud auth application-default login
Here I understand that you changed the roles. So the initial role (set of permissions) given to the identity for which you generated the default credentials, was not enough.
Fourth:
You gave both users a large set of permissions (the Owner role has almost all permissions) Then you regenerated the default credentials.
But it did not work: because as stated in error message : (propagation of new permission may take a few minutes)
Finally:
When you came back with a new account it did work because he was already set with proper permissions. But if you retry with the old one it will work also, of course if you did not change his Owner role.
I had almost similar kind of issue when I was working on a Cloud Function in VS Code local development.In my case Application Default Credentials (ADC) was pointing to different service account which didn't had enough permission.
I followed the below steps to resolve this issue:
Executed these commands (locally) in Google Cloud SDK Shell
First list the current configuration to verify the config values
gcloud config list
This will output:
[accessibility]
screen_reader = False
[core]
account = <service_account>#appspot.gserviceaccount.com
disable_usage_reporting = True
project = ProjectId
Then I had to run a command to list all Credentialed Accounts
gcloud auth list
This will output the service accounts and an asterisk (*) at the beginning of an account which is active.
If this active service account is different than what is expected then must
activate the respective service account by running the following command.
gcloud auth activate-service-account <different_service_account>#appspot.gserviceaccount.com --key-file=PATH_TO_SERVICE_ACCOUNT_KEY_FILE
More details can be found here
Then restart the application resolved my issue.
Though this is not exactly the solution for this question, but definitely may help anyone who got into this error.
I am trying to use Cognito User Pool to authenticate with a PC application using an HTTPS call. I want to obtain the various tokens that I can then use to access the AWS resources without storing AWS secrets in the PC application.
The AWS documentation documents the InitiateAuth method and shows the AWS Endpoints, but it is not immediately apparent how to make the call over HTTPS. Most calls would require an AWS signature, but the InitiateAuth call should not, if I am just submitting Username and Password.
After some poking around, I was able to use the AWS CLI to successfully obtain tokens with this command:
aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id the_cognito_client_id --auth-parameters USERNAME=the_users_email,PASSWORD=the_users_password
Where the_cognito_client_id is an approximately 26 character long string shown as App client id under General Settings / App clients.
Note that the USER_PASSWORD_AUTH flow is not enabled by default, so you will initially get an error with this. Go to the Cognito Console for the specific User Pool and look for General Settings/App Clients, click "Show Details" for your specific app client, and check the "Enable username password based authentication (ALLOW_USER_PASSWORD_AUTH)" and save.
Once you get back tokens and you know your call is working, you can use the aws history show command to show you the details of the actual https call. The first time you call
aws history show
You will get a message
Could not locate history. Make sure cli_history is set to enabled in the ~/.aws/config file
Go to that file and add
cli_history=enabled
Then, run your initiate-auth call again with the cli. Then, when you run
aws history show
You will get back the details of how the call was made. (At this point, you might consider removing the cli_history setting so you don't log all of your calls, with all of the credentials, in the future.) You will see
to URL: https://cognito-idp.us-east-1.amazonaws.com/
which tells you the URL to use, and you will see that it is a POST. You will note that "InitiateAuth" is not anywhere in that URL. However, you will see the headers include:
"X-Amz-Target": "AWSCognitoIdentityProviderService.InitiateAuth"
and
"Content-Type": "application/x-amz-json-1.1"
You need both of those headers, including the non-standard Content-Type, to make the HTTPS call work. You can use Postman to put the call together, although Postman does not like the non-standard Content-Type, so you have to turn off the standard Content-Type and manually add these two headers to the call. At that point, Postman is able to obtain the tokens as well.
Postman also provides an export to CURL function (click the link that says "Code"), which gives you:
curl --location --request POST 'https://cognito-idp.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityProviderService.InitiateAuth' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"AuthFlow": "USER_PASSWORD_AUTH",
"AuthParameters": {
"PASSWORD": "the_users_password",
"USERNAME": "the_users_email"
},
"ClientId": "the_cognito_client_id"
}'
Submitting that on the command line also gives you the tokens you need.
To refresh using the refresh token, just use InitiateAuth, but the AuthFlow is REFRESH_TOKEN_AUTH and the only member of AuthParameters is REFRESH_TOKEN (which is, of course, the RefreshToken)
Now, I just need to figure out how to do USER_SRP_AUTH using HTTPS.
The other answer explains how to get the Tokens using the Username and Password. Next, we need to get the temporary credentials from the Cognito Identity Pool. This appears to require two steps. First, we need to call cognito-identity get-id and then cognito-identity get-credentials-for-identity
The get-id call requires the Identity Pool ID, which can be obtained from the Cognito Console for the Identity Pool. The identity-pool-id is available under "edit identity pool" as "identity pool ID." The Login key name comes from the User pool ID and is available under Authentication Providers / Cognito under the Edit Identity Pool. The Login key name is actually provided as "iss" in the encoded ID Token. You can use a service like https://jwt.io/ to decode the ID Token, and it will include an entry like
"iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx"
You can use that, without the https://, as the key for Login. The value for Login is the JWT returned as "IdToken" from InitiateAuth.
Therefore, using the CLI, the first call is
aws cognito-identity get-id --identity-pool-id "us-east-1:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" --logins "{\"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx\": \"ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool\"}"
Which returns the ID:
{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555"
}
Then, we need to get the temporary credentials using get-credentials-for-identity. This call uses the IdentityId returned from get-id rather than the Identity Pool ID used by get-id. The Login JSON is the same as for get-id. Therefore, the call is:
aws cognito-identity get-credentials-for-identity --identity-id "us-east-1:11111111-2222-3333-4444-555555555555" --logins "{\"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx\": \"ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool\"}"
Which returns something like:
{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555",
"Credentials": {
"AccessKeyId": "ASIAXXXXXXXXXXXXXXXX",
"SecretKey": "Im0JN4PrvZZZZZZZZZZZZZZZZZZZZZZZZZZ/Y/XX",
"SessionToken": "ExtremelyLongSessionToken",
"Expiration": "2020-06-07T15:28:51-07:00"
}
}
You can then use that AccessKeyID and SecretKey to make any needed AWS calls; these temporary credentials are good for an hour.
Using the same methodology with aws history show and Postman described above, we can get the CURL equivalents:
curl --location --request POST 'https://cognito-identity.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityService.GetId' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"IdentityPoolId": "us-east-1:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
"Logins": {
"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx": "ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool"
}
}'
and then
curl --location --request POST 'https://cognito-identity.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityService.GetCredentialsForIdentity' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555",
"Logins": {
"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx": "ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool"
}
}'
Note that the CLI converts the expiration timestamp to local time. When you call this with CURL, the "Expiration" value will be something like 1.591572729E9 . You can convert that to a human time with a site like https://www.epochconverter.com/
Hello I am using below rest api commamd to create a service account key in GCP. Running the command from cloud shell though not sure doing it correctly.
curl POST https://iam.googleapis.com/v1/projects/project_iD/serviceAccounts/serviceaccountID.iam.gserviceaccount.com/keys?key=key generated by API Key credentials
I am a service account admin but when I run this command in cloud shell I get below error. Idealy I have all access for service account still says list permisssion is required. Can anybody help?
curl: (6) Could not resolve host: POST { "error": { "code": 403, "message": "Permission iam.serviceAccountKeys.list is required to perform this operation on service account projects/pserviceaccountID#dev.iam.gserviceaccount.com.", "status": "PERMISSION_DENIED" } }
There are two parts to your error. The first:
curl: (6) Could not resolve host: POST
Is telling you that the curl command cannot look up the hostname "POST" because you omitted the -X parameter, the first part of your command should read:
curl -X POST
Next, the URL you have is not quite the right format, as there should be no URL parameters (in this case the ?key=key portion), as it is a POST request -- the parameters from the API would be included in the body of the request.
However, I suspect even in that case you will have a permission denied error, as curl will not manage the oauth authentication and authorization that is necessary for this request to work -- you're effectively appearing to the API as unauthenticated. I'd recommend in this case that you use one of the client libraries to do the request, or use the gcloud command directly instead of curl. These will both greatly simplify the management of the authentication.
There are examples in C#, Go, Node, Python and others in the documentation for the API itself, take a look here: https://cloud.google.com/iam/reference/rest/v1/projects.serviceAccounts.keys/create
The documentation for using gcloud to accomplish this is here: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-gcloud
That said, if you really want to do this with curl from cloud shell (where you have an authenticated gcloud session) this is the sequence of commands you need:
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl -X POST --header "Authorization: Bearer ${ACCESS_TOKEN}" \
https://iam.googleapis.com/v1/projects/PROJECTID/serviceAccounts/SERVICEACCOUNTNAME#PROJECTID.iam.gserviceaccount.com/keys
If you aren't on a cloud shell machine, you need to make sure you have gcloud auth first:
gcloud auth login
It looks like your service account that's making the request doesn’t have the required permissions. You should either give the SA the required IAM roles described in [1], or you can use your own user by doing gcloud auth login user#email.com to make the call.
You’ll also need the Service Account Key Admin, as SA Admin doesn’t have iam.serviceAccountKeys.list [2].
[1] https://cloud.google.com/iam/docs/creating-managing-service-account-keys#required_permissions
[2] https://cloud.google.com/iam/docs/permissions-reference
I tried to access Google's Datastore through their REST Api. It says that they allow authentication through the API-key. However it doesn't seems that I can get it to work any where. I copied the snippet generated from their Try this API page.
curl --request POST \
'https://datastore.googleapis.com/v1/projects/PROJECT_ID:runQuery?key=[YOUR_API_KEY]' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"query":{"filter":{"compositeFilter":{"op":"AND","filters":[{"propertyFilter":{"property":{"name":"id"},"op":"EQUAL","value":{"stringValue":"ID"}}}]}},"kind":[{"name":"NAME"}]},"partitionId":{"namespaceId":"NAMESPACE_ID","projectId":"PROJECT_ID"}}' \
--compressed
But it keeps returning me an 401 error.
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
It looks like it require me to use OAuth instead, which is not what their documentation says. Anyone experienced something similar?
You are using an API key which is incorrect.
This link details which services support API Keys. Cloud Datastore is not one of them.
Using API Keys
You want to use an Access Token which is derived from Service Account credentials.
Review this document.
Using OAuth 2.0
The steps to generate an Access Token:
Load the service account credentials json file.
Extract the client_email, private_key and private_key_id.
Create a json payload.
Call the authorization URL: https://www.googleapis.com/oauth2/v4/token
This returns a json object. Extract the access_token.
Use the access_token instead of an API Key.
There are examples on the Internet in various languages. The link will get you started. The process appears complicated, and it is, but once you understand it, generating Access Tokens is easy and they can be reused until they expire (typically 60 minutes which you control).
This document on Google Cloud Storage authentication is the same for Cloud Datastore. The key is understanding "Authorization: Bearer" which is a header you need to include with your curl request.
Authentication