I tried to access Google's Datastore through their REST Api. It says that they allow authentication through the API-key. However it doesn't seems that I can get it to work any where. I copied the snippet generated from their Try this API page.
curl --request POST \
'https://datastore.googleapis.com/v1/projects/PROJECT_ID:runQuery?key=[YOUR_API_KEY]' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"query":{"filter":{"compositeFilter":{"op":"AND","filters":[{"propertyFilter":{"property":{"name":"id"},"op":"EQUAL","value":{"stringValue":"ID"}}}]}},"kind":[{"name":"NAME"}]},"partitionId":{"namespaceId":"NAMESPACE_ID","projectId":"PROJECT_ID"}}' \
--compressed
But it keeps returning me an 401 error.
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
It looks like it require me to use OAuth instead, which is not what their documentation says. Anyone experienced something similar?
You are using an API key which is incorrect.
This link details which services support API Keys. Cloud Datastore is not one of them.
Using API Keys
You want to use an Access Token which is derived from Service Account credentials.
Review this document.
Using OAuth 2.0
The steps to generate an Access Token:
Load the service account credentials json file.
Extract the client_email, private_key and private_key_id.
Create a json payload.
Call the authorization URL: https://www.googleapis.com/oauth2/v4/token
This returns a json object. Extract the access_token.
Use the access_token instead of an API Key.
There are examples on the Internet in various languages. The link will get you started. The process appears complicated, and it is, but once you understand it, generating Access Tokens is easy and they can be reused until they expire (typically 60 minutes which you control).
This document on Google Cloud Storage authentication is the same for Cloud Datastore. The key is understanding "Authorization: Bearer" which is a header you need to include with your curl request.
Authentication
Related
I'm new to Google APIs and I've been trying for days to use a service account to upload content to a Google Cloud Storage bucket. I'm able to accomplish this, but only with a temporary access token obtained from the Google API playground, and I need to be able to get new access tokens so the service account can always upload content.
I've been experimenting with the following, but I keep getting access denied, even though the account in question has 'owner' permissions.
curl -X POST / -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \ -H "Content-Type: application/json; charset=utf-8" \ -d #Documents/request.json \ https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/content-uploader#kol-hatorah-kulah.iam.gserviceaccount.com:generateAccessToken
response:
{ "error": { "code": 403, "message": "The caller does not have permission", "status": "PERMISSION_DENIED" } }
When I run gcloud config list I get the correct project, and the account is my work email, which is also in Google Cloud as an owner.
Thanks in advance!
DISCLAIMER - my solution works for Workload Identity Federation
related problems.
I've had hard time with this error, but finally found it!
For me it was wrong attribute mapping.
I was following some tutorial (which probably went outdated) and mapping was different than from official github action task documentation (here)
I had repository_owner and aud. Changed it for repository and...
It works!
To sum up my mapping looks like this:
"google.subject" = "assertion.sub"
"attribute.actor" = "assertion.actor"
"attribute.repository" = "assertion.repository"
So if you got here because of same tutorial... you've been served!
Your curl command is attempting to use a service account identity to generate an Access Token. The command is failing because you do not have permission.
Add the role roles/iam.serviceAccountTokenCreator to the identity running the command.
I am trying to use Cognito User Pool to authenticate with a PC application using an HTTPS call. I want to obtain the various tokens that I can then use to access the AWS resources without storing AWS secrets in the PC application.
The AWS documentation documents the InitiateAuth method and shows the AWS Endpoints, but it is not immediately apparent how to make the call over HTTPS. Most calls would require an AWS signature, but the InitiateAuth call should not, if I am just submitting Username and Password.
After some poking around, I was able to use the AWS CLI to successfully obtain tokens with this command:
aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id the_cognito_client_id --auth-parameters USERNAME=the_users_email,PASSWORD=the_users_password
Where the_cognito_client_id is an approximately 26 character long string shown as App client id under General Settings / App clients.
Note that the USER_PASSWORD_AUTH flow is not enabled by default, so you will initially get an error with this. Go to the Cognito Console for the specific User Pool and look for General Settings/App Clients, click "Show Details" for your specific app client, and check the "Enable username password based authentication (ALLOW_USER_PASSWORD_AUTH)" and save.
Once you get back tokens and you know your call is working, you can use the aws history show command to show you the details of the actual https call. The first time you call
aws history show
You will get a message
Could not locate history. Make sure cli_history is set to enabled in the ~/.aws/config file
Go to that file and add
cli_history=enabled
Then, run your initiate-auth call again with the cli. Then, when you run
aws history show
You will get back the details of how the call was made. (At this point, you might consider removing the cli_history setting so you don't log all of your calls, with all of the credentials, in the future.) You will see
to URL: https://cognito-idp.us-east-1.amazonaws.com/
which tells you the URL to use, and you will see that it is a POST. You will note that "InitiateAuth" is not anywhere in that URL. However, you will see the headers include:
"X-Amz-Target": "AWSCognitoIdentityProviderService.InitiateAuth"
and
"Content-Type": "application/x-amz-json-1.1"
You need both of those headers, including the non-standard Content-Type, to make the HTTPS call work. You can use Postman to put the call together, although Postman does not like the non-standard Content-Type, so you have to turn off the standard Content-Type and manually add these two headers to the call. At that point, Postman is able to obtain the tokens as well.
Postman also provides an export to CURL function (click the link that says "Code"), which gives you:
curl --location --request POST 'https://cognito-idp.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityProviderService.InitiateAuth' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"AuthFlow": "USER_PASSWORD_AUTH",
"AuthParameters": {
"PASSWORD": "the_users_password",
"USERNAME": "the_users_email"
},
"ClientId": "the_cognito_client_id"
}'
Submitting that on the command line also gives you the tokens you need.
To refresh using the refresh token, just use InitiateAuth, but the AuthFlow is REFRESH_TOKEN_AUTH and the only member of AuthParameters is REFRESH_TOKEN (which is, of course, the RefreshToken)
Now, I just need to figure out how to do USER_SRP_AUTH using HTTPS.
The other answer explains how to get the Tokens using the Username and Password. Next, we need to get the temporary credentials from the Cognito Identity Pool. This appears to require two steps. First, we need to call cognito-identity get-id and then cognito-identity get-credentials-for-identity
The get-id call requires the Identity Pool ID, which can be obtained from the Cognito Console for the Identity Pool. The identity-pool-id is available under "edit identity pool" as "identity pool ID." The Login key name comes from the User pool ID and is available under Authentication Providers / Cognito under the Edit Identity Pool. The Login key name is actually provided as "iss" in the encoded ID Token. You can use a service like https://jwt.io/ to decode the ID Token, and it will include an entry like
"iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx"
You can use that, without the https://, as the key for Login. The value for Login is the JWT returned as "IdToken" from InitiateAuth.
Therefore, using the CLI, the first call is
aws cognito-identity get-id --identity-pool-id "us-east-1:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" --logins "{\"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx\": \"ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool\"}"
Which returns the ID:
{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555"
}
Then, we need to get the temporary credentials using get-credentials-for-identity. This call uses the IdentityId returned from get-id rather than the Identity Pool ID used by get-id. The Login JSON is the same as for get-id. Therefore, the call is:
aws cognito-identity get-credentials-for-identity --identity-id "us-east-1:11111111-2222-3333-4444-555555555555" --logins "{\"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx\": \"ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool\"}"
Which returns something like:
{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555",
"Credentials": {
"AccessKeyId": "ASIAXXXXXXXXXXXXXXXX",
"SecretKey": "Im0JN4PrvZZZZZZZZZZZZZZZZZZZZZZZZZZ/Y/XX",
"SessionToken": "ExtremelyLongSessionToken",
"Expiration": "2020-06-07T15:28:51-07:00"
}
}
You can then use that AccessKeyID and SecretKey to make any needed AWS calls; these temporary credentials are good for an hour.
Using the same methodology with aws history show and Postman described above, we can get the CURL equivalents:
curl --location --request POST 'https://cognito-identity.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityService.GetId' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"IdentityPoolId": "us-east-1:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
"Logins": {
"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx": "ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool"
}
}'
and then
curl --location --request POST 'https://cognito-identity.us-east-1.amazonaws.com/' \
--header 'X-Amz-Target: AWSCognitoIdentityService.GetCredentialsForIdentity' \
--header 'Content-Type: application/x-amz-json-1.1' \
--data-raw '{
"IdentityId": "us-east-1:11111111-2222-3333-4444-555555555555",
"Logins": {
"cognito-idp.us-east-1.amazonaws.com/us-east-1_zAgxxxxxx": "ThisIsTheVeryLong.IDTokenReturneFrom.TheCognitoUserPool"
}
}'
Note that the CLI converts the expiration timestamp to local time. When you call this with CURL, the "Expiration" value will be something like 1.591572729E9 . You can convert that to a human time with a site like https://www.epochconverter.com/
I am using SCIM api for registering a user
curl -v -k --user admin:admin --data '{"schemas":[],"name":{"familyName":"Smith","givenName":"Paul"},"userName":"Paul","password":"password","emails":[{"primary":true,"value":"paul#somemail.com"}],"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User":{askPassword:"true"}}' --header "Content-Type:application/json" https://localhost:9443/scim2/Users
I got a
Create Password for New Account
email to my primary email address with a confirmation code.
How to validate this confirmation code with wso2is using SCIM 2?
In REST API there is an option to validate code.
I tried REST API
curl -k -v -X POST -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" -d '{ "code": "84325529-8aa7-4851-8751-5980a7f2d9f7","properties": []}' "https://localhost:9443/api/identity/user/v1.0/validate-code"
it returns in case of invalid code
{
"code": "18001",
"message": "Bad Request",
"description": "Invalid Code '84325529-8aa7-4851-8751-5980a7f2d9f7'"
}
in the case valid code it returns
{
"code": "18001",
"message": "Bad Request",
"description": "Invalid Code '%s.'"
}
no console error
Validate code worked with REST API work flow
There is no such SCIM APIs in WSO2 Identity Server to validate the confirmation codes. WSO2 Identity Server has a soap API (UserInformationRecoveryService-verifyConfirmationCode) and a REST API (/validate-code) that can fulfill your requirement.
You can find the SOAP API here.
I am editing my answer here since I talked with the developers and I realised there is a different API that you need to use for validating the confirmation code: documentation.
So you need to send the following request:
curl -k -v -X POST -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" -d '{ "code": "84325529-8aa7-4851-8751-5980a7f2d9f7","step": "UPDATE_PASSWORD","properties": []}' "https://localhost:9443/api/identity/recovery/v0.9/validate-code"
and the response should be
"HTTP/1.1 202 Accepted"
I tried it and it works.
Note: if you are using SCIM identity through a system like a self-managed GitLab, GitLab 15.3 (August 2022) offers a better integration:
User SCIM identity visible in UI
Previously, the SCIM identity for a user could only be accessed using the SCIM API.
Now, a user’s SCIM identity is visible to GitLab administrators in the Identities tab of the User list. With
this, troubleshooting of SCIM-related issues is simplified. Administrators can validate what identity, if any, is
being used for a specific account without requiring GitLab Support or an API query.
See Documentation and Issue.
As per google's docs, I'm generating my oauth access token like this:
export TOKEN=$(~/go/bin/oauth2l fetch -jwt -json ~/.google/my-service-key.json cloud-platform)
I'm then doing requests to Google's REST API like this:
curl -v -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d (...) $URL
The response I'm getting back from Google is that I'm not providing an OAuth token, when I clearly am:
Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Other posts suggest to use gcloud auth application-default print-access-token instead of the OAuth token, but I know this to be the incorrect approach, as Google's API responds back that it wants a service account OAuth token and not an identity.
Any idea what's happening here?
Sometimes an old (bad) token gets cached from before you rotated the service_account.json.
Delete Cache
Try this:
rm ~/.oauth2l
Token vs JWT
And try getting an API token before you sign the JWT:
oauth2l fetch cloud-platform
Scope vs Audience
Also, the API token requires a scope (shown above), whereas the JWT requires an audience aud, which is a URL:
oauth2l fetch --jwt https://www.googleapis.com/auth/cloud-platform
ENVs
You may also want to make sure that you don't have competing configuration, see if GOOGLE_APPLICATION_CREDENTIALS is set.
echo $GOOGLE_APPLICATION_CREDENTIALS
unset GOOGLE_APPLICATION_CREDENTIALS
Or potentially use it instead of --json ./service_account.json:
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/google/service_account.json
HTH
I have successfully configured the API Manager, ID and IS according to the documentation: https://docs.wso2.com/display/AM260/JWT+Grant#JWTGrant-UsingtheJWTgrant.
I invoke the WSO2 token endpoint to exchange an external JWT for a WSO2 access token:
curl -i -X POST -k -d 'grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiIsImF1ZCI6WyJ3c28yIl0sImVudmlyb25tZW50Ijp7InByb2ZpbGVzIjpbImRldiJdfSwiYXV0aCI6eyJyb2xlcyI6WyJVU0VSIiwiQURNSU4iXX0sInVzZXJfbmFtZSI6IjAwMDAwMDk5Iiwic2NvcGUiOlsib3BlcmF0ZSJdLCJpc3MiOiJQQVMiLCJleHAiOjE1NDUzNDgyODcsImdlbmVyYXRlZEJ5IjoiUEFTIiwianRpIjoiOWQ4ZWU3ZTgtNDBlZS00MTZjLTlkYjgtYjU2NDZhYTZhN2JmIiwiY2xpZW50X2lkIjoiZnJvbnQtcG9saXphcyJ9.Ccs1OxjteRsvHTump-ZTawEsqlTrIeO0LJUzt5Ita8udvMOa_tB1rHOtI8GAa2mDCPMD_Z_jtZ2SlXPs10GvsYlF4jS_wcCVAPtHsoigzuNtg5t7CVfeCI2Bzhak721LdYBcjB9s0Jn24G9eb2jqx8NF0RPlKgmhbxwdY0b8XeigLp-kGCsFKY_fDIjFUM0oifzCWOmtaCRMtMx3CKVZOWq9dBIokheCi2foL8YkBCz57yo4vb782AYWXdiHj38TPPe4IguARuoc9FSymyiL1gWHJmyMZFvAeAJkDnHHEnnezqPmcWQweC1ylLwUYGNVLM8YSfuBDtcGBWSO0F-WKw' -H 'Content-Type: application/x-www-form-urlencoded' https://localhost:9443/oauth2/token -d 'client_id=w_paekjnDDY8zcCfCRgj_81g2eYa'
This answers successfully with an access token, a refresh token etc etc.
I have created an application in the WSO2 APIM store. In the production/sandbox tabs, the only checked Grant Type item is JWT.
The point is, I use the previously gotten access token (which is itself an JWT token) to invoke an API subscribed with the above application:
curl -k -X GET "https://192.168.179.129:8243/myapp/api/v1/customers" -H "accept: application/json;charset=UTF-8" -H "Authorization: Bearer eyJ4NXQiOiJOVEF4Wm1NeE5ETXlaRGczTVRVMVpHTTBNekV6T0RKaFpXSTRORE5sWkRVMU9HRmtOakZpTVEiLCJraWQiOiJOVEF4Wm1NeE5ETXlaRGczTVRVMVpHTTBNekV6T0RKaFpXSTRORE5sWkRVMU9HRmtOakZpTVEiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImF1ZCI6IndfcGFla2puRERZOHpjQ2ZDUmdqXzgxZzJlWWEiLCJuYmYiOjE1NDUzNDgxOTgsImF6cCI6IndfcGFla2puRERZOHpjQ2ZDUmdqXzgxZzJlWWEiLCJzY29wZSI6ImRlZmF1bHQiLCJpc3MiOiJodHRwczpcL1wvbG9jYWxob3N0Ojk0NDNcL29hdXRoMlwvdG9rZW4iLCJleHAiOjE1NDUzNDgyODcsImlhdCI6MTU0NTM0ODE5OCwianRpIjoiNjRlM2I5N2UtOTNlNC00YzQ2LThlNmQtMzlmZjQzOWQxM2Y0In0.UBLOsCCD3t4Wf8nXBnDkkGXxefYySelDzEcs1F_IrbshMJXohxcL92Av1nmcpdNdjin7GdC8Y305rrkBt9T1L_cMAHLYYcI5cI1J7wmAgEd1CEv9gI7IUYfAdbga2AeV4kIlNsgiV6PKnU34WnY7rEVqXD908eEHY5UvaNXc0Bz6C8d-p39-SqKUblGHPh9vdkpcCGcK0CgGKjtiU2lai_JkRALdgEgonT37R5eqmuxPxUouWNz9TCJgTuonKPA-9bYOsMvbzGlm--0m0j9gdxnv-3N1Kv_2JqSCR4pToDClhSKgFCE1L025LIICM-sLd_PDU5pwYge_iKseiIDZfA" -d 'client_id=w_paekjnDDY8zcCfCRgj_81g2eYa'
I get the following error (900908) - Resource forbidden:
<ams:fault xmlns:ams="http://wso2.org/apimanager/security"><ams:code>900908</ams:code><ams:message>Resource forbidden </ams:message>
<ams:description>Access failure for API: /myapp/api/v1, version: v1 status: (900908) - Resource forbidden </ams:description></ams:fault>%
I must be missing the final step which is how to allow those access tokens gotten in the JWT grant to be used to access an API subscribed by an application.
The error code 900908 means the API is not subscribed by the application. Please double check.
Technically Bee was right in his answer, but I would like to point specifically at what I was doing wrong in case it happens to others:
The problem was that the client_id/client_secret I was using when exchanging the JWT to get the access token were the ones from the Service Provider I had created. WRONG!
The ones was that need to be sent are those from the subscribed application. With that the resource forbidden error doesn't show up anymore.