I have a service to service set up that I completed using the google cloud tutorial (https://cloud.google.com/run/docs/authenticating/service-to-service#nodejs)
Changed the cloudrun Service account to have roles/run.invoker (they both share the same role)
Make a request to get the access token: http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://XXXX-XXXX-XXXX-xx.a.run.app'
(failing) Use that access token to make a request at https://XXXX-XXXX-XXXX-xx.a.run.app/my-endpoint with the access token: axios.post('https://XXXX-XXXX-XXXX-xx.a.run.app/my-endpoint', {myData}, {headers: {Authorization: 'Bearer eyJhbGciOiJSUz.....'}})
However, on step 3, making the call to my service, I receive a 403 error, any thoughts on what I missed?
Note: I have tried deploying my invoked service with --allow-unauthenticated and without it. I am not using a custom domain, I am using the CloudRun created url.
PS: If I change the ingress from internal and load balancer to all it works, however I'm not sure if this is correct to do.
The HTTP 403 Forbidden error message when accessing your Cloud Run service means that your client is not authorized to invoke this service.
You have not granted the service account permission to call the receiving service. Your question states that you added roles/run.invoker but the error message indicates you did not complete this step correctly.
Go to the Google Cloud Console.
Select the receiving service (this is the Cloud Run service you are calling).
Click Show Info Panel in the top right corner to show the Permissions tab.
In the Add members field, enter the identity of the calling service.
Select the Cloud Run Invoker role from the Select a role drop-down menu.
Click Add.
Note: When requesting the Identity Token, do not specify the custom domain. Your question's wording is confusing on that point.
[UPDATE]
The OP has enabled internal and load balancer. This requires setting up Serverless VPC Access.
Connecting to a VPC network
Solution was to add a VPC Connector and route all traffic through it. I added this to the deploy script --vpc-egress all-traffic. Originally I had --vpc-egress private-ranges-only to connect to redis MemoryStore, however this was insufficient to connect to my other service (internal only ingress).
Credit to excellent insight from #JohnHanley and #GuillaumeBlaquiere
Interesting Note About NodeJS: My container wouldn't start when I switched the --vpc-egress to all-traffic, and I had no idea why because there were no logs. It turns out running node v16.2 caused some weird issues with --vpc-egress all-traffic that I couldn't debug, so downgrading to 14.7 allowed the container to start.
Related
I'm trying to enable IAP for the first time for an app I have set up and working. I think I'm missing an obvious step in the process, but can't seem to figure it out. In the console if I go to APIs and Services->Credentials, under OAuth 2.0 Client IDs there's an entry named IAP-backend-service. I retrieve the client ID and client secret from there.
The app I'm trying to secure is tekton. I'm deploying my backend through terraform using the google_compute_backend resource. To enable IAP, I'm adding the following block to that resource:
iap {
oauth2_client_id = "replaced"
oauth2_client_secret = "replaced"
}
This automatically enables IAP for it which I can see in the console. However, as soon as I do this, I can no longer access my app. I just get connection refused or connection reset without it trying to even prompt me for IAP creds. Is there another step I need to do for this to work?
I have firewall rules in place to allow tcp/80 and tcp/9097 from the IAP range (35.235.240.0/20)
Thanks!
I am trying to setup a python app on GCP Cloudrun I need only authenticated users to be able to access the Cloudrun URL but I am facing 403 issue when I set up this app. Is there any alternative way to access the Cloudrun instance using browser provided it is configured to allow only authenticated users?
This is the flow which i am trying to implement :
HTTP(S) Load balancer -> Frontend forwarding rule -> Cloudrun Backend -> Python app deployed on Cloudrun
I have saw few other questions and tried that solution but it does not work few such similar questions would be :
403 "Error: Forbidden" when opening the URL of my Cloud Run service
If you're currently getting 403, it means you don't have the necessary permission to access the service (the App was deployed to cloud with the option to use 'authenticated' invocation which means you can't access it by directly typing the URL in the browser). You can do any of the following
Generate a token and then use curl to invoke your url using that token. See Google Documentation on this and a more detailed explanation from Google here. But you can't be doing this each time you wish to invoke the service. It's more for testing.
Update: The solution below was to allow him to actually see the App run in the browser but reread the question and see that OP wants only authenticated users to access the App.
2. Redeploy the App to Cloud Run but make sure you choose the option to allow for 'unauthenticated invocation'. See step 3.iv of this [blog article][3] we wrote on deploying to cloud run
After trying out IAP as said by #guillaumeblaquiere i was able to fix this issue. Thanks a lot as there is ver less documentation on how to fix this i have recorded steps that i implemented to fix this issue :
Accessing applications on Authenticated Cloud Run using IAP
I'm hosting the backend for an internal admin tool on Cloud Run. Since only admins of the GCP project should be able to access this tool, I followed the instructions here to enable IAP for Cloud Run by setting up a load balancer with a static external IP (and custom domain), restricting ingress to "Internal and Cloud Load Balancing", and allowing public unauthenticated access for the Cloud Run service since IAP is handling the authentication and authorization.
Now I'm trying to set up some cron jobs on Cloud Scheduler, for which I've provided an endpoint corresponding to my custom domain (say https://customdomain.com/endpoint), along with a service account email that allows OIDC tokens to be generated. The audience for the OIDC token is set automatically to the same custom domain URL. However, as reported on this thread, there seems to be a bug with Cloud Scheduler that only allows run.app audiences - anything else (including custom domains) results in a 401 UNAUTHENTICATED. This happens even if I set my target URL to https://customdomain.com/endpoint but my audience to https://cloud-run-service.a.run.app/endpoint. Of course, I can't change my target URL to https://cloud-run-service.a.run.app/endpoint since it doesn't allow direct traffic not coming through the load balancer.
Has anyone been in this situation or know of any workarounds? Thanks!
I understand your issue is,
In Cloud Scheduler, the OIDC token that is sent to the Cloud Run Service only works if the Audience is the Cloud Run-provided URL, not the Custom Domain URL.
Doesn't work: URL: https://service-url.customdomain.com | Audience: https://https://service-url.customdomain.com
Works: URL: https://service-url.customdomain.com | Audience: https://example-abcdefg.a.run.app
Works: URL: https://example-abcdefg.a.run.app | Audience: https://example-abcdefg.a.run.app
Google is aware of the issue and is working on allowing them to specify custom audiences for Cloud Run services, which will solve your problem.
Right now as per the latest update on May, 2022 we're about to ship custom audiences for Cloud Run. Please fill out this form if you are interested in being an early tester for "custom audiences for Cloud Run."
Currently, to authenticate the caller via Cloud IAM, you must pass in JWT token with the audience field set to the full URL of the service, such as https://example-abcdefg.a.run.app.
With this capability, you can specify a custom domain as the audience field in the OAuth token instead of the original service URL enable a service deployed in multiple regions to accept a common audience field
Issue tracker reference :
https://issuetracker.google.com/182490050
I believe you can still set the target URL (while configuring Cloud Scheduler) to the run.app/endpoint of your Cloud Run service by making use of service accounts
First create a service account for Cloud Scheduler
Then give this service account permission to invoke your Cloud Run Service
See Google's documentation here
After hours of painful debugging, here's the solution for anyone with the same issue. While it's still true that custom domains mapped to the Cloud Run service don't work as the OIDC audience, neither does the Cloud Run-provided run.app URL when using IAP in front of a load balancer. It turns out the expected audience in such cases is the IAP Client ID. You can find this under Credentials -> APIs and Services -> OAuth 2.0 Client IDs -> <IAP service name>. Just manually set the OIDC audience to this exact string and things should start working!
I am trying to add Amazon Connect integration to an existing lex-web-ui instance running in a React app. The bot has not been created using the CloudFormation deployment template provided in the aws-lex-web-ui repo, and I am instead using one that has been manually created. Same goes for cognito and connect instances.
The Amazon Connect configuration for this requires apiGatewayEndpoint (provisioned by CF template) - however, due to the above, this is not present in my current config. What exactly is this gateway endpoint, what is its purpose and what does it connect to? It looks like it's absolutely needed for the config otherwise it throws a refusal error.
I've configured the connect instance and contact flow ID, the last piece missing is this API GW url.
I'm having issues performing requests using jest to an AWS ElasticSearch cluster v5.3.
Reason is:
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details
I am using windows 10 with java 11, spring boot 2, webflux, jest and the aws http request signer that they point to in their documentation.
I've checked and doubled checked the access and secret keys of the IAM user. I also added policies for the IAM user of full control over the cluster, still the 403 message.
Removing or adding the Content-Length header yields the same error.
Not sure where to go from here.
Any help would be appreciated.
Thx
So from I discovered, is that the network issue had something to do with the corporate proxy. I created a tunnel between my laptop and the ElasticSearch cluster, removed the proxy from the http client used by jest, and things work smoothly now.
I wasn't able to figure out exactly how the proxy affected the request signature though, but I'll stick with the tunnel solution.