I'm hosting the backend for an internal admin tool on Cloud Run. Since only admins of the GCP project should be able to access this tool, I followed the instructions here to enable IAP for Cloud Run by setting up a load balancer with a static external IP (and custom domain), restricting ingress to "Internal and Cloud Load Balancing", and allowing public unauthenticated access for the Cloud Run service since IAP is handling the authentication and authorization.
Now I'm trying to set up some cron jobs on Cloud Scheduler, for which I've provided an endpoint corresponding to my custom domain (say https://customdomain.com/endpoint), along with a service account email that allows OIDC tokens to be generated. The audience for the OIDC token is set automatically to the same custom domain URL. However, as reported on this thread, there seems to be a bug with Cloud Scheduler that only allows run.app audiences - anything else (including custom domains) results in a 401 UNAUTHENTICATED. This happens even if I set my target URL to https://customdomain.com/endpoint but my audience to https://cloud-run-service.a.run.app/endpoint. Of course, I can't change my target URL to https://cloud-run-service.a.run.app/endpoint since it doesn't allow direct traffic not coming through the load balancer.
Has anyone been in this situation or know of any workarounds? Thanks!
I understand your issue is,
In Cloud Scheduler, the OIDC token that is sent to the Cloud Run Service only works if the Audience is the Cloud Run-provided URL, not the Custom Domain URL.
Doesn't work: URL: https://service-url.customdomain.com | Audience: https://https://service-url.customdomain.com
Works: URL: https://service-url.customdomain.com | Audience: https://example-abcdefg.a.run.app
Works: URL: https://example-abcdefg.a.run.app | Audience: https://example-abcdefg.a.run.app
Google is aware of the issue and is working on allowing them to specify custom audiences for Cloud Run services, which will solve your problem.
Right now as per the latest update on May, 2022 we're about to ship custom audiences for Cloud Run. Please fill out this form if you are interested in being an early tester for "custom audiences for Cloud Run."
Currently, to authenticate the caller via Cloud IAM, you must pass in JWT token with the audience field set to the full URL of the service, such as https://example-abcdefg.a.run.app.
With this capability, you can specify a custom domain as the audience field in the OAuth token instead of the original service URL enable a service deployed in multiple regions to accept a common audience field
Issue tracker reference :
https://issuetracker.google.com/182490050
I believe you can still set the target URL (while configuring Cloud Scheduler) to the run.app/endpoint of your Cloud Run service by making use of service accounts
First create a service account for Cloud Scheduler
Then give this service account permission to invoke your Cloud Run Service
See Google's documentation here
After hours of painful debugging, here's the solution for anyone with the same issue. While it's still true that custom domains mapped to the Cloud Run service don't work as the OIDC audience, neither does the Cloud Run-provided run.app URL when using IAP in front of a load balancer. It turns out the expected audience in such cases is the IAP Client ID. You can find this under Credentials -> APIs and Services -> OAuth 2.0 Client IDs -> <IAP service name>. Just manually set the OIDC audience to this exact string and things should start working!
Related
I am trying to setup a python app on GCP Cloudrun I need only authenticated users to be able to access the Cloudrun URL but I am facing 403 issue when I set up this app. Is there any alternative way to access the Cloudrun instance using browser provided it is configured to allow only authenticated users?
This is the flow which i am trying to implement :
HTTP(S) Load balancer -> Frontend forwarding rule -> Cloudrun Backend -> Python app deployed on Cloudrun
I have saw few other questions and tried that solution but it does not work few such similar questions would be :
403 "Error: Forbidden" when opening the URL of my Cloud Run service
If you're currently getting 403, it means you don't have the necessary permission to access the service (the App was deployed to cloud with the option to use 'authenticated' invocation which means you can't access it by directly typing the URL in the browser). You can do any of the following
Generate a token and then use curl to invoke your url using that token. See Google Documentation on this and a more detailed explanation from Google here. But you can't be doing this each time you wish to invoke the service. It's more for testing.
Update: The solution below was to allow him to actually see the App run in the browser but reread the question and see that OP wants only authenticated users to access the App.
2. Redeploy the App to Cloud Run but make sure you choose the option to allow for 'unauthenticated invocation'. See step 3.iv of this [blog article][3] we wrote on deploying to cloud run
After trying out IAP as said by #guillaumeblaquiere i was able to fix this issue. Thanks a lot as there is ver less documentation on how to fix this i have recorded steps that i implemented to fix this issue :
Accessing applications on Authenticated Cloud Run using IAP
I have a service to service set up that I completed using the google cloud tutorial (https://cloud.google.com/run/docs/authenticating/service-to-service#nodejs)
Changed the cloudrun Service account to have roles/run.invoker (they both share the same role)
Make a request to get the access token: http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://XXXX-XXXX-XXXX-xx.a.run.app'
(failing) Use that access token to make a request at https://XXXX-XXXX-XXXX-xx.a.run.app/my-endpoint with the access token: axios.post('https://XXXX-XXXX-XXXX-xx.a.run.app/my-endpoint', {myData}, {headers: {Authorization: 'Bearer eyJhbGciOiJSUz.....'}})
However, on step 3, making the call to my service, I receive a 403 error, any thoughts on what I missed?
Note: I have tried deploying my invoked service with --allow-unauthenticated and without it. I am not using a custom domain, I am using the CloudRun created url.
PS: If I change the ingress from internal and load balancer to all it works, however I'm not sure if this is correct to do.
The HTTP 403 Forbidden error message when accessing your Cloud Run service means that your client is not authorized to invoke this service.
You have not granted the service account permission to call the receiving service. Your question states that you added roles/run.invoker but the error message indicates you did not complete this step correctly.
Go to the Google Cloud Console.
Select the receiving service (this is the Cloud Run service you are calling).
Click Show Info Panel in the top right corner to show the Permissions tab.
In the Add members field, enter the identity of the calling service.
Select the Cloud Run Invoker role from the Select a role drop-down menu.
Click Add.
Note: When requesting the Identity Token, do not specify the custom domain. Your question's wording is confusing on that point.
[UPDATE]
The OP has enabled internal and load balancer. This requires setting up Serverless VPC Access.
Connecting to a VPC network
Solution was to add a VPC Connector and route all traffic through it. I added this to the deploy script --vpc-egress all-traffic. Originally I had --vpc-egress private-ranges-only to connect to redis MemoryStore, however this was insufficient to connect to my other service (internal only ingress).
Credit to excellent insight from #JohnHanley and #GuillaumeBlaquiere
Interesting Note About NodeJS: My container wouldn't start when I switched the --vpc-egress to all-traffic, and I had no idea why because there were no logs. It turns out running node v16.2 caused some weird issues with --vpc-egress all-traffic that I couldn't debug, so downgrading to 14.7 allowed the container to start.
I'm having some trouble setting uptime checks for some Cloud Run services that don't allow unauthenticated invocations.
For context, I'm using Cloud Endpoints + ESPv2 as an API gateway that's connected to a few Cloud Run services.
The ESPv2 container/API gateway allows unauthenticated invocations, but the underlying Cloud Run services do not (since requests to these backends flow via the API gateway).
Each Cloud Run service has an internal health check endpoint that I'd like to hit periodically via Cloud Monitoring uptime checks.
This serves the purpose of ensuring that my Cloud Run services are healthy, but also gives the added benefit of reduced cold boot times as the containers are kept 'warm'
However, since the protected Cloud Run services expect a valid authorisation header all of the requests from Cloud Monitoring fail with a 403.
From the Cloud Monitoring UI, it looks like you can only configure a static auth header, which won't work in this case. I need to be able to dynamically create an auth header per request sent from Cloud Monitoring.
I can see that Cloud Scheduler supports this already. I have a few internal endpoints on the Cloud Run services (that aren't exposed via the API gateway) that are hit via Cloud Scheduler, and I am able to configure an OIDC auth header on each request. Ideally, I'd be able to do the same with Cloud Monitoring.
I can see a few workarounds for this, but all of them are less than ideal:
Allow unauthenticated invocations for the underlying Cloud Run services. This will make my internal services publicly accessible and then I will have to worry about handling auth within each service.
Expose the internal endpoints via the API gateway/ESPv2. This is effectively the same as the previous workaround.
Expose the internal endpoints via the API gateway/ESPv2 AND configure some sort of auth. This sort of works but at the time of writing the only auth methods supported by ESPv2 are API Keys and JWT. JWT is already out of the question but I guess an API key would work. Again, this requires a bit of set up which I'd rather avoid if possible.
Would appreciate any thought/advice on this.
Thanks!
This simple solution may work on your use case as it is easier to just use a TCP uptime check on port 443:
Create your own Cloud Run service using https://cloud.google.com/run/docs/quickstarts/prebuilt-deploy.
Create a new uptime check on TCP port 443 Cloud Run URL.
Wait a couple of minutes.
Location results: All locations passed
Virginia OK
Oregon OK
Iowa OK
Belgium OK
Singapore OK
Sao Paulo OK
I would also like to advise that Cloud Run is a Google fully managed product and it has a 99.95 % monthly up time SLA, with no recent incidents in the past few months, but proactively monitoring this on your end is a very good thing too.
We use Cloud Endpoints in the Google Cloud for service-to-service authentication in applications deployed to the Google Kubernetes Engine.
We have an Extensible Service Proxy sidecar in front of all of our applications, with a Swagger (OpenApi) 2.0 YAML descriptor specifying which endpoints can be accessed by which other services.
Every service has its own Service Account, and it's straightforward to create the necessary security definition:
securityDefinitions:
service-foo:
authorizationUrl: ""
flow: "implicit"
type: "oauth2"
x-google-issuer: "service-foo#my-gcloud-project.iam.gserviceaccount.com"
x-google-jwks_uri: "https://www.googleapis.com/robot/v1/metadata/x509/service-foo-my-gcloud-project.iam.gserviceaccount.com"
This works well for service-to-service communication in GKE.
But during development, sometimes we want to send requests to these service from our development machines. And with this setup, we need to have access to a raw Service Account key to generate the JWT token, which is inconvenient, and also raises security concerns.
It would be very nice if we could create a security definition with which a developer could access a service using not a Service Account, but their own Google Account.
I tried to create a JWT token using the gcloud CLI with the following command:
$ gcloud auth print-identity-token
This prints a valid JWT token, where the Audience is 32555940559.apps.googleusercontent.com, and the token has an email field with my own Google account email address.
I tried creating a security definition for this, I tried the following.
my-dev-account:
authorizationUrl: ""
flow: "implicit"
type: "oauth2"
x-google-issuer: "https://accounts.google.com"
x-google-jwks_uri: "https://www.googleapis.com/oauth2/v3/certs"
x-google-audiences: "32555940559.apps.googleusercontent.com"
And this technically works, the ESP allows the request go through with the JWT token produced by gcloud auth print-identity-token. But the problem is that this doesn't limit access to my own account in any way, because 32555940559.apps.googleusercontent.com is the Client ID of the GCloud SDK itself, so anyone with a Google account would have access.
Is it maybe possible to specify in the security definition that the email field should be limited to a certain value?
Or am I completely on the wrong track, but is there another way to allow ESP access for a developer Google account?
Make a small service in app engine standard behind a Identity Aware proxy (IAP). There you can limit who has access to it.
And the purpose of this service would be replicate that same request to the specified endpoint with the correct authentication token in the header.
GET /home?next=https://your-kubernetes-endpoint/resource
If you add a new member to the team, you can grant access through the IAP.
If you add a new endpoint, you change the value of the query parameter next.
You can generate an id_token with the audience that you want like this:
gcloud auth print-identity-token --audiences=<MySpecificAudience>
BUT, you can't generate this with a user account. You need a service account key file, that isn't so good for security reason (key management and so on...)
The solution could be to impersonate a service account when you request a token with a specific audience
gcloud auth print-identity-token --audiences=<MySpecificAudience> \
--impersonate-service-account=<SA-Name>#<ProjectId>.iam.gserviceaccount.com \
--include-email
The --include-email adds also the email.... of the impersonated service account. I don't know if that match your requirements (security, traceability,...)
I am using a GCE instance with one of the python/jupyter notebook images provided by GCP to do some data analytics. The data resides in BigQuery. The instance can use a service account to access BiqQuery by default, but for data protection reasons I have to use personal user accounts requesting the data.
I know can use an programmatic oauth flow in python to request credentials to authenticate to BigQuery with a personal account, but that flow has to be run interactively every time you restart the ipython kernel and involves opening that authorization URL in a browser and then pasting the secret, which is annoying.
Since I am using IAP to log onto the GCE instance with my account, is there a way to get personal credentials from the IAP to use for authentication to BigQuery automatically?
As I understand, IAP is like a login screen between user and the application (in your case, jupyter server running on GCE) for the purpose of authenticating user to the app, e.g. jupyter server.
IAP adds an id_token (JWT) as signed headers to your http requests after you authenticate, the JWT is specifically made for the IAP instance, if you base64 decode the jwt, you'll see the audience claim is set to the client_id of the IAP instance, and it can't be used for other purposes.
Also bigquery requires an access_token instead of an id_token. I think doing an oauth flow might be the only way. but happy to be corrected...