I am new to GCP pub/sub and am trying to get everything setup. I am facing a permission error when creating a subscription topic. Does anyone know what I am missing in my steps, or how to resolve? Here is what I have done so far.
Add roles, Service Account Admin and Service Account Key Admin
Created Service Account (completed the full flow)
Tried to create a subscription topic <-- this is where I am receiving an error.
Here is the error at a closer look.
UPDATE: After receiving some comments, it seems my issues are all related to roles and permissions. The problem is that I do not know which roles/permissions work.
I am able to create a subscription topic so long that I do NOT enable authentication and add a service account. Since my original post, I have added these other additional permissions: Pub/Sub Admin, Service Account Admin, Service Account Key Admin, Service Account Token Creator, Service Account User.
Related
I have a DataStudio dashboard which contains a dataset resource connection to a BigQuery table which is currently authenticated using an individual user's account.
I want to change that to use a service account.
In order to do that, I followed this guide, which means I have:
Created a service account
Added the BigQuery Job User role to the service account
Added the Service Account Token Creator role to the service account
Added the BigQuery Data Viewer role to the service account (associated to the correct BigQuery table)
Ensured that my account is added as a Service Account User on the service account
On the DataStudio dashboard, I then navigate to:
Resource > Manage Added Data Sources > Select the specific data source > Click on the blade with the current user account being used to authenticate.
When I try to insert the service account details in the window, I see the error message below.
I've followed the link (which leads back to the original guide). The link says I need to add the Service Account Token Creator to the service account, which I've confirm through the following:
Now I'm stumped! What could be the reason why I cannot authenticate the connection using the service account?
I figured it out.
The problem was that the Service Account Token Creator role was assigned to the service account and not the service agent.
This link describes the process of setting up a service account for authenticating Data Studio data sources. Although it's got all the info there, the devil is in the detail in Step 2: Allow the Looker Studio service agent to access your service account.
The scenario is the following: I have two projects in the same organization and folder; let's call them project1 and project2. I have a Pub/Sub topic on project2 and I have a Cloud Run service on project1 that I want to use as the endpoint for a push subscription for the topic. However, I don't want everyone to be able to access my Cloud Run service: I only want authorised service accounts to be able to invoke the service. Thus I created a service account on project1 that has the role/run.invoker role for the Cloud Run service. Now, when I go and create my subscription in the topic in project2 with the Cloud Run service as the endpoint and I try to associate to it the service account in project1 using the following command:
gcloud --project=project2 pubsub subscriptions create test_subscrption --topic=topic-name --topic-project=project2 --push-auth-service-account=service_account1 --push-endpoint=https://my-cloud-run-service.run.app/
I get the following:
ERROR: Failed to create subscription [projects/project2/subscriptions/test_subscrption]: User not authorized to perform this action.
ERROR: (gcloud.pubsub.subscriptions.create) Failed to create the following: [test_subscrption].
Note that the account with which I'm logged into GCP has the Owner permission at the organization level. Moreover, if I create a service account in project2 and use that instead of the service account on project1, the subscription is created without problems. I'm aware that there are some permissions to set before associating a service account to a push subscription (serviceAccountTokenCreator as described here and iam.serviceAccounts.actAs as described here), but I can guarantee I did set those permission before trying to create my subscription, and I still got the same error. From all this I'm concluding that it is not possible to associate a service account from a different project to a push subscription in Pub/Sub (more to the point, if I try to create the subscription from the GCP UI in the dropdown menu for the service account to associate to the push subscription I only see the service accounts that are part of the same project as the topic and subscription). Can anybody confirm this, or tell me how to achieve my goal? My current solution has been to create a service account in project2 and associate that to the push subscription instead, but I would like to know if my original plan was even possible at all.
As mentioned in the comment, when you do it with the UI, you can only select the service account of the current project and it is impossible to use an external service account. Instead, you can create the push subscription in project1 and create the topic in project2.
My current setup in GCP includes a Logging Sink to a Pub/Sub topic which is exporting certain Healthcare API log entries to a third party logging tools. Despite having, what I believe to be, all required permissions, the activity page shows Stackdriver config error with an error message something like this
Access to the topic was denied
The specified topic does not allow the service account associated with the log sink to publish to it. Grant publish permission for the service account specified in the sink's writerIdentity field on the topic.
Now the service account in question has the following set of permissions already assigned to it
Healthcare Dataset Administrator
Healthcare FHIR Resource Editor
Healthcare FHIR Resource Reader
Healthcare FHIR Store Administrator
Service Account Admin
Service Account Key Admin
Service Account User
Workload Identity User
Logs Configuration Writer
Logs Writer
Pub/Sub Editor
Pub/Sub Publisher
According to this thread, its is a known issue but its already been past the 24 hour threshold mentioned in there so I am a bit confused.
Are there any particular permissions that I am missing here? I have tried recreating the sink once or twice to no use. Appreciate any help from the community on this one. Thanks.
EDIT:
Here is what I found on running gcloud logging sinks describe <sink name>
createTime: '2021-02-23T00:42:35.363983703Z'
destination: <PUBSUB_TOPIC>
filter: resource.labels.service="healthcare.googleapis.com"
name: <SINK_NAME>
updateTime: '2021-03-01T18:44:50.694055502Z'
writerIdentity: <SA_NAME>
We have two projects in our GCP account; one for our Dev environment and one for our Test environment at the moment. Terraform manages most of our infrastructure, so we have minimal clicking around the GUI, or CLI commands.
I have assumed we enabled the Pub/Sub API by deploying to it with Terraform in both of our environments, although we may have needed to do this manually. We noticed that Google created a default Pub/Sub service account for us in our Dev environment, but not in our Test environment. This docs page suggests it should make this service account.
Additionally, we have noticed multiple Pub/Sub subscriptions working, apparently without any service account. We believe that the service account is only needed for this particular Subscription because it is a push to an e-mail server. Therefore, it needs a service account with the 'Service Account Token Creator' role.
We've attempted to redeploy the whole infrastructure and disable/re-enable the Pub/Sub API. Neither seemed to kick GCP into creating the Service Account. Further to this, we attempted to make the default service account manually. Still, GCP constrains the name a user can give a service account themselves, so we're unable to create a service account with the name that the Pub/Sub service would expect.
We wonder if there is some configuration of the project we may have missed or if anyone has seen this previously?
Does it not exist or does you not see it?
I'm pretty sure that it exists but without any role granted on it and you don't see it in the UI. Try to grant a role on this default service account, and it will appear in the IAM page!
I've created a service account I intend to use in our development environment, and since its credentials are checked into source control, I want to lock down its access to the bare minimum.
When spinning up new hosts, our app creates a new PubSub subscription, listens on the newly created subscription, then deletes it when finished. We chose this model over having a number of preconfigured/hardcoded subscriptions because the number of hosts scales up and down with time, we don't want host-specific config (In the form of a host-subscription map) if possible, etc. We want to have this same behavior in our local development environments ideally for debugging.
I created a specific "development" topic and granted our development service account the Pub/Sub Admin role, allowing it to attach subscriptions to that topic. I also granted the service account the project-level permission that allows it to create subscriptions. The service account can successfully create a new subscription and attach it to the topic, but when it tries to read from the new subscription, I receive a permission denied error.
I'm guessing this is because, when checking the newly created subscription, I notice that the service account that created it is not granted any permissions of any kind to that subscription. I would've expected the account that created a subscription to at least have read rights to it.
Is there a way for a service account to create subscriptions, attach them to a topic, and then read from that subscription... while not giving the service account access to any other subscriptions or topics? I'm aware I could just give the service account a "higher level" role with access to read all subscriptions, but that defeats the purpose of having a development-only account.
After your service account creates the subscription, it can call setIamPolicy on the subscription to grant itself read access to it. This works because you gave your service account the pubsub admin role on the project, which includes the ability to call setIamPolicy on any subscription in the project.
https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions/setIamPolicy