I detached a subscription using the GUI at https://console.cloud.google.com/. The detachment went fine. Now I want to delete the subscription and need the subscription ID. I can't find it anywhere in the UI or when I run gcloud pubsub subscriptions list in the built in Cloud Shell terminal.
I expected the subscription to still appear in the subscription list in the project view. I assumed I would then be able to delete it from the UI.
I also expected that the subscription would show up when I listed the subscriptions from the cloud shell terminal but it was not in the list.
Related
The scenario is the following: I have two projects in the same organization and folder; let's call them project1 and project2. I have a Pub/Sub topic on project2 and I have a Cloud Run service on project1 that I want to use as the endpoint for a push subscription for the topic. However, I don't want everyone to be able to access my Cloud Run service: I only want authorised service accounts to be able to invoke the service. Thus I created a service account on project1 that has the role/run.invoker role for the Cloud Run service. Now, when I go and create my subscription in the topic in project2 with the Cloud Run service as the endpoint and I try to associate to it the service account in project1 using the following command:
gcloud --project=project2 pubsub subscriptions create test_subscrption --topic=topic-name --topic-project=project2 --push-auth-service-account=service_account1 --push-endpoint=https://my-cloud-run-service.run.app/
I get the following:
ERROR: Failed to create subscription [projects/project2/subscriptions/test_subscrption]: User not authorized to perform this action.
ERROR: (gcloud.pubsub.subscriptions.create) Failed to create the following: [test_subscrption].
Note that the account with which I'm logged into GCP has the Owner permission at the organization level. Moreover, if I create a service account in project2 and use that instead of the service account on project1, the subscription is created without problems. I'm aware that there are some permissions to set before associating a service account to a push subscription (serviceAccountTokenCreator as described here and iam.serviceAccounts.actAs as described here), but I can guarantee I did set those permission before trying to create my subscription, and I still got the same error. From all this I'm concluding that it is not possible to associate a service account from a different project to a push subscription in Pub/Sub (more to the point, if I try to create the subscription from the GCP UI in the dropdown menu for the service account to associate to the push subscription I only see the service accounts that are part of the same project as the topic and subscription). Can anybody confirm this, or tell me how to achieve my goal? My current solution has been to create a service account in project2 and associate that to the push subscription instead, but I would like to know if my original plan was even possible at all.
As mentioned in the comment, when you do it with the UI, you can only select the service account of the current project and it is impossible to use an external service account. Instead, you can create the push subscription in project1 and create the topic in project2.
My GCP Dataflow job immediately fails with this error message:
Workflow failed. Causes:
Subscription 'projects/project/subscriptions/subscription' not found.
Please supply an existing subscription.
The subscription does exist, I was able to click on it in the dataflow UI. I've been able to run this job previously, and it worked. It's only when I try redeploying it that it gives the error.
EDIT:
this is how I am getting the pubsub subscription
pipeline
.apply(PubsubIO.readProtos(...)
.fromSubscription(options.getSubscription()))
Workflow failed. Causes: Subscription ‘<subscription_name>’ not found. Please supply an existing subscription.
From the error message, it seemed like the pubsub subscription topic is not visible to the dataflow pipeline. Examing the permission on the service account shows that it has "pubsub.subscription.consume" permission already. Nothing looked out of order. So, we reported this issue to the Google Cloud team, and it turns out other developers are also facing a similar issue. Google Cloud has created a bug for the same.
Meanwhile, those who are facing this issue in the dataflow deployment can fix it by adding “pubsub.subscriptions.get” permission to the dataflow service account.
I am trying to subscribe to Gmail for mail notifications using Google's Pub/Sub and I've gone through the documentation and tutorials provided by Google. At one point, the docs state that I need to publish the correct rights to my PubSub topic:
You need to grant publish privileges to serviceAccount:gmail-api-push#system.gserviceaccount.com. You can do this using the Cloud Pub/Sub Developer Console permissions interface following the resource-level access control instructions.
My understanding is that the publish privilege is equivalent to pubsub.topics.publish or just "PubSub Publisher". But what about the serviceAccount:gmail-api-push#system.gserviceaccount.com? I cant' find that service account anywhere and the console is not letting me create it due to character restrictions. Is there another way to get this account?
You do not need to create the gmail service account, it already exists. You are correct about “granting publish privileges” as the same as “Pub/Sub Publisher”.
Under the topic permissions interface, I would pass in the service account gmail-api-push#system.gserviceaccount.com (note: take out the leading “serviceAccount:” part) in the ‘new members’ field, and select “Pub/Sub Publisher” as the role.
I am guessing you couldn’t find the service account because you were looking up serviceAccount:gmail-api-push#system.gserviceaccount.com.
Permissions are set by Pub/Sub Topic in the Pub/Sub Topics section of the GCP Console.
Go to the Google Cloud Console.
From the Hamburget menu, go to Big Data -> Pub/Sub -> Topics.
Shortcut to this location: https://console.cloud.google.com/projectselector2/cloudpubsub/topicList
Click the checkbox for the Topic.
At the right side of the console, select "SHOW INFO PANEL".
Click Add members
Enter the service account and select the desired role. Usually Pub/Sub Publisher.
I would like to define the steps in making a pubsub log export sink from one project to another. As a secondary goal, I would like the sink to bridge organizations. So far I have followed these steps as outlined in the gcloud help pages and the auth documentation.
First I created a sink:
# from project A
gcloud logging sinks create \
<sink_name> \
pubsub.googleapis.com/projects/<project_B>/topics/<topic_name> \
--log-filter <filter>
The CLI returns successfully and gives a little advice on setting up permissions for a service account it created:
Created [https://logging.googleapis.com/v2/projects/<project_A>/sinks/<sink_name>].
Please remember to grant `serviceAccount:<new_user>#logging-<project_A_account_number>.iam.gserviceaccount.com` Pub/Sub Publisher role to the topic.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export
Following this advice, I gave the new service account the appropriate permissions for that topic.
gcloud projects add-iam-policy-binding <project_B> \
--member serviceAccount:<new_user>#logging-<project_A_account_numbe_id>.iam.gserviceaccount.com \
--role roles/pubsub.publisher
This command returns without issue.
In spite of everything seeming OK, no logs flow through the sink.
Here are some clues: The Exports tab on the Logs Viewer reports a permissions error in the sink. The Project Activity tab reports a permissions issue.
Image: Logs Viewer, Exports
Image: Project, Activity
Is there a solution to make this work? Is it possible to generalize this to send logs to a sink in this project from other gcloud organizations?
I have been able to reproduce the scenario you wanted. Let me set the basics for the scenario description:
Project A (where my logs are stored): project-a
Project B (where my Pub/Sub Topic and Subscription are): project-b
Topic: projects/project-b/topics/myTopic
Subscription: mySub
Sink: test-sink
Then, this is the processed I followed:
In project A: create a filter for logs in the Logging > Logs tab in the Google Cloud Console.
Create an export with the elements in the image below (remember to append pubsub.googleapis.com/ to the name of your topic in the other project):
Move to the Exports tab and copy the Writer Identity, which should have the format test-sink#XXXXXXXX.iam.gserviceaccount.com
In project B: go to the IAM & admin > IAM tab in the Console and add a new member being the previous service account obtained in step 3 with the role Pub/Sub Editor enabled.
Create a Pub/Sub subscription with the command gcloud beta pubsub subscriptions create --topic myTopic mySub
Do some operation that results in logs read by the filter you specified in Project A.
Consume the logs written to the topic using the subscription, with the command gcloud beta pubsub subscriptions pull mySub.
There you will find the logs that are written from Project A to your tropic in Project B. I have reproduced this same scenario writing logs from a simple App Engine application (and therefore with the appropriate log filter searching for App Engine logs), and when I make requests to the App Engine app, some logs are created and then written in myTopic, which I can read using mySub.
Regarding your second question, I cannot make sure whether this same procedure works in a cross-organization scenario but I see no issues with that.
Matt, your gcloud commands to create a sink and grant publisher role to the service account used by the sink looks correct. The error you saw may be transient and it should have resolved after some time.
A possible reason for the error could be that there was a time delay between the two commands, and the sink tried to export logs immediately before you granted the IAM role. Can you confirm that the error resolved itself eventually?
To answer your last question, yes you can export logs from one project to a destination in another project in a different organization.
I am trying to add notification action in cloud watch through service catalog, I created new SNS topic and refer the topic in cloud watch actions section it failed, and also when ever I am creating cloud watch action through service catalog it ask me to enable IAM to make action even though I manually created alarm to stop an EC2 instance.