I have created a pubsub topic and subscriber and configured a public API for receiving the push notification. I have able to receive sample push notification in the API through topic. I would like to know if there is any way in which I can receive all GCP asset creation/ updation details as push notification via pub sub. I need all the details like asset creation date, who created, project etc...Is this doable in GCP pubsub
You have 2 solutions to achieve that:
Use Cloud Asset inventory
Use Audit logs
Cloud Asset inventory monitoring is a built in solution that publish a message on each asset change
The tradeoff that, for now, all the assets aren't supported. The most common yes, but depends on your needs.
The other solution is to create a Cloud Logging sink to PubSub, with a filter only on the System Audit Logs
Related
I have created an alert policy in google cloud platform. I am getting the emails notifications based on the alert policy.
Now I want to configure an external custom monitoring system for this alert. I want to know the REST APIs that this monitoring system can call at after every 10-20 sec and get the status of this alert.
Please help
If you want to use only Cloud Monitoring, and Alerting Policy you won't achieve this.
It is not possible to configure alert policies to notify (repeatedly) while the policy's conditions are met. Alert policies that are created through Google Cloud Console send a notification only when the condition is met. You can also receive a notification when the condition stops being met.
Additional information can be found in Notifications per incident documentation.
In Cloud Monitoring API v3 - Alerting policies, you can find information that only the creation of the incident is sent.
An alerting policy is a configuration resource that describes the criteria for generating incidents and how to notify you when those incidents are created.
In general, if you want to use Notification Channels to send notifications outside you can use Webhooks or PubSub.
Note
Webhooks only support public endpoints. If you need notifications sent to an endpoint that isn't public, then create a Pub/Sub notification channel and configure a subscription to the Pub/Sub topic. For more information, see Webhook notifications fail when configured for a private endpoint.
As you didn't provide more information it's hard to say if you are not using some built-in features in 3rd party software to integrate with GCP Cloud Monitoring. One of the example is Grafana:
Grafana ships with built-in support for Google Cloud Monitoring. Add it as a data source to build dashboards for your Google Cloud Monitoring metrics.
GCP also might use Prometheus features. Maybe this might give you something similar to what you want.
Prometheus is a monitoring tool often used with Kubernetes. If you configure Cloud Operations for GKE and include Prometheus support, then the metrics that are generated by services using the Prometheus exposition format can be exported from the cluster and made visible as external metrics in Cloud Monitoring.
There are some workarounds, however they won't fulfill what you want.
It is possible to create multiple conditions that identify the same issue. Every time a condition is met, a notification will be received.
It is possible to get users notified when a condition is NOT met, however this might cause spam messages.
The last thing I want to mention is that there is already a Feature Request to add multiple notifications until the condition is gone. More details in FR: Repeat Notifications until condition is gone.
Additional Documentation:
Monitoring Alerts in GCP by integrating Cloud Operations with Notification Channels
Conclusion
Alert policies that are created through Google Cloud Console send a notification only when the condition is met. You can also enable notification to get solved notification.
There is Feature Request to add repeatedly notifications - here
To send notifications to other apps/resources you can use Webhooks or PubSub.
Let me try to explain to you what we are trying to do.
Saying it quickly: We want to give Pub/Sub Publisher (in our GoogleCloud) privileges to a GMail-API that is outside of our GoogleCloud.
What we have:
Following instructions here: https://developers.google.com/gmail/api/quickstart/ruby
We've created a project for GMail-API and the credentials in GoogleCloud, let's call it Cloud-A. (We'll not own this side in a production environment; this project and cloud will be managed by our customers' IT department.)
Next, we followed this other guide https://developers.google.com/gmail/api/guides/push
We created a Pub/Sub topic in our GoogleCloud (Cloud-B), we own this portion and it's the topic where we want to subscribe in order to listen for messages/notifications. (This topic is in a different account from the GMail-API that will be publishing messages that is Cloud-A).
So, following that last guide, it says that we need to give permissions to gmail-api-push#system.gserviceaccount.com and from my understanding what that means is that I'm giving privileges to GMail-API from Cloud-B to publish messages in Pub/Sub Cloud-B.
What I can't find out is a way to give permissions to Gmail-API from Cloud-A to publish messages in Pub/Sub Cloud-B.
To wrap up, I want to listen to Pub/Sub in Cloud-B that will receive notifications from Gmail-API in Cloud-A.
We used this https://github.com/googleapis/google-api-ruby-client/blob/master/generated/google/apis/gmail_v1/service.rb#L144 and I get an error saying that the topic doesn't exist (Probably because it is in Cloud-B and I'm configuring Cloud-A Gmail-Api)
I hope I was clear enough, we are not looking to given another project inside the same Google Cloud access to a Pub/Sub, it isn't even a service that we wrote since it is Gmail-Api and the only thing we are allowed to do is to send it the topic name we want it to publish in.
I'm not familiar with how the GMail-API publishes to Pub/Sub, but, if you have already figured out how to publish from GMail-API in project Cloud-A to a Pub/Sub topic in Cloud-A, you may try the following workarounds:
Alternative A:
Create the topic (topic-A) in project Cloud-A.
Create a pull subscription (subs-A) associated to topic-A also in project Cloud-A.
Create a service account (account-B) in project Cloud-B and grant it the Pub/Sub subscriber role for subscription subs-A.
Make your consumers (e.g. AppEngine, GKE, GCE) use service account account-B to pull messages from subs-A.
Alternative B:
Create the topic (topic-A) in project Cloud-A.
Create a push subscription (subs-A) associated to topic-A pointing to an endpoint of a service hosted in project Cloud-B (e.g. GCE, GKE, AppEngine, Cloud Function, etc.)
Alternative C:
Create the topic (topic-A) in project Cloud-A.
Create a pull or push subscription (subs-B) in project Cloud-B associated to topic-A in project Cloud-A. The user creating this subscription should have the Pub/Sub Editor role granted for topic-A.
Consume the messages from subs-B.
I'm exporting google analytics data to BigQuery.
Is it possible to set a notification that export of GA data is complete?
It is kind of possible. You need Stackdriver Logging and Pub/Sub. You export logging events that the user "analytics-processing-dev#system.gserviceaccount.com" triggers from Stackdriver to Pub/Sub. From Pub/Sub you push the messages to a URL with a service behind - e.g. you can build a chatbot or mailing service.
But there is no out of the box notification system afaik.
I find the docs for gcloud stuff detailed but hardly the easiest to follow. Is anyone out there able to tell me what I'm doing that means my topic never received the logs I believe it should. Many thanks:
Create a project called logproj, enable PubSub API and create a topic called mytopic.
Create organization wide aggregated sink with the destination of the PubSub topic I just created, with the sink containing admin activity logs only:
gcloud logging sinks create mysink pubsub.googleapis.com/projects/logproj-12345/topics/mytopic --include-children --organization=123456789123 --log-filter='"logName:activity" AND logName:"/logs/cloudaudit.googleapis.com%2Factivity"'
The above command completes successfully and gives me a service account called o78732a92983-3234626#gcp-sa-logging.iam.gserviceaccount.com that I need to add to the topic and give PubSub Publisher role, which I do via the Web UI.
In order to test the topic is receiving organization wide activity logs, I create a test project. Then use the log viewer and select the pubsub topic, but the only logs I can see are from me creating the topic in the logproj project.
What am I missing?
Thanks in advance.
Messages published to Pub/Sub cannot be found in the Stackdriver logs. If you want to view the messages published to your topic:
Create a subscription for your topic
Use command-line to pull messages: https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull
Alternatively, you could implement a simple subscriber. Code examples can be found here: https://cloud.google.com/pubsub/docs/pull
I would like to be able to forward a Google Cloud Platform (GCP) Pubsub subscription's messages to another GCP topic, which is possibly in another GCP Project.
Is this possible via Push endpoint URL (What would the url of the destination topic be?)
Some other easy configuration via API?
(workaround) Alternatively, how would you provide this functionality?
There is a Dataflow template called Cloud_PubSub_to_Cloud_PubSub that read messages from a Pubsub subscription (inputSubscription) and publish the messages to a Pubsub topic (outputTopic).
Be sure that the service account used to launch the Dataflow job has the right permissions (roles/pubsub.subscriber in the project where inputSubscription is defined or directly on inputSubscription, and roles/pubsub.publisher in the project containing outputTopic or directly on outputTopic).
You can take a look at the code on GitHub if you are familiar with Java.
The only way to do this would be to publish the messages to the other topic in a subscriber you write when it receives the messages. There is no automated way to forward messages from one topic to another.