Does anyone successfully set up GCP Cloud Build trigger via pub/sub notification with Cloud Storage OBJECT_FINALIZE event (new object uploaded)?
I have followed the GCP documentation, but nothing worked. This is what I did:
Create a topic
Create a notification via gsutil as per documentation which sends a message to the topic upon OBJECT_FINALIZE event on Google Cloud Storage (https://cloud.google.com/storage/docs/reporting-changes)
Create the Cloud Build trigger with Pub/Sub subscription to the topic above
I also ensure that Cloud Build API is enabled and the cloud build service account has permissions to read the message from the topic
But when i uploaded new file to the Cloud Storage bucket, the build pipeline was never triggered.
Please help if you already set this up before. Thanks.
Related
Here’s what I do:
Set up a Cloud Pub/Sub using my own Google Cloud Platform (GCP) project.
Create a topic
Allow Google Play to publish notifications on the topic.
Tell my Google Play Console about my topic.
Get the Authentication Key for your Backend .
Added backend service as a subscriber to the topic .
----> faild here notifications not triger my cloud functions
Verify that receiving the notifications.
Update your backend with the new subscription change .
I have a template that creates a few resources in GCP, and I want it to either call an HTTP endpoint or publish a message to a topic whenever the deployment completes. I've been checking different services all day, and couldn't find anything about it.
In AWS, it is quite easy to deploy an SNS message that is published to an SNS topic, which is subscribed to SQS Queue, and that triggers a lambda function. In Azure ARM templates, we can use az CLI to invoke a web request and call an endpoint directly.
I couldn't find any similar in GCP. Is there any way of either calling an HTTP endpoint, Cloud Function or perhaps publishing a message to a topic whenever a deployment is finished?
I really appreciate any help.
The best approach in GCP is to Create a Logging Sink using a filter and the Logging query language to only send the Deployment Manager logs to a PubSub topic.
Here is an example of a sink sending deployment manager logs to a PubSub topic previously created:
gcloud logging sinks create $SINK_NAME pubsub.googleapis.com/projects/$YOUR_PROJECT/topics/$TOPIC \
--log-filter='resource.type="deployment"' --description="my sink"
Be careful to Set the destination permissions or will not see the logs in the PubSub topic.
Once you are getting the logs in the PubSub topic, you can configure Cloud Pub/Sub Triggers to fire up an HTTP call based on content of the log.
I have created a set of cloud functions that work to ingest data into google cloud storage. The functions have been set with a get http request to only accept internal traffic.
However, when I use cloud scheduler to to invoke the functions I continually get permissions errors even while after specifying a service account for each of the functions with the proper permissions. I have set each of the functions to be in the us-central1 region and have researched the docs and Stack overflow with no success so far. Can I receive some assistance with this?
Cloud Scheduler is a serverless product. This means it doesn't belong to your project and not send the request to your Cloud Function through the VPC. In addition, Cloud Scheduler isn't yet supported in VPC SC
Thus, you can't. The workaround is to allow all ingress traffic on cloud function and to uncheck allow-unauthenticated access. Therefore, your function is callable from elsewhere (from internet) BUT you need a valid authentication to invoke it.
Use your service account and add it to Cloud Scheduler for invoking your function. Grant it the sufficient role for this
Alternative
However, if you would like initially not deploy your function publicly accessible on internet (allow internal traffic only ingress mode), there is an alternative.
Change your Cloud Scheduler to publish a PubSub message instead of calling directly your function. Then, deploy your function linked to PubSub topic instead of in HTTP target mode.
You might have some update to perform in your code, especially if you have parameters to handle (initially in the query or the body, now all is in the PubSub message published by Cloud Scheduler). But your function in only callable by your PubSub topic and no other way.
According to the documentation, in order to trigger a Cloud Function from Cloud Scheduler you have to use Pub/Sub. These are the steps:
Create the Cloud Function and make it trigger by a Pub/Sub topic.
Create the Pub/Sub topic.
Create the Cloud Scheduler job that will invoke the Pub/Sub trigger.
Once you do that you will be able to test-run the Cloud Scheduler job and verify whether it's working now. The final schema is something like this:
Cloud Scheduler job => Pub/Sub topic => Cloud Function
Once it's working remember to revert the roles granted to the Cloud Scheduler service account, as this method doesn't require them.
Here I found a blog post that does the same but with a more practical approach that you can follow from a CLI.
Let me try to explain to you what we are trying to do.
Saying it quickly: We want to give Pub/Sub Publisher (in our GoogleCloud) privileges to a GMail-API that is outside of our GoogleCloud.
What we have:
Following instructions here: https://developers.google.com/gmail/api/quickstart/ruby
We've created a project for GMail-API and the credentials in GoogleCloud, let's call it Cloud-A. (We'll not own this side in a production environment; this project and cloud will be managed by our customers' IT department.)
Next, we followed this other guide https://developers.google.com/gmail/api/guides/push
We created a Pub/Sub topic in our GoogleCloud (Cloud-B), we own this portion and it's the topic where we want to subscribe in order to listen for messages/notifications. (This topic is in a different account from the GMail-API that will be publishing messages that is Cloud-A).
So, following that last guide, it says that we need to give permissions to gmail-api-push#system.gserviceaccount.com and from my understanding what that means is that I'm giving privileges to GMail-API from Cloud-B to publish messages in Pub/Sub Cloud-B.
What I can't find out is a way to give permissions to Gmail-API from Cloud-A to publish messages in Pub/Sub Cloud-B.
To wrap up, I want to listen to Pub/Sub in Cloud-B that will receive notifications from Gmail-API in Cloud-A.
We used this https://github.com/googleapis/google-api-ruby-client/blob/master/generated/google/apis/gmail_v1/service.rb#L144 and I get an error saying that the topic doesn't exist (Probably because it is in Cloud-B and I'm configuring Cloud-A Gmail-Api)
I hope I was clear enough, we are not looking to given another project inside the same Google Cloud access to a Pub/Sub, it isn't even a service that we wrote since it is Gmail-Api and the only thing we are allowed to do is to send it the topic name we want it to publish in.
I'm not familiar with how the GMail-API publishes to Pub/Sub, but, if you have already figured out how to publish from GMail-API in project Cloud-A to a Pub/Sub topic in Cloud-A, you may try the following workarounds:
Alternative A:
Create the topic (topic-A) in project Cloud-A.
Create a pull subscription (subs-A) associated to topic-A also in project Cloud-A.
Create a service account (account-B) in project Cloud-B and grant it the Pub/Sub subscriber role for subscription subs-A.
Make your consumers (e.g. AppEngine, GKE, GCE) use service account account-B to pull messages from subs-A.
Alternative B:
Create the topic (topic-A) in project Cloud-A.
Create a push subscription (subs-A) associated to topic-A pointing to an endpoint of a service hosted in project Cloud-B (e.g. GCE, GKE, AppEngine, Cloud Function, etc.)
Alternative C:
Create the topic (topic-A) in project Cloud-A.
Create a pull or push subscription (subs-B) in project Cloud-B associated to topic-A in project Cloud-A. The user creating this subscription should have the Pub/Sub Editor role granted for topic-A.
Consume the messages from subs-B.
When deploying a background Cloud Function for Cloud Pub/Sub via:
gcloud functions deploy function_name --runtime python37 --trigger-topic some_topic
A subscription gets automatically created with a push endpoint (likely App Engine standard endpoint, but those are claimed to be without the need of
domain verification https://cloud.google.com/pubsub/docs/push#other-endpoints).
For the generated subscription/endpoint there doesn't seem like a way to register/verify the domain
(https://www.google.com/webmasters/verification, https://console.cloud.google.com/apis/credentials/domainverification)
As a result the user cannot alter the subscription, for instance:
gcloud alpha pubsub subscriptions update some_subscription --ack-deadline=10
Would yield something like "INVALID_ARGUMENT: The supplied HTTP URL is not registered in the subscription's parent project"
Doing it via the web interface on GCP would also yield an error: "pubsub error INVALID_ARGUMENT"
When connecting to a Cloud Pub/Sub topic via Cloud Functions, a subscription is created on your behalf. This subscription is managed by Cloud Functions and therefore it is not possible to update the acknowledgement deadline.