I'm trying to trigger a Cloud Function when I receive a message via Pub/Sub Subscriber. This subscriber is calling to another project (I'm able to see the messages in my project). But now, I need to trigger a Cloud Function to consume the message.
Do you have any idea how I should follow?
You can't plug a Cloud Functions in project A on a PubSub topic in a project B. You need to deploy a HTTP functions and to create a push subscription to call the function.
Note: you can create your push subscription in the project A or in the project B. All depends who will be charged for the subscription cost.
Related
I have an API that's from a third party (for me and for gcp), I want to know if it is possible to set a pull that get data from the API into PUBSUB, or from a wrapper API that I can set in my environment.
Basic schema:
(third party API) -Pull-> PUBSUB -Cloud Function-> BigQuery
I think in the code of the third party api, you can use the Google Pub Sub client, pull the messages and acknowledge them if there is no error.
The Cloud Pub/Sub client libraries use streaming pull to receive messages instead of pull.
The link for Pub Sub Python client for example :
https://googleapis.dev/python/pubsub/latest/pubsub/subscriber/index.html#pulling-a-subscription-asynchronously
You can also check this topic : How to pull messages from a subscription with GCP Pub/Sub?
You can write a cloud function, either scheduled via Cloud Scheduler or HTTP-triggered, that pulls the data and publishes to PubSub. Your next Cloud Function can then be PubSub triggered to move the data from your PubSub message to BigQuery.
There is an application already pushing to Google PubSub topic T1 in a different project and I would like to have the Google Cloud function triggered by the publishing on T1. I read the official document https://cloud.google.com/functions/docs/calling/pubsub that it is not possible to trigger cross-project and tried creating the cloud function too from the GCP console but do not see the topics from other projects even though I have access to those projects
I am looking for alternatives. Is it possible to achieve it in a different way?
The trick is to use HTTP communication and not native event communication. I mean you have to change your Background functions (that you plug on a PubSub topic) in a HTTP function.
By doing that, you also have to create a PubSub push subscription on your PubSub topic and to add the Cloud Functions HTTP URL as target.
Don't forget to configure the security part to allow only secure and authenticated communication between PubSub and your Function.
Alternatively, you can follow Piotr advice (in comment) and use other HTTP oriented product, like Cloud RUn. But the principle is still the same: PubSub push subscription is the key!
I was trying to create a google cloud function which can be triggered by a pubsub topic. I used the google deployment manager template for creating it. But I couldn't find any way to set the subscriber message filter for that cloud function.
I tried to create the subscription before and after the cloud function resource was created. But in no ways it worked.
If I create it before creating cloud function then after the resource is created it overrides the subscriber function and removes the filter.
Is it possible to set subscriber message filter for any pubsub cloud function using google deployment manager template?
In fact, it's not possible at all. When you deploy a Cloud functions in mode trigger-topic, you can't set a filter on the subscription (which is automatically created). And it's not a limitation of deployment manager.
If you want to trigger a Cloud Function on a PubSub message with the filter activated, you need to
Deploy your function in HTTP mode (trigger-http)
Create a Push Subscription to call in HTTP the Cloud Functions with the message in parameter. When you create the Push Subscription, this time, you can set the filter that you want
Secure the communication between the Push Subscription and the Cloud Functions (the security is automatically built in the trigger-topic mode (alias background function))
Let me try to explain to you what we are trying to do.
Saying it quickly: We want to give Pub/Sub Publisher (in our GoogleCloud) privileges to a GMail-API that is outside of our GoogleCloud.
What we have:
Following instructions here: https://developers.google.com/gmail/api/quickstart/ruby
We've created a project for GMail-API and the credentials in GoogleCloud, let's call it Cloud-A. (We'll not own this side in a production environment; this project and cloud will be managed by our customers' IT department.)
Next, we followed this other guide https://developers.google.com/gmail/api/guides/push
We created a Pub/Sub topic in our GoogleCloud (Cloud-B), we own this portion and it's the topic where we want to subscribe in order to listen for messages/notifications. (This topic is in a different account from the GMail-API that will be publishing messages that is Cloud-A).
So, following that last guide, it says that we need to give permissions to gmail-api-push#system.gserviceaccount.com and from my understanding what that means is that I'm giving privileges to GMail-API from Cloud-B to publish messages in Pub/Sub Cloud-B.
What I can't find out is a way to give permissions to Gmail-API from Cloud-A to publish messages in Pub/Sub Cloud-B.
To wrap up, I want to listen to Pub/Sub in Cloud-B that will receive notifications from Gmail-API in Cloud-A.
We used this https://github.com/googleapis/google-api-ruby-client/blob/master/generated/google/apis/gmail_v1/service.rb#L144 and I get an error saying that the topic doesn't exist (Probably because it is in Cloud-B and I'm configuring Cloud-A Gmail-Api)
I hope I was clear enough, we are not looking to given another project inside the same Google Cloud access to a Pub/Sub, it isn't even a service that we wrote since it is Gmail-Api and the only thing we are allowed to do is to send it the topic name we want it to publish in.
I'm not familiar with how the GMail-API publishes to Pub/Sub, but, if you have already figured out how to publish from GMail-API in project Cloud-A to a Pub/Sub topic in Cloud-A, you may try the following workarounds:
Alternative A:
Create the topic (topic-A) in project Cloud-A.
Create a pull subscription (subs-A) associated to topic-A also in project Cloud-A.
Create a service account (account-B) in project Cloud-B and grant it the Pub/Sub subscriber role for subscription subs-A.
Make your consumers (e.g. AppEngine, GKE, GCE) use service account account-B to pull messages from subs-A.
Alternative B:
Create the topic (topic-A) in project Cloud-A.
Create a push subscription (subs-A) associated to topic-A pointing to an endpoint of a service hosted in project Cloud-B (e.g. GCE, GKE, AppEngine, Cloud Function, etc.)
Alternative C:
Create the topic (topic-A) in project Cloud-A.
Create a pull or push subscription (subs-B) in project Cloud-B associated to topic-A in project Cloud-A. The user creating this subscription should have the Pub/Sub Editor role granted for topic-A.
Consume the messages from subs-B.
I would like to be able to forward a Google Cloud Platform (GCP) Pubsub subscription's messages to another GCP topic, which is possibly in another GCP Project.
Is this possible via Push endpoint URL (What would the url of the destination topic be?)
Some other easy configuration via API?
(workaround) Alternatively, how would you provide this functionality?
There is a Dataflow template called Cloud_PubSub_to_Cloud_PubSub that read messages from a Pubsub subscription (inputSubscription) and publish the messages to a Pubsub topic (outputTopic).
Be sure that the service account used to launch the Dataflow job has the right permissions (roles/pubsub.subscriber in the project where inputSubscription is defined or directly on inputSubscription, and roles/pubsub.publisher in the project containing outputTopic or directly on outputTopic).
You can take a look at the code on GitHub if you are familiar with Java.
The only way to do this would be to publish the messages to the other topic in a subscriber you write when it receives the messages. There is no automated way to forward messages from one topic to another.