I'd like to call a Cloud Run app from inside a Cloud Function multiple times, given some logic. I've googled this quite a lot and don't find good solutions. Is this supported?
I've seen the Workflows Tutorials, but AFAIK they are meant to pass messages in series between different GPC services. My Cloud Function runs on a schedule every minute and it would only need to call the Cloud Run app a few times per day given some event. I've thought about having the entire app run in Cloud Run instead of the Cloud function. However, I think having it all in Cloud Run would be more expensive than running the Cloud function.
I went through your question, I have an alternative in my mind if you agree to the solution. You can use Cloud Scheduler to securely trigger a Cloud Run service asynchronously on a schedule.
You need to create a service account to associate with Cloud
Scheduler, and give that service account the permission to invoke
your Cloud Run service, i.e. Cloud Run invoker (You can use an
existing service account to represent Cloud Scheduler, or you can
create a new one for that matter)
Next, you have to create a Cloud Scheduler job that invokes your
service at specified times. Specify the frequency, or job interval,
at which the job is to run, using a configuration string. Specify the
fully qualified URL of your Cloud Run service, for example
https://myservice-abcdef-uc.a.run.app The job will send requests to
this URL.
Next, specify the HTTP method: the method must match what your
previously deployed Cloud Run service is expecting. When you deploy
the service using Cloud Scheduler, make sure you do not allow
unauthenticated invocations. Please go through this
documentation for details and try to implement the steps.
Back to your question, yes it's possible to call your Cloud Run service from inside Cloud Functions. Here, your Cloud Run service calls from another backend service i.e. Cloud Functions directly( synchronously) over HTTP, using its endpoint URL. For this use case, you should make sure that each service is only able to make requests to specific services.
Go through this documentation suggested by #John Hanley as it provides you with the steps you need to follow.
Related
I'm trying to trigger a Cloud Function on a schedule using Cloud Scheduler. I have set up a service account with the Cloud Function Invoker role, set it as the service account on the scheduler, and set the auth header to "Add OIDC token". The URL is the same as the trigger URL for the cloud function.
When I run the scheduled job manually, the result comes back as "Success", so there doesn't appear to be any authentication issue. However, the result I'd expect (new data being appended to a BigQuery table) does not happen.
You might assume at this point that there is a problem with the Cloud Function, however when I run the function manually (without the scheduler), it works exactly as expected.
When I check the Cloud Function logs after running the scheduler manually, clearly the function has not been called, so it seems somehow the interaction between the Scheduler and the Function is not working. The strange thing is that I have set this up in exactly the same way as I've done with other scheduled functions in the past, which worked just fine, so I can't find a reason why this wouldn't be working.
Any ideas where I could be going wrong?
Here there seems to be an issue with the calls not reaching cloud function when scheduler tries to use the function-invoker service account to trigger the cloud function running as function-runner.The problem with Cloud Scheduler is that it cannot be used to trigger the function if it is set to “allow internal traffic”.
Internal-only HTTP functions can only be invoked by HTTP requests that are created within a VPC network, such as those from Kubernetes Engine, Compute Engine, or the App Engine Flexible Environment.
This means that events created by or routed through Pub/Sub, Eventarc, Cloud Scheduler, and Cloud Tasks cannot trigger these functions[1].
Please check the Load Balancer configurations to manage the traffic in Cloud Functions. So even though you choose the option of “Allow internal traffic and traffic from Cloud Load Balancing”, it is only using the part of “allow internal traffic” because there is no option to manage the load balancing. A workaround would be to create the load balancer[2] to manage the traffic in Cloud Functions[3], or you could select the option of “Allow all traffic” if it is acceptable to you.
[1] https://cloud.google.com/functions/docs/networking/network-settings#ingress_settings
[2] https://cloud.google.com/iap/docs/load-balancer-howto
[3] https://cloud.google.com/load-balancing/docs/https/setting-up-https-serverless
I've created a job in Google Cloud Scheduler to download data from my demo app using HTTP GET it seemed to run successfully. My question is where did it store that data? And how can I store it into Google Cloud Storage? Below is a screenshot of my job:
I'm new to Google Cloud and working with a free trial account. Please advise.
Cloud Scheduler does not process data. The data returned by your example request is discarded.
Write a Cloud Function scheduled by Cloud Scheduler. There are other services such as Firebase and Cloud Run that work well also.
What you're trying to do is that you're trying to create a scheduler job that calls a GET request, and Cloud Scheduler will do exactly that, except the part where data is stored. Since Cloud Scheduler is a managed-cron service, it doesn't matter if the URL returns data. Cloud Scheduler will call the endpoint on a timely manner and that's it. Data returned by the request will be discarded, as mentioned by John Hanley.
What you can do is to integrate scheduling with Cloud Functions, but to be clear, your app needs to do the following first:
Download from external link and save the object to /tmp within the function.
The rest of the file system is read-only, and /tmp is the only writeable part. Any files saved in /tmp are stored within the function's memory. You can clear /tmp for every successful upload to GCS.
Upload the file to Cloud Storage (using Cloud Storage client library).
Now that your app is capable of storing and uploading data, then you can make a decision where to deploy it.
You can deploy your code using Firebase CLI and use scheduled functions. The advantage is that Pub/Sub and Cloud Scheduler is taken care of automatically and configuration is done on your code. Downside is that you are limited to Node Runtime compared to GCP Cloud Functions, where there are many different programming languages available. Learn more about scheduled functions.
Second, you can deploy through gcloud CLI but for this, you need to setup Pub/Sub notifications and Cloud Scheduler. You can check more about this by navigating to this link.
I'm working on a Cloud Run docker application that handles a few long-running data integration processes.
I'm struggling to come up with a way to locally run/test my submissions to Cloud Tasks before actually deploying the container to Cloud Run.
Is there any way to do this?
A local emulator for Cloud Tasks is not available yet, in some cases you can substitute Cloud Tasks with Pub/Sub.
Also, consider to use non Google solutions such as Cloud-Tasks-In-Process-Emulator, gcloud-tasks-emulator 0.5.1 or Cloud tasks emulator.
As I can understand you want to test the cloud task locally! Yes it is possible by using ngrok. By using ngrok you can access your local application on public and for cloud task you need the public url for handling task.
Usecase: Our requirement is to run a service continuously every few minutes. This service reads a value from datastore, and hits a public url using that value from datastore (Stateful). This service doesnt have Front End. No body would be accessing this service publicly. A new value is stored in datastore as a result of response from the url. Exactly one server is required to run.
We are in need to decide one of the below for our use case.
Compute Engine (IaaS -> we dont want to maintain the infra for this simple stateful application)
Kubernetes Engine (still feeling overkill )
App Engine : PaaS-> App Engine is usually used for Mobile apps, Gaming, Websites. App Engine provides a url with web address. Is it right choice for our usecase? If we choose app engine, is it possible to stop the public app engine url? Also, as one instance would be running continuously in app engine, what is cost effective - standard or flexible?
Cloud Functions -> Event Driven(looks not suitable for our application)
Google Cloud Scheduler-> We thought we could use cloud scheduler + cloud functions. But during outage, jobs are queued up. In our case, after outage, only one server/instance/job could be up and running.
Thanks!
after outage, only one server/instance/job could be up and running
Limiting Cloud Function concurrency is enough? If so, you can do this:
gcloud functions deploy FUNCTION_NAME --max-instances 1 FLAGS...
https://cloud.google.com/functions/docs/max-instances
I also recommend taking a look at Google Cloud Run, is a serverless docker platform, it can be limited to a maximum of 1 instances responding to a maximum of 1 request concurrently. It would require Cloud Scheduler too, making regular HTTP requests to it.
With both services configured with max concurrency of 1, only one server/instance/job will be up and running, but, after outages, jobs may be scheduled as soon as another finish. If this is problematic, adding a lastRun datetime field on datastore job row and not running if it's too recent, or disable retry of cloud scheduler, like said here:
Google Cloud Tasks HTTP trigger - how to disable retry
I want to start/stop a set of Compute engine instances in Google Cloud Platform using Google Cloud Scheduler. How can I do it?
In order to start and stop a Compute Engine using the Cloud Scheduler you can follow Google this tutorial, or this other
I won’t be copy-pasting the required code here because the tutorial it's very complete but I will resume here the steps to follow.
Set up your Compute Engine instances
Deploy the starter Cloud Function. You can see an example in here
Deploy the stop Cloud Function. You can see an example in here
Set up the Cloud Scheduler jobs
If you need any help with the tutorial please just let me know!
I still wonder why gcp has still not have this feature in the first place.
Anyways These simple steps did the job for me
Create a new JobScheduler.
Fill in the required details
Choose frequency which suits your requirement.
Choose the target to Pub/Sub.
Choose the topic name (Create a new topic if not created ).
In the payload section use this stop script
gcloud compute instances stop instance-name.
To verify the change you can run the job manually and check
I use vm instance API directly. No need for a cloud function.
Here is the link to the api description:
https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop
The API Call: POST https://compute.googleapis.com/compute/v1/projects/{project}/zones/{zone}/instances/{resourceId}/stop
You can start the engine in a similiar way.
Example how to configure the scheduler:
You can look at Google Article to achieve your goal https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule.
Also, If these VM instances are stateless then I would suggest to look at Google Cloud Run service which can help you to save cost and operation overhead to configure auto-shutdown/auto-startup.
Hope this helps.
The new Google Compute Engine feature of Instance Schedules can now be used to start and stop instances through the Cloud Console UI, using gcloud or via the API:
https://cloud.google.com/compute/docs/instances/schedule-instance-start-stop