I'm trying to trigger a Cloud Function on a schedule using Cloud Scheduler. I have set up a service account with the Cloud Function Invoker role, set it as the service account on the scheduler, and set the auth header to "Add OIDC token". The URL is the same as the trigger URL for the cloud function.
When I run the scheduled job manually, the result comes back as "Success", so there doesn't appear to be any authentication issue. However, the result I'd expect (new data being appended to a BigQuery table) does not happen.
You might assume at this point that there is a problem with the Cloud Function, however when I run the function manually (without the scheduler), it works exactly as expected.
When I check the Cloud Function logs after running the scheduler manually, clearly the function has not been called, so it seems somehow the interaction between the Scheduler and the Function is not working. The strange thing is that I have set this up in exactly the same way as I've done with other scheduled functions in the past, which worked just fine, so I can't find a reason why this wouldn't be working.
Any ideas where I could be going wrong?
Here there seems to be an issue with the calls not reaching cloud function when scheduler tries to use the function-invoker service account to trigger the cloud function running as function-runner.The problem with Cloud Scheduler is that it cannot be used to trigger the function if it is set to “allow internal traffic”.
Internal-only HTTP functions can only be invoked by HTTP requests that are created within a VPC network, such as those from Kubernetes Engine, Compute Engine, or the App Engine Flexible Environment.
This means that events created by or routed through Pub/Sub, Eventarc, Cloud Scheduler, and Cloud Tasks cannot trigger these functions[1].
Please check the Load Balancer configurations to manage the traffic in Cloud Functions. So even though you choose the option of “Allow internal traffic and traffic from Cloud Load Balancing”, it is only using the part of “allow internal traffic” because there is no option to manage the load balancing. A workaround would be to create the load balancer[2] to manage the traffic in Cloud Functions[3], or you could select the option of “Allow all traffic” if it is acceptable to you.
[1] https://cloud.google.com/functions/docs/networking/network-settings#ingress_settings
[2] https://cloud.google.com/iap/docs/load-balancer-howto
[3] https://cloud.google.com/load-balancing/docs/https/setting-up-https-serverless
Related
We are currently deploying multiple instances (a front-end, a back-end, a database, etc..). All are deployed and configured using a CloudFormation script so we can deploy our solution quickly. One of our central server has multiple connections to other services and, for some, we open very simple REST endpoints that reply with 200 or 500 if the server can connect to another service or the database (get on /dbConnectionStatus for example).
We would like to have perform calls on those endpoints periodically and have a view on these. A little bit like the health check but without restarting the instance in case of trouble and possibly multiple endpoints to check on a service.
Is there an AWS service that can achieve that? If not what alternative do you suggest?
AWS CloudWatch Synthetic Monitoring can do what you want. By default it will just perform checks against your endpoints, and log the success or failure, without triggering a redeployment or something like a load balancer health check would.
CONTEXT:
We have a platform where users can create their own projects - multiple projects per user. We need to provide them with a browser-based IDE to edit those projects.
We decided to go with coder-server. For this we need to configure an auto-scalable cluster on AWS. When the user clicks "Edit Project" we will bring up a new container each time.
https://hub.docker.com/r/codercom/code-server
QUESTION:
How to pass parameters from the url query (my-site.com/edit?project=1234) into a startup script to pre-configure the workspace in a docker container when it starts?
Let's say the stack is AWS + ECS + Fargate. We could use kubernetes instead of ECS if it helps.
I don't have any experience in cluster configuration. Will appreciate any help or at least a direction where to dig further.
The above can be achieved using multiple ways in AWS ECS. The basic requirements for such systems are to launch and terminate containers on the fly while persisting the changes in the files. (I will focus on launching the containers)
Using AWS SDK's:
The task can be easily achieved using AWS SDKs, Using a base task definition. AWS SDK allows starting tasks with overrides on the base task definition.
E.G. If task definition has a memory of 2GB then the SDK can override the memory to parameterised value while launching a task from task def.
Refer to the boto3 (AWS SDK for Python) docs.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
Overall Solution
Now that we know how to run custom tasks with python SDK (on demand). The overall flow for your application is your API calling AWS lambda function whit parameters to spin up and wait to keep checking task status and update and rout traffic to it once the status is healthy.
API calls AWS lambda functions with parameters
Lambda function using AWS SDK create a new task with overrides from base task definition. (assuming the base task definition already exists)
Keep checking the status of the new task in the same function call and set a flag in your database for your front end to be able to react to it.
Once the status is healthy you can add a rule in the application load balancer using AWS SDK to route traffic to the IP without exposing the IP address to the end client (AWS application load balancer can get expensive, I'll advise using Nginx or HAProxy on ec2 to manage dynamic routing)
Note:
Ensure your Image is lightweight, and the startup times are less than 15 mins as lambda cannot execute beyond that. If that's the case create a microservice for launching ad-hoc containers and hosting them on EC2
Using Terraform:
If you looking for infrastructure provisioning terraform is the way to go. It has a learning curve so recommend it as a secondary option.
Terraform is popular for parametrising using variables and it can be plugged in easily as a backend for an API. The flow of your application still remains the same from step 1, but instead of AWS Lambda API will be calling your ad-hoc container microservice, which in turn calls terraform script and passing variables to it.
Refer to the Terrafrom docs for AWS
https://registry.terraform.io/providers/hashicorp/aws/latest
I'd like to call a Cloud Run app from inside a Cloud Function multiple times, given some logic. I've googled this quite a lot and don't find good solutions. Is this supported?
I've seen the Workflows Tutorials, but AFAIK they are meant to pass messages in series between different GPC services. My Cloud Function runs on a schedule every minute and it would only need to call the Cloud Run app a few times per day given some event. I've thought about having the entire app run in Cloud Run instead of the Cloud function. However, I think having it all in Cloud Run would be more expensive than running the Cloud function.
I went through your question, I have an alternative in my mind if you agree to the solution. You can use Cloud Scheduler to securely trigger a Cloud Run service asynchronously on a schedule.
You need to create a service account to associate with Cloud
Scheduler, and give that service account the permission to invoke
your Cloud Run service, i.e. Cloud Run invoker (You can use an
existing service account to represent Cloud Scheduler, or you can
create a new one for that matter)
Next, you have to create a Cloud Scheduler job that invokes your
service at specified times. Specify the frequency, or job interval,
at which the job is to run, using a configuration string. Specify the
fully qualified URL of your Cloud Run service, for example
https://myservice-abcdef-uc.a.run.app The job will send requests to
this URL.
Next, specify the HTTP method: the method must match what your
previously deployed Cloud Run service is expecting. When you deploy
the service using Cloud Scheduler, make sure you do not allow
unauthenticated invocations. Please go through this
documentation for details and try to implement the steps.
Back to your question, yes it's possible to call your Cloud Run service from inside Cloud Functions. Here, your Cloud Run service calls from another backend service i.e. Cloud Functions directly( synchronously) over HTTP, using its endpoint URL. For this use case, you should make sure that each service is only able to make requests to specific services.
Go through this documentation suggested by #John Hanley as it provides you with the steps you need to follow.
Usecase: Our requirement is to run a service continuously every few minutes. This service reads a value from datastore, and hits a public url using that value from datastore (Stateful). This service doesnt have Front End. No body would be accessing this service publicly. A new value is stored in datastore as a result of response from the url. Exactly one server is required to run.
We are in need to decide one of the below for our use case.
Compute Engine (IaaS -> we dont want to maintain the infra for this simple stateful application)
Kubernetes Engine (still feeling overkill )
App Engine : PaaS-> App Engine is usually used for Mobile apps, Gaming, Websites. App Engine provides a url with web address. Is it right choice for our usecase? If we choose app engine, is it possible to stop the public app engine url? Also, as one instance would be running continuously in app engine, what is cost effective - standard or flexible?
Cloud Functions -> Event Driven(looks not suitable for our application)
Google Cloud Scheduler-> We thought we could use cloud scheduler + cloud functions. But during outage, jobs are queued up. In our case, after outage, only one server/instance/job could be up and running.
Thanks!
after outage, only one server/instance/job could be up and running
Limiting Cloud Function concurrency is enough? If so, you can do this:
gcloud functions deploy FUNCTION_NAME --max-instances 1 FLAGS...
https://cloud.google.com/functions/docs/max-instances
I also recommend taking a look at Google Cloud Run, is a serverless docker platform, it can be limited to a maximum of 1 instances responding to a maximum of 1 request concurrently. It would require Cloud Scheduler too, making regular HTTP requests to it.
With both services configured with max concurrency of 1, only one server/instance/job will be up and running, but, after outages, jobs may be scheduled as soon as another finish. If this is problematic, adding a lastRun datetime field on datastore job row and not running if it's too recent, or disable retry of cloud scheduler, like said here:
Google Cloud Tasks HTTP trigger - how to disable retry
I trying to create some firewall rules in google compute, everything goes well, but some time later, they are just disappears.
I tried to add rules on default network, and also custom created - in both cases result same.
Tried both: through web UI, and through gcloud tool
If you believe that someone or something is reverting your Firewall changes, you can take multiple approaches to verify that.
inspect Cloud Console Activity logs
same using CLI: gcloud beta logging read "resource.type=gce_firewall_rule"
check GCE Operations section in Cloud Console
check GCE API requests in Cloud Console Logging, using this advanced filter:
resource.type="gce_firewall_rule"
jsonPayload.event_subtype:"compute.firewalls"