I have a Cloud Run application built in Python and Docker. The application serves a dashboard that runs queries against data and displays visualizations and statistics. Currently, if I want the app to load quickly I have to set the minimum number of instances to a number greater than 0, I typically use 10. This is great for serving the app immediately, however it can become outdated. I would essentially like to be able to keep a minimum number of instances available to serve the app immediately, but I would like it if they would refresh, or shut down and start up, once every few hours or at least once a day. Is there a way to achieve this?
I have tried looking into Cloud Scheduler to somehow get the Cloud Run application to refresh on a schedule, but I was unclear on how to make the whole thing shut down and reload, especially without serving another revision.
I think about a design but I never tested it. Try to do that.
At startup, store in a global variable the time.Now(). You have the startup time
Implement a Healthcheck probe. The healthcheck answer OK (HTTP 200) if the NOW minus startup time (the global variable) is below X (1 hour for instance). Else it answer KO (HTTP 500)
Deploy your Cloud Run service with the new Health check feature
Like that, after X duration, the instance will autodeclare itself unhealthy and Cloud Run will evict it and create a new one.
It should work. Let me know, I'm interested in the result!
Related
Hi have a basic CI/CD setup
Whenever I push new code on my GitHub repository (trigger).
It should push it via Google Build to Google Run
But currently, whenever the trigger is initiated, it creates a new service on my cloud account.
Now, I am not sure why it's happening!
What I want is to update the current running service on Google Run
or
Create a new service, migrate 100% traffic to it and delete the old one.
This is my Deployment screen looks like. I have highlighted the section which I believe might be the reason for this duplicate services.
When you deploy a new version of your service (a new version can be a new image, or the same image with different parameters (concurrency, min/max instance, CPU, env vars,...)), it's named a revision. There is no limit on the number of revision (in fact yes, you have a limit at 1000, then the oldest will be deleted when a new one is created).
The pricing model of Cloud Run (and other serverless product) is simple: you pay when your service is running. In your case, you haven't a min instance parameter, and therefore your revision will run (and you will start to pay) when request invokes your service (and you pay only when you go over the free tier)
Same thing for Cloud Build: you start to pay when you use it. And you use it every time you push your code on GitHub (that invoke a Cloud Build trigger and then deploy your code on Cloud Run). Here again, you have a comfortable free tiers of 120 minutes free per day for Cloud Build.
I have a rest API running on cloud run that implements a cache, which needs to be cleared maybe once a week when I update a certain property in the database. Is there any way to send a HTTP request to all running instances of my application? Right now my understanding is even if I send multiple requests and there are 5 instances, it could all go to one instance. So is there a way to do this?
Let's go back to basics:
Cloud Run instances start based on a revision/image.
If you have the above use case, where suppose you have 5 instances running and you suddenly need to re-start them as restarting the instances resolves your use case, such as clearing/rebuilding the cache, what you need to do is:
Trigger a change in the service/config, so a new revision gets
created.
This will automatically replace, so will stop and relaunch all your instances on the fly.
You have a couple of options here, choose which is suitable for you:
if you have your services defined as yaml files, the easiest is to run the replace service command:
gcloud beta run services replace myservice.yaml
otherwise add an Environmental variable like a date that you increase, and this will yield a new revision (as a change in Env means new config, new revision) read more.
gcloud run services update SERVICE --update-env-vars KEY1=VALUE1,KEY2=VALUE2
As these operations are executed, you will see a new revision created, and your active instances will be replaced on their next request with fresh new instances that will build the new cache.
You can't reach directly all the active instance, it's the magic (and the tradeoff) of serverless: you don't really know what is running!! If you implement cache on Cloud Run, you need a way to invalidate it.
Either based on duration; when expired, refresh it
Or by invalidation. But you can't on Cloud Run.
The other way to see this use case is that you have a cache shared between all your instance, and thus you need a shared cache, something like memory store. You can have only 1 Cloud Run instance which invalidate it and recreate it and all the other instances will use it.
I use shell or vps from google cloud, and every time I want to run a process in this shelll when I exit the process in the shell also stops or dies.
I have used the SCREEN and TMUX commands, when I get out of the shell for 20 minutes the process shuts down by itself.
can I get the solution from the masters?
To answer shortly to your question: it's not possible.
With more detail, you need to understand what is Cloud Shell: it's a micro VM that google start freely for you. You have a quota per week and the VM is restarted at least every 12h
Non-interactive usage: Cloud Shell is intended for interactive use only. Non-interactive sessions will be ended automatically after a warning. Note that Cloud Shell sessions are capped at 12 hours, after which sessions are automatically terminated. You can use a new session immediately after.
When you aren't in interaction with the Cloud Shell console, it stops automatically after 20 minutes. Why? To save resource and save money (on Google side), because you don't pay this computing resource.
You have more detail on Cloud Shell environment limitations in the documentation
Anyway, if you want to run a batch job, on a small VM like Cloud Shell, you can use the Free tier compute engine instance for this. (choose correctly the region, else you will pay!)
At the moment I have a load balancer which runs a Compute Engine Instance Group which has a minimum of 1 server and a maximum of 5 servers.
This is running auto scaling and use a pre-build ubuntu template with all the base stuff needed.
When an instance boots up it will log a runner into the GitLab project, and then trigger the job to update the instance to the latest copy of the code.
This is fine and works well.
The issue comes when I make a change to the git branch and push the changes, it only seems to be being picked up by one of the random 5 instances that have loaded.
I was under the impression that GitLab would push out to all the runners logged, but this doesn't seem to be the case.
I have seen answers on here that show multiple runners, but on a single server, I haven't come across my particular situation.
Has anyone come across this before? I would assume that this is a pretty normal situation, and weird that it doesn't just work.
For each job that runs in GitLab, only 1 runner receives the job. The mechanism is PULL based -- the runners constantly ask GitLab if there's any jobs available to run. GitLab never initiates communication with the runners.
Therefore, your load balancer rules do nothing to affect which runner receives a job and there is no "fairness" in distributing jobs across server. Runners will keep asking for jobs every few seconds as long as they are able to take them (according to concurrency settings in the config.toml) and GitLab will hand them out on a first-come, first-served basis.
If you set the concurrency to 1 and start multiple jobs, you should see multiple servers pick up the jobs.
I have a software that process some files. What I need is:
start a default image on google cloud (I think docker should be a good solution) using an API or a run command
download files from google storage
process it, run my software using those downloaded files
upload the result to google storage
shut the image down, expecting not to be billed anymore
What I do know is how to create my image hehe. But I can't find any info saying me what google cloud service should I use or even if I could do it like I'm thinking. I think I'm not using the right keywords to find what i need.
I was looking at Kubernetes, but i couldn't figure out how to manipulate those instances to execute a one time processing.
[EDIT]
Explaining better the process I have an app that receive images and send it to Google storage. After that, I need to process that images, apply filters, georeferencing, split image etc. So I want to start a docker image to process it and upload the results to google cloud again.
If you are using any of the runtimes supported by Google Cloud Functions, they are easiest way to do those kind of operations (i.e. fetch something from Google Cloud Storage, perform some actions on those files and upload them again). The Cloud Functions will be triggered by an event of your choice, and after the job, it will die.
Next option in terms of complexity would be to deploy a Google App Engine application in standard environment. It allows you to deploy your own application written in any of the supported languages for this environment. While there is traffic in your application, you will have instances serving, but the number of instances running can go down to 0 when they are not serving, which would mean less cost.
Another option would be Google App Engine in flexible environment. This product allows you to deploy your application in any custom runtime. This option has always at least one instance running, so it would never shut down.
Lastly, you can use Google Compute Engine to "create and run virtual machines on Google infrastructure". Otherwise than GAE, this is not that managed by Google, which means that most of the configuration is up to you. In this case, you would need to programmatically indicate your VM to shut down after you have finished your operations.
Based on your edit where you stated that you already have an app that is inserting images into Google Cloud Storage, your easiest option would be to use Cloud Functions that are triggered by additions, changes, or deletions to objects in Cloud Storage buckets.
You can follow the Cloud Functions tutorial for Cloud Storage to get an idea of the generic process and then implement your own code that handles your specific tasks. There are other tutorials like the Imagemagick tutorial for Cloud Functions that might also be relevant to the type of processing you intend to do.
Cloud Functions is probably your lightest weight approach. You could of course do more full scale applications, but that is likely overkill, more expensive, and more complex. You can write your processing code in Node.js, Python, or Go.