I have a rest API running on cloud run that implements a cache, which needs to be cleared maybe once a week when I update a certain property in the database. Is there any way to send a HTTP request to all running instances of my application? Right now my understanding is even if I send multiple requests and there are 5 instances, it could all go to one instance. So is there a way to do this?
Let's go back to basics:
Cloud Run instances start based on a revision/image.
If you have the above use case, where suppose you have 5 instances running and you suddenly need to re-start them as restarting the instances resolves your use case, such as clearing/rebuilding the cache, what you need to do is:
Trigger a change in the service/config, so a new revision gets
created.
This will automatically replace, so will stop and relaunch all your instances on the fly.
You have a couple of options here, choose which is suitable for you:
if you have your services defined as yaml files, the easiest is to run the replace service command:
gcloud beta run services replace myservice.yaml
otherwise add an Environmental variable like a date that you increase, and this will yield a new revision (as a change in Env means new config, new revision) read more.
gcloud run services update SERVICE --update-env-vars KEY1=VALUE1,KEY2=VALUE2
As these operations are executed, you will see a new revision created, and your active instances will be replaced on their next request with fresh new instances that will build the new cache.
You can't reach directly all the active instance, it's the magic (and the tradeoff) of serverless: you don't really know what is running!! If you implement cache on Cloud Run, you need a way to invalidate it.
Either based on duration; when expired, refresh it
Or by invalidation. But you can't on Cloud Run.
The other way to see this use case is that you have a cache shared between all your instance, and thus you need a shared cache, something like memory store. You can have only 1 Cloud Run instance which invalidate it and recreate it and all the other instances will use it.
Related
I have a Cloud Run application built in Python and Docker. The application serves a dashboard that runs queries against data and displays visualizations and statistics. Currently, if I want the app to load quickly I have to set the minimum number of instances to a number greater than 0, I typically use 10. This is great for serving the app immediately, however it can become outdated. I would essentially like to be able to keep a minimum number of instances available to serve the app immediately, but I would like it if they would refresh, or shut down and start up, once every few hours or at least once a day. Is there a way to achieve this?
I have tried looking into Cloud Scheduler to somehow get the Cloud Run application to refresh on a schedule, but I was unclear on how to make the whole thing shut down and reload, especially without serving another revision.
I think about a design but I never tested it. Try to do that.
At startup, store in a global variable the time.Now(). You have the startup time
Implement a Healthcheck probe. The healthcheck answer OK (HTTP 200) if the NOW minus startup time (the global variable) is below X (1 hour for instance). Else it answer KO (HTTP 500)
Deploy your Cloud Run service with the new Health check feature
Like that, after X duration, the instance will autodeclare itself unhealthy and Cloud Run will evict it and create a new one.
It should work. Let me know, I'm interested in the result!
Hi have a basic CI/CD setup
Whenever I push new code on my GitHub repository (trigger).
It should push it via Google Build to Google Run
But currently, whenever the trigger is initiated, it creates a new service on my cloud account.
Now, I am not sure why it's happening!
What I want is to update the current running service on Google Run
or
Create a new service, migrate 100% traffic to it and delete the old one.
This is my Deployment screen looks like. I have highlighted the section which I believe might be the reason for this duplicate services.
When you deploy a new version of your service (a new version can be a new image, or the same image with different parameters (concurrency, min/max instance, CPU, env vars,...)), it's named a revision. There is no limit on the number of revision (in fact yes, you have a limit at 1000, then the oldest will be deleted when a new one is created).
The pricing model of Cloud Run (and other serverless product) is simple: you pay when your service is running. In your case, you haven't a min instance parameter, and therefore your revision will run (and you will start to pay) when request invokes your service (and you pay only when you go over the free tier)
Same thing for Cloud Build: you start to pay when you use it. And you use it every time you push your code on GitHub (that invoke a Cloud Build trigger and then deploy your code on Cloud Run). Here again, you have a comfortable free tiers of 120 minutes free per day for Cloud Build.
I have two AWS Elasticache instances, One of the instances (lets say instance A) has very important data sets and connections on it, downtime is unacceptable. Because of this situation, instead of doing normal migration (like preventing source data from new writes, getting dump on it, and restore it to the new one) I'm trying to sync instance A's data to another Elasticache instance (lets say instance B). As I said, this process should be downtime-free. In order to do that, I tried RedisShake, but because AWS restrict users to run certain commands (bgsave, config, replicaof,slaveof,sync etc), RedisShake is not working with AWS Elasticache. It's giving the error below.
2022/04/04 11:58:42 [PANIC] invalid psync response, continue, ERR unknown command `psync`, with args beginning with: `?`, `-1`,
[stack]:
2 github.com/alibaba/RedisShake/redis-shake/common/utils.go:252
github.com/alibaba/RedisShake/redis-shake/common.SendPSyncContinue
1 github.com/alibaba/RedisShake/redis-shake/dbSync/syncBegin.go:51
github.com/alibaba/RedisShake/redis-shake/dbSync.(*DbSyncer).sendPSyncCmd
0 github.com/alibaba/RedisShake/redis-shake/dbSync/dbSyncer.go:113
github.com/alibaba/RedisShake/redis-shake/dbSync.(*DbSyncer).Sync
... ...
I've tried rump for that matter, But it doesn't have enough stability to handle any important processes. First of all, it's not working as a background process, when the first sync finished, it's being closed with signal: exit done, so it will not be getting ongoing changes after the first finish.
Second of all, it's recognizing created/modified key/values in each run, for example, in first run key apple equals to pear, it's synced to the destination as is, but when I deleted the key apple and its value in source and ran the rump syncing script again, it's not being deleted in destination. So basically it's not literally syncing the source and the destination. Plus, last commit to the rump github repo is about 3 years ago. It seems a little bit outdated project to me.
After all this information and attempts, my question is, is there a way to sync two Elasticache for Redis instances, as I said, there is no room for downtime in my case. If you guys with this kind of experience have a bulletproof suggestion, I would be much appreciated. I tried but unfortunately didn't find any.
Thank you very much,
Best Regards.
If those two Elasticache Redis clusters exist in the same account but different regions, you can consider using AWS Elasticache global-datastore.
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Console.html
It has some restrictions on the regions, type of nodes and that both the clusters should have same configurations in terms of number of nodes, etc.
Limitations - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Getting-Started.html
Otherwise, there's a simple brute-force mechanism and you would be able to code it yourself I believe.
Create a client EC2 (let's call this Sync-er) pub-sub channel from your EC Redis instance A.
Whenever there is a new data, Sync-er would make WRITE commands on EC Redis instance B.
NOTE - You'll have to make sure that the clusters are in connectable VPCs.
Elasticache is only available to the resources within the VPC. If your Instance A and Instance B are in different VPCs, you'll have to peer them or connect them via TransitGateway.
I have a cloud run with multiple instances running or idle.
I want all the instances to do an async job periodically (to rebuild a cache).
Example of async job:
Periodically check if there is a new version of a JSON file on the object storage bucket
Do some processing on the JSON and store it as a variable (cache) that will be used by the API endpoints. So I do not need to contact database on each request.
Options on how to do it:
setInterval() to call rebuildCacheIfNeeded(). Cloud run cannot do async tasks in the background (they are assigned CPU resources only while handling a request).
webcron will not work. Only one instance would handle the request and the cache would be rebuild only on that instance.
Pub / sub on new file added to the bucket. Can pub/sub be setup in the way that all instances are awaken and all will rebuild the cache? If yes, this would be the best solution.
Call rebuildCacheIfNeeded() on each request and keep the http connection until the cache is rebuild. I would like to avoid this for obvious reasons.
Kill all instances of cloud run when new file is added to the bucket. Cloud run should be stateless, so this solution is the only one that complies with statelessness rule. But how kill all instances without running whole redeploy?
Any other possible solutions that I am missing?
Thank you
Please do not suggest "Just use a database"... The cached data is small and I would like to avoid a database latencies and possible point of failure.
You are trying to use side-effects of a service that is neither predictable nor manageable. That will lead to problems today and possibly failure when features are updated or new features are released. Design your application to use documented features.
There is no documented method to achieve your objective.
I have a simple pure C program, that takes an integer as input, runs for a while (let's say an hour) and then returns me a text file. I want to run this program 1000 times with input integers from 1 to 1000.
Currently I'm running this program in parallel (4 processors), what takes 250 hours. The program is such that it fits in a AWS micro instance (I've tested it). Would it be possible to use 1000 micro instances in AWS to do the whole job in one hour? (at a cost of ~20$ - $0.02/instance)?
If it is possible, does anybody have some guidlines on how to do that?
If it is not, does anybody have an also low-budget alternative to that?
Thanks a lot!
In order to achieve this, you will need to:
Create a S3 bucket to store your bootstrapping scripts, application, input data and output data
Custom Lightweight AMI: you might want to create a custom lightweight AMI which knows about downloading the bootstrapping script
Bootstrapping Script: will download your software from your S3 bucket, parse a custom instance tag which will contain the integer [1..1000] and download any additional data.
Your application: does the processing stuff.
End of processing script is which uploads the result to a another result S3 bucket and terminates the instance, you might also want to send a SNS notification to communicate the end of processing status.
If you need result consolidation you might want to create another instance and use it as a coordinator, waiting for all "end of processing" notifications in order to finish the processing. In this case you might consider using in the future the Amazon's Hadoop map reduce engine, since it will do almost all this heavy lifting for free.
You didn't specify what language you'd like to do this in and since the app you use to deploy your program doesn't have to be in the same language I would recommend C#. There are a number of examples around the web on how to programmatically spawn new Amazon Instances using the SDK. For example:
How to start an Amazon EC2 instance programmatically in .NET
You can create your own AMIs with the program already present but that might end up being a pain if you want to make adjustments to it since it will require you to recreate the entire AMI. I'd recommend creating an extra instance or simply hosting the program in a location which is accessible from a public URL. Then I would create some kind of service which would be installed on the AMI ahead of time to allow me to specify the URL to download the app from along with whatever command line parameters I wanted for that particular instance.
Hope this helps.
Be carefull by default you can only spawn 20 instance by zone. You need to ask amazon in order to use more instances.
If you want low cost but don't care about delay you should use spot instances.