I have two AWS Elasticache instances, One of the instances (lets say instance A) has very important data sets and connections on it, downtime is unacceptable. Because of this situation, instead of doing normal migration (like preventing source data from new writes, getting dump on it, and restore it to the new one) I'm trying to sync instance A's data to another Elasticache instance (lets say instance B). As I said, this process should be downtime-free. In order to do that, I tried RedisShake, but because AWS restrict users to run certain commands (bgsave, config, replicaof,slaveof,sync etc), RedisShake is not working with AWS Elasticache. It's giving the error below.
2022/04/04 11:58:42 [PANIC] invalid psync response, continue, ERR unknown command `psync`, with args beginning with: `?`, `-1`,
[stack]:
2 github.com/alibaba/RedisShake/redis-shake/common/utils.go:252
github.com/alibaba/RedisShake/redis-shake/common.SendPSyncContinue
1 github.com/alibaba/RedisShake/redis-shake/dbSync/syncBegin.go:51
github.com/alibaba/RedisShake/redis-shake/dbSync.(*DbSyncer).sendPSyncCmd
0 github.com/alibaba/RedisShake/redis-shake/dbSync/dbSyncer.go:113
github.com/alibaba/RedisShake/redis-shake/dbSync.(*DbSyncer).Sync
... ...
I've tried rump for that matter, But it doesn't have enough stability to handle any important processes. First of all, it's not working as a background process, when the first sync finished, it's being closed with signal: exit done, so it will not be getting ongoing changes after the first finish.
Second of all, it's recognizing created/modified key/values in each run, for example, in first run key apple equals to pear, it's synced to the destination as is, but when I deleted the key apple and its value in source and ran the rump syncing script again, it's not being deleted in destination. So basically it's not literally syncing the source and the destination. Plus, last commit to the rump github repo is about 3 years ago. It seems a little bit outdated project to me.
After all this information and attempts, my question is, is there a way to sync two Elasticache for Redis instances, as I said, there is no room for downtime in my case. If you guys with this kind of experience have a bulletproof suggestion, I would be much appreciated. I tried but unfortunately didn't find any.
Thank you very much,
Best Regards.
If those two Elasticache Redis clusters exist in the same account but different regions, you can consider using AWS Elasticache global-datastore.
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Console.html
It has some restrictions on the regions, type of nodes and that both the clusters should have same configurations in terms of number of nodes, etc.
Limitations - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Getting-Started.html
Otherwise, there's a simple brute-force mechanism and you would be able to code it yourself I believe.
Create a client EC2 (let's call this Sync-er) pub-sub channel from your EC Redis instance A.
Whenever there is a new data, Sync-er would make WRITE commands on EC Redis instance B.
NOTE - You'll have to make sure that the clusters are in connectable VPCs.
Elasticache is only available to the resources within the VPC. If your Instance A and Instance B are in different VPCs, you'll have to peer them or connect them via TransitGateway.
Related
I have a rest API running on cloud run that implements a cache, which needs to be cleared maybe once a week when I update a certain property in the database. Is there any way to send a HTTP request to all running instances of my application? Right now my understanding is even if I send multiple requests and there are 5 instances, it could all go to one instance. So is there a way to do this?
Let's go back to basics:
Cloud Run instances start based on a revision/image.
If you have the above use case, where suppose you have 5 instances running and you suddenly need to re-start them as restarting the instances resolves your use case, such as clearing/rebuilding the cache, what you need to do is:
Trigger a change in the service/config, so a new revision gets
created.
This will automatically replace, so will stop and relaunch all your instances on the fly.
You have a couple of options here, choose which is suitable for you:
if you have your services defined as yaml files, the easiest is to run the replace service command:
gcloud beta run services replace myservice.yaml
otherwise add an Environmental variable like a date that you increase, and this will yield a new revision (as a change in Env means new config, new revision) read more.
gcloud run services update SERVICE --update-env-vars KEY1=VALUE1,KEY2=VALUE2
As these operations are executed, you will see a new revision created, and your active instances will be replaced on their next request with fresh new instances that will build the new cache.
You can't reach directly all the active instance, it's the magic (and the tradeoff) of serverless: you don't really know what is running!! If you implement cache on Cloud Run, you need a way to invalidate it.
Either based on duration; when expired, refresh it
Or by invalidation. But you can't on Cloud Run.
The other way to see this use case is that you have a cache shared between all your instance, and thus you need a shared cache, something like memory store. You can have only 1 Cloud Run instance which invalidate it and recreate it and all the other instances will use it.
I have multiple questions about this issue since AWS is still new to me.
What I am trying to accomplish: I want to create a test/sandbox db for my live db instance. I want to do this to create a sandbox environment for my developers, so they can add, edit, delete whatever they want on the test db without effecting the companies live db that's being used for our site.
Specifications: I want my test db to be continually updated alongside my live db. So I want a parent and child relationship or master/slave. So basically whatever the master has the slave will have but the slave cannot edit the master in any way whatsoever. Also I want the slave to have a different security group than the masters.
Things I have tried: first I thought I could copy a db snapshot of my live db and transfer it to a new db instance. But how would the new instance be continually updated. I couldn't even find what to do with the copied snapshot so that was a dead end.
Then I read that I could create a read replica of my master db, so I did that, I promoted it as per the documentation. Now can I read and write to this replica? Is this the correct path for what I want? Can I assign the replica a different security group from the master? And will the replica get updated data from master?
Is what I'm asking possible? Could someone please explain to me what the correct course of action is here or if I'm going about it incorrectly.
Again a bit lost as I'm new to AWS, any help would be greatly appreciated! :)
Documentation for reference:
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CopySnapshot.html
Not possible with RDS. Either you use replication (and can't write to replica) or make a clone that will not update.
You can, however, setup you own MySQL/MariaDB master-slave configuration and write to slave. Should work fine as long as you don't have conflicting values in indexes (usually because of autoincrementing primary key).
I have a pipeline setup wherein I have 3 main stages:
1) take input from a zipped file, unzip this file in s3. run some basic verification on each file to guarantee its integrity, move to step 2
2) kick off 2 simultaneous processing tasks on separate EC2 instances (parallelization of this step saves us a lot of time, so we need it for efficiency sake). Each EC2 instance will run data processing steps on some of the files in s3 which were unzipped in step 1, the files required are different for each instance.
3) after the 2 simultaneous processes are both done, spin up another EC2 instance to do the final data processing. Once this is done, run a cleanup job to remove the unzipped files from s3, leaving only the original zip file in its place.
So, one of the problems we're running into is that we have 4 EC2 instances that run this pipeline process, but there are some global parameters we would like for each EC2 instance to have access to. If we were running on a single instance, we could of course use shell variables to accomplish this task, but really need the separate instances for efficiency. Currently our best idea is to store a flat file in the s3 bucket which has access to these global variables and just read them on initialization and write back to them if they need to change. This is gross and there seems like there should be a better way, however we can't figure one out yet. I saw there's a way to set parameters which can be accessed at any part of the pipeline, but it looks like you can only set this on a per pipeline level, not on the granularity of each run of the pipeline. Does anyone have any resources that could help here? Much appreciated.
We were able to solve this by using DynamoDB to keep track of variables/state. The pipeline itself doesn't have any mechanism to do this, other than parameter values, which unfortunately only work per pipeline, not per job. You'll need to setup a DynamoDB instance and then use the pipeline job ID to keep track of state, connecting via the CLI tools or some SDK.
I've followed the "Getting Started Guide", the Two-Shards / Two-Replicas secnario and everything worked perfectly.
Then I started using the Collections API which is the preferable way of managing collection,shards and replication.
I launched two instances locally (afterwards with AWS, same problem)
I created a new collection with two shards using the following command:
/admin/collections?action=CREATE&name=collection1&numShards=2&collection.configName=collection
This successfully created two shards, one on each instance.
Then I launched another instance, expecting it to automatically set itself as a replica for the first shard, just like in the example. That didn't happen.
Is there something I'm missing?
There were two ways I was able to achieve this:
Manually, using the Collections API, I added a replica to shard1 and then another to shard2.
This is not good enough, as I will need to have this done automatically with Auto Scaling, so I'll need to micro-manage each server "role" - which replicas of which shards of which collections its handling, which complicates things a lot
The second way, which I couldn't find a documentation for is to launch an instance with a folder named "collectionX", inside a file named core.properties. In it the following line:
collection=collection1
Each instance I launched that way was automatically added as a replica in a round-robin way.
(Also working with several collections)
That's actually not a bad way at all, as I can pass parameter when I launch an AMI/instance in AWS.
Thanks everyone.
Amir
1) You are running the wrong command; the complete command is as follows:
curl 'http://localhost:8080/solr/admin/collections?action=CREATE&name=corename&numShards=2&replicationFactor=2&maxShardsPerNode=10'
Here I have given replication factor and due to which it will create the replica for your shards.
I have a simple pure C program, that takes an integer as input, runs for a while (let's say an hour) and then returns me a text file. I want to run this program 1000 times with input integers from 1 to 1000.
Currently I'm running this program in parallel (4 processors), what takes 250 hours. The program is such that it fits in a AWS micro instance (I've tested it). Would it be possible to use 1000 micro instances in AWS to do the whole job in one hour? (at a cost of ~20$ - $0.02/instance)?
If it is possible, does anybody have some guidlines on how to do that?
If it is not, does anybody have an also low-budget alternative to that?
Thanks a lot!
In order to achieve this, you will need to:
Create a S3 bucket to store your bootstrapping scripts, application, input data and output data
Custom Lightweight AMI: you might want to create a custom lightweight AMI which knows about downloading the bootstrapping script
Bootstrapping Script: will download your software from your S3 bucket, parse a custom instance tag which will contain the integer [1..1000] and download any additional data.
Your application: does the processing stuff.
End of processing script is which uploads the result to a another result S3 bucket and terminates the instance, you might also want to send a SNS notification to communicate the end of processing status.
If you need result consolidation you might want to create another instance and use it as a coordinator, waiting for all "end of processing" notifications in order to finish the processing. In this case you might consider using in the future the Amazon's Hadoop map reduce engine, since it will do almost all this heavy lifting for free.
You didn't specify what language you'd like to do this in and since the app you use to deploy your program doesn't have to be in the same language I would recommend C#. There are a number of examples around the web on how to programmatically spawn new Amazon Instances using the SDK. For example:
How to start an Amazon EC2 instance programmatically in .NET
You can create your own AMIs with the program already present but that might end up being a pain if you want to make adjustments to it since it will require you to recreate the entire AMI. I'd recommend creating an extra instance or simply hosting the program in a location which is accessible from a public URL. Then I would create some kind of service which would be installed on the AMI ahead of time to allow me to specify the URL to download the app from along with whatever command line parameters I wanted for that particular instance.
Hope this helps.
Be carefull by default you can only spawn 20 instance by zone. You need to ask amazon in order to use more instances.
If you want low cost but don't care about delay you should use spot instances.