is there any way to auto scale AWS RDS?. I know about read replicas. What we are experiencing is that our peaks on RDS are relatively very small. Is there any possibility that we can scale short term on peak and then rollback after it?
(What I was thinking is to use AWS cloudwatch to get RDS metric and if it is > 90% then launch an EC2 instance in auto scaling group with database application install in it and connect it with a our architecture and destroy it when peak goes down) is it possible? TIA
What db engines are you using? If thats MySQL or PgSQL you can use Aurora read replica auto scaling: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Integrating.AutoScaling.html
This is a managed service by AWS to auto scale out read replica when read traffic is high and scale in when traffic is low.
But notice that the write instance still cannot auto scale, this is basically a general limitation for relational db in general--> the write instance can only scale vertically with downtime.
However Aurora Serverless v2 : https://aws.amazon.com/rds/aurora/serverless/, which is in preview is going to solve the issue as you can scale in/out both write and read capacity within seconds.
What you're looking for is called vertical scaling and it's possible with minimal downtime in Multi-AZ RDS. This is different than horizontal scaling with read replicas, your master instance is replaced by a scaled copy in the other AZ. You can find a short write up with the console steps here. It should be fairly straightforward to create a cloudwatch event that does this automatically.
Related
Quick question for those who have used Amazon's MSK (Managed Streaming for Kafka):
The docs/FAQ make it clear that you can easily scale up the number of brokers in an MSK cluster. Can you just as easily scale the number back down? I'm not talking about auto-scaling, but manual scaling.
Currently Amazon MSK doesn't support the scaling down through the APIs. It is one of the known limitation as of now.
You will have to do it manually but that itself is an hassle as you will have to manage all the replicas to make sure those are in sync.
Another option could be you spin up another smaller MSK service and run the mirror maker to copy the data.
You aren't currently able to scale down a cluster https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-nodes-count.html#UpdateBrokerCount.
Just wanted to provide some potentially useful information; Confluent Cloud does allow for cluster shrink.
https://docs.confluent.io/cloud/current/clusters/shrink.html
Apologies if this isn't the right place to ask this question.
I'm looking to create an automated copy (backup) of my AWS RDS (MySQL) database daily and have this backup restored daily to another RDS instance and made available to another set of applications
I already have daily backups running and I can create a new rds instance from a backup but I want this to happen automatically within AWS.
Looking through AWS documentation and I can't find anything that fits this purpose but maybe there's a service that I'm not aware of.
AWS Aurora for MySQL and PostgreSQL support AutoScaling.
Autoscaling dynamically adjusts the number of Replicas available for cluster based on different metrics and policy. When sudden workload increase it'll add move read replicas and when it'll decrease it'll also remove so you don't have to pay for it.
Aurora AutoScaling
AWS RDS doesn't support autoscaling but you can always scale horizontally and vertically manually.
Scaling Your Amazon RDS Instance Vertically Horizontally
Our current server consisting of an 2x EC2 instances and RDS (Read/Write) database is in Mumbai Region. However I would like to copy everything (2x EC2 & RDS (R/W)) across to Sydney, and other to other regions.
Ideally I would like to replicate the contents in those instances as well.
Does anyone know a quick and easy way of doing this?
Edit 25/01/2019:
However I would like to copy everything including what ever is inside the instances (2x EC2s and the RDSs)
Edit 29/01/2019:
The purpose is to "scale/expand out". I want to have the same infrastructure replicated 1-to-1 (exactly/identically) across various regions.
It is simple!
- For EC2 - you need to create an AMI of those instances then right click on the AMI you've just created and choose "copy AMI" to the designated region.
For RDS
If you just wanna copy data to another region then take a snapshot then copy that snapshot to destination region
If you want to make the RDS replicate to another region continuously then you need to create a read-replica from your RDS instance.
Option for replicating environment depends on how much downtime can you tolerate.
If you are okay with downtime
1. Copy the AMI of EC2 instance and snapshot of RDS to another regions
2. Bring up your new environment.
This is perfect for non critial workload
If this is critical application
1. Copy the AMI of ec2 instance ( I am assuming this would be your web/app instnaces) For real time replication use rsync or robocopy .. or solution like cloudendure .
2. Create a new RDS instance in sydney
3. USE DMS migration tool .. create source and target relationship
4. once insync cut off the relation bring new environment in sydney
As suggested by previous answers for EC2 you can create AMIs and then move the AMI to a different region.
For RDS, you can either create read replicas (and read replicas of read replicas, but beware of latency), read replicas are used to mainly improve read performance of your app.
You can also create a Multi AZ backup which will act as a disaster recovery site. However, note that Multi-AZ is only used in case of a failover. Moreover, Multi-AZ involves Synchronous data copy and read replicas are asynchronous, so read replicas can demonstrate eventual consistency behavior.
But the real question here is - What are you trying to achieve?
Are you trying to "scale out" your infrastructure to support huge traffic to your application? Or are you simply trying to setup disaster recovery (DR)?
If your answer is DR, then the approach is pretty straight forward with Multi AZ and EC2 instance snapshots. But if the answer is scaling out and performance, you really need to be thinking of better strategies such as using Cloudfront (CDN) if it is a web app, using Elasticache in-memory cache for frequently read data, or RDS read replicas, using Elastic Load Balancers with Dynamic/Step scale-out/scale-in. Other, methods would be to evaluate the type of RDS storage subsystem used i.e. using Provisional IOPs vs. Using General Purpose SSD, checking if there are any NAT “instance” bottlenecks in your VPC and so on.
It may be tempting to spin up all these redundant copies of EC2 AMIs or RDS read replicas with a click of a button, but you really need to be thinking about the cost you are going to incur on a monthly basis for completely un-used resources.
I want to autoscale AWS RDS automatically with scripts based on the metric monitoring.
RDS doesn't really do this for Read-Write
Multi AZ Write-Read database copies are intended for failover from primary to secondary if there is an availability problem. They don't address the problem of performance
Read replicas can be used to increase performance but they are read only
It might be possible to look at a load metric and use a Cloudwatch alarm to start an extra read replica. Read replicas can be used via an ELB or NLB
But probably this isn't a good idea. While an existing RDS is making a read replica, performance is degraded. RDS read replicas are quite slow to come up and become available so it's unlikely to respond in a good way to transient demand
You can make an API call to Modify an RDS Instance, including changing the instance class.
Amazon RDS will provision a new instance of the desired class and will then re-point the Endpoint to the new instance. Existing connections will be terminated, but applications can reconnect and all the data will be there.
Rather than scaling the RDS instance, you could always consider a caching layer, such as Amazon ElastiCache that supports Redis and Memcached. Most applications are read-heavy, which is ideal for using a cache. This can significantly improve application performance without having to scale the database.
In simple, it can be possible with Aurora 5.7 DB RDS instances only, they provide an option to auto-scale based on cloud watch metric conditions i.e CPU utilization etc.
One of my RDS instance take high cpu some times due to heavy load. That time we manually need to upgrade instance. So I think to use auto scaling feature of RDS to automatically increase instances, but not getting any details about it.
Please some one confirm that If Autoscaling supports for RDS instances or not..
Thanks in advance.
The previous answers are now out of date. The newer AWS RDS Aurora does support autoscaling. Aurora Auto Scaling is available for both Aurora MySQL and Aurora PostgreSQL.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Integrating.AutoScaling.html
No, it does not.
The scaling you're describing is vertical scaling, which is upgrading the class or type of an instance. Autoscaling, in general, is for horizontal scaling: adding more instances. RDS supports neither horizontal nor vertical autoscaling, but it does support manual horizontal scaling (by adding read replicas) and manual vertical scaling (by upgrading/downgrading an existing instance).
AWS RDS supports Vertical scalling only.
Horizantal scalling can be applied only for read intensive RDS instances by creating read replicas under an ELB
https://aws.amazon.com/blogs/database/scaling-your-amazon-rds-instance-vertically-and-horizontally/
Another solution can be set a difference between allocated storage and max allocated storage. It give a storage buffer that you can use whenever a database faces storage crunch. Ofcourse you have to set a max value for this beforehand
Best solution is using a script in crontab using cli for modify db instance class in crontab. Is also possible in the script to retrieve the cpu usage and for example scaling up if cpu is higher than 90% and down when goes less than 60% but if RDS is not multi AZ this scaling generate downtime