What could be the problem of the reader instance not showing in Cloudwatch that it uses less capacity than the writer? (tried all the solutions from the AWS documentation)
Related
I have been using sagemaker recently and am using inference with GPU-based instances.
I am thinking of turning off sagemaker inference instances at night—for example, 8 pm to 8 am.
I want to do that using cdk. Not sure if it is a crazy idea or not?
Any help?
Amazon SageMaker supports different inference options that fits various use cases. You can use SageMaker Asynchronous endpoints to save cost during idle time (after operational hours), you don't have to use AWS CDK/ AWS CloudFormation while using this option.
Amazon SageMaker supports automatic scaling (autoscaling) your asynchronous endpoint. Autoscaling dynamically adjusts the number of instances provisioned for a model in response to changes in your workload. Unlike other hosted models Amazon SageMaker supports, with Asynchronous Inference you can also scale down your asynchronous endpoints instances to zero. Requests that are received when there are zero instances are queued for processing once the endpoint scales up.
Refer documentation, samples and blogs here.
is there any way to auto scale AWS RDS?. I know about read replicas. What we are experiencing is that our peaks on RDS are relatively very small. Is there any possibility that we can scale short term on peak and then rollback after it?
(What I was thinking is to use AWS cloudwatch to get RDS metric and if it is > 90% then launch an EC2 instance in auto scaling group with database application install in it and connect it with a our architecture and destroy it when peak goes down) is it possible? TIA
What db engines are you using? If thats MySQL or PgSQL you can use Aurora read replica auto scaling: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Integrating.AutoScaling.html
This is a managed service by AWS to auto scale out read replica when read traffic is high and scale in when traffic is low.
But notice that the write instance still cannot auto scale, this is basically a general limitation for relational db in general--> the write instance can only scale vertically with downtime.
However Aurora Serverless v2 : https://aws.amazon.com/rds/aurora/serverless/, which is in preview is going to solve the issue as you can scale in/out both write and read capacity within seconds.
What you're looking for is called vertical scaling and it's possible with minimal downtime in Multi-AZ RDS. This is different than horizontal scaling with read replicas, your master instance is replaced by a scaled copy in the other AZ. You can find a short write up with the console steps here. It should be fairly straightforward to create a cloudwatch event that does this automatically.
I am relatively experienced with many AWS services - but I do have a large gap around Aurora/RDS
I'm trying to create a multi-region multi-master (write replicas) setup
The purpose is to give low latency to users (if each read and write replica is in the user's region) and to give resilience (if there is a region outage, the users can have their requests routed to another region (the latency will be higher, but reduced service is better than no service))
I'm trying to learn about AWS Aurora and I've created a toy cluster to learn. It seems I can create a cluster that is served out of multiple regions (and Aurora replicates data between regions automatically). I've also read that it is possible to have a multi-master setup (in my toy cluster, it only had one write partition, I couldn't work out how to create another write partition in another region, which made me question if it's possible?)
Here is a diagram of what I'm thinking:
https://imgur.com/DzoSpHL
Thank you in advance!
The purpose is to give low latency to users (if each read and write replica is in the user's region)
I couldn't work out how to create another write partition in another region, which made me question if it's possible?
That is not possible (at least not currently) because of multi-master Aurora limitations.
all DB instances in a multi-master cluster must be in the same AWS Region.
and others such as
you can have a maximum of two DB instances in a multi-master cluster
You can't enable cross-Region replicas from multi-master clusters.
You can read more here
The best thing you can do in your scenario is to create single master and place read replicas into those additional regions (possibly with some caching in necessary).
As mentioned earlier it is not possible with Aurora.
However DynamoDB supports multi-active multi-region:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html
As others have said, with Amazon Aurora, you cannot deploy multi-Region and multi-master. However you can deploy multi-Region using Aurora Global Database. Then one writer endpoint would be in one Region, while reader endpoints would be available in all the other Regions. Then you can also use write forwarding (assuming you are using the MySQL flavor of Aurora) in the read-only Regions. I know latency is a concern for you, so note the write actually goes back to the primary Region, so writes will incur that extra latency.
Quick question for those who have used Amazon's MSK (Managed Streaming for Kafka):
The docs/FAQ make it clear that you can easily scale up the number of brokers in an MSK cluster. Can you just as easily scale the number back down? I'm not talking about auto-scaling, but manual scaling.
Currently Amazon MSK doesn't support the scaling down through the APIs. It is one of the known limitation as of now.
You will have to do it manually but that itself is an hassle as you will have to manage all the replicas to make sure those are in sync.
Another option could be you spin up another smaller MSK service and run the mirror maker to copy the data.
You aren't currently able to scale down a cluster https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-nodes-count.html#UpdateBrokerCount.
Just wanted to provide some potentially useful information; Confluent Cloud does allow for cluster shrink.
https://docs.confluent.io/cloud/current/clusters/shrink.html
One of my RDS instance take high cpu some times due to heavy load. That time we manually need to upgrade instance. So I think to use auto scaling feature of RDS to automatically increase instances, but not getting any details about it.
Please some one confirm that If Autoscaling supports for RDS instances or not..
Thanks in advance.
The previous answers are now out of date. The newer AWS RDS Aurora does support autoscaling. Aurora Auto Scaling is available for both Aurora MySQL and Aurora PostgreSQL.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Integrating.AutoScaling.html
No, it does not.
The scaling you're describing is vertical scaling, which is upgrading the class or type of an instance. Autoscaling, in general, is for horizontal scaling: adding more instances. RDS supports neither horizontal nor vertical autoscaling, but it does support manual horizontal scaling (by adding read replicas) and manual vertical scaling (by upgrading/downgrading an existing instance).
AWS RDS supports Vertical scalling only.
Horizantal scalling can be applied only for read intensive RDS instances by creating read replicas under an ELB
https://aws.amazon.com/blogs/database/scaling-your-amazon-rds-instance-vertically-and-horizontally/
Another solution can be set a difference between allocated storage and max allocated storage. It give a storage buffer that you can use whenever a database faces storage crunch. Ofcourse you have to set a max value for this beforehand
Best solution is using a script in crontab using cli for modify db instance class in crontab. Is also possible in the script to retrieve the cpu usage and for example scaling up if cpu is higher than 90% and down when goes less than 60% but if RDS is not multi AZ this scaling generate downtime