I have the following setup in google cloud.
Http load balancer will balance between 2 instance groups in different regions
Each instance group have minimum of 2 instances and maximum of 5 instances
I want to schedule a snapshot for each running instances at any moment. Since i don't know upfront how many instances will be running at any moment, what is the better way to create snapshot of every running instances. Is that possible?
My main question is about what are you trying to save. If you are running MIGs (Managed Instance Groups) all of your VMs are clones of the Instance Template. Maybe another approach is better here. If you are trying to save logs, send your logs to Cloud Monitoring ; if you are trying to preserve objects, send your files to a Storage Bucket or use Cloud Filestore. So, what are you trying to preserve? Maybe Snapshots are no the best solution for your case
Related
I have 1 perfect AWS EC2 instance in working condition which is running NODE js
which have its elastic IP.
Now I need to auto-scale if that instance in use over 80%.
I do know few ways to do that but am not able to figure out, when new instance launches how will my node code will be on that instance.
Later when I want to update my node code then on which instance should I connect with?
If anyone can help me with this then it would be awesome.
Thanks :)
If you want to use the EC2 AutoScaling, you must know that the instances are in a stateless mode ( can be started/Terminated) by the autoscaling group regarding the metrics and indications of elasticity ( cpu, number of request...) memory is
a custom metric
So to use the AutoScaling group, you must generate from your current working machine an AMI ( Amazon Machine Image) that contains the code of your node.js and create a launch configuration/template, for the next machines EC2 can use it to bootstrap.
IF you want to update your code, that means you must update your Launch configuration ( updated AMI, or use the user data bootstrap to get the update node.js from S3 or other location) or use maybe CodeDeploy wich is the perfect candidate for this king of subject.
I've an ECS cluster with running one task for my backend instance. I would like to be able to stop/start the EC2 instance whenever I want. Is it possible?? I was trying to stop instance directly but it terminates after few second when stopped and after that new instance is created automatically. I tried to change the Auto Scale Group to match desired=min=0 capacity but when I do that the instance gets auto terminated. I just want to turn off the Ec2 instance when its not needed to be used but at the same time I want data to persist betweet turning on and off. I am fighting with this for a few days now and wasn't able to achieve my goals.
Also how to link EBS volume with VOLUME /root/.local/share/XYZ from docker file image to persist the data from the XYZ folder
I would suggest you to do modifications in autoscaling group, when you want to turn off instance put 0 in auto scaling and when you want to turn on change value in autoscaling,
... you can do that with aws cli , and you can shcdule the period also by putting aws cli command in cron job
I would suggest using EFS. Here is an article from AWS on how to persist data from ECS containers using EFS.
Using Amazon EFS to Persist Data from Amazon ECS Containers
Start/Stop instances and auto-scale don't really fit together.
Auto-scale is specifically designed to solve scalein/scaleout.
One way to address this could be using customized termination policy (but I never tried this in ECS setup).
One note though, if your customized termination policy never terminates the instances and you continue adding instances to keep always, you might get good amount EC2 bill.
Last day I wanted, according to AWS recommendations, put my ec2 instance inside of an autoscaling group. I created my ec2 instance by using the standard linux AMI instance and then I installed a full LAMP server.
The next morning I tried accessing my apache and guess what? My LAMP wasn't there anymore! Everything was wiped away.
I guess this is because, for some reason, the autoscaling group deleted my instance and recreated it vanilla.
Now I still want to autoscale my instance but, of course, I want to keep my LAMP and the stored data.
So here's my questions:
How to create a customized image starting from my actual instance?
Would it be correct to create the mysql DB using AWS RDS so to not keep it linked to my instance?Is it more or less expensive than dedicating a EBS storage?
I also want to keep my /var/www/html data somewhere shared between instances: while it is true that, on production, I won't update those files often it is also true that I don't want to lose them just because the autoscaling resets my instance state. I also don't want to re-create an image each time I update said files... What's the best way?Maybe an s3 bucket? Or, still, an EBS storage shared between instances?
I would assume that the reason that your "LAMP [server] wasn't there anymore" was because the web server failed health checks and was terminated and replaced by AutoScaling.
Elastic Beanstalk would be a good way to manage some of the complexity here. If that's not an option then you should read up on AutoScaling, ALB, and health checks.
In response to your specific questions:
you can create an Amazon Machine Image (AMI) from an instance. When you, or AutoScaling, launch a new instance from that AMI, you can get the instance up to date by running a script in userdata
move the DB from the web/app server to RDS, or to a DB server that you manage yourself
maintain the html/js/css etc. in S3 and sync them to your web server periodically (there are other options, but that's simple)
I want to create a web app for my organization where users can schedule in advance at what times they'd like their EC2 instances to start and stop (like creating events in a calendar), and those instances will be automatically started or stopped at those times. I've come across four different options:
AWS Datapipeline
Cron running on EC2 instance
Scheduled scaling of Auto Scaling Group
AWS Lambda scheduled events
It seems to me that I'll need a database to store the user's scheduled times for autostarting and autostopping an instance, and that I'll have to pull that data from the database regularly (to make sure that's the latest updated schedule). Which would be the best of the four above options for my use case?
Edit: Auto Scaling only seems to be for launching and terminating instances, so I can rule that out.
Simple!
Ask users to add a tag to their instance(s) indicating when they should start and stop (figure out some format so they can easily specify Mon-Fri or Every Day)
Create an AWS Lambda function that scans instances for their tags and starts/stops them based upon the tag content
Create an Amazon CloudWatch Event rule that triggers the Lambda function every 15 minutes (or whatever resolution you want)
You can probably find some sample code if you search for AWS Stopinator.
Take a look at ParkMyCloud if you're looking for an external SaaS app that can help your users easily schedule (or override that schedule) your EC2, RDS, and ASG instances. It also connects to SSO, provides an API, and shows you all of your resources across regions/accounts/clouds. There's a free trial available if you want to test it out.
Disclosure: I work for ParkMyCloud.
Is it possible to do AutoScaling with Static IPs in AWS ? The newly created instances should either have a pre-defined IP or pick from a pool of pre-defined IPs.
We are trying to setup ZooKeeper in production, with 5 zooKeeper instances. Each one should have a static-IP which are to hard-coded in the Kafka's AMI/Databag that we use. It should also support AutoScaling, so that if one of the zooKeeper node goes down, a new one is spawned with the same IP or from a pool of IPs. For this we have decided to go with 1 zoo-keeper instance per AutoScaling group, but the problem is with the IP.
If this is the wrong way, please suggest the right way. Thanks in advance !
One method would be to maintain a user data script on each instance, and have each instance assign itself an elastic IPs from a set of EIPs assigned for this purpose. This user data script would be referenced in the ASGs Launch Configuration, and would run on launch.
Say the user script is called "/scripts/assignEIP.sh", using the AWS CLI you would have it consult the pool to see which ones are available and which ones are not (already in use). Then it would assign itself one of the available EIPS.
For ease of IP management, you could keep the pool of IPs in a simple text properties file on S3, and have the instance download and consult that list when the instance starts.
Keep in mind that each instance will need an to be assigned IAM instance profile that will allow each instance to consult and assign EIPs to itself.