Let's say I've created an AMI from one of my EC2 instances. Now, I can add this manually to then LB or let the AutoScaling group to do it for me (based on the conditions I've provided). Up to this point everything is fine.
Now, let's say my developers have added a new functionality and I pull the new code on the existing instances. Note that the AMI is not updated at this point and still has the old code. My question is about how I should handle this situation so that when the autoscaling group creates a new instance from my AMI it'll be with the latest code.
Two ways come into my mind, please let me know if you have any other solutions:
a) keep AMIs updated all the time; meaning that whenever there's a pull-request, the old AMI should be removed (deleted) and replaced with the new one.
b) have a start-up script (cloud-init) on AMIs that will pull the latest code from repository on initial launch. (by storing the repository credentials on the instance and pulling the code directly from git)
Which one of these methods are better? and if both are not good, then what's the best practice to achieve this goal?
Given that anything (almost) can be automated using the AWS using the API; it would again fall down to the specific use case at hand.
At the outset, people would recommend having a base AMI with necessary packages installed and configured and have init script which would download the the source code is always the latest. The very important factor which needs to be counted here is the time taken to checkout or pull the code and configure the instance and make it ready to put to work. If that time period is very big - then it would be a bad idea to use that strategy for auto-scaling. As the warm up time combined with auto-scaling & cloud watch's statistics would result in a different reality [may be / may be not - but the probability is not zero]. This is when you might consider baking a new AMI frequently. This would enable you to minimize the time taken for the instance to prepare themselves for the war against the traffic.
I would recommend measuring and seeing which every is convenient and cost effective. It costs real money to pull down the down the instance and relaunch using the AMI; however thats the tradeoff you need to make.
While, I have answered little open ended; coz. the question is also little.
People have started using Chef, Ansible, Puppet which performs configuration management. These tools add a different level of automation altogether; you want to explore that option as well. A similar approach is using the Docker or other containers.
a) keep AMIs updated all the time; meaning that whenever there's a
pull-request, the old AMI should be removed (deleted) and replaced
with the new one.
You shouldn't store your source code in the AMI. That introduces a maintenance nightmare and issues with autoscaling as you have identified.
b) have a start-up script (cloud-init) on AMIs that will pull the
latest code from repository on initial launch. (by storing the
repository credentials on the instance and pulling the code directly
from git)
Which one of these methods are better? and if both are not good, then
what's the best practice to achieve this goal?
Your second item, downloading the source on server startup, is the correct way to go about this.
Other options would be the use of Amazon CodeDeploy or some other deployment service to deploy updates. A deployment service could also be used to deploy updates to existing instances while allowing new instances to download the latest code automatically at startup.
Related
I am trying to figure it out which one is the best practice when creating a new server on AWS EC2.
To do that I choose Tableau Server. First time, as Tableau docs recommend I did the install myself, but I would like to keep as automatic as possible everything, the idea behind that is if ec2 get destroyed how can I recover everything fast?
I am using terraform to store as a code all the AWS infrastructure, but the installation itself is not automatic yet.
To do that, I have two options, ansible (never worked before) or in this particular case Tableau has an automated install script in python, which I could add in the EC2 template launch configuration,and then using terraform I can raise it in minutes.
Which one should be the choosen why? Both seems to acomplish the final goal.
Also it raises some kind of doubs such as:
It retrieves the server up and with a full instalation of the software, but to get all users, and all the Tableau setup I have to raise anyways an snapshot, right? Is there any other tool to do that?
Then, if the manual install of the software is fast enough, why then I should use IaC to keep the install as code, instead of document the script of installation? And just keep the Infrastructure as code?
I need to move more than 50 compute instances from a Google Cloud project to another one, and I was wondering if there's some tool that can take care of this.
Ideally, the needed steps could be the following (I'm omitting regions and zones for the sake of simplicity):
Get all instances in source project
For each instance get machine sizing and the list of attached disks
For each disk create a disk-image
Create a new instance, of type machine sizing, in target project using the first disk-image as source
Attach remaining disk-images to new instance (in the same order they were created)
I've been checking on both Terraform and Ansible, but I have the feeling that none of them supports creating disk images, meaning that I could only use them for the last 2 steps.
I'd like to avoid writing a shell script because it doesn't seem a robust option, but I can't find tools that can help me doing the whole process either.
Just as a side note, I'm doing this because I need to change the subnet for all my machines, and it seems like you can't do it on already created machines but you need to clone them to change the network.
There is no tool by GCP to migrate the instances from one project to another one.
I was able to find, however, an Ansible module to create Images.
In Ansible:
You can specify the “source_disk” when creating a “gcp_compute_image” as mentioned here
Frederic
I have an EBS backed EC2 web server. I am trying to get to the point where I can automate launching an identical server (identical in terms of software and config).
From my understanding there are two ways to achieve this:
Install all required software, make config changes and then create a custom AMI from the EBS volume. Use this AMI to launch any future instances.
Use the original AMI and use a Bootstrap Script install all the required software and make necessary config changes.
To me, creating a custom AMI and not having to write a script seems a lot easier initially. But Im aware that any modifications to the server will mean creating a new AMI. I can also see how a bootstrap script is much more transparent and allows one to easily inspect what is going to be installed/configured.
I wondered if anyone could point out anymore advantages/disadvantages between the two approaches. Is there one which is considered best practice?
If you create custom AMI:
It'll be faster to provision new servers.
It'll be easy to maintain the unification of the environment.
It'll be difficult to update the AMI in-case of any modifications.
If you use bootstrapping script:
It'll be easier to modify in case of any changes.
It'll be hard to maintain the unification of the environment (e.g. can face versioning issues).
Slow to provision new servers.
BTW, how about using AMI for things that are less likely to change and then using bootstrap scripts e.g. user data to do things that'll change over time.
Logicmonitor collector installers must be fetched with a valid token and the installer expires after a period of time. So there's no simple way to pull a collector installer binary onto a new ec2 instance and then run it.
Instead, it is necessary to use a script which uses Logicmonitor's REST API to generate a new collector installer URL, then fetch that and run it immediately. I'm guessing that, since there is a logicmonitor provider in terraform, at least one other person has gone through this process and already has a working script. Frankly, Logicmonitor's docs provide the bulk of it, so it isn't that hard to generate for myself, but if someone out there already has a nice template or module which adds the necessary pieces to an instance's userdata, you'll save me a couple of hours of copypasta and trial and error work. Something that uses cloud-init would be particularly useful, but I can convert. Basic example:
runcmd:
- export COLLECTOR_URL = `get_collector_url.sh ${api_key} ${other_var} ${yet_another}`
- curl -o LogicmonitorCollector.bin $COLLECTOR_URL
- chmod +x LogicmonitorCollector.bin
- ./LogicmonitorCollector.bin
I can pull the script for get_collector_url.sh out of github or an S3 bucket in an earlier statement easily enough.
This is the kind of thing I'd love to eventually build into a resource in the logicmonitor provider. I am new to terraform and don't know what is involved in adding a new resource, but this seems like a common need for anyone planning to bring up all of their infrastructure via terraform, since I don't want to have to manually install collectors on the instances terraform will be launching in my mgmt VPC for handling things like bastion duty and monitoring collectors.
Note - Logicmonitor claims the installer binary itself will expire after 2 hours. I've been taking them at their word. If what they actually mean it that the token in the installer URL will expire after 2 hours, I could just download the installer once and stick it in an AMI so that I need never download it again - just change the config to update collector id. But I'm guessing there are reasons their docs don't provide instructions for just pulling the binary once.
We have a load balanced setup in AWS with two instances. We do pretty frequent code updates, utilizing SVN. I need to know how easy it is to update the code changes across all the instances in our cluster. Can we simply do 'snapshots' and create new volumes each time for the instances?...or?...
I would not do updates via EBS snapshots. Think of EBS volumes as a hard disk - you would not change your harddisk if you have an update for your software.
As you have your code in a version control system, code updates should be quite simple like logging in to your (multiple) servers and doing a git pull or svn update. This should fetch the latest code files from your servers. Depending on the type of application you would have to do some other tasks afterwards, running build scripts, emptying cache etc.
The problem is that this kind of setup does not scale well. If you have n servers, you will have to login and do this command n times. Therefore it makes sense to look into some remote management tools that you can use in one step. With a lot of these tools, you also get a complete configuration management stack: you define a set of recipes or tasks (like installed packages, configuration files, fetch the latest code, necessary build steps) for each of your servers, and when you boot up a new server it fetches the lastest version of its configuration and installs itself.
Popular configuration management tools include Puppet or Salt. Both tools have remote execution included which should make your task to publish your code base easier, you would only have to fire one command on your master server and it automatically executes this task on all its minions / slave servers.