Automating "Cloning" / "snapshot" of an already existing AWS EC2 instance.
I am able to create a AWS EC2 instance manually through Cloud Formation within the console. Alternatively , from Jenkins too I was able to perform the same operation.
Clone / Snapshot : Manually , through the options of "Snapshot" / "Create Image" I was able to spin up a new instance from the existing one. My question is can this be automated through Jenkins or script etc? The solution should be able to use either the snapshot or create image or any other options available and create a new instance from an existing one.
If the process can be automated , my request to please guide / provide steps / scripts / documents that can help me achieve the same.
Absolutely everything on AWS can be automated in multiple ways, including:
AWS Command-Line Interface (CLI)
SDKs and Programming Toolkits for AWS for multiple languages
Through IT management tools like Chef, Jenkins, Ansible, etc (which use SDKs to call AWS services on your behalf)
Please note that AWS CloudFormation is a service for deploying services, such as networking, compute and database in an automatic and reproducible manner. It is not typically used for operational activities like taking snapshots.
Related
I could not find a definite 'yes' or 'no' anywhere, so I thought maybe I ask here. Is it possible to run a custom script which would automatically create a new environment on AWS with all the settings like (Network, Capacity, Security etc.)? I need to create a lot of new environments as I am switching from individual load balancers to shared ones, and all the settings are the same (apart from the environment and application name), so it involves a lot of manual work.
From What is AWS CloudFormation? - AWS CloudFormation:
AWS CloudFormation is a service that helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; CloudFormation handles that.
If you want to create the CloudFormation template programmatically, you can use AWS CDK - AWS Cloud Development Kit (CDK):
The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.
The AWS CDK supports TypeScript, JavaScript, Python, Java, C#/.Net, and Go. Developers can use one of these supported programming languages to define reusable cloud components known as Constructs. You compose these together into Stacks and Apps.
Or, you can simply write your own script in a programming language that calls an AWS SDK to individually create resources in AWS. Everything in AWS can be done via API calls.
They seem to be serving the same purpose. They can both be broken down into steps, each step being a script.
A Command or Automation document can also both be part of SSM Associations in the State Manager.
So my question is simple. In which case would I need to create a Command document instead of an Automation document ?
From documentation:
Using Run Command, a capability of AWS Systems Manager, you can
remotely and securely manage the configuration of your managed nodes.
So with command documents you are executing commands on your managed instances (i.e. yum update)
Automation, a capability of AWS Systems Manager, simplifies common
maintenance, deployment, and remediation tasks for AWS services like
Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database
Service (Amazon RDS), Amazon Redshift, Amazon Simple Storage Service
(Amazon S3), and many more.
With Automation document you can interact with any AWS service to execute some actions (i.e. launch EC2 instance, crate AMI from running instance, crate RDS snapshot etc.)
Moreover you can define retries, crate process branches (i.e. when some step fails, go different path than when it success)
I am trying to find a solution for configuration management using AWS OpsWorks. What I can see is AWS offers three services for OpsWorks
Chef Automate
Puppet
AWS stacks
I have read basics of all three of them but unable to compare between three of them. I am unable to understand when to use which solution.
I want to implemnet a solution for my multiple EC2 instances, using which I can deliver updates to all my instances from a central repository(github). And, rollback changes if needed.
So following are my queries:
Which of the three solutions is best for this use case?
What should I use if my instances are in different regions?
I am unable to find anything useful on these topics so that I can make my decision. It would be great if I can get links to some useful articles as well.
Thanks in advance.
Terraform, Packer and Ansible are a great resource, I use them everyday to configure AMI's and build out all my infrastructure.
Terraform - Configuration Management for Infrastructure, it allows you to provision all the AWS, Azure, GCE components you needs to run your application.
Packer - Creates reusable images by pre installing software that is common to your applications.
Ansible - pre and post provisioning configuration management. You can use Ansible with Packer to provision software in an AMI, then if needed, use Ansible to configure it after provisioning. There is no need for a chef server or puppet master, you can run Ansible from your desktop if you have access to the cloud servers.
This examples provisions all the infrastructure for a Wordpress site, and uses Ansible to configure it post provisioning.
https://github.com/strongjz/tf-wordpress
All of this as well can automated in a Jenkins pipeline or with other Continous Deployment tools like CircleCI etc.
Ansible has no restriction on regions, neither does Terraform. Packer is a local build tool or on a CD server.
Examples:
https://www.terraform.io/intro/examples/aws.html
https://github.com/ansible/ansible-examples
https://www.packer.io/intro/getting-started/build-image.html
I might be naive but looking for a good solution to automatically spin up an AWS server with an API.
The use case is to create AWS EC2 instances on a click and maintain the deployments. Ansible is a probable candidate but looking for the core solution of spinning up a new EC2 machine.
Appreciate your help.
Rather than directly calling an API (eg from Java, .Net, Python, etc), you can also use the AWS Command-Line Interface (CLI).
The command you want is run-instances, which will launch a new Amazon EC2 instance.
See: AWS CLI documentation for run-instances
We have multiple amazon ec2 instances behind a load balancer. Our build script is written in phing and is integrated with git.
We are looking for a tool (like Jenkins or Amazon code deploy) which could display all the active instances currently behind load balancer and then allow us to select some of them (or select a group defined previously) and then trigger either of the following (whichever is better) -
a build script hosted on the same dedicated server where the tool is hosted.
or the respective build scripts hosted on the selected ec2 instances.
We should be able to do the following -
specify a git branch name, optionally, when we trigger the build script for any group of instances.
be able to roll out in batches of boxes, so as to get some time to monitor load, and then move to next batch if all is good. Best way, I guess, would be to specify a size of the batch (e.g. 10), so that the process waits for a user prompt after rollout on every batch completes.
So, if we have to rollout two different git branches to two groups of instances, we should be able to run them in two steps (if we do not specify batch size).
Would like to know about experiences of people who dealt with something similar.
For CodeDeploy, it supports Git (more precisely, GitHub). It also allows you to deploy only to tagged EC2 instances. If combined with custom DeploymentConfig (http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-create-deployment-configuration.html), you can also control how fast (the size of the batch) to deploy.
I would re-structure the question:
The choices you have for application deployment
and whether the tool has option to perform rolling deployments.
Jenkins is software for CI/CD, which will have to use plugins,custom scripting or leverage an existing orchestration software setup for doing the deployments.
For software orchestration, you have many choices, some of the more famous tools are Chef, puppet, ansible etc.. All of these would need you to manage some kind of centralized setup. All such software support application deployment.
You need to make a decision on whether you would want to invest in maintaining such a setup.
If you decide against such a setup, you have the option of using managed services such as AWS OpsWorks, AWS CodeDeploy, hosted chef etc.
In choosing any of these services, you delegate the management of orchestration software to a vendor, which will ensure the service is up all the time.
AWS code deploy and AWS OpsWorks are managed services on aws and work pretty well on AWS setups.
AWS OpsWorks uses chef under the hood.
AWS CodeDeploy only provides a subset of what OpsWorks provides and is responsible only for deployments. With AWS code deploy you get convenient visualization of your software deployments through AWS console.
With AWS code deploy, you can achieve the goal of partial roll out to ec2 instances.
You can do the same with other tools as well but CodeDeploy on AWS environment will take least amount of work.
CodeDeploy also allows you to deploy from GIT. Please refer to the following aws documentation
http://docs.aws.amazon.com/codedeploy/latest/userguide/github-integ-tutorial.html
The pitfall with code deploy is the fact that the agent that will run on instances has been tested for and is supported for only a limited number of OS combinations.(http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-run-agent.html#how-to-run-agent-supported-oses)
Also in future if you decide to move away from AWS, you will have to redo the deployment related work.
CodeDeploy service only charges you for the underneath AWS resources.
Please find the link to pricing documentation below:
https://aws.amazon.com/codedeploy/pricing/