Create and delete EC2 instance with CloudFormation - amazon-web-services

I am quite new to AWS and want to know how to achieve following task with CloudFormation.
I want to spin up an EC2 instance with tomcat and deploy a java application on it. This java application will perform some operation. Once the operation is done, I want to delete all the resources created by this CloudFormation stack.
All these activities should be automatic. For example -- I will create the CloudFormation stack JSON file. At particular time of a day, a job should be kicked off (I don't know where in AWS to configure such job or how). But I know through Jenkins we can create a CloudFormation stack that will create all resources.
Then, after some time (lets say 2 hrs), another job should kick off and delete all resources created by CloudFormation.
Is this possible in AWS? If yes, any hints on how to do this?

Just to confirm, what you intend to do is have an EC2 instance get created on a schedule, and then have it shut down after 2 hours. The common way of accomplishing that is to use an Auto-Scaling Group (ASG) with a ScheduledAction to scale up and a ScheduledAction to scale down.
ASGs have a "desired capacity" (the number of instances in the ASG). You would want this to be "0" by default, change it to "1" at your desired time, and change it back to "0" two hours after that. What that will do is automatically start and subsequently terminate your EC2 instance on your schedule.
They also use a LaunchConfiguration, which is a template for your EC2 instances that will start on the schedule.
MyASG:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AvailabilityZones: !GetAZs !Ref "AWS::Region"
LaunchConfigurationName: !Ref MyLaunchConfiguration
MaxSize: 1
MinSize: 0
DesiredCapacity: 0
ScheduledActionUp:
Type: AWS::AutoScaling::ScheduledAction
Properties:
AutoScalingGroupName: !Ref MyASG
DesiredCapacity: 1
Recurrence: "0 7 * * *"
ScheduledActionDown:
Type: AWS::AutoScaling::ScheduledAction
Properties:
AutoScalingGroupName: !Ref MyASG
DesiredCapacity: 0
Recurrence: "0 9 * * *"
MyLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: ami-xxxxxxxxx # <-- Specify the AMI ID that you want
InstanceType: t2.micro # <-- Chaneg the instance size if you want
KeyName: my-key # <-- Change to the name of an EC2 SSH key that you've added
UserData:
Fn::Base64: !Sub |
#!/bin/bash
yum install -y aws-cfn-bootstrap
# ...
# ... run some commands to set up the instance, if you need to
# ...
Metadata:
AWS::CloudFormation::Init:
config:
files:
"/etc/something/something.conf":
mode: 000600
owner: root
group: root
content: !Sub |
#
# Add the content of a config file, if you need to
#
Depending on what you want your instances to interact with, you might also need to add a Security Group and/or an IAM Instance Profile along with an IAM Role.
If you're using Jenkins to deploy the program that will run, you would add a step to bake an AMI, build and push a docker image, or take whatever other action you need to deploy your application to the place that it will be used by your instance.
I note that in your question you say that you want to delete all of the resources created by CloudFormation. Usually, when you deploy a stack like this, the stack remains deployed. The ASG will remain there until you decide to remove the stack, but it won't cost anything when you're not running EC2 instances. I think I understand your intent here, so the advice that I'm giving aligns with that.

You can use Lambda to execute events on a regular schedule.
Write a Lambda function that calls CloudFormation to create your stack of resources. You might even consider including a termination Lambda function in your CloudFormation stack and configure it to run on a schedule (2 hours after the stack was created) to delete the stack that the termination Lambda function itself is part of (have not tried this, but believe that it will work). Or you could trigger stack deletion from cron on the EC2 instance running your Java app, of course).

If all you want is an EC2 instance, it's probably easier to simply create the EC2 instance rather than a CloudFormation stack.
Something (eg an AWS Lambda function triggered by Amazon CloudWatch Events) calls the EC2 API to create the instance
User Data is passed to the EC2 instance to install the desired software OR use a custom AMI with all software pre-installed
Have the instance terminate itself when it has finished processing -- this could be as simple as calling the Operating System to shutdown the machine, with the EC2 Shutdown Behavior set to Terminate.

Related

Why does ECS CloudFormation template create an EC2 Spot Fleet

I created a cluster in ECS with basic settings, nothing specific about the configuration except that I am using 1 On Demand t2.micro EC2 instance for the cluster.
I wanted to see what exactly was created and took a look at the CloudFormation template the cluster created.
I noticed in the template it has a configuration for EcsSpotFleet
EcsSpotFleet:
Condition: CreateWithSpot
Type: AWS::EC2::SpotFleet
Properties:
SpotFleetRequestConfigData:
AllocationStrategy: !Ref SpotAllocationStrategy
IamFleetRole: !Ref IamSpotFleetRoleArn
TargetCapacity: !Ref AsgMaxSize
SpotPrice: !If [ CreateWithSpotPrice, !Ref SpotPrice, !Ref 'AWS::NoValue' ]
TerminateInstancesWithExpiration: true
LaunchSpecifications:
....
I am wondering why is this created? Because I know the Cluster instances are created with ASG + LC. My only explanation is this fleet is used for running the CloudFormation stack. I cannot find an explanation to this in the documentation, not even sure if instances are needed for CloudFormation stack run.
p.s. I am very new to AWS, also have very little knowledge on CloudFormation.
Not all code in CloudFormation will be executed. It still depends on the "Condition" flag.
AWS usually create a template that covers most of the user cases and enables/disable parts of the template using the "Condition"
You can read more about Condition in AWS documentation here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html

Cloudformation Attach instance to Auto scaling group

In my CloudFormation file, I created an instance, a Launch Configuration and an Auto Scaling Group.
LogstashInstance:
Type: AWS::EC2::Instance
Properties:
IamInstanceProfile:
Ref: LogstashInstanceProfile
InstanceType: t2.micro
KeyName: chuongtest
ImageId: ami-0cd31be676780afa7
UserData:
SecurityGroupIds:
- Ref: LogstashSecurityGroup
SubnetId: subnet-0e5691582096fe1e6
Tags:
- Key: Name
Value: Logstash Instance
LogstashLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
EbsOptimized: false
IamInstanceProfile:
Ref: LogstashInstanceProfile
ImageId: ami-0cd31be676780afa7
InstanceMonitoring: true
InstanceType: t2.micro
KeyName: chuongtest
LaunchConfigurationName: LogstashLaunchConfiguration
SecurityGroups:
- Ref: LogstashSecurityGroup
UserData:
LogstashAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName: LogstashAutoScalingGroup
AvailabilityZones:
- ap-southeast-1b
DesiredCapacity: 1
LaunchConfigurationName:
Ref: LogstashLaunchConfiguration
MaxSize: 1
MinSize: 1
Tags:
- Key: Name
PropagateAtLaunch: "false"
Value: Logstash ASG
- Key: Instances
PropagateAtLaunch: "true"
Value: Logstash
My idea is to create an instance, attach it to the ASG and use the ASG to keep the work continue.
But this code will launch 2 instances.
The first instances has userdata different from the later instances so I can not delete the instance.
I have looked in the documentation but I can not find anything make sense. Is there a way to configure in the template, or the only way is scripting?
I have looked in the documentation but I can not find anything make sense. Is there a way to configure in the template, or the only way is scripting?
It does not make sense, because its not practical. ASG will be launching instances only from the associated AWS::AutoScaling::LaunchConfiguration. Thus, there is not much sense on "manually" attaching instances in your cause the the ASG. ASG will not re-launch it when it fails.
If the "manually" attached instances gets terminated for some reason, e.g. hardware failure, ASG will launch a replacement based on the launch configuration.
userdata different from the later instances
In this case the best would be to provide some condition in your UserData which would determine which version of your UserData to run.
Alternatively, just have two ASGs. One for the first instance, while the other group for the remaining instances.
Edit
Having discussed via the comments I would highly recommend the approach of using EFS as it is designed for the workload design you're doing.
In addition it is multi AZ vs the approach of rotating a single EBS volume as instances fail in an autoscaling group which would not persist should the AZ fail and potentially lead to loss of data.
Original
If that first instance is different from the rest you should be careful about adding it to the autoscaling group (as you want it to persist), especially in how it handles the replacement of instances.
Autoscaling groups are designed to scale similar instances that are generally immutable, so by having your host in this autoscaling group it will be treated as if it is one of the other hosts.
An Auto Scaling group contains a collection of Amazon EC2 instances that are treated as a logical grouping for the purposes of automatic scaling and management.
If you must add it to the autoscaling group here is the general approach:
Create a Custom Resource which involves a Lambda attaching the EC2 to the ASG using attach_instances
Add set_instance_protection to your instance to prevent it being replaced during a scale in event (or the instance failing)
Also ensure you set both the MinSize and DesiredCapacity to 0 to remove any extra instance that gets launched.
As I stated earlier you should not add this instance to the ASG if it can be avoided, tt will probably lead to confusion.

Is it possible to execute commands and then update security groups in a CloudFormation template?

I would like to perform the following operations in order with CloudFormation.
Start up an EC2 instance.
Give it privileges to access the full internet using security group A.
Download particular versions of Java and Python
Remove its internet privileges by removing security group A and adding a security group B.
I observe that there is a DependsOn attribute for specifying the order in which to create resources, but I was unable to find a feature that would allow me to update the security groups on the same EC2 instance twice over the course of creating a stack.
Is this possible with CloudFormation?
Not in CloudFormation natively, but you could launch the EC2 instance with a configured userdata script that itself downloads Java/Python and the awscli, as necessary, and then uses the awscli to switch security groups for the current EC2 instance.
However, if all you need is Java and Python pre-loaded then why not simply create an AMI with them already installed and launch from that AMI?
The best way out is to utilise a Cloudformation custom resource here. You can create a lambda function that does exactly what you need. This lambda function can then be called as a custom resource function in the cloud formation template.
You can pass your new security group ID and instance ID to the lambda function and code the lambda function to use AWS SDK and do the modifications that you need.
I have leveraged it to post an update to my web server about the progress of the cloud formation template. Below is the sample code of the template.
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles: [!Ref 'EC2Role']
MarkInstanceProfileComplete:
Type: 'Custom::EC2InstanceProfileDone'
Version: '1.0'
DependsOn: EC2InstanceProfile
Properties:
ServiceToken: !Ref CustomResourceArn
HostURL: !Ref Host
LoginType: !Ref LoginType
SecretId: !Ref SecretId
WorkspaceId: !Ref WorkspaceId
Event: 2
Total: 3
Here the resource MarkInstanceProfileComplete is a custom resource that calls a Lambda function. It takes the event count and total count as input and processes them to calculate percentage progress. Based on that it sends out a request to my web server. For all we care, this Lambda function can do potentially anything you want it to do.

Need Ansible to create new Ec2 instance Everytime

I Have an Ansible script That runs a Cloudformation script. The problem is The ansible script should run over and over a specified time to create an unlimited amount of servers on AWS just like running a Cloudformation createstack on AWS, but when ran more than once it keeps updating the same resource created. It just changes the Name. I have been tring to fix this for 2 days. I need a way to create a NEW server in aws no matter how many times I run the ansible script. I believe the issue is the instance id. Since it sees one is created it doesn't attempt to create a new one. Here is my Cloudformation code uploaded to s3.
Parameters:
KeyPair:
Type: AWS::EC2::KeyPair::KeyName
Description: Connects to this
Resources:
ec2:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-04681a1dbd79675a5
InstanceType: t2.micro
KeyName: !Ref KeyPair
And This is my Ansible code to run the ec2 server in s3 bucket. Ran like ansible-playbook provision.yml
please help.
- hosts: localhost
tasks:
- name: first Task Provision ec2
cloudformation:
stack_name: 'provisiong-ec2'
state: present
region: us-east-1
disable_rollback: true
template_url: https://s3.amazonaws.com/randombuckets/ansy2-cloudformation.template
template_parameters:
KeyPair: rabbit
It's not creating a new instance because the stack_name has not changed and your CFT only builds one host (which is already built).
Your immediate options are:
Create your instances using AutoScalingGroups (ASG) within CloudFormation. You can pass in the minimum number of hosts (MinSize) as a parameter and ASG will take care of the rest. You'll need to build in some logic to increment the count by one each iteration.
(not advised) Change the stack name every time you run the Ansible playbook
(not advised) Add another host to your CFT every time you want to run Ansible

Cloudformation doesn't change back a setting I manually modified

Trying to understand CFs behavior better.
I have a template that defines an ECS service:
ECSService:
Type: AWS::ECS::Service
Properties:
Cluster: mycluster
...
DesiredCount: 2
I go to the service the CF creates and set the DesiredCount to 0.
Then I deploy the CF template again but it doesn't change back the DesiredCount to 2.
Why doesn't it assert the full config?
The functionality you're looking for is called "Drift Detection".
This feature is not yet a part of CloudFormation, but it is currently in beta and is a planned release for 2018, according to Amazon.
It's generally a good practice to not modify resources managed by a Cloudformation stack. If you need to update a resource, perform a stack update.
Update (11/19): Good news! AWS has released this feature: https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/