How can I make user defined instances from AWS CloudFormation?
Example:
Question: How many instances do you want?
Answer: 4
AWS CloudFormation should install 4 same instances.
You don't really need custom-resource for that. You can just create your instances as part of AWS::AutoScaling::AutoScalingGroup. ASG will create required number of instances once you set DesiredCapacity and MaxSize. So if you want 4 instances, you set these values to 4.
You have two options
Autoscaling Group - DesiredCapacity and MaxSize - AWS::AutoScaling::AutoScalingGroup
Spot fleet ( cost savings by 90%) - TargetCapacity - AWS::EC2::SpotFleet
Related
It is possible to configure an autoscaling group to use Launch Template instead of Launch Configuration. I want to use it because when I configure an autoscaling group using Launch Configuration I can not set mixed instances policy (ondemand + spot instances).
I have created launch template (name 'test') with such configuration:
AMI Ubuntu 16,
some security groups and IAM Role with EC2, AutoScaling and CodeDeploy full access
and all other settings in default *Not specified* state.
Launch Template ID lt-0e1111
Default Version Number 1
Launch Template Name test
Owner arn:aws:iam::11111111:user/itsp
Version 1 (Default)
Name test
Created by arn:aws:iam::179961111910:user/itsp
Description Not specified
Date Created February 8, 2019 at 12:47:54 AM UTC+3
AMI ID ami-0bf1924ed30c48a6a
Availability Zone Not specified
Security Groups Not specified
Instance Type t2.nano
Key pair name itsp
Security Group Ids sg-07e746cf0e4935b84,sg-0a45f7e6ac321614d,sg-27bf6d4b
Advanced Details:
Purchasing option Not specified
Spot request type Not specified
Spot interruption behavior Not specified
IAM Instance Profile Not specified
Stop - Hibernate behavior Not specified
Monitoring Not specified
EBS-optimized instance Not specified
Tenancy Host ID Not specified
RAM Disk ID Not specified
User Data Not specified
Elastic Inference Accelerator Not specified
Maximum Spot price Not specified
Spot expiry date -
Spot block duration Not specified
Shutdown behavior Not specified
Termination Protection Not specified
Placement Group Name Not specified
Tenancy Not specified
Tenancy Affinity Not specified
Kernel ID Not specified
Elastic Graphics Not specified
Then I am creating an AutoScaling group with that launch template. Here is my autoscaling group configuration:
Launch Template test
Launch Template Version 1
Launch Template Description -
Instance Types -
Spot Diversity 0
Optional On-Demand Base 0
On-Demand Percentage 0%
Desired Capacity 1
Min 1
Max 1
Availability Zone(s) eu-central-1a
Subnet(s) subnet-53075e38
Classic Load Balancers -
Target Groups -
Health Check Type EC2
Health Check Grace Period 300
Instance Protection -
Termination Policies Default
Suspended Processes -
Placement Groups -
Default Cooldown 300
Enabled Metrics -
Creation Time Fri Feb 08 00:50:16 GMT+300 2019
Service-Linked Role arn:aws:iam::11111111111:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling
Such configuration appears to be set when I choose
*Fleet Composition - Adhere to the launch template
The launch template determines the instance type and purchase option (On-Demand or Spot).*
option and my lauch template configuration is described above.
So there are no spots in my autoscaling group.
Then I configure a CodeDeploy application 'Application' and DeploymentGroup 'DG_TEST' inside of it with such configuration:
Deployment type Blue/green
Service role ARN arn:aws:iam::11111111111111:role/CodeDeployRole
Deployment configuration CodeDeployDefault.AllAtOnce
Rollback enabled -
Environment configuration: Amazon EC2 Auto Scaling groups -> Name 'test'
Ok, after everything is set up I set desired instances count to 1 in my Autoscaling group and new on-demand instance spawns in my 'test' group.
I am now starting a blue-green deployment with AWS CLI command:
aws deploy create-deployment --application-name Application --s3-location bucket=codedeploybucket,key=application,bundleType=zip --deployment-group-name DG_TEST
And I can see new autoscaling group with same lauch template appears in my green fleet.
So we just checked all our services are working as expected.
Now I want to tweak my autoscaling group configuration a little to mix ondemand and spot instances in it:
Fleet Composition - Combine purchase options and instances
Choose a mix of On-Demand Instances and Spot Instances and multiple instance types. Spot Instances are automatically launched at the lowest price available.
Instance Types:
t2.nano (1vCPUs, 0.5GiB)
t2.small (1vCPUs, 2GiB)
t2.micro (1vCPUs, 1GiB)
Maximum Spot Price - Use default (recommended)
Spot Allocation Strategy - Diversify Spot Instances across your 2 lowest priced instance types per Availability Zone
Optional On-Demand Base - Designate the first 0 instances as On-Demand
On-Demand Percentage 50% On-Demand and 50% Spot
Desired Capacity 1
Min 1
Max 1
I save it and try same AWS CLI command
aws deploy create-deployment --application-name Application --s3-location bucket=codedeploybucket,key=application,bundleType=zip --deployment-group-name DG_TEST
And now I have my deployment failed with such comment:
The following validation error occurred: Valid requests must contain either LaunchTemplate, LaunchConfigurationName, InstanceId or MixedInstancesPolicy parameter. (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: 11111114-2b2b-11e9-93b7-11111111111)
Another strange thing is that if I configure my Launch Templae to use spots only:
Launch Template ID lt-00dc4c187d1111159
Default Version Number 1
Launch Template Name test2
Owner
arn:aws:iam::179961111910:user/itsp
Version 1 (Default)
Name test2
Created by arn:aws:iam::179961111910:user/itsp
Description Not specified
Date Created February 8, 2019 at 1:14:13 AM UTC+3
AMI ID ami-0bf1924ed30c48a6a
Availability Zone Not specified
Security Groups Not specified
Instance Type t2.micro
Key pair name itsp
Security Group Ids sg-0a45f7e6ac321614d,sg-07e746cf0e4935b84,sg-27bf6d4b
Advanced details
Purchasing option spot
Spot request type one-time
Spot interruption behavior Not specified
IAM Instance Profile CodeDeployRole (arn:aws:iam::1111111111:instance-profile/CodeDeployRole)
Stop - Hibernate behavior Not specified
Monitoring Not specified
EBS-optimized instance Not specified
Tenancy Host ID Not specified
RAM Disk ID Not specified
User Data Not specified
Elastic Inference Accelerator Not specified
Maximum Spot price Not specified
Spot expiry date -
Spot block duration Not specified
Shutdown behavior Not specified
Termination Protection Not specified
Placement Group Name Not specified
Tenancy Not specified
Tenancy Affinity Not specified
Kernel ID Not specified
Elastic Graphics Not specified
and use that option in my autoscaling group:
*Fleet Composition - Adhere to the launch template
The launch template determines the instance type and purchase option (On-Demand or Spot).*
Then after same deployment command I receive another error in CodeDeploy (failed deployment reason):
The IAM role arn:aws:iam::179966034910:role/CodeDeployRole does not give you permission to perform operations in the following AWS service: AmazonAutoScaling. Contact your AWS administrator if you need help. If you are an AWS administrator, you can grant permissions to your users or groups by creating IAM policies.
And that is definitely not the case because my role for CodeDeploy have full EC2, Autoscaling and CodeDeploy access.
Am I doing something wrong or that are bugs? How can I configure mixed autoscaling group (ondemand+spot instances) with codedeploy using launch templates? Thanks in advance!
I faced a similar issue when working with CodeDeploy and LaunchTemplates.
It looks like LaunchTemplates are still not supported by CodeDeploy.
We have to stick to LaunchConfigurations. They still don't support versioning, but for the most part, they are pretty much the same :/
I could't find where AWS has documented that little incompatibility issue when migrating LaunchConfigurations to LaunchTemplates... but could find here more info.
Sheers!
I am a little hazy about the need for both. I am using both in ECS CloudFormation. It appears that AWS::AutoScaling::AutoScalingGroup is for scaling EC2 instances whereas AWS::ApplicationAutoScaling::ScalableTarget is for scaling containers/tasks. Is my understanding correct?
That is pretty much correct.
AWS::AutoScaling::AutoScalingGroup is used for creating an Auto Scaling group of EC2 instances
AWS::ApplicationAutoScaling::ScalableTarget is used to specify an application resource that can scale - that is any of a range of resources including ECS service, EMR cluster or DynamoDB. A list of all the resources that can be scaled can be found in the ResourceId parameter description for RegisterScalableTarget.
I am quite new to AWS and want to know how to achieve following task with CloudFormation.
I want to spin up an EC2 instance with tomcat and deploy a java application on it. This java application will perform some operation. Once the operation is done, I want to delete all the resources created by this CloudFormation stack.
All these activities should be automatic. For example -- I will create the CloudFormation stack JSON file. At particular time of a day, a job should be kicked off (I don't know where in AWS to configure such job or how). But I know through Jenkins we can create a CloudFormation stack that will create all resources.
Then, after some time (lets say 2 hrs), another job should kick off and delete all resources created by CloudFormation.
Is this possible in AWS? If yes, any hints on how to do this?
Just to confirm, what you intend to do is have an EC2 instance get created on a schedule, and then have it shut down after 2 hours. The common way of accomplishing that is to use an Auto-Scaling Group (ASG) with a ScheduledAction to scale up and a ScheduledAction to scale down.
ASGs have a "desired capacity" (the number of instances in the ASG). You would want this to be "0" by default, change it to "1" at your desired time, and change it back to "0" two hours after that. What that will do is automatically start and subsequently terminate your EC2 instance on your schedule.
They also use a LaunchConfiguration, which is a template for your EC2 instances that will start on the schedule.
MyASG:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AvailabilityZones: !GetAZs !Ref "AWS::Region"
LaunchConfigurationName: !Ref MyLaunchConfiguration
MaxSize: 1
MinSize: 0
DesiredCapacity: 0
ScheduledActionUp:
Type: AWS::AutoScaling::ScheduledAction
Properties:
AutoScalingGroupName: !Ref MyASG
DesiredCapacity: 1
Recurrence: "0 7 * * *"
ScheduledActionDown:
Type: AWS::AutoScaling::ScheduledAction
Properties:
AutoScalingGroupName: !Ref MyASG
DesiredCapacity: 0
Recurrence: "0 9 * * *"
MyLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: ami-xxxxxxxxx # <-- Specify the AMI ID that you want
InstanceType: t2.micro # <-- Chaneg the instance size if you want
KeyName: my-key # <-- Change to the name of an EC2 SSH key that you've added
UserData:
Fn::Base64: !Sub |
#!/bin/bash
yum install -y aws-cfn-bootstrap
# ...
# ... run some commands to set up the instance, if you need to
# ...
Metadata:
AWS::CloudFormation::Init:
config:
files:
"/etc/something/something.conf":
mode: 000600
owner: root
group: root
content: !Sub |
#
# Add the content of a config file, if you need to
#
Depending on what you want your instances to interact with, you might also need to add a Security Group and/or an IAM Instance Profile along with an IAM Role.
If you're using Jenkins to deploy the program that will run, you would add a step to bake an AMI, build and push a docker image, or take whatever other action you need to deploy your application to the place that it will be used by your instance.
I note that in your question you say that you want to delete all of the resources created by CloudFormation. Usually, when you deploy a stack like this, the stack remains deployed. The ASG will remain there until you decide to remove the stack, but it won't cost anything when you're not running EC2 instances. I think I understand your intent here, so the advice that I'm giving aligns with that.
You can use Lambda to execute events on a regular schedule.
Write a Lambda function that calls CloudFormation to create your stack of resources. You might even consider including a termination Lambda function in your CloudFormation stack and configure it to run on a schedule (2 hours after the stack was created) to delete the stack that the termination Lambda function itself is part of (have not tried this, but believe that it will work). Or you could trigger stack deletion from cron on the EC2 instance running your Java app, of course).
If all you want is an EC2 instance, it's probably easier to simply create the EC2 instance rather than a CloudFormation stack.
Something (eg an AWS Lambda function triggered by Amazon CloudWatch Events) calls the EC2 API to create the instance
User Data is passed to the EC2 instance to install the desired software OR use a custom AMI with all software pre-installed
Have the instance terminate itself when it has finished processing -- this could be as simple as calling the Operating System to shutdown the machine, with the EC2 Shutdown Behavior set to Terminate.
I would like to start a new instance inside an Auto-Scaling group so the new instance will be 100% identical to other instances in this group.
In other words I want to do the same thing AWS does when scaling rules are triggered.
What API method I need to call to achieve this?
You can achieve this with the DescribeAutoScalingGroups API and the SetDesiredCapacity API.
Pseudocode:
Call DescribeAutoScalingGroups to get the current desired size of your group.
Call SetDesiredCapacity, setting DesiredCapacity = CurrentDesiredCapacity + 1
SetDesiredCapacity allows you to change the "desired capacity" of an AutoScaling group. If you increase the current desired capacity by 1, the AutoScaling Group will launch a new instance based on the AMI associated with your launch configuration. This is what scaling policies do within the bounds of your min/max capacity.
SetDesiredCapacity API Example: (from docs)
https://autoscaling.amazonaws.com/?Action=SetDesiredCapacity
&AutoScalingGroupName=my-asg
&HonorCooldown=false
&DesiredCapacity=2
&Version=2011-01-01
&AUTHPARAMS
set-desired-capacity CLI Example: (from docs)
aws autoscaling set-desired-capacity --auto-scaling-group-name my-auto-scaling-group --desired-capacity 2 --no-honor-cooldown
Reference:
AWS Documentation - Manual Scaling
API - SetDesiredCapacity
CLI - set-desired-capacity
I'm using Ansible to configure AWS Auto Scaling Groups (ASG). Looking at the ec2_asg_module options, there's none for enabling Monitoring in cloudWatch. However, that option can be enabled either form the AWS CLI or the AWS Console.
In the Console, it is labeled as "Group Metric Collection".
Keep in mind that I do not want to monitor the EC2 instances, but the Auto Scaling Group itself.
Thank you.
I submitted a PR last year to add 2 AWS modules : boto3 and boto3_wait.
These 2 modules allow you to interact with AWS API using boto3.
For instance, you could enable group metrics on the ASG by calling enable_metrics_collection method on AutoScaling service :
- name: Enable group metrics
boto3:
service: autoscaling
region: us-east-1
operation: enable_metrics_collection
parameters:
AutoScalingGroupName: my-auto-scaling-group
Granularity: 1Minute
Feel free to give the PR a thumbs-up if you like it! ;)