aws cloudformation template for spot request in ec2 instance - amazon-web-services

I need a sample cloud formation template to add spot requests while provisioning the ec2 instance in AWS.I have tried with console to provision spot instances but I couldn't find any exact template for add spot request in ec2

You Can create a SpotFleet resource, here's an example
SpotFleet:
Type: AWS::EC2::SpotFleet
Properties:
SpotFleetRequestConfigData:
IamFleetRole: !GetAtt [IAMFleetRole, Arn]
SpotPrice: '1000'
TargetCapacity:
Ref: TargetCapacity
LaunchSpecifications:
- EbsOptimized: 'false'
InstanceType:
Ref: InstanceType
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
SubnetId:
Ref: Subnet1
WeightedCapacity: '8'
- EbsOptimized: 'true'
InstanceType:
Ref: InstanceType
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
Monitoring:
Enabled: 'true'
SecurityGroups:
- GroupId:
Fn::GetAtt:
- SG0
- GroupId
SubnetId:
Ref: Subnet0
IamInstanceProfile:
Arn:
Fn::GetAtt:
- RootInstanceProfile
- Arn
WeightedCapacity: '8'

You need to create Spot-fleet resource.
Example :
"SpotFleet": {
"Type": "AWS::EC2::SpotFleet",
"Properties": {
"SpotFleetRequestConfigData": {
"IamFleetRole": { "Fn::GetAtt": [ "IAMFleetRole", "Arn"] },
"SpotPrice": "1000",
"TargetCapacity": { "Ref": "TargetCapacity" },
"LaunchSpecifications": [
{
"EbsOptimized": "false",
"InstanceType": { "Ref": "InstanceType" },
"ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },
{ "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] }
]},
"SubnetId": { "Ref": "Subnet1" },
"WeightedCapacity": "8"
},
{
"EbsOptimized": "true",
"InstanceType": { "Ref": "InstanceType" },
"ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },
{ "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] }
]},
"Monitoring": { "Enabled": "true" },
"SecurityGroups": [ { "GroupId": { "Fn::GetAtt": [ "SG0", "GroupId" ] } } ],
"SubnetId": { "Ref": "Subnet0" },
"IamInstanceProfile": { "Arn": { "Fn::GetAtt": [ "RootInstanceProfile", "Arn" ] } },
"WeightedCapacity": "8"
}
]
}
}
}
More details can be found in this link :
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-spotfleet.html

Related

EKS Managed Nodegroup with Capacity Reservation in Launch Template through CloudFormation does not use Capacity Reservation

I am creating a Managed Nodegroup for EKS using CloudFormation.
I have an EC2 Launch Template with a CapacityReservationSpecification defined.
The Launch Template is linked to the Managed Nodegroup using CloudFormation. When the Managed Node Group is initialised the Launch Template is copied with an eks-*** prefix in the name. The CapacityReservationSpecification is not copied to the newly generated Launch Template. Cloud Formation script Example:
LaunchTemplate:
Resources:
LaunchTemplateAux:
Type: 'AWS::EC2::LaunchTemplate'
Properties:
LaunchTemplateData:
InstanceType: t3.medium
CapacityReservationSpecification:
CapacityReservationTarget:
CapacityReservationResourceGroupArn: {{reservation_group_arn}}
MetadataOptions:
HttpPutResponseHopLimit: 2
HttpTokens: optional
SecurityGroupIds:
- xxxxx
LaunchTemplateName: !Sub '${AWS::StackName}Aux'
NodeGroup:
ManagedNodeGroupAux:
Type: 'AWS::EKS::Nodegroup'
Properties:
AmiType: AL2_x86_64
ClusterName: test-cluster
Labels:
alpha.eksctl.io/cluster-name: test-cluster
alpha.eksctl.io/nodegroup-name: test-ng-aux
LaunchTemplate:
Id: !Ref LaunchTemplateAux
NodeRole: node-instance-role::NodeInstanceRole'
NodegroupName: test-nodegroup
ScalingConfig:
DesiredSize: 1
MaxSize: 2
MinSize: 1
Subnets:
- xxx
The resulting launch templates are as follows. Obtained using the following command aws ec2 describe-launch-template-versions --launch-template-id <template-id>
My Launch template Output:
{
"LaunchTemplateVersions": [
{
"LaunchTemplateId": "lt-xx",
"LaunchTemplateName": "test-cluster-ngAux",
"VersionNumber": 1,
"CreateTime": "2022-03-24T12:35:05+00:00",
"CreatedBy": "xxx:user/xxx",
"DefaultVersion": true,
"LaunchTemplateData": {
"InstanceType": "t3.medium",
"SecurityGroupIds": [
"sg-xxx"
],
"CapacityReservationSpecification": {
"CapacityReservationTarget": {
"CapacityReservationResourceGroupArn": "arn:aws:resource-groups:xxxxx:group/my-group"
}
},
"MetadataOptions": {
"HttpTokens": "optional",
"HttpPutResponseHopLimit": 2
}
}
}
]
}
Launch template copied by EKS API:
{
"LaunchTemplateVersions": [
{
"LaunchTemplateId": "lt-xxx",
"LaunchTemplateName": "eks-xxx",
"VersionNumber": 1,
"CreateTime": "2022-03-24T12:35:46+00:00",
"CreatedBy": "xxx:assumed-role/AWSServiceRoleForAmazonEKSNodegroup/EKS",
"DefaultVersion": true,
"LaunchTemplateData": {
"IamInstanceProfile": {
"Name": "xxx"
},
"ImageId": "ami-0c37e3f6cdf6a9007",
"InstanceType": "t3.medium",
"UserData": "xxx",
"TagSpecifications": [
{
"ResourceType": "volume",
"Tags": [
{
"Key": "eks:cluster-name",
"Value": "test-cluster"
},
{
"Key": "eks:nodegroup-name",
"Value": "test-cluster-ng-aux"
}
]
},
{
"ResourceType": "instance",
"SecurityGroupIds": [
"xxx"
],
"MetadataOptions": {
"HttpTokens": "optional",
"HttpPutResponseHopLimit": 2
}
}
}
]
}
This seems to be a bug in AWS. They have informed me that they will fix it.
https://repost.aws/questions/QUaid5sRdmRu2OFi7SQyxytg#ANF8uJ5RulQtmrVMaabAKZZg

AWS Cloudformation how to use conditional resource as DependsOn for other resource

I am trying to create EC2 instance with an optional value for IAMInstanceprofile.
When createiam is False I am expecting ec2stack to create without iam, when its True it should wait for iamstack and use its value.
"Parameters": {
"createiam" : {
"Type" : "String",
"Default" : "False"
}
},
"Conditions" : {
"Create_iam" : {"Fn::Equals" : [{"Ref" : "createiam"}, "True"]}
},
"Resources" : {
"iamstack" : {
"Type" : "AWS::CloudFormation::Stack",
"Condition": "Create_iam"
},
"ec2stack": {
"Type" : "AWS::CloudFormation::Stack",
"DependsOn" : "iamstack"
}
}
While running stack with option False for crateiam getting an error saying iamstack resource not found. Is there a way to add a condition for Dependson value?
I am able to achieve this with the WaitConditionHandle.
Found the references here.
Stack overflow question
Blog link with explination
Code flow now:
"Parameters": {
"createiam" : {
"Type" : "String",
"Default" : "False"
}
},
"Conditions" : {
"Create_iam" : {"Fn::Equals" : [{"Ref" : "createiam"}, "True"]}
},
"Resources" : {
"IAMWaithandle": {
"Condition": "Create_iam",
"DependsOn": "iamstack",
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"IAMWaitcondiftion": {
"Type": "AWS::CloudFormation::WaitCondition",
"Properties": {
"Handle": {
"Fn::If": [
"Create_iam",
{
"Ref": "IAMWaithandle"
},
{
"Ref": "WaitHandle"
}
]
},
"Timeout": "1",
"Count": 0
}
},
"iamstack" : {
"Type" : "AWS::CloudFormation::Stack",
"Condition": "Create_iam"
},
"ec2stack": {
"Type" : "AWS::CloudFormation::Stack",
"DependsOn" : "IAMWaitcondiftion"
}
}
Is there a way to add a condition for Dependson value? -> Yes, there is.
In this example (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html), It worked.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvType:
Description: Environment type.
Default: test
Type: String
AllowedValues:
- prod
- test
ConstraintDescription: must specify prod or test.
Conditions:
CreateProdResources: !Equals
- !Ref EnvType
- prod
Resources:
EC2Instance:
Type: 'AWS::EC2::Instance'
Properties:
ImageId: ami-05b891753d41ff88f #ami-0ff8a91507f77f867
MountPoint:
Type: 'AWS::EC2::VolumeAttachment'
Condition: CreateProdResources
Properties:
InstanceId: !Ref EC2Instance
VolumeId: !Ref NewVolume
Device: /dev/sdh
NewVolume:
Type: 'AWS::EC2::Volume'
Condition: CreateProdResources
Properties:
Size: 11
AvailabilityZone: !GetAtt
- EC2Instance
- AvailabilityZone
Bucket:
Type: 'AWS::S3::Bucket'
Condition: CreateProdResources
DependsOn: MountPoint
In your case, you need to share your complete code.

Invalid template resource property 'ElbDomainName'

For the below cloudformation template:
AWSTemplateFormatVersion: "2010-09-09"
Description: "Todobackend Stack"
# Stack Parameters
Parameters:
VpcId:
Type: "AWS::EC2::VPC::Id"
Description: "The target VPC Id"
SubnetId:
Type: "AWS::EC2::Subnet::Id"
Description: "The target Subnet Id in availability zone a"
KeyPair:
Type: "String"
Description: "The key pair that is allowed SSH access"
InstanceCount:
Type: "Number"
Description: "The desired number of application instances"
DbSubnets:
Type: "List<AWS::EC2::Subnet::Id>"
Description: "The target DB Subnet Group subnet Ids"
DbAvailabilityZone:
Type: "String"
Description: "The target availability zone for the database instance"
DbUsername:
Type: "String"
Description: "The RDS database username"
DbPassword:
Type: "String"
Description: "The RDS database password"
NoEcho: "true"
# Stack Resources
Resources:
# Configure auto scaing group
AutoScalingGroup:
Type: "AWS::AutoScaling::AutoScalingGroup"
Properties:
VPCZoneIdentifier: [ { "Ref": "SubnetId" } ]
LaunchConfigurationName: { "Ref": "AutoScalingLaunchConfiguration" }
MinSize: 0
MaxSize: 2
DesiredCapacity: { "Ref": "InstanceCount" }
Tags:
- Key: "Name"
Value: { "Fn::Join": ["", [ { "Ref": "AWS::StackName" }, " -instance" ] ] }
PropagateAtLaunch: "true"
AutoScalingLaunchConfiguration:
Type: "AWS::AutoScaling::LaunchConfiguration"
Properties:
ImageId: ami-05958d7635caa4d04
InstanceType: t2.micro
keyName: { "Ref": "KeyPair" }
IamInstanceProfile: { "Ref": "EC2InstanceProfile" }
SecurityGroups:
- { "Ref": "EC2InstanceSecurityGroup" }
UserData: {
"Fn::Base64": { "Fn::Join": ["", [
"#!/bin/bash\n",
"echo ECS_CLUSTER=", { "Ref": "EcsCluster"}, " >> /etc/ecs/ecs.config\n"
] ] }
}
EC2InstanceSecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: "todobackend-sg"
VpcId: { "Ref": "VpcId" }
SecurityGroupIngress:
- IpProtocol: "tcp"
FromPort: "8080"
ToPort: "8080"
SourceSecurityGroupId: { "Ref": "ElbSecurityGroup" }
- IpProtocol: "tcp"
FromPort: "22"
ToPort: "22"
CidrIp: "0.0.0.0/0"
Tags:
- Key: "Name"
Value: { "Fn::Join": ["", [ { "Ref": "AWS::StackName" }, "-instance-sg" ] ] }
EC2InstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: "/"
Roles: [ { "Ref": "EC2InstanceRole" } ]
EC2InstanceRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument: {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": [ "ec2.amazonaws.com" ] },
"Action": [ "sts:AssumeRole"]
}
]
}
Path: "/"
ManagePolicyArns:
- "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceForEc2Role"
# Configure RDS
DbInstance:
Type: "AWS::RDS::DBInstance"
Properties:
DBSubnetGroupName: { "Ref": "DbSubnetGroup" }
MultiAZ: "false"
AvailabilityZone: { "Ref": "DBAvailabilityZone" }
AllocatedStorage: 8
StorageType: "gp2"
DBInstanceClass: "db.t2.micro"
DBName: "todobackend"
Engine: "MySQL"
EngineVersion: "5.6"
MasterUserName: { "Ref": "DbUsername" }
MasterUserPassword: { "Ref": "DbPassword" }
VPCSecurityGroups:
- { "Ref": "DbSecurityGroup" }
Tags:
- Key: "Name"
Value: { "Fn::Join": ["", [ { "Ref": "AWS::Stackname" }, "-db" ] ] }
DbSecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: "Todobackend DB Security Group"
VpcId: { "Ref": "VpcId" }
SecurityGroupIngress:
- IpProtocol: "tcp"
FromPort: "3306"
ToPort: "3306"
SourceSecurityGroupId: { "Ref": "EC2InstanceSecurityGroup" }
DbSubnetGroup:
Type: "AWS::RDS::DBSubnetGroup"
Properties:
DBSubnetGroupDescription: "Todobackend DB Subnet Group"
SubnetIds: { "Ref": "DbSubnets" }
Tags:
- Key: "Name"
Value: { "Fn::Join": ["", [ { "Ref": "AWS::StackName" }, "-db-subnet-group" ] ] }
# Configure ELB
ElasticLoadBalancer:
Type: "AWS::ElasticLoadBalancing::LoadBalancer"
Properties:
CrossZone: "false"
SecurityGroups: [ { "Ref": "ElbSecurityGroup" } ]
Listeners:
- LoadBalancerPort: "80"
InstancePort: "8000"
Protocol: "http"
HealthCheck:
Target: "HTTP:8000/todos"
HealthyThreshold: "2"
UnhealthyThreshold: "10"
Interval: "30"
Timeout: "5"
Subnets: [ { "Ref": "SubnetId" } ]
Tags:
- Key: "Name"
Value: { "Fn::Join": ["", [ { "Ref": "AWS::StackName" }, "-elb" ] ] }
ElbSecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: "Todobackend ELB Security Group"
VpcId: { "Ref": "VpcId" }
SecurityGroupIngress:
- IpProtocol: "tcp"
FromPort: "80"
ToPort: "80"
CidrIp: "0.0.0.0/0"
Tags:
- Key: "Name"
Value: { "Fn::Join": ["", [ { "Ref": "AWS::StackName" }, "-elb-sg" ] ] }
# Configure ECS
EcsCluster:
Type: "AWS::ECS::EcsCluster"
TodobackendTaskDefinition:
Type: "AWS::ECS::TaskDefinition"
Properties:
ContainerDefinitions:
- Name: todobackend
Image: shamdockerhub/todobackend
Memory: 450
Environment:
- Name: DJANGO_SETTINGS_MODULE
Value: todobackend.settings.release
- Name: MYSQL_HOST
Value: { "Fn::GetAtt": ["DbInstance", "Endpoint.Address"] }
- Name: MYSQL_USER
Value: { "Ref": "DbUsername" }
- Name: MYSQL_PASSWORD
Value: { "Ref": "DbPassword" }
MountPoints:
- ContainerPath: /var/www/todobackend
SourceVolume: webroot
Command:
- uwsgi
- "--socket /var/www/todobackend/todobackend.sock"
- "--chmod-socket=666"
- "--module todobackend.wsgi"
- "--master"
- "--die-on-term"
- Name: nginx
Image: shamdockerhub/todobackend-nginx
Memory: 300
PortMappings:
- ContainerPort: "8000"
HostPort: "8000"
MountPoints:
- ContainerPath: /var/www/todobackend
SourceVolume: webroot
Volumes:
- Name: webroot
Host:
SourcePath: /ecs/webroot
TodobackendService:
Type: "AWS::ECS::Service"
Properties:
TaskDefinition: { "Ref": "TodobackendTaskDefinition" }
Cluster: { "Ref": "EcsCluster" }
LoadBalancers:
- ContainerName: "nginx"
ContainerPort: "8000"
LoadBalancerName: { "Ref": "ElasticLoadBalancer" }
Role: { "Ref": "EcsServiceRole" }
DesiredCount: 0
EcsServiceRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument: {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [ "ecs.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole"]
}
]
}
Path: "/"
ManagePolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
TodobackendAdhocTaskDefinition: # Application management task
Type: "AWS::ECS::TaskDefinition"
Properties:
ContainerDefinitions:
- Name: todobackend
Image: shamdockerhub/todobackend
Memory: 245
Environment:
- Name: DJANGO_SETTINGS_MODULE
Value: todobackend.settings.release
- Name: MYSQL_HOST
Value: { "Fn::GetAtt": ["DbInstance", "Endpoint.Address"] }
- Name: MYSQL_USER
Value: { "Ref": "DbUsername" }
- Name: MYSQL_PASSWORD
Value: { "Ref": "DbPassword" }
MountPoints:
- ContainerPath: /var/www/todobackend
SourcePath: webroot
Volumes:
- Name: webroot
Host:
SourcePath: /ecs/webroot
# Stack outputs
Outputs:
ElbDomainName:
Description: "Public DNS name of Elastic Load Balancer"
Value: { "Fn::GetAtt": [ "ElasticLoadBalancer", "DNSName" ] }
EcsCluster:
Description: "Amazon resource name (ARN) of Todobackend Ecs Cluster"
Value: { "Ref": "EcsCluster" }
TodobackendTaskDefinition:
Description: "Amazon resource name (ARN) of Todobackend Task definition"
Value: { "Ref": "TodobackendTaskDefinition"}
TodobackendAdhocTaskDefinition:
Description: "Amazon resource name(ARN) of Todobackend Adhoc Task Definition"
Value: { "Ref": "TodobackendAdhocTaskDefinition" }
TodobackendService:
Description: "Amazon resource name (ARN) of Todobackend service"
Value: { "Ref": "TodobackendService" }
below is the error from Outputs block:
Invalid template resource property 'ElbDomainName'
We need dns name of elastic load balancer to be part of Outputs block of stack
Why 'ElbDomainName' an invalid property?
The entire Outputs: block is indented a level too far. Outputs: should not be underneath Resources: indentation-wise
Outputs documentation

Why is AWS CloudFormation throwing "Encountered unsupported property InstanceGroups"?

When I deploy the below AWS CloudFormation script, I am getting the following error: "Encountered unsupported property InstanceGroups"
I have used InstanceGroups in the past without any issues. Here is an example of how others using it: https://noise.getoto.net/tag/amazon-emr/
I am using EMR 5.17.0, which I have used to set up before.
{
"Description": "Spark ETL EMR CloudFormation",
"Resources": {
"EMRCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Applications": [
{
"Name": "Hadoop"
},
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"AutoScalingRole": "EMR_AutoScaling_DefaultRole",
"BootstrapActions": [
{
"Path": "s3://somepath/scripts/install_pip36_dependencies.sh",
"Args": [
"relay==0.0.1"
],
"Name": "install_pip36_dependencies"
}
],
"Configurations": [
{
"Classification": "yarn-site",
"Properties": {
"yarn.scheduler.fair.preemption": "False",
"yarn.resourcemanager.am.max-attempts": "1"
},
"Configurations": []
},
{
"Classification": "core-site",
"Properties": {
"fs.s3.canned.acl": "BucketOwnerFullControl"
},
"Configurations": []
}
],
"EbsRootVolumeSize": 10,
"InstanceGroups": [
{
"Name": "Master",
"Market": "ON_DEMAND",
"InstanceRole": "MASTER",
"InstanceType": "m5.2xlarge",
"InstanceCount": 1,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "64"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Core",
"Market": "ON_DEMAND",
"InstanceGroupType": "CORE",
"InstanceType": "m5.2xlarge",
"InstanceCount": 5,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 100,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
},
{
"Name": "Task - 3",
"Market": "ON_DEMAND",
"InstanceGroupType": "TASK",
"InstanceType": "m5.2xlarge",
"InstanceCount": 2,
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"SizeInGB": 32,
"VolumeType": "gp2"
},
"VolumesPerInstance": 1
}
],
"EbsOptimized": "True"
}
}
],
"LogUri": "s3://somepath/emr-logs/",
"Name": "EMR CF",
"ReleaseLabel": "emr-5.17.0",
"ServiceRole": "EMR_DefaultRole",
"VisibleToAllUsers": "True"
}
}
}
}
When the CF script is loaded, it should create an AWS EMR cluster
Aws recommends that you set MasterInstanceGroup and CoreInstanceGroup under Instances
I give you an example of the Instances property of an EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper:
Instances:
Ec2KeyName: !Ref KeyName
Ec2SubnetId: !ImportValue MySubnetPrivateA
EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
AdditionalMasterSecurityGroups:
- !ImportValue EmrMasterAdditionalSgId
EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
AdditionalSlaveSecurityGroups:
- !ImportValue EmrSlaveAdditionalSgId
ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref MasterInstanceType
Market: ON_DEMAND
Name: Master
CoreInstanceGroup:
InstanceCount: !Ref NumberOfCoreInstances
InstanceType: !Ref CoreInstanceType
Market: ON_DEMAND
Name: Core
TerminationProtected: false
VisibleToAllUsers: true
JobFlowRole: !Ref EMRClusterinstanceProfile
ReleaseLabel: !Ref ReleaseLabel
LogUri: !Ref LogUri
Name: !Ref EMRClusterName
AutoScalingRole: EMR_AutoScaling_DefaultRole
ServiceRole: !Ref EMRClusterServiceRole
Tags:
-
Key: "cluster_name"
Value: "master.emr.my.com"
You can see the complete AWS template here.

Cloudformation unable to create resource policy for apigateway

The resource policy is working fine when i directly pass it to the console.
Below is resource policy example :-
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:us-west-2:339159142535:ooxmwl6q4e/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
""14.98.8.190/32""
]
}
}
}
]
}
Now how to create a cloudformation template for this to get created and get attached to the apigateway
I tried to create a policy but as per new policy "Principal" is depricated.
I created a role also but no help. Below is role snippet :-
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"Apifirewall": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "Apifirewall",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": [
"arn:aws:execute-api:us-west-2:339159142535:ooxmwl6q4e/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"14.98.8.190/32"
]
}
}
}
]
}
}
]
}
}
},
"Outputs": {
"Apifirewall": {
"Value": {
"Fn::GetAtt": [
"Apifirewall",
"Arn"
]
}
}
}
}
APIGateway resource policy is not binding to IAM Policy, it's different kind of resource.
So to implement it on your RestApi your should use the Policy parameter on AWS::ApiGateway::RestApi resource on
{
"Type" : "AWS::ApiGateway::RestApi",
"Properties" : {
"ApiKeySourceType" : String,
"BinaryMediaTypes" : [ String, ... ],
"Body" : JSON object,
"BodyS3Location" : S3Location,
"CloneFrom" : String,
"Description" : String,
"EndpointConfiguration" : EndpointConfiguration,
"FailOnWarnings" : Boolean,
"MinimumCompressionSize" : Integer,
"Name" : String,
"Parameters" : { String:String, ... },
"Policy" : JSON object
}
}
Below is the entire CFT for api deployment with lambda integration
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"AppEnv": {
"Type": "String",
"Description": "Application environment, for this deployment"
},
"DeployTag": {
"Type": "String",
"Description": "Distinct deployment tag ex: BLUE, GREEN"
}
},
"Resources": {
"LambdaExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/AWSLambdaFullAccess"
]
}
},
"RecommenderLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "recommender_field_validation_lambda.lambda_handler",
"FunctionName": "recommenderlambda2",
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
},
"Environment": {
"Variables": {
"S3_BUCKET": "belcorp.recommender.test",
"REGION_NAME": "us-west-2",
"TOPIC_ARN": {
"Fn::ImportValue": "RecommenderTopicARN"
},
"TABLE_NAME": {
"Fn::ImportValue": "recommederrequestinfo"
}
}
},
"Code": {
"S3Bucket": "belcorp.recommender.lambdas",
"S3Key": "recommender_field_validation_lambda.zip"
},
"Runtime": "python3.6",
"Timeout": 25
}
},
"LambdaPermission": {
"DependsOn": "RecommenderLambda",
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:invokeFunction",
"FunctionName": "recommenderlambda2",
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Join": [
"",
[
"arn:aws:execute-api:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":",
{
"Ref": "RecommenderApi"
},
"/*"
]
]
}
}
},
"RecommenderApi": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"EndpointConfiguration": {
"Types": [
"EDGE"
]
},
"Description": "RecommenderAPI",
"Name": {
"Fn::Sub": "RecommenderApi-${AppEnv}-${DeployTag}"
},
"Policy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:*/*"
},
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"14.98.8.190/32"
]
}
}
}
]
}
}
},
"ApiGatewayAccount": {
"Type": "AWS::ApiGateway::Account",
"Properties": {
"CloudWatchRoleArn": {
"Fn::ImportValue": "cloudwatchRole"
}
}
},
"ApiDeployment": {
"Type": "AWS::ApiGateway::Deployment",
"DependsOn": [
"OfferPostMethod",
"OrderPostMethod"
],
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"StageName": "dev"
}
},
"ProcessInput": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ParentId": {
"Fn::GetAtt": [
"RecommenderApi",
"RootResourceId"
]
},
"PathPart": "process-input"
}
},
"OfferLevel": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ParentId": {
"Ref": "ProcessInput"
},
"PathPart": "offer-level"
}
},
"OrderLevel": {
"Type": "AWS::ApiGateway::Resource",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ParentId": {
"Ref": "ProcessInput"
},
"PathPart": "order-level"
}
},
"OfferPostMethod": {
"DependsOn": "RecommenderLambda",
"Type": "AWS::ApiGateway::Method",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ResourceId": {
"Ref": "OfferLevel"
},
"HttpMethod": "POST",
"AuthorizationType": "NONE",
"Integration": {
"Type": "AWS_PROXY",
"IntegrationHttpMethod": "POST",
"Uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":lambda:path/2015-03-31/functions/",
{
"Fn::GetAtt": [
"RecommenderLambda",
"Arn"
]
},
"/invocations"
]
]
},
"IntegrationResponses": [
{
"StatusCode": 200,
"ResponseTemplates": {
"application/json": "$input.json('$.body')"
}
}
]
}
}
},
"OrderPostMethod": {
"DependsOn": "RecommenderLambda",
"Type": "AWS::ApiGateway::Method",
"Properties": {
"RestApiId": {
"Ref": "RecommenderApi"
},
"ResourceId": {
"Ref": "OrderLevel"
},
"HttpMethod": "POST",
"AuthorizationType": "NONE",
"Integration": {
"Type": "AWS_PROXY",
"IntegrationHttpMethod": "POST",
"Uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":lambda:path/2015-03-31/functions/",
{
"Fn::GetAtt": [
"RecommenderLambda",
"Arn"
]
},
"/invocations"
]
]
},
"IntegrationResponses": [
{
"StatusCode": 200,
"ResponseTemplates": {
"application/json": "$input.json('$.body')"
}
}
]
}
}
}
},
"Outputs": {
"RootUrl": {
"Description": "Root URL of the API gateway",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "RecommenderApi"
},
".execute-api.",
{
"Ref": "AWS::Region"
},
".amazonaws.com"
]
]
}
},
"OfferUrl": {
"Description": "Root URL of the API gateway",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "RecommenderApi"
},
".execute-api.",
{
"Ref": "AWS::Region"
},
".amazonaws.com",
"/dev/process-input/offer-level"
]
]
}
},
"OrderUrl": {
"Description": "Root URL of the API gateway",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Ref": "RecommenderApi"
},
".execute-api.",
{
"Ref": "AWS::Region"
},
".amazonaws.com",
"/dev/process-input/order-level"
]
]
}
}
}
}
Too long for a comment. This is the transformed YAML from this answer, which a commenter pointed out can be done in CloudFormation Designer:
AWSTemplateFormatVersion: 2010-09-09
Parameters:
AppEnv:
Type: String
Description: 'Application environment, for this deployment'
DeployTag:
Type: String
Description: 'Distinct deployment tag ex: BLUE, GREEN'
Resources:
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
RecommenderLambda:
Type: 'AWS::Lambda::Function'
Properties:
Handler: recommender_field_validation_lambda.lambda_handler
FunctionName: recommenderlambda2
Role: !GetAtt
- LambdaExecutionRole
- Arn
Environment:
Variables:
S3_BUCKET: belcorp.recommender.test
REGION_NAME: us-west-2
TOPIC_ARN: !ImportValue RecommenderTopicARN
TABLE_NAME: !ImportValue recommederrequestinfo
Code:
S3Bucket: belcorp.recommender.lambdas
S3Key: recommender_field_validation_lambda.zip
Runtime: python3.6
Timeout: 25
LambdaPermission:
DependsOn: RecommenderLambda
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:invokeFunction'
FunctionName: recommenderlambda2
Principal: apigateway.amazonaws.com
SourceArn: !Join
- ''
- - 'arn:aws:execute-api:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':'
- !Ref RecommenderApi
- /*
RecommenderApi:
Type: 'AWS::ApiGateway::RestApi'
Properties:
EndpointConfiguration:
Types:
- EDGE
Description: RecommenderAPI
Name: !Sub 'RecommenderApi-${AppEnv}-${DeployTag}'
Policy:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action: 'execute-api:Invoke'
Resource: !Sub 'arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:*/*'
Condition:
IpAddress:
'aws:SourceIp':
- 14.98.8.190/32
ApiGatewayAccount:
Type: 'AWS::ApiGateway::Account'
Properties:
CloudWatchRoleArn: !ImportValue cloudwatchRole
ApiDeployment:
Type: 'AWS::ApiGateway::Deployment'
DependsOn:
- OfferPostMethod
- OrderPostMethod
Properties:
RestApiId: !Ref RecommenderApi
StageName: dev
ProcessInput:
Type: 'AWS::ApiGateway::Resource'
Properties:
RestApiId: !Ref RecommenderApi
ParentId: !GetAtt
- RecommenderApi
- RootResourceId
PathPart: process-input
OfferLevel:
Type: 'AWS::ApiGateway::Resource'
Properties:
RestApiId: !Ref RecommenderApi
ParentId: !Ref ProcessInput
PathPart: offer-level
OrderLevel:
Type: 'AWS::ApiGateway::Resource'
Properties:
RestApiId: !Ref RecommenderApi
ParentId: !Ref ProcessInput
PathPart: order-level
OfferPostMethod:
DependsOn: RecommenderLambda
Type: 'AWS::ApiGateway::Method'
Properties:
RestApiId: !Ref RecommenderApi
ResourceId: !Ref OfferLevel
HttpMethod: POST
AuthorizationType: NONE
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt
- RecommenderLambda
- Arn
- /invocations
IntegrationResponses:
- StatusCode: 200
ResponseTemplates:
application/json: $input.json('$.body')
OrderPostMethod:
DependsOn: RecommenderLambda
Type: 'AWS::ApiGateway::Method'
Properties:
RestApiId: !Ref RecommenderApi
ResourceId: !Ref OrderLevel
HttpMethod: POST
AuthorizationType: NONE
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt
- RecommenderLambda
- Arn
- /invocations
IntegrationResponses:
- StatusCode: 200
ResponseTemplates:
application/json: $input.json('$.body')
Outputs:
RootUrl:
Description: Root URL of the API gateway
Value: !Join
- ''
- - 'https://'
- !Ref RecommenderApi
- .execute-api.
- !Ref 'AWS::Region'
- .amazonaws.com
OfferUrl:
Description: Root URL of the API gateway
Value: !Join
- ''
- - 'https://'
- !Ref RecommenderApi
- .execute-api.
- !Ref 'AWS::Region'
- .amazonaws.com
- /dev/process-input/offer-level
OrderUrl:
Description: Root URL of the API gateway
Value: !Join
- ''
- - 'https://'
- !Ref RecommenderApi
- .execute-api.
- !Ref 'AWS::Region'
- .amazonaws.com
- /dev/process-input/order-level
If you are using YAML for CloudFormation, the Policy can be in YAML. There is no need to use JSON for it. For example:
Parameters:
ApiAllowedIps:
Type: CommaDelimitedList
RestApi:
Type: AWS::ApiGateway::RestApi
Properties:
...
Policy:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: '*'
Principal: '*'
Resource: '*'
Condition:
IpAddress:
aws:SourceIp: !Ref ApiAllowedIps