CloudFormation conditional EMR instances - amazon-web-services

I struggle a bit with the use of conditionals in CF templates, and I would like to conditionally specify EMR cluster instances groups or fleets in the most concise way.
This builds w/o error. It chooses to use instance groups if prod, or instance fleets if non-prod using two separate conditionals:
Parameters:
EnvironmentName:
Type: String
Description: 'Example: ci, qa, stage, prod'
Conditions:
IsPreProd: !Or
[!Equals [!Ref EnvironmentName, ci], !Equals [!Ref EnvironmentName, qa]]
IsProd: !Or
[!Equals [!Ref EnvironmentName, stage], !Equals [!Ref EnvironmentName, prod]]
Resources:
EMRCluster:
Type: 'AWS::EMR::Cluster'
Properties:
Instances:
CoreInstanceGroup:
!If
- IsProd
- InstanceCount: 1
InstanceType: m5.8xlarge
Market: ON_DEMAND
Name: CoreInstance
- !Ref "AWS::NoValue"
CoreInstanceFleet:
!If
- IsPreProd
- InstanceTypeConfigs:
- InstanceType: m5.8xlarge
TargetOnDemandCapacity: 1
TargetSpotCapacity: 1
LaunchSpecifications:
SpotSpecification:
TimeoutAction: SWITCH_TO_ON_DEMAND
TimeoutDurationMinutes: 10
- !Ref "AWS::NoValue"
I would like to use just a single conditional, like below, except the build fails telling me 'YAML not well-formed' on the line where the 'If' is. If I implement it like above, I would end up having four separate conditionals since I also have to add master instance groups or fleets as well. Is is possible to do it like this all as one conditional?
Parameters:
EnvironmentName:
Type: String
Description: 'Example: ci, qa, stage, prod'
Conditions:
IsProd: !Or
[!Equals [!Ref EnvironmentName, stage], !Equals [!Ref EnvironmentName, prod]]
Resources:
EMRCluster:
Type: 'AWS::EMR::Cluster'
Properties:
Instances:
- !If
- IsProd
- CoreInstanceGroup:
InstanceCount: 1
InstanceType: m5.8xlarge
Market: ON_DEMAND
Name: CoreInstance
- CoreInstanceFleet:
InstanceTypeConfigs:
- InstanceType: m5.8xlarge
TargetOnDemandCapacity: 1
TargetSpotCapacity: 1
LaunchSpecifications:
SpotSpecification:
BlockDurationMinutes: 60
TimeoutAction: SWITCH_TO_ON_DEMAND
TimeoutDurationMinutes: 10

Instances is not a list. You don't need - before !If:
Resources:
EMRCluster:
Type: 'AWS::EMR::Cluster'
Properties:
Instances:
!If
- IsProd
- CoreInstanceGroup:
InstanceCount: 1
InstanceType: m5.8xlarge
Market: ON_DEMAND
Name: CoreInstance
- CoreInstanceFleet:
InstanceTypeConfigs:
- InstanceType: m5.8xlarge
TargetOnDemandCapacity: 1
TargetSpotCapacity: 1
LaunchSpecifications:
SpotSpecification:
BlockDurationMinutes: 60
TimeoutAction: SWITCH_TO_ON_DEMAND
TimeoutDurationMinutes: 10

Related

Conditional statement in cloud-formation

Background
I have the following cloud formation template that i am trying to use to spin up a EKS CLUSTER. I am having issues with the logging settings. I want to make them conditional so in the future a user can set a specific logging like say api to true or false and based on that it will be enabled or disabled.
Parameters:
ClusterName:
Type: String
ClusterVersion:
Type: Number
AllowedValues: [1.21, 1.20, 1.19, 1.18]
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
ApiLogging:
Type: String
AllowedValues: [true, false]
AuditLogging:
Type: String
AllowedValues: [true, false]
AuthenticatorLogging:
Type: String
AllowedValues: [true, false]
ControllerManagerLogging:
Type: String
AllowedValues: [true, false]
SchedulerLogging:
Type: String
AllowedValues: [true, false]
Conditions:
ApiLoggingEnabled: !Equals [!Ref ApiLogging, 'true']
AuditLoggingEnabled: !Equals [!Ref AuditLogging, 'true']
AuthenticatorLoggingEnabled: !Equals [!Ref AuthenticatorLogging, 'true']
ControllerManagerLoggingEnabled: !Equals [!Ref ControllerManagerLogging, 'true']
SchedulerLoggingEnabled: !Equals [!Ref SchedulerLogging, 'true']
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Sub ${ClusterName}
Version: !Sub ${ClusterVersion}
RoleArn: !Sub ${RoleArnValue}
ResourcesVpcConfig:
SecurityGroupIds: !Ref ListOfSecurityGroupIDs
SubnetIds: !Ref ListOfSubnetIDs
Logging:
ClusterLogging:
EnabledTypes:
- Type: !If [ApiLoggingEnabled, api, 'AWS::NoValue']
- Type: !If [AuditLoggingEnabled, audit, 'AWS::NoValue']
- Type: !If [AuthenticatorLoggingEnabled, authenticator, 'AWS::NoValue']
- Type: !If [ControllerManagerLoggingEnabled, controllerManager, 'AWS:NoValue']
- Type: !If [SchedulerLoggingEnabled, scheduler, 'AWS:NoValue']
Outputs:
ClusterArn:
Description: Arn of EKS CLUSTER
Value: !Ref EKSCluster
However i get the following Error My template works fine when i get rid of the logging stuff but i want to fix that. I am not sure what i did wrong.
Properties validation failed for resource EKSCluster with message: #/Logging/ClusterLogging/EnabledTypes/2/Type: #: only 1 subschema matches out of 2 #/Logging/ClusterLogging/EnabledTypes/2/Type: failed validation constraint for keyword [enum] #/Logging/ClusterLogging/EnabledTypes/3/Type: #: only 1 subschema matches out of 2 #/Logging/ClusterLogging/EnabledTypes/3/Type: failed validation constraint for keyword [enum] #/Logging/ClusterLogging/EnabledTypes/4/Type: #: only 1 subschema matches out of 2 #/Logging/ClusterLogging/EnabledTypes/4/Type: failed validation constraint for keyword [enum]
It should be !Ref 'AWS::NoValue':
Logging:
ClusterLogging:
EnabledTypes:
- Type: !If [ApiLoggingEnabled, api, !Ref 'AWS::NoValue']
- Type: !If [AuditLoggingEnabled, audit, !Ref 'AWS::NoValue']
- Type: !If [AuthenticatorLoggingEnabled, authenticator, !Ref 'AWS::NoValue']
- Type: !If [ControllerManagerLoggingEnabled, controllerManager, !Ref 'AWS:NoValue']
- Type: !If [SchedulerLoggingEnabled, scheduler, !Ref 'AWS:NoValue']

CloudFormation Joining List of Strings with FindInMap

I am facing issue with Cloudformation on joining a list of strings after getting it from a map.
Error caused is Security Group expects a list of string.
"123456":
us-east-1:
#VPC1
VpcId: vpc-xxxx
VpcCidr: 10.78.160.0/20
SecurityGroups: sg-123
AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
"123457":
us-east-1:
#VPC
VpcId: vpc-xxxx
VpcCidr: 10.78.160.0/20
SecurityGroups: sg-123,sg-124
AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
LaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Metadata:
AWS::CloudFormation::Init:
config:
Properties:
InstanceType:
Ref: InstanceType
SecurityGroups: !Join [ ",", [!FindInMap [!Ref "AWS::AccountId", !Ref "AWS::Region", SecurityGroups], !Ref EC2SG ]]
IamInstanceProfile:
please let me know how to get the list for securitygroups and adding it to another resource
Solved by using this question
How do I use nested lists or append to a list in Cloudformation?
Lets take mappings:
Mappings:
"111222333444":
us-east-1:
SecurityGroups:
- sg-abc
- sg-xyz
us-west-2:
SecurityGroups:
- sg-1234
- sg-3456
Security groups can be referred as
SecurityGroupIds:
"Fn::FindInMap":
[!Ref AWS::AccountId, !Ref AWS::Region, SecurityGroups]
Define as array in mappings and pass the full array, this way we can avoid splitting and joining.
Full Template , tested on a Lambda
AWSTemplateFormatVersion: "2010-09-09"
Description: "Test"
Mappings:
"111222333444":
us-east-1:
SecurityGroups:
- sg-abc
- sg-xyz
us-west-2:
SecurityGroups:
- sg-1234
- sg-3456
Resources:
TestLambda:
Type: "AWS::Lambda::Function"
Properties:
Handler: com.test.MainClass::handleRequest
Runtime: java8
FunctionName: "Test-Lambda"
Code:
S3Bucket: code-bucket
S3Key: !Sub "artifacts/test.jar"
Description: "Test Lambda"
MemorySize: 512
Timeout: 60
Role: !Ref my-role-arn
VpcConfig:
SecurityGroupIds:
"Fn::FindInMap":
[!Ref AWS::AccountId, !Ref AWS::Region, SecurityGroups]
SubnetIds:
- subnet-1
- subnet-2

CloudFormation: conditional AutoScalingGroup notifications

I want to receive AutoScaling Event notifications using SNS, but only in my PROD environment. How can I configure my CloudFormation template to do so?
Should it be like this:
Parameters:
Environment:
Description: Environment of the application
Type: String
Default: dev
AllowedValues:
- dev
- prod
Conditions:
IsDev: !Equals [ !Ref Environment, dev]
IsProd: !Equals [ !Ref Environment, prod]
Resources:
mySNSTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- Endpoint: "my#email.com"
Protocol: "email"
myProdAutoScalingGroupWithNotifications:
Type: AWS::AutoScaling::AutoScalingGroup
Condition: IsProd
Properties:
NotificationConfigurations:
- NotificationTypes:
- "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
- "autoscaling:EC2_INSTANCE_TERMINATE"
- "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
TopicARN: !Ref "mySNSTopic"
myDevAutoScalingGroupWithoutNotifications:
Type: AWS::AutoScaling::AutoScalingGroup
Condition: IsDev
Properties:
Or does CloudFormation support the following too:
Parameters:
Environment:
Description: Environment of the application
Type: String
Default: dev
AllowedValues:
- dev
- prod
Conditions:
IsProd: !Equals [ !Ref Environment, prod]
Resources:
mySNSTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- Endpoint: "my#email.com"
Protocol: "email"
myAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
NotificationConfigurations:
- Condition: IsProd
NotificationTypes:
- "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
- "autoscaling:EC2_INSTANCE_TERMINATE"
- "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
TopicARN: !Ref "mySNSTopic"
It should be double using Fn::If function:
NotificationConfigurations:
- !If
- IsProd
- NotificationTypes:
- "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
- "autoscaling:EC2_INSTANCE_TERMINATE"
- "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
TopicARN: !Ref "mySNSTopic"
- !Ref "AWS::NoValue"
Can also try the following form:
NotificationConfigurations:
!If
- IsProd
- - NotificationTypes:
- "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
- "autoscaling:EC2_INSTANCE_TERMINATE"
- "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
TopicARN: !Ref "mySNSTopic"
- !Ref "AWS::NoValue"
Please be careful about indentation. You may need to adjust it to match your template.

aws cloudformation elasticsearch stack

I have written the below template to pick an environment based upon the user input. But I am getting error as "An error occurred (ValidationError) when calling the CreateStack operation: Template format error: [/Resources/Type] resource definition is malformed" .Please guide me what need to be changed and whether syntax is in right format.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvironmentValue:
AllowedValues:
- PROD
- TEST
Description: 'Please select an Environment'
Type: String
Mappings:
Environment:
PROD:
VPC: vpc-xxxxxxxx
Subnets: 'subnet-xxxxx,subnet-xxxxx,subnet-xxxx'
Securitygroups: 'sg-xxxx,sg-xxxx'
TEST:
VPC: vpc-xxxxx
Subnets: 'subnet-xxxx,subnet-xxxxx'
Securitygroups: 'sg-xxxx,sg-xxxxx'
#Conditions:
# CreatePRODStack: !Equals [!Ref EnvironmentValue, PROD]
# CreateTESTStack: !Equals [!Ref EnvironmentValue, TEST]
Resources:
Type: 'AWS::Es:Domain'
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions: !FindInMap [Environment, !Ref 'EnvironmentValue', VPC]
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
Type: 'AWS::IAM::Policy'
Properties:
PolicyDocument: YAML
PolicyName: prodtest
When the user gives input as Prod, the stack for Prod should be created in Cloudformation
I'm seeing a few issues here:
1 - You haven't named your resources.
2 - Your indentinging looks incorrect, which is important for yaml
3 - I believe your Type for the Elasticsearch domain is incorrect. You have
Type: 'AWS::Es:Domain'
but I think it should be
Type: AWS::Elasticsearch::Domain
Using your Domain as an example, I think it should be more along the lines of:
ElasticsearchDomain:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions: !FindInMap [Environment, !Ref 'EnvironmentValue', VPC]
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
There may be other issues I'm missing here, but there are definitely syntax errors in here
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticsearch-domain.html
The type tag is expected to be 'AWS::Elasticsearch::Domain' and there are multiple formatting errors as per the yaml declarations. The Properties should be at the same level as Type. Then VPCOptions should have the two properties following it. With the given example the template should look like the following
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EnvironmentValue:
AllowedValues:
- PROD
- TEST
Description: 'Please select an Environment'
Type: String
Mappings:
Environment:
PROD:
VPC: vpc-xxxxxxxx
Subnets: 'subnet-xxxxx,subnet-xxxxx,subnet-xxxx'
Securitygroups: 'sg-xxxx,sg-xxxx'
TEST:
VPC: vpc-xxxxx
Subnets: 'subnet-xxxx,subnet-xxxxx'
Securitygroups: 'sg-xxxx,sg-xxxxx'
#Conditions:
# CreatePRODStack: !Equals [!Ref EnvironmentValue, PROD]
# CreateTESTStack: !Equals [!Ref EnvironmentValue, TEST]
Resources:
ElasticSearchCluster:
Type: 'AWS::Es:Domain'
Properties:
DomainName: EPD34
ElasticsearchVersion: 6.5
ElasticsearchClusterConfig:
DedicatedMasterEnabled: 'true'
InstanceCount: '2'
ZoneAwarenessEnabled: 'true'
InstanceType: r4.xlarge.elasticsearch
DedicatedMasterType: r4.xlarge.elasticsearch
DedicatedMasterCount: '2'
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 100
VolumeType: gp2
VPCOptions:
SubnetIds: !FindInMap [Environment, !Ref 'EnvironmentValue', Subnets]
Securitygroups: !FindInMap [Environment, !Ref 'EnvironmentValue', Securitygroups]
SnapshotOptions:
AutomatedSnapshotStartHour: '0'
IAMPolicyEntry:
Type: 'AWS::IAM::Policy'
Properties:
PolicyDocument: YAML
PolicyName: prodtest

CodeDeploy does not deploy to new ASG group because of CloudFormation resources creation order

I am trying to develop an entire AWS architecture by usin CloudFormation only, however I am having some issues with the integration of CodeDeploy with CloudFormation and AutoScaling Group.
The problem is that, since I need to associate the CodeDeploy DeploymentGroup to an AutoScaling Group in order for the auto-deployment to work, CloudFormation recognizes the group as being required before creating the deployment group.
What happens is that the ASG gets created, instances start to spin up BEFORE the deployment group has been created, which means that these instances will never get deployed. I tried to think of a Lambda function to forcefully deploy these instances, however the problems persists because the CodeDeploy Deployment Group will still not be available yet most likely, or if it was, it's not reliable.
This problem only occurs when the stack is created for the first time.
This is my CloudFormation template:
[...]
UpdateApiAutoscalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName:
Fn::Join:
- ''
- - !ImportValue UpdateApiCodeDeployApplication
- -autoscaling-group
- !Ref Environment
DesiredCapacity: !Ref MinimumApiAmount
HealthCheckGracePeriod: 30
HealthCheckType: ELB
LaunchConfigurationName: !Ref UpdateApiAutoscalingLaunchConfiguration
TargetGroupARNs:
- !Ref UpdateApiTargetGroup
MaxSize: !Ref MaximumApiAmount
MinSize: !Ref MinimumApiAmount
VPCZoneIdentifier:
- Fn::Select:
- 0
- !Split
- ","
- Fn::ImportValue:
!Sub "PrivateSubnets-${Environment}"
Tags:
- Key: Environment
Value: !Ref Environment
PropagateAtLaunch: true
- Key: CompanySshAccess
Value: 1
PropagateAtLaunch: true
- Key: Application
Value: update-api
PropagateAtLaunch: true
# Defines how the Update API servers should be provisioned in the scaling group.
UpdateApiAutoscalingLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
AssociatePublicIpAddress: false
IamInstanceProfile: !Ref UpdateApiInstanceRole
ImageId: !FindInMap [Api, Image, !Ref Environment]
InstanceType: !FindInMap [Api, InstanceType, !Ref Environment]
LaunchConfigurationName: !Sub 'update-api-launchconfig-${Environment}'
SecurityGroups:
- Fn::ImportValue: !Sub 'InternalBastionSecurityGroupId-${Environment}'
- !GetAtt LoadBalancerProtectedSecurityGroup.GroupId
UpdateApiCodeDeploymentGroup:
Type: AWS::CodeDeploy::DeploymentGroup
Properties:
DeploymentGroupName: !Ref Environment
DeploymentConfigName: "atleast-one-instance-online"
ServiceRoleArn: !Ref CodeDeployServiceRoleArn # TODO: create CodeDeployServiceRole using CloudFormation
ApplicationName: !ImportValue UpdateApiCodeDeployApplication
LoadBalancerInfo:
ElbInfoList:
- Name: !GetAtt UpdateApiLoadBalancer.LoadBalancerName
DeploymentStyle:
DeploymentOption: !FindInMap [Api, DeploymentStyleOption, !Ref Environment]
DeploymentType: !FindInMap [Api, DeploymentStyleType, !Ref Environment]
AutoScalingGroups:
- !Ref UpdateApiAutoscalingGroup
[...]