Invalid template resource property 'Ref' - amazon-web-services

Question

I am new to writing yaml scripts and I keep getting this error 'Invalid template resource property 'Ref'' when I try to create a stack on Cloudformation. Is there something I am missing in my code. Please do let me know. Thanks! I have written this template to bring up an emr cluster using yaml.
AWSTemplateFormatVersion: 2010-09-09
Description: EMR Cluster for Spark
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Common Parameters
Parameters:
- SystemValue
- SubSystemValue
- RevenueStreamValue
- EnvironmentValue
- KMSArn
- Label:
default: EMR Parameters
default: Lambda Parameters
Parameters:
- EMRClusterName
- KeyName
- MasterInstanceType
- CoreInstanceType
- NumberOfCoreInstances
- VpcId
- VPCSubnetIds
- LogUriFolder
- S3DataUri
- ReleaseLabel
- ApplicationsList
- PackageBucket
- EbsRootVolumeSize
- FunctionName1Value
- Label:
default: S3 Parameters
Parameters:
- EDWBucketName
ParameterLabels:
SystemValue:
default: System
SubSystemValue:
default: SubSystem
RevenueStreamValue:
default: Revenue Stream
EnvironmentValue:
default: Environment Value
FunctionName1Value:
default: Lambda-1 Name
EMRClusterName:
default: EMR Cluster Name
KeyName:
default: Key Name
MasterInstanceType:
default: Master Instance Type
CoreInstanceType:
default: Core Instance Type
NumberOfCoreInstances:
default: Number Of Core Instances
VpcId:
default: VPC ID
VPCSubnetIds:
default: VPC Subnet ID
LogUriFolder:
default: Log Uri Folder
S3DataUri:
default: S3 Data Uri
ReleaseLabel:
default: Release Label
ApplicationsList:
default: Applications List
KMSArn:
default: KMS Arn
EDWBucketName:
default: EDW Bucket Name
PackageBucket:
default: Package Bucket Name
EbsRootVolumeSize:
default: Ebs Root Volume Size
#########################################################################
Parameters:
KMSArn:
Type: String
Description: Enter KMS ARN based on your subsystem.
Default: 'a36ef9be-97e1-4949-9b04-c1979eda5955'
SystemValue:
Type: String
Description: Enter System Name
Default: 'Messaging'
SubSystemValue:
Type: String
Description: Enter SubSystem Name
Default: 'EDW'
RevenueStreamValue:
Type: String
Description: Enter Revenue Stream Name
Default: 'edw'
FunctionName1Value:
Type: String
Default: 'EMRCluster'
Description: Enter 1st Lambda Function Name(Do not append System & Sub-System Name).
EDWBucketName:
Type: String
Default: 'crx-dev-messaging-edw'
Description: Enter crx-[env]-messaging-edw
PackageBucket:
Type: String
Default: 'crx-dev-deployments'
Description: Enter crx-[env]-deployments
EnvironmentValue:
Type: String
Default: 'dev'
LambdaRuntime:
Type: String
Default: 'Python-2.7'
TagEnvironmentValue:
Type: String
Default: 'dev'
VpcId:
Type: 'AWS::EC2::VPC::Id'
Default: vpc-7c368507
VpcCIDR:
Type: String
Default: '10.10.16.0/20'
VPCSubnetIds:
Type: AWS::EC2::Subnet::Id
Default: subnet-4d527c62
Env:
Type: String
Default: 'dev'
KeyName:
Type: AWS::EC2::KeyPair::KeyName
Description: Ane<Env>-Messaging-EDW
Default: development
MasterInstanceType:
Type: String
Description: Instance type to be used for the master instance.
Default: 'm5.xlarge'
AllowedValues:
- m5.xlarge
- m1.medium
CoreInstanceType:
Type: String
Description: Instance type to be used for core instances.
Default: 'm5.xlarge'
AllowedValues:
- m5.xlarge
- m1.medium
NumberOfCoreInstances:
Description: Must be a valid number - 2
Type: Number
Default: '2'
LogUriFolder:
Type: String
Description: S3 Folder name for spark logs (spark-logs)
Default: 's3n://crx-dev-messaging-edw/spark-logs/'
S3DataUri:
Type: String
Description: Must be a valid S3 bucket URL
Default: 's3n://crx-messaging-edw'
EbsRootVolumeSize:
Type: String
Description: Specify the volume size
Default: '200'
ReleaseLabel:
Type: String
Description: Must be a valid EMR release version emr-6.0.0
Default: 'emr-6.0.0'
ApplicationsList:
Type: String
Description: Please select which application will be installed on the cluster
this would be either Ganglia,spark,Hadoop etc.,
Default: 'Spark'
AllowedValues:
- Hadoop
- Ganglia
- Spark
#########################################################################
Resources:
Bucket1:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub 'crx-${EnvironmentValue}-${SubSystemValue}'
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: !Ref KMSArn
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
EbsRootVolumeSize: !Ref EbsRootVolumeSize
SvcAccessSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub SG-${Env}-${SystemValue}-${SubSystemValue}-ServiceAccess
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-Service Access Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-ServiceAccess'
EmrMasterSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRMaster'
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-EMR Managed Master Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRMaster'
EmrSlaveSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRSlave'
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-EMR Managed Slave Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRSlave'
LambdaSecurityGroup:
Type: AWS::EC2::SecurityGroup
Condition : HasFunctionName1
Properties:
GroupName: !Sub 'SG-${SGEnvironmentValue}-${SystemValue}-${SubSystemValue}'
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-Lambda Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${SGEnvironmentValue}-${SystemValue}-${SubSystemValue}'
Lambda1:
Type: AWS::Lambda::Function
DependsOn:
- LambdaExecutionRole
- ManagerApiPolicy
Properties:
Description: Script to launch EMR
FunctionName: !Sub ' ${SystemValue}-${SubSystemValue}-${FunctionName1Value}'
Handler: lambda_function.lambda_handler
Code:
S3Bucket: !Ref PackageBucket
S3Key: emr-launch.zip
Role: !GetAtt
- LambdaExecutionRole
- Arn
Runtime: !Ref LambdaRuntime
Timeout: '900'
MemorySize: 512
Environment:
Variables:
parameterPath: !Sub '/crx/${SystemValue}/${SubSystemValue}/'
region: !Ref 'AWS::Region'
VpcConfig:
SubnetIds: !Ref VPCSubnetIds
SecurityGroupIds:
- !GetAtt
- LambdaSecurityGroup
- GroupId
Tags:
- Value: !Sub '${SystemValue}-${SubSystemValue}-${FunctionName1Value}'
Key: Name
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Value: !Ref RevenueStreamValue
Key: RevenueStream
- Value: !Ref TagEnvironmentValue
Key: Environment
LambdaExecutionRole:
Type: AWS::IAM::Role
DependsOn:
- LambdaSecurityGroup
Properties:
RoleName: !Sub '${SystemValue}-${SubSystemValue}'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- states.amazonaws.com
- events.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagerApiPolicy:
Type: 'AWS::IAM::ManagedPolicy'
DependsOn:
- LambdaExecutionRole
Properties:
ManagedPolicyName: !Sub '${SystemValue}-${SubSystemValue}'
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: common
Effect: Allow
Action:
- 'ec2:CreateNetworkInterface'
- 'ec2:Describe*'
- 'ec2:Get*'
- 'ec2:DeleteNetworkInterface'
- 'kms:GenerateDataKey'
- 'kms:ListAliases'
- 'kms:ListKeys'
- 'kms:ReEncryptTo'
- 'kms:CreateKey'
- 'kms:GenerateRandom'
- 'kms:ReEncryptFrom'
- 'logs:CreateLogGroup'
- 'logs:PutLogEvents'
- 'logs:CreateLogStream'
- 'ssm:DescribeParameters'
- 'lambda:InvokeFunction'
Resource: '*'
- Sid: ssm
Effect: Allow
Action:
- 'ssm:GetParametersByPath'
- 'ssm:GetParameters'
- 'ssm:GetParameter'
Resource:
- !Join
- ''
- - 'arn:aws:ssm:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':parameter/crx/'
- !Sub '${SystemValue}/${SubSystemValue}*'
- Sid: kms
Effect: Allow
Action: 'kms:*'
Resource:
- !Ref KMSArn
- !Join
- ''
- - 'arn:aws:kms:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':alias/'
- !Sub '${SystemValue}-${SubSystemValue}'
- Sid: s3
Effect: Allow
Action:
- 's3:PutObject'
- 's3:GetObject'
- 's3:List*'
Resource:
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-${PackageBucketName}'
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-${PackageBucketName}/*'
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-public-assets'
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-public-assets/*'
Roles:
- !Ref LambdaExecutionRole
PermissionForEventsToInvokeLambda:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName:
!Ref EmrCloudWatchEventLambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn:
Fn::GetAtt:
- EventRuleEMRtest
- Arn
EMRClusterServiceRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role
Path: /
EMRClusterinstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref EMRClusterinstanceProfileRole

Answer 1

Line 192 in the above template
EbsRootVolumeSize: !Ref EbsRootVolumeSize
is not inside any of the resources block.

Answer 2

The CloudFormation Linter and Visual Studio Code extension catch these template errors:
W2030 You must specify a valid Default value for LambdaRuntime (Python-2.7).
Valid values are ['dotnetcore1.0', 'dotnetcore2.0', 'dotnetcore2.1', 'dotnetcore3.1', 'go1.x', 'java8', 'java11', 'nodejs', 'nodejs4.3-edge', 'nodejs4.3', 'nodejs6.10', 'nodejs8.10', 'nodejs10.x', 'nodejs12.x', 'provided', 'python2.7', 'python3.6', 'python3.7', 'python3.8', 'ruby2.5', 'ruby2.7']
template.yaml:111:5
E3001 Invalid resource attribute Ref for resource EbsRootVolumeSize
template.yaml:191:3
E3001 Type not defined for resource EbsRootVolumeSize
template.yaml:191:3
E3004 Circular Dependencies for resource EbsRootVolumeSize. Circular dependency with [EbsRootVolumeSize]
template.yaml:191:3
E8002 Condition HasFunctionName1 is not defined.
template.yaml:270:5
E1019 Parameter SGEnvironmentValue for Fn::Sub not found at Resources/LambdaSecurityGroup/Properties/GroupName/Fn::Sub
template.yaml:272:7
E1019 Parameter SGEnvironmentValue for Fn::Sub not found at Resources/LambdaSecurityGroup/Properties/Tags/4/Value/Fn::Sub
template.yaml:293:11
E3002 Property SubnetIds should be of type List or Parameter should be a list for resource Lambda1
template.yaml:318:9
E1019 Parameter PackageBucketName for Fn::Sub not found at Resources/ManagerApiPolicy/Properties/PolicyDocument/Statement/3/Resource/0/Fn::Join/1/2/Fn::Sub
template.yaml:422:13
E1019 Parameter PackageBucketName for Fn::Sub not found at Resources/ManagerApiPolicy/Properties/PolicyDocument/Statement/3/Resource/1/Fn::Join/1/2/Fn::Sub
template.yaml:422:13
E1012 Ref EmrCloudWatchEventLambda not found as a resource or parameter
template.yaml:451:7
E1010 Invalid GetAtt EventRuleEMRtest.Arn for resource PermissionForEventsToInvokeLambda
template.yaml:456:9
E3001 Invalid resource attribute ManagedPolicyArns for resource EMRClusterServiceRole
template.yaml:472:5
E3001 Invalid resource attribute Path for resource EMRClusterServiceRole
template.yaml:474:5
E1012 Ref EMRClusterinstanceProfileRole not found as a resource or parameter
template.yaml:481:9

Answer 3

It is caused by this:
EbsRootVolumeSize: !Ref EbsRootVolumeSize
Whatever it is, it should not be in the location it is currently present.
Also ManagedPolicyArns and Path in EMRClusterServiceRole resource have incorrect indentations. Could be due to copy-paste to SO though.
Other issue is that condition HasFunctionName1 is not defined. Same goes for SGEnvironmentValue and EventRuleEMRtest.