I am trying to create a database on glue using cloud formation but it fails with the below error. Am I missing something?
Property validation failure: [The property {/DatabaseInput} is required, The property {/CatalogId} is required]
This is how my template code block looks like
GlueDatabase:
Type: AWS::Glue::Database
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseInput: !Ref TeamName
According to the docs the DatabaseInput should have the following structure:
GlueDatabase:
Type: AWS::Glue::Database
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseInput:
Description: String
LocationUri: String
Name: String
Parameters: Json
Thus the question is, what TeamName is in your tempalte?
Related
I have the following parameter in my CloudFormation script:
CloudFormationURL:
Type: String
Description: S3 URL for nested CloudFormation templates
Default: ""
This parameter covers the CloudFormation scripts in a nested folder of my deployment config.
I use it with a resource like this:
CloudWatchDashboardStack:
Type: "AWS::CloudFormation::Stack"
Properties:
TemplateURL: !Sub "${CloudFormationURL}/cloudwatch-dashboard.cfn.yaml"
Parameters:
AppName: !Ref AppName
DeployPhase: !Ref DeployPhase
DeveloperPrefix: !Ref DeveloperPrefix
Environment: !Ref Environment
Which works fine, and has worked for months.
I needed to add another resource, so I added this:
BatchDNSResources:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
AppName: !Ref AppName
Environment: !Ref Environment
DeveloperPrefix: !Ref DeveloperPrefix
DeployPhase: !Ref DeployPhase
AppVersion: !Ref AppVersion
SharedBucketName: !Ref SharedBucketName
S3Version: !Ref S3Version
HostedZone: !Ref HostedZone
VPCStackName: !FindInMap
- EnvironmentMap
- !Ref Environment
- VpcStackName
Company: !Ref Company
CostCenter: !Ref CostCenter
Team: !Ref Team
TemplateURL: !Sub "${CloudFormationURL}/batch-dns.cfn.yaml"
CloudFormation throws this error and then fails:
Parameters: [CloudFormationURL] must have values
Checking the changeset for the stack I can see the following value for the CloudFormationURL:
s3://application-shared-dev/application-name/qa/cf/nested/KShyDj205UK8mz6W_XUA5TnEF8nqPWHS
Checking the application predeploy logs I can see:
upload: deploy/cloudformation/templates/nested/batch-dns.cfn.yaml to s3://application-shared-dev/application-name/qa/cf/nested/KShyDj205UK8mz6W_XUA5TnEF8nqPWHS/batch-dns.cfn.yaml
And I can see the file in the S3 bucket.
If I remove BatchDNSResource the stack completes successfully.
What the heck am I missing here?
Sometimes, the smallest things will get you.
I had copied the Parameters from the master CloudFormation script, including this one, into the nested script:
CloudFormationURL:
Type: String
Description: S3 URL for nested CloudFormation templates
Default: ""
If you look closely, you will see that I did not pass the parameter into the nested script when calling the resource:
BatchDNSResources:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
AppName: !Ref AppName
Environment: !Ref Environment
DeveloperPrefix: !Ref DeveloperPrefix
DeployPhase: !Ref DeployPhase
AppVersion: !Ref AppVersion
SharedBucketName: !Ref SharedBucketName
S3Version: !Ref S3Version
HostedZone: !Ref HostedZone
VPCStackName: !FindInMap
- EnvironmentMap
- !Ref Environment
- VpcStackName
Company: !Ref Company
CostCenter: !Ref CostCenter
Team: !Ref Team
TemplateURL: !Sub "${CloudFormationURL}/batch-dns.cfn.yaml"
Because the CloudFormation console was saying the issue was with the BatchDNSResources I kept looking at the master script for the problem and missing the reference in the other script. There are two ways to solve this problem:
Keep CloudFormationURL as a parameter in the nested script (if you need it for some reason) and pass the value from the master script.
Remove the parameter from the nested script (if it is not needed)
Sometimes just asking for an extra set of eyeballs and getting a little rest will help you to find the issues. I want to leave this question/answer in place here because when I was searching for the error here and elsewhere no one ever mentioned (probably out of embarrassment) that the mistake is simply overlooking something like this. I hope this answer prompts others to check everything when they run across this type of error.
Parameters:
ClusterName:
Type: String
ClusterVersion:
Type: Number
AllowedValues: [1.21, 1.20, 1.19, 1.18]
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Sub ${ClusterName}
Version: !Sub ${ClusterVersion}
RoleArn: !Sub ${RoleArnValue}
ResourcesVpcConfig:
SecurityGroupIds:
- !Sub ${ListOfSecurityGroupIDs}
SubnetIds:
- !Sub ${ListOfSubnetIDs}
Above is the .yaml clouldformation template I have created so i can spin up eks cluster. Then i am using aws cli to spin up the cluster using the following command.
aws cloudformation deploy --template-file eks.yaml --stack-name cluster-test --parameter-overrides ClusterName=Dev ClusterVersion=1.21 ListOfSubnetIDs=subnet-11111d11b11b011f4,subnet-99999d237f87f11d7,subnet-222222c110c7e4be7,subnet-88888884de8d25176 ListOfSecurityGroupIDs=sg-01111111a21221 RoleArnValue=arn:aws:iam::123456546456:role/cluster-ServiceRole-WMIC72AOWSP0 --capabilities CAPABILITY_NAMED_IAM
I get the following error
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template error: variable ListOfSecurityGroupIDs in Fn::Sub expression does not resolve to a string
I am not sure why. Am i using !sub in correctly? Would really appreciate input on this.
Since you want to reference the parameters you provided the template as they are, you should use the Ref function.
Here's an example of a valid template:
Parameters:
ClusterName:
Type: String
RoleArnValue:
Type: String
ListOfSubnetIDs:
Description: Array of Subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ListOfSecurityGroupIDs:
Description: Array of security group ids
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
EKSCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Ref ClusterName
RoleArn: !Ref RoleArnValue
ResourcesVpcConfig:
SecurityGroupIds: !Ref ListOfSecurityGroupIDs
SubnetIds: !Ref ListOfSubnetIDs
and here's how I deployed it:
aws cloudformation deploy --template-file eks.yml --stack-name cluster-test --parameter-overrides ClusterName=Dev ListOfSubnetIDs=subnet-be0a99c4,subnet-c71046ae ListOfSecurityGroupIDs=sg-009690ac6b3bff6df,sg-009a3f1cb63943941 -RoleArnValue=...
Sub should be used when you want to perform string manipulation. Checkout the examples from the documentation.
I having issues creating an AWS log group that belongs to a bigger CloudFormation template. So just for testing, I'm creating just the log group with the following template
Parameters:
LogGroupName:
Type: String
Description: 'cloudwatch log group name'
Default: "test-log-group"
LogGroupRetention:
Type: Number
Description: Retention period for log groups in cloudwatch
Default: 30
DelPolicy:
Type: String
Description: 'Deletion policy'
Default: "Retain"
Resources:
LLGO1WY:
Type: 'AWS::Logs::LogGroup'
Properties:
awslogs-region: !Ref 'AWS::Region'
LogGroupName: !Ref LogGroupName
RetentionInDays: !Ref LogGroupRetention
DeletionPolicy: !Ref DelPolicy
When I import the template during the manual stack creation ("Create Stack" button), I get the following when I get to the "Import Overview" page.
There was an error creating this change set
The following resources to import [LLGO1WY] must have DeletionPolicy attribute specified in the template.
If you look the documentation for AWS::Logs::LogGroup, it doesn't even have a DeletionPolicy defined as a property. Note that if I remove that property, I get the same error. Any clues?
AWS::Logs::LogGroup does not have DeletionPolicy property. DeletionPolicy is a top level attribute which you can't parametrize.
I guess you wanted maybe:
Resources:
LLGO1WY:
Type: 'AWS::Logs::LogGroup'
DeletionPolicy: Retain # <--- This is not property and must be here
Properties:
awslogs-region: !Ref 'AWS::Region'
LogGroupName: !Ref LogGroupName
RetentionInDays: !Ref LogGroupRetention
I am trying to create the glue security configuration using cloudformation script but I am getting the following error:
Property validation failure: [Value of property {/EncryptionConfiguration/S3Encryptions} does not match type {Array}]
What is the right way to give the S3encryption?
AWSTemplateFormatVersion: 2010-09-09
Description: Script creates resources for GlueSecurityConfiguration
Resources:
GlueSecurityConfiguration:
Type: AWS::Glue::SecurityConfiguration
Properties:
EncryptionConfiguration:
S3Encryptions:
KmsKeyArn: !Ref KMSArn
S3EncryptionMode: SSE-KMS
JobBookmarksEncryption:
KmsKeyArn: !Ref KMSArn
CloudWatchEncryption:
KmsKeyArn: !Ref KMSArn
Name: !Sub '${SystemValue}-${SubSystemValue}'
I think it should be
- KmsKeyArn: !Ref KMSArn
S3EncryptionMode: SSE-KMS
since S3Encryptions expects an array.
From the documentation, AWS::Athena::NamedQuery, it is unclear how to attach Athena to an S3 bucket specified in the same stack.
If I had to guess from the example, I would imagine that you can write a template like,
Resources:
MyS3Bucket:
Type: AWS::S3::Bucket
... other params ...
AthenaNamedQuery:
Type: AWS::Athena::NamedQuery
Properties:
Database: "db_name"
Name: "MostExpensiveWorkflow"
QueryString: >
CREATE EXTERNAL TABLE db_name.test_table
(...) LOCATION s3://.../path/to/folder/
Would a template like the above work? Upon stack creation, will the table db_name.test_table be available to run queries on?
Turns out the way you connect the S3 and Athena is to make a Glue table! How silly of me!! Of course Glue is how you connect things!
Sarcasm aside, this is a template that worked for me when using AWS::Glue::Table and AWS::Glue::Database,
Resources:
MyS3Bucket:
Type: AWS::S3::Bucket
MyGlueDatabase:
Type: AWS::Glue::Database
Properties:
DatabaseInput:
Name: my-glue-database
Description: "Glue beats tape"
CatalogId: !Ref AWS::AccountId
MyGlueTable:
Type: AWS::Glue::Table
Properties:
DatabaseName: !Ref MyGlueDatabase
CatalogId: !Ref AWS::AccountId
TableInput:
Name: my-glue-table
Parameters: { "classification" : "csv" }
StorageDescriptor:
Location:
Fn::Sub: "s3://${MyS3Bucket}/"
InputFormat: "org.apache.hadoop.mapred.TextInputFormat"
OutputFormat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
SerdeInfo:
Parameters: { "separatorChar" : "," }
SerializationLibrary: "org.apache.hadoop.hive.serde2.OpenCSVSerde"
StoredAsSubDirectories: false
Columns:
- Name: column0
Type: string
- Name: column1
Type: string
After this, the database and table were in the AWS Athena Console!