Terraform / Cloudformation - Pass parameter as YAML - amazon-web-services

I use Terraform to launch a Cloudformation stack to create Glue Databrew resources that don't exist yet on Terraform.
The thing is that I've a variable in Terraform that corresponds to the list of my data sources and in order to create the databrew resources associated to this data, I loop over my list to create one instance of my Cloudformation template for each data source.
Inside this template, I've a resource that I want to be different per data source. It correspond to the AWS::DataBrew::Ruleset resource.
It looks like this :
DataBrewDataQualityRuleset:
Type: AWS::DataBrew::Ruleset
Properties:
Name: !Ref RuleSetName
Description: Data Quality ruleset
Rules:
- Name: Check columns for missing values
Disabled: false
CheckExpression: AGG(MISSING_VALUES_PERCENTAGE) == :val1
SubstitutionMap:
- ValueReference: ":val1"
Value: '0'
ColumnSelectors:
- Regex: ".*"
- Name: Check two
Disabled: false
CheckExpression: :col IN :list
SubstitutionMap:
- ValueReference: ":col"
Value: "`group`"
- ValueReference: ":list"
Value: "[\"Value1\", \"Value2\"]"
TargetArn: !Sub SomeArn
What I want to do is, extract the Rules part of the component and create one file where I will put all my rules per data sources. In fact having something like below :
DataBrewDataQualityRuleset:
Type: AWS::DataBrew::Ruleset
Properties:
Name: !Ref RuleSetName
Description: Data Quality ruleset
Rules: !Ref Rules
TargetArn: !Sub SomeArn
And in my terraform, my Rules parameter would be my actual set of rules for one particular data source.
I've thought about having one YAML file from which I would loop on terraform but I'm not sure it's doable and if cloudformation would accept YAML as parameter type.
Below you'll also find my terraform component :
resource "aws_cloudformation_stack" "databrew_jobs" {
for_each = var.data_sources
name = "datachecks-${each.value.stack_name}"
parameters = {
Bucket = "test_bucket"
DataSetKey = "raw/${each.value.job_name}"
DataSetName = "dataset-${each.value.stack_name}"
RuleSetName = "ruleset-${each.value.stack_name}"
JobName = "profile-job-${each.value.stack_name}"
DataSourceName = "${each.value.stack_name}"
JobResultKey = "databrew-results/${each.value.job_name}"
RoleArn = iam_role_test.arn
}
template_body = file("${path.module}/databrew-job.yaml")
}
Do you have any idea how could I achieve this ?
Thanks in advance !

Related

AWS CDK - get ProvisioningArtifactIds from ServiceCatalog product

In CDK I couldn't find from documentation how I can get ProvisioningArtifactIds e.g. pa-4abcdjnxjj6ne? CloudFormation will return the value, but not CDK.
(Edit: in CF, ProductId: !Ref Product and ProvisioningArtifactId: !GetAtt Product.ProvisioningArtifactIds)
I need it, so I can make a CloudFormation Product
product = aws_servicecatalog.CloudFormationProduct(self, "Product",
product_name="My Product",
owner="Product Owner",
product_versions=[servicecatalog.CloudFormationProductVersion(
product_version_name="v1",
cloud_formation_template=servicecatalog.CloudFormationTemplate.from_product_stack(S3BucketProduct(self, "S3BucketProduct"))
)])
and later in the same stack make the association
aws_servicecatalog.CfnServiceActionAssociation(
self,
"serviceId",
product_id=product.product_id,
provisioning_artifact_id="????", # ????
service_action_id=myServiceAction.attr_id,
)
The CloudFormation docs list the return values for the resource.
The !Ref Product value is easy. The CDK source code shows it is exposed directly on the L2 CloudFormationProduct as product_id.
# !Ref Product
ref = product.product_id
The !GetAtt Product.ProvisioningArtifactIds value is trickier. First reference the underlying L1 CfnCloudFormationProduct construct with escape hatch syntax. Then call get_att:
# !GetAtt Product.ProvisioningArtifactIds
cfn_product = product.node.default_child
get_att = cfn_product.get_att("ProvisioningArtifactIds")

How to create an Athena stack and consume Glue Data catalog?

I have to create an athena template in cloud formation, the task is to replicate the next Terraform script using CF:
resource "aws_athena_workgroup" "sample_athena_wg" {
name = "sample_athena_wg"
}
resource "aws_athena_database" "sample_athena_database" {
name = "sample_athena_database"
bucket = "sample_bucket_id"
}
resource "aws_athena_named_query" "test_query" {
name = "Test"
workgroup = aws_athena_workgroup.sample_athena_wg.id
database = aws_athena_database.sample_athena_database.name
query = "SELECT * FROM ${aws_athena_database.sample_athena_database.name} limit 10;"
}
The problem is that there is no such a resource in CF called "AWS::ATHENA::DATABASE" or something like that, and I don't really know what the terraform resource "aws_athena_database" is creating behind the scenes. When I deploy the Terraform script, it seems like this creates a glue database, but I do know what else this creates.
I found that Terraform creates an Athena Workgroup and GLue database behind the scenes when you try to create a aws_athena_database. I could replicate this resource in CF like this:
AthenaWorkgroup:
Type: AWS::Athena::WorkGroup
Properties:
Name: ....
State: ENABLEDr
WorkGroupConfiguration:
BytesScannedCutoffPerQuery: ...
EnforceWorkGroupConfiguration: ...
PublishCloudWatchMetricsEnabled: ...
ResultConfiguration:
OutputLocation: !Ref S3LocationPath
GlueDatabase:
Type: AWS::Glue::Database
Properties:
DatabaseInput:
Name: ...

Passing CloudFormation parameters to AppSync Resolver

So I have my CloudFormation template defined to include a Parameters section with several parameters including
Parameters:
DefaultLimit:
Type: Number
I also have a GraphQL API defined in which I am using AppSync PIPELINE resolvers to run multiple operations in sequence.
QueryResolver:
Type: AWS::AppSync::Resolver
DependsOn: AppSyncSchema
Properties:
ApiId: !GetAtt [AppSyncAPI, ApiId]
TypeName: Query
FieldName: getData
Kind: PIPELINE
PipelineConfig:
Functions:
- !GetAtt AuthFunction.FunctionId
- !GetAtt ScanDataFunction.FunctionId
RequestMappingTemplate: |
{
# Inject value of DefaultLimit in $context object
}
ResponseMappingTemplate: "$util.toJson($context.result)"
That all works as expected, except for injecting CFN parameter values in mapping templates.
The issue I am having is this -- I would like to pass the value of DefaultLimit to the before RequestMappingTemplate so that the value is available to the ScanDataFunction. The goal is for that value be used as the default limit value when the second function does, say a DynamoDB scan operation, and returns paginated results.
My current approach is to hardcode a default value of 20 for limit in the request mapping template of the ScanDataFunction. I am using a DynamoDB resolver for this operation. Instead, I would like to inject the parameter value because it would give me the flexibility to set different default values for different deployment environments.
Any help with this would be appreciated.
The | character in YAML starts a block and what you enter indented after that is all treated as text.
CloudFormation isn't going to process any of that. The solution I have generally seen is to use the Join intrinsic function. It ends up looking pretty bad can be difficult to maintain so I recommend using it sparingly. Below is a rough possible example:
Parameters:
DefaultLimit:
Type: Number
Resourece:
QueryResolver:
Type: AWS::AppSync::Resolver
DependsOn: AppSyncSchema
Properties:
ApiId: !GetAtt [AppSyncAPI, ApiId]
TypeName: Query
FieldName: getData
Kind: PIPELINE
PipelineConfig:
Functions:
- !GetAtt AuthFunction.FunctionId
- !GetAtt ScanDataFunction.FunctionId
RequestMappingTemplate:
Fn::Join:
- ""
- - "Line 1 of the template\n"
- "Line 2 of the template, DefaultList="
- Ref: DefaultLimit
- "\nLine 3 of the template"
ResponseMappingTemplate: "$util.toJson($context.result)"
Untested code warning

AWS CloudFormation & Service Catalog - Can I require tags with user values?

Our problem seems very basic and I would expect common.
We have tags that must always be applied (for billing). However, the tag values are only known at the time the stack is deployed... We don't know what the tag values will be when developing the stack, or when creating the product in the Service Catalog...
We don't want to wait until AFTER the resource is deployed to discover the tag is missing, so as cool as AWS config may be, we don't want to rely on its rules if we don't have to.
So things like Tag Options don't work, because it appears that they expect we know the tag value months prior to some deployment (which isn't the case.)
Is there any way to mandate tags be used for a cloudformation template when it is deployed? Better yet, can we have service catalog query for a tag value when deploying? Tags like "system" or "project", for instance, come and go over time and are not known up-front for many types of cloudformation templates we develop.
Isn't this a common scenario?
I am worried that I am missing something very, very simple and basic which mandates tags be used up-front, but I can't seem to figure out what. Thank you in advance. I really did Google a lot before asking, without finding a satisfying answer.
I don't know anything about service catalog but you can create Conditions and then use it to conditionally create (or even fail) your resource creation. Conditional Resource Creation e.g.
Parameters:
ResourceTag:
Type: String
Default: ''
Conditions:
isTagEmpty:
!Equals [!Ref ResourceTag, '']
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
Condition: isTagEmpty
Properties:
DBInstanceClass: <DB Instance Type>
Here RDS DB instance will only be created if tag is non-empty. But cloudformation will still return success.
Alternatively, you can try & fail the resource creation.
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !If [isTagEmpty, !Ref "AWS::NoValue", <DB instance type>]
I haven't tried this but it should fail as DB instance type will be invalid if tag is null.
Edit: You can also create your stack using the createStack CFN API. Write some code to read & validate the input (e.g. read from service catalog) & call the createStack API. I am doing the same from Lambda (nodejs) reading some input from Parameter Store. Sample code -
module.exports.create = async (event, context, callback) => {
let request = JSON.parse(event.body);
let subnetids = await ssm.getParameter({
Name: '/vpc/public-subnets'
}).promise();
let securitygroups = await ssm.getParameter({
Name: '/vpc/lambda-security-group'
}).promise();
let params = {
StackName: request.customerName, /* required */
Capabilities: [
'CAPABILITY_IAM',
'CAPABILITY_NAMED_IAM',
'CAPABILITY_AUTO_EXPAND',
/* more items */
],
ClientRequestToken: 'qwdfghjk3912',
EnableTerminationProtection: false,
OnFailure: request.onfailure,
Parameters: [
{
ParameterKey: "SubnetIds",
ParameterValue: subnetids.Parameter.Value,
},
{
ParameterKey: 'SecurityGroupIds',
ParameterValue: securitygroups.Parameter.Value,
},
{
ParameterKey: 'OpsPoolArnList',
ParameterValue: request.userPoolList,
},
/* more items */
],
TemplateURL: request.templateUrl,
};
cfn.config.region = request.region;
let result = await cfn.createStack(params).promise();
console.log(result);
}
Another option: add a AWS Custom Resource backed by Lambda. Check for tags in this section & return failure if it doesn't satisfy the constraints. Make all other resource creation depend on this resource (so that they all create if your checks pass). Link also contains example. You will also have to add handling for stack update & deletion (like a default success). I think this is your best bet as of now.

AWS Cloudformation - cannot set parameters group name

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-dbparametergroup.html
and
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbclusterparametergroup.html
Unless I am doing it wrong - I cannot set parameters groups name via CF templates, while I can easily set it via GUI and via CLI (https://docs.aws.amazon.com/cli/latest/reference/rds/create-db-parameter-group.html). Additionally, for whatever reason, db cluster parameters group expects a non-empty parameters.
Is there a way to pass both name and parameters via CloudFormation?
For the name, add a tag called Name. For the parameters I end up just adding a unimportant setting to its default value:
ParameterGroup:
Type: AWS::RDS::DBParameterGroup
Properties:
...
Parameters:
application_name: ''
Tags:
- Key: Name
Value: "MyParameterGroupName"