We are automating a lambda via SAM to run on a Schedule Event. We use YAML but we are unable to work out how to use !Sub to make the Input be dynamic.
If you read the sam documentation it says that Input needs to be a JSON formatted string
The following code works for us:
Events:
Event1:
Type: Schedule
Properties:
Schedule: rate(1 minute)
Input: >-
{
"sqsUrl": "https://sqs.12344.url",
"snsArn": "arn:val"
}
But we need to insert dynamic params into the Input like so:
Events:
Event1:
Type: Schedule
Properties:
Schedule: rate(1 minute)
Input: >-
{
"sqsUrl": "https://sqs.${AWS::AccountId}.url",
"snsArn": "arn:val"
}
We have tried to do this in multiple ways, using a !Sub but the deployment always fails saying that it needs to be valid JSON.
What is the correct way to make this JSON string use variables?
Thanks,
Mark
So, you should wrap all Input value (in your case this is json-string and of course it should be wrapped with some quotes) with the !Sub function.
Then, it will look like:
Input:
Fn::Sub: '{"sqsUrl": "https://sqs.${AWS::AccountId}.url","snsArn": "arn:val"}'
I've used something like:
!Sub |
{
"sqsUrl": "https://sqs.${AWS::AccountId}.url",
"snsArn": "arn:val"
}
the | (and >- among others) define the way yaml handles the line breaks in the string.
Related
My system design
I have created 4 Glue Jobs: testgluejob1, testgluejob2, testgluejob3 and common-glue-job.
EventBridge rule detects SUCCEEDED state of glue jobs such as testgluejob1, testgluejob2, testgluejob3.
After getting Glue Job's SUCCEEDED notification, Glue Trigger run to start common-glue-job.
Problem
I want to use the jobname string in common-glue-job script as parameter
Is it possible to pass parameters to Glue Workflow or Trigger from EventBridge?
The things I tried
Trigger can pass parameters to common-glue-job
https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-glue-trigger.html
Type: AWS::Glue::Trigger
...
Actions:
- JobName: prod-job2
Arguments:
'--job-bookmark-option': job-bookmark-enable
If set Run Properties for Glue Workflow, I cat get it from common-glue-job by using boto3 and get_workflow_run_properties() function. But I have no idea how to put Run Properties from EventBridge by CFn
https://docs.aws.amazon.com/glue/latest/dg/workflow-run-properties-code.html
I set Target InputTransformer of EventBridge Rule, but I'm not sure how to use this value in common-glue-job.
DependsOn:
- EventBridgeGlueExecutionRole
- GlueWorkflowTest01
Type: AWS::Events::Rule
Properties:
Name: EventRuleTest01
EventPattern:
source:
- aws.glue
detail-type:
- Glue Job State Change
detail:
jobName:
- !Ref GlueJobTest01
state:
- SUCCEEDED
Targets:
-
Arn: !Sub arn:aws:glue:${AWS::Region}:${AWS::AccountId}:workflow/${GlueWorkflowTest01}
Id: GlueJobTriggersWorkflow
RoleArn: !GetAtt 'EventBridgeGlueExecutionRole.Arn'
InputTransformer:
InputTemplate: >-
{
"--ORIGINAL_JOB": <jobName>
}
InputPathsMap:
jobName : "$.detail.jobName"
Any help would be greatly appreciated.
If I understand you correctly, you already have information in EventBridge event, but cannot access it from your Glue job. I used the following workaround to do this:
You need to get an event ID from Glue workflow properties
event_id = glue_client.get_workflow_run_properties(Name=self.args['WORKFLOW_NAME'],
RunId=self.args['WORKFLOW_RUN_ID'])['RunProperties']['aws:eventIds'][1:-1]
Get all NotifyEvent events for the last several minutes. It's up to you to decide how much time can pass between the workflow start and your job start.
response = event_client.lookup_events(LookupAttributes=[{'AttributeKey': 'EventName',
'AttributeValue': 'NotifyEvent'}],
StartTime=(datetime.datetime.now() - datetime.timedelta(minutes=5)),
EndTime=datetime.datetime.now())['Events']
Check which event has an enclosed event with the event ID we get from Glue workflow.
for i in range(len(response)):
event_payload = json.loads(response[i]['CloudTrailEvent'])['requestParameters']['eventPayload']
if event_payload['eventId'] == event_id:
event = json.loads(event_payload['eventBody'])
In event variable you get full content of the event that triggered workflow.
So I have my CloudFormation template defined to include a Parameters section with several parameters including
Parameters:
DefaultLimit:
Type: Number
I also have a GraphQL API defined in which I am using AppSync PIPELINE resolvers to run multiple operations in sequence.
QueryResolver:
Type: AWS::AppSync::Resolver
DependsOn: AppSyncSchema
Properties:
ApiId: !GetAtt [AppSyncAPI, ApiId]
TypeName: Query
FieldName: getData
Kind: PIPELINE
PipelineConfig:
Functions:
- !GetAtt AuthFunction.FunctionId
- !GetAtt ScanDataFunction.FunctionId
RequestMappingTemplate: |
{
# Inject value of DefaultLimit in $context object
}
ResponseMappingTemplate: "$util.toJson($context.result)"
That all works as expected, except for injecting CFN parameter values in mapping templates.
The issue I am having is this -- I would like to pass the value of DefaultLimit to the before RequestMappingTemplate so that the value is available to the ScanDataFunction. The goal is for that value be used as the default limit value when the second function does, say a DynamoDB scan operation, and returns paginated results.
My current approach is to hardcode a default value of 20 for limit in the request mapping template of the ScanDataFunction. I am using a DynamoDB resolver for this operation. Instead, I would like to inject the parameter value because it would give me the flexibility to set different default values for different deployment environments.
Any help with this would be appreciated.
The | character in YAML starts a block and what you enter indented after that is all treated as text.
CloudFormation isn't going to process any of that. The solution I have generally seen is to use the Join intrinsic function. It ends up looking pretty bad can be difficult to maintain so I recommend using it sparingly. Below is a rough possible example:
Parameters:
DefaultLimit:
Type: Number
Resourece:
QueryResolver:
Type: AWS::AppSync::Resolver
DependsOn: AppSyncSchema
Properties:
ApiId: !GetAtt [AppSyncAPI, ApiId]
TypeName: Query
FieldName: getData
Kind: PIPELINE
PipelineConfig:
Functions:
- !GetAtt AuthFunction.FunctionId
- !GetAtt ScanDataFunction.FunctionId
RequestMappingTemplate:
Fn::Join:
- ""
- - "Line 1 of the template\n"
- "Line 2 of the template, DefaultList="
- Ref: DefaultLimit
- "\nLine 3 of the template"
ResponseMappingTemplate: "$util.toJson($context.result)"
Untested code warning
I've written a macro (called BucketChecker) that takes in an s3 bucket name and checks if it already exists.
The fragment will return true or false accordingly.
I would like to use this macro in a Conditions section as described in this article: https://cloudnineapps.com/blogs/cloud-computing/how-to-create-dynamic-condition-expressions-in-aws-cloudformation-using-macros/.
The idea is to use this as a condition in my template like this:
Conditions:
CreateBucket:
Fn::Equals: ['false', 'Fn::Transform': {
Name: BucketChecker,
Parameters: {
Operation: 'bucketExists',
BucketName: 'my-bucket'
}}]
Resources:
MyBucket:
Type: "AWS::S3::Bucket"
Condition: CreateBucket # -> only create it if doesn't yet exist
Properties:
BucketName: 'my-bucket'
But cfn-linter gives me the error: E8003 Fn::Equals element must be a supported function (Ref, Fn::FindInMap, Fn::Sub, Fn::Join, Fn::Select, Fn::Split)
My main question is: is it even possible to achieve this using CloudFormation? If yes, what's wrong with my template?
The linked article seems to be doing exactly the same, but mine doesn't work.
Based on the comments.
The cfn-linter was incorrectly classifying the Fn::Transform as malformed.
Deploying the stack confirmed that there are no issues with the Fn::Transform.
Here is my cloud formation template which passes event patter to the sub stack which in fact creates the rule depending on the event data.
cloudwatchRule:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "${s3Path}/cw-rule.yml"
Parameters:
eventPattern: !Join
- ' '
- - '{"source":["aws.iam"],"detail-type":["AWS API Call via CloudTrail"],"detail":{"eventSource":["iam.amazonaws.com"],"eventName":['
- Fn::Split:
- ','
- !Sub ${ssmParamWhichContainsEventNames}
- ']}}'
ruleState: "ENABLED"
#The value of ssmParamWhichContainsEventNames is of format #"CreateServiceSpecificCredential,DeactivateMFADevice"
When I run this I get the following error
Template error: every Fn::Join object requires two parameters, (1) a string delimiter and (2) a list of strings to be joined or a function that returns a list of strings (such as Fn::GetAZs) to be joined.. Rollback requested by user.
I have tried various techniques to format the order of !Join !Split !Sub
I have also tried using Fn::Join (full function format) but it keeps failing.
eventName in the eventPattern parameter expects the input in following format.
"eventName":["event1","event2","event3","event4"]
My SSM variable has event names in the format "event1,event2,event3..." To make it compatible with eventName and make the cloudwatch rule run, I'll have to transform "event1,event2,event3..." to '"event1","event2","event3"...'
One option is that I convert the SSM to my acceptable format but this is the thing I want to avoid for some reason.
Can anyone help me figure out the way to transform the "CreateServiceSpecificCredential,DeactivateMFADevice" to ' "CreateServiceSpecificCredential","DeactivateMFADevice" ' (each value enclosed within double quotes and whole string enclosed within single quotes
I keep feeling that I'm not correctly writing the intrinsic functions in the above code in the correct order.
In case somebody is still looking for how to make it happen. Here is the cloudformation template piece that split comma separated string and make it into an array of individual elements to be injected into the "eventName" attribute:
cloudwatchRule:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "${s3Path}/cw-rule.yml"
Parameters:
eventPattern: !Sub
- |
{"source":["aws.iam"],"detail-type":["AWS API Call via CloudTrail"],"detail":{"eventSource":["iam.amazonaws.com"],"eventName":[
"${eventNames}"
]}}
- eventNames: !Join
- '","'
- !Split
- ','
- !Ref ssmParamWhichContainsEventNames
ruleState: "ENABLED"
Our problem seems very basic and I would expect common.
We have tags that must always be applied (for billing). However, the tag values are only known at the time the stack is deployed... We don't know what the tag values will be when developing the stack, or when creating the product in the Service Catalog...
We don't want to wait until AFTER the resource is deployed to discover the tag is missing, so as cool as AWS config may be, we don't want to rely on its rules if we don't have to.
So things like Tag Options don't work, because it appears that they expect we know the tag value months prior to some deployment (which isn't the case.)
Is there any way to mandate tags be used for a cloudformation template when it is deployed? Better yet, can we have service catalog query for a tag value when deploying? Tags like "system" or "project", for instance, come and go over time and are not known up-front for many types of cloudformation templates we develop.
Isn't this a common scenario?
I am worried that I am missing something very, very simple and basic which mandates tags be used up-front, but I can't seem to figure out what. Thank you in advance. I really did Google a lot before asking, without finding a satisfying answer.
I don't know anything about service catalog but you can create Conditions and then use it to conditionally create (or even fail) your resource creation. Conditional Resource Creation e.g.
Parameters:
ResourceTag:
Type: String
Default: ''
Conditions:
isTagEmpty:
!Equals [!Ref ResourceTag, '']
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
Condition: isTagEmpty
Properties:
DBInstanceClass: <DB Instance Type>
Here RDS DB instance will only be created if tag is non-empty. But cloudformation will still return success.
Alternatively, you can try & fail the resource creation.
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !If [isTagEmpty, !Ref "AWS::NoValue", <DB instance type>]
I haven't tried this but it should fail as DB instance type will be invalid if tag is null.
Edit: You can also create your stack using the createStack CFN API. Write some code to read & validate the input (e.g. read from service catalog) & call the createStack API. I am doing the same from Lambda (nodejs) reading some input from Parameter Store. Sample code -
module.exports.create = async (event, context, callback) => {
let request = JSON.parse(event.body);
let subnetids = await ssm.getParameter({
Name: '/vpc/public-subnets'
}).promise();
let securitygroups = await ssm.getParameter({
Name: '/vpc/lambda-security-group'
}).promise();
let params = {
StackName: request.customerName, /* required */
Capabilities: [
'CAPABILITY_IAM',
'CAPABILITY_NAMED_IAM',
'CAPABILITY_AUTO_EXPAND',
/* more items */
],
ClientRequestToken: 'qwdfghjk3912',
EnableTerminationProtection: false,
OnFailure: request.onfailure,
Parameters: [
{
ParameterKey: "SubnetIds",
ParameterValue: subnetids.Parameter.Value,
},
{
ParameterKey: 'SecurityGroupIds',
ParameterValue: securitygroups.Parameter.Value,
},
{
ParameterKey: 'OpsPoolArnList',
ParameterValue: request.userPoolList,
},
/* more items */
],
TemplateURL: request.templateUrl,
};
cfn.config.region = request.region;
let result = await cfn.createStack(params).promise();
console.log(result);
}
Another option: add a AWS Custom Resource backed by Lambda. Check for tags in this section & return failure if it doesn't satisfy the constraints. Make all other resource creation depend on this resource (so that they all create if your checks pass). Link also contains example. You will also have to add handling for stack update & deletion (like a default success). I think this is your best bet as of now.