AWS Step Functions: Input Parameter - amazon-web-services

I have a very simple workflow using Fargate containers. The containers simply return inputs.
Workflow Input is:
{
"value": "FALSE"
}
Choice operates as expected. $.value is processed as FALSE and choice runs the ECS RunTask-FALSE container.
The container should receive and return the string "FALSE", however it returns the string "*.value". For some reason the Input value is not being parsed. When I look at the Task run in ECS portal I see:
How can I pass the starting parameter to this Task ?
{
"Comment": "A description of my state machine",
"StartAt": "Choice",
"States": {
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.value",
"StringEquals": "FALSE",
"Next": "ECS RunTask-FALSE"
}
],
"Default": "ECS RunTask-TRUE"
},
"ECS RunTask-FALSE": {
"Type": "Task",
"Resource": "arn:aws:states:::ecs:runTask.sync",
"Parameters": {
"LaunchType": "FARGATE",
"Cluster": "arn:aws:ecs:us-east-2:xxxxxxxxxxx:cluster/portal",
"TaskDefinition": "arn:aws:ecs:us-east-2:xxxxxxxxxxx:task-definition/simple:4",
"Overrides": {
"ContainerOverrides": [
{
"Name": "simple",
"Command": ["$.value"]
}
]
},
"NetworkConfiguration": {
"AwsvpcConfiguration": {
"Subnets": [
"subnet-001f85595e8af43cf",
"subnet-05e742358ac59ae04"
],
"SecurityGroups": [
"sg-0948e5328861ae667"
],
"AssignPublicIp": "ENABLED"
}
}
},
"End": true
},
"ECS RunTask-TRUE": {
"Type": "Task",
"Resource": "arn:aws:states:::ecs:runTask.sync",
"Parameters": {
"LaunchType": "FARGATE",
"Cluster": "arn:aws:ecs:us-east-2:xxxxxxxxxxx:cluster/portal",
"TaskDefinition": "arn:aws:ecs:us-east-2:xxxxxxxxxxx:task-definition/simple:4",
"NetworkConfiguration": {
"AwsvpcConfiguration": {
"Subnets": [
"subnet-001f85595e8af43cf",
"subnet-05e742358ac59ae04"
],
"SecurityGroups": [
"sg-0948e5328861ae667"
],
"AssignPublicIp": "ENABLED"
}
}
},
"End": true
}
}
}

"Command.$": "States.Array($.value)"
Parameters: For key-value pairs where the value is selected using a path, the key name must end in .$.
States.Array Intrinsic Function: The interpreter returns a JSON array containing the values of the arguments in the order provided.

Related

Map executing another step function does not pass any input

Can't figure this one out : the step function executed inside a map ALWAYS has an empty dict in input. I need the input to be {"id": "xxxx"}
Here is the map element :
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "InsideStep",
"States": {
"InsideStep": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution.sync:2",
"Parameters": {
"StateMachineArn": "arn:aws:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"End": true
}
}
},
"Next": "Consolidate logs",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "Log error",
"ResultPath": "$.error"
}
],
"ItemsPath": "$.ids_list",
"MaxConcurrency": 10,
"Parameters": {
"id.$": "$$.Map.Item.Value"
}
}
Thanks for any help
Here is the working solution, thanks to luk2302 suggestions
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "InsideStep",
"States": {
"InsideStep": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution.sync:2",
"Parameters": {
"StateMachineArn": "arn:aws:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"Input": {
"id.$": "$"
}
},
"End": true
}
}
},
"Next": "Consolidate logs",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "Log error",
"ResultPath": "$.error"
}
],
"ItemsPath": "$.ids_list",
"MaxConcurrency": 10
}

AWS Step and Batch Dynamic Command

I have a batch Job with a single Job Definition that executes depending on a parameter on the environment command option.
The original value is "--param2=XXX" but I need this to be dynamic according to the Input parameters of the Step Functions.
{
"param2": "--param2=YYY"
}
I haven't been able to replace the value in the Step Function with the Input Value
{
"Step1": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:us-east-2:zzzzzzzzz:job-definition/XXXXXX",
"JobQueue": "arn:aws:batch:us-east-2:zzzzzzzz:job-queue/YYYYYY",
"JobName": "Step1",
"ContainerOverrides": {
"Environment": [
{
"Name": "envparam",
"Value": "0"
}
],
"Command": [
"python",
"run.py",
"--param=val",
"$.param2"
]
}
},
"Next": "Step2"
}
}
I found a solution adding a Parameter to Batch and it is referenced using Ref::Param2
This is the complete code
{
"Step1": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobDefinition": "arn:aws:batch:us-east-2:zzzzzzzzz:job-definition/XXXXXX",
"JobQueue": "arn:aws:batch:us-east-2:zzzzzzzz:job-queue/YYYYYY",
"JobName": "Step1",
"Parameters": {
"Param2.$": "$.param2"
},
"ContainerOverrides": {
"Environment": [
{
"Name": "envparam",
"Value": "0"
}
],
"Command": [
"python",
"run.py",
"--param=val",
"Ref::Param2"
]
}
},
"Next": "Step2"
}
}

Configure the LoadBalancer in AWS CloudWatch Alarm

I have a web application on AWS and I am trying to configure my autoscaling based on the requests.
My AppLoadBalancer resource is as below:
"AppLoadBalancer": {
"Properties": {
"LoadBalancerAttributes": [
{
"Key": "idle_timeout.timeout_seconds",
"Value": "60"
}
],
"Name": "sample-app-v1",
"Scheme": "internet-facing",
"SecurityGroups": [
"sg-1abcd234"
],
"Subnets": {
"Fn::FindInMap": [
"LoadBalancerSubnets",
{
"Ref": "AWS::Region"
},
"Subnets"
]
},
"Tags": [
{
"Key": "Name",
"Value": "sample-app-v1"
},
{
"Key": "StackName",
"Value": "sample-app"
},
{
"Key": "StackVersion",
"Value": "v1"
}
]
},
"Type": "AWS::ElasticLoadBalancingV2::LoadBalancer"
}
I am trying to configure a CloudWatch Alarm like this:
"RequestCountTooHighAlarm": {
"Properties": {
"AlarmActions": [
{
"Ref": "ScaleUp"
}
],
"AlarmDescription": "Scale-up if request count >= 8000 for last 5 minute",
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"Dimensions": [
{
"Name": "LoadBalancer",
"Value": [
{
"Fn::GetAtt": [
"AppLoadBalancer",
"LoadBalancerFullName"
]
}
]
}
],
"EvaluationPeriods": 1,
"MetricName": "RequestCount",
"Namespace": "AWS/ApplicationELB",
"OKActions": [
{
"Ref": "ScaleDown"
}
],
"Period": 300,
"Statistic": "SampleCount",
"Threshold": 8000
},
"Type": "AWS::CloudWatch::Alarm"
}
However, my stack continues to fail and I don't know what is wrong here. Here is the error which I am getting.
ERROR: RequestCountTooHighAlarm CREATE_FAILED: Value of property Value must be of type String
ERROR: sample-app-v1 CREATE_FAILED: The following resource(s) failed to create: [RequestCountTooHighAlarm].
Can somebody suggest?
The property mentioned requires a string. You have it defined as a list:
"Value": [
{
"Fn::GetAtt": [
"AppLoadBalancer",
"LoadBalancerFullName"
]
} ]
The [] brackets defines a list in JSON. Remove the outside brackets in the Value value, and use only the Fn::GetAt portion. That call will return a string.

Cloud Formation: S3 linked to Lambda gives The ARN is not well formed

I'm trying to use CloudFormation to deploy an S3 bucket that on ObjectCreate invokes a Lambda function.
Here are my resources:
"ExampleFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "index.lambda_handler",
"Code": {
"S3Bucket": "bucketname",
"S3Key": "something.zip"
},
"Runtime": "python3.6",
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
}
}
},
"InputDataBucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": "input-data",
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Function": {
"Ref": "ExampleFunction"
},
"Event": "s3:ObjectCreated:*",
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": "zip"
}
]
}
}
}
]
}
}
},
"LambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"ExampleFunction",
"Arn"
]
},
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
},
"SourceArn": {
"Fn::Join": [
":",
[
"arn",
"aws",
"s3",
"",
"",
{
"Ref": "InputDataBucket"
}
]
]
}
}
}
I've tried to follow the documentation of the Notification Configuration, that says that there can be a circular dependency. However, if I follow the instructions I get the same error. Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-notificationconfig.html
When I try to create the stack, the S3 always breaks it with error "The ARN is not well formed"
I've tried many things, but I always receive this same error.
I can get this to work as long as I know the S3 bucket name in advance (mybucketname below). If you don't know the bucket name in advance, then you can enhance this to request the bucket name as a stack parameter and it should still work. If you need the bucket name to be auto-generated (so you can't predict the name in advance) then this will not work and you'll have to go the create/update route.
Key thing here is to manually create the S3 bucket ARN from the known bucket name, rather than relying on "Ref": "InputDataBucket" to get the bucket name for you.
Also worth reading this support article.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "stackoverflow-48037497",
"Resources" : {
"ExampleFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "index.lambda_handler",
"Code": {
"S3Bucket": "bucketname",
"S3Key": "something.zip"
},
"Runtime": "python3.6",
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
}
}
},
"LambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"DependsOn": [ "ExampleFunction" ],
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"ExampleFunction",
"Arn"
]
},
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
},
"SourceArn": "arn:aws:s3:::mybucketname"
}
},
"InputDataBucket": {
"Type": "AWS::S3::Bucket",
"DependsOn": [ "ExampleFunction", "LambdaInvokePermission" ],
"Properties": {
"BucketName": "mybucketname",
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Function": { "Fn::GetAtt" : [ "ExampleFunction", "Arn" ] },
"Event": "s3:ObjectCreated:*"
}
]
}
}
}
}
}

Setup Lambda to trigger from CloudWatch using CloudFormation

I want to use CloudFormation to trigger Lambda when my CloudWatch function is called. I have the below, but it does not work.
CloudWatch rule created fine
"CloudWatchNewEc2": {
"Type": "AWS::Events::Rule",
"DependsOn": ["LambdaNewEc2"],
"Properties": {
"Description": "Triggered on new EC2 instances",
"EventPattern": {
"source": [
"aws.ec2"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RunInstances"
]
}
},
"Targets": [
{
"Arn": {
"Fn::GetAtt": ["LambdaNewEc2", "Arn"]
},
"Id": "NewEc2AutoTag"
}
]
}
},
Lambda created but is not triggered
"LambdaNewEc2": {
"Type": "AWS::Lambda::Function",
"DependsOn": ["S3Lambda", "IAMRoleLambda"],
"Properties": {
"Code": {
"S3Bucket": {"Ref": "LambdaBucketName"},
"S3Key": "skynet-lambda.zip"
},
"Description": "When new EC2 instances are created, auto tag them",
"FunctionName": "newEc2AutoTag",
"Handler": "index.newEc2_autoTag",
"Role": {"Fn::GetAtt": ["IAMRoleLambda", "Arn"]},
"Runtime": "nodejs6.10",
"Timeout": "30"
}
}
},
It seems like CloudWatch Target is not sufficient?
UPDATE (Full CloudFormation template)
{
"Parameters": {
"Environment": {
"Type": "String",
"Default": "Staging",
"AllowedValues": [
"Testing",
"Staging",
"Production"
],
"Description": "Environment name"
},
"BucketName": {
"Type": "String",
"Default": "skynet-staging",
"Description": "Bucket Name"
},
"LambdaBucketName": {
"Type": "String",
"Default": "skynet-lambda",
"Description": "Lambda Bucket Name"
},
"Owner": {
"Type": "String",
"Description": "Owner"
}
},
"Resources": {
"S3Web": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Ref": "BucketName"
},
"WebsiteConfiguration": {
"IndexDocument": "index.html",
"RoutingRules": [
{
"RedirectRule": {
"ReplaceKeyPrefixWith": "#"
},
"RoutingRuleCondition": {
"HttpErrorCodeReturnedEquals": "404"
}
}
]
},
"AccessControl": "PublicRead",
"Tags": [
{
"Key": "Cost Center",
"Value": "Skynet"
},
{
"Key": "Environment",
"Value": {
"Ref": "Environment"
}
},
{
"Key": "Owner",
"Value": {
"Ref": "Owner"
}
}
]
}
},
"S3Lambda": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Ref": "LambdaBucketName"
},
"VersioningConfiguration": {
"Status": "Enabled"
},
"Tags": [
{
"Key": "Cost Center",
"Value": "Skynet"
},
{
"Key": "Owner",
"Value": {
"Ref": "Owner"
}
}
]
}
},
"CloudWatchNewEc2": {
"Type": "AWS::Events::Rule",
"DependsOn": ["LambdaNewEc2"],
"Properties": {
"Description": "Triggered on new EC2 instances",
"EventPattern": {
"source": [
"aws.ec2"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RunInstances"
]
}
},
"Targets": [
{
"Arn": {
"Fn::GetAtt": ["LambdaNewEc2", "Arn"]
},
"Id": "NewEc2AutoTag"
}
]
}
},
"IAMRoleLambda": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "skynet-lambda-role",
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [ "lambda.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/AmazonEC2FullAccess",
"arn:aws:iam::aws:policy/AWSLambdaFullAccess",
"arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess",
"arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
]
}
},
"LambdaNewEc2": {
"Type": "AWS::Lambda::Function",
"DependsOn": ["S3Lambda", "IAMRoleLambda"],
"Properties": {
"Code": {
"S3Bucket": {"Ref": "LambdaBucketName"},
"S3Key": "skynet-lambda.zip"
},
"Description": "When new EC2 instances are created, auto tag them",
"FunctionName": "newEc2AutoTag",
"Handler": "index.newEc2_autoTag",
"Role": {"Fn::GetAtt": ["IAMRoleLambda", "Arn"]},
"Runtime": "nodejs6.10",
"Timeout": "30"
}
}
},
"Outputs": {
"WebUrl": {
"Value": {
"Fn::GetAtt": [
"S3Web",
"WebsiteURL"
]
},
"Description": "S3 bucket for web files"
}
}
}
I managed to deploy your template into a CloudFormation stack (by removing the LambdaBucket and pointing to my own zip file). It seems to create all resource correctly.
It took about 10 minutes for the RunInstances event to appear in CloudTrail. It then successfully triggered the Rule, but the CloudWatch metrics for my rule showed a failed invocation because I faked a Lambda function for your template.
Once I edited the rule to point to a better function and re-tested, it worked fine.
Bottom line: Seems to work!