I need my StateMachine to evaluate a value of a value from the output of a Lambda Function. I am not able to access the value since it is nested within the JSON object. I am not able to find the right StateMachine syntax to capture the right values.
State Machine:
{
"StartAt": "Pass_1",
"States": {
"take_db_snapshot": {
"Type":"Task",
"Resource":"arn:aws:lambda:XXXXXXXXXX:function:se-db-reload",
"Next":"Pass_2"
},
"Pass_2": {
"Comment": "Pass for logging and debugging. No Action.",
"Type": "Pass",
"Next": "snapshot_waiter"
},
"snapshot_waiter": {
"Type":"Choice",
"Choices": [
{
"And": [
{
"Variable":"$$.event.db-reload-step",
"StringEquals": "create_instance"
},
{
"Variable":"$$.event.status-code",
"NumericEquals":200
}
],
"Next":"Pass_3"
},
{
"Variable":"$$.event.status-code",
"NumericEquals":408,
"Next":"snapshot_waiter"
},
{
"Variable":"$$.event.status-code",
"Comment":"Snapshot creation Error, try again",
"NumericEquals":400,
"Next":"take_db_snapshot"
}
]
}
.
.
.
This is the output from Pass_2, or the Input into the snapshot_waiter State.
{
"name": "Pass_2",
"input": {
"event": {
"db-reload-step": "take_snapshot_waiter",
"status-code": 200,
}
}
},
"inputDetails": {
"truncated": false
}
}
Related
I am using AWS step function to iterate over a list in an input document where for each iteration, I need to invoke an external service. So I want to iterate over each item and run a step using lambda:invoke.waitForTaskToken and pass the TaskToken into the execution of each iteration.
The problem I'm running into is how to use both an ItemSelector at the Map state level but also inject the TaskToken during the internal step. I need to use an ItemSelector because I want each item to also contain information from the input to Map state. The AWS Docs state:
The ItemSelector field replaces the Parameters field within the Map state. If you use the Parameters field in your Map state definitions to create custom input, we highly recommend that you replace them with ItemSelector.
But they also say:
During an execution, the context object is populated with relevant data for the Parameters field from where it is accessed. The value for a Task field is null if the Parameters field is outside of a task state.
These two statements seem to imply that what I'm trying to do is impossible.
So, what I want is something like:
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemSelector": {
"dynamic.$": "$.dynamic",
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value"
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda",
"Payload": {
"item.$": "$",
"token.$": "$$.Task.Token"
}
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
But this doesn't work because the ItemSelector overrides the Payload of the internal TestPass state. Is there a way to get this to work?
ETA: I figured I would try putting $$.Task.Token in ItemSelector just in case it would magically work but it ended up throwing an error because $$.Task does not exist in the context object at that level.
Example with this (invalid) configuration:
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemSelector": {
"dynamic.$": "$.dynamic",
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value",
"token.$": "$$.Task.Token"
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda"
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
Based on my research I don't think what I'm trying to do is possible. What I ended up implementing is a workaround. I modified the function providing input to this step function to put the dynamic info that I needed into every item in the list I am iterating over. So my step function definition now looks something like this
{
"StartAt": "ExampleMapState",
"States": {
"ExampleMapState": {
"Type": "Map",
"ItemsPath": "$.items",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "TestPass",
"States": {
"TestPass": {
"Type": "Task",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-west-2:123456789012:function:echo-lambda",
"Payload": {
"item.$": "$",
"token.$": "$$.Task.Token"
}
},
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"End": true
}
}
},
"End": true
}
}
}
And an example input to this step function looks like:
{
"dynamic": "info",
"items": [
{
"dynamic": "info",
"resize": "true",
"format": "jpg"
},
{
"dynamic": "info",
"resize": "false",
"format": "png"
},
{
"dynamic": "info",
"resize": "true",
"format": "jpg"
}
]
}
It's not great because I have to repeat info into every item ahead of time but it works.
I am trying to access item properties which I am iterating over using Map state. I keep getting this error:
Value ($['Map'].snapshot_id.S) for parameter snapshotId is invalid. Expected: 'snap-...'. (Service: Ec2, Status Code: 400, Request ID: 6fc02935-c161-49df-8606-bc6f3e2934a6)
I have gone through the docs, which seems to suggest access using $.Map.snapshot_id.S but doesn't seem to work. Meanwhile, an input item to Map is:
{
"snapshot_id": {
"S": "snap-01dd5ee46df84119e"
},
"file_type": {
"S": "bash"
},
"id": {
"S": "64e6893261d94669b7a8ca425233d68b"
},
"script_s3_link": {
"S": "df -h"
}
}
Map state definition:
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Parallel State",
"States": {
"Parallel State": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "CreateVolume",
"States": {
"CreateVolume": {
"Type": "Task",
"Parameters": {
"AvailabilityZone": "us-east-2b",
"SnapshotId": "$$.Map.snapshot_id.S"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:createVolume",
"Next": "AttachVolume"
},
"AttachVolume": {
"Type": "Task",
"Parameters": {
"Device": "MyData",
"InstanceId": "MyData",
"VolumeId": "MyData"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:attachVolume",
"End": true
}
}
}
],
"End": true
}
}
},
"Next": "Final Result",
"ItemsPath": "$.Items",
"MaxConcurrency": 40
},
TL;DR "SnapshotId.$": "$.snapshot_id"
By default, each inline map iteration's input is one item from the ItemsPath array, accessible simply as $.
Your state machine definition implies that $.Items is an array of objects with a snapshot_id key (and other keys). If so, each iteration's snapshot id is at $.snapshot_id.
Finally, adding .$ to the parameter name (SnapshotId.$) tells Step Functions the value is not a literal, but rather a path value to be substituted.
I have the flow where i want to edit the column in the csv table and replace the "," by a "."
How do I do that? Because the replace function expression in logicApp does not return the column:
It asks me to take the complete body when I use the replace function.
Where as details column is available which I want to edit:
How should I replace the "," from the details column?
I did this then, Then i don't see the variable I initialize.
For instance I've taken this as my sample .csv file which I'm retrieving from my storage account.
Firstly I have used Parse CSV file like you did the same, then initialised and used the Append the string variable connector taking the Productsname column. Lastly, have used the replace function expression to replace ' , ' with a ' . '.
NOTE: I have used '|' following productsname variable for future purpose.
Here is my Logic App workflow
THE COMPOSE CONNECTOR EXPRESSION :-
split(replace(variables('Productname'),',','.'),'|')
OUTPUT:
Here is my workflow that you can refer to:
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": "#split(replace(variables('Productname'),',','.'),'|')",
"runAfter": {
"For_each_2": [
"Succeeded"
]
},
"type": "Compose"
},
"For_each_2": {
"actions": {
"Append_to_string_variable": {
"inputs": {
"name": "Productname",
"value": "#{items('For_each_2')?['Productname']}|"
},
"runAfter": {},
"type": "AppendToStringVariable"
}
},
"foreach": "#body('Parse_CSV')",
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "Foreach"
},
"Get_blob_content_(V2)": {
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azureblob']['connectionId']"
}
},
"method": "get",
"path": "/v2/datasets/#{encodeURIComponent(encodeURIComponent('AccountNameFromSettings'))}/files/#{encodeURIComponent(encodeURIComponent('JTJmY29udGFpbmVyMjQwOCUyZlByb2R1Y3RzLmNzdg=='))}/content"
},
"metadata": {
"JTJmY29udGFpbmVyMjQwOCUyZlByb2R1Y3RzLmNzdg==": "/container2408/Products.csv"
},
"runAfter": {},
"type": "ApiConnection"
},
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "Productname",
"type": "string"
}
]
},
"runAfter": {
"Parse_CSV": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Parse_CSV": {
"inputs": {
"body": {
"content": "#{base64(body('Get_blob_content_(V2)'))}",
"headers": "Productid,Productname"
},
"host": {
"connection": {
"name": "#parameters('$connections')['plumsail']['connectionId']"
}
},
"method": "post",
"path": "/flow/v1/Documents/jobs/ParseCsv"
},
"runAfter": {
"Get_blob_content_(V2)": [
"Succeeded"
]
},
"type": "ApiConnection"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {
"$connections": {
"value": {
"azureblob": {
"connectionId": "/subscriptions/<subscription id>/resourceGroups/<Your resource group name>/providers/Microsoft.Web/connections/azureblob",
"connectionName": "azureblob",
"id": "/subscriptions/<subscription id>/providers/Microsoft.Web/locations/northcentralus/managedApis/azureblob"
},
"plumsail": {
"connectionId": "/subscriptions/<subscription id >/resourceGroups/<Your resource group name>/providers/Microsoft.Web/connections/plumsail",
"connectionName": "plumsail",
"id": "/subscriptions/<subscription id>/providers/Microsoft.Web/locations/northcentralus/managedApis/plumsail"
}
}
}
}
}
I used items function express and did it directly.
#replace(item()?['details'],',','')
This was a bit strange it didn't work at first but now it is working.
Step Function is defined like that:
{
"StartAt": "Decision_Maker",
"States": {
"Decision_Maker":{
"Type": "Choice",
"Choices": [
{
"Variable": "$.body.MyData",
"StringEquals": "null", //that doesn't work :(
"Next": "Run_Task1"
}],
"Default": "Run_Task2"
},
"Run_Task1": {
"Type": "Task",
"Resource": "url_1",
"Next": "Run_Task2"
},
"Run_Task2": {
"Type": "Task",
"Resource": "url_2",
"End": true
}
}
}
Basically it's a choice between 2 tasks.
Input data is like this:
{
"body": {
"prop1": "value1",
"myData": {
"otherProp": "value"
}
}
}
The problem is that sometimes there's no myData in JSON. So input may come like this:
{
"body": {
"prop1": "value1",
"myData": null
}
}
How do I check whether or not myData is null?
As of August 2020, Amazon States Language now has an isNull and isPresent Choice Rule. Using these you can natively check for null or the existence of a key in the state input inside a Choice state.
Example:
{ "Variable": "$.possiblyNullValue", "IsNull": true }
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-choice-state.html#amazon-states-language-choice-state-rules
The order matters. Set the "IsPresent": false first, then the "IsNull": true, finally the scalar comparison last.
"Check MyValue": {
"Comment": "Check MyValue",
"Type": "Choice",
"Default": "ContinueWithMyValue",
"Choices": [
{
"Or": [
{
"Variable": "$.MyValue",
"IsPresent": false
},
{
"Variable": "$.MyValue",
"IsNull": true
},
{
"Variable": "$.MyValue",
"BooleanEquals": false
}
],
"Next": "HaltProcessing"
},
{
"Variable": "$.MyValue",
"BooleanEquals": true,
"Next": "ContinueWithMyValue"
}
]
},
As per my experience, Choice Type can't deal nulls. Best way could be pre-processing your input using a lambda in the very first state and return the event formatting it as "null". Below code snippet might help.
def lambda_handler(event, context):
if event['body']['MyData']:
return event
else:
event['body']['MyData']="null"
return event
Note: This also handles empty string.
I am trying to set the environment variable which takes it's value at runtime through my CloudFormation template json for CustomResource. So that later it executes a python lambda and I can read the environment variable in the lambda and process some data.
I want my python lambda to be able to read this variable inside os.environ
Following is my Cloudformation for CustomResource
"TriggerRedshiftSetupLambda": {
"Type": "AWS::CloudFormation::CustomResource",
"Version": 1.0,
"Properties": {
"Environment": {
"Variables": {
"AHost": {
"Fn::GetAtt" : [ "A", "Endpoint.Address" ]
},
"APort": {
"Fn::GetAtt" : [ "A", "Endpoint.Port" ]
}
}
},
"ServiceToken": {
"Fn::GetAtt" : [ "ASetupLambda", "Arn" ]
}
}
}
Here is my lambda code using the variable
def lambda_handler(event, context):
print(os.environ)
print(os.environ['AHost'])
The 1st print statement prints the entire environment variables list but doesn't have any key / value pair for 'AHost'
Am I doing something wrong? How to initialize environment variables through customresource for lambda correctly?
Setting environment variables through the custom resource definition seems not to be supported. What you are setting is the properties section for the actual invocation (so event data).
So taking your template, your configuration should be accessible under the following path.
event['ResourceProperties']['Environment']['Variables']['AHost']
As stated by #jens above it's not possible to set environment variables under os.environ using the CustomResource CloudFormation.
Instead, the Lambda CloudFormation needs to define those values -
"RedshiftSetupLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": { "Fn::Sub": "XYZ-${Branch}" },
"S3Key": { "Fn::Sub": "XYZ-${Commit}.zip" }
},
"Description": "Setup Lambda",
"FunctionName": { "Fn::Sub": "${BlockId}-setup-${Branch}" },
"Handler": "setup.lambda_handler",
"KmsKeyArn": {
"Fn::ImportValue": {
"Fn::Sub": "${BlockId}-Common-RolesKeys-${Branch}-KMSKeyArn"
}
},
"Role": {
"Fn::ImportValue": {
"Fn::Sub": "${BlockId}-Common-RolesKeys-${Branch}-LambdaIAMRoleArn"
}
},
"Runtime": "python2.7",
"Timeout": 30,
"VpcConfig": {
"SecurityGroupIds": [ {"Ref": "SecurityGroup"} ],
"SubnetIds": [
{ "Fn::ImportValue": "VPCCreate-PrivateSubnet1Id" },
{ "Fn::ImportValue": "VPCCreate-PrivateSubnet2Id" },
{ "Fn::ImportValue": "VPCCreate-PrivateSubnet3Id" }
]
},
"Environment": {
"Variables": {
"DB_USERNAME": {
"Ref": "MasterUsername"
},
"AHOST": {
"Fn::GetAtt": ["RedshiftCluster", "Endpoint.Address"]
},
"APORT": {
"Fn::GetAtt": ["RedshiftCluster", "Endpoint.Port"]
},
"CLUSTER_IDENTIFIER": {
"Ref": "RedshiftCluster"
}
}
}
}
}
They can be accessed this way:
print(os.environ['AHOST'])