AWS Stepfunction, ValidationException - amazon-web-services

i got the error "The provided key element does not match the schema", while getting data from AWS dynamoDB using stepfunction.
stepfunction Defination
{
"Comment": "This is your state machine",
"StartAt": "Choice",
"States": {
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.data.Type",
"StringEquals": "GET",
"Next": "DynamoDB GetItem"
},
{
"Variable": "$.data.Type",
"StringEquals": "PUT",
"Next": "DynamoDB PutItem"
}
]
},
"DynamoDB GetItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "KeshavDev",
"Key": {
"Email": {
"S": "$.Email"
}
}
},
"End": true
},
"DynamoDB PutItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "KeshavDev",
"Item": {
"City": {
"S.$": "$.City"
},
"Email": {
"S.$": "$.Email"
},
"Address": {
"S.$": "$.Address"
}
}
},
"InputPath": "$.data",
"End": true
}
}
}
Input
{
"data": {
"Type": "GET",
"Email": "demo#gmail.com"
}
}
Error
{ "resourceType": "dynamodb", "resource": "getItem", "error":
"DynamoDB.AmazonDynamoDBException", "cause": "The provided key
element does not match the schema (Service: AmazonDynamoDBv2; Status
Code: 400; Error Code: ValidationException; Request ID:
a78c3d7a-ca3f-4483-b986-1735201d4ef2; Proxy: null)" }

I see some potential issues with the getItem task when compared to AWS documentation.
I think the Key field needs to be S.$ similar to what you have in your putItem task.
There is no ResultPath attribute to tell the state machine where to put the results.
Your path may not be correct, try $.data.Email
"DynamoDB GetItem": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "KeshavDev",
"Key": {
"Email": {
"S.$": "$.data.Email"
}
}
},
"ResultPath": "$.DynamoDB",
"End": true
},
To be honest, I'm not sure if one of all of these are contributing to the validation error those are some things to experiment with.
On another note, there are some open source validators for Amazon State Language but for this case, they were not very helpful and said that your code was valid.

its working, above JD D mentoned steps and also by adding both key in step function definition.
DynamoDb have two key.
primary partition key
primary sort key

Related

Access Map State ( item ) in Step functions

I am trying to access item properties which I am iterating over using Map state. I keep getting this error:
Value ($['Map'].snapshot_id.S) for parameter snapshotId is invalid. Expected: 'snap-...'. (Service: Ec2, Status Code: 400, Request ID: 6fc02935-c161-49df-8606-bc6f3e2934a6)
I have gone through the docs, which seems to suggest access using $.Map.snapshot_id.S but doesn't seem to work. Meanwhile, an input item to Map is:
{
"snapshot_id": {
"S": "snap-01dd5ee46df84119e"
},
"file_type": {
"S": "bash"
},
"id": {
"S": "64e6893261d94669b7a8ca425233d68b"
},
"script_s3_link": {
"S": "df -h"
}
}
Map state definition:
"Map": {
"Type": "Map",
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "INLINE"
},
"StartAt": "Parallel State",
"States": {
"Parallel State": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "CreateVolume",
"States": {
"CreateVolume": {
"Type": "Task",
"Parameters": {
"AvailabilityZone": "us-east-2b",
"SnapshotId": "$$.Map.snapshot_id.S"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:createVolume",
"Next": "AttachVolume"
},
"AttachVolume": {
"Type": "Task",
"Parameters": {
"Device": "MyData",
"InstanceId": "MyData",
"VolumeId": "MyData"
},
"Resource": "arn:aws:states:::aws-sdk:ec2:attachVolume",
"End": true
}
}
}
],
"End": true
}
}
},
"Next": "Final Result",
"ItemsPath": "$.Items",
"MaxConcurrency": 40
},
TL;DR "SnapshotId.$": "$.snapshot_id"
By default, each inline map iteration's input is one item from the ItemsPath array, accessible simply as $.
Your state machine definition implies that $.Items is an array of objects with a snapshot_id key (and other keys). If so, each iteration's snapshot id is at $.snapshot_id.
Finally, adding .$ to the parameter name (SnapshotId.$) tells Step Functions the value is not a literal, but rather a path value to be substituted.

Error in using InputPath to select parts of input in a Step Functions workflow

I am creating a Step Functions workflow which has various steps. I am referring to this topic in their documentation InputPath, ResultPath and OutputPath Examples. I am trying to check the identity and address of a person in my workflow as they've shown in their document. I'm passing the input for the Verify identity step within the state machine definition inside Parameters. My workflow looks like this.
Note: But when I run this, am getting the error -> An error occurred while executing the state 'Verify identity' (entered at the event id #19). Invalid path '$.identity' : Property ['identity'] not found in path $
What am I doing wrong here? Can someone please explain?
Thanks..
{
"StartAt": "Step1",
"States": {
"Step1": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
...something...
},
"Next": "Step2"
},
"Step2": {
"Type": "Choice",
"Choices": [
Do something...
],
"Default": "Step3.1"
},
"Step3.1": {
"Type": "Task",
...something...
}
},
"Next": "Step3.3"
},
...something...,
"Step4": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "Verify identity",
"States": {
"Verify identity": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"InputPath": "$.identity",
"Parameters": {
"Payload": {
"identity": {
"email": "jdoe#example.com",
"ssn": "123-45-6789"
},
"firstName": "Jane",
"lastName": "Doe"
},
"FunctionName": "{Lambda ARN}"
},
"End": true
}
}
},
{
"StartAt": "Verify address",
"States": {
"Verify address": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"Payload": {
"street": "123 Main St",
"city": "Columbus",
"state": "OH",
"zip": "43219"
},
"FunctionName": "{Lambda ARN}"
},
"End": true
}
}
}
],
"Next": "Step5"
},
"Step5": {
"Type": "Task",
"Parameters": {
something...
},
"End": true
}
}```
You don't have an explicit transition in your example to call Step4 but assuming the order you have defined (step1 -> step2 -> step3.1 -> step3.3 -> step4)
This means the output from step3.3 should be something like
{
"cat": "meow",
"dog": "woof",
"identity": { // this is whats missing
"email": "jdoe#example.com",
"ssn": "123-45-6789"
}
}
this is what will get passed to each branch of your parallel state (Step4)
However, since you have anInputPath defined for Step4."Verify identity", the effective input to the task becomes
{
"email": "jdoe#example.com",
"ssn": "123-45-6789"
}
The error youre seeing
An error occurred while executing the state 'Verify identity' (entered at the event id #19). Invalid path '$.identity' : Property ['identity'] not found in path $
means the "identity" key (aka $.identity) isn't getting added to the output of Step3.3 (aka $)

Step function with Redshift cluster

Building a step function to orchestrate an ETL pipeline but keep getting this error. Here is my code and following below AWS docs.
https://docs.aws.amazon.com/step-functions/latest/dg/sample-etl-orchestration.html
"GetStateOfCluster": {
"Type": "Task",
"Resource": "lambda,
"TimeoutSeconds": 180,
"HeartbeatSeconds": 60,
"Next": "IsClusterAvailable",
"InputPath": "$",
"ResultPath": "$.clusterStatus"
},
"IsClusterAvailable": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.clusterStatus",
"StringEquals": "available",
"Next": "runetljobs"
},
{
"Variable": "$.clusterStatus",
"StringEquals": "unavailable",
"Next": "ClusterUnavailable"
},
{
"Variable": "$.clusterStatus",
"StringEquals": "paused",
"Next": "InitializeResumeCluster"
},
{
"Variable": "$.clusterStatus",
"StringEquals": "resuming",
"Next": "ClusterWait"
}
],
"Default": "DefaultState"
},
"DefaultState": {
"Type": "Fail",
"Error": "DefaultStateError",
"Cause": "No Matches!"
},
"ClusterUnavailable": {
"Type": "Fail",
"Cause": "Redshift cluster is not available",
"Error": "Error"
},
"ClusterWait": {
"Type": "Wait",
"Seconds": 900,
"Next": "InitializeCheckCluster"
},
"InitializeResumeCluster": {
"Type": "Pass",
"Next": "ResumeCluster",
"Result": {
"input": {
"redshift_cluster_id": "redshift cluster id",
"operation": "resume"
}
}
},
"ResumeCluster": {
"Type": "Task",
"Resource": "lambda",
"TimeoutSeconds": 180,
"HeartbeatSeconds": 60,
"Next": "ClusterWait",
"InputPath": "$",
"ResultPath": "$"
},
It's directly going to default even cluster status shows 'available', rather it should go to runetljob stage. In the doc, they dont have default, if we dont add default, error is,
"cause": "An error occurred while executing the state 'IsClusterAvailable' (entered at the event id #14). Failed to transition out of the state. The state does not point to a next state."
You don't see the state "runetljobs" defined in you state definition.

Workflow has no terminal state

I am creating a workflow with AWS Step Function where I am first checking if a record exists in database, then based on the records there are two branches and each of them end at either Succeed or Failed state, but I am still getting Workflow has no end state error.
Following is the JSON for workflow
{
"Comment": "A demo state machine",
"StartAt": "FindCategory",
"States": {
"FindCategory": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxx:function:xxxxxx",
"Next": "Exists?"
},
"Exists?": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.exists",
"BooleanEquals": true,
"Next": "Yes"
},
{
"Variable": "$.exists",
"BooleanEquals": false,
"Next": "No"
}
]
},
"Yes": {
"Type": "Pass",
"Next": "GetQuestions"
},
"GetQuestions": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxxxx",
"Next": "ReplyWithPolls"
},
"ReplyWithPolls": {
"Type": "Map",
"MaxConcurrency": 2,
"Iterator": {
"StartAt": "SendPoll",
"States": {
"SendPoll": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx:xxxxxxx",
"Next": "SendPoll"
}
}
},
"Next": "Succeed"
},
"No": {
"Type": "Pass",
"Next": "FailState"
},
"Succeed": {
"Type": "Succeed"
},
"FailState": {
"Type": "Fail",
"Error": "404",
"Cause": "Category not found"
}
}
}
I believe the problem is that your SendPoll state results in an infinite loop. It references itself as next. Instead, the state in the iterator should be a terminal state.
Replace the "Next" field in "SendPoll" state with an "End" field.
"SendPoll": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx:xxxxxxx",
"End": true
}

AWS State Machine ASL: Use the Result Selector only if data is returned

I am trying to configure a task state to find an organization, if the organization is found the Result Selector will select the name, id, and createTime. This will then be added back to the ticket under the org node (ResultPath).
The issue I am encountering is that if the organization isn't found then the state machine execution will be cancelled because the Result Selector is attempting to select nodes that do not exist in the result. Does anyone know if there is a way to use the Result Selector only if data is returned? I want the state machine to continue even if the org isn't found.
"Find Organization": {
"Type": "Task",
"Resource": "${aws_lambda_function.test_function.arn}",
"Next": "Find User Account",
"Parameters": {
"name": "FindOrganization",
"request.$": "$.item"
},
"ResultSelector": {
"name.$": "$.name",
"id.$": "$.id",
"createTime.$": "$.createTime"
},
"ResultPath": "$.org",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error",
"Next": "Publish SNS Failure"
}
]
}
This is exactly when Choice State comes into play.
An example modification to your original ASL definition would be
{
"Comment": "An example of the Amazon States Language using a choice state.",
"StartAt": "Find Organization",
"States": {
"Find Organization": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Next": "Check If Organization Exists",
"Parameters": {
"name": "FindOrganization",
"request.$": "$.item"
},
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.error",
"Next": "Publish SNS Failure"
}
]
},
"Check If Organization Exists": {
"Type": "Choice",
"Choices": [
{
"Variable": "$",
"IsNull": false,
"Next": "Organization Exists"
},
{
"Variable": "$",
"IsNull": true,
"Next": "Organization Not Exists"
}
],
"Default": "Organization Exists"
},
"Organization Exists": {
"Type": "Pass",
"Parameters": {
"name.$": "$.name",
"id.$": "$.id",
"createTime.$": "$.createTime"
},
"ResultPath": "$.org",
"Next": "Find User Account"
},
"Organization Not Exists": {
"Type": "Fail"
},
"Find User Account": {
"Type": "Succeed"
},
"Publish SNS Failure": {
"Type": "Fail"
}
}
}