I have this dynamodb:Query in my step function:
{
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:dynamodb:query",
"Next": "If nothing returned by query Or Study not yet Zipped",
"Parameters": {
"TableName": "TEST-StudyProcessingTable",
"ScanIndexForward": false,
"Limit": 1,
"KeyConditionExpression": "OrderID = :OrderID",
"FilterExpression": "StudyID = :StudyID",
"ExpressionAttributeValues": {
":OrderID": {
"S.$": "$.body.order_id"
},
":StudyID": {
"S.$": "$.body.study_id"
}
}
},
"ResultPath": "$.processed_files"
}
The results comes in as an array called Items which is nested under my ResultPath
processed_files.Items:
{
"body": {
"order_id": "1001",
"study_id": "1"
},
"processed_files": {
"Count": 1,
"Items": [
{
"Status": {
"S": "unzipped"
},
"StudyID": {
"S": "1"
},
"ZipFileS3Key": {
"S": "path/to/the/file"
},
"UploadSet": {
"S": "4"
},
"OrderID": {
"S": "1001"
},
"UploadSet#StudyID": {
"S": "4#1"
}
}
],
"LastEvaluatedKey": {
"OrderID": {
"S": "1001"
},
"UploadSet#StudyID": {
"S": "4#1"
}
},
"ScannedCount": 1
}
}
My question is how do i access the items inside this array from a choice state in a step function?
I need to query then decide something based on the results by checking the item in a condition in a choice state.
The problem is that since this is an array I can't access it using regular JsonPath (like with Items.item), and in my next step the choice condition does NOT accept an index like processed_files.Items['0'].Status
Ok so the answer was so simple all you need to do is use a number instead of string for the array index like this.
processed_files.Items[0].Status
I was originally mislead by an error I received which said that it expected a ' or '[' after the first '['. I mistakenly thought this meant it only accepts strings.
I was wrong, it works like any other array.
I hope this helps somebody one day.
Related
I have 2 tables, with 1 matching data in which I want to utilize that as a matching field and retrieve some data.
First table is this:
{
"mainFieldName": {
"S": "someString"
},
"fieldA": {
"L": [
{
"M": {
"AccountId": {
"S": "12345"
},
"PrincipalId": {
"S": "randomIdString"
},
"PrincipalType": {
"S": "GROUP"
}
}
},
{
"M": {
"AccountId": {
"S": "12345"
},
"PrincipalId": {
"S": "secondRandomString"
},
"PrincipalType": {
"S": "GROUP"
}
}
}
]
},
"fieldC": {
"L": [
{
"M": {
"name": {
"S": "xxx"
},
"final_json": {
"S": "some json data"
}
}
}
]
}
}
Second table:
{
"userId": {
"S": "randomString"
},
"group": {
"L": [
{
"M": {
"GroupId": {
"S": "randomGroupId"
}
}
}
]
}
}
I want to find the matched field for first table's fieldA.PrincipalId and second table's group.GroupId, if match, returning data is first table's fieldC.final_json
My params i tried is this, it's executed successfully but no results returned. I have confirmed there should be some matched inputs.
response = table1.scan(
TableName=TABLE_1,
FilterExpression="#gid.#pid = :id",
ExpressionAttributeValues={
':id': {'S': groupId}
},
ExpressionAttributeNames={
'#gid': 'groupId',
'#pid': 'PrincipalId'
}
)
It always return empty results
I managed to find a resolution to this. To simplify, I changed to a flatter table structure by pre-processing the Json to appending to a list.
My first table becomes:
{
"id": {
"S": "randomString"
},
"fieldA": {
"S": "randomString"
},
"table_1_groupId": {
"L": [
{
"S": "randomGroupIdString"
}
]
},
"fieldB": {
"S": "asdfsafd"
},
"fieldC": {
"L": [
{
"M": {
"name": {
"S": "randomString"
},
"jsonData": {
"S": "randomJsonData"
},
"type": {
"S": "type_a"
}
}
}
]
}
}
Second table stays the same.
With that i am able to use DynamoDB query which is more efficient as well, with FilterExpressions
My query is:
response = table1.query(
TableName=TABLE_1,
KeyConditionExpression="id = :id",
FilterExpression="contains(groupId, :gid)",
ExpressionAttributeValues={
':id': defaultId,
':gid': groupId
},
)
My output returns list of all data (since I haven't added any filter to the output) once they have the field 'GroupId' in Table2 match with table_1_groupId in table 1
I do have an Step Function A - which executes a lambda and pull some results.
Same Function has a Map Stage which is iterating over results and should call another Step Function from Map State.
While calling another Step Function B from the map state i am not able to pass the parameter or that one record as Input to Step Function B.
Please suggest how can i use Input for second step function.
Below is the example i am using , orderServiceResponse has a List of orders which I need to iterate and pass that one order to next step function.
"Validate-All" : {
"Type" : "Map",
"InputPath" : "$.orderServiceResponse",
"ItemsPath" : "$.orders",
"MaxConcurrency" : 5,
"ResultPath" : "$.orders",
"Iterator" : {
"StartAt" : "Validate" ,
"States" :{
"Validate" : {
"Type" : "Task"
"Resource" : "arn:aws:states:::states:startExecution.sync:2",
"Parameters" {
"Input" : "$.orders",
"StateMachineArn" : "{arn of Step Function B }
},
"End" : true
}
}
TL;DR Use Parameters with Map Context to add the full input object to each Map element iteration.
You have an array of data you want to process elementwise in a Map State. By default, Map only passes
the array element's data to the map iterator. But we can add additional context to each iteration.
Here is an example - the important bits are commented:
{
"StartAt": "MapState",
"States": {
"MapState": {
"Type": "Map",
"ResultPath": "$.MapResult",
"Next": "Success",
// the map's elements of each get the following:
"Parameters": {
"Index.$": "$$.Map.Item.Index", // the array element's data (we only get this by default)
"Order.$": "$$.Map.Item.Value", // the array element's index 0,1,2...
"FullInput.$": "$" // a copy of the the full input object <-- this is what you were looking for
},
"Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "Fail" }],
// substitute your iterator:
"Iterator": {
"StartAt": "MockTask",
"States": {
"MockTask": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxxxx",
"Parameters": {
"expression": "`Order ${$.Order.OrderID} was ordered by ${$.FullInput.CustomerName}`",
"expressionAttributeValues": {
"$.Order.OrderID.$": "$.Order.OrderID",
"$.FullInput.CustomerName.$": "$.FullInput.CustomerName"
}
}
}
}
},
"ItemsPath": "$.Orders"
},
"Success": { "Type": "Succeed" },
"Fail": { "Type": "Fail" }
}
}
Execution Input, 3 Orders:
{
"CustomerID": 1,
"CustomerName": "Morgan Freeman",
"OtherInfo": { "Foo": "Bar" },
"Orders": [{ "OrderID": "A", "Status": "Fulfilled" }, { "OrderID": "B", "Status": "Pending" }, { "OrderID": "C", "Status": "Cancelled" }]
}
Map Iteration 0 Input:
{
"Order": { "OrderID": "A", "Status": "Fulfilled" },
"Index": 0,
"FullInput": { "CustomerID": 1, "CustomerName": "Morgan Freeman", "OtherInfo": { "Foo": "Bar" }, "Orders": [{...
Execution Output MapResult key
{
"MapResult": [
"Order A was ordered by Morgan Freeman",
"Order B was ordered by Morgan Freeman",
"Order C was ordered by Morgan Freeman"
]
...
}
In one of my AWS DynamoDB tables I have several items that require an update within in a pipeline.
The following json shows an item in the AWS DynamoDB table:
{
"DeploymentConfigs": {
"L": [
{
"M": {
"Config": {
"M": {
"Name": {
"S": "batch-komo"
},
"Replicas": {
"N": "3"
}
}
}
}
},
{
"M": {
"Config": {
"M": {
"Name": {
"S": "online-komo"
},
"Replicas": {
"N": "3"
}
}
}
}
}
]
},
"environment": {
"S": "komo-claimcenter"
}
}
How can update an object in DeploymentConfigs?
Primary partition key is environment.
E.g. the object
{
"M": {
"Config": {
"M": {
"Name": {
"S": "batch-komo"
},
"Replicas": {
"N": "3"
}
}
}
}
}
shall be updated to
{
"M": {
"Config": {
"M": {
"Name": {
"S": "batch-komo"
},
"Replicas": {
"N": "5"
}
}
}
}
}
I do not know to achieve in the AWS CLI.
To update a nested part inside an item, you use an UpdateExpression with an attribute path. For example, SET DeploymentConfigs[0].Config.Replicas = :val.
The problem, however, is that your top-level attribute DeploymentConfigs is a list, so to modify one of its items, you need to know in index (in this example 0). If you don't know the index you have a problem - there is no way to "modify the list item which has name='batch-komo'" or something like that... If you need to do something like this, you have to read the entire top-level attribute, modify it in the client, and write it (or just the small change, now that you know the index) back.
I have a step where I want to update a object on a DynamoDB table.
Everything works except its creating a new object with the ID value of "$.id", instead of updating where the ID I pass in.
This is my first state machine attempt so what have I done wrong here?
"update-table-processing": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:updateItem",
"ResultPath": "$.updateResult",
"Parameters": {
"TableName": "Projects",
"Key": {
"id": {
"S": "$.id"
}
},
"UpdateExpression": "SET step = :updateRef",
"ExpressionAttributeValues": {
":updateRef": {
"S": "processing"
}
},
"ReturnValues": "ALL_NEW"
},
"Next": "create-project"
},
Do I somehow need to tell DynamoDB to evaluate "$.id" rather than treating it as a "S", or is this happening because I've not mapped the input correctly that the "$.id" value is empty?
My input looks like:
{
"id": "f8185735-c90d-4d4e-8689-cec68a48b1bc"
}
In order to specify data from your input you have to use a Key-Value pair, with the key value ending in a ".$". So to fix this you need to change it to:
"Key": {
"id": {
"S.$": "$.id"
}
},
Using the above it should correctly resolve to the value from your input instead of the string value "$.id".
References - https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html#input-output-parameters
How do I create a mapping for a list of numbers object in API gateway? I am trying to post a list of integers using POST request. I tried working with NS attribute but got the Error.
Error:
{
"__type": "com.amazon.coral.service#SerializationException"
}
However, it works well when I have N attribute and post a single integer value.
Is there any way to resolve this issue?
I believe you are trying to map your request payload to DynamoDB JSON String. You can apply a velocity template like this one,
{
"TableName":"ABC",
"Item": {
"id": {
"S": "$context.requestId"
},
"name": {
"S": "$input.path('$.name')"
},
"price": {
"L": [
#set($prices=$input.path('$.price'))
#foreach($p in $prices)
{
"N": "$p"
}#if ($velocityCount < $prices.size()), #end
#end
]
}
}
}
Method Request Body:
{
"name":"Test",
"price": [1, 2, 3]
}
Endpoint Request Body:
{
"TableName": "ABC",
"Item": {
"id": {
"S": "test-invoke-request"
},
"name": {
"S": "Test"
},
"price": {
"L": [
{
"N": "1"
},
{
"N": "2"
},
{
"N": "3"
}
]
}
}
}