Dynamodb update multiple items in one transaction - amazon-web-services

In my dynamodb table, I have a scenario where when I add a new item to my table, I need to also increment a counter for another entry in my table.
For instance, when USER#1 follows USER#2, I would like to increment followers count for USER#2.
HashKey
RangeKey
counter
USER1
USER2
USER3
USER2
USER2
USER2
2
I do not want to use auto-increment as I want to control how the increment happens.
Naturally, everything works as expected if I make two update calls to dynamodb. One to create the relationship between users and another to update the count for the other user.
The question is, if it is a good approach to make two such calls or would a transactWrite be a better alternative.
If so how could I make an increment using transactwrite api.
I can add items using the following approach. But I am not sure how I can increment
"TransactItems": [
{
"Put": {
"TableName": "Table",
"Item": {
"hashKey": {"S":"USER1"},
"rangeKey": {"S":"USER2"}
}
}
},
{
"Update": {
"TableName": "TABLE",
"Key": {
"hashKey": {"S":"USER2"},
"rangeKey": {"S":"USER2"}
},
"ConditionExpression": "#cefe0 = :cefe0",
"ExpressionAttributeNames": {"#cefe0":"counter"},
"ExpressionAttributeValues": ?? how do I increment here
}
}
]

Transactions would defintely be the best way to approach it, you can increment using SET in the UpdateExpression:
"TransactItems": [
{
"Put": {
"TableName": "Table",
"Item": {
"hashKey": {"S":"USER1"},
"rangeKey": {"S":"USER2"}
}
}
},
{
"Update": {
"TableName": "TABLE",
"Key": {
"hashKey": {"S":"USER2"},
"rangeKey": {"S":"USER2"}
},
"UpdateExpression": "SET #cefe0 = #cefe0 + :cefe0",
"ExpressionAttributeNames": {"#cefe0":"counter"},
"ExpressionAttributeValues": {"cefe0": {"N": "1"}}
}
}
]

Related

how to update table item in dynamodb from sqs msg payload but not overwrite the table structure?

i have the following chain: lambda->sns->sqs-> ec2 script->dynamodb table.
the test event for lambda
{
"body": {
"MessageAttributes": {
"vote": {
"Type": "String",
"StringValue": "b"
},
"voter": {
"Type": "String",
"StringValue": "count"
}
}
}
}
originally the table should have 1 partition and 2 attributes - a and b, created with this json code:
{
"voter": {
"S": "count"
},
"a": {
"N": "11"
},
"b": {
"N": "20"
}
}
the issue is in ec2 script that runs continuously. It has a function update_count(vote) that should just increase the existing a or b by 1. The code:
def update_count(vote):
logging.info('update count....')
table.update_item(
Key={'voter': 'count'},
UpdateExpression="ADD #vote = #vote + :incr",
ExpressionAttributeNames={'#vote': vote},
ExpressionAttributeValues={':incr': 1}
)
when i send my test event it overwrites my dynamodb table and it has the following structure:
An UpdateItem will never overwrite attributes which are not set in the UpdateExpression.
I would ensure that you have no other code which is inadvertantly overwriting your item, perhaps from a Lambda function listening to a stream, or another function being called unknowingly in your application.
You can enable DynamoDB Dataplane logs on Cloudtrail to understand what is happening:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/logging-using-cloudtrail.html

AWS, dynamodb startwith condiftion gsi

I need to execute query like this:
select * from table where sampling_date like "2020-05-%"
To do this, I'm calling for
db.query({
TableName: "Tubes",
Select: "ALL_ATTRIBUTES",
IndexName: "sampling_date_idx",
KeyConditionExpression: " sampling_date > :sampling_date ",
ExpressionAttributeValues:{ ':sampling_date': {'S': '2020-05-'}}
}, function(error: AWSError, data: QueryOutput){
console.log(error, data);
})
And I get this error message:
{"errorType":"Error","errorMessage":"{\"message\":\"Query key condition not supported\",\"code\":\"ValidationException\",
My table:
this.tubes = new dynamodb.Table(this, "tubes", {
tableName: "Tubes",
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
partitionKey: {
name: "id",
type: dynamodb.AttributeType.STRING
},
pointInTimeRecovery: true,
removalPolicy: cdk.RemovalPolicy.RETAIN
});
this.tubes.addGlobalSecondaryIndex({
indexName: "sampling_date_idx",
sortKey: {
name: 'sampling_date_srt',
type: AttributeType.STRING
},
partitionKey: {
name: "sampling_date",
type: AttributeType.STRING,
},
})
I think there are two issues in your current code -
In KeyConditionExpression, there must be an equality condition on a single partition key value. In your case, it must include "sampling_date = :sampling_date".
Please read "KeyConditionExpression" section in -
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
In short, you only can perform equality test against partition key.
I am not sure which language you use. I suspect your syntax for ExpressionAttributeValues is not correct.
The syntax given in AWS doc is -
"ExpressionAttributeValues": {
"string" : {
"B": blob,
"BOOL": boolean,
"BS": [ blob ],
"L": [
"AttributeValue"
],
"M": {
"string" : "AttributeValue"
},
"N": "string",
"NS": [ "string" ],
"NULL": boolean,
"S": "string",
"SS": [ "string" ]
}
}
In your case, it may be something like -
"ExpressionAttributeValues": {
":sampling_date": {"S": "2020-05-01"}
}
My experience is in C#, it may be something like -
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{ ":sampling_date", new AttributeValue{S = "2005-05-01"} }
}
To solve your problem, you may need to use another attribute as the index's partition key. sampling_date only can be used as sort key.
sampling_date is the partition key for your GSI sampling_date_idx.
DynamoDB documentation says that in key condition expressions:
You must specify the partition key name and value as an equality condition.
Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.KeyConditionExpressions
So sampling_date can only be used with the "equal to" comparison operator. None of the other operators like less than, greater than, between, contains, begins with, etc. can be used with sampling_date.
However, these operators can be used with a sort key!
So if you can redesign your table and/or indexes such that sampling_date becomes a sort key of some index, you can use begins_with on it.
Here's a suggestion:
Create a GSI with partition key = sampling_year & sort key = sampling_date.
Then if your table has the following items:
{
"id": "id1",
"sampling_year": 2020,
"sampling_date": "2020-04-01"
}
{
"id": "id2",
"sampling_year": 2020,
"sampling_date": "2020-05-01"
}
{
"id": "id3",
"sampling_year": 2020,
"sampling_date": "2020-06-01"
}
And you use the following Node.js code:
let AWS = require("aws-sdk")
let dc = new AWS.DynamoDB.DocumentClient()
dc.query({
TableName: "Tubes",
IndexName: "sampling_year-sampling_date-index",
KeyConditions: {
"sampling_year": {
ComparisonOperator: "EQ",
AttributeValueList: [2020]
},
"sampling_date": {
ComparisonOperator: "BEGINS_WITH",
AttributeValueList: ["2020-05-"]
}
}
}
You'll get your desired output:
{
"id": "id2",
"sampling_year": 2020,
"sampling_date": "2020-05-01"
}
Try
KeyConditionExpression: `begins_with(sampling_date, :sampling_date)`
See available condition expressions here...
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html

Azure Cosmos query to convert into List

This is my JSON data, which is stored into cosmos db
{
"id": "e064a694-8e1e-4660-a3ef-6b894e9414f7",
"Name": "Name",
"keyData": {
"Keys": [
"Government",
"Training",
"support"
]
}
}
Now I want to write a query to eliminate the keyData and get only the Keys (like below)
{
"userid": "e064a694-8e1e-4660-a3ef-6b894e9414f7",
"Name": "Name",
"Keys" :[
"Government",
"Training",
"support"
]
}
So far I tried the query like
SELECT c.id,k.Keys FROM c
JOIN k in c.keyPhraseBatchResult
Which is not working.
Update 1:
After trying with the Sajeetharan now I can able to get the result, but the issue it producing another JSON inside the Array.
Like
{
"id": "ee885fdc-9951-40e2-b1e7-8564003cd554",
"keys": [
{
"serving": "Government"
},
{
"serving": "Training"
},
{
"serving": "support"
}
]
}
Is there is any way that extracts only the Array without having key value pari again?
{
"userid": "e064a694-8e1e-4660-a3ef-6b894e9414f7",
"Name": "Name",
"Keys" :[
"Government",
"Training",
"support"
]
}
You could try this one,
SELECT C.id, ARRAY(SELECT VALUE serving FROM serving IN C.keyData.Keys) AS Keys FROM C
Please use cosmos db stored procedure to implement your desired format based on the #Sajeetharan's sql.
function sample() {
var collection = getContext().getCollection();
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT C.id,ARRAY(SELECT serving FROM serving IN C.keyData.Keys) AS keys FROM C',
function (err, feed, options) {
if (err) throw err;
if (!feed || !feed.length) {
var response = getContext().getResponse();
response.setBody('no docs found');
}
else {
var response = getContext().getResponse();
var map = {};
for(var i=0;i<feed.length;i++){
var keyArray = feed[i].keys;
var array = [];
for(var j=0;j<keyArray.length;j++){
array.push(keyArray[j].serving)
}
feed[i].keys = array;
}
response.setBody(feed);
}
});
if (!isAccepted) throw new Error('The query was not accepted by the server.');
}
Output:

DynamoDB UpdateItem calculation with numbers?

I would like my UpdateItem function to add the new value to the previous one but I can't figure out how to use numbers for the calculation and access the current value to be added to the new one.
Here is my working function, what I want to do is included in comment. How can I do it?
{
"TableName": "Votes",
"Key": {
"votesId": {
"S": "$input.path('$.votesId')"
}
},
"UpdateExpression": "set buy = :val1",
"ExpressionAttributeValues" : {
":val1": {
"N": "$input.path('$.buy')"
//Would like: "N": "buy + $input.path('$.buy')"
}
},
"ReturnValues": "ALL_NEW"
}
Here is how I test it:
{
"votesId":1,
"down":0,
"up":0,
"hold":0,
"buy":0,
"sell":0
}
You can use UpdateExpression to indicate which attributes are to be updated and what actions to perform.
Specifically in your case, UpdateExpression supports the following:
SET myNum = myNum + :val

Elasticsearch _reindex fails

I am working on AWS Elasticsearch. It doesn't allow open/close index, so setting change can not be applied on the index.
In order to change the setting of a index, I have to create a new index with new setting and then move the data from the old index into new one.
So first I created a new index with
PUT new_index
{
"settings": {
"max_result_window":3000000,
"analysis": {
"filter": {
"german_stop": {
"type": "stop",
"stopwords": "_german_"
},
"german_keywords": {
"type": "keyword_marker",
"keywords": ["whatever"]
},
"german_stemmer": {
"type": "stemmer",
"language": "light_german"
}
},
"analyzer": {
"my_german_analyzer": {
"tokenizer": "standard",
"filter": [
"lowercase",
"german_stop",
"german_keywords",
"german_normalization",
"german_stemmer"
]
}
}
}
}
}
it succeeded. Then I try to move data from old index into new one with query:
POST _reindex
{
"source": {
"index": "old_index"
},
"dest": {
"index": "new_index"
}
}
It failed with
Request failed to get to the server (status code: 504)
I checked the indices with _cat api, it gives
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open old_index AGj8WN_RRvOwrajKhDrbPw 5 1 2256482 767034 7.8gb 7.8gb
yellow open new_index WnGZ3GsUSR-WLKggp7Brjg 5 1 52000 0 110.2mb 110.2mb
Seemingly some data are loaded into there, just wondering why the _reindex doesn't work.
You can check the status of reindex with api:
GET _tasks?detailed=true&actions=*reindex
There is a "status" object in response which has field "total":
total is the total number of operations that the reindex expects to perform. You can estimate the progress by adding the updated, created, and deleted fields. The request will finish when their sum is equal to the total field.
Link to ES Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html