Here is my DynamoDB structure.
{"books": [
{
"name": "Hello World 1",
"id": "1234"
},
{
"name": "Hello World 2",
"id": "5678"
}
]}
I want to set ConditionExpression to check whether id existed before adding new items to books array. Here is my ConditionExpression. I am using API gateway to access DynamoDB.
"ConditionExpression": "NOT contains(#lu.books.id,:id)",
"ExpressionAttributeValues": {":id": {
"S": "$input.path('$.id')"
}
}
Result when I test the API: no matter id existed or not, success to add items to array.
Any suggestion on how to do it? Thanks!
Unfortunately, you can't. However, there is a workaround.
Store the books in separate rows. For example
PK SK
BOOK_LU#<ID> BOOK_NAME#<book name>#BOOK_ID#<BOOK_ID>
Now you can use the 'if_not_exists' conditional expression
"ConditionExpression": "if_not_exists(id, :id)'",
"ExpressionAttributeValues": {":id": {
"S": "$input.path('$.id')"
}
}
The con is if you were previously fetching the list as part of another object you will have to change that.
The pro is that now you can easily work with the books + you won't hit the max row size limits if the books became too many.
Related
I have the following dictionary
plaintext_item = {
"website": "https://example.com",
"description": "This is a sample data",
"website_username": {
"testuser1": "password12",
"testuser2": "password13",
}
}
In the above dictionary I want to encrypt both the passwords but not their usernames and store it in dynamoDb.
what I tried?
This was my first approach but didn't work
actions = AttributeActions(
default_action=CryptoAction.ENCRYPT_AND_SIGN,
attribute_actions={
"website": CryptoAction.DO_NOTHING,
plaintext_item["website_username"]["testuser1"]: CryptoAction.ENCRYPT_AND_SIGN,
"description": CryptoAction.DO_NOTHING,
}
)
Then I tried this below 2nd approach like how we update nested value in dynamodb, this too didn't work
actions = AttributeActions(
default_action=CryptoAction.ENCRYPT_AND_SIGN,
attribute_actions={
"website": CryptoAction.DO_NOTHING,
"website_username.testuser1": CryptoAction.ENCRYPT_AND_SIGN,
"description": CryptoAction.DO_NOTHING,
})
In both the above cases the whole object is getting encrypted and stored, I looked for some documentation but I am not able to find anything related, I am able to encrypt normal dictionaries like {"a":2,"b":3} but not nested ones.
When I do a table.put_item I get the error message "Aggregated size of all range keys has exceeded the size limit of 1024". What options do I have so I can save my data?
Change a setting in DynamoDB to allow a larger limit?
Split or compress the item and save to DynamoDB?
Store the item in s3?
Use another kind of database?
Other options?
Here is the specific code snippet:
def put_record(item):
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('table_name')
table.put_item(Item=item)
Here is an example of an item stored in DynamoDB. The two string variables p and r combined could be up to 4000 tokens.
{
"uuid": {
"S": "5bf19498-344c"
},
“p”: {
"S": “What is the next word after Mary had a”
},
“pp”: {
"S": "0"
},
"response_length": {
"S": "632"
},
"timestamp": {
"S": "04/03/2022 06:30:55 AM CST"
},
"s": {
"S": "1"
},
"c": {
"S": "test"
},
"f": {
"S": "0"
},
"t": {
"S": "0.7"
},
"to": {
"S": "1"
},
"b": {
"S": "1"
},
"r": {
"S": “lamb”
}
}
I read this
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ServiceQuotas.html
and couldn't figure out how the 1024 is calculated but I'm assuming the two string variables are causing the error.
The put_item doesn't cause an error when Item is a smaller size; only when the size is larger than the 1024 limit.
It is hard to estimate how many of the saves will be large but I need to be able to save the large items. So from an architecture perspective willing to consider any and all options.
Appreciate the assistance!
The error message "Aggregated size of all range keys has exceeded the size limit of 1024" is baffling because there can only be one sort key, so what does "aggregate" refer to? The following post was also surprised by this message: https://www.stuffofminsun.com/2019/05/07/dynamodb-keys/.
I am guessing you actually do try to write an item where its "p" (which you said is your sort key) itself is over 1024 characters. I don't see how it's the size of p+r combined that matters. You can take a look (and/or include in the question) at the one specific request that fails, and check what is the length of p itself. Please also double-check that you really set "p", and not something else, as the sort key.
Finally, if you really need sort keys over 1024 characters in length, you can consider Scylla Alternator - an open-source DynamoDB-compatible database which doesn't have this specific limitation.
Basically, i have the following input:
{
"name": "abc",
"choice": "choice1"
}
My dynamoDB table has the following structure:
Partition key - "name"
Complex json with choices:
{
"choices":
{
"choice1": ......,
"choice2": ......
}
}
I want to directly read from dynamodb, and get a subitem under the relevant choice:
{
"StartAt": "Read Next Message from DynamoDB",
"States": {
"Read Next Message from DynamoDB": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:getItem",
"Parameters": {
"TableName": "my_table",
"Key": {
"customerName": {"S.$": "$.name"}
}
},
"OutputPath": "$.Item.choices.M.choice1.M.myvalue.S",
"Next": "World"
},
"World": {
"Type": "Pass",
"End": true
}
}
}
basically i want to do something like "$.Item.choices.M.{$.choice}.M.myvalue.S", and take one of the output's keys from the input. is this possible?
I think what you're looking for is JsonPath interpolation, but that is not supported as per this thread on AWS forums.
As far as I know Step Functions allow only path reference through $, . and [] operators (Reference Path).
I don't know how much control you have on the DynamoDB table's data but I think your problem can be solved easily if your choice types are modeled in following way
{
"choices": [{
"choiceType": "choice1",
........
},
{
"choiceType": "choice2",
........
}]
}
Now you can use the map state to iterate over the choices array. Note that don't forget to pass the expected choiceType to each iteration.
First state of the map iterator can be a choice state which compares choiceType and moves to appropriate next state. So, basically your rest of the workflow is modeled as iterator of the map state in step 1.
Now, if you don't have the control over DynamoDB table, then you can process the query result in an AWS Lambda.
I need to store a list of map in cassandra. Is that possible?
This is a json representation of my data:
{
"deviceId" : "261e92b8-91af-40da-8ba4-c39d821472ec",
"sensors": [
{
"fieldSensorId": "sensorID",
"name": "sensorName",
"location": "sensor location",
"unit": "value units",
"notes": "notes"
},
{
"fieldSensorId": "sensorID 2",
"name": "sensorName 2",
"location": "sensor location 2",
"unit": "value units",
"notes": "notes"
}
]
}
CQL:
CREATE TABLE device_sensors (
device_id text,
sensors list<frozen <map<text,text>>>,
time timeuuid,
PRIMARY KEY (device_id)
)
Still im not able to insert any data. What is the right way of storing such data in cassandra? Later i will need to query the sensors list
Is it maybe wiser to create a sensors table and use sensor > to reference the sensors?
I think that the problem is that you declare devide_id as text in CQL, but you have declared itUUID in the source code, and Spring maps it into corresponding type when trying to insert data. Can you try to add #CassandraType(type = Name.TEXT) to the deviceId declaration. You can also remove the #Column declaration - the #PrimaryKeyColumn should be enough.
Or you can change the table definition to declare device_idas UUID.
Is there a better way to return the record(s) after DS.Store#pushPayload is called? This is what I'm doing...
var payload = { id: 1, title: "Example" }
store.pushPayload('post', payload);
return store.getById('post', payload.id);
But, with regular DS.Store#push you get the inserted record returned. The only difference between the two, from what I can tell, is that DS.Store#pushPayload serializes the payload data with the correct serializers.
DS.Store#pushPayload is able to take an array of items, not just one, and may contain side-loaded data. It processes a full payload and expects root keys in the payload:
{
"posts": [{
"id": 1,
"title": "title",
"comments": [1]
}],
"comments": [
//.. and so on ...
]
}
DS.Store#push expects a single record which has been normalized and contains no side loaded data (notice there is no root key):
{
"id": 1,
"title": "title",
"comments": [1]
}
For this reason, it makes sense for push to return the record, but for pushPayload to return nothing.
When you use pushPayload, a second lookup of store.find('post', 1) (or store.getById('post', 1)) is the way to go, I don't believe there is a better way.
As of this PR pushPayload can now return an array of all the records pushed into the store, once the 'ds-pushpayload-return' feature flag has been enabled.
At the moment, this feature isn't available in a standard or beta release-- you'll have to use
"ember-data": "emberjs/data#master",
(i.e. Canary) in your package.json in order to access it. I'm not sure when the feature will be generally available.