I am working on a lambda function that gets called from API Gateway and updates information in dynamoDB. I have half of this working really dynamically, and im a little stuck on updating. Here is what im working with:
dynamoDB table with a partition key of guild_id
My dummy json code im using:
{
"guild_id": "126",
"guild_name": "Posted Guild",
"guild_premium": "true",
"guild_prefix": "z!"
}
Finally the lambda code:
import json
import boto3
def lambda_handler(event, context):
client = boto3.resource("dynamodb")
table = client.Table("guildtable")
itemData = json.loads(event['body'])
guild = table.get_item(Key={'guild_id':itemData['guild_id']})
#If Guild Exists, update
if 'Item' in guild:
table.update_item(Key=itemData)
responseObject = {}
responseObject['statusCode'] = 200
responseObject['headers'] = {}
responseObject['headers']['Content-Type'] = 'application/json'
responseObject['body'] = json.dumps('Updated Guild!')
return responseObject
#New Guild, Insert Guild
table.put_item(Item=itemData)
responseObject = {}
responseObject['statusCode'] = 200
responseObject['headers'] = {}
responseObject['headers']['Content-Type'] = 'application/json'
responseObject['body'] = json.dumps('Inserted Guild!')
return responseObject
The insert part is working wonderfully, How would I accomplish a similar approach with update item? Im wanting this to be as dynamic as possible so I can throw any json code (within reason) at it and it stores it in the database. I am wanting my update method to take into account adding fields down the road and handling those
I get the follow error:
Lambda execution failed with status 200 due to customer function error: An error occurred (ValidationException) when calling the UpdateItem operation: The provided key element does not match the schema.
A "The provided key element does not match the schema" error means something is wrong with Key (= primary key). Your schema's primary key is guild_id: string. Non-key attributes belong in the AttributeUpdate parameter. See the docs.
Your itemdata appears to include non-key attributes. Also ensure guild_id is a string "123" and not a number type 123.
goodKey={"guild_id": "123"}
table.update_item(Key=goodKey, UpdateExpression="SET ...")
The docs have a full update_item example.
Related
I'm new to DynamoDb and the intricacies of querying it - I understand (hopefully correctly) that I need to either have a partition Key or Global Secondary Index (GSI) in order to query against that value in the table.
I know I can use Appsync to query on a GSI by setting up a resolver - and this works. However, I have a setup using the Java AWS CDK (I'm writing in Kotlin) where I'm using Appsync and routing my queries into lambda Resolvers (so that once this works, I can do more complicated things later).
The Crux of the issue is that when I setup a Lambda to resolve my query, I end up with this Error msg: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Query condition missed key schema element: testName returned from the Lambda.
I think these should be the key snippets..
My DynamoDbBean..
#DynamoDbBean
data class Test(
#get:DynamoDbPartitionKey var id: String = "",
#get:DynamoDbSecondaryPartitionKey(indexNames = ["testNameIndex"])
var testName: String = "",
)
Using the CDK I Created GSI on
testTable.addGlobalSecondaryIndex(
GlobalSecondaryIndexProps.builder()
.indexName("testNameIndex")
.partitionKey(
Attribute.builder()
.name("testName")
.type(AttributeType.STRING)
.build()
)
.projectionType(ProjectionType.ALL)
.build())
Then, within my Lambda I am trying to query my DynamoDb table, using a fixed value here testName = A.
My Sample data in the Test table would be like so..
{
"id" : "SomeUUID",
"testName" : "A"
}
private var client: AmazonDynamoDB = AmazonDynamoDBClientBuilder.standard().build()
private var dynamoDB: DynamoDB = DynamoDB(client)
Lambda Resolver Snippets...
val table: Table = dynamoDB.getTable(TABLE_NAME)
val index: Index = table.getIndex("testNameIndex")
...
QuerySpec().withKeyConditionExpression("testNameIndex = :testName")
.withValueMap(ValueMap().withString(":testName", "A"))
val iterator: Iterator<Item> = index.query(querySpec).iterator()
while (iterator.hasNext()) {
logger.info(iterator.next().toJSONPretty())
}
This is what results in this Error msg: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Query condition missed key schema element: testName
Am I on the wrong lines here? I know there is some mixing of libs between the 'enhanced' Dynamo sdk and the dynamodbv2 sdk - so if there is a better way of doing this query, I'd love to know!
Thanks!
Your QuerySpec's withKeyConditionExpression is initialized wrong. You need to init it like.
not testNameIndex it should be testName
QuerySpec().withKeyConditionExpression("testName = :testName")
.withValueMap(ValueMap().withString(":testName", "A"))
I have a use case where I have to filter incoming data from Kinesis Firehose based on the type of the event. I should write only certain events to S3 and ignore the rest of the events. I am using lambda to filter the records. I am using following python code to achieve this:
def lambda_handler(event, context):
# TODO implement
output = []
for record in event['records']:
payload = base64.b64decode(record["data"])
payload_json = json.loads(payload)
event_type = payload_json["eventPayload"]["operation"]
if event_type == "create" or event_type == "update":
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': base64.b64encode(payload)}
output.append(output_record)
else:
output_record = {
'recordId': record['recordId'],
'result': 'Dropped'}
output.append(output_record)
return {'records': output}
I am only trying to process "create" and "update" events and dropping the rest of the events. I got the sample code from AWS docs and built it from there.
This is giving the following error:
{"attemptsMade":1,"arrivalTimestamp":1653289182740,"errorCode":"Lambda.MissingRecordId","errorMessage":"One or more record Ids were not returned. Ensure that the Lambda function returns all received record Ids.","attemptEndingTimestamp":1653289231611,"rawData":"some data","lambdaArn":"arn:$LATEST"}
I am not able to get what this error means and how to fix it.
Bug: The return statement needs to be outside of the for loop. This is the cause of the error. The function is processing multiple recordIds, but only 1 recordId is returned. Unindent the return statement.
The data key must be included in output_record, even if the event is being dropped. You can base64 encode the original payload with no transformations.
Additional context: event['records'] and output must be the same length (length validation). Each dictionary in output must have a recordId key whose value equals a recordId value in a dictionary in event['record'] (recordId validation).
From AWS documentation:
The record ID is passed from Kinesis Data Firehose to Lambda during the invocation. The transformed record must contain the same record ID. Any mismatch between the ID of the original record and the ID of the transformed record is treated as a data transformation failure.
Reference: Amazon Kinesis Data Firehose Data Transformation
I have a function create which uses 3 dynamodb tables. How do i mock three Dynamo db tables?
def create():
//This function uses a dynamodb table "x"
// Then it calls my_table() function
def my_table():
// This function basically uses two dynamodb table "y" and "z"
// This function returns a value which is used in create() function.
My test file has following code -
#patch.dict(os.environ, {"DYNAMODB_TABLE": "x",
'second_TABLE': "y",
'Third_TABLE': "z"
})
def test_create():
dynamodb_test()
event = { // my event values}
result = create(event)
assert result == 200
def dynamodb_test():
with mock_dynamodb2():
dynamodb = boto3.client('dynamodb', region_name='us-east-1')
dynamodb.create_table(
TableName=os.environ["DYNAMODB_TABLE"],
KeySchema=[
{
'AttributeName': 'id',
'KeyType': 'HASH'
}
],
AttributeDefinitions=[
{
'AttributeName': 'id',
'AttributeType': 'S'
}
],
ProvisionedThroughput={
'ReadCapacityUnits': 1,
'WriteCapacityUnits': 1
}
)
yield dynamodb
whenever i am testing test_create() function using pytest , i am getting
botocore.exceptions.ClientError: An error occurred (ExpiredTokenException) when
calling the Scan operation: The security token included in the request is expired
I think its trying to access the actual aws dynamo db but i want it to use mock dynamodb. How can i achieve this ?
Moto only works when two conditions are met:
The logic to be tested is executed inside a Moto-context
The Moto-context is started before any boto3-clients (or resources) are created
The Moto-context in your example, with mock_dynamodb2(), is localized to the dynamodb_test-function. After the function finishes, the mock is no longer active, and Boto3 will indeed try to access AWS itself.
Solution
The following test-function would satisfy both criteria:
#patch.dict(os.environ, {"DYNAMODB_TABLE": "x",
'second_TABLE': "y",
'Third_TABLE': "z"
})
# Initialize the mock here, so that it is effective for the entire test duration
#mock_dynamodb2
def test_create():
dynamodb_test()
event = { // my event values}
# Ensure that any boto3-clients/resources created in the logic are initialized while the mock is active
from ... import create
result = create(event)
assert result == 200
def dynamodb_test():
# There is no need to start the mock-context again here, so create the table immediately
dynamodb = boto3.client('dynamodb', region_name='us-east-1')
dynamodb.create_table(...)
The test code you provided does not talk about creating tables y and z - if the logic expects them to exist, you'd have to create them manually as well of course (just like table x was created in dynamodb_test.
Documentation for the import quirk can be found here: http://docs.getmoto.org/en/latest/docs/getting_started.html#recommended-usage
I believe this post is almost identical to yours. You could try this or utilize some of the other existing tools like localstack or dynamodb-local. The Python client for localstack for example: https://github.com/localstack/localstack-python-client
EDIT:
I see your title explains you want to use moto. I don't see you importing moto any of the moto modules into your code. See the last snippet in this page and replace s3 with either dynamodb, dynamodb2 (whichever you are using)
I have table already and I want to add a new attribute to that table.
I am trying to do that with the update_item functionality of dynamDB.
use case: Bid table holds details on the bid of the product. The user accepts the bid, once the bid is accepted, have to add a few attributes to that record like the user information. Not sure if it is the right way or should I have a new table for this.
pratition key is : Pickup,
sort key is : DropOff
A Demo example that I am trying currently
currently trying to alter the same table and facing the error.
import json
import boto3
def lambda_handler(event, context):
dynamo_client = boto3.resource('dynamodb')
users = dynamo_client.Table('LoadsandBids')
item = event['body']
print("Fuirst")
users.update_item(
Key={
'Pickup': event['body']['Pickup'],
'DropOff' : event['body']['DropOff']
},
UpdateExpression='SET #attr1 = :val1',
ExpressionAttributeNames={'#attr1': 'new_field'},
ExpressionAttributeValues={':val1': event['body']['new']},
ReturnValues='UPDATED_NEW'
)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
getting an error:
"An error occurred (ValidationException) when calling the UpdateItem operation: The provided key element does not match the schema",
Could anyone help me out of this and also suggest it I my approach is good or not?
I am trying to get_item from a table in dynamodb.
def read_table_item(table_name, pk_name, pk_value):
"""
Return item read by primary key.
"""
table = dynamodb.Table(table_name)
response = table.get_item( Key={pk_name: pk_value})
return response
print (read_table_item(table_name,pk_name="_id",pk_value={"S":str(1)}))
The error I get is
"botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the GetItem operation: The provided key element does not match the schema"
It will be helpful if somebody review the above piece and help us rectify the issue.
Thanks
it seems to me what ever you are passing to pk_name is not in the DynamodbSchema. See following for more informaton.
Incorrect dynamo db key mapping