I'm using save expression on an encrypted attribute named transactionAmount while updating data in dynamo DB. However the update query is failing with ConditionalCheckFailedException. The data is encrypted on client side during initial persistence in dynamodb in way same as described here. Following is the code:
Data Transfer Object:
public final class SampleDTO {
#DynamoDBHashKey(attributeName = CommonDynamoDBSchemaConstants.UNIQUE_KEY)
#Getter(onMethod = #__({ #DoNotTouch }))
private String uniqueKey;
#DynamoDBAttribute(attributeName = CommonDynamoDBSchemaConstants.EVENT_RUNNING_TIME_EPOCH)
#Getter(onMethod = #__({ #DoNotTouch }))
private Long eventRunningTimeInEpoch;
#DynamoDBAttribute(attributeName = CommonDynamoDBSchemaConstants.INSTRUMENT_TYPE)
#DynamoDBTypeConverted(converter = InstrumentTypeConverter.class)
#Getter(onMethod = #__({ #DoNotTouch }))
private InstrumentType instrumentType;
#DynamoDBAttribute(attributeName = CommonDynamoDBSchemaConstants.TRANSACTION_AMOUNT)
private String transactionAmount;
}
Data Access Code:
// fetches data from dynamoDB based on unique key passed to it.
SampleDTO sampleDTO = getSampleDTO("testLedgerUniqueKey");
sampleDTO.setInstrumentType(InstrumentType.MACHINE);
DynamoDBSaveExpression saveExpression = new DynamoDBSaveExpression();
Map<String, ExpectedAttributeValue> expressionAttributeValues =
new HashMap<String, ExpectedAttributeValue>();
expressionAttributeValues.put(
CommonDynamoDBSchemaConstants.LEDGER_UNIQUE_KEY,
new ExpectedAttributeValue(true)
.withValue(new AttributeValue(sampleDTO.getLedgerUniqueKey())));
expressionAttributeValues.put(
CommonDynamoDBSchemaConstants.TRANSACTION_AMOUNT,
new ExpectedAttributeValue(true).withValue(
new AttributeValue(sampleDTO.getTransactionAmount())));
saveExpression.setExpected(expressionAttributeValues);
saveExpression.setConditionalOperator(ConditionalOperator.AND);
dynamoDBMapper.save(sampleDTO, saveExpression, null /*dynamoDBMapperConfig*/);
ConditionalCheckFailedException:
You are trying to update a record that does not exist with your query condition. Please verify your query condition to make sure your query returns a record.
Reference:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.MessagesAndCodes
You specified a condition that evaluated to false. For example, you
might have tried to perform a conditional update on an item, but the
actual value of the attribute did not match the expected value in the
condition.
Hope it helps.
Related
We are using Spark job with emr-dynamodb-connector to load the data from S3 files into Dyanamodb.
https://github.com/awslabs/emr-dynamodb-connector
But if document is already present in dynamodb, my code is replacing it.
Is there a way to avoid updating existing records (based on id) if they are present in Dynamodb. If id is present in dynamodb, i simply don't want to update it, just skip that id and write rest of records. Code i am using is
JobConf ddbConf = new JobConf(spark.sparkContext().hadoopConfiguration());
ddbConf.set("dynamodb.output.tableName", tableName);
ddbConf.set("dynamodb.throughput.write.percent", "50");
ddbConf.set("mapred.input.format.class", "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat");
ddbConf.set("mapred.output.format.class", "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat");
JavaPairRDD<Text, DynamoDBItemWritable> ddbInsertFormattedRDD = finalDatasetToBeSaved.toJavaRDD().mapToPair(new PairFunction<Row, Text, DynamoDBItemWritable>() {
#Override
public Tuple2<Text, DynamoDBItemWritable> call(Row row) throws Exception {
Map<String, AttributeValue> ddbMap = new HashMap<String, AttributeValue>();
for (int i = 0 ; i <= schemaDdb.length - 1; i++) {
Object value = row.get(i);
if (value != null) {
AttributeValue att = new AttributeValue();
if(schemaDdb[i]._2.toString().equalsIgnoreCase("IntegerType")){
att.setN(value.toString());
}else{
att.setS(value.toString());
}
ddbMap.put((String)schemaDdb[i]._1, att);
}
}
DynamoDBItemWritable item = new DynamoDBItemWritable();
item.setItem(ddbMap);
return new Tuple2<Text, DynamoDBItemWritable>(new Text(""), item);
}
});
ddbInsertFormattedRDD.saveAsHadoopDataset(ddbConf);
By saying Is there a way to avoid updating existing records (based on id) if they are already present, Do you want to add another document instead of replacing/updating it?
If yes, I am afraid it wont be possible with primary key, since that should be unique and distinguishes it from other. You need to make a key non-primary in order to do this.
If you want to ignore the insertion (if item exists), you can use condition-expression attribute_not_exists(your-key) as defined in the documentation
I have a DynamoDB table, let us say here: ReportingTable. Its has following keys to uniquely identify items:
reportingtablePrimaryKey - partition key of table.
merchantId - sort Key of table.
transactionType-timestamp-index - Global Secondary Index of table containing following attributes.
transactionType - partition key of our GSI. We are always saving four types of values here. [Cancel, Refund, Shipment, MFNShipment]
timestamp- timestamp in epoch when item came into our system and was saved in dynamodb.
Now, the thing I am trying to achieve is, I am to calculate number of items present in DynamoDB table which lie between two timestamps (start and end timestamp).
For that, I came-up with the approach of using our GSI transactionType-timestamp-index and where for the list of values of transactionType and timestamp range, I will pass the key condition which will read all the records and to overcome returned response limiting issue and I will use lastEvaluatedKey in loop to get the other records till end.
Following is the code I am using:
private static int getNumberOfRecordsFromTable(final AmazonDynamoDB dynamoDBclient, final String tableName,
final String gsiIndex, final List<String> transactionTypes,
final long startTimeEpoch, final long endTimeEpoch) {
int numberOfRecords=0;
Map<String, AttributeValue> lastEvaluatedKey = null;
Map<String, AttributeValue> valueMap = new HashMap<>();
valueMap.put(":transaction_type", new AttributeValue().withSS(transactionTypes));
valueMap.put(":start_time_epoch", new AttributeValue().withN(String.valueOf(startTimeEpoch)));
valueMap.put(":end_time_epoch", new AttributeValue().withN(String.valueOf(endTimeEpoch)));
Map<String, String> nameMap = new HashMap<>();
nameMap.put("#timestamp","timestamp");
nameMap.put("#transactionType","transactionType");
final String conditionExpression = "(#transactionType = :transaction_type) " +
"AND (#timestamp BETWEEN :start_time_epoch AND :end_time_epoch)";
QueryRequest queryRequest = new QueryRequest()
.withTableName(tableName)
.withIndexName(gsiIndex)
.withKeyConditionExpression(conditionExpression)
.withExpressionAttributeNames(nameMap)
.withExpressionAttributeValues(valueMap)
.withProjectionExpression("#transactionType, #timestamp")
.withExclusiveStartKey(lastEvaluatedKey)
.withConsistentRead(false);
do {
int numberOfRecordsFetched=0;
QueryResult queryResult = dynamoDBclient.query(queryRequest);
lastEvaluatedKey = queryResult.getLastEvaluatedKey();
numberOfRecordsFetched = queryResult.getScannedCount();
queryRequest.setExclusiveStartKey(lastEvaluatedKey);
numberOfRecords = numberOfRecords + numberOfRecordsFetched;
} while (lastEvaluatedKey != null);
log.info("Number of {} type messages fetched :: {}", transactionType, numberOfRecords);
return numberOfRecords;
}
I am getting the following error:
Exception in thread "main" com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One or more parameter values were invalid: Condition parameter type does not match schema type (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: FJLJTP7NFVKPTSDF2AJRUL0PTJVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:3443)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:3419)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeQuery(AmazonDynamoDBClient.java:2318)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:2293)
at com.amazon.gstreporting.mtrschedulercli.scripts.GSTDatabaseRecordsCount.getNumberOfRecordsFromTable(GSTDatabaseRecordsCount.java:231)
at com.amazon.gstreporting.mtrschedulercli.scripts.GSTDatabaseRecordsCount.countDynamoDBRecords(GSTDatabaseRecordsCount.java:192)
at com.amazon.gstreporting.mtrschedulercli.scripts.GSTDatabaseRecordsCount.main(GSTDatabaseRecordsCount.java:123)
Could anyone help me in it?
The reason I was getting error - I was passing list of transactionType which ideally should been passed one by one into the query.
After, adding one more root for-loop to go over every transactionType I was able to fix it.
Please see the code change for reference:
private static int getNumberOfRecordsFromTable(final AmazonDynamoDB dynamoDBclient, final String tableName,
final String gsiIndex, final List<String> transactionTypes,
final long startTimeEpoch, final long endTimeEpoch) {
int numberOfRecords=0;
for (String transactionType: transactionTypes) {
Map<String, AttributeValue> lastEvaluatedKey = null;
Map<String, AttributeValue> valueMap = new HashMap<>();
valueMap.put(":transaction_type", new AttributeValue().withS(transactionType));
valueMap.put(":start_time_epoch", new AttributeValue().withN(String.valueOf(startTimeEpoch)));
valueMap.put(":end_time_epoch", new AttributeValue().withN(String.valueOf(endTimeEpoch)));
Map<String, String> nameMap = new HashMap<>();
nameMap.put("#timestamp","timestamp");
nameMap.put("#transactionType","transactionType");
final String conditionExpression = "(#transactionType = :transaction_type) " +
"AND (#timestamp BETWEEN :start_time_epoch AND :end_time_epoch)";
QueryRequest queryRequest = new QueryRequest()
.withTableName(tableName)
.withIndexName(gsiIndex)
.withKeyConditionExpression(conditionExpression)
.withExpressionAttributeNames(nameMap)
.withExpressionAttributeValues(valueMap)
.withProjectionExpression("#transactionType, #timestamp")
.withExclusiveStartKey(lastEvaluatedKey)
.withConsistentRead(false);
do {
int numberOfRecordsFetched=0;
QueryResult queryResult = dynamoDBclient.query(queryRequest);
lastEvaluatedKey = queryResult.getLastEvaluatedKey();
numberOfRecordsFetched = queryResult.getScannedCount();
queryRequest.setExclusiveStartKey(lastEvaluatedKey);
numberOfRecords = numberOfRecords + numberOfRecordsFetched;
} while (lastEvaluatedKey != null);
log.info("Number of {} type messages fetched :: {}", transactionType, numberOfRecords);
}
return numberOfRecords;
}
I'm trying to create an initial DB state in DB Unit like this...
public function getDataSet() {
$primary = new \PHPUnit\DbUnit\DataSet\CompositeDataSet();
$fixturePaths = [
"test/Seeds/Upc/DB/UpcSelect.xml",
"test/Seeds/Generic/DB/ProductUpcSelect.xml"
];
foreach($fixturePaths as $fixturePath) {
$dataSet = $this->createXmlDataSet($fixturePath);
$primary->addDataSet($dataSet);
}
return $primary;
}
Then after my query I'm attempting to call this user-defined function...
protected function compareDatabase(String $seedPath, String $table) {
$expected = $this->createFlatXmlDataSet($seedPath)->getTable($table);
$result = $this->getConnection()->createQueryTable($table, "SELECT * FROM $table");
$this->assertTablesEqual($expected, $result);
}
The idea here is that I have an initial DB state, run my query, then compare the actual table state with the XML data set representing what I expect the table to look like. This process is described in PHPUnit's documentation for DBUnit but I keep having an exception thrown...
PHPUnit\DbUnit\InvalidArgumentException: There is already a table named upc with different table definition
Test example...
public function testDeleteByUpc() {
$mapper = new UpcMapper($this->getPdo());
$mapper->deleteByUpc("someUpcCode1");
$this->compareDatabase("test/Seeds/Upc/DB/UpcAfterDelete.xml", 'upc');
}
I seem to be following the docs...how is this supposed to be done?
This was actually unrelated to creating a second XML Dataset. This exception was thrown because the two fixtures I loaded in my getDataSet() method both had table definitions for upc.
I have a table called friends:
Friend 1 | Friend 2 | Status
Friend 1 is my HASH attribute and Friend 2 is my range attribute.
I would like to update an item's staus attribute where friend 1 = 'Bob' and friend 2 = 'Joe'. Reading through the documentation on http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/JavaDocumentAPICRUDExample.html I can only see how to update an item by 1 key, how do I include the other key?
Here you go:
DynamoDBQueryExpression<Reply> queryExpression = new DynamoDBQueryExpression<Reply>()
.withKeyConditionExpression("Id = :val1 and ReplyDateTime > :val2")
.withExpressionAttributeValues(
...
where Id is the Hash Key and ReplyDateTime is the Range Key.
Reference:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.QueryScanExample.html
I'm writing example where you can make update of multiple item in single table. I have primary key as id and range key as Datetime.
Actually there is no feature available in dynamodb so what i'm doing here is first query all the variable with hash key and range key of which i want to make update. Once all data are stored in List then loading data with it's hash key and rangekey and changing or updating field using set and save it.
Since I'm editing in hash key so, hash key original will be there we need to delete it. If you need to update in next attribute no need. I haven't added deleting code write yourself. You can query if you have confusion your entry with hash key will be still and new entry with new hash key will be added.
Code is below:
public static void main(String[] args) {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
DynamoDBMapper mapper = new DynamoDBMapper(client);
client.setEndpoint("http://localhost:8000/");
String fromDate = "2016-01-13";
String toDate = "2016-02-05";
User user = new User();
user.setId("YourHashKey");
LocalDate frmdate = LocalDate.parse(fromDate, DateTimeFormatter.ISO_LOCAL_DATE);
LocalDate todate = LocalDate.parse(toDate, DateTimeFormatter.ISO_LOCAL_DATE);
LocalDateTime startfrm = frmdate.atStartOfDay();
LocalDateTime endto = todate.atTime(23, 59, 59);
Condition rangeCondition = new Condition().withComparisonOperator(ComparisonOperator.BETWEEN.toString()).withAttributeValueList(new AttributeValue().withS(startfrm.toString()), new AttributeValue().withS(endto.toString()));
DynamoDBQueryExpression<User> queryExpression = new DynamoDBQueryExpression<User>().withHashKeyValues(user).withRangeKeyCondition("DATETIME", rangeCondition);
List<User> latestReplies = mapper.query(User.class, queryExpression);
for (User in : latestReplies) {
System.out.println(" Hashid: " + in.getId() + " DateTime: " + in.getDATETIME() + "location:" + in.getLOCID());
User ma = mapper.load(User.class, in.getId(), in.getDATETIME());
ma.setLOCID("Ohelig");
mapper.save(ma);
}
}
I have Product table which has a related table Images with a relation 1:M.
Class Product {
private Integer productId;
private String productName;
....
....
....
private List<Image> productImageList;
....
....
....
}
Class Image{
private Integer imageId;
private String imageName;
}
Class ProductLite{
private Integer productId;
private String productName;
private String imageName;
}
I am trying a JPQL query where I want to query to fetch products and the first image from the productImageList and returning a ProductLite object using the new constructor.
#TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)
public List<ProductLite> getAllProductLite() {
Query q = em.createQuery("SELECT NEW com.mycomp.application.entity.ProductLite(p.productId, p.productName, p.productImageList.get(0).getImageName())"
+ " from Product p"
+ " ORDER by p.productName");
List<ProductLite> prods = q.getResultList();
return prods;
}
But for some reason I am not able to get it to work. I get a NoViableException. So I tried moving the logic of getting the first image (getImage() method) to the Product Entity so in the query I could just call the getImage(). Even that does not seem to work.
java.lang.IllegalArgumentException: An exception occurred while creating a query in EntityManager:
Exception Description: Syntax error parsing the query [SELECT NEW com.meera.application.entity.ProductLite(distinct p.productId, p.productName, p.getImage()) from Product p, IN(p.productImageList) pil where p.category.categoryCode = :categoryCode ORDER by p.productName ], line 1, column 52: unexpected token [distinct].
Internal Exception: NoViableAltException(23#[452:1: constructorItem returns [Object node] : (n= scalarExpression | n= aggregateExpression );])
Any help is appreciated.
First, you cannot call methods in entity class from your JP QL query. Second, to use the order of entities in list, you need persisted order.
To create column for order to the join table between image and product, you have to add
#OrderColumn-annotation to the productImageList. For example:
#OrderColumn(name = "myimage_order")
//or dont't define name and let it default to productImageList_order
#OneToMany
private List<Image> productImageList;
Then you have to modify query to use that order to choose only first image:
SELECT NEW com.mycomp.application.entity.ProductLite(
p.productId, p.productName, pil.imageName)
FROM Product p JOIN p.productImageList pil
WHERE INDEX(pil) = 0
ORDER by p.productName