DynamoDBMapper ConditionalCheckFailedException when range-key attribute for GSI is not present in update request - amazon-web-services

I'm trying to reason about the cause of a ConditionalCheckFailedException I receive when using DynamoDBMapper with a specific save expression and with UPDATE_SKIP_NULL_ATTRIBUTES SaveBehavior.
My schema is as follows:
Member.java
#Data
#DynamoDBTable(tableName = "members")
public class Member implements DDBTable {
private static final String GROUP_GSI_NAME = "group-gsi";
#DynamoDBHashKey
#DynamoDBAutoGeneratedKey
private String memberId;
#DynamoDBVersionAttribute
private Long version;
#DynamoDBIndexHashKey(globalSecondaryIndexName = GROUP_GSI_NAME)
private String groupId;
#DynamoDBAutoGeneratedTimestamp(strategy = DynamoDBAutoGenerateStrategy.CREATE)
#DynamoDBIndexRangeKey(globalSecondaryIndexName = GROUP_GSI_NAME)
private Date joinDate;
#DynamoDBAttribute
private String memberName;
#Override
#DynamoDBIgnore
public String getHashKeyColumnName() {
return "memberId";
}
#Override
#DynamoDBIgnore
public String getHashKeyColumnValue() {
return memberId;
}
}
I use the following class to create/update/get the records in the members table.
DDBModelDAO.java
public class DDBModelDAO<T extends DDBTable> {
private final Class<T> ddbTableClass;
private final AmazonDynamoDB amazonDynamoDB;
private final DynamoDBMapper dynamoDBMapper;
public DDBModelDAO(Class<T> ddbTableClass, AmazonDynamoDB amazonDynamoDB, DynamoDBMapper dynamoDBMapper) {
this.ddbTableClass = ddbTableClass;
this.amazonDynamoDB = amazonDynamoDB;
this.dynamoDBMapper = dynamoDBMapper;
}
public T loadEntry(final String hashKey) {
return dynamoDBMapper.load(ddbTableClass, hashKey);
}
public T createEntry(final T item) {
dynamoDBMapper.save(item, getSaveExpressionForCreate(item));
return item;
}
public T updateEntry(final T item) {
dynamoDBMapper.save(item, getSaveExpressionForUpdate(item),
DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES.config());
return item;
}
private DynamoDBSaveExpression getSaveExpressionForCreate(final T item) {
// No record with the same hash key must be present when creating
return new DynamoDBSaveExpression().withExpectedEntry(item.getHashKeyColumnName(),
new ExpectedAttributeValue(false));
}
private DynamoDBSaveExpression getSaveExpressionForUpdate(final T item) {
// The hash key for the record being updated must be present.
return new DynamoDBSaveExpression().withExpectedEntry(item.getHashKeyColumnName(),
new ExpectedAttributeValue(new AttributeValue(item.getHashKeyColumnValue()))
.withComparisonOperator(ComparisonOperator.EQ)
);
}
}
I wrote a test class to insert and update records into the members table, which is as follows:
public static void main(String[] args) {
DDBTestClient testClient = new DDBTestClient();
AmazonDynamoDB amazonDynamoDB = testClient.buildAmazonDynamoDB();
DynamoDBMapper dynamoDBMapper = testClient.buildDynamoDBMapper(amazonDynamoDB);
DDBModelDAO<Member> memberDAO = new DDBModelDAO<>(Member.class, amazonDynamoDB, dynamoDBMapper);
DDBModelDAO<Group> groupDAO = new DDBModelDAO<>(Group.class, amazonDynamoDB, dynamoDBMapper);
try {
// Create a group
Group groupToCreate = new Group();
groupToCreate.setGroupName("group-0");
Group createdGroup = groupDAO.createEntry(groupToCreate);
System.out.println("Created group: " + createdGroup);
Thread.sleep(3000);
// Create a member for the group
Member memberToCreate = new Member();
memberToCreate.setGroupId(createdGroup.getGroupId());
memberToCreate.setMemberName("member-0");
Member createdMember = memberDAO.createEntry(memberToCreate);
System.out.println("Created member: " + createdMember);
Thread.sleep(3000);
// Update member name
createdMember.setMemberName("member-updated-0");
createdMember.setGroupId(null);
//createdMember.setJoinDate(null); // <---- Causes ConditionalCheckFailedException
memberDAO.updateEntry(createdMember);
System.out.println("Updated member");
} catch (Exception exception) {
System.out.println(exception.getMessage());
}
}
As can be seen above, if I do not pass a valid value for joinDate(which happens to be the range-key for the groups GSI), in the updateEntry call, DynamoDB returns a ConditionalCheckFailedException. This is the case, even when I use a save behavior of UPDATE_SKIP_NULL_ATTRIBUTES, as can be seen in DDBModelDAO.java.
Can someone help me understand, why I'm required to send the range-key attribute for the GSI, for a conditional write to succeed?

Not sure if this answers your question:
Interface DynamoDBAutoGenerator:
"DynamoDBAutoGenerateStrategy.CREATE, instructs to generate when
creating the item. The mapper, determines an item is new, or
overwriting, if it's current value is null. There is a limitation when
performing partial updates using either,
DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES, or DynamoDBMapperConfig.SaveBehavior.APPEND_SET. A new value will only be generated if the mapper is also generating the key."
So the last part is important: "A new value will only be generated if the mapper is also generating the key"
That should explain why you only see the behavior that you are experiencing.
Does this make sense?

Related

How to validate an expression in Apache Calcite

I want to know how to validate a expression in Calcite. I create a schema and add my table into it, then I want to use Validator to validate a SqlNode which is from an expression. Here is the code.
public class DummyTable extends AbstractTable {
private final String tableName;
public DummyTable(String tableName) {
this.tableName = tableName;
}
#Override
public RelDataType getRowType(RelDataTypeFactory typeFactory) {
RelDataTypeFactory.Builder builder = typeFactory.builder();
builder.add("a", SqlTypeName.BIGINT);
builder.add("b", SqlTypeName.BIGINT);
return builder.build();
}
public String getTableName() {
return tableName;
}
}
public class ConditionValidator {
private final String sql;
private final SchemaPlus rootSchema;
private final FrameworkConfig frameworkConfig;
private final RelDataTypeFactory relDataTypeFactory;
private final CalciteCatalogReader catalogReader;
private final SqlOperatorTable sqlOperatorTable;
private final SqlValidator validator;
public ConditionValidator(String sql) {
this.sql = sql;
SchemaPlus rootSchema = Frameworks.createRootSchema(true);
DummyTable testTable = new DummyTable("test_table");
rootSchema.add(testTable.getTableName(), testTable);
this.rootSchema = rootSchema;
this.frameworkConfig = Frameworks.newConfigBuilder()
.parserConfig(SqlParser.config()
.withLex(Lex.MYSQL)
.withConformance(SqlConformanceEnum.DEFAULT))
.defaultSchema(rootSchema)
.operatorTable(SqlStdOperatorTable.instance())
.build();
this.relDataTypeFactory = new SqlTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
Properties properties = new Properties();
properties.setProperty(CalciteConnectionProperty.CASE_SENSITIVE.camelName(), "true");
this.catalogReader = new CalciteCatalogReader(
CalciteSchema.from(rootSchema),
CalciteSchema.from(rootSchema).path(rootSchema.getName()),
relDataTypeFactory,
new CalciteConnectionConfigImpl(properties));
this.sqlOperatorTable = SqlOperatorTables.chain(frameworkConfig.getOperatorTable(), catalogReader);
this.validator = SqlValidatorUtil.newValidator(sqlOperatorTable, catalogReader, relDataTypeFactory, frameworkConfig.getSqlValidatorConfig());
}
public SqlNode validate() {
SqlParser sqlParser = SqlParser.create(sql, frameworkConfig.getParserConfig());
SqlNode sqlNode;
try {
sqlNode = sqlParser.parseExpression();
} catch (Exception e) {
throw new RuntimeException(e);
}
SqlNode validate = validator.validate(sqlNode);
return validate;
}
}
And now I can validate a SqlNode from a SQL query, for example, when input sql is "select a, b from test_table where a > 1 and b = 1", validation is ok (if validating a query, you should use the way below to parse the sql:
sqlNode = sqlParser.parseQuery(); // not 'sqlParser.parseExpression();
).
However, when I validate an expression like "a > 1 and b = 1", such an exception occurs: "Column 'a' not found in any table". I assume there maybe something wrong with validate scope, but I can't find the solution. Can someone help me? Thanks a lot!

DynamoDB: find item by Global Secondary Index using Spring Data

In my project, I want to fetch data by GSI. And my GSI is such that it will always be unique, so it will return at most one item at a time. But when I used #DynamoDBIndexHashKey annotation on the field that I have made GSI, I get a ResourceNotFoundException:
"Exception: | Exception : class com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException | Exception message : Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException;"
But if I only use #DynamoDBAttribute which is to say only defines the column, everything works fine.
Repository:
#EnableScan
public interface MyRepository extends DynamoDBCrudRepository< MyModel, String> {
MyModel findByMetaData(String metaData);
MyModel findByPrimaryKey(String primaryKey);
}
Model:
#Data
#DynamoDBTable(tableName = "my_model")
public class MyModel {
#DynamoDBHashKey(attributeName = "primary_key")
#DynamoDBAttribute(attributeName = "primary_key")
private String primaryKey;
#DynamoDBIndexHashKey(attributeName = "meta_data", globalSecondaryIndexName = "meta_data")
#DynamoDBAttribute(attributeName = "meta_data")
private String metaData;
And this is how DynamoDB client is being created.
#Bean
public AmazonDynamoDB amazonDynamoDB(AWSCredentialsProvider awsCredentialsProvider,
#Value("${aws.region}") String region) {
return AmazonDynamoDBClientBuilder.standard()
.withCredentials(awsCredentialsProvider)
.withRegion(region)
.build();
}
#Bean
#Primary
public DynamoDBMapper mapper(AmazonDynamoDB amazonDynamoDB,
#Value("${table.name.prefix}") String tableNamePrefix) {
DynamoDBMapperConfig config = new DynamoDBMapperConfig.Builder().
withTableNameOverride(DynamoDBMapperConfig.TableNameOverride.withTableNamePrefix(tableNamePrefix + "-"))
.build();
return new DynamoDBMapper(amazonDynamoDB, config);
}
Also find by primary partition key is working fine.
I have created a secondary index "meta_data" on my table too.
So is there a problem in code or in the way I have created global secondary index in AWS?

What is the difference between DynamoDBMapper and Table for DynamoDB Tables

In AWS DynamoDB, There are two options available to do the CRUD operations on the Table.
DynamoDBMapper :
com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;.
AmazonDynamoDB dbClient = AmazonDynamoDBAsyncClientBuilder.standard().withCredentials(creds)
.withRegion("us-east-1").build();
// creds is AWSCredentialsProvider
DynamoDBMapper mapper = new DynamoDBMapper(dbClient);
mapper.save(item);
Table: com.amazonaws.services.dynamodbv2.document.Table;.
static DynamoDB dynamoDB =new DynamoDB(dbClient);
Table table = dynamoDB.getTable("TABLE_NAME");
Item item =new Item().withPrimaryKey("","")
.withString("":, "");
table.putItem(item);
Both seem to do the same operations.
Is DynamoDBMapper a layer over Table? If so what are the differences in using each of these?
If you want to map Java classes to DynamoDB tables (which is a useful feature), consider moving away from the old V1 API (com.amazonaws.services.dynamodbv2 is V1). V2 packages are software.amazon.awssdk.services.dynamodb.*.
Replace this old API with the DynamoDB V2 Enhanced Client. You can learn about this here:
Map items in DynamoDB tables
You can find code examples for using the Enhanced Client here.
Here is a Java V2 code example that shows you how to use the Enhanced Client to put data into a Customer table. As you see, you can map a Java Class to columns in a DynamoDB table and then create a Customer object when adding data to the table.
package com.example.dynamodb;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import java.time.Instant;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
/*
* Prior to running this code example, create an Amazon DynamoDB table named Customer with a key named id and populate it with data.
* Also, ensure that you have setup your development environment, including your credentials.
*
* For information, see this documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class EnhancedPutItem {
public static void main(String[] args) {
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.build();
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
putRecord(enhancedClient) ;
ddb.close();
}
// Puts an item into a DynamoDB table
public static void putRecord(DynamoDbEnhancedClient enhancedClient) {
try {
DynamoDbTable<Customer> custTable = enhancedClient.table("Customer", TableSchema.fromBean(Customer.class));
// Create an Instant
LocalDate localDate = LocalDate.parse("2020-04-07");
LocalDateTime localDateTime = localDate.atStartOfDay();
Instant instant = localDateTime.toInstant(ZoneOffset.UTC);
// Populate the Table
Customer custRecord = new Customer();
custRecord.setCustName("Susan Blue");
custRecord.setId("id103");
custRecord.setEmail("sblue#noserver.com");
custRecord.setRegistrationDate(instant) ;
// Put the customer data into a DynamoDB table
custTable.putItem(custRecord);
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
System.out.println("done");
}
#DynamoDbBean
public static class Customer {
private String id;
private String name;
private String email;
private Instant regDate;
#DynamoDbPartitionKey
public String getId() {
return this.id;
};
public void setId(String id) {
this.id = id;
}
#DynamoDbSortKey
public String getCustName() {
return this.name;
}
public void setCustName(String name) {
this.name = name;
}
public String getEmail() {
return this.email;
}
public void setEmail(String email) {
this.email = email;
}
public Instant getRegistrationDate() {
return regDate;
}
public void setRegistrationDate(Instant registrationDate) {
this.regDate = registrationDate;
}
}
}

Illegal query expression: No hash key condition is found in the query in AWS Query

I have table in AWS mobile hub and I am using the following model for it
public class UserstopcoreDO {
private String _userId;
private String _usertoplevel;
private String _usertopscore;
private String _username;
#DynamoDBHashKey(attributeName = "userId")
#DynamoDBAttribute(attributeName = "userId")
public String getUserId() {
return _userId;
}
public void setUserId(final String _userId) {
this._userId = _userId;
}
#DynamoDBAttribute(attributeName = "usertoplevel")
public String getUsertoplevel() {
return _usertoplevel;
}
#DynamoDBAttribute(attributeName = "username")
public String getUsername() {
return _username;
}
public void setUsername(final String _username) {
this._username = _username;
}
public void setUsertoplevel(final String _usertoplevel) {
this._usertoplevel = _usertoplevel;
}
#DynamoDBIndexHashKey(attributeName = "usertopscore", globalSecondaryIndexName = "usertopscore")
public String getUsertopscore() {
return _usertopscore;
}
public void setUsertopscore(final String _usertopscore) {
this._usertopscore = _usertopscore;
}
}
In the table, I have 1500+ records and now I want to fetch Top 10 records from it so for that I write the below query
final DynamoDBQueryExpression<UserstopcoreDO> queryExpression = new DynamoDBQueryExpression<>();
queryExpression.withLimit(10);
queryExpression.setScanIndexForward(false);
final PaginatedQueryList<UserstopcoreDO> results = mapper.query(UserstopcoreDO.class, queryExpression);
Iterator<UserstopcoreDO> resultsIterator = results.iterator();
if (resultsIterator.hasNext()) {
final UserstopcoreDO item = resultsIterator.next();
try {
Log.d("Item :",item.getUsertopscore());
} catch (final AmazonClientException ex) {
Log.e(LOG_TAG, "Failed deleting item : " + ex.getMessage(), ex);
}
}
But when I run the code it gives me an error
Caused by: java.lang.IllegalArgumentException: Illegal query expression: No hash key condition is found in the query
but in my condition, I did not need any condition because I want to fetch top 10 records instead of one specific record. So how to handle that condition ?
If you want to "query" DynamoDB without specifying all HashKeys, use a Scan instead, i.e. DynamoDBScanExpression. You probably also want to change your "usertopscore" to be a RangeKey instead of a HashKey.
From https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/datamodeling/DynamoDBQueryExpression.html every DynamoDBQueryExpression requires all the Hash Keys be set.
Also see boto dynamodb2: Can I query a table using range key only?
Please set the hash key in the query expression. Below is the example of query expression for main table and GSI (need to set the index name).
Querying the main table:-
Set the hash key value of the table.
UserstopcoreDO hashKeyObject = new UserstopcoreDO();
hashKeyObject.setUserId("1");
DynamoDBQueryExpression<UserstopcoreDO> queryExpressionForMainTable = new DynamoDBQueryExpression<UserstopcoreDO>()
.withHashKeyValues(hashKeyObject);
Querying the Index:-
Set the index name and hash key value of the index.
UserstopcoreDO hashIndexKeyObject = new UserstopcoreDO();
hashIndexKeyObject.setUsertoplevel("100");
DynamoDBQueryExpression<UserstopcoreDO> queryExpressionForGsi = new DynamoDBQueryExpression<UserstopcoreDO>()
.withHashKeyValues(hashIndexKeyObject).withIndexName("usertopscore");
GSI attributes in mapper:-
#DynamoDBIndexHashKey(attributeName = "usertoplevel", globalSecondaryIndexName = "usertopscore")
public String getUsertoplevel() {
return _usertoplevel;
}
#DynamoDBIndexRangeKey(attributeName = "usertopscore", globalSecondaryIndexName = "usertopscore")
public String getUsertopscore() {
return _usertopscore;
}

CommonDomain - how to unit test aggregate root

I have a small system that uses Jonathan Oliver's CommonDomain and EventStore.
How can I unit test my aggregate roots in order to verify that correct events are raised?
Consider following aggregate root:
public class Subscriber : AggregateBase
{
private Subscriber(Guid id)
{
this.Id = id;
}
private Subscriber(Guid id, string email, DateTimeOffset registeredDate)
: this(id)
{
this.RaiseEvent(new NewSubscriberRegistered(this.Id, email, registeredDate));
}
public string Email{ get; private set; }
public DateTimeOffset RegisteredDate { get; private set; }
public static Subscriber Create(Guid id, string email, DateTimeOffset registeredDate)
{
return new Subscriber(id, email, registeredDate);
}
private void Apply(NewSubscriberRegistered #event)
{
this.Email = #event.Email;
this.RegisteredDate = #event.RegisteredDate;
}
}
I would like to write a following test:
// Arrange
var id = Guid.NewGuid();
var email = "test#thelightfull.com";
var registeredDate = DateTimeOffset.Now;
// Act
var subscriber = Subscriber.Create(id, email, registeredDate);
// Assert
var eventsRaised = subscriber.GetEvents(); <---- How to get the events?
// Assert that NewSubscriberRegistered event was raised with valid data
I could set up whole EventStore with memory persistence and synchronous dispatcher, hook up mock event handler and store any published events for verification, but it seems a bit of overkill.
There is an interface IRouteEvents in CommonDomain. Looks like I could mock it to get the events directly from AggregateBase but how would I actually pass it to my Subscriber class? I don't want to 'pollute' my domian with testing-related code.
I've found out that AggregateBase explicitly implements IAggregate interface, which exposes ICollection GetUncommittedEvents(); method.
So the unit test looks like that:
var eventsRaised = ((IAggregate)subscriber).GetUncommittedEvents();
and no dependency on EventStore is required.
I just pushed up NEventStoreExample with code I gathered in various places (StackOverflow, Documently, Greg Young's skillcast).
It's a very basic implementation of NEventStore that uses CommonDomain to rebuild aggregate state and an EventSpecification base test class to test aggregate behaviour.
Here is a fairly simple test fixture that uses NUnit and ApprovalTests to test CommonDomain aggregate roots . (ApprovalTests is not required - just kinda makes life simple).
The assumption is that
1) the fixture is instantiated with an aggregate (perhaps already set in a certain state) along with a series of 'given' events to be applied.
2) the test will then invoke a specific command handler as part of the TestCommand method - current expectation is a Func that returns the command that is handled
3) the aggregate snapshot, commands, and events all contain 'rich' ToString methods
The TestCommand method then compares the expected with the approved interactions within the aggregate.
public class DomainTestFixture<T>
where T : AggregateBase
{
private readonly T _agg;
private readonly StringBuilder _outputSb = new StringBuilder();
public DomainTestFixture(T agg, List<object> giveEvents)
{
_agg = agg;
_outputSb.AppendLine(string.Format("Given a {0}:", agg.GetType().Name));
giveEvents.ForEach(x => ((IAggregate) _agg).ApplyEvent(x));
_outputSb.AppendLine(
giveEvents.Count == 0
? string.Format("with no previously applied events.")
: string.Format("with previously applied events:")
);
giveEvents.ForEach(x => _outputSb.AppendLine(string.Format(" - {0}", x)));
((IAggregate) _agg).ClearUncommittedEvents();
var snapshot = ((IAggregate) _agg).GetSnapshot();
_outputSb.AppendLine(string.Format("which results in the state: {0}", snapshot));
}
public void TestCommand(Func<T, object> action)
{
var cmd = action.Invoke(_agg);
_outputSb.AppendLine(string.Format("When handling the command: {0}", cmd));
_outputSb.AppendLine(string.Format("Then the {0} reacts ", _agg.GetType().Name));
var raisedEvents = ((IAggregate) _agg).GetUncommittedEvents().Cast<object>().ToList();
_outputSb.AppendLine(
raisedEvents.Count == 0
? string.Format("with no raised events")
: string.Format("with the following raised events:")
);
raisedEvents.ForEach(x => _outputSb.AppendLine(string.Format(" - {0}", x)));
var snapshot = ((IAggregate) _agg).GetSnapshot();
var typ = snapshot.GetType();
_outputSb.AppendLine(string.Format("and results in the state: {0}", snapshot));
Approvals.Verify(_outputSb.ToString());
Assert.Pass(_outputSb.ToString());
}
}
and an example usage
[Test]
public void Test_Some_Aggregate_Handle_Command()
{
var aggId = Guid.Empty;
var tester = new DomainTestFixture<PartAggregate>(
new PartAggregate(aggId, null),
new List<object>()
{
new PartOrdered(),
new PartReceived()
}
);
tester.TestCommand(
(agg) =>
{
var cmd = new RejectPart();
agg.Handle(cmd);
return cmd;
});
}