In my project, I want to fetch data by GSI. And my GSI is such that it will always be unique, so it will return at most one item at a time. But when I used #DynamoDBIndexHashKey annotation on the field that I have made GSI, I get a ResourceNotFoundException:
"Exception: | Exception : class com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException | Exception message : Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException;"
But if I only use #DynamoDBAttribute which is to say only defines the column, everything works fine.
Repository:
#EnableScan
public interface MyRepository extends DynamoDBCrudRepository< MyModel, String> {
MyModel findByMetaData(String metaData);
MyModel findByPrimaryKey(String primaryKey);
}
Model:
#Data
#DynamoDBTable(tableName = "my_model")
public class MyModel {
#DynamoDBHashKey(attributeName = "primary_key")
#DynamoDBAttribute(attributeName = "primary_key")
private String primaryKey;
#DynamoDBIndexHashKey(attributeName = "meta_data", globalSecondaryIndexName = "meta_data")
#DynamoDBAttribute(attributeName = "meta_data")
private String metaData;
And this is how DynamoDB client is being created.
#Bean
public AmazonDynamoDB amazonDynamoDB(AWSCredentialsProvider awsCredentialsProvider,
#Value("${aws.region}") String region) {
return AmazonDynamoDBClientBuilder.standard()
.withCredentials(awsCredentialsProvider)
.withRegion(region)
.build();
}
#Bean
#Primary
public DynamoDBMapper mapper(AmazonDynamoDB amazonDynamoDB,
#Value("${table.name.prefix}") String tableNamePrefix) {
DynamoDBMapperConfig config = new DynamoDBMapperConfig.Builder().
withTableNameOverride(DynamoDBMapperConfig.TableNameOverride.withTableNamePrefix(tableNamePrefix + "-"))
.build();
return new DynamoDBMapper(amazonDynamoDB, config);
}
Also find by primary partition key is working fine.
I have created a secondary index "meta_data" on my table too.
So is there a problem in code or in the way I have created global secondary index in AWS?
Related
In AWS DynamoDB, There are two options available to do the CRUD operations on the Table.
DynamoDBMapper :
com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;.
AmazonDynamoDB dbClient = AmazonDynamoDBAsyncClientBuilder.standard().withCredentials(creds)
.withRegion("us-east-1").build();
// creds is AWSCredentialsProvider
DynamoDBMapper mapper = new DynamoDBMapper(dbClient);
mapper.save(item);
Table: com.amazonaws.services.dynamodbv2.document.Table;.
static DynamoDB dynamoDB =new DynamoDB(dbClient);
Table table = dynamoDB.getTable("TABLE_NAME");
Item item =new Item().withPrimaryKey("","")
.withString("":, "");
table.putItem(item);
Both seem to do the same operations.
Is DynamoDBMapper a layer over Table? If so what are the differences in using each of these?
If you want to map Java classes to DynamoDB tables (which is a useful feature), consider moving away from the old V1 API (com.amazonaws.services.dynamodbv2 is V1). V2 packages are software.amazon.awssdk.services.dynamodb.*.
Replace this old API with the DynamoDB V2 Enhanced Client. You can learn about this here:
Map items in DynamoDB tables
You can find code examples for using the Enhanced Client here.
Here is a Java V2 code example that shows you how to use the Enhanced Client to put data into a Customer table. As you see, you can map a Java Class to columns in a DynamoDB table and then create a Customer object when adding data to the table.
package com.example.dynamodb;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import java.time.Instant;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
/*
* Prior to running this code example, create an Amazon DynamoDB table named Customer with a key named id and populate it with data.
* Also, ensure that you have setup your development environment, including your credentials.
*
* For information, see this documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class EnhancedPutItem {
public static void main(String[] args) {
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.build();
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
putRecord(enhancedClient) ;
ddb.close();
}
// Puts an item into a DynamoDB table
public static void putRecord(DynamoDbEnhancedClient enhancedClient) {
try {
DynamoDbTable<Customer> custTable = enhancedClient.table("Customer", TableSchema.fromBean(Customer.class));
// Create an Instant
LocalDate localDate = LocalDate.parse("2020-04-07");
LocalDateTime localDateTime = localDate.atStartOfDay();
Instant instant = localDateTime.toInstant(ZoneOffset.UTC);
// Populate the Table
Customer custRecord = new Customer();
custRecord.setCustName("Susan Blue");
custRecord.setId("id103");
custRecord.setEmail("sblue#noserver.com");
custRecord.setRegistrationDate(instant) ;
// Put the customer data into a DynamoDB table
custTable.putItem(custRecord);
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
System.out.println("done");
}
#DynamoDbBean
public static class Customer {
private String id;
private String name;
private String email;
private Instant regDate;
#DynamoDbPartitionKey
public String getId() {
return this.id;
};
public void setId(String id) {
this.id = id;
}
#DynamoDbSortKey
public String getCustName() {
return this.name;
}
public void setCustName(String name) {
this.name = name;
}
public String getEmail() {
return this.email;
}
public void setEmail(String email) {
this.email = email;
}
public Instant getRegistrationDate() {
return regDate;
}
public void setRegistrationDate(Instant registrationDate) {
this.regDate = registrationDate;
}
}
}
I have table in AWS mobile hub and I am using the following model for it
public class UserstopcoreDO {
private String _userId;
private String _usertoplevel;
private String _usertopscore;
private String _username;
#DynamoDBHashKey(attributeName = "userId")
#DynamoDBAttribute(attributeName = "userId")
public String getUserId() {
return _userId;
}
public void setUserId(final String _userId) {
this._userId = _userId;
}
#DynamoDBAttribute(attributeName = "usertoplevel")
public String getUsertoplevel() {
return _usertoplevel;
}
#DynamoDBAttribute(attributeName = "username")
public String getUsername() {
return _username;
}
public void setUsername(final String _username) {
this._username = _username;
}
public void setUsertoplevel(final String _usertoplevel) {
this._usertoplevel = _usertoplevel;
}
#DynamoDBIndexHashKey(attributeName = "usertopscore", globalSecondaryIndexName = "usertopscore")
public String getUsertopscore() {
return _usertopscore;
}
public void setUsertopscore(final String _usertopscore) {
this._usertopscore = _usertopscore;
}
}
In the table, I have 1500+ records and now I want to fetch Top 10 records from it so for that I write the below query
final DynamoDBQueryExpression<UserstopcoreDO> queryExpression = new DynamoDBQueryExpression<>();
queryExpression.withLimit(10);
queryExpression.setScanIndexForward(false);
final PaginatedQueryList<UserstopcoreDO> results = mapper.query(UserstopcoreDO.class, queryExpression);
Iterator<UserstopcoreDO> resultsIterator = results.iterator();
if (resultsIterator.hasNext()) {
final UserstopcoreDO item = resultsIterator.next();
try {
Log.d("Item :",item.getUsertopscore());
} catch (final AmazonClientException ex) {
Log.e(LOG_TAG, "Failed deleting item : " + ex.getMessage(), ex);
}
}
But when I run the code it gives me an error
Caused by: java.lang.IllegalArgumentException: Illegal query expression: No hash key condition is found in the query
but in my condition, I did not need any condition because I want to fetch top 10 records instead of one specific record. So how to handle that condition ?
If you want to "query" DynamoDB without specifying all HashKeys, use a Scan instead, i.e. DynamoDBScanExpression. You probably also want to change your "usertopscore" to be a RangeKey instead of a HashKey.
From https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/datamodeling/DynamoDBQueryExpression.html every DynamoDBQueryExpression requires all the Hash Keys be set.
Also see boto dynamodb2: Can I query a table using range key only?
Please set the hash key in the query expression. Below is the example of query expression for main table and GSI (need to set the index name).
Querying the main table:-
Set the hash key value of the table.
UserstopcoreDO hashKeyObject = new UserstopcoreDO();
hashKeyObject.setUserId("1");
DynamoDBQueryExpression<UserstopcoreDO> queryExpressionForMainTable = new DynamoDBQueryExpression<UserstopcoreDO>()
.withHashKeyValues(hashKeyObject);
Querying the Index:-
Set the index name and hash key value of the index.
UserstopcoreDO hashIndexKeyObject = new UserstopcoreDO();
hashIndexKeyObject.setUsertoplevel("100");
DynamoDBQueryExpression<UserstopcoreDO> queryExpressionForGsi = new DynamoDBQueryExpression<UserstopcoreDO>()
.withHashKeyValues(hashIndexKeyObject).withIndexName("usertopscore");
GSI attributes in mapper:-
#DynamoDBIndexHashKey(attributeName = "usertoplevel", globalSecondaryIndexName = "usertopscore")
public String getUsertoplevel() {
return _usertoplevel;
}
#DynamoDBIndexRangeKey(attributeName = "usertopscore", globalSecondaryIndexName = "usertopscore")
public String getUsertopscore() {
return _usertopscore;
}
I'm trying to reason about the cause of a ConditionalCheckFailedException I receive when using DynamoDBMapper with a specific save expression and with UPDATE_SKIP_NULL_ATTRIBUTES SaveBehavior.
My schema is as follows:
Member.java
#Data
#DynamoDBTable(tableName = "members")
public class Member implements DDBTable {
private static final String GROUP_GSI_NAME = "group-gsi";
#DynamoDBHashKey
#DynamoDBAutoGeneratedKey
private String memberId;
#DynamoDBVersionAttribute
private Long version;
#DynamoDBIndexHashKey(globalSecondaryIndexName = GROUP_GSI_NAME)
private String groupId;
#DynamoDBAutoGeneratedTimestamp(strategy = DynamoDBAutoGenerateStrategy.CREATE)
#DynamoDBIndexRangeKey(globalSecondaryIndexName = GROUP_GSI_NAME)
private Date joinDate;
#DynamoDBAttribute
private String memberName;
#Override
#DynamoDBIgnore
public String getHashKeyColumnName() {
return "memberId";
}
#Override
#DynamoDBIgnore
public String getHashKeyColumnValue() {
return memberId;
}
}
I use the following class to create/update/get the records in the members table.
DDBModelDAO.java
public class DDBModelDAO<T extends DDBTable> {
private final Class<T> ddbTableClass;
private final AmazonDynamoDB amazonDynamoDB;
private final DynamoDBMapper dynamoDBMapper;
public DDBModelDAO(Class<T> ddbTableClass, AmazonDynamoDB amazonDynamoDB, DynamoDBMapper dynamoDBMapper) {
this.ddbTableClass = ddbTableClass;
this.amazonDynamoDB = amazonDynamoDB;
this.dynamoDBMapper = dynamoDBMapper;
}
public T loadEntry(final String hashKey) {
return dynamoDBMapper.load(ddbTableClass, hashKey);
}
public T createEntry(final T item) {
dynamoDBMapper.save(item, getSaveExpressionForCreate(item));
return item;
}
public T updateEntry(final T item) {
dynamoDBMapper.save(item, getSaveExpressionForUpdate(item),
DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES.config());
return item;
}
private DynamoDBSaveExpression getSaveExpressionForCreate(final T item) {
// No record with the same hash key must be present when creating
return new DynamoDBSaveExpression().withExpectedEntry(item.getHashKeyColumnName(),
new ExpectedAttributeValue(false));
}
private DynamoDBSaveExpression getSaveExpressionForUpdate(final T item) {
// The hash key for the record being updated must be present.
return new DynamoDBSaveExpression().withExpectedEntry(item.getHashKeyColumnName(),
new ExpectedAttributeValue(new AttributeValue(item.getHashKeyColumnValue()))
.withComparisonOperator(ComparisonOperator.EQ)
);
}
}
I wrote a test class to insert and update records into the members table, which is as follows:
public static void main(String[] args) {
DDBTestClient testClient = new DDBTestClient();
AmazonDynamoDB amazonDynamoDB = testClient.buildAmazonDynamoDB();
DynamoDBMapper dynamoDBMapper = testClient.buildDynamoDBMapper(amazonDynamoDB);
DDBModelDAO<Member> memberDAO = new DDBModelDAO<>(Member.class, amazonDynamoDB, dynamoDBMapper);
DDBModelDAO<Group> groupDAO = new DDBModelDAO<>(Group.class, amazonDynamoDB, dynamoDBMapper);
try {
// Create a group
Group groupToCreate = new Group();
groupToCreate.setGroupName("group-0");
Group createdGroup = groupDAO.createEntry(groupToCreate);
System.out.println("Created group: " + createdGroup);
Thread.sleep(3000);
// Create a member for the group
Member memberToCreate = new Member();
memberToCreate.setGroupId(createdGroup.getGroupId());
memberToCreate.setMemberName("member-0");
Member createdMember = memberDAO.createEntry(memberToCreate);
System.out.println("Created member: " + createdMember);
Thread.sleep(3000);
// Update member name
createdMember.setMemberName("member-updated-0");
createdMember.setGroupId(null);
//createdMember.setJoinDate(null); // <---- Causes ConditionalCheckFailedException
memberDAO.updateEntry(createdMember);
System.out.println("Updated member");
} catch (Exception exception) {
System.out.println(exception.getMessage());
}
}
As can be seen above, if I do not pass a valid value for joinDate(which happens to be the range-key for the groups GSI), in the updateEntry call, DynamoDB returns a ConditionalCheckFailedException. This is the case, even when I use a save behavior of UPDATE_SKIP_NULL_ATTRIBUTES, as can be seen in DDBModelDAO.java.
Can someone help me understand, why I'm required to send the range-key attribute for the GSI, for a conditional write to succeed?
Not sure if this answers your question:
Interface DynamoDBAutoGenerator:
"DynamoDBAutoGenerateStrategy.CREATE, instructs to generate when
creating the item. The mapper, determines an item is new, or
overwriting, if it's current value is null. There is a limitation when
performing partial updates using either,
DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES, or DynamoDBMapperConfig.SaveBehavior.APPEND_SET. A new value will only be generated if the mapper is also generating the key."
So the last part is important: "A new value will only be generated if the mapper is also generating the key"
That should explain why you only see the behavior that you are experiencing.
Does this make sense?
I have my table with UserId(hashkey)+id(Rangekey). they are only together unique as expected.
[DynamoDBTable("Item")]
public class Item
{
[DynamoDBHashKey]
public string UserId { get; set; }
[DynamoDBRangeKey]
public string id { get; set; }
}
now I want to get all the table items for a UserId. I tried as below using xamarin sdk loadasync function with parameter of an existing UserId.
var client = new
Amazon.DynamoDBv2.AmazonDynamoDBClient(AWS.DynamoDBhelper.Credentials,
Amazon.RegionEndpoint.USEast1);
Amazon.DynamoDBv2.DataModel.DynamoDBContext context = new
Amazon.DynamoDBv2.DataModel.DynamoDBContext(client);
Items= await context.LoadAsync<List<Item>>("14354365");
But i get exception as below, it looks like It expects not list because it assumes there should be table with table with list.
How can I achieve this?
There is also QueryAsync function but I am not sure what is the difference between LoadAsync.
ex = {System.InvalidOperationException: Must have one hash key defined
for the table List`1
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.MakeKey (System.Object
hashKey, System.Object rangeKey,
Amazon.DynamoDBv2.DataModel.ItemStorageConfig storageConfig, Amaz...
Your code:
var client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(
AWS.DynamoDBhelper.Credentials,
Amazon.RegionEndpoint.USEast1
);
Amazon.DynamoDBv2.DataModel.DynamoDBContext context = new
Amazon.DynamoDBv2.DataModel.DynamoDBContext(client);
Items= await context.LoadAsync<List<Item>>("14354365");`
Should read:
var client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(
AWS.DynamoDBhelper.Credentials,
Amazon.RegionEndpoint.USEast1
);
Amazon.DynamoDBv2.DataModel.DynamoDBContext context = new
Amazon.DynamoDBv2.DataModel.DynamoDBContext(client);
items= await context.QueryAsync<Item>("14354365").GetRemainingAsync();
You should be able to use LoadAsync to get an object by both its hash + range key.
The .NET library has this:
public Task<T> LoadAsync<T>(
object hashKey,
object rangeKey,
DynamoDBOperationConfig operationConfig,
CancellationToken cancellationToken = default(CancellationToken))
Use Load when the method requires only the primary key of the item you want to retrieve.
I have 2 simple tables.
create table Owner
(
Id int primary key,
Name nvarchar(100),
);
create table Status
(
Id int primary key,
BrandName nvarchar(50)
OwnerId int foreign key references Owner(Id),
);
In app I map these tables to model classes:
public class Owner
{
public int Id {get;set;}
public string Name{get;set;}
public Status Status {get;set;}
}
public class Status
{
public int Id {get;set;}
public string Brand {get;set;}
public int OwnerId {get;set;}
}
I use dapper and dapper extension.
I would like map relation one to one in dapper in classmapper. It’s possible?
My goal is when I added owner object which has set up also property Status to db via repository
it also insert record do status table.
What is best way to achieve this behavior?
public class OwnerMapper : ClassMapper<Owner>
{
public OwnerMapper()
{
Table("Owner");
Map(p=>p.Id).Column("Id").Key(KeyType.Assigned);
Map(p=>p.Name).Column("Name");
//how map property status
}
}
public class StatusMapper : ClassMapper<Status>
{
public StatusMapper()
{
Table("Status");
Map(p=>p.Id).Column("Id").Key(KeyType.Identity);
Map(p=>p.Brand).Column("BrandName");
Map(p=>OwnerId).Column("OwnerId");
}
}
You can try Dapper's multi-mapping feature:
[Test]
public void MyTest()
{
var connection = new SqlConnection("conn string here");
connection.Open();
const string sql =
"select Id = 1, Name ='Bill Gates', Id = 1, Brand = 'Apple', OwnerId = 1";
var result = connection.Query<Owner, Status, Owner>(sql,
(owner, status) =>
{
owner.Status = status;
return owner;
},
commandType: CommandType.Text
).FirstOrDefault();
Assert.That(result, Is.Not.Null);
Assert.That(result.Status, Is.Not.Null);
Assert.That(result.Status.Brand, Is.EqualTo("Apple"));
connection.Close();
}