DynamoDb scan not working properly - amazon-web-services

I am trying to fetch items from the dynamoDb. However when I fetch single item using partiton key everything works fine but when I try to fetch all the items in dynamo db using scan I encouter an error. Following is the code what I am trying to do:
public List<PageCmsDomain> getAllPages() {
DynamoDBUtil dynamoDBUtil = new DynamoDBUtil();
AmazonDynamoDB dynamoDBClient = dynamoDBUtil.getDynamoDBClient();
DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient);
List<PageCmsDomain> scanResult = mapper.scan(PageCmsDomain.class, new
DynamoDBScanExpression());
return scanResult;
Following error comes when I execute this code:
com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException: PageCmsDomain[restrictions]; could not unconvert attribute
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperTableModel.unconvert(DynamoDBMapperTableModel.java:271)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.privateMarshallIntoObject(DynamoDBMapper.java:456)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.marshallIntoObjects(DynamoDBMapper.java:484)
at com.amazonaws.services.dynamodbv2.datamodeling.PaginatedScanList.<init>(PaginatedScanList.java:64)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.scan(DynamoDBMapper.java:1458)
at com.amazonaws.services.dynamodbv2.datamodeling.AbstractDynamoDBMapper.scan(AbstractDynamoDBMapper.java:216)
at com.astro.ott.dynamodb.cmsapi.impl.PageCmsDaoImpl.getAllPages(PageCmsDaoImpl.java:74)
at com.astro.ott.cmsapi.impl.PageCmsGetServiceImpl.getPages(PageCmsGetServiceImpl.java:41)
at com.astro.ott.cms_api.cms_api.PageCmsTestCase.get(PageCmsTestCase.java:81)
Caused by: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException: not supported; requires #DynamoDBTyped or #DynamoDBTypeConverted
at com.amazonaws.services.dynamodbv2.datamodeling.StandardModelFactories$Rules$NotSupported.get(StandardModelFactories.java:660)
at com.amazonaws.services.dynamodbv2.datamodeling.StandardModelFactories$Rules$NotSupported.get(StandardModelFactories.java:650)
at com.amazonaws.services.dynamodbv2.datamodeling.StandardModelFactories$AbstractRule.unconvert(StandardModelFactories.java:714)
at com.amazonaws.services.dynamodbv2.datamodeling.StandardModelFactories$AbstractRule.unconvert(StandardModelFactories.java:691)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTypeConverter$DelegateConverter.unconvert(DynamoDBTypeConverter.java:109)
at com.amazonaws.services.dynamodbv2.datamodeling.StandardTypeConverters$Vector$ToMap.unconvert(StandardTypeConverters.java:433)
at com.amazonaws.services.dynamodbv2.datamodeling.StandardTypeConverters$Vector$ToMap$1.unconvert(StandardTypeConverters.java:417)
at com.amazonaws.services.dynamodbv2.datamodeling.StandardTypeConverters$Vector$ToMap$1.unconvert(StandardTypeConverters.java:410)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTypeConverter$DelegateConverter.unconvert(DynamoDBTypeConverter.java:109)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTypeConverter$NullSafeConverter.unconvert(DynamoDBTypeConverter.java:128)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTypeConverter$ExtendedConverter.unconvert(DynamoDBTypeConverter.java:88)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperFieldModel.unconvert(DynamoDBMapperFieldModel.java:146)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperFieldModel.unconvertAndSet(DynamoDBMapperFieldModel.java:164)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperTableModel.unconvert(DynamoDBMapperTableModel.java:267)
... 32
And I am using same PageCmsDomain while fetching single item also.

Specified code will work only if restrictions field is null or empty map. Otherwise it will throw exception as DynamoDB does not allow to save or fetch objects that have field with Object type (even as a generic type, in this case - map's value type Object), instead it should have specific type (e.g. String, Integer, Map<String, Integer>).
If restrictions map's value type can't be changed to specific one,
you can fix that by adding DynamoDBTypeConvertedJson annotation to store serialized value of restrictions field:
#DynamoDBAttribute(attributeName = "restrictions")
#DynamoDBTypeConvertedJson
private Map<String, Object> restrictions;
Another option is to specify your own custom converter class and annotate restrictions field with
#DynamoDBTypeConverted(converter = CustomConverter.class)

Related

Update specific attribute using UpdateItemEnhancedRequest DynamoDb java sdk2

We have a DynamoDB table which has an attribute counter, which will be decremented asynchronously by multiple lambda based on an event. I am trying to update the counter using UpdateItemEnhancedRequest (using the Dynamodb Enhanced Client. - JAVA SDK 2). I am able to build the condition for updating the counter but it updates the entire item and not just the counter. Can somebody please guide on how to update a single attribute using DynamoDb Enhanced Client?
Code Sample
public void update(String counter, T item) {
AttributeValue value = AttributeValue.builder().n(counter).build();
Map<String, AttributeValue> expressionValues = new HashMap<>();
expressionValues.put(":value", value);
Expression myExpression = Expression.builder()
.expression("nqctr = :value")
.expressionValues(expressionValues)
.build();
UpdateItemEnhancedRequest<T> updateItemEnhancedRequest =
UpdateItemEnhancedRequest.builder(collectionClassName)
.item(item)
.conditionExpression(myExpression)
.build();
getTable().updateItem(updateItemEnhancedRequest);
}
When you update a specific column, you need to specify which column to update. Assume we have this table:
Now assume we want to update the archive column. You need to specify the column in your code. Here we change the archive column of the item that corresponds to the key to Closed (a single column update). Notice we specify the column name by using the HashMap object named updatedValues.
// Archives an item based on the key
public String archiveItem(String id){
DynamoDbClient ddb = getClient();
HashMap<String,AttributeValue> itemKey = new HashMap<String,AttributeValue>();
itemKey.put("id", AttributeValue.builder()
.s(id)
.build());
HashMap<String, AttributeValueUpdate> updatedValues =
new HashMap<String,AttributeValueUpdate>();
// Update the column specified by name with updatedVal
updatedValues.put("archive", AttributeValueUpdate.builder()
.value(AttributeValue.builder()
.s("Closed").build())
.action(AttributeAction.PUT)
.build());
UpdateItemRequest request = UpdateItemRequest.builder()
.tableName("Work")
.key(itemKey)
.attributeUpdates(updatedValues)
.build();
try {
ddb.updateItem(request);
return"The item was successfully archived";
NOTE: This is not the Enhanced Client.
This code is from the AWS Tutorial that show how to build a Java web app by using Spring Boot. Full tutorial here:
Creating the DynamoDB web application item tracker
TO update a single column using the Enhanced Client, call the Table method. This returns a DynamoDbTable instance. Now you can call the updateItem method.
Here is the logic to update the the archive column using the Enhanced Client. Notice you get a Work object, call its setArchive then pass the Work object. workTable.updateItem(r->r.item(work));
Java code:
// Update the archive column by using the Enhanced Client.
public String archiveItemEC(String id) {
DynamoDbClient ddb = getClient();
try {
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(getClient())
.build();
DynamoDbTable<Work> workTable = enhancedClient.table("Work", TableSchema.fromBean(Work.class));
//Get the Key object.
Key key = Key.builder()
.partitionValue(id)
.build();
// Get the item by using the key.
Work work = workTable.getItem(r->r.key(key));
work.setArchive("Closed");
workTable.updateItem(r->r.item(work));
return"The item was successfully archived";
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
return "";
}
This answer shows both ways to update a single column in a DynamoDB table. The above tutorial now shows this way.
In your original solution, you're misinterpreting the meaning of the conditionExpression attribute. This is used to validate conditions that must be true on the item that matches the key in order to perform the update, not the expression to perform the update itself.
There is a way to perform this operation with the enhanced client without needing to fetch the object before making an update. The UpdateItemEnhancedRequest class has an ignoreNulls attribute that will exclude all null attributes from the update. This is false by default, which is what causes a full overwrite of the object.
Let's assume this is the structure of your item (without all the enhanced client annotations and boilerplate, you can add those):
class T {
public String partitionKey;
public Int counter;
public String someOtherAttribute
public T(String partitionKey) {
this.partitionKey = partitionKey;
this.counter = null;
this.someOtherAttribute = null
}
}
You can issue an update of just the counter, and only if the item exists, like this:
public void update(Int counter, String partitionKey) {
T updateItem = new T(partitionKey)
updateItem.counter = counter
Expression itemExistsExpression = Expression.builder()
.expression("attribute_exists(partitionKey)")
.build();
UpdateItemEnhancedRequest<T> updateItemEnhancedRequest =
UpdateItemEnhancedRequest.builder(collectionClassName)
.item(item)
.conditionExpression(itemExistsExpression)
.ignoreNulls(true)
.build();
getTable().updateItem(updateItemEnhancedRequest);
}

Google Cloud Data flow : CloudBigtableScanConfiguration.withScan(), how to pass dynamic filter values?

SourceLocation is prefix for my Bigtable, which is fetched from application.properties. Is there a way to fetch it dynamically while running the data flow template?
My Pipeline:
pipeline.apply("ReadTable", Read.from(CloudBigtableIO.read(configSetUp(options))))
CloudBigtableScanConfiguration
private static CloudBigtableScanConfiguration configSetUp(LocationSetupOptions options) {
ValueProvider<Integer> pageFilter = options.getPageFilter();
Scan scan = new Scan(Bytes.toBytes(options.getSourceLocation().get()));
FilterList filterList = new FilterList();
PrefixFilter prefixFilter = new PrefixFilter(Bytes.toBytes(options.getSourceLocation().get()));
filterList.addFilter(new PageFilter(Long.valueOf(pageFilter.get())));
filterList.addFilter(prefixFilter);
scan.setFilter(filterList);
return new CloudBigtableScanConfiguration.Builder()
.withProjectId(options.getProjectId())
.withInstanceId(options.getInstanceId())
.withTableId(options.getTableId())
.withScan(scan)
.build();}
There are two clients for Bigtable CloudBigtableIO and BigtableIO. The CloudBigtableIO parameters are not updated to be modified by templates via a ValueProvider but BigtableIO is compatible with ValueProviders.
In your particular case if you are looking for ValueProvider to be used along with template then I would recommend that you move to using BigtableIO. A sample can be found here AvroToBigtable.
UPDATE
The #Default.InstanceFactory can be used to specify a user-provided factory method to generate default values for a parameter. With this, you could read the default value from a resource file inside of your DefaultValueFactory implementation.
As an example, you can check how WindowedWordCount defines DefaultToCurrentSystemTime to annotate the minTimestampMillis parameter:

DynamoDb not able to save map data(java)

I am trying to save data into the dynamoDb but that data contains some Map attributes as well.But I am getting error while saving that data. Following is my domain class which I am using for saving data from request:
#DynamoDBTable(tableName = "ottMiddleware_rails")
public class RailsCmsDomain {
#DynamoDBHashKey(attributeName = "railId")
private String railId;
#DynamoDBTyped
#DynamoDBAttribute(attributeName = "railLogic")
private Map<String, Object> railLogic;
#DynamoDBAttribute(attributeName = "railSourceType")
private String railSourceType;
#DynamoDBAttribute(attributeName = "railTitle")
private RailCmsTitleDomain railTitle;
#DynamoDBTyped
#DynamoDBAttribute(attributeName = "restrictions")
private Map<String, Object> restrictions;
I am giving the following request:
{
"railId": "railOne",
"railLogic": {
"programType": 1,
"railSourceUrl": "http://myUrl"
},
"railSourceType": "myRail",
"railTitle": {
"tam": "Raan Phan",
"def": "சிறப்பு கட்டமைப்பு"
},
"restrictions": {
"clients": [
"abc",
"xyz"
],
"periodStart": 1506572217
}
}
I am using following code to save my data into the dynamoDb
public Boolean saveUpdateRailsDetails(RailsCmsDomain railsDomain) {
DynamoDBUtil dynamoDBUtil = new DynamoDBUtil();
AmazonDynamoDB dynamoDBClient = dynamoDBUtil.getDynamoDBClient();
DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient);
mapper.save(railsDomain);
return true;
}
Please suggest how can I save map into dynamoDb. I am taking data as map because in later stages there are chances more data can be added to those attributes which are map and that data can be of any data type. I am getting following error:
errorMessage": "not supported; requires #DynamoDBTyped or
#DynamoDBTypeConverted"
"errorType":
"com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException"
I would like to post this answer as an option though it is not going to store the data as map in DynamoDB. However, it will resolve the error.
You can define the restrictions attribute as mentioned below. It will store the data as JSON string in DynamoDB.
#DynamoDBTypeConvertedJson
private Map<String, Object> restrictions;
Drawbacks:-
When you would like to update the restrictions attribute, you need to get the current value from database, update it and save the data in database.
DynamoDB data:-
restrictions attribute saved as JSON string.
When I needed to store map in dynamodb I just annotated the getter method of the map with #DynamoDBTypeConverted(converter = MapConverter.class) and in the converter just serialized the map into a json string, this way you can retrieve the map and everything but are not going to be able to use it in queryExpressions if you need.
So in case you want to be able to query the objects in the map maybe consider annotating the object you are using in your map with #DynamoDBDocument this way it's going to be automatically serialized by dynamodb as a document and use Set instead of map into your parent object.
The problem is that the DynamoDB Mapper doesn't know how to marshall/unmarshall the Object in this line:
private Map<String, Object> railLogic;
If you don't want to change Object to String, then use the DynamoDBTypeConverted annotation to provide a custom marshaller for Map<String, Object> which you can then (probably) reuse for restrictions. Relevant documentation here.

How to query based on one-to-many relationships in Spring Data Neo4j

I have defined simple one-to-many relationship as:
#Relationship(type = "BELONG")
private Set<Category> categories;
I want to query all the objects based on the exact set of Category. i.e. implement something like:
Page<SomeObject> findAllByCategories(Set<Category> categories, PageRequest page);
What is the best way to do it in Spring Data Neo4j?
You have a couple of options on how you can do this.
Your calling code can simply call the following on your SomeObjectRepository:
Iterable<Long> ids = categories.stream().map(SomeObject::getId).collect(Collectors.toSet());
Iterable<SomeObject> someObjects = someObjectRepository.findAll(ids, 1);
Note this method does not support paging though.
Another way is to inject a Session into your calling code and either write a custom Cypher query to retrieve the objects you want or use the load all by instance method:
#Autowired
private Session session;
...
// Option 1: write some cypher
Map<String, Object> params = new HashMap<>():
params.put("ids", ids) // use a stream like above to collect the ids.
Iterable<SomeObject> someObjects = session.query(SomeObject.class, "MATCH (n:SomeObject) ...", params);
//Option 2: use the load by instances method which also allows pagination:
Collection<SomeObject> someObjects = session.loadAll(categories, new Pagination(0, 25));
My recommendation would be option 2 as it pretty much does exactly what you want.

How do we retrieve values of Custom Lists(dropdown) in Netsuite

I have record type "XYZ" which has field called "award area" which is of type list/record. "award area" is of type custom list and is a drop down control.
Using Suitetalk how can I retrieve those values from that drop down?
Thank you
I think something like this should work. It's for translating the results from the internalId's returned into the actual text type, you maybe be able to leverage it in another way. Maybe you could create a lookup list with something like this(C#):
public Dictionary<string, Dictionary<long, string>> getCustomFieldLists()
{
return
nsService.search(new CustomListSearch())
.recordList.Select(a => (CustomList) a)
.ToDictionary(a => a.name,
a => a.customValueList.customValue
.ToDictionary(b => b.valueId, c => c.value));
}
var valueLookup = getCustomFieldLists()["award area"];
Here's how I did it for myself, because I was irritated with the fact the NetSuite doesn't just provide us an easy way to access these. And I wanted the following data for reference:
The Internal ID of the Custom List
The Name of the Custom List
The Internal ID of the Custom List Item
The name Value of the Custom List Item
I wanted/needed access to all of those things, and I wanted to be able to obtain the name Value of the Custom List Item by just providing the Internal ID of the Custom List and the Internal ID of the Custom List Item. So, in my homemade integration client, similar to David Rogers' answer, but without all the fancy Linq, I figured out that the best solution was a Dictionary>>.
This way, for the outer Dictionary, I could set the key to the internal IDs of the Custom Lists, and for the inner Dictionary I could set the key to the internal IDs of the Custom List Items themselves. Then, I would get the name of the Custom List for "free" as the beginning part of the Tuple, and the actual name Value for "free" as the value of the internal Dictionary.
Below is my method code to generate this object:
/// <summary>
/// Gets the collection of all custom lists, and places it in the public CustomListEntries object
/// </summary>
/// <returns></returns>
private Dictionary<string, Tuple<string, Dictionary<long, string>>> GetCustomLists()
{
Dictionary<string, Tuple<string, Dictionary<long, string>>> customListEntries = new Dictionary<string, Tuple<string, Dictionary<long, string>>>();
SearchPreferences sp = SuiteTalkService.searchPreferences; //Store search preferences to reset back later, just need body fields this one time
SuiteTalkService.searchPreferences = new SearchPreferences() { bodyFieldsOnly = false };
SearchResult sr = SuiteTalkService.search(new CustomListSearch());
SuiteTalkService.searchPreferences = sp; //Restore search preferences
foreach (CustomList cl in sr.recordList)
{
Dictionary<long, string> customListItems = new Dictionary<long, string>();
if (cl.customValueList == null) continue;
foreach (CustomListCustomValue clcv in cl.customValueList.customValue)
{
customListItems.Add(clcv.valueId, clcv.value);
}
customListEntries.Add(cl.internalId, new Tuple<string, Dictionary<long, string>>(cl.name, customListItems));
}
return customListEntries;
}
Then, in the constructors of my Integration class, I can set my object to the return result:
public Dictionary<string, Tuple<string, Dictionary<long, string>>> CustomListEntries = GetCustomLists();
And finally, whenever I need access TO those values, since I set all of this up ahead of time, I can do the following:
dr[Class] = SuiteTalkIntegrator.CustomListEntries[lorr.typeId].Item2[long.Parse(lorr.internalId)];
In this case above, my "lorr" object is a ListOrRecordRef object that I obtained from the SearchColumnSelectCustomField.searchValue from the search results of a SavedSearch. I don't know if this will work for anyone else that finds this code, but since I was frustrated in finding an easy answer to this problem, I thought I'd share my solution with everyone.
Frankly, I'm most frustrated that this functionality isn't just given to us out of the box, but I've noticed that NetSuite has made a lot of bad design choices in their SuiteTalk API, like not making a custom class of "RecordField" for their record fields and not placing their record fields under an IEnumerable of RecordField so that programmers can loop through all values in a record in a generic way without having to EXPLICITLY name them and re-construct the same code logic over and over again... ugh...