DyanmoDB can't satisfy KeySchema - amazon-web-services

I am trying to build a DynamoDB table using boto,which will save the various aspects of an IAM policy in the table. I have defined the attributes for keyschema, I do not understand the error.I am very new to DYanmoDB and AWS. This is my code:
table =dynamodb.create_table(
TableName='GoodTable',
KeySchema=[
{
'AttributeName': 'Name',
'KeyType': 'HASH'
},
{
'AttributeName': 'Instance Type',
'KeyType': 'RANGE'
},
{
'AttributeName': 'Region',
'KeyType': 'RANGE'
},
{
'AttributeName': 'Volume Size',
'KeyType': 'RANGE'
},
],
AttributeDefinitions=[
{
"AttributeName": "Name",
"AttributeType": "S"
},
{
"AttributeName": "Instance Type",
"AttributeType": "S"
},
{
"AttributeName": "Region",
"AttributeType": "S"
},
{
"AttributeName": "Volume Size",
"AttributeType": "N"
}
],
ProvisionedThroughput={
"ReadCapacityUnits": 1,
"WriteCapacityUnits": 1
}
)
time.sleep(20)
table = dynamodb.Table('GoodTable')
response = table.put_item(
Item= {
'Name': 'GoodName',
}
)
response = table.put_item(
Item= {
'Instance Type': 't2.micro',
}
)
response = table.put_item(
Item= {
'Region': 'us-east-1',
}
)
response = table.put_item(
Item= {
'Volume Size': '20',
}
)
This is the error I am getting:
botocore.exceptions.ClientError: An error occurred (ValidationException) when
calling the CreateTable operation: 1 validation error detected: Value '[com.amazonaws.dynamodb.v20120810.KeySchemaElement#ad4dcbcd, com.amazonaws.dynamodb.v20120810.KeySchemaElement#126b7ad8, com.amazonaws.dynamodb.v20120810.KeySchemaElement#ca666a07, com.amazonaws.dynamodb.v20120810.KeySchemaElement#6478bc3a]' at 'keySchema' failed to satisfy constraint: Member must have length less than or equal to 2

You can only have 2 fields as a primary key in DynamoDB. You can have only one Hash Key and One Range Key Max.
CreateTable
For a composite primary key (partition key and sort key), you must provide exactly two elements, in this order: The first element must have a KeyType of HASH, and the second element must have a KeyType of RANGE.
You can setup Secondary Indexes in DynamoDB

There are two issues in your code:
As already pointed out, you can't have more then two key attributes unless using global or local secondary indies.
dynamodb.Table('GoodTable') is incorrect as you need resource.
You can check the modified code:
table = dynamodb.create_table(
TableName='GoodTable',
KeySchema=[
{
'AttributeName': 'Name',
'KeyType': 'HASH'
}
],
AttributeDefinitions=[
{
"AttributeName": "Name",
"AttributeType": "S"
}
],
ProvisionedThroughput={
"ReadCapacityUnits": 1,
"WriteCapacityUnits": 1
}
)
import time
time.sleep(20)
table = boto3.resource('dynamodb').Table('GoodTable')
response = table.put_item(
Item= {
'Name': 'GoodName',
'Instance Type': 't2.micro',
}
)
response = table.put_item(
Item= {
'Name': 'GoodName2',
'Instance Type': 't2.micro',
}
)
response = table.put_item(
Item= {
'Name': 'GoodName3',
'Region': 'us-east-1',
}
)
response = table.put_item(
Item= {
'Name': 'GoodName4',
'Volume Size': '20',
}
)

The field name is "KeySchema" not "TableSchema" or anything else, it defines only the key. The table is "schema-less", which means that each record can have a different structure and there is no need to define it. You must define only the key. In DynamoDB the key is either only a HASH column or HASH + RANGE columns. You should think about which of those two possibilities you want to use. If you use HASH + RANGE you have to query the table with both as well. Reading many records is costly.
So think a bit about what you want to store and how you would query that. Design the hash key accordingly.
There is a big argument from the single table complex hash key data model by the AWS Principal NoSQL technologist Rick Houlihan https://youtu.be/HaEPXoXVf2k?t=2573 . When I watched the video I started designing my DynamoDB tables differently and it improved my life.
Then natural tendency is to select one column which is kind of unique and use it as a hash key, but it really limits your query options. A well designed hash key can help you querying without additional indexes, so your solution is cheaper and more efficient.
As I mentioned above - beside the key, there is no structure defined. But it does not mean that each record should be completely random. However, it does make sense to store multiple item types in one table where each item type has the same structure - see the video, it's worth it.
In your case - using the instance name as a hash can be risky, because it may not be unique across the regions. You can even have two instances with the same name in the same region, because the name is just a tag. If you do not know or do not want to store the instance ID you have to come up with some other clever solution.
For example the hash can be: INSTANCE:: and the sort key can be instance creation time. There is an additional work to compose and decompose the key for each record. I solved it by creating a python class which wraps put_item/get_item in a method which handles the keys.

Related

Amazon SP-API Listings API putListingsItem How To Update price and quantity? Node.js

I am using amazon-sp-api (JavaScript client for the Amazon Selling Partner API) but this is not limited to this client. All I want to do is use the Amazon SP-API Listings API's putListingsItem call to update the price and quantity of an item I have listed.
productType
According to the ListingsItemPutRequest docs, productType and attributes are required for this call.
Firstly, to obtain the correct productType value, you are supposed to search for a product definitions type using the Product Type Definitions API. So, I do that, and call searchDefinitionsProductTypes, just to discover my product has no matching product type.
Ultimately, I gave the value PRODUCT for productType field. Using PRODUCT, I made the getDefinitionsProductType call and got an object containing an array of propertyNames, shown below:
"propertyNames": [
"skip_offer",
"fulfillment_availability",
"map_policy",
"purchasable_offer",
"condition_type",
"condition_note",
"list_price",
"product_tax_code",
"merchant_release_date",
"merchant_shipping_group",
"max_order_quantity",
"gift_options",
"main_offer_image_locator",
"other_offer_image_locator_1",
"other_offer_image_locator_2",
"other_offer_image_locator_3",
"other_offer_image_locator_4",
"other_offer_image_locator_5"
]
},
On seeing this, I decide list_price and fulfillment_availability must be the price and quantity and then try using these in my code below.
attributes
The attributes value is also required. However, their current docs show no clear example of what to put for these values, which are where I must put price and quantity somewhere.
I found this link about patchListingsItem and tried to implement that below but got an error.
code:
// trying to update quantity... failed.
a.response = await a.sellingPartner.callAPI({
operation:'putListingsItem',
path:{
sellerId: process.env.SELLER_ID,
sku: `XXXXXXXXXXXX`
},
query: {
marketplaceIds: [ `ATVPDKIKX0DER` ]
},
body: {
"productType": `PRODUCT`
"requirements": "LISTING_OFFER_ONLY",
"attributes": {
"fulfillment_availability": {
"fulfillment_channel_code": "AMAZON_NA",
"quantity": 4,
"marketplace_id": "ATVPDKIKX0DER"
}
}
});
console.log( `a.response: `, a.response )
error:
{
"sku": "XXXXXXXXXXXX",
"status": "INVALID",
"submissionId": "34e1XXXXXXXXXXXXXXXXXXXX",
"issues": [
{
"code": "4000001",
"message": "The provided value for 'fulfillment_availability' is invalid.",
"severity": "ERROR",
"attributeName": "fulfillment_availability"
}
]
}
I also tried using list_price :
// list_price attempt... failed.
a.response = await a.sellingPartner.callAPI({
operation:'putListingsItem',
path:{
sellerId: process.env.SELLER_ID,
sku: `XXXXXXXXXXXX`
},
query: {
marketplaceIds: [ `ATVPDKIKX0DER` ]
},
body: {
"productType": `PRODUCT`
"requirements": "LISTING_OFFER_ONLY",
"attributes": {
"list_price": {
"Amount": 90,
"CurrencyCode": "USD"
}
});
console.log( `a.response: `, a.response )
Error (this time seems I got warmer... maybe?):
{
"sku": "XXXXXXXXXXXX",
"status": "INVALID",
"submissionId": "34e1XXXXXXXXXXXXXXXXXXXX",
"issues": [
{
"code": "4000001",
"message": "The provided value for 'list_price' is invalid.",
"severity": "ERROR",
"attributeName": "list_price"
}
]
}
How do you correctly specify the list_price or the quantity so this call will be successful?
Just tryin to update a single item's price and quantity.
The documentation for this side of things is terrible. I've managed to get some of it through a fair bit of trial and error though.
Fulfillment and Availability can be set with this block of JSON
"fulfillment_availability": [{
"fulfillment_channel_code": "DEFAULT",
"quantity": "9999",
"lead_time_to_ship_max_days": "5"
}]
and List price gets set, oddly, with this block. I'm still trying to find out how to set the List Price with Tax however.
"purchasable_offer": [{
"currency": "GBP",
"our_price": [{"schedule": [{"value_with_tax": 285.93}]}],
"marketplace_id": "A1F83G8C2ARO7P"
}]
Hope this helps you out :)

How to Search Nested Array of Object in DynamoDB

I am new to dynamoDb, i would like to search nested array properties. For ex my table has sample data given below
[{
id: '123',
name: 'test',
subShops: [
{
shopId: '234',
shopName: 'New Shop'
},
{
shopId: '345',
shopName: 'New Shop 2'
}
]
},
{
id: '1234',
name: 'test2',
subShops: [
{
shopId: '2345',
shopName: 'New Shop 3'
},
{
shopId: '3456',
shopName: 'New Shop 4'
}
]
}
]
I want to search where name : ['test', 'test2', 'test3'] or subShops[].shopeName where ['New Shop', 'New Shop 2', ''New Shop 3].
I have existing code for only name : ['test', 'test2', 'test3']
const params: AWS.DynamoDB.DocumentClient.ScanInput = {
TableName: VENDOR_TABLE_INFO.Name,
ExpressionAttributeNames: { "#Id": "name" },
FilterExpression: `#Id in (${Object.keys(keyValues).toString()}) or contains (subShops, :category2)`,
ExpressionAttributeValues: {
...keyValues,
':category2': {
...keyValues
}
}
};
Please notice that DynamoDB (DDB) is mainly a hyperscale key-value serverless datastore with very limited query pattern and flexibility, you need to be ok with that to use it.
In each DDB table you can only define one hash key (pk), and up to 5 local secondary index (sort key) for querying. And you can have up to 20 Global Secondary Index (GSI)
In you example, you have hash key of "id", and then if you need to query by "name" only, you need to build a GSI with name as hash key, and included the needed fields in the projection. There is no way to query by "shopname" in sub shop array unless you "flaten" the JSON tree structure.
In short, if you want JSON tree level data query/manipulation and all of your data is JSON documents, i would suggest you to use Amazon DocumentDB which is MongoDB 4 compatible, or directly use MongoDB itself.

Not Getting total cost in AWS boto3 costExplorer get_cost_and_usage() function

I'm calling boto3 get_cost_and_usage function with below request to fetch daily cost grouped by services and "Name" tag:
result = client.get_cost_and_usage(
TimePeriod = {
'Start': '2020-10-31',
'End': '2020-11-02'
},
Granularity = 'DAILY',
Metrics = ["BlendedCost"],
GroupBy = [
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'TAG',
'Key': 'Name'
}
]
)
In the response, I'm getting all the costs for each service and Name tags for each day, however the total cost for each day is empty [ "Total": {} ]:
"ResultsByTime": [
{
"Estimated": true,
"Groups": [
{
...
}
"TimePeriod": {
"End": "2020-11-01",
"Start": "2020-10-31"
},
"Total": {}
},
...
]
Please tell me if anything is wrong here? I also tried with "UnblendedCost", "AmortizedCost" in Metrics and grouped by only services but it is the same issue i.e., total cost is not coming.
Kindly help me out here.
I encountered the same issue with Groupby not getting the total amount.
I am calculating the total cost by adding every service cost in the loop like below.
I added line below inside the for loop:
total_cost=float(cost_spent_on_each_service)+total_cost
and declared total_cost = 0 outside of the loop.

An error occurred (ResourceInUseException) when calling the CreateTable operation: Table already exists:

Getting below error while creating a table in dynamodb to put data from s3 bucket and load into the table having more than 4-columns.
error in cloudwatchlogs:"module initialization error: An error occurred (ResourceInUseException) when calling the CreateTable operation: Table already exists: "
sample code:
import boto3
s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
def txt_reader(event,context):
bucket_path = event['Records'][0]['s3']['bucket']['name']
key_path = event['Records'][0]['s3']['object']['key']
obj = s3.get_object(Bucket = bucket_path,Key = key_path)
body_rows = obj['Body'].read().decode('utf-8').split('\n')
# Create the DynamoDB table.
table_name = dynamodb.create_table(
TableName='TFM',
KeySchema=[
{
'AttributeName': 'CN',
'KeyType': 'HASH'
},
{
'AttributeName': 'SN',
'KeyType': 'RANGE'
}
],
AttributeDefinitions=[
{
'AttributeName': 'CN',
'AttributeType': 'S'
},
{
'AttributeName': 'SN',
'AttributeType': 'S'
},
{
'AttributeName': 'WF',
'AttributeType': 'S'
},
],
#defining local secondary index on column WF
LocalSecondaryIndexes=[
{
'IndexName': 'WF',
'KeySchema': [
{
'KeyType': 'HASH',
'AttributeName': 'CN'
},
{
'KeyType': 'RANGE',
'AttributeName': 'WF'
}
],
'Projection': {
'ProjectionType': 'ALL',
}
}
],
ProvisionedThroughput={
'ReadCapacityUnits': 5,
'WriteCapacityUnits': 5
}
)
table=dynamodb.Table(table_name)
#using a method batch_writer as batch below
with table.batch_writer() as batch:
for row in body_rows:
batch.put_item(Item = {
'CN':row.split('|')[0],
'SN':row.split('|')[1],
'WF':row.split('|')[2],
'sf':row.split('|')[3],
'Con':row.split('|')[4],
'LCI':row.split('|')[5]
})
MY Queries: Please help me by taking some random '|' seperated values for the given columns in a txt file and run the code in lambda.
Note: Services to be used are Dynamodb as a resource, S3 as a Client. In this case, i am getting the error but i can see, everytime i save the code and upload the txt file in s3, table is getting created also and then having this above given error. I delete the table, save the lambda code and upload the file in S3 and again the same error i get. Here S3 is acting as a trigger. I have created one S3-lambda-cloudwatchlogs-dynamodb role also.
already given above
Normally, table create is one time operation in most cases. You should create table in another script, have it run once and then run the other part of the code on lambda.
If anyways, you have to create and destroy table every time, then check the status of the table prior to put_item operation. Because once you execute table create API, if not be created instantaneously. Its a request to create table, which will take a while to create. Here is the doc describing table_status, based on which can take decision when you should run put_item operation
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.table_status

dynamodb: Conditionally create/update an item and return current value

Is it possible in dynamoDB to conditionally create/update an item and return its current value?
In the below we have three states, one in which the user is not in the table yet, one in which the user is in the table but does not have a CreatedAt field, and one in which the user exists and has a CreatedAt field.
out, err := db.UpdateItem(&dynamodb.UpdateItemInput{
TableName: aws.String("User"),
Key: map[string]*dynamodb.AttributeValue{
"ID": {S: aws.String(u.ID)},
},
ExpressionAttributeNames: map[string]*string{
"#c": aws.String("CreatedAt"),
},
ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
":c": {S: aws.String(u.CreatedAt.Format(time.RFC3339))},
}
ConditionExpression: aws.String("attribute_not_exists(#c)"),
UpdateExpression: aws.String("SET #c = :c"),
ReturnValues: aws.String("ALL_NEW"),
})
if isConditionError(err) {
// out.Attributes == nil
// .: out.Attributes["CreatedAt"] == ""
}
When the user is not present, this creates the user as expected.
When the user does not have a CreatedAt field, it adds that field as expected.
When the condition fails (i.e. the user has a value for CreatedAt), I would like to know what the database says in the CreatedAt field for that user. Unfortunately regardless whether I specify ALL_NEW or ALL_OLD for ReturnValues, out.Attributes is always nil.
I would like to avoid calling the database twice to get this information, and since I know dynamo is reading it to evaluate the condition I think this is reasonable. How would I do this?