I am following aws appsync tutorial and i'm stuck at delta sync step (https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-delta-sync.html).
I have finished the example but the result is not as expected. In the update step, dynamodb does not create 2 records (a record when the item was created and record for when the item was updated) as in the example. And when using delta query, an error message is received:
"data": null,
"errors": [
{
"path": [
"syncPosts"
],
"locations": null,
"message": "Cannot return null for non-nullable type: 'PostConnection' within parent 'Query' (/syncPosts)"
}
]
}
My detal table ttl is 1 min, and delta queries only select from base table, not delta table.
Can someone look into this and help me out? Thanks
I found solution, i was setting the delta table partition_key field to ds_pk and delta table sort_key field to ds_sk. The result is as expected after the changed.
Related
This question is a follow up to another SO question.
Summary: I have an API returning a nested JSON array. Data is being extracted via APEX REST Data Sources. The Row Selector in the Data Profile is set to "." (to select the "root node").
The lines array has been manually added to a column (LINES) to the Data Profile, set data type to JSON Document, and used lines as the selector.
SAMPLE JSON RESPONSE FROM API
[ {
"order_number": "so1223",
"order_date": "2022-07-01",
"full_name": "Carny Coulter",
"email": "ccoulter2#ovh.net",
"credit_card": "3545556133694494",
"city": "Myhiya",
"state": "CA",
"zip_code": "12345",
"lines": [
{
"product": "Beans - Fava, Canned",
"quantity": 1,
"price": 1.99
},
{
"product": "Edible Flower - Mixed",
"quantity": 1,
"price": 1.50
}
]
},
{
"order_number": "so2244",
"order_date": "2022-12-28",
"full_name": "Liam Shawcross",
"email": "lshawcross5#exblog.jp",
"credit_card": "6331104669953298",
"city": "Humaitá",
"state": "NY",
"zip_code": "98670",
"lines": [
{
"order_id": 5,
"product": "Beans - Green",
"quantity": 2,
"price": 4.33
},
{
"order_id": 1,
"product": "Grapefruit - Pink",
"quantity": 5,
"price": 5.00
}
]
},
]
The order attributes have been synchronized to a local table (Table name: SOTEST_LOCAL)
The table has the correct data. As you can see below, the LINES column contains the JSON array.
I then created an ORDER_LINES child table to extract the JSON from LINES column in the SOTEST_LOCAL table. (Sorry for the table names.. I should've named the tables as ORDERS_LOCAL and ORDER_LINES_LOCAL)
CREATE TABLE "SOTEST_ORDER_LINES_LOCAL"
( "LINE_ID" NUMBER,
"ORDER_ID" NUMBER,
"LINE_NUMBER" NUMBER,
"PRODUCT" VARCHAR2(200) COLLATE "USING_NLS_COMP",
"QUANTITY" NUMBER,
"PRICE" NUMBER,
CONSTRAINT "SOTEST_ORDER_LINES_LOCAL_PK" PRIMARY KEY ("LINE_ID")
USING INDEX ENABLE
) DEFAULT COLLATION "USING_NLS_COMP"
/
ALTER TABLE "SOTEST_ORDER_LINES_LOCAL" ADD CONSTRAINT "SOTEST_ORDER_LINES_LOCAL_FK" FOREIGN KEY ("ORDER_ID")
REFERENCES "SOTEST_LOCAL" ("ORDER_ID") ON DELETE CASCADE ENABLE
/
QuickSQL version..
SOTEST_ORDER_LINES_LOCAL
LINE_ID /pk
ORDER_ID /fk SOTEST_LOCAL references ORDER_ID
LINE_NUMBER
PRODUCT
QUANTITY
PRICE
So per Carsten's answer in the previous question, I can write SQL to extract the JSON array from the LINES column in the SOTEST_LOCAL table to the child table SOTEST_ORDER_LINES_LOCAL.
My question is two parts.
Where exactly do I write the SQL? Would I write it in SQL Workshop in SQL Commands?
The REST data is being synchronized to make a request every hour. So would I need to write a function that runs every time there is new data being merged?
there are multiple options for this:
Create a trigger on the local synchronization table
You could create an trigger on your ORDERS table, which runs AFTER INSERT, UPDATE or DELETE on your ORDERS table, and which maintains the LINES table. The nice things about this one is that the maintenance of the child table is independent from APEX or the REST Synchronization; it would also work if you just inserted rows with plain SQL*Plus.
Here's some pseudo-code on how the trigger could look like.
create or replace trigger tr_maintain_lines
after insert or update or delete on ORDERS_LOCAL
for each row
begin
if inserting then
insert into SOTEST_ORDER_LINES_LOCAL ( order_id, line_id, line_number, product, quantity, price)
( select :new.id,
seq_lines.nextval,
j.line#,
j.product,
j.quantity,
j.price
from json_table(
:new.lines,
'$[*]' columns (
line# for ordinality,
product varchar2(255) path '$.product',
quantity number path '$.quantity',
price number path '$.price' ) ) );
elsif deleting then
delete SOTEST_ORDER_LINES_LOCAL
where order_id = :old.id;
elsif updating then
--
-- handle the update case here.
-- I would simply delete and re-insert LINES rows.
end if;
end;
Handle child table maintenance in APEX itself.
You could turn off the schedule of your REST Source synchronization, and have it only running when called with APEX_REST_SOURCE_SYNC.SYNCHRONIZE_DATA (https://docs.oracle.com/en/database/oracle/apex/22.1/aeapi/SYNCHRONIZE_DATA-Procedure.html#GUID-660DE4D1-4BAF-405A-A871-6B8C201969C9).
Then create an APEX Automation, which runs on your desired schedule, and this automation has two Actions. One would be the REST Source Synchronization, the other one would call PL/SQL code to maintain the child tables.
Have a look into this blog posting which talks a bit about more complex synchronization scenarios (although it does exactly fit scenario): https://blogs.oracle.com/apex/post/synchronize-parent-child-rest-sources
I hope this helps
I was looking to set up TTL for my DynamoDB table and set it to be the "TTL" key. Used python3 + boto3 to do a batch write with the TTL field set to str(int(time.time()) + 172800). I also tried int(time.time()) + 172800.
In either case, the epoch time stamp in the TTL column was 48 hours in the future. When using the second version, hovering over the value in the DynamoDB table showed a pop up with the date and time of the time stamp and I confirmed it was 2 days in the future.
However, when I came back ~5 minutes later and refreshed the table, all of the entries were gone.
I repeated the process and actively refreshed to keep an eye on the values and they were all getting deleted gradually with later time stamps being deleted last. I checked the cloudwatch logs and they showed my scans and writes in the correct GMT time.
I'm just wondering what might cause this to happen. For reference the table's creation date was July 27, 2019; is it possible that the clock for the db is off?
Example timestamp that was deleted: 1588959677 which should translate to 5/8/2020 sometime.
Let me know if I need to provide more information and thanks for the help.
Edit: When I batch write I run the following:
boto3.resource('dynamodb', region_name).batch_write_item(RequestItems=put_data)
where:
put_data = { tablename: { "PutRequest": { "Item" { "id": "id", "TTL": ttl_integer_value, "another_id": "id", "flag": "true", "timestamp": original_time_value, "description": "some description" }}}}
I tried changing it to:
put_data = { tablename: { "PutRequest": { "Item" { "TTL": {"N": ttl_integer_value}... }}}}
but it threw an error with the key value not being valid
Also, if I hover over the integer value in the table, the appropriate date and time shows in a popup. Wouldn't that be an indication of the correct format?
I want to query a DDB GSI with key condition, and apply filter on returned result using contains function.
Data I have in DDB table:
{
"lookupType": "PRODUCT_GROUP",
"name": "Spring framework study set",
"structureJson": {
"productIds": [
"FBCUPOQsrp",
"Y4LDaiViLY",
"J6N3UWq9CK"
]
},
"ownerId": "mT9R9y6zGO"
}
{
"lookupType": "PRODUCT_GROUP",
"name": "Relational databases study set",
"structureJson": {
"productIds": [
"QWQWQWQWQW",
"XZXZXZXZXZ"
]
},
"ownerId": "mT9R9y6zGO"
}
...
I have a compound GSI (ownerId - HASH, lookupType - RANGE).
When I try to query the DDB (query structure is in "2" field) I get the result (the result is in "3"):
{
"0":[
],
"2":{
"TableName":"Products",
"IndexName":"ProductsOwnerIdLookupTypeIndex",
"KeyConditionExpression":"#ownerId = :ownerId and #lookupType = :lookupType",
"FilterExpression":"contains(#structureMember, :memberId)",
"ExpressionAttributeNames":{
"#ownerId":"ownerId",
"#lookupType":"lookupType",
"#structureMember":"structureJson.productIds"
},
"ExpressionAttributeValues":{
":ownerId":"mT9R9y6zGO",
":lookupType":"PRODUCT_GROUP",
":memberId":"FBCUPOQsrp"
}
},
"3":{
"Items":[
],
"Count":0,
"ScannedCount":2
}
}
The returned result set is empty, despite I have data with given field values.
How I see the query (or what I want to achieve):
When I query the GSI with ownerId = mT9R9y6zGO and lookupType = PRODUCT_GROUP it will find 2 items - Spring and Relational DB sets
As the second step DDB will scan the returned query result with contains (structureJson.productIds, FBCUPOQsrp) filter expression and it should return one result to me, but I get empty set
Is something wrong with the query or do I miss some point in DDB query workflow?
I have an AWS DynamoDb cart table with the following item structure -
{
"cart_id": "5e4d0f9f-f08c-45ae-986a-f1b5ac7b7c13",
"user_id": 1234,
"type": "OTHER",
"currency": "INR",
"created_date": 132432423,
"expiry": 132432425,
"total_amount": 90000,
"total_quantity": 2,
"items": [
{
"amount": 90000,
"category": "Laptops",
"name": "Apple MacBook Pro",
"quantity": 1
}
]
}
-
{
"cart_id": "12340f9f-f08c-45ae-986a-f1b5ac7b1234",
"user_id": 1234,
"type": "SPECIAL",
"currency": "INR",
"created_date": 132432423,
"expiry": 132432425,
"total_amount": 1000,
"total_quantity": 2,
"items": [
{
"amount": 1000,
"category": "Special",
"name": "Special Item",
"quantity": 1
}
]
}
The table will have cart_id as Primary key,
user_id as an Index or GSI,
type as an Index or GSI.
I want to be able to query the cart table,
to find the items which have user_id = 1234 AND type != "SPECIAL".
I don't know if this means for the query -
--key-condition-expression "user_id = 1234 AND type != 'SPECIAL'"
I understand that an AWS DynamoDb table cannot be queried using multiple indexes at the same time,
I came across the following question, it has a similar use case and the answer is recommending creating a composite key,
Querying with multiple local Secondary Index Dynamodb
Does it mean that while putting a new item in the table,
I will need to maintain another column like user_id_type,
with its value as 1234SPECIAL and create an Index / GSI for user_id_type ?
Sample item structure -
{
"cart_id": "5e4d0f9f-f08c-45ae-986a-f1b5ac7b7c13",
"user_id": 1234,
"type": "OTHER",
"user_id_type" : "1234OTHER",
"currency": "INR",
"created_date": 132432423,
"expiry": 132432425,
"total_amount": 90000,
"total_quantity": 2,
"items": [
{
"amount": 90000,
"category": "Laptops",
"name": "Apple MacBook Pro",
"quantity": 1
}
]
}
References -
1. Querying with multiple local Secondary Index Dynamodb
2. Is there a way to query multiple hash keys in DynamoDB?
Your assumption is correct. Maybe you can add into that a delimitter field1_field2 or hash them if either of them is too big in size hashOfField1_hashOfField2
That mean spending some more processing power on your side, however. As DynamoDB does not natively support It.
Composite key in DynamoDB with more than 2 columns?
Dynamodb: query using more than two attributes
Additional info on your use case
KeyConditionExpression only allowed for the hash key.
You can put it in the FilterExpression
Why is there no **not equal** comparison in DynamoDB queries?
Does it mean that while putting a new item in the table,
I will need to maintain another column like user_id_type,
with its value as 1234SPECIAL and create an Index / GSI for user_id_type?
The answer is it depends on how many columns (dynamodb is schema-less, by a column I mean data field) you need and are you happy with 2 round trips to DB.
your query:
user_id = 1234 AND type != "SPECIAL"
1- if you need all information in the cart but you are happy with two round trips:
Solution: Create a GSI with user_id (HASH) and type (RANGE), then add cart_id (base table Hash key) as projection.
Explanation: so, you need one query on index table to get the cart_id given user_id and type
--key-condition-expression "user_id = 1234 AND type != 'SPECIAL'"
then you need to use cart_id(s) from the result and make another query to the base table
2- if you do not need all of cart information.
Solution: you need to create a GSI and make user_id HASH and type as RANGE and add more columns (columns you need) to projections.
Explanation: projection is additional columns you want to have in your index table. So, add some extra columns, which are more likely to be used as a result of the query, to avoid an extra round trip to the base table
Note: adding too many extra columns can double your costs, as any update on base table results in updates in GSI tables projection fields)
3- if you want just one round trip and you need all data
then you need to manage it by yourself and your suggestion can be applied
One possible answer is to create a single index with a sort key. Then you can do this:
{
TableName: "...",
IndexName: "UserIdAndTypeIndex",
KeyConditionExpression: "user_id = :user_id AND type != :type",
ExpressionAttributeValues: {
":user_id": 1234,
":type": "SPECIAL"
}
}
You can build GraphQL schema with AWS AppSync from your DynamoDB table and than query it in your app with GraphQL. Link
A part of our application submits 100s of BQ queries to build an aggregated dashboard view. Due to the large number of queries, we submit all queries with BATCH priority.
In last 24 hours we've had quite a few queries (more than 300) fail due to resources exceeded. This is odd as some of the failed queries save results to an output table specifically to avoid this type of issue. We spot checked a few others and are able to run them without issue in the BQ web UI.
Sample job ID (no dest table): job_GPghMyEBvlYFBGBHI5Szv0HMIpMY
Query:
SELECT order_sub.value as value, count(*) as count
FROM
(
SELECT order_id, drop_ship as value
FROM (TABLE_QUERY(d34f20170905T212849Zc4abca, 'table_id IN ("order_detail_2018")'))
WHERE order_date BETWEEN TIMESTAMP("2018-01-01") AND TIMESTAMP("2018-08-31")
) order_sub
JOIN EACH
(
SELECT customer_id, order_id, sequence
FROM temp_30_days.query_q6109c34f20180918T151206Ze52003_buyers
) cust_sub
ON order_sub.order_id = cust_sub.order_id
GROUP BY value
ORDER BY value
Error Message:
"status": {
"state": "DONE",
"errorResult": {
"reason": "resourcesExceeded",
"message": "Resources exceeded during query execution."
},
"errors": [
{
"reason": "resourcesExceeded",
"message": "Resources exceeded during query execution."
}
]
}
Sample job ID (with dest table): job_2YsCt-cxql5TP0jklb1aRENzINdZ
Query:
SELECT
customer_id, sequence
FROM
(
SELECT
customer_id,
sequence,
random,
MAX(random) OVER (PARTITION BY customer_id) AS max_value
FROM
(
SELECT customer_id, sequence, RAND() as random
FROM (TABLE_QUERY(d34f20170905T212849Zc4abca, 'table_id IN ("cust_char_2015", "cust_char_2016", "cust_char_2017")'))
WHERE end_date >= TIMESTAMP("2016-12-31")
AND effective_date <= TIMESTAMP ("2017-08-31")
AND (frequency_end = "98")
)
)
WHERE max_value = random
GROUP EACH BY
customer_id, sequence
Error Message:
"status": {
"errorResult": {
"message": "Resources exceeded during query execution.",
"reason": "resourcesExceeded"
},
"errors": [
{
"message": "Resources exceeded during query execution.",
"reason": "resourcesExceeded"
}
],
"state": "DONE"
}
FWIW status.cloud.google.com shows green lights for BigQuery, is anyone else experiencing similar issues?