I'm inserting from a source table to the target table. If the source record already exists in the target, then it will update else insert. I have done this without using Update strategy. In session properties, I have set as treat rows as 'Update' and in the mapping target properties, I selected 'Insert' and 'Update Else Insert' checkbox. Also, I have chosen a primary key in the target table as well. But while running the session every time, it is always inserting the same rows again and again. I.e. duplicate rows are inserting instead of update the record. am I doing anything wrong?
The database table should have primary keys defined for this to work.
Related
I want to put an order from my lex bot into dynamoDB however the PutItem operation overwrites each time(If the customer name is already in the table).
I know from the documentation that it will do this if the primary key is the same.
My goal is to have each order put into the database so they will be easily searchable in the future.
I have attached some screenshots below. Any help is appreciated
https://imgur.com/a/mLpEkOi
def putDynam(orderNum, table_custName, slotKey, slotVal):
client = boto3.resource('dynamodb')
table = client.Table('blah')
input = {'Customer': table_custName, 'OrderNumber':orderNum[0], 'Bun Type': slotVal[5], 'CheeseDecision': slotVal[1], 'Cheese Type': slotVal[0], 'Pickles': slotVal[4], 'SauceDecision': slotVal[3], 'Sauce Type': slotVal[2]}
action = table.put_item(Item=input)
The primary key is used for identifying each item in the table. There can only be 1 record with a specific primary key (primary keys are unique).
Customer name is not a good primary key, because it's not unique.
In this case you could have an order with some generated Id (orderNumber in your example?), that could be the primary key, and Customer (preferably CustomerId) as a property.
Or you could have a composite primary key made up of CustomerId and OrderId.
If you want to query orders by customer, you could use an index if it's not in the primary key.
I recommend you read up on how DynamoDB works first. You can start with this data modelling tutorial from AWS.
So, basically, the customer name has to be unique, since it's your Primary Key. You can't have two rows with the same primary key. A way could be to have an incremental value that serves as id, and each insert would simply have i+1 as its id.
You can see this stack overflow question for more information: https://stackoverflow.com/a/12460690/11593346
Per https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.put_item, you can:
"perform a conditional put operation (add a new item if one with the specified primary key doesn't exist)"
Note
To prevent a new item from replacing an existing item, use a conditional expression that contains the attribute_not_exists function with the name of the attribute being used as the partition key for the table. Since every record must contain that attribute, the attribute_not_exists function will only succeed if no matching item exists.
Also see DynamoDB: updateItem only if it already exists
If you really need to know whether the item exists or not so you can trigger your exception logic, then run a query first to see if the item already exists and don't even call put_item. You can also explore whether using a combination of ConditionExpression and one of the ReturnValues options (for put_item or update_item) may return enough data for you to know if an item existed.
I'm writing an ODBC class to connect to a remote SQL Server database. I have most of it working.
The class has the ability to generate queries such as the following:
UPDATE Customers SET Id=?,Name=?,TaxId=?,ContactFName=?,ContactLName=?,Phone_Office=?,Phone_Mobile=?,Phone_Home=?,Email=?,Website=?,Address1_Physical=?,Address2_Physical=?,City_Physical=?,State_Physical=?,Zip_Physical=?,Address1_Billing=?,Address2_Billing=?,City_Billing=?,State_Billing=?,Zip_Billing=?,StartingBalance=?,Discount=?,BillingSequence=?,BillingCategory=?,ShowOnReport=?,Active=?,CreateDate=?
As you can see, it's an UPDATE query. Yet, running this query gives me an error:
Microsoft ODBC Driver 17 for SQL Server (SQL Server) : ReturnCode: -1 : Violation of PRIMARY KEY constraint 'PK_Customers'. Cannot insert duplicate key in object 'dbo.Customers'. The duplicate key value is (82). (State: 23000, NativeError: 2627) : The statement has been terminated. (State: 01000, NativeError: 3621)
I'm confused why I'm getting an error about inserting when I'm doing an update. Has anyone seen this?
Notes:
Id is the primary key. I first read all column values from the database, and then update those I want to change. The ID does not change.
The error above was put together by my code, but is based on the messages returned by SQLGetDiagRec().
There's no WHERE clause on the UPDATE statement, so it's trying to update EVERY SINGLE ROW in the database, and since ID is one of the columns being changed, it's trying to set every row's ID to the same value. This is resulting in an attempt to create a duplicate primary key.
Make sure your UPDATE statement has an appropriate WHERE clause... like "WHERE ID = ?"... and it's probably best practice to NOT include the ID in that UPDATE statement if it's not changing.
That is the message you should expect when an UPDATE statement violates a primary key. EG
use tempdb
go
drop table if exists t
create table t(id int primary key)
insert into t(id) values (1),(2)
go
update t set id = 2 where id = 1
--Msg 2627, Level 14, State 1, Line 11
--Violation of PRIMARY KEY constraint 'PK__t__3213E83F127C5D76'. Cannot insert duplicate key in object 'dbo.t'. The duplicate key value is (2).
--The statement has been terminated.
In the UPDATE I see a field called ID. If you are making a change to the ID and it's the primary key then the DBMS will fuss because you are trying to store duplicate keys.
I want to check whether id(primary-key) of new item is uniq or not before adding into dynamoDB
what could be best option for both performance and cost wise.
Possible options to check uniqueness of primary-key can be...
1) Get (if empty array returns, it means there are no matching data. which also means it is uniq)
2) Scan (obvious, worst idea for both performance and cost)
3) Query
++ my another thought is, if there has any way to forcibly ignore incoming request in DynamoDB settings(discard incoming request or send error message), logic could be much simpler.
In normal RDB, if we try to add new item with existing primary key, Database will return error message without changing original data stored in database.
however, in DynamoDB, whether we Put item or Update item with existing primary key, it just silently changes original data stored in database.
have any idea?
As you mentioned, DynamoDB will update an item with the primary key you provide if it already exists. The article below shows you how you can make a conditional PUT request which will fail upon trying to insert an item that already exists (based on the primary key).
http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/API_PutItem.html
To prevent a new item from replacing an existing item, use a conditional expression that contains the attribute_not_exists function with the name of the attribute being used as the partition key for the table. Since every record must contain that attribute, the attribute_not_exists function will only succeed if no matching item exists.
I am using Informatica to finally write in the oracle table after performing certain logical operations on the data.
The problem is that if a certain ID was already previously processed and is present in the target table then it is not inserted again.
Please suggest a workaround.
Hi, This is because you might have same primary key in the source which is available in the target. Look into primary key columns and try loading them.
Altering your target table
alter table target_table_name drop constraint constraint name;
I am trying to update a specific field in the "Claims" table of my 2010 Access Database. I keep receiving an error message that says there are key violations. Here is the SQL:
UPDATE Claims SET Claims.LS_Name = "JPN"
WHERE (((Claims.Responsibility2)=0));
Is there any reason, based on the above code, that it is not working?
Thanks in advance!
My first guess would be that there is a unique key on Claims.LS_Name and your update hits more than one row.
OK, I ran into this issue as well in Microsoft Access and think I have ways to solve versions of this. The intermediate table may not be necessary. My experience is that autonumber is the issue so conversion to number seems to work, but you have to delete relationships for Access to allow this change.
remove all relationships to the destination table.
change from autonumber to number.
make new empty tablethatincrements starting with the next higher key https://superuser.com/questions/288087/how-do-i-set-the-first-value-of-autonumber-in-access
update/query to the tablethatincrements table instead of destination.
update/merge from tablethatincrementsto to the destination table, including new higher keys.
remake relationships
OR In certain situations. (for mine the new keys matched the old)
something like this (may not be exact steps)
remove all relationships to the destination table.
delete ID/primary key in destination
merge/update to destination
create new ID/Primary key in destination (so it can be autonumber to renumber keys)
remake relationships
Anyway my next research item is to see how to never use autoincrement and do the unique keys using other methods so relationships do not have to be removed to change from autoincrement to number.