having WSO2AM 2.1.0 with analytics (wso2am-analytics-2.1.0) with PostgreSQL the database team complains about too many database errors in the analytics database (too many is approx 1.5GB / day and we have almost no traffic)
Database logs:
<PGSCYANAES-scyanaes-2017-10-10 09:35:22 CEST>ERROR: duplicate key value
violates unique constraint "anx___7lv6kvss__pkey"
<PGSCYANAES-scyanaes-2017-10-10 09:35:22 CEST>DETAIL: Key
(record_id)=(64d8b03f-9e60-3d2e-837e-3df5103620e7) already exists.
<PGSCYANAES-scyanaes-2017-10-10 09:35:22 CEST>STATEMENT: INSERT INTO
ANX___7Lv6kvss_ (partition_key, timestamp, data, record_id) VALUES ($1, $2, $3, $4)
<PGSCYANAES-scyanaes-2017-10-10 09:35:22 CEST>ERROR: duplicate key value
violates unique constraint "anx___7lv6kvss__pkey"
I've seen these entries in the wso2carbon.log before, but after recreating the databases and resintling the analytics server, the wso2carbon.log looks clean. However - the database team keeps claiming these errors are still occuring (a lot)
Edit:
we found out there's a configutation file with SQL statements rdbms-config.xml
and there are several statements causing the errors in the db logs:
<recordInsertQuery>INSERT INTO {{TABLE_NAME}} (partition_key, timestamp, data, record_id) VALUES (?, ?, ?, ?) </recordInsertQuery>
This statement seems to cause the most of the error logs.
I've changed the statement to:
INSERT INTO {{TABLE_NAME}} (partition_key, timestamp, data, record_id)
VALUES (?, ?, ?, ?)
ON CONFLICT (record_id)
DO UPDATE SET partition_key=EXCLUDED.partition_key, timestamp=EXCLUDED.timestamp, data=EXCLUDED.data
Until now it looks all working, but I may be not aware of all consequences and side-effects of this update. Is there any problem that can arise using the new upsert statement?
Thank you in advance for any hint / help
Related
I have two tables, SRC and TGT. I want to populate all my foreign key values in TGT from the target lookups. However, I'm getting nulls into the target. I have used all natural keys for lookup conditions. Can anyone please explain why I am seeing nulls?
For instance, I want to populate foreign key values for MPNG_ID, SESN-ID, WRKFLW-ID from lookup tables based on repository name and version number.
Did you validate if all your Join Conditions are working fine? Probably try to replicate same query in DB and check if its giving you the same result. I assume some Join might have messed up which might result in Null values. Ensure that even the spaces are in sync if you are using Varchar value.
I'm writing an ODBC class to connect to a remote SQL Server database. I have most of it working.
The class has the ability to generate queries such as the following:
UPDATE Customers SET Id=?,Name=?,TaxId=?,ContactFName=?,ContactLName=?,Phone_Office=?,Phone_Mobile=?,Phone_Home=?,Email=?,Website=?,Address1_Physical=?,Address2_Physical=?,City_Physical=?,State_Physical=?,Zip_Physical=?,Address1_Billing=?,Address2_Billing=?,City_Billing=?,State_Billing=?,Zip_Billing=?,StartingBalance=?,Discount=?,BillingSequence=?,BillingCategory=?,ShowOnReport=?,Active=?,CreateDate=?
As you can see, it's an UPDATE query. Yet, running this query gives me an error:
Microsoft ODBC Driver 17 for SQL Server (SQL Server) : ReturnCode: -1 : Violation of PRIMARY KEY constraint 'PK_Customers'. Cannot insert duplicate key in object 'dbo.Customers'. The duplicate key value is (82). (State: 23000, NativeError: 2627) : The statement has been terminated. (State: 01000, NativeError: 3621)
I'm confused why I'm getting an error about inserting when I'm doing an update. Has anyone seen this?
Notes:
Id is the primary key. I first read all column values from the database, and then update those I want to change. The ID does not change.
The error above was put together by my code, but is based on the messages returned by SQLGetDiagRec().
There's no WHERE clause on the UPDATE statement, so it's trying to update EVERY SINGLE ROW in the database, and since ID is one of the columns being changed, it's trying to set every row's ID to the same value. This is resulting in an attempt to create a duplicate primary key.
Make sure your UPDATE statement has an appropriate WHERE clause... like "WHERE ID = ?"... and it's probably best practice to NOT include the ID in that UPDATE statement if it's not changing.
That is the message you should expect when an UPDATE statement violates a primary key. EG
use tempdb
go
drop table if exists t
create table t(id int primary key)
insert into t(id) values (1),(2)
go
update t set id = 2 where id = 1
--Msg 2627, Level 14, State 1, Line 11
--Violation of PRIMARY KEY constraint 'PK__t__3213E83F127C5D76'. Cannot insert duplicate key in object 'dbo.t'. The duplicate key value is (2).
--The statement has been terminated.
In the UPDATE I see a field called ID. If you are making a change to the ID and it's the primary key then the DBMS will fuss because you are trying to store duplicate keys.
On a regular occasion, my Django webapps produce SQL errors on M2M tables.
Each time it turns out the ID sequence is reset to a value within the range of existing rows.
The app performs normal SQL queries such as:
INSERT INTO "myapp_project" ("name") VALUES ('test1') RETURNING "myapp_project"."id"'
which cause errors such as:
IntegrityError: duplicate key value violates unique constraint "myapp_project_pkey"
DETAIL: Key (id)=(29) already exists.
Then it turns out that the myapp_project_id_seq is pointing to an old ID number:
select currval('myapp_project_id_seq')
29
Which can then be reset by using:
select setval('myapp_project_id_seq', (select max(id) from myapp_project))
However, I can't explain why this is happening. It typically happens on M2M tables in Django. In this case, a normal table with admin-only input. Can someone enlighten me about this?
This typically happens when you (or somebody) sometimes write values to id explicitly, instead of getting values from the sequence (by default or with nextval()).
Your repair code is missing a pair of parentheses.
SELECT setval('myapp_project_id_seq', (SELECT max(id) FROM myapp_project));
This is a tiny bit shorter & cheaper while doing the same, exactly:
SELECT setval('myapp_project_id_seq', max(id)) FROM myapp_project;
I am trying to run multiple alter table statements for adding foreign keys on my database
I am using RazorSQL
this are my sql statement
ALTER TABLE SPO999.AVTVRSTEPLACILAPOD
ADD CONSTRAINT SQL100419145030510 FOREIGN KEY
(AVP_VRSTEPLACILA)
REFERENCES SPO999.VRSTEPLACILA
(VP_ID_VP)
ON DELETE NO ACTION
--ON UPDATE NO ACTION
ENFORCED
ENABLE QUERY OPTIMIZATION;
-- DDL Statements for foreign keys on Table SPO999.AVTVRSTEPLACILAVRPL
ALTER TABLE SPO999.AVTVRSTEPLACILAVRPL
ADD CONSTRAINT SQL100419145030630 FOREIGN KEY
(AVV_VRSTEPLACILA)
REFERENCES SPO999.VRSTEPLACILA
(VP_ID_VP)
ON DELETE NO ACTION
--ON UPDATE NO ACTION
ENFORCED
ENABLE QUERY OPTIMIZATION;
If I run one at a time it works, while if trying to run both at I get an SQL Error -104 a token,character or clause is invalid or missing.
I can not find a problem/solution
any suggestions?
thank you
I think it is likely that you can't have these two foreign key definitions referencing the same column within a single transaction.
Try adding commit; between the statements.
(Your SQL editor, depending on its settings, may automatically send a commit after each chunk of statements that you execute. That would explain the difference between the two cases).
Background: Running a PostgreSQL database for a Django app (Django 1.1.1, Python2.4, psycopg2 and Postgres 8.1) I've restored the database from a SQL dump several times. Each time I do that and then try to add a new row, either shell, admin, or site front end, I get this error:
IntegrityError: duplicate key violates unique constraint "app_model_pkey"
The data dump is fine and is resetting the sequences. But if I try adding the row again, it's successful! So I can just try jamming a new row into every table and then everything seems to be copacetic.
Question: Given that (1) the SQL dump is good and Postgres is reading it in correctly (per earlier question), and (2) Django's ORM does not seem to be failing systemically getting next values, what is going on in this specific instance?
Django doesn't hold or directly read the sequence values in any way. I've explained it f.ex. in this question: 2088210/django-object-creation-and-postgres-sequences.
Postgresql does increment the sequence when you try to add a row, even if the result of the operation is not successful (raises a duplicate key error) the sequence incrementation doesn't rollback. So, that's the reason why it works the second time you try adding a row.
I don't know why your sequences are not set properly, could you check what is the sequence value before dump and after restore, and do the same with the max() pk of the table? Maybe it's an 8.1 bug with the restore? I don't know. What I'm sure of: it's not Django's fault.
I am guessing that your sequence is out of date.
You can fix that like this:
select setval('app_model_id_seq', max(id)) from app_model;