I have below sql which is getting stuck in oracle database for more than 2 hours. This stuck happens only when it is executed via the C++ application. Interestingly, at the same time when it was stuck I can execute it through sql developer manually and it returns within seconds. My table has millions of rows and about 100 columns. Can someone please point out how can I overcome this issue?
select *
from MY_TABLE
INNER JOIN ( (select max(concat(DATE ,concat('',to_char(INDEX, '0000000000')))) AS UNIQUE_ID
from MY_TABLE
WHERE ((DATE < '2018/01/29')
OR (DATE = '2018/01/29' AND INDEX <= 100000))
AND EXISTS ( select ID
from MY_TABLE
where DATE = '2018/01/29'
AND INDEX > 100000
AND LATEST =1)
group by ID ) SELECTED_SET )
ON SELECTED_SET.UNIQUE_ID = concat(DATE, concat('',to_char(INDEX, '0000000000')))
WHERE (FIELD_1 = 1 AND FIELD_2 = 1 AND FIELD_3='SomeString');
UPDATE:
db file sequential read is present on the session.
SELECT p3, count(*) FROM v$session_wait WHERE event='db file sequential read' GROUP BY p3;
.......................................
| P3 | COUNT(*) |
.......................................
| 1 | 2 |
.......................................
"I can execute it through sql developer manually and it returns within seconds"
Clearly the problem is not intrinsic to the query. So it must be a problem with your application.
Perhaps you have a slow network connection between your C++ application and the database. To check this you should talk to your network admin team. They are likely to be resistant to the suggestion that the network is the problem. So you may need to download and install Wireshark, and investigate it yourself.
Or your C++ is just very inefficient in handling the data. Is the code instrumented? Do you know what it's been doing for those two hours?
"the session is shown as 'buffer busy wait'"
Buffer busy waits indicate contention for blocks between sessions. If your application has many sessions running this query then you may have a problem. Buffer busy waits can indicate that there are sessions waiting on a full table scan to complete; but as the query returned results when you ran it in SQL Developer I think we can discount this. Perhaps there are other sessions updating MY_TABLE. How many sessions are reading or writing to it?
Also, what is the output of this query?
SELECT p3, count(*)
FROM v$session_wait
WHERE event='buffer busy wait'
GROUP BY p3
;
Worked with our DBA and he disabled the plan directives at system level using
alter system set "_optimizer_dsdir_usage_control"=0;
As per him, SQL plan directives were created as cardinality mis-estimates after executing the sql. After that timing was greatly improved and the problem is solved.
Related
I'm new to redshift and facing 'D': 'Serializable isolation violation on table error.
This is occurring during the data archive process time frame(mid night load) and impacting the regular batch job data load for same table.
I've written below the archive load process. Is there any best way to avoid such error in my below script?
Can you help? Thanks in advance.
INSERT INTO abc.data_arch
( col1,col2......
)
With cte_365days_older
AS
(
Select * from abc.abc_data_365
WHERE abc_ts < dateadd(day, -365, current_date)
--LIMIT 1
)
SELECT
col1,col2......
FROM cte_365days_older src
WHERE not exists (select 1 from abc.data_arch trgt
where src.abc_hkey = trgt.abc_hkey
)
order by col1,col2
;
DELETE
from abc.abc_data_365
WHERE abc_ts < dateadd(day, -365, current_date);
COMMIT;
VACUUM DELETE ONLY abc.abc_data_365;
COMMIT;
Error:-
'D': 'Serializable isolation violation on table - 342561, transactions forming the cycle are: 2234341, 2034548 (pid:3235)'
OK, with the regular batch job data running at the same time, it looks like the conflict is between the transaction which is doing the INSERT, and the VACUUM command running in parallel in another session (as per Max's answer).
Regarding the data architecture, DELETE is a relatively expensive operation, so to avoid the error you might consider an alternative approach:
Create a VIEW on your abc_data_365 table, which only contains the most
recent 365 days of data.
Run the DELETE less frequently (maybe monthly), and do it immediately after the INSERT has finished for that day
Then let your end-users SELECT from the VIEW, not the base table.
End-users will probably see almost no performance degradation (if you run it monthly then at worst a SELECT will be scanning 1/12 of the table extra)
You'll be asking Redshift to do the expensive DELETE far less frequently
Something like this:
CREATE VIEW v_abc_data_365 AS
SELECT * FROM abc_data_365
WHERE abc_ts > DATEADD(DAY, -365, current_date);
I had an issue with updating my historical table in db.
I have 3 steps in ETL job:
Detect changes - Matillion ETL finds differencies between two tables and writes it into tmp table
Close current - this step updates all rows with current_flag = 'Y' to 'N' and sets end_date to dateadd(day,-1,current_date)
Insert rows - all rows from tmp table in step 1 (filtering only the ones with N and C indicator) are being appended to Historical table with start_date equal to dateadd(day,-1,current_date) and end date equal to '2099-01-01'
WHat happened is that step 2 and 3 were executed manually by mistake with data from yesterday (from tmp table) and after 15minutes job with all 3 steps was executed again as supposed to with new data.
I have different update_timestamps (2021-03-19 01:59:02 and 2021-03-19 02:11:57) which can be used to fix the issue. But i am having difficulties on fixing data that were updated manually (having start_date='2021-03-18' and end_date='2021-03-18' and current_flag='N'), I believe those should be updated again to start_date='2021-03-18' and end_date='2099-01-01' and current_flag='Y') but I am not sure what to do with new job data (newer timestamp - that executed correctly) as it was done on wrong data.
Should I delete it and rerun after fixing start_date='2021-03-18' and end_date='2021-03-18' and current_flag='N' manually?
Also, I feel like i'm missing some steps here?
Thanks
Im new in azure analytics. Im using analytics to get feedbacks from users. There are about 50 events that im sending to azure in a second and im trying to get a combined result from two inputs but couldnt get a working output. My problem is in sql query for output.
Now I'm sending in the inputs.
Recommandations:
{"appId":"1","sequentialId":"28","ItemId":"1589018","similaristyValue":"0.104257207028537","orderId":"0"}
ShownLog:
{"appId":"1","sequentialId":"28","ItemId":"1589018"}
I need to join them with sequentialId and ItemId and calculate the difference between two ordered sequential.
For example: I send 10 Recommandations events and after that (like after 2 sec) i send 3 ShownLog event. So what i need to do is i have to get sum of first 3 (because i send 3 shownlog event) event's similaristyValue ordered by "orderid" from "Recommandations". I also need to get the sum of similarityValues from "ShownLog". At the end i need an input like (for every sequential ID):
sequentialID Difference
168 1.21
What i ve done so far is. I save all the inputs my azure sql and i ve managed to write the sql i want. You may find the mssql query for it:
declare #sumofSimValue float;
declare #totalItemCount int;
declare #seqId float;
select
#sumofSimValue = sum(b.[similarityValue]),
#totalItemCount = count(*),
#seqId = a.sequentialId
from EventHubShownLog a inner join EventHubResult b on a.sequentialId=b.sequentialId and a.ItemId=b.ItemId group by a.sequentialId
--select #sumofSimValue,#totalItemCount,#seqId
SELECT #seqId, SUM([similarityValue])-#sumofSimValue
FROM (
SELECT TOP(#totalItemCount) [similarityValue]
FROM [EventHubResult] where sequentialId=#seqId order by orderId
) AS T
But it gives lots of error in analytics. Also it lacks the logic of azure analytcs. I hope i could tell the problem.
Can you tell me how can i do such a job for my system? How can i use the time windows or how can i join them properly?
For every shown log, you have to select sum of similarity value. Is that the intention? Why not just join and select sum? It would only select as many rows as there are shown logs.
One thing to decide is the maximum time difference between recommendation events and shown log events, with that you can use Azure Stream analytics join, https://msdn.microsoft.com/en-us/library/azure/dn835026.aspx
Is there a way to find the resource intensive and time consuming queries in WX2?
I tried to check SYS.IPE_COMMAND and SYS.IPE_TRANSACTION tables but of no help.
The best way to identify such queries when they are still running is to connect as SYS with Kognitio Console and use Tools | Identify Problem Queries. This runs a number of queries against Kognitio virtual tables to understand how long current queries have been running, how much RAM they are using, etc. The most intensive queries are at the top of the list, ranked by the final column, "Relative Severity".
For queries which ran in the past, you can look in IPE_COMMAND to see duration but only for non-SELECT queries - this is because SELECT queries default to only logging the DECLARE CURSOR statement, which basically just measures compile time rather than run time. To see details for SELECT queries you should join to IPE_TRANSACTION to find the start and end time for the transaction.
For non-SELECT queries, IPE_COMMAND contains a breakdown of the time taken in a number of columns (all times in ms):
SM_TIME shows the compile time
TM_TIME shows the interpreter time
QUEUE_TIME shows the time the query was queued
TOTAL_TIME aggregates the above information
If it is for historic view image commands as mentioned in the comments, you can query
... SYS.IPE_COMMAND WHERE COMMAND IMATCHING 'create view image' AND TOTAL_TIME > 300000"
If it is for currently running commands you can look in SYS.IPE_CURTRANS and join to IPE_TRANSACTION to find the start time of the transaction (assuming your CVI runs in its own transaction - if not, you will need to look in IPE_COMMAND to find when the last statement in this TNO completed and use that as the start time)
Currently, my development database is taking up way more disk space than it probably should. The project uses Django, Docker-Machine, and PostgreSQL. The weird thing is that when I use pg_database_size(), it says that around 9MB of disk space is used (the uploaded file is only 20KB). However, when I sum up the sizes of all the tables (i.e. summing up the results of doing pg_total_relation_size() on each table name returned by the query SELECT relname FROM pg_class WHERE relkind='r' AND relname !~ '^(pg_|sql_)';), the total size reported is only 1.8MB. I'm using psycopg2 to query my database.
Why exactly is there such a difference in sizes? I saw a similar sort of question where the answer seemed to be to vacuum any LOBs that weren't properly deleted. I did the following:
docker exec -it <name_of_database_container> vacuumlo -U postgres -W -v postgres
and it reported that it successfully removed 0 LOBs, so it seems like LOBs were never my issue. What else could be going on? Am I not correctly getting all the table names using the SELECT relname.... query?
Your pg_total_relation_size query is excluding system tables that start with pg_. Those tables do still take up space.
The minimum possible (non-broken) database is around 7.2MB in my hands. So the numbers you give work out.
The documentation for pg_total_relation_size says that it includes the size of indexes with it, but that wasn't the case with my database. That may have been because I created the indexes with an explicit name rather than anonymous indexes.
Here are some queries to help track down where space is going:
-- total database size on disk
select * from pg_size_pretty(pg_database_size('etlq'));
-- total sizes by schema
select nspname, pg_size_pretty(sum(pg_total_relation_size(C.oid)))
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
GROUP BY nspname;
-- total size by relation
select nspname, relname, pg_size_pretty(pg_total_relation_size(C.oid)), pg_total_relation_size(c.oid)
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE true
and pg_total_relation_size(C.oid) > 10 * 1024 * 1024 -- greater than 10MB
ORDER BY pg_total_relation_size(C.oid) DESC, nspname;