SQLite how to limit the number of records - c++

I want to limit the number of records in my SQLite table to for example 100 records, and then when I INSERT the 101th record, the first record (the oldest) be removed from the table. In other word, I want to prevent the table from growing more than 100 records and always have the last 100 records. Is there any setting or query with SQLite or should I handle it manually?
thanks in advance

You can do it with a trigger.
Say that your table is this:
CREATE TABLE tablename (
id INTEGER PRIMARY KEY,
name TEXT,
inserted_at TEXT DEFAULT (strftime('%Y-%m-%d %H:%M:%f', 'now'))
);
In the column inserted_at you will have the timestamp of the insertion of each row.
This is not necessary if you declared the column id as:
id INTEGER PRIMARY KEY AUTOINCREMENT
because in this case you could identify the 1st inserted row by the minimum value of the id.
Now create this trigger:
CREATE TRIGGER keep_100_rows AFTER INSERT ON tablename
WHEN (SELECT COUNT(*) FROM tablename) > 100
BEGIN
DELETE FROM tablename
WHERE id = (SELECT id FROM tablename ORDER BY inserted_at, id LIMIT 1);
-- or if you define id as AUTOINCREMENT
-- WHERE id = (SELECT MIN(id) FROM tablename);
END;
END;
Every time that you insert a new row, the trigger will check if the table has more than 100 rows and if it does it will delete the 1st inserted row.
See the demo (for max 3 rows).

Related

Retrieving the row with the greatest timestamp in questDB

I'm currently running QuestDB 6.1.2 on linux. How do I get the row with maximum value from a table? I have tried the following on a test table with around 5 million rows:
select * from table where cast(timestamp as symbol) in (select cast(max(timestamp) as symbol) from table );
select * from table inner join (select max(timestamp) mm from table ) on timestamp >= mm
select * from table where timestamp = max(timestamp)
select * from table where timestamp = (select max(timestamp) from table )
where 1 is correct but runs in ~5s, 2 is correct and runs in ~500ms but looks unnecessarily verbose for a query, 3 compiles but returns an empty table, and 4 is incorrect syntax although that's how sql usually does it
select * from table limit -1 works. QuestDB returns rows sorted by timestamp as default, and limit -1 takes the last row, which happens to be the row with the greatest timestamp. To be explicit about ordering by timestamp, select * from table order by timestamp limit -1 could be used instead. This query runs in around 300-400ms on the same table.
As a side note, the third query using timestamp=max(timestamp) doesn't work yet since QuestDB does not support subqueries in where yet (questDB 6.1.2).

How to avoid error "Cannot insert rows out of order" in QuestDB?

I'm trying to migrate data to QuestDB and inserting historical records, I create table as
create table records(
type INT,
interval INT,
timestamp TIMESTAMP,
name STRING) timestamp(timestamp)
and insert data from CSV by curl uploading it.
I get back error "Cannot insert rows out of order". I read that out of order was supported in QuestDB but somehow I cannot make it work.
You can insert rows out of order on partitioned tables only, create new partitioned table and copy data into it
create table records2(
type INT,
interval INT,
timestamp TIMESTAMP,
name STRING
)
timestamp(timestamp) partition by DAY
insert into records2
select * from records
drop table records
rename table records2 to records
After this you'll be able to insert out of order into table records

Why bigquery can't handle a query processing 4TB data?

I'm trying to run this query
SELECT
id AS id,
ARRAY_AGG(DISTINCT users_ids) AS users_ids,
MAX(date) AS date
FROM
users,
UNNEST(users_ids) AS users_ids
WHERE
users_ids != " 1111"
AND users_ids != " 2222"
GROUP BY
id;
Where users table is splitted table with id column and user_ids (comma separated) column and date column
on a +4TB and it give me resources
Resources exceeded during query execution: Your project or organization exceeded the maximum disk and memory limit available for shuffle operations.
.. any idea why?
id userids date
1 2,3,4 1-10-20
2 4,5,6 1-10-20
1 7,8,4 2-10-20
so the final result I'm trying to reach
id userids date
1 2,3,4,7,8 2-10-20
2 4,5,6 1-10-20
Execution details:
It's constantly repartitioning - I would guess that you're trying to cramp too much stuff into the aggregation part. Just remove the aggregation part - I don't even think you have to cross join here.
Use a subquery instead of this cross join + aggregation combo.
Edit: just realized that you want to aggregate the arrays but with distinct values
WITH t AS (
SELECT
id AS id,
ARRAY_CONCAT_AGG(ARRAY(SELECT DISTINCT uids FROM UNNEST(user_ids) as uids WHERE
uids != " 1111" AND uids != " 2222")) AS users_ids,
MAX(date) OVER (partition by id) AS date
FROM
users
GROUP BY id
)
SELECT
id,
ARRAY(SELECT DISTINCT * FROM UNNEST(user_ids)) as user_ids
,date
FROM t
Just the draft I assume id is unique but it should be something along those lines? Grouping by arrays is not possible ...
array_concat_agg() has no distinct so it comes in a second step.

Updating Records Via RowVersion , using 'SQL WHERE' to filter for a MAX Value

Trying to update a table based off a RowVersion value in existing table. My data lake updates once a week , with new data stored as a .json file, which holds any new RowVersions.
I need to:
1)Query the existing table in my data warehouse to find the most up to date RowVersion( ie max)
2)Use that value to only filter/select the records in my data warehouse that are greater than the RowVersion I just identified
3)Update my table to include the new Rows
My Question is - the SQL Below, I am not sure how to select the Max RowNumber in the current table and then use that to filter/specify what I want returned when querying my S3 Bucket:
create or replace temporary table UPDATE_CAR_SALES AS
SELECT
VALUE:CAR::string AS CARS,
VALUE:RowVersion::INT AS ROW_VERSION
having row_version > max(row_version)
from '#s3_bucket',
lateral flatten( input => $1:value);
It's not clear to me how you store the data. Is the CARS column unique? Do you need to find maximum row version for each car or for all cars/rows? Anyway you can use a sub-query to filter the rows having row version is higher than the max value:
create or replace temporary table UPDATE_CAR_SALES AS
SELECT
VALUE:CAR::string AS CARS,
VALUE:RowVersion::INT AS ROW_VERSION
FROM #s3_bucket, lateral flatten( input => $1 )
where ROW_VERSION > (SELECT MAX(RowVersion)
from MAIN_TABLE);
If you need to filter the rows, based on row version of each car (of the existing table):
create or replace temporary table UPDATE_CAR_SALES AS
SELECT * FROM (SELECT
VALUE:CAR::string AS CARS,
VALUE:RowVersion::INT AS ROW_VERSION
FROM #s3_bucket, lateral flatten( input => $1 )) temp_table
where temp_table.ROW_VERSION > (SELECT MAX(RowVersion)
from MAIN_TABLE where cars = temp_table.CARS );
I needed to put the main query in brackets to be able to use alias. Hope it helps.

Filter sql query results using static values from "Select list" page item in Oracle APEX 5.0

I'm looking for a way to filter and display records using static values from a "Select list" page item. I have created a bar chart in APEX 5.0 using the following query:
select to_char(to_date(time_stamp,'YYYY-MM-DD-HH24:MI:SS'),'YYYY-MM-DD-HH24:MI:SS') as label, col2 as value from table1 where :P5_NEW_1 = col1 ;
The time_stamp column of table1 is of datatype varchar2 in my database and contains the date values in the format YYYY-MM-DD-HH24:MI:SS
e.g., values like below are stored in the time_stamp column
2015-08-26-10:17:15
2015-08-26-13:17:15
2015-09-17-12:45:54
2015-09-17-14:12:32
2015-10-06-10:01:42
2015-10-06-11:01:28
2015-10-06-05:01:28
and so on...
I have added a "Select list" item named "interval" on my form that contains a pre-populated list of values like 1hr, 6hrs.
Now I want to modify the above query such that following should happen:
For the value of 1hr selected from the drop down list, the query should check the time_stamp column and display the records for the last one hour
(i.e., records which fall in range to_char(sysdate - 1/24 ,'YYYY-MM-DD-HH24:MI:SS') to to_char(sysdate ,'YYYY-MM-DD-HH24:MI:SS')
And for the value of 6hrs selected from the drop-down list, the query should display records for the last 6 hours.
(i.e., records which fall in range to_char(sysdate - 6/24 ,'YYYY-MM-DD-HH24:MI:SS' to to_char(sysdate ,'YYYY-MM-DD-HH24:MI:SS') )
How do I modify my SQL query to add the above condition in my current SQL query, using static values from page item select list?
Hi you can try this:
SELECT to_char(to_date(time_stamp,'YYYY-MM-DD-HH24:MI:SS'),'YYYY-MM-DD-HH24:MI:SS') AS label, col2 AS VALUE
FROM table1
WHERE :P5_NEW_1 = col1 AND
to_date(time_stamp,'yyyy-mm-dd hh24:mi:ss') BETWEEN
(SYSDATE - (to_number(:SELECT_LIST_ID,'99')/24)) AND SYSDATE;
Just make sure that the static values has a return value equal to hr itself
For example, if static value is equal to 1HR then return value should be 1
:SELECT_LIST_ID is the select list id which holds the static values for the interval.
Hope this helps.