I am looking for an example of how to use QuestDB embedded in Java process. I have a table
CREATE TABLE market_tick ( tickTimeStamp TIMESTAMP, token SYMBOL)
timestamp(tickTimeStamp)
PARTITION BY MONTH
and I append rows like
TableWriter tableWriter = questDB.getTableWrite( "market_tick" ) ;
TableWriter.Row row = tableWriter.newRow() ;
row.putTimestamp( 0, timeStamp );
String token = String.valueOf(event.getToken()) ;
row.putSym(1, token );
tableWriter.commit();
and I have exception
io.questdb.cairo.CairoException: [5] Cannot remove directory: C:\trade\questdb\market_tick\1970-01\token.k
at io.questdb.std.ThreadLocal.initialValue(ThreadLocal.java:36)
at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
at io.questdb.cairo.CairoException.instance(CairoException.java:42)
How to insert rows using Java embedded API?
Since the table market_tick has designated timestamp, you cannot use tableWriter.newRow() to create new row but have to use tableWriter.newRow(timeStamp). The rest of the code is correct
TableWriter tableWriter = questDB.getTableWrite( "market_tick" ) ;
TableWriter.Row row = tableWriter.newRow(timeStampMicroSeconds) ;
String token = String.valueOf(event.getToken()) ;
row.putSym(1, token );
tableWriter.commit();
There is a scenario where I receive a string to the bigquery function and need to use it as a column name.
here is the function
CREATE OR REPLACE FUNCTION METADATA.GET_VALUE(column STRING, row_number int64) AS (
(SELECT column from WORK.temp WHERE rownumber = row_number)
);
When I call this function as select METADATA.GET_VALUE("TXCAMP10",149); I get the value as TXCAMP10 so we can say that it is processed as SELECT "TXCAMP10" from WORK.temp WHERE rownumber = 149 but I need it as SELECT TXCAMP10 from WORK.temp WHERE rownumber = 149 which will return some value from temp table lets suppose the value as A
so ultimately I need value A instead of column name i.e. TXCAMP10.
I tried using execute immediate like execute immediate("SELECT" || column || "from WORK.temp WHERE rownumber =" ||row_number) from this stack overflow post to resolve this issue but turns out I can't use it in a function.
How do I achieve required result?
I don't think you can achieve this result with the help of UDF in standard SQL in BigQuery.
But it is possible to do this with stored procedures in BigQuery and EXECUTE IMMEDIATE statement. Consider this code, which simulates the situation you have:
create or replace table d1.temp(
c1 int64,
c2 int64
);
insert into d1.temp values (1, 1), (2, 2);
create or replace procedure d1.GET_VALUE(column STRING, row_number int64, out result int64)
BEGIN
EXECUTE IMMEDIATE 'SELECT ' || column || ' from d1.temp where c2 = ?' into result using row_number;
END;
BEGIN
DECLARE result_c1 INT64;
call d1.GET_VALUE("c1", 1, result_c1);
select result_c1;
END;
After some research and trial-error methods, I used this workaround to solve this issue. It may not be the best solution when you have too many columns but it surely works.
CREATE OR REPLACE FUNCTION METADATA.GET_VALUE(column STRING, row_number int64) AS (
(SELECT case
when column_name = 'a' then a
when column_name = 'b' then b
when column_name = 'c' then c
when column_name = 'd' then d
when column_name = 'e' then e
end from WORK.temp WHERE rownumber = row_number)
);
And this gives the required results.
Point to note: the number of columns you use in the case statement should be of the same datatype else it won't work
We've recently had the need to add columns to a few of our existing SQLite database tables. This can be done with ALTER TABLE ADD COLUMN. Of course, if the table has already been altered, we want to leave it alone. Unfortunately, SQLite doesn't support an IF NOT EXISTS clause on ALTER TABLE.
Our current workaround is to execute the ALTER TABLE statement and ignore any "duplicate column name" errors, just like this Python example (but in C++).
However, our usual approach to setting up database schemas is to have a .sql script containing CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS statements, which can be executed using sqlite3_exec or the sqlite3 command-line tool. We can't put ALTER TABLE in these script files because if that statement fails, anything after it won't be executed.
I want to have the table definitions in one place and not split between .sql and .cpp files. Is there a way to write a workaround to ALTER TABLE ADD COLUMN IF NOT EXISTS in pure SQLite SQL?
I have a 99% pure SQL method. The idea is to version your schema. You can do this in two ways:
Use the 'user_version' pragma command (PRAGMA user_version) to store an incremental number for your database schema version.
Store your version number in your own defined table.
In this way, when the software is started, it can check the database schema and, if needed, run your ALTER TABLE query, then increment the stored version. This is by far better than attempting various updates "blind", especially if your database grows and changes a few times over the years.
One workaround is to just create the columns and catch the exception/error that arise if the column already exist. When adding multiple columns, add them in separate ALTER TABLE statements so that one duplicate does not prevent the others from being created.
With sqlite-net, we did something like this. It's not perfect, since we can't distinguish duplicate sqlite errors from other sqlite errors.
Dictionary<string, string> columnNameToAddColumnSql = new Dictionary<string, string>
{
{
"Column1",
"ALTER TABLE MyTable ADD COLUMN Column1 INTEGER"
},
{
"Column2",
"ALTER TABLE MyTable ADD COLUMN Column2 TEXT"
}
};
foreach (var pair in columnNameToAddColumnSql)
{
string columnName = pair.Key;
string sql = pair.Value;
try
{
this.DB.ExecuteNonQuery(sql);
}
catch (System.Data.SQLite.SQLiteException e)
{
_log.Warn(e, string.Format("Failed to create column [{0}]. Most likely it already exists, which is fine.", columnName));
}
}
If you are doing this in a DB upgrade statement, perhaps the simplest way is to just catch the exception thrown if you are attempting to add a field that may already exist.
try {
db.execSQL("ALTER TABLE " + TABLE_NAME + " ADD COLUMN foo TEXT default null");
} catch (SQLiteException ex) {
Log.w(TAG, "Altering " + TABLE_NAME + ": " + ex.getMessage());
}
SQLite also supports a pragma statement called "table_info" which returns one row per column in a table with the name of the column (and other information about the column). You could use this in a query to check for the missing column, and if not present alter the table.
PRAGMA table_info(foo_table_name)
Sample output:
cid
name
type
notnull
dflt_value
pk
0
id
integer
0
null
1
1
type
text
0
null
0
2
data
json
0
null
0
http://www.sqlite.org/pragma.html#pragma_table_info
threre is a method of PRAGMA is table_info(table_name), it returns all the information of table.
Here is implementation how to use it for check column exists or not,
public boolean isColumnExists (String table, String column) {
boolean isExists = false
Cursor cursor;
try {
cursor = db.rawQuery("PRAGMA table_info("+ table +")", null);
if (cursor != null) {
while (cursor.moveToNext()) {
String name = cursor.getString(cursor.getColumnIndex("name"));
if (column.equalsIgnoreCase(name)) {
isExists = true;
break;
}
}
}
} finally {
if (cursor != null && !cursor.isClose())
cursor.close();
}
return isExists;
}
You can also use this query without using loop,
cursor = db.rawQuery("PRAGMA table_info("+ table +") where name = " + column, null);
For those want to use pragma table_info()'s result as part of a larger SQL.
select count(*) from
pragma_table_info('<table_name>')
where name='<column_name>';
The key part is to use pragma_table_info('<table_name>') instead of pragma table_info('<table_name>').
This answer is inspired by #Robert Hawkey 's reply. The reason I post it as a new answer is I don't have enough reputation to post it as comment.
I come up with this query
SELECT CASE (SELECT count(*) FROM pragma_table_info(''product'') c WHERE c.name = ''purchaseCopy'') WHEN 0 THEN ALTER TABLE product ADD purchaseCopy BLOB END
Inner query will return 0 or 1 if column exists.
Based on the result, alter the column
In case you're having this problem in flex/adobe air and find yourself here first, i've found a solution, and have posted it on a related question: ADD COLUMN to sqlite db IF NOT EXISTS - flex/air sqlite?
My comment here: https://stackoverflow.com/a/24928437/2678219
I took the answer above in C#/.Net, and rewrote it for Qt/C++, not to much changed, but I wanted to leave it here for anyone in the future looking for a C++'ish' answer.
bool MainWindow::isColumnExisting(QString &table, QString &columnName){
QSqlQuery q;
try {
if(q.exec("PRAGMA table_info("+ table +")"))
while (q.next()) {
QString name = q.value("name").toString();
if (columnName.toLower() == name.toLower())
return true;
}
} catch(exception){
return false;
}
return false;
}
You can alternatively use the CASE-WHEN TSQL statement in combination with pragma_table_info to know if a column exists:
select case(CNT)
WHEN 0 then printf('not found')
WHEN 1 then printf('found')
END
FROM (SELECT COUNT(*) AS CNT FROM pragma_table_info('myTableName') WHERE name='columnToCheck')
Here is my solution, but in python (I tried and failed to find any post on the topic related to python):
# modify table for legacy version which did not have leave type and leave time columns of rings3 table.
sql = 'PRAGMA table_info(rings3)' # get table info. returns an array of columns.
result = inquire (sql) # call homemade function to execute the inquiry
if len(result)<= 6: # if there are not enough columns add the leave type and leave time columns
sql = 'ALTER table rings3 ADD COLUMN leave_type varchar'
commit(sql) # call homemade function to execute sql
sql = 'ALTER table rings3 ADD COLUMN leave_time varchar'
commit(sql)
I used PRAGMA to get the table information. It returns a multidimensional array full of information about columns - one array per column. I count the number of arrays to get the number of columns. If there are not enough columns, then I add the columns using the ALTER TABLE command.
All these answers are fine if you execute one line at a time. However, the original question was to input a sql script that would be executed by a single db execute and all the solutions ( like checking to see if the column is there ahead of time ) would require the executing program either have knowledge of what tables and columns are being altered/added or do pre-processing and parsing of the input script to determine this information. Typically you are not going to run this in realtime or often. So the idea of catching an exception is acceptable and then moving on. Therein lies the problem...how to move on. Luckily the error message gives us all the information we need to do this. The idea is to execute the sql if it exceptions on an alter table call we can find the alter table line in the sql and return the remaining lines and execute until it either succeeds or no more matching alter table lines can be found. Heres some example code where we have sql scripts in an array. We iterate the array executing each script. We call it twice to get the alter table command to fail but the program succeeds because we remove the alter table command from the sql and re-execute the updated code.
#!/bin/sh
# the next line restarts using wish \
exec /opt/usr8.6.3/bin/tclsh8.6 "$0" ${1+"$#"}
foreach pkg {sqlite3 } {
if { [ catch {package require {*}$pkg } err ] != 0 } {
puts stderr "Unable to find package $pkg\n$err\n ... adjust your auto_path!";
}
}
array set sqlArray {
1 {
CREATE TABLE IF NOT EXISTS Notes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
note text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
);
CREATE TABLE IF NOT EXISTS Version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
);
INSERT INTO Version(version) values('1.0');
}
2 {
CREATE TABLE IF NOT EXISTS Tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
tag text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
);
ALTER TABLE Notes ADD COLUMN dump text;
INSERT INTO Version(version) values('2.0');
}
3 {
ALTER TABLE Version ADD COLUMN sql text;
INSERT INTO Version(version) values('3.0');
}
}
# create db command , use in memory database for demonstration purposes
sqlite3 db :memory:
proc createSchema { sqlArray } {
upvar $sqlArray sql
# execute each sql script in order
foreach version [lsort -integer [array names sql ] ] {
set cmd $sql($version)
set ok 0
while { !$ok && [string length $cmd ] } {
try {
db eval $cmd
set ok 1 ; # it succeeded if we get here
} on error { err backtrace } {
if { [regexp {duplicate column name: ([a-zA-Z0-9])} [string trim $err ] match columnname ] } {
puts "Error: $err ... trying again"
set cmd [removeAlterTable $cmd $columnname ]
} else {
throw DBERROR "$err\n$backtrace"
}
}
}
}
}
# return sqltext with alter table command with column name removed
# if no matching alter table line found or result is no lines then
# returns ""
proc removeAlterTable { sqltext columnname } {
set mode skip
set result [list]
foreach line [split $sqltext \n ] {
if { [string first "alter table" [string tolower [string trim $line] ] ] >= 0 } {
if { [string first $columnname $line ] } {
set mode add
continue;
}
}
if { $mode eq "add" } {
lappend result $line
}
}
if { $mode eq "skip" } {
puts stderr "Unable to find matching alter table line"
return ""
} elseif { [llength $result ] } {
return [ join $result \n ]
} else {
return ""
}
}
proc printSchema { } {
db eval { select * from sqlite_master } x {
puts "Table: $x(tbl_name)"
puts "$x(sql)"
puts "-------------"
}
}
createSchema sqlArray
printSchema
# run again to see if we get alter table errors
createSchema sqlArray
printSchema
expected output
Table: Notes
CREATE TABLE Notes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
note text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, dump text)
-------------
Table: sqlite_sequence
CREATE TABLE sqlite_sequence(name,seq)
-------------
Table: Version
CREATE TABLE Version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, sql text)
-------------
Table: Tags
CREATE TABLE Tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
tag text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
)
-------------
Error: duplicate column name: dump ... trying again
Error: duplicate column name: sql ... trying again
Table: Notes
CREATE TABLE Notes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
note text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, dump text)
-------------
Table: sqlite_sequence
CREATE TABLE sqlite_sequence(name,seq)
-------------
Table: Version
CREATE TABLE Version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, sql text)
-------------
Table: Tags
CREATE TABLE Tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
tag text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
)
-------------
select * from sqlite_master where type = 'table' and tbl_name = 'TableName' and sql like '%ColumnName%'
Logic: sql column in sqlite_master contains table definition, so it certainly contains string with column name.
As you are searching for a sub-string, it has its obvious limitations. So I would suggest to use even more restrictive sub-string in ColumnName, for example something like this (subject to testing as '`' character is not always there):
select * from sqlite_master where type = 'table' and tbl_name = 'MyTable' and sql like '%`MyColumn` TEXT%'
I solve it in 2 queries. This is my Unity3D script using System.Data.SQLite.
IDbCommand command = dbConnection.CreateCommand();
command.CommandText = #"SELECT count(*) FROM pragma_table_info('Candidat') c WHERE c.name = 'BirthPlace'";
IDataReader reader = command.ExecuteReader();
while (reader.Read())
{
try
{
if (int.TryParse(reader[0].ToString(), out int result))
{
if (result == 0)
{
command = dbConnection.CreateCommand();
command.CommandText = #"ALTER TABLE Candidat ADD COLUMN BirthPlace VARCHAR";
command.ExecuteNonQuery();
command.Dispose();
}
}
}
catch { throw; }
}
Apparently... in SQLite... the "alter table" statement does not generate exceptions if the column already exists.
Found this post in the support forumn and tested it.
I'm trying to execute a raw sql string using my custom Connection class..
(The Execute method simply executes the given sql statement..)
MyConnection.Execute(
"DECLARE #i int "
"SET #i = 0 "
"WHILE (#i < 10000) BEGIN "
"INSERT INTO my_table VALUES (1, 2, 3) "
"SET #i = #i + 1");
Running this statement gives me a syntax error:
Code = 80040e14
Code meaning = I
Source = ASEOLEDB
Description = [42000]
[ASEOLEDB] Incorrect syntax near '1'.
As you can see, I'm connecting to Sybase Adaptive Server Enterprise, which accepts T-SQL..
I'm assuming there is something wrong with my sql string,, or with how it's formatted?
Thanks
PS I'm sure there's nothing wrong with my "Execute" method, as a single line delete statement works fine..
i found this example in the web
there is no END at your code and no ; but the ';', i think, you don't need.
DECLARE #MyCounter int;
SET #MyCounter = 0;
WHILE (#MyCounter < 26)
BEGIN;
INSERT INTO TestTable VALUES (#MyCounter, CHAR( ( #MyCounter + ASCII('a') ) ) );
SET #MyCounter = #MyCounter + 1;
END;
BR
Alex
In database I have partitioning table by column 'status' for better performance. My database administrator ask me about put in query value for that column directly in sql (not bind by parameter).
I can change binding by set hint QueryHints.BIND_PARAMETERS on false, but then all parameters are inside sql.
Can I set not bind only on 'status' parameter ?
Example result when BIND_PARAMETERS = true
SELECT t0.* FROM S_JOBS_ORG_UNIT_CFG t0
WHERE ((((t0.ORG_ID = ?) AND (t0.SCHEDULER_NEXT_ACTIVATION < SYSDATE)) AND (t0.ACTIVE = ?))
AND NOT EXISTS (SELECT ? FROM S_JOBS t1 WHERE (((t1.ORDER_ID = t0.ORDER_ID) AND (t1.ORG_ID = t0.ORG_ID)) AND NOT ((t1.STATUS = ?)))) )
bind => [472100, Y, 1, E]
and result when BIND_PARAMETERS = false
SELECT t0.* FROM S_JOBS_ORG_UNIT_CFG t0
WHERE ((((t0.ORG_ID = 472100) AND (t0.SCHEDULER_NEXT_ACTIVATION < SYSDATE)) AND (t0.ACTIVE = Y))
AND NOT EXISTS (SELECT 1 FROM S_JOBS t1 WHERE (((t1.ORDER_ID = t0.ORDER_ID) AND (t1.ORG_ID = t0.ORG_ID)) AND NOT ((t1.STATUS = E)))) )
Code:
Query jobOrgUnitCfgQuery = entityManager.createQuery(
"SELECT c FROM JobOrgUnitCfg c WHERE c.orgId = :orgId and c.schedulerNextActivation < current_timestamp and c.active = :active and " +
" not exists (SELECT j FROM Job j WHERE j.orderId = c.orderId and j.orgId = c.orgId and j.status <> 'E')");
jobOrgUnitCfgQuery.setParameter("orgId", orgId);
jobOrgUnitCfgQuery.setParameter("active", Boolean.TRUE);
return jobOrgUnitCfgQuery.getResultList();
I think your best bet is just to programmatically build your query like you're doing with a hard coded status and escape the other paramaters manually to avoid SQL Injection.