We've recently had the need to add columns to a few of our existing SQLite database tables. This can be done with ALTER TABLE ADD COLUMN. Of course, if the table has already been altered, we want to leave it alone. Unfortunately, SQLite doesn't support an IF NOT EXISTS clause on ALTER TABLE.
Our current workaround is to execute the ALTER TABLE statement and ignore any "duplicate column name" errors, just like this Python example (but in C++).
However, our usual approach to setting up database schemas is to have a .sql script containing CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS statements, which can be executed using sqlite3_exec or the sqlite3 command-line tool. We can't put ALTER TABLE in these script files because if that statement fails, anything after it won't be executed.
I want to have the table definitions in one place and not split between .sql and .cpp files. Is there a way to write a workaround to ALTER TABLE ADD COLUMN IF NOT EXISTS in pure SQLite SQL?
I have a 99% pure SQL method. The idea is to version your schema. You can do this in two ways:
Use the 'user_version' pragma command (PRAGMA user_version) to store an incremental number for your database schema version.
Store your version number in your own defined table.
In this way, when the software is started, it can check the database schema and, if needed, run your ALTER TABLE query, then increment the stored version. This is by far better than attempting various updates "blind", especially if your database grows and changes a few times over the years.
One workaround is to just create the columns and catch the exception/error that arise if the column already exist. When adding multiple columns, add them in separate ALTER TABLE statements so that one duplicate does not prevent the others from being created.
With sqlite-net, we did something like this. It's not perfect, since we can't distinguish duplicate sqlite errors from other sqlite errors.
Dictionary<string, string> columnNameToAddColumnSql = new Dictionary<string, string>
{
{
"Column1",
"ALTER TABLE MyTable ADD COLUMN Column1 INTEGER"
},
{
"Column2",
"ALTER TABLE MyTable ADD COLUMN Column2 TEXT"
}
};
foreach (var pair in columnNameToAddColumnSql)
{
string columnName = pair.Key;
string sql = pair.Value;
try
{
this.DB.ExecuteNonQuery(sql);
}
catch (System.Data.SQLite.SQLiteException e)
{
_log.Warn(e, string.Format("Failed to create column [{0}]. Most likely it already exists, which is fine.", columnName));
}
}
If you are doing this in a DB upgrade statement, perhaps the simplest way is to just catch the exception thrown if you are attempting to add a field that may already exist.
try {
db.execSQL("ALTER TABLE " + TABLE_NAME + " ADD COLUMN foo TEXT default null");
} catch (SQLiteException ex) {
Log.w(TAG, "Altering " + TABLE_NAME + ": " + ex.getMessage());
}
SQLite also supports a pragma statement called "table_info" which returns one row per column in a table with the name of the column (and other information about the column). You could use this in a query to check for the missing column, and if not present alter the table.
PRAGMA table_info(foo_table_name)
Sample output:
cid
name
type
notnull
dflt_value
pk
0
id
integer
0
null
1
1
type
text
0
null
0
2
data
json
0
null
0
http://www.sqlite.org/pragma.html#pragma_table_info
threre is a method of PRAGMA is table_info(table_name), it returns all the information of table.
Here is implementation how to use it for check column exists or not,
public boolean isColumnExists (String table, String column) {
boolean isExists = false
Cursor cursor;
try {
cursor = db.rawQuery("PRAGMA table_info("+ table +")", null);
if (cursor != null) {
while (cursor.moveToNext()) {
String name = cursor.getString(cursor.getColumnIndex("name"));
if (column.equalsIgnoreCase(name)) {
isExists = true;
break;
}
}
}
} finally {
if (cursor != null && !cursor.isClose())
cursor.close();
}
return isExists;
}
You can also use this query without using loop,
cursor = db.rawQuery("PRAGMA table_info("+ table +") where name = " + column, null);
For those want to use pragma table_info()'s result as part of a larger SQL.
select count(*) from
pragma_table_info('<table_name>')
where name='<column_name>';
The key part is to use pragma_table_info('<table_name>') instead of pragma table_info('<table_name>').
This answer is inspired by #Robert Hawkey 's reply. The reason I post it as a new answer is I don't have enough reputation to post it as comment.
I come up with this query
SELECT CASE (SELECT count(*) FROM pragma_table_info(''product'') c WHERE c.name = ''purchaseCopy'') WHEN 0 THEN ALTER TABLE product ADD purchaseCopy BLOB END
Inner query will return 0 or 1 if column exists.
Based on the result, alter the column
In case you're having this problem in flex/adobe air and find yourself here first, i've found a solution, and have posted it on a related question: ADD COLUMN to sqlite db IF NOT EXISTS - flex/air sqlite?
My comment here: https://stackoverflow.com/a/24928437/2678219
I took the answer above in C#/.Net, and rewrote it for Qt/C++, not to much changed, but I wanted to leave it here for anyone in the future looking for a C++'ish' answer.
bool MainWindow::isColumnExisting(QString &table, QString &columnName){
QSqlQuery q;
try {
if(q.exec("PRAGMA table_info("+ table +")"))
while (q.next()) {
QString name = q.value("name").toString();
if (columnName.toLower() == name.toLower())
return true;
}
} catch(exception){
return false;
}
return false;
}
You can alternatively use the CASE-WHEN TSQL statement in combination with pragma_table_info to know if a column exists:
select case(CNT)
WHEN 0 then printf('not found')
WHEN 1 then printf('found')
END
FROM (SELECT COUNT(*) AS CNT FROM pragma_table_info('myTableName') WHERE name='columnToCheck')
Here is my solution, but in python (I tried and failed to find any post on the topic related to python):
# modify table for legacy version which did not have leave type and leave time columns of rings3 table.
sql = 'PRAGMA table_info(rings3)' # get table info. returns an array of columns.
result = inquire (sql) # call homemade function to execute the inquiry
if len(result)<= 6: # if there are not enough columns add the leave type and leave time columns
sql = 'ALTER table rings3 ADD COLUMN leave_type varchar'
commit(sql) # call homemade function to execute sql
sql = 'ALTER table rings3 ADD COLUMN leave_time varchar'
commit(sql)
I used PRAGMA to get the table information. It returns a multidimensional array full of information about columns - one array per column. I count the number of arrays to get the number of columns. If there are not enough columns, then I add the columns using the ALTER TABLE command.
All these answers are fine if you execute one line at a time. However, the original question was to input a sql script that would be executed by a single db execute and all the solutions ( like checking to see if the column is there ahead of time ) would require the executing program either have knowledge of what tables and columns are being altered/added or do pre-processing and parsing of the input script to determine this information. Typically you are not going to run this in realtime or often. So the idea of catching an exception is acceptable and then moving on. Therein lies the problem...how to move on. Luckily the error message gives us all the information we need to do this. The idea is to execute the sql if it exceptions on an alter table call we can find the alter table line in the sql and return the remaining lines and execute until it either succeeds or no more matching alter table lines can be found. Heres some example code where we have sql scripts in an array. We iterate the array executing each script. We call it twice to get the alter table command to fail but the program succeeds because we remove the alter table command from the sql and re-execute the updated code.
#!/bin/sh
# the next line restarts using wish \
exec /opt/usr8.6.3/bin/tclsh8.6 "$0" ${1+"$#"}
foreach pkg {sqlite3 } {
if { [ catch {package require {*}$pkg } err ] != 0 } {
puts stderr "Unable to find package $pkg\n$err\n ... adjust your auto_path!";
}
}
array set sqlArray {
1 {
CREATE TABLE IF NOT EXISTS Notes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
note text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
);
CREATE TABLE IF NOT EXISTS Version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
);
INSERT INTO Version(version) values('1.0');
}
2 {
CREATE TABLE IF NOT EXISTS Tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
tag text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
);
ALTER TABLE Notes ADD COLUMN dump text;
INSERT INTO Version(version) values('2.0');
}
3 {
ALTER TABLE Version ADD COLUMN sql text;
INSERT INTO Version(version) values('3.0');
}
}
# create db command , use in memory database for demonstration purposes
sqlite3 db :memory:
proc createSchema { sqlArray } {
upvar $sqlArray sql
# execute each sql script in order
foreach version [lsort -integer [array names sql ] ] {
set cmd $sql($version)
set ok 0
while { !$ok && [string length $cmd ] } {
try {
db eval $cmd
set ok 1 ; # it succeeded if we get here
} on error { err backtrace } {
if { [regexp {duplicate column name: ([a-zA-Z0-9])} [string trim $err ] match columnname ] } {
puts "Error: $err ... trying again"
set cmd [removeAlterTable $cmd $columnname ]
} else {
throw DBERROR "$err\n$backtrace"
}
}
}
}
}
# return sqltext with alter table command with column name removed
# if no matching alter table line found or result is no lines then
# returns ""
proc removeAlterTable { sqltext columnname } {
set mode skip
set result [list]
foreach line [split $sqltext \n ] {
if { [string first "alter table" [string tolower [string trim $line] ] ] >= 0 } {
if { [string first $columnname $line ] } {
set mode add
continue;
}
}
if { $mode eq "add" } {
lappend result $line
}
}
if { $mode eq "skip" } {
puts stderr "Unable to find matching alter table line"
return ""
} elseif { [llength $result ] } {
return [ join $result \n ]
} else {
return ""
}
}
proc printSchema { } {
db eval { select * from sqlite_master } x {
puts "Table: $x(tbl_name)"
puts "$x(sql)"
puts "-------------"
}
}
createSchema sqlArray
printSchema
# run again to see if we get alter table errors
createSchema sqlArray
printSchema
expected output
Table: Notes
CREATE TABLE Notes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
note text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, dump text)
-------------
Table: sqlite_sequence
CREATE TABLE sqlite_sequence(name,seq)
-------------
Table: Version
CREATE TABLE Version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, sql text)
-------------
Table: Tags
CREATE TABLE Tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
tag text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
)
-------------
Error: duplicate column name: dump ... trying again
Error: duplicate column name: sql ... trying again
Table: Notes
CREATE TABLE Notes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
note text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, dump text)
-------------
Table: sqlite_sequence
CREATE TABLE sqlite_sequence(name,seq)
-------------
Table: Version
CREATE TABLE Version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
, sql text)
-------------
Table: Tags
CREATE TABLE Tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name text,
tag text,
createdDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) ) ,
updatedDate integer(4) DEFAULT ( cast( strftime('%s', 'now') as int ) )
)
-------------
select * from sqlite_master where type = 'table' and tbl_name = 'TableName' and sql like '%ColumnName%'
Logic: sql column in sqlite_master contains table definition, so it certainly contains string with column name.
As you are searching for a sub-string, it has its obvious limitations. So I would suggest to use even more restrictive sub-string in ColumnName, for example something like this (subject to testing as '`' character is not always there):
select * from sqlite_master where type = 'table' and tbl_name = 'MyTable' and sql like '%`MyColumn` TEXT%'
I solve it in 2 queries. This is my Unity3D script using System.Data.SQLite.
IDbCommand command = dbConnection.CreateCommand();
command.CommandText = #"SELECT count(*) FROM pragma_table_info('Candidat') c WHERE c.name = 'BirthPlace'";
IDataReader reader = command.ExecuteReader();
while (reader.Read())
{
try
{
if (int.TryParse(reader[0].ToString(), out int result))
{
if (result == 0)
{
command = dbConnection.CreateCommand();
command.CommandText = #"ALTER TABLE Candidat ADD COLUMN BirthPlace VARCHAR";
command.ExecuteNonQuery();
command.Dispose();
}
}
}
catch { throw; }
}
Apparently... in SQLite... the "alter table" statement does not generate exceptions if the column already exists.
Found this post in the support forumn and tested it.
Related
I have columns in bigquery like this:
expected output:
I am trying to merge columns into json using bigquery.
I am taking letter before underscore(common name ) as column then converting.
I am trying this query:
with selectdata as (
SELECT a_firstname, a_middlename,a_lastname FROM `account_id.Dataset.Table_name`
)
select TO_JSON_STRING(t) AS json_data FROM selectdata AS t;
How can I join columns with condition or with case to achieve this output in bigquery
Consider below approach
create temp function extract_keys(input string) returns array<string> language js as """
return Object.keys(JSON.parse(input));
""";
create temp function extract_values(input string) returns array<string> language js as """
return Object.values(JSON.parse(input));
""";
select * except(row_id) from (
select format('%t',t) row_id,
split(key, '_')[offset(0)] as col,
'{' || string_agg(format('"%s":"%s"', split(key, '_')[safe_offset(1)], value)) || '}' as value
from your_table t, unnest(extract_keys(to_json_string(t))) key with offset
join unnest(extract_values(to_json_string(t))) value with offset
using(offset)
group by row_id, col
)
pivot (any_value(value) for col in ('a','b','c'))
if applied to sample data in your question - output is
I need a custom function that takes two parameters, Column1 and Column2, so:
For each Row, return the value of Column1 but only if exists a Value in the Column2 else return null
I have tried this:
let ColumnsFilter = (Tabla,C1,C2)=>
Table.AddColumn(Tabla, "Custom", each if [C2] <> null then [C1] else null)
in
ColumnsFilter
And calling the function:
#"Previous Step" = .....
#"P" = ColumnsFilter(#"Previous Step","Column1","Column2")
in
P
And is not working. clearly I am not using the syntax properly.
In summary I need a table as input and a table as output adding custom columns.
How can I write this?
(Please don't tell me to use the assisted of Power Query, I need to write similar functions manually)
Since you're passing column names as text and individual rows are a record type, you have to use Record.Field to pull the right column (field) from the current row (record).
let
ColumnsFilter = (Tabla as table, C1 as text, C2 as text) as table =>
Table.AddColumn(Tabla, "Custom",
each if Record.Field(_, C2) <> null then Record.Field(_, C1) else null
)
in
ColumnsFilter
There is a scenario where I receive a string to the bigquery function and need to use it as a column name.
here is the function
CREATE OR REPLACE FUNCTION METADATA.GET_VALUE(column STRING, row_number int64) AS (
(SELECT column from WORK.temp WHERE rownumber = row_number)
);
When I call this function as select METADATA.GET_VALUE("TXCAMP10",149); I get the value as TXCAMP10 so we can say that it is processed as SELECT "TXCAMP10" from WORK.temp WHERE rownumber = 149 but I need it as SELECT TXCAMP10 from WORK.temp WHERE rownumber = 149 which will return some value from temp table lets suppose the value as A
so ultimately I need value A instead of column name i.e. TXCAMP10.
I tried using execute immediate like execute immediate("SELECT" || column || "from WORK.temp WHERE rownumber =" ||row_number) from this stack overflow post to resolve this issue but turns out I can't use it in a function.
How do I achieve required result?
I don't think you can achieve this result with the help of UDF in standard SQL in BigQuery.
But it is possible to do this with stored procedures in BigQuery and EXECUTE IMMEDIATE statement. Consider this code, which simulates the situation you have:
create or replace table d1.temp(
c1 int64,
c2 int64
);
insert into d1.temp values (1, 1), (2, 2);
create or replace procedure d1.GET_VALUE(column STRING, row_number int64, out result int64)
BEGIN
EXECUTE IMMEDIATE 'SELECT ' || column || ' from d1.temp where c2 = ?' into result using row_number;
END;
BEGIN
DECLARE result_c1 INT64;
call d1.GET_VALUE("c1", 1, result_c1);
select result_c1;
END;
After some research and trial-error methods, I used this workaround to solve this issue. It may not be the best solution when you have too many columns but it surely works.
CREATE OR REPLACE FUNCTION METADATA.GET_VALUE(column STRING, row_number int64) AS (
(SELECT case
when column_name = 'a' then a
when column_name = 'b' then b
when column_name = 'c' then c
when column_name = 'd' then d
when column_name = 'e' then e
end from WORK.temp WHERE rownumber = row_number)
);
And this gives the required results.
Point to note: the number of columns you use in the case statement should be of the same datatype else it won't work
I don't understand whats wrong with the code, I have read a lot of times but I can't find the error
pstmt = con->prepareStatement("SELECT (?) FROM votos WHERE id = (?)");
pstmt->setString(1, eleccion);
pstmt->setInt(2, p->getId());
res = pstmt->executeQuery();
while(res->next())
{
p->setVoto(res->getInt(1));
}
When the eleccion and id variables are Provincial and 1 respectively the getInt(1) function should return 1, but it returns 0.
The command (in the mysql command line):
SELECT Provincial from Votos WHERE id=1
Returns a table with one row and one column with the value 1
Side notes:
Spelling was checked
The getId() function works correctly
The compiler doesn't give any error
You can't use a placeholder in a prepared query for a column name. It's returning the value of the string eleccion, not using it as the name of a column in the table. You need to do string concatenation to substitute the column name.
std::string sql = std::string("SELECT `") + eleccion + "` FROM votos WHERE id = ?";
pstmt = con->prepareStatement(sql.c_str());
pstmt->setInt(1, p->getId());
res = pstmt->executeQuery();
while(res->next())
{
p->setVoto(res->getInt(1));
}
If the value of eleccion is coming from the user or some other untrusted source, make sure you validate it before concatenating, to prevent SQL injection.
[Posting question from customer]
I would like to do periodic loads into SQL Data Warehouse using Polybase. My plan is to create a Stored Procedure that takes in the "LOCATION" parameter for Create External Table, but the parameter is failing due to syntax. Here's a sample of the code I've written:
alter PROCEDURE CETFromNewLocation
as
begin
Declare #location varchar(100)='data/2015/2015831';
Create External TABLE stg_tbl (
[DateId] int NULL
)
WITH (
LOCATION= #location,
DATA_SOURCE = my_external_source,
FILE_FORMAT = my_external_file_format,
REJECT_TYPE = VALUE,
REJECT_VALUE = 0
)
end
The error message I'm getting is:
> Parse error at line: 25, column: 16: Incorrect syntax near
> '#location'.
You cannot use variables in DDL statements. Try using dynamic sql - form the string for your DDL and execute it using sp_executesql stored procedure like below.
ALTER PROCEDURE CETFromNewLocation AS
BEGIN
DECLARE #location varchar(100)
SET #location = 'data/2015/2015831'
DECLARE #CreateExternalTableString varchar(100)
SET #CreateExternalTableString =
'Create External TABLE stg_tbl (
[DateId] int NULL
)
WITH (LOCATION = ' + #location + ',
DATA_SOURCE = my_external_source,
FILE_FORMAT = my_external_file_format,
REJECT_TYPE = VALUE,
REJECT_VALUE = 0
)'
EXEC sp_executesql #CreateExternalTableString
END