when does an sqlite compiled statement get 'executed' - c++

I'm writing a light wrapper for the SQLite API.
Basically, I'm curious about how/when an SQLite pre-compiled statement gets executed...
When i go:
char buffer[] = "INSERT INTO example VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)";
sqlite3_stmt* stmt;
sqlite3_prepare_v2(mDb, buffer, strlen(buffer), &stmt, NULL);
for (unsigned i = 0; i < mVal; i++)
{
std::string id = getID();
sqlite3_bind_text(stmt, 1, id.c_str(), id.size(), SQLITE_STATIC);
sqlite3_bind_double(stmt, 2, getDouble());
sqlite3_bind_double(stmt, 3, getDouble());
sqlite3_bind_double(stmt, 4, getDouble());
sqlite3_bind_int(stmt, 5, getInt());
sqlite3_bind_int(stmt, 6, getInt());
sqlite3_bind_int(stmt, 7, getInt());
if (sqlite3_step(stmt) != SQLITE_DONE)
{
printf("Commit Failed!\n");
}
sqlite3_reset(stmt);
}
sqlite3_finalize(stmt);
At what point is the actual sql executed? Is it during the call to sqlite3_prepare_v2, or during the first sqlite3_step?
Any clarity is greatly appreciated :)
Cheers
Jarrett

According to the SQLite documentation for sqlite3_prepare, we see:
To execute an SQL query, it must first be compiled into a byte-code program using
one of these routines:
sqlite3_prepare, sqlite3_prepare_v2, sqlite3_prepare16, sqlite3_prepare16_v2.
And for sqlite3_step:
After a prepared statement has been prepared using either sqlite3_prepare_v2() or
sqlite3_prepare16_v2() or one of the legacy interfaces sqlite3_prepare() or
sqlite3_prepare16(), this function must be called one or more times to evaluate
the statement.
More information:
SQLite has a virtual machine that does all necessary actions to execute your code on the selected database. sqlite3_prepare (and its family) compiles your SQL statement into byte code that can be executed on this virtual machine. On the other hand, sqlite3_step executes that byte-code in the VM.

Related

Very long query execution using C++, Sql Server and ODBC connection

I'm using the following code to connect to a sql-server database. I can run all other queries with no difficulties using the same function. However, one query executes in Sql management studio and in R using RODBC in ~11 seconds but takes over an hour (at the "SQLExecDirect" function) using c++. Has anyone else had this issue and how did you resolve it?
std::string sqlQuery="[myquery]";
SQLHANDLE sqlconnectionhandle;
SQLAllocHandle(SQL_HANDLE_DBC, sqlenvhandle, &sqlconnectionhandle);
SQLCHAR retconstring[1024];
SQLDriverConnect (sqlconnectionhandle, NULL,
conn_str,
len, retconstring, 1024, NULL,SQL_DRIVER_NOPROMPT);
SQLHANDLE sqlstatementhandle;
SQLAllocHandle(SQL_HANDLE_STMT, sqlconnectionhandle, &sqlstatementhandle);
//this is where the program "hangs" for over an hour
if(SQL_SUCCESS!=SQLExecDirect(sqlstatementhandle, (SQLCHAR*)(sqlQuery.c_str()), SQL_NTS)){
show_error(SQL_HANDLE_STMT, sqlstatementhandle, errorMsg);
return;
}
int numRow=0;
while(SQLFetch(sqlstatementhandle)==SQL_SUCCESS){
for(int i=1; i<=numFields+1; ++i){
double myVal;
SQLGetData(sqlstatementhandle, i, SQL_C_DOUBLE, &myVal, 0, NULL);
cb(myVal, numRow, i-1); //callback function defined elsewhere
}
numRow++;
}
SQLFreeHandle(SQL_HANDLE_STMT, sqlstatementhandle );
SQLDisconnect(sqlconnectionhandle);
SQLFreeHandle(SQL_HANDLE_DBC, sqlconnectionhandle);
That is because the ArithAbort property on your database is off.
See also http://www.sommarskog.se/query-plan-mysteries.html
You can check it and correct it with this small script
declare #value sql_variant
select #value = SESSIONPROPERTY('ARITHABORT')
if #value <> 1
begin
USE master
ALTER DATABASE [your_database] SET ARITHABORT ON WITH NO_WAIT
use your_database
end
I figured this out: my connection string's default database was different than the one that I had as the default database in SSMS. Switching this made the query run in the same speed as SSMS.

RAM consumption inside a C++ program using Sqlite3 blob

I'm using sqlite3 dbms inside a C++ program that I use mainly to store files as blob objects (I know that's not the best option).
Obviously I write them incrementally since they can be sometimes large (40-80MB) in order to do so I have to create first a placeholder of the blob using the binding function sqlite3_bind_zeroblob(...) and after that I open the blob writing and reading incrementally from and to it.
The problem I'm facing is that when i create the blob placeholder (during sqlite3_step) the RAM consumption of my app reaches 80-160MB for 2-3 seconds, once it has been created the RAM consumption goes back to 2-3MB at most.
I do not get why! If they created a way to write to a blob incrementally there is for sure a way to create that stupid placeholder without wasting 160MB of RAM, but I didn't find it. Have you got any suggestion?
sqlite3_stmt* stm = NULL;
sqlite3_blob *BLOB = NULL;
rc = sqlite3_prepare_v2(db, sql.c_str(), -1, &stm, NULL);
rc = sqlite3_bind_blob(stm, 1, wpath.c_str(), wpath.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_bind_text(stm, 2, hash.c_str(), hash.size(), SQLITE_STATIC);
rc = sqlite3_bind_zeroblob(stm, 3, size);
rc = sqlite3_bind_int(stm, 4, versione);
rc = sqlite3_bind_blob(stm, 5, last.c_str(), last.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_step(stm);
if (rc != SQLITE_DONE) {
fprintf(stderr, " This file was already present in the database!\n", rc);
return;
}
else {
fprintf(stdout, "Record FILE created successfully\n");
}
It is an issue reported HERE.
And the oficial answer is:
In order for zeroblobs to work as above (using a fixed amount of
memory no matter how big they are) all zeroblobs must be at the end
of the row. In other words, the columns of the table that are
receiving the zeroblobs must be the last columns in the table. If
any non-zero content follows the zeroblob, then the zeroblob is
expanded into a literal sequence of zero bytes, meaning memory must
be allocated for the entire zeroblob.
So you need to change the order to fix it:
sqlite3_stmt* stm = NULL;
sqlite3_blob *BLOB = NULL;
rc = sqlite3_prepare_v2(db, sql.c_str(), -1, &stm, NULL);
rc = sqlite3_bind_blob(stm, 1, wpath.c_str(), wpath.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_bind_text(stm, 2, hash.c_str(), hash.size(), SQLITE_STATIC);
rc = sqlite3_bind_int(stm, 3, versione);
rc = sqlite3_bind_blob(stm, 4, last.c_str(), last.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_bind_zeroblob(stm, 5, size);
rc = sqlite3_step(stm);
if (rc != SQLITE_DONE) {
fprintf(stderr, " This file was already present in the database!\n", rc);
return;
}
else {
fprintf(stdout, "Record FILE created successfully\n");
}

sqlite - delete and get all values of some column from affected rows [duplicate]

I heard of using sqlite3_prepare_v2 instead of sqlite_exec to get integers from database, but I failed to find any examples. This page wasn't helpful also. Now I am getting strings from database, so I need to parse them with atoi and this seems to be slow and ineffective.
There a lot of similar questions on SO, but they are about obj-c and iOS SDK. I need C/C++ hint or example.
Thanks in advance.
After sqlite3_prepare has succeeded, you must not forget to clean up the statement with sqlite3_finalize.
To get the result records, call sqlite3_step until it does not return SQLITE_ROW.
To get the values of the current result record, call the sqlite3_column_* functions:
sqlite3_stmt *stmt;
if (sqlite3_prepare_v2(db, "SELECT 42", -1, &stmt, NULL) != SQLITE_OK)
...error...
else {
for (;;) {
int rc = sqlite3_step(stmt);
if (rc == SQLITE_DONE)
break;
if (rc != SQLITE_ROW) {
...error...
break;
}
printf("value: %d\n", sqlite3_column_int(stmt, 0));
}
sqlite3_finalize(stmt);
}
sqlite3_column_int(result, columnNum); will return one column from the current row of your result as an int.
Your prepare function is to prepare your query, it has nothing to do with how the results are interpreted. All data in sqlite3 is stored textually, you use the appropriate function to retrieve a value in the type you believe it should be.

Getting int values from SQLite

I heard of using sqlite3_prepare_v2 instead of sqlite_exec to get integers from database, but I failed to find any examples. This page wasn't helpful also. Now I am getting strings from database, so I need to parse them with atoi and this seems to be slow and ineffective.
There a lot of similar questions on SO, but they are about obj-c and iOS SDK. I need C/C++ hint or example.
Thanks in advance.
After sqlite3_prepare has succeeded, you must not forget to clean up the statement with sqlite3_finalize.
To get the result records, call sqlite3_step until it does not return SQLITE_ROW.
To get the values of the current result record, call the sqlite3_column_* functions:
sqlite3_stmt *stmt;
if (sqlite3_prepare_v2(db, "SELECT 42", -1, &stmt, NULL) != SQLITE_OK)
...error...
else {
for (;;) {
int rc = sqlite3_step(stmt);
if (rc == SQLITE_DONE)
break;
if (rc != SQLITE_ROW) {
...error...
break;
}
printf("value: %d\n", sqlite3_column_int(stmt, 0));
}
sqlite3_finalize(stmt);
}
sqlite3_column_int(result, columnNum); will return one column from the current row of your result as an int.
Your prepare function is to prepare your query, it has nothing to do with how the results are interpreted. All data in sqlite3 is stored textually, you use the appropriate function to retrieve a value in the type you believe it should be.

How to write to a varchar(max) column using ODBC

Summary: I'm trying to write a text string to a column of type varchar(max) using ODBC and SQL Server 2005. It fails if the length of the string is greater than 8000. Help!
I have some C++ code that uses ODBC (SQL Native Client) to write a text string to a table. If I change the column from, say, varchar(100) to varchar(max) and try to write a string with length greater than 8000, the write fails with the following error
[Microsoft][ODBC SQL Server
Driver]String data, right truncation
So, can anyone advise me on if this can be done, and how?
Some example (not production) code that shows what I'm trying to do:
SQLHENV hEnv = NULL;
SQLRETURN iError = SQLAllocEnv(&hEnv);
HDBC hDbc = NULL;
SQLAllocConnect(hEnv, &hDbc);
const char* pszConnStr = "Driver={SQL Server};Server=127.0.0.1;Database=MyTestDB";
UCHAR szConnectOut[SQL_MAX_MESSAGE_LENGTH];
SWORD iConnectOutLen = 0;
iError = SQLDriverConnect(hDbc, NULL, (unsigned char*)pszConnStr,
SQL_NTS, szConnectOut,
(SQL_MAX_MESSAGE_LENGTH-1), &iConnectOutLen,
SQL_DRIVER_COMPLETE);
HSTMT hStmt = NULL;
iError = SQLAllocStmt(hDbc, &hStmt);
const char* pszSQL = "INSERT INTO MyTestTable (LongStr) VALUES (?)";
iError = SQLPrepare(hStmt, (SQLCHAR*)pszSQL, SQL_NTS);
char* pszBigString = AllocBigString(8001);
iError = SQLSetParam(hStmt, 1, SQL_C_CHAR, SQL_VARCHAR, 0, 0, (SQLPOINTER)pszBigString, NULL);
iError = SQLExecute(hStmt); // Returns SQL_ERROR if pszBigString len > 8000
The table MyTestTable contains a single colum defined as varchar(max). The function AllocBigString (not shown) creates a string of arbitrary length.
I understand that previous versions of SQL Server had an 8000 character limit to varchars, but not why is this happening in SQL 2005?
Thanks,
Andy
You sure you load the SQL Native Driver for 2005, not the old driver for 2000? The native driver name is {SQL Server Native Client 10.0} for 2k8 or {SQL Native Client} for 2k5
The error message ODBC SQL Server Driver seem to indicate the old 2k driver (I may be wrong, haven't touch ODBC in like 10 years now).
Turns out that although the fix works for SQLSetParam, it does not work for SQLBindParameter.
For example:
int iLength = 18001;
char* pszBigString = new char[iLength + 1];
memset(pszBigString, 'a', iLength);
pszBigString[iLength] = 0;
LONG_PTR lLength = SQL_NTS;
::SQLBindParameter(hStmt, 1, SQL_PARAM_INPUT,
SQL_C_CHAR,
SQL_VARCHAR,
iLength, 0, pszBigString, iLength * sizeof(TCHAR),
&lLength);
will result in the same 22001 "String data, right truncation" error, regardless of which driver is used.
In fact, my experiments have shown that you do not actually need to install version 10 of the client driver. Instead you should use SQL_LONGVARCHAR instead of SQL_VARCHAR if you expect the lengths of your strings to exceed 8000 characters. You could potentially perform a mass find-and-replace, but it's possible that using SQL_LONGVARCHAR might incur some sort of penalty (although that's pure speculation; it's an 'extended datatype').
I have tested this successfully with both drivers on Windows XP:
{SQL Server} 2000.85.1117.00 (04/08/2004)
{SQL Server Native Client 10.0} 2007.100.1600.22 (10/07/2008)