RAM consumption inside a C++ program using Sqlite3 blob - c++

I'm using sqlite3 dbms inside a C++ program that I use mainly to store files as blob objects (I know that's not the best option).
Obviously I write them incrementally since they can be sometimes large (40-80MB) in order to do so I have to create first a placeholder of the blob using the binding function sqlite3_bind_zeroblob(...) and after that I open the blob writing and reading incrementally from and to it.
The problem I'm facing is that when i create the blob placeholder (during sqlite3_step) the RAM consumption of my app reaches 80-160MB for 2-3 seconds, once it has been created the RAM consumption goes back to 2-3MB at most.
I do not get why! If they created a way to write to a blob incrementally there is for sure a way to create that stupid placeholder without wasting 160MB of RAM, but I didn't find it. Have you got any suggestion?
sqlite3_stmt* stm = NULL;
sqlite3_blob *BLOB = NULL;
rc = sqlite3_prepare_v2(db, sql.c_str(), -1, &stm, NULL);
rc = sqlite3_bind_blob(stm, 1, wpath.c_str(), wpath.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_bind_text(stm, 2, hash.c_str(), hash.size(), SQLITE_STATIC);
rc = sqlite3_bind_zeroblob(stm, 3, size);
rc = sqlite3_bind_int(stm, 4, versione);
rc = sqlite3_bind_blob(stm, 5, last.c_str(), last.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_step(stm);
if (rc != SQLITE_DONE) {
fprintf(stderr, " This file was already present in the database!\n", rc);
return;
}
else {
fprintf(stdout, "Record FILE created successfully\n");
}

It is an issue reported HERE.
And the oficial answer is:
In order for zeroblobs to work as above (using a fixed amount of
memory no matter how big they are) all zeroblobs must be at the end
of the row. In other words, the columns of the table that are
receiving the zeroblobs must be the last columns in the table. If
any non-zero content follows the zeroblob, then the zeroblob is
expanded into a literal sequence of zero bytes, meaning memory must
be allocated for the entire zeroblob.
So you need to change the order to fix it:
sqlite3_stmt* stm = NULL;
sqlite3_blob *BLOB = NULL;
rc = sqlite3_prepare_v2(db, sql.c_str(), -1, &stm, NULL);
rc = sqlite3_bind_blob(stm, 1, wpath.c_str(), wpath.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_bind_text(stm, 2, hash.c_str(), hash.size(), SQLITE_STATIC);
rc = sqlite3_bind_int(stm, 3, versione);
rc = sqlite3_bind_blob(stm, 4, last.c_str(), last.size()*sizeof(wchar_t), SQLITE_STATIC);
rc = sqlite3_bind_zeroblob(stm, 5, size);
rc = sqlite3_step(stm);
if (rc != SQLITE_DONE) {
fprintf(stderr, " This file was already present in the database!\n", rc);
return;
}
else {
fprintf(stdout, "Record FILE created successfully\n");
}

Related

Error while Bind BLOB values in ODBC(PostgreSQL), C/C++

I have some problems about ODBC programming with C/C++, PostgreSQL connector.
I just want to put some BLOB data in my PostgreSQL with C/C++, ODBC
here's my DB Table Create Query
CREATE TABLE BLOB_TEST
(
id number(20),
data BYTEA
);
And, here's my code
.
.
.
int retcode = 0;
SQLCHAR sql[1024] =
"BEGIN \n"
"insert into BLOB_TEST "
"values(9, ?); \n"
"EXCEPTION \n"
"when DUP_VAL_ON_INDEX then \n"
"dbms_output.put_line(1); \n"
"END; ";
char * temp_str = "this is a BLOB input TEST";
retcode = SQLPrepareA(hstmt, sql, SQL_NTS);
.
.
.
SQLBindParameter(hstmt,
1, /* Parameter number, starting at 1 */
SQL_PARAM_INPUT, /* in, out, inout */
SQL_C_BINARY, /* C data type of the parameter */
SQL_LONGVARBINARY, /* SQL data type of the parameter : char(8)*/
0, /* size of the column or expression, precision */
0, /* The decimal digits, scale */
temp_str, /* A pointer to a buffer for the parameter’s data */
0, /* Length of the ParameterValuePtr buffer in bytes */
NULL /* indicator */
);
.
.
.
retcode = SQLExecute(hstmt);
if (retcode == SQL_SUCCESS)
{
printf("Query Execute Success\n");
}
else
{
SQLGetDiagRecA(SQL_HANDLE_STMT, hstmt, ++rec, state, &native, message, sizeof(message), &length);
printf("%s : %ld : %ld : %s\n", state, rec, native, message);
printf("Query Execute ERROR : %d\n", retcode);
}
SQLExecute return -1(SQL_ERROR) and ERROR Message says:
SQLSTATE 42601, Missing ";" at the end of Expression
I know that PostgreSQL BLOB(BYTEA) type matched SQL_LONGVARBINARY Option when using SQLBindParameter, But that makes ERROR...
Is there any odd expression in my prepared query?
Or, Is there any way to check value-combind Query that SQLExcute function made?
I'm very confused, Cause the query that I prepared works well when using PgAdmin Querying tools...
So, I Want to check value-combind Query that SQLExcute function made.
You are trying to run a PL/SQL block on a database that is not Oracle. How is that supposed to work?
PostgreSQL has a DO statement that serves a similar purpose, but you cannot use parameters with it. You should send only the INSERT statement and do the exception handling in your C client code.

Getting segmentation fault when using pointers in C

I'm currently working on a project where I want to use SQLite to store some data. Everything is working well except when I want to insert new data into the table. When I run the application, I get segmentation fault, but I can't find the problem.
void sqlite(char *id, char *sensorname, char *sensorvalue){
sqlite3 *db;
char *zErrMsg = 0;
int rc;
char *sql;
const char* data = "Callback function called";
/* Open database */
rc = sqlite3_open("/home/macho/Documents/sensor_database.db", &db);
if( rc ){
fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
exit(0);
}else{
fprintf(stderr, "Opened database successfully\n");
}
sql = "INSERT INTO sensors (id,sensorname,sensorvalue) VALUES(";
char* split = ",";
strcat(sql, id);
strcat(sql, ",");
strcat(sql, sensorname);
strcat(sql, ",");
strcat(sql, sensorvalue);
strcat(sql, ");");
rc = sqlite3_exec(db, sql, callback, (void*)data, &zErrMsg);
if( rc != SQLITE_OK ){
fprintf(stderr, "SQL error: %s\n", zErrMsg);
sqlite3_free(zErrMsg);
}else{
fprintf(stdout, "Operation done successfully\n");
}
sqlite3_close(db);
}
And in the main, I'm calling the sqlite() function:
sqlite("1","sensor","sensor1");
Any idea what the problem can be?
Thanks!
You assign sql a static (read-only) string and then attempt to append to it. Instead, create a large writeable array either on the stack or use malloc and then assemble your query in that. So
char sql[4096];
strcpy(sql, "INSERT INTO sensors ...
...
Note that you should check for overflow of the buffer based on the lengths of the values.
BTW, the code as written is just asking for an SQL injection attack if accessible to users. Look up Bobby Tables.

sqlite - delete and get all values of some column from affected rows [duplicate]

I heard of using sqlite3_prepare_v2 instead of sqlite_exec to get integers from database, but I failed to find any examples. This page wasn't helpful also. Now I am getting strings from database, so I need to parse them with atoi and this seems to be slow and ineffective.
There a lot of similar questions on SO, but they are about obj-c and iOS SDK. I need C/C++ hint or example.
Thanks in advance.
After sqlite3_prepare has succeeded, you must not forget to clean up the statement with sqlite3_finalize.
To get the result records, call sqlite3_step until it does not return SQLITE_ROW.
To get the values of the current result record, call the sqlite3_column_* functions:
sqlite3_stmt *stmt;
if (sqlite3_prepare_v2(db, "SELECT 42", -1, &stmt, NULL) != SQLITE_OK)
...error...
else {
for (;;) {
int rc = sqlite3_step(stmt);
if (rc == SQLITE_DONE)
break;
if (rc != SQLITE_ROW) {
...error...
break;
}
printf("value: %d\n", sqlite3_column_int(stmt, 0));
}
sqlite3_finalize(stmt);
}
sqlite3_column_int(result, columnNum); will return one column from the current row of your result as an int.
Your prepare function is to prepare your query, it has nothing to do with how the results are interpreted. All data in sqlite3 is stored textually, you use the appropriate function to retrieve a value in the type you believe it should be.

Getting int values from SQLite

I heard of using sqlite3_prepare_v2 instead of sqlite_exec to get integers from database, but I failed to find any examples. This page wasn't helpful also. Now I am getting strings from database, so I need to parse them with atoi and this seems to be slow and ineffective.
There a lot of similar questions on SO, but they are about obj-c and iOS SDK. I need C/C++ hint or example.
Thanks in advance.
After sqlite3_prepare has succeeded, you must not forget to clean up the statement with sqlite3_finalize.
To get the result records, call sqlite3_step until it does not return SQLITE_ROW.
To get the values of the current result record, call the sqlite3_column_* functions:
sqlite3_stmt *stmt;
if (sqlite3_prepare_v2(db, "SELECT 42", -1, &stmt, NULL) != SQLITE_OK)
...error...
else {
for (;;) {
int rc = sqlite3_step(stmt);
if (rc == SQLITE_DONE)
break;
if (rc != SQLITE_ROW) {
...error...
break;
}
printf("value: %d\n", sqlite3_column_int(stmt, 0));
}
sqlite3_finalize(stmt);
}
sqlite3_column_int(result, columnNum); will return one column from the current row of your result as an int.
Your prepare function is to prepare your query, it has nothing to do with how the results are interpreted. All data in sqlite3 is stored textually, you use the appropriate function to retrieve a value in the type you believe it should be.

when does an sqlite compiled statement get 'executed'

I'm writing a light wrapper for the SQLite API.
Basically, I'm curious about how/when an SQLite pre-compiled statement gets executed...
When i go:
char buffer[] = "INSERT INTO example VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)";
sqlite3_stmt* stmt;
sqlite3_prepare_v2(mDb, buffer, strlen(buffer), &stmt, NULL);
for (unsigned i = 0; i < mVal; i++)
{
std::string id = getID();
sqlite3_bind_text(stmt, 1, id.c_str(), id.size(), SQLITE_STATIC);
sqlite3_bind_double(stmt, 2, getDouble());
sqlite3_bind_double(stmt, 3, getDouble());
sqlite3_bind_double(stmt, 4, getDouble());
sqlite3_bind_int(stmt, 5, getInt());
sqlite3_bind_int(stmt, 6, getInt());
sqlite3_bind_int(stmt, 7, getInt());
if (sqlite3_step(stmt) != SQLITE_DONE)
{
printf("Commit Failed!\n");
}
sqlite3_reset(stmt);
}
sqlite3_finalize(stmt);
At what point is the actual sql executed? Is it during the call to sqlite3_prepare_v2, or during the first sqlite3_step?
Any clarity is greatly appreciated :)
Cheers
Jarrett
According to the SQLite documentation for sqlite3_prepare, we see:
To execute an SQL query, it must first be compiled into a byte-code program using
one of these routines:
sqlite3_prepare, sqlite3_prepare_v2, sqlite3_prepare16, sqlite3_prepare16_v2.
And for sqlite3_step:
After a prepared statement has been prepared using either sqlite3_prepare_v2() or
sqlite3_prepare16_v2() or one of the legacy interfaces sqlite3_prepare() or
sqlite3_prepare16(), this function must be called one or more times to evaluate
the statement.
More information:
SQLite has a virtual machine that does all necessary actions to execute your code on the selected database. sqlite3_prepare (and its family) compiles your SQL statement into byte code that can be executed on this virtual machine. On the other hand, sqlite3_step executes that byte-code in the VM.