I am attempting to make my saving code more efficient through the use of re-using the prepared statement, the use of transactions, and parametrized queries. However whenever I run the save code only one record ends up in my table. I think it has to do with where the sqlite3_step is placed.
But I also want some other eyes to look over my code and let me know if I am doing things correctly; I could not find an example doing this anywhere so it is a jumble of about six forums posts and Stack Overflow topics I found on the various subjects:
//Planet Data
//Delete previous save data
dData("Generated_Planets", bErrors);
if (plData.size() > 0)
{
sqlite3_exec(dBase, "BEGIN TRANSACTION", NULL, NULL, &error);
sqlStr = "Insert Into Generated_Planets (ID, Name, Affiliation, Disposition, Race, Player_Owned, Is_Destroyed, EKS, Planet_Size, Current_Pop, Max_Pop) Values (?,?,?,?,?,?,?,?,?,?,?);";
if (sqlite3_prepare_v2(dBase, sqlStr.c_str(), sqlStr.size(), &statement, 0) == SQLITE_OK)
{
cout << "Saving planet data";
//Save new data
for (i = 0; i <= plData.size(); i++)
{
if (i == plData.size())
{
finalize(statement, bErrors);
}
else
{
sqlI1 = plData.at(i).pID;
sqlS1 = plData.at(i).pName;
sqlS2 = plData.at(i).pAffiliation;
sqlS3 = plData.at(i).pDispo;
sqlS4 = plData.at(i).pRace;
sqlI2 = plData.at(i).bIsPOwned;
sqlI3 = plData.at(i).bIsDestroyed;
sqlF1 = plData.at(i).pEKS;
sqlF2 = plData.at(i).pSize;
sqlLLI1 = plData.at(i).pCPop;
sqlLLI2 = plData.at(i).pMPop;
find = "'";
temp = "\"";
foundAt = sqlS1.find(find);
if (foundAt != string::npos)
{
sqlS1.replace(foundAt,1,temp);
}
//Bind parameters
sqlite3_bind_int(statement,1,sqlI1);
sqlite3_bind_text(statement,2,sqlS1.c_str(),sqlS1.size(),SQLITE_TRANSIENT);
sqlite3_bind_text(statement,3,sqlS2.c_str(),sqlS2.size(),SQLITE_TRANSIENT);
sqlite3_bind_text(statement,4,sqlS3.c_str(),sqlS3.size(),SQLITE_TRANSIENT);
sqlite3_bind_text(statement,5,sqlS4.c_str(),sqlS4.size(),SQLITE_TRANSIENT);
sqlite3_bind_int(statement,6,sqlI2);
sqlite3_bind_int(statement,7,sqlI3);
sqlite3_bind_double(statement,8,sqlF1);
sqlite3_bind_double(statement,9,sqlF2);
sqlite3_bind_int64(statement,10,sqlLLI1);
sqlite3_bind_int64(statement,11,sqlLLI2);
sqlite3_step(statement);
cout << ".";
}
}
}
else
{
*bErrors = true;
createBInfo();
d.createBReport("SQL Code 2",sqlite3_errmsg(dBase),bLocale + to_string(__LINE__),bTDate,"./SC_Log.txt");
}
sqlite3_exec(dBase, "END TRANSACTION", NULL, NULL, &error);
sFlags_Temp.push_back(saveFlag());
sFlags_Temp.at(sFlags_Temp.size()-1).sfName = "GPlanets";
sFlags_Temp.at(sFlags_Temp.size()-1).sfValue = 1;
cout << "Done" << endl << endl;
}
It looks like you're missing a call to sqlite3_reset after sqlite3_step and before binding the next set of parameters. And you're ignoring the return codes that would have told you about it.
See the documentation for sqlite3_step here: http://sqlite.org/c3ref/step.html
Related
Why am I get this crazy high value in the field member of LPWFSIDCSTATUS? Does this mean that the ATM does not have an anti-fraud module? Am I missing anything?
Field values such as fwDevice or fwMedia produce the correct values, so I'm not sure why wAntiFraudModule or wDevicePosition is behaving this way.
Using visual studio debugger
Here's a code segment of what I've got going on.
LPWFSRESULT lpResult = NULL;
LPWFSPTRSTATUS lpStatus = NULL;
HRESULT result = WFMAllocateBuffer(sizeof(WFSPTRSTATUS), WFS_MEM_ZEROINIT | WFS_MEM_SHARE, (void**)&lpResult);
if (result != WFS_SUCCESS)
std::cout << "FAILED TO ALLOCATE\n";
dev_status = WFSGetInfo(XFS::GetServiceInstance(), WFS_INF_PTR_STATUS, NULL, 40000, &lpResult);
if (dev_status != WFS_SUCCESS)
std::cout << "Failed to get status for printer\n";
lpStatus = (LPWFSPTRSTATUS)lpResult->lpBuffer;
WORD my_fraud = lpStatus->wAntiFraudModule;
std::cout << "Anti-Fraud status: " << my_fraud << std::endl;
WFSFreeResult(lpResult);
I'm working on a wrapper for MariaDB Connector C. There is a typical situation when a developer doesn't know a length of a data stored in a field. As I figured out, one of the ways to obtain a real length of the field is to pass a buffer of lengths to mysql_stmt_bind_result and then to fetch each column by calling mysql_stmt_fetch_column. But I can't understand how the function mysql_stmt_fetch_column works because I'm getting a memory corruption and app abortion.
Here is how I'm trying to reach my goal
// preparations here
...
if (!mysql_stmt_execute(stmt))
{
int columnNum = mysql_stmt_field_count(stmt);
if (columnNum > 0)
{
MYSQL_RES* metadata = mysql_stmt_result_metadata(stmt);
MYSQL_FIELD* fields = mysql_fetch_fields(metadata);
MYSQL_BIND* result = new MYSQL_BIND[columnNum];
std::memset(result, 0, sizeof (MYSQL_BIND) * columnNum);
std::vector<unsigned long> lengths;
lengths.resize(columnNum);
for (int i = 0; i < columnNum; ++i)
result[i].length = &lengths[i];
if (!mysql_stmt_bind_result(stmt, result))
{
while (true)
{
int status = mysql_stmt_fetch(stmt);
if (status == 1)
{
m_lastError = mysql_stmt_error(stmt);
isOK = false;
break;
}
else if (status == MYSQL_NO_DATA)
{
isOK = true;
break;
}
for (int i = 0; i < columnNum; ++i)
{
my_bool isNull = true;
if (lengths.at(i) > 0)
{
result[i].buffer_type = fields[i].type;
result[i].is_null = &isNull;
result[i].buffer = malloc(lengths.at(i));
result[i].buffer_length = lengths.at(i);
mysql_stmt_fetch_column(stmt, result, i, 0);
if (!isNull)
{
// here I'm trying to read a result and I'm getting a valid result only from the first column
}
}
}
}
}
}
If I put an array to the mysql_stmt_fetch_column then I'm fetching the only first field valid, all other fields are garbage. If I put a single MYSQL_BIND structure to this function, then I'm getting an abortion of the app on approximately 74th field (funny thing that it's always this field). If I use another array of MYSQL_BIND then the situation is the same as the first case.
Please help me to understand how to use it correctly! Thanks
Minimal reproducible example
I'm trying to store and fetch some data from LMDB. Data seems to be stored, I can see the keys in my database, but it gives me MDB_NOTFOUND when I try to fetch the value with the same ID I have just stored it under.
Database opening
MDB_env* environment;
MDB_dbi main;
MDB_dbi order;
mdb_env_create(&environment);
mdb_env_set_maxdbs(environment, 2);
mdb_env_open(environment, path.toStdString().c_str(), 0, 0664);
int rc;
MDB_txn *txn;
mdb_txn_begin(environment, NULL, 0, &txn);
mdb_dbi_open(txn, "main", MDB_CREATE, &main);
mdb_dbi_open(txn, "order", MDB_CREATE | MDB_INTEGERKEY, &order);
mdb_txn_commit(txn);
Insertion
void Core::Archive::addElement(const Shared::Message& message) {
QByteArray ba;
QDataStream ds(&ba, QIODevice::WriteOnly);
message.serialize(ds);
uint64_t stamp = message.getTime().toMSecsSinceEpoch();
const std::string& id = message.getId().toStdString();
MDB_val lmdbKey, lmdbData;
lmdbKey.mv_size = id.size();
lmdbKey.mv_data = (uint8_t*)id.c_str();
lmdbData.mv_size = ba.size();
lmdbData.mv_data = (uint8_t*)ba.data();
MDB_txn *txn;
mdb_txn_begin(environment, NULL, 0, &txn);
int rc;
rc = mdb_put(txn, main, &lmdbKey, &lmdbData, 0);
if (rc == 0) {
MDB_val orderKey;
orderKey.mv_size = 8;
orderKey.mv_data = (uint8_t*) &stamp;
rc = mdb_put(txn, order, &orderKey, &lmdbKey, 0);
if (rc) {
mdb_txn_abort(txn);
} else {
rc = mdb_txn_commit(txn);
if (rc) {
qDebug() << "A transaction error: " << mdb_strerror(rc);
}
}
} else {
qDebug() << "An element couldn't been added to the archive, skipping" << mdb_strerror(rc);
mdb_txn_abort(txn);
}
}
Fetching
Shared::Message Core::Archive::getElement(const QString& id) {
MDB_val lmdbKey, lmdbData;
lmdbKey.mv_size = id.toStdString().size();
lmdbKey.mv_data = (uint8_t*)id.toStdString().c_str();
MDB_txn *txn;
int rc;
mdb_txn_begin(environment, NULL, MDB_RDONLY, &txn);
rc = mdb_get(txn, main, &lmdbKey, &lmdbData);
if (rc) {
qDebug() <<"Get error: " << mdb_strerror(rc);
mdb_txn_abort(txn);
throw NotFound(id.toStdString(), jid.toStdString());
} else {
//it never comes here
}
}
Testing code
Core::Archive ar();
ar.open("Test");
Shared::Message msg1;
msg1.generateRandomId();
msg1.setBody("oldest");
msg1.setTime(QDateTime::currentDateTime().addDays(-7));
Shared::Message msg2;
msg2.generateRandomId();
msg2.setBody("Middle");
msg2.setTime(QDateTime::currentDateTime().addDays(-4));
Shared::Message msg3;
msg3.generateRandomId();
msg3.setBody("newest");
msg3.setTime(QDateTime::currentDateTime());
ar.addElement(msg2);
ar.addElement(msg3);
ar.addElement(msg1);
Shared::Message d0 = ar.getElement(msg1.getId());
My logs show stored keys. I can see the required key, I can even compare it with the requested key if I use cursors to scroll over the whole storage, it even shows they are equal, but mdb_cursor_get or mdb_get constantly give me MDB_NOTFOUND. What am I doing wrong?
I got it. No matter what I put into database, I have to read it as a char*
Had to modify fetching code
lmdbKey.mv_data = (uint8_t*)id.toStdString().c_str();
I had to change it to
lmdbKey.mv_data = (char*)id.toStdString().c_str();
and it worked
I have the following code here that executes a query. Originally, I used SQL Injection to return row results. Hearing I should use parametrization, I rearranged my code and read the MySQL docs on how to do so. I'm using MySQL's C library in a C++ application.
However, it's not returning the results.
I know my SQL statement is 100% fine. It has been tested. The only thing I changed was changing %d (injection) to ?, which accepts the player's ID.
This returns -1. It's a SELECT statement though, so maybe it's normal?
// Get the number of affected rows
affected_rows = mysql_stmt_affected_rows(m_stmt);
This returns 2. This is correct. I have two fields being returned.
// Store the field count
m_fieldCount = mysql_field_count(&m_conn);
This returns 0 (success)
if (mysql_stmt_store_result(m_stmt))
Finally, this returns null.
m_result = mysql_store_result(&m_conn);
I need m_result so I can read the rows. "mysql_stmt_store_result" sounds similar, but doesn't return MYSQL_RESULT.
m_result = mysql_store_result(&m_conn);
/// <summary>
/// Executes a query.
/// </summary>
/// <param name="query">The query to execute.</param>
/// <returns>Returns true on success, else false.</returns>
bool SQLConnection::executeQuery_New(const char *query)
{
int param_count = 0;
int affected_rows = 0;
// Validate connection.
if (!m_connected)
return false;
// Initialize the statement
m_stmt = mysql_stmt_init(&m_conn);
if (!m_stmt) {
fprintf(stderr, " mysql_stmt_init(), out of memory\n");
return false;
}
// Prepare the statement
if (mysql_stmt_prepare(m_stmt, query, strlen(query))) {
fprintf(stderr, " mysql_stmt_prepare(), INSERT failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Get the parameter count from the statement
param_count = mysql_stmt_param_count(m_stmt);
if (param_count != m_bind.size()) {
fprintf(stderr, " invalid parameter count returned by MySQL\n");
return false;
}
// Bind buffers
// The parameter binds are stored in std::vector<MYSQL_BIND>
// I need to convert std::vector<MYSQL_BIND> m_bind to MYSQL_BIND *bnd
MYSQL_BIND *bind = new MYSQL_BIND[m_bind.size() + 1];
memset(bind, 0, sizeof(bind) * m_bind.size());
for (int i = 0; i < param_count; i++)
bind[i] = m_bind[i];
if (mysql_stmt_bind_param(m_stmt, &bind[0]))
{
fprintf(stderr, " mysql_stmt_bind_param() failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Execute the query
if (mysql_stmt_execute(m_stmt)) {
fprintf(stderr, " mysql_stmt_execute(), 1 failed\n");
fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
return false;
}
// Get the number of affected rows
affected_rows = mysql_stmt_affected_rows(m_stmt);
//if (affected_rows == -1) {
// fprintf(stderr, " mysql_stmt_execute(), 1 failed\n");
// fprintf(stderr, " %s\n", mysql_stmt_error(m_stmt));
// return false;
//}
// Store the field count
m_fieldCount = mysql_field_count(&m_conn);
// Store the result
if (mysql_stmt_store_result(m_stmt))
{
fprintf(stderr, " failed retrieving result\n");
fprintf(stderr, " %s\n", mysql_error(&m_conn));
int d = mysql_errno(&m_conn);
return false;
}
// This looks similar to the last above statement, but I need m_result. I used mysql_store_result earlier when using injection and it worked fine, but here in this case it returns null.
m_result = mysql_store_result(&m_conn);
// Close the statement
if (mysql_stmt_close(m_stmt)) {
/* mysql_stmt_close() invalidates stmt, so call */
/* mysql_error(mysql) rather than mysql_stmt_error(stmt) */
fprintf(stderr, " failed while closing the statement\n");
fprintf(stderr, " %s\n", mysql_error(&m_conn));
return false;
}
// Delete bind array
if (bind) {
delete[] bind;
bind = NULL;
}
return true;
}
How I'm adding an int parameter (player's id):
void SQLConnection::addParam(int buffer, enum_field_types type, unsigned long length)
{
MYSQL_BIND bind;
memset(&bind, 0, sizeof(bind));
bind.buffer = (char *)&buffer;
bind.buffer_type = type;
bind.is_null = 0;
bind.length = &length;
m_bind.push_back(bind);
}
My variables and their types:
class SQLConnection
{
private:
MYSQL m_conn;
MYSQL_ROW m_row;
MYSQL_RES *m_result;
char m_errorMessage[ERROR_MSG_MAX];
bool m_connected;
MYSQL_STMT *m_stmt;
std::vector<MYSQL_BIND> m_bind;
int m_fieldCount;
// ...
And finally its calling function at the end of the SQL statement:
...WHERE player_id = ?;");
conn.addParam(m_id, MYSQL_TYPE_LONGLONG, 0);
if (!conn.executeQuery_New(buffer)) {
conn.close();
return "";
}
// Close the connection.
conn.close();
std::string s = conn.getField("max_value_column_name");
The error code I get is 2014:
https://dev.mysql.com/doc/refman/5.7/en/commands-out-of-sync.html
Just for the sake of interest, this is a prior function I used. This worked fine for injection. Using the new function above with parameterization is the one causing the issues.
bool SQLConnection::executeQuery(const char *query)
{
// Validate connection.
if (!m_connected)
return false;
// Execute the query
int status = mysql_query(&m_conn, query);
if (status != 0) {
sprintf(m_errorMessage, "Error: %s", mysql_error(&m_conn));
return false;
}
// Store the result
m_result = mysql_store_result(&m_conn);
return true;
}
After I started having language religious wars in my head about using C# over C++, I thought I'd give one last attempt here. Any help is appreciated.
Edit:
This is how I read in column names prior to parameterization (maybe this code needs to be updated after calling mysql_stmt_store_result(m_stmt)?
std::string SQLConnection::getField(const char *fieldName)
{
MYSQL_FIELD *field = NULL;
unsigned int name_field = 0;
mysql_stmt_data_seek(m_stmt, 0);
mysql_stmt_fetch_column(m_stmt, &bind, 0, 0);
//mysql_data_seek(m_result, 0);
//mysql_field_seek(m_result, 0);
const unsigned int num_fields = mysql_stmt_field_count(m_stmt);
// const unsigned int num_fields = mysql_num_fields(m_result);
std::vector<char *> headers(num_fields);
for (unsigned int i = 0; (field = mysql_fetch_field(m_result)); i++)
{
headers[i] = field->name;
if (strcmp(fieldName, headers[i]) == 0)
name_field = i;
}
while ((m_row = mysql_fetch_row(m_result))) {
return std::string(m_row[name_field]);
}
return "";
}
Edit:
What I'm finding is in this last function there are equivalent functions for statements, like mysql_num_fields() is mysql_stmt_field_count(). I'm thinking these need to be updated because it's using m_stmt and not m_result anymore, which gives reason to update the functions so m_stmt is used. It's not very apparent how to update the second half of the function though.
You may need a better understanding of how stmt works.You can't get the final results by mysql_store_result() when you're using stmt.
You shoud bind several buffers for the statement you're using to accept the result set. You can finish this by mysql_stmt_bind_result(), just like mysql_stmt_bind_param().
Then you can use the buffers bound by mysql_stmt_bind_result() to return row data. You can finish this by mysql_stmt_fetch().
Do the fetch method repeatedly, so you can get the whole result set row by row.
The basic sequence of calls:
mysql_stmt_init
mysql_stmt_prepare
mysql_stmt_bind_param
mysql_stmt_execute
mysql_stmt_bind_result
mysql_stmt_store_result
mysql_stmt_fetch (repeatedly, row by row)
mysql_stmt_free_result
It works for me.
It's a long time since I finished this part of my project, you'd better read the manual carefully and find more examples of stmt.
Sorry for my poor English. Good luck!
I'm trying to reduce duplicate sqlite code in my object. To do so I wanted one method that had the sqlite interaction and just return a sqlite3_stmt. My problem is that the sqlite3_stmt pointer is always null. My question is, am I doing something wrong or can this not be done with sqlite?
Usage:
SqliteIO sqldb;
string GET_CONFIG = "select* from config; ";
sqlite3_stmt *statement;
assert(sqldb.sqlForResults(GET_CONFIG, statement));
assert(statement != NULL); //this fails
Method:
bool SqliteIO::sqlForResults(string pSql, sqlite3_stmt *statement ){
bool lbRetVal=false;
if(mDB == NULL){
cerr << "No database to query";
Logger::error("SqlliteIO::getAllScheuled() null database handle");
} else {
sqlite3_busy_timeout(mDB, 2000);
if(sqlite3_prepare_v2(mDB, pSql.c_str(), -1, &statement, 0) == SQLITE_OK) {
lbRetVal = true;
}
string error = sqlite3_errmsg(mDB);
if(error != "not an error"){
cerr << pSql << " " << error << endl;
Logger::error("SqlliteIO::sqlForResults() "+ error + " " +pSql );
}
}
return lbRetVal;
sqlite3_stmt *statement; // this never gets initialized
assert(sqldb.sqlForResults(GET_CONFIG, statement));
assert(statement != NULL); //this fails
the statement variable in that block of code never gets initialized (you're passing a copy of it to sqlForResults()).
frankly, you're lucky that assertion is failing, as statement could have any garbage value (I'm guessing variables are zero-initialized in debug mode for your compiler)
The statement parameter of SqliteIO::sqlForResults() needs to have the type sqlite3_stmt**, not sqlite3_stmt*. So the call site would now look like:
SqliteIO sqldb;
string GET_CONFIG = "select* from config; ";
sqlite3_stmt *statement = NULL;
assert(sqldb.sqlForResults(GET_CONFIG, &statement));
Without the reference/address-of operator & before statement (like in the original question), you would be passing the value of statement to sqlForResults, which was NULL or 0 in this case. The relevant changes in sqlForResults are:
bool SqliteIO::sqlForResults(string pSql, sqlite3_stmt **statement ){
bool lbRetVal=false;
if(mDB == NULL){
// ...
} else {
// ...
if(sqlite3_prepare_v2(mDB, pSql.c_str(), -1, statement, 0) == SQLITE_OK) {
// ...
}
// ...
}
return lbRetVal;
The parameter list now accepts a sqlite_stmt** arg, and we pass that directly to sqlite3_prepare_v2. In the original question, sqlForResults passed &statement to sqlite3_prepare_v2 - which is the address of the statement parameter for this SqliteIO::sqlForResults method call - which isn't what you want.
I know this is an old question, but I hope this helps.
EDIT: for clarity and to elude some more code from sqlForResults