So I've written some C++ code to select data from one database and INSERT it into another. There's over 50,000 entries in the source db. The table definitions for the destination database are as follows:
CREATE TABLE evidence_master
( id INTEGER NOT NULL,
person_id TEXT NOT NULL,
message TEXT,
eid INTEGER NOT NULL,
frequency INTEGER,
string TEXT,
CONSTRAINT id_pk PRIMARY KEY (id),
CONSTRAINT eid_fk FOREIGN KEY (eid) REFERENCES evidences(eid)
);
CREATE TABLE evidences
(
eid INTEGER NOT NULL,
evidence_name TEXT,
CONSTRAINT eid_pk PRIMARY KEY (eid)
);
The source database is the one found here: http://wing.comp.nus.edu.sg/SMSCorpus/
My problem is, when it gets to the 252nd entry, it claims it cannot open the database (EDIT: the destination database). Despite having performed the work on 251 entries successfully.
Why might this be happening?
EDIT: here's the code that selects from the source:
dbOpenFail=sqlite3_open("../../../db/smscorpus.db", &dbCorpus);
if(dbOpenFail)
{
cerr << "Can't open sms corpus: " << sqlite3_errmsg(dbCorpus);
}
if(sqlite3_prepare(dbCorpus,
queryCorpus.data(),
queryCorpus.length() + 1,
&preparedCorpusQuery, NULL) != SQLITE_OK)
{
cerr << "Failed to prepare corpus query: " << queryCorpus << zErrMsg << endl;
sqlite3_free(zErrMsg);
}
rowSelectCorpus=sqlite3_step(preparedCorpusQuery);
while(rowSelectCorpus != SQLITE_DONE) //need to go through 51652 messages from corpus
{
if(rowSelectCorpus == SQLITE_ROW)
{
personID = sqlite3_column_text(preparedCorpusQuery, 0); //get sender ID
messageContent = sqlite3_column_text(preparedCorpusQuery, 1); //get message itself
db_entry d1(string(reinterpret_cast<const char *>(personID)),
string(reinterpret_cast<const char *>(messageContent)));
insertResult = d1.insertIntoDatabase();
if(insertResult == -1)
{
return 0; //exit
}
}
rowSelectCorpus=sqlite3_step(preparedCorpusQuery);
}
Some C runtime libraries have a default limit of 256 open file handles per process.
Apparently, your program is opening the database for every INSERT operation, and forgets to close it.
You should open every database only once.
Related
So I'm trying to make a login function that uses SQLite3, and what I want is the user to input his username and password and the program is going to search in the table for his username than his password and compare it, and if it matches he get access if don't he needs to input his credentials again. And this is how my function look right now:
void logIn(){
std::cout << "Username: "; std::cin >> user;
std::cout << "Password: "; std::cin >> psw;
//Open database
rc = sqlite3_open("Users.db", &db);
//create sql statement
sql = "SELECT Username FROM Users_info"
"WHERE Username='?';";
//Execute sql statement
sqlite3_prepare(db, sql, -1, &st, NULL);
sqlite3_bind_text(st, 1, user.c_str(), name.length(), SQLITE_TRANSIENT);
int rc = sqlite3_exec(db, sql, callback, (void*)data.c_str(), NULL);
//show SQL output
if (rc != SQLITE_OK) {
std::cout << "Error making Log In" ;
sqlite3_free(zErrMsg);
}
else
std::cout << "Log In succesfully" ;
sqlite3_close(db);
And as you can see in these lines I'm trying to use the WHERE clause
//create sql statement
sql = "SELECT Username FROM Users_info"
"WHERE Username='?';";
Because for example, in my table I have this "test" user and if I use this exact code in SQLite3 It's going to return the exact user that I want, and with that in mind, what I want is to somehow my program to compare the user input to the return of the sql code but I don't have any idea to do so. Hope this makes my doubt clear enough to be clarified.
I use sqlite on a c++ project, but I have a problem when i use WHERE on a column with TEXT values
I created a sqlite database:
CREATE TABLE User( id INTEGER PRIMARY KEY AUTOINCREMENT, name VARCHAR(24))
When i try to get the value of the column with VARCHAR values, it doesn't work, and return me a STATUS_CODE 101 just after the sqlite3_step :
int res = 0;
sqlite3_stmt *request;
char *sqlSelection = (char *)"SELECT * FROM User WHERE name='bob' ";
int id = 0;
res = sqlite3_prepare_v2(db, sqlSelection, strlen(sqlSelection), &request, NULL);
if (!res){
while (res == SQLITE_OK || res == SQLITE_ROW){
res = sqlite3_step(request);
if (res == SQLITE_OK || res == SQLITE_ROW ){
id = sqlite3_column_int(request, 0);
printf("User exist %i \n",id);
}
}
sqlite3_finalize(request);
I also tried with LIKE but it also doesn't work
SELECT * FROM User WHERE name LIKE '%bob%'
But when I execute the same code but for an INTERGER value
SELECT * FROM User WHERE id=1
It work fine.
In DB Browser for SQLite all requests work fine.
To solve the problem I searched what status code 101 means.
Here is what they said.
(101) SQLITE_DONE
The SQLITE_DONE result code indicates that an operation has completed.
The SQLITE_DONE result code is most commonly seen as a return value
from sqlite3_step() indicating that the SQL statement has run to
completion. But SQLITE_DONE can also be returned by other multi-step
interfaces such as sqlite3_backup_step().
https://sqlite.org/rescode.html
So, you're getting 101 because there is no more result from SELECT SQL.
The solution was to replace the VARCHAR fields by TEXT.
SQLite for c++ seems to don't manage VARCHAR fields when they are used after the WHERE
After I create a table, insert data into it, and then delete a non-existent row, the operation succeeds even though the row does not exist. When I delete a row that actually exists, it succeeds as well and the row is actually deleted. Why doesn't an error occur when I try to delete a row that does not exist?
I am using sqlite3 with c++ on eclipse.
I've been working with some code found on the web, as well as my own.
Other operations, like SELECT and INSERT work fine. DELETE works when rows exist and even when they don't exist.
// Creating a table
sql = "CREATE TABLE COMPANY(" \
"ID INT PRIMARY KEY NOT NULL," \
"NAME TEXT NOT NULL," \
"AGE INT NOT NULL," \
"ADDRESS CHAR(50)," \
"SALARY REAL );";
// Inserting data
sql = "INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) " \
"VALUES (1, 'Paul', 32, 'California', 20000.00 ); " \
"INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) " \
"VALUES (2, 'Allen', 25, 'Texas', 15000.00 ); " \
"INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY)" \
"VALUES (3, 'Teddy', 23, 'Norway', 20000.00 );" \
"INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY)" \
"VALUES (4, 'Mark', 25, 'Rich-Mond ', 65000.00 );";
// Deleting (this is where the deletion should fail because there is no
ID 30)
rc = sqlite3_open("test.db", &db);
if( rc ) {
cout << "ERROR ----> " << zErrMsg << endl;
return(0);
} else {
fprintf(stderr, "Opened database successfully\n");
}
/* Create SQL statement */
sql = "DELETE FROM COMPANY WHERE ID = 30";
/* Execute SQL statement */
rc = sqlite3_exec(db, sql, callback, (void*)data, &zErrMsg);
if( rc != SQLITE_OK ) {
cout << "ERROR DELETING" << endl;
fprintf(stderr, "SQL error: %s\n", zErrMsg);
sqlite3_free(zErrMsg);
} else {
fprintf(stdout, "Deletion operation done successfully\n");
}
sqlite3_close(db);
I expect the message "ERROR DELETING" to be displayed, but "Deletion operation done successfully" always displays, even when the ID deleted does not exist.
Why doesn't an error occur when I try to delete a row that does not exist?
Databases return number of rows affected (deleted in your case), they don't throw error, unless there is something wrong with the query.
The only guaranteed method to check if the row has been deleted is to execute a SELECT statement with the same conditions. Transaction control is an important factor here, i.e. what happens if something causes a rollback? Are you explicitly committing transactions, or allowing them to auto-commit? You shouldn't just rely on return codes for this. If you do a COUNT as well, then you will know for sure, at least within your session:
SELECT COUNT(*) FROM COMPANY WHERE ID = 30
If after the DELETE, the COUNT is 0, the row no longer exists.
Note: This approach will be helpful if you need concurrency, which will force you to use a different database engine (e.g. PostgreSQL).
I guess the answer is "this is by design".
If you need to know if zero (or more) rows were affected by the last DELETE statement (or INSERT or UPDATE for that matter) then you could look into using:
int sqlite3_changes(sqlite3*);
That will allow you to react accordingly if no rows were deleted.
See https://www.sqlite.org/c3ref/changes.html
I have been writing a wrapper for the Cassandra cppdriver for CQL3.0 and I have come across some odd behavior, and I am not sure if it is typical or a bug.
For reference, I am working with with the cppdriver code release on 4 September (from the repository), libuv0.10 and off of the songs / playlist example posted on the datastax website (http://www.datastax.com/documentation/cql/3.1/cql/ddl/ddl_music_service_c.html)
The problem that I am having is with executing query strings. There seems to be some threshold of characters after which the query string being sent to Cassandra becomes garbage. The code that I am using to construct and send the string to the cppdriver library (and parse the results) is provided below. I added a function (cass_session_print_query) to the cassandra.h and session.cpp files to print out generated statement.
map<string, vector<string> > retresults;
int i = 0, ccount;
stringstream ss;
vector<string> keys = get.GetList();
vector<string>::iterator kit = keys.begin();
map<int, pair<string, string> > primkeys = get.GetMap();
map<int, pair<string, string> >::iterator mit = primkeys.begin();
if (!keys.empty())
{
ss << "SELECT " << (*kit);
++kit;
for ( ; kit != keys.end(); ++kit)
ss << "," << (*kit);
ss << " FROM " << tablename;
if (!primkeys.empty())
{
ss << " WHERE ";
ss << mit->second.first << " = ?";
++mit;
for ( ; mit != primkeys.end(); ++mit)
ss << " and " << mit->second.first << " = ?";
mit = primkeys.begin();
}
ss << ";";
cass_bool_t has_more_pages = cass_false;
const CassResult* result = NULL;
CassString query = cass_string_init(ss.str().c_str());
CassStatement* statement = cass_statement_new(query, primkeys.size());
for ( ; mit != primkeys.end(); ++mit)
cass_statement_bind_string(statement, i++, cass_string_init(mit->second.second.c_str()));
cass_statement_set_paging_size(statement, 100);
do
{
cass_session_print_query(statement);
CassIterator* iterator;
CassFuture* future = cass_session_execute(session_, statement);
if (cass_future_error_code(future) != 0)
{
CassString message = cass_future_error_message(future);
fprintf(stderr, "Error: %.*s\n", (int)message.length, message.data);
break;
}
result = cass_future_get_result(future);
ccount = cass_result_column_count(result);
vector<string> cnames;
for (i = 0; i < ccount; i++)
cnames.push_back(cass_result_column_name(result, i).data);
iterator = cass_iterator_from_result(result);
ListVector::iterator vit;
while (cass_iterator_next(iterator))
{
const CassRow* row = cass_iterator_get_row(iterator);
for (vit = cnames.begin(); vit != cnames.end(); ++vit)
{
CassString value;
char value_buffer[256];
cass_value_get_string(cass_row_get_column_by_name(row, (*vit).c_str()), &value);
if (value.length == 0 || value.data == NULL)
continue;
memcpy(value_buffer, value.data, value.length);
value_buffer[value.length] = '\0';
retresults[(*vit)].push_back(value_buffer);
}
}
has_more_pages = cass_result_has_more_pages(result);
if (has_more_pages)
cass_statement_set_paging_state(statement, result);
cass_iterator_free(iterator);
cass_result_free(result);
} while (has_more_pages);
}
return retresults;
With this, an initial query string of SELECT id,album,title,artist,data FROM songs; results in a Cassandra query string of SELECT id,album,title,artist,data FROM songs;. However, if I add one more column to the SELECT portion SELECT id,album,title,artist,data,tags FROM songs; the query string in the Cassandra cppdriver library becomes something like: ,ar����,dat�� jOM songX. This results in the following error from Cassandra / library: Error: line 1:49 no viable alternative at character '�'.
I have also tried fewer columns, but with a WHERE clause, and that results in the same problem.
Is this a bug? Or am I building and sending strings to the cppdriver library incorrectly?
You should cass_future_wait() on the execute future before testing the error code.
Unrelated: there are also a couple of things that should be freed (future, statement), but I'm assuming that was omitted to keep this concise.
So, it looks like (for whatever reason) I HAVE to parse out the row key from the results. I checked the example, and I was able to not parse out the row key information and everything still worked. I am not yet entirely sure what is forcing me to do this (compared to the provided paging example), but for others, you need to include the following within the while (cass_iterator_nex(iterator)) block to "magically" fix my code above.
CassUuid key;
char key_buffer[CASS_UUID_STRING_LENGTH];
const CassRow* row = cass_iterator_get_row(iterator);
cass_value_get_uuid(cass_row_get_column(row, 0), key);
cass_uuid_string(key, key_buffer);
This is really a long shot, but since you mentioned the Music Service example, did you possibly download and use cql_collections.zip query strings? If so, the strings (now fixed) had minor syntax errors:
-use music
-CREATE TABLE music.songs ( id uuid PRIMARY KEY, album text, artist text, data blob, reviews list, tags set, title text, venue map
+use music;
+CREATE TABLE music.songs ( id uuid PRIMARY KEY, album text, artist text, data blob, reviews list, tags set, title text, venue map);
AeroBuffalo's code worked for me except that I had to put '&' in front of the second parameter of cass_value_get_uuid() function. It required reference type.
cass_value_get_uuid(cass_row_get_column(row, 0), &key);
I create "database.db" and everything goes ok but why wouldn't it delete the table at the end? Everytime i run it, i get "table already exist" error message on creating the table.
int main()
{
sqlite3 *db; //Database Handle
char *zErr;
int rc;
char *sql;
rc = sqlite3_open("database.db", &db);
if(rc)
{
cout << "Can't open database: " << sqlite3_errmsg(db) << endl;;
sqlite3_close(db);
exit(1);
}
sql = "create table test(PID int primary key, Name text)"; //sql query
rc = sqlite3_exec(db, sql, NULL, NULL, &zErr); //execute sql statement
if(rc != SQLITE_OK)
{
if (zErr != NULL)
{
cout << "SQL error: " << zErr << endl;
sqlite3_free(zErr);
}
}
else
{
sql = "insert into test values (1,'John')";
rc = sqlite3_exec(db, sql, NULL, NULL, &zErr);
sql = "insert into test values (2,'Smith')";
rc = sqlite3_exec(db, sql, NULL, NULL, &zErr);
}
//delete the table on exit.
rc = sqlite3_exec(db, "delete from test", NULL, NULL, &zErr);
sqlite3_close(db);
return 0;
}
Also, can the primary keys be auto-generated following the last greater key existing in database?
You need to use drop table. delete deletes rows in the table.
rc = sqlite3_exec(db, "drop table test", NULL, NULL, &zErr);
SQLite will auto-increment an integer field declared as the primary key. See here.
You use a DELETE command to delete rows of a table.
You use a DROP command to drop a whole table or other DB item.
To create the table you can also use CREATE TABLE IF NOT EXISTS if you aren't sure if it exists.
To get your auto generated row ids use: id INTEGER PRIMARY KEY in your CREATE TABLE command.
Use DROP TABLE.
From SQLite docs:
The usual algorithm is to give the newly created row a ROWID that is one larger than the largest ROWID in the table prior to the insert.
The DML "delete from test" does delete all the rows from the table test. If you wish to drop the table, try "delete table test" or "drop table" instead.