glibc memory corruption with libmysqlcppconn prepared statement - c++

Question

I'm currently writing a web-crawler/spider in C++ on Linux and I'm having some problems with updating a database. I'm fairly new to C/C++, just FYI.
The database updates are executed by a seperate thread (using pthreads) but the same problem exists if executed in main() so I, perhaps naively, discarded the threading stuff as the cause of anything.
I'm using libmysqlcppconn for the database API.
I am compiling with gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1) with -O2 -Wall -pedantic and it compiles cleanly.
Nevertheless, when the function commitChangesToDatabase() below is called, it basically picks out items from a std::map (url_queue), throws them in a std::vector (updates) and erases said item from the original std::map, then proceeds to iterate over the std::vector, executing a MySQL prepared statement for each item in the vector. Here is where it fails hard.
It randomly either:
Crashes without any error output (no segfault, no stacktrace, no nothing)
Crashes with a glibc memory corruption detected (see output here: http://pastie.org/private/wlkuorivq5tptlcr7ojg)
Reports that the MySQL server has gone away (caught exception), but keeps trying (doesn't crash)
I have tried switching the prepared statement to a simple executeUpdate(), but to no avail.
I have tried eliminating the step with picking out items and rather just execute the updates whenever I find an item to update, in the first loop over url_queue.
Other functions in this application uses prepared statements as well (another UPDATE) and that works fine. Those functions are also run by seperate threads.
I would run the application through valgrind, but quite frankly, I don't understand most of the output so it wouldn't help me much - but if anyone wants the output from it, let me know which options to run it with and I'll provide it.
I have no clue how to proceed from here. Anyone have a clue what's wrong?
struct queue_item_t {
int id;
int sites_id;
int priority;
int depth;
int handler;
int state; // 0 = Pending, 1 = Working, 2 = Completed, 3 = Checked
double time_allowed_crawl;
bool status;
bool was_redirected;
double time;
double time_end;
double time_curl;
double size;
std::string hash;
std::string url;
std::string file;
std::string host;
};
void commitChangesToDatabase()
{
map< string, queue_item_t >::iterator it, end;
sql::PreparedStatement *pstmt;
int i = 0;
if (!url_queue.size()) {
return;
}
pthread_mutex_lock(&dbCommitMutex);
pthread_mutex_lock(&itemMutex);
cout << "commitChangesToDatabase()" << endl;
pstmt = dbPrepareStatement("UPDATE crawler_queue SET process_hash = NULL, date_crawled = NOW(), url = ?, hash = ? WHERE id = ?");
for (it = url_queue.begin(); it != url_queue.end();)
{
if (it->second.state == 2)
{
pstmt->setString(1, it->second.url);
pstmt->setString(2, it->second.hash);
pstmt->setInt(3, it->second.id);
try {
pstmt->executeUpdate();
++i;
} catch (sql::SQLException &e) {
cerr << "# ERR: SQLException in " << __FILE__;
cerr << "(" << __FUNCTION__ << ") on line " << __LINE__ << endl;
cerr << "# ERR: " << e.what();
cerr << " (MySQL error code: " << e.getErrorCode();
cerr << ", SQLState: " << e.getSQLState() << " )" << endl;
}
url_queue.erase(it++);
}
else {
++it;
}
}
delete pstmt;
cout << "~commitChangesToDatabase()" << endl;
pthread_mutex_unlock(&itemMutex);
pthread_mutex_unlock(&dbCommitMutex);
}
// this function is defined in another file but is written here just to show the contents of it
sql::PreparedStatement *dbPrepareStatement(const std::string &query)
{
return con->prepareStatement(query);
}
Edit:
Some seem to believe the problem is with the iteration over the url_queue collection, however I have ruled that out but commenting out everything that operates on the database, but not the iteration. Furthermore, the iteration here is a simplified (but working) version of the original which picks out items from the map, throws in a vector and erases from the map, as demonstrated below, and that part of the program works fine - it only crashes whenever the database is used.
for (it = url_queue.begin(); it != url_queue.end();)
{
if (it->second.state == 2)
{
update_item.type = (!it->second.was_redirected ? 1 : 2);
update_item.item = it->second;
updates.push_back(update_item);
url_queue.erase(it++);
}
else {
++it;
}
}
Edit 2:
Output from valgrind --leak-check=yes: http://pastie.org/private/2ypk0bmawwsqva3ikfazw

Answer 1

It seems, the iterator is incremented unnecessarily; first in loop body, and also in for statement. In this code, it is possible to increment the end iterator, this is a problematic operation and might be source of the problem.
The following loop structure is more suitable for this case:
it = url_queue.begin();
while( it != url_queue.end() ){
//loop body
}

Answer 2

I don't think it is a good idea to mess with iterators. replace:
else {
++it;
}
by:
else continue;
or just remove it.