MySQL server-side timeout - c++

I have some connection code that causes a timeout when queries take too long. The connection options are setup like this (timeout is an integer):
sql::ConnectOptionsMap com;
com["hostName"] = url; // Some string
com["userName"] = user; // Some string
com["password"] = pwd; // Some string
com["OPT_RECONNECT"] = true;
com["OPT_READ_TIMEOUT"] = timeout; // Usually 1 (second)
com["OPT_WRITE_TIMEOUT"] = timeout; // Usually 1 (second)
After testing the timeout setup above what I found is that a throw does occur but MySQL continues trying to execute the query. In other words, the below try goes to the catch after the configured timeout with error code 2013 but it doesn't stop MySQL from trying to execute the query (2013 is an error code related to lost connection):
// Other code
try
{
stmt = con->createStatement();
stmt->execute("DROP DATABASE IF EXISTS MySQLManagerTest_TimeoutRead");
stmt->execute("CREATE DATABASE MySQLManagerTest_TimeoutRead");
stmt->execute("USE MySQLManagerTest_TimeoutRead");
stmt->execute("CREATE TABLE foo (bar INT)");
for (int i = 0; i < 100; i++)
stmt->execute("INSERT INTO foo (bar) VALUES (" + LC(i) + ")");
// A bit of playing is needed in the loop condition
// Make it longer than a second but not too long
// Using 10000 seems to take roughly 5 seconds
stmt->execute(
"CREATE FUNCTION waitAWhile() "
"RETURNS INT READS SQL DATA "
"BEGIN "
"DECLARE baz INT DEFAULT 0; "
"WHILE baz < 10000 DO "
"SET baz = baz + 1; "
"END WHILE; "
"RETURN baz; "
"END;"
);
res = stmt->executeQuery("SELECT 1 FROM foo WHERE bar = waitAWhile()");
} catch (sql::SQLException &e) {
std::cout << e.getErrorCode() << std::endl;
}
// Other code
I was able to notice that MySQL did not stop by running "top" at the same time as the above testing code. Making the above MySQL waitAWhile() function instead be an infinite loop further confirmed that MySQL was not stopping because I had to kill the MySQL process to make it stop
This kind of timeout is not what I wanted, I wanted MySQL to give up on a query if it took too long. Can this be done (so that both my execution and MySQL stop doing work)? Additionally, can this be specified only for INSERT queries?

You can do it in regular SQL by having SQL Server set a max query execution time. However, it doesn't look like MySQL supports this; see the following accepted SO answer for more details:
MySQL - can I limit the maximum time allowed for a query to run?

Related

SQLite in C++. DB is BUSY (Multithread)

I've got a problem. I'm using SQLite3 in my C++ project. In the log, I've got errors: DB is locked error code 5. As I know, error code 5 means that DB is busy. To solve this, I started to use WAL journal mode. But it doesn't help.
In my program, I've got 2 connections to the same DB. I use mutexes for both DB connections.
I'm opening connections with this code:
if (sqlite3_open_v2(db_path.c_str(), &this->db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX, 0) ) {
LOG4CPLUS_FATAL(this->logger, "Can not open/create DB " << sqlite3_errmsg(db));
sqlite3_close(this->db);
}
if (sqlite3_exec(this->db, "PRAGMA journal_mode = WAL;", 0, 0, &err)) {
LOG4CPLUS_ERROR(this->logger, "SQL det journal mode error: " << err);
sqlite3_free(err);
}
The first connection is used for inserting data to the DB. It happens 4 times every second.
The second connection is used for starting transaction, selecting, updating, deleting data, and committing. It happens every 5 seconds.
I'm getting errors from the first connection.
Please help me to solve this problem.
Update:
First connection:
void readings_collector::flushToDb()
{
this->db_mutex.lock();
LOG4CPLUS_DEBUG(this->logger, "Flush to DB start.");
const char *query = "INSERT INTO `readings` (`sensor_id`, `value`, `error`, `timestamp`) VALUES (?,?,?,?)";
sqlite3_stmt *stmt = NULL;
int rc = sqlite3_prepare_v2(this->db, query, -1, &stmt, NULL);
if (SQLITE_OK != rc) {
LOG4CPLUS_ERROR(this->logger, "sqlite prepare insert statment error: " << sqlite3_errmsg(this->db));
}
LOG4CPLUS_TRACE(this->logger, "--------------------");
LOG4CPLUS_TRACE(this->logger, this->readings.size());
while(!this->readings.empty()) {
sensor::reading temp_reading = this->readings.front();
this->readings.pop();
LOG4CPLUS_TRACE(this->logger, "Reading " << temp_reading.sensor_id << " : " << temp_reading.value << " : " << temp_reading.error << " : " << temp_reading.timestamp);
sqlite3_clear_bindings(stmt);
sqlite3_bind_int(stmt, 1, temp_reading.sensor_id);
sqlite3_bind_text(stmt, 2, temp_reading.value.c_str(), sizeof(temp_reading.value.c_str()), NULL);
sqlite3_bind_int(stmt, 3, temp_reading.error);
sqlite3_bind_int(stmt, 4, temp_reading.timestamp);
rc = sqlite3_step(stmt);
if (SQLITE_DONE != rc) {
LOG4CPLUS_ERROR(this->logger, "sqlite insert statment exec error: " << sqlite3_errmsg(this->db) << "; status: " << rc);
}
}
sqlite3_finalize(stmt);
LOG4CPLUS_TRACE(this->logger, "Flush to DB finish.");
this->db_mutex.unlock();
}
Second connection:
void dataSend_task::sendData()
{
this->db_mutex.lock();
char *err = 0;
LOG4CPLUS_INFO(this->logger, "Send data function");
if (sqlite3_exec(this->db, "BEGIN TRANSACTION", 0, 0, &err)) {
LOG4CPLUS_ERROR(this->logger, "SQL exec error: " << err);
sqlite3_free(err);
}
if (sqlite3_exec(this->db, this->SQL_UPDATE_READINGS_QUERY, 0, 0, &err)) {
LOG4CPLUS_ERROR(this->logger, "SQL exec error: " << err);
sqlite3_free(err);
}
this->json.clear();
this->readingsCounter = 0;
if (sqlite3_exec(this->db, this->SQL_SELECT_READINGS_QUERY, +[](void *instance, int x, char **y, char **z) {
return static_cast<dataSend_task *>(instance)->callback(0, x, y, z);
}, this, &err)) {
LOG4CPLUS_ERROR(this->logger, "SQL exec error: " << err);
sqlite3_free(err);
} else {
LOG4CPLUS_TRACE(this->logger, "Json data: " << this->json);
if (this->curlSend()) {
if (sqlite3_exec(this->db, this->SQL_DELETE_READINGS_QUERY, 0, 0, &err)) {
LOG4CPLUS_ERROR(this->logger, "SQL exec error: " << err);
sqlite3_free(err);
}
}
}
if (sqlite3_exec(this->db, "COMMIT", 0, 0, &err)) {
LOG4CPLUS_ERROR(this->logger, "SQL exec error: " << err);
sqlite3_free(err);
}
this->db_mutex.unlock();
this->json.clear();
}
As you've no doubt realized, SQLite only allows one connection at a time to be be updating the database.
From the code you have pasted, it looks as though you have two separate mutexes, one for the readings_collector instance, another for the dataSend_task instance. These would protect against multiple executions of each of the two functions but not against both of those functions running at once.
It wasn't that clear from your question what the purpose of the mutexes is, but it certainly isn't going to prevent both of those connections from simultaneously trying to update the database.
I can suggest two approaches to fix your problem.
The first would be to use a single shared mutex between those two instances, so that only one of them at a time can be updating the database.
The second would be to take advantage of the facilities SQLite provides for resolving contention when accessing the database. SQLite allows you to install a 'busy handler' which will be called in the event that an attempt is made to access a database which is already locked by another thread or process. The busy handler can take whatever action is desired, but the simplest case is normally just to wait a while and try again, which is catered for by the built in busy handler which you can install by calling sqlite3_busy_timeout.
For example, immediately after opening your database connection, you could do this:
sqlite3_busy_timeout(this->db, 1000); // Wait 1000mS if busy
It is also possible to set such a timeout by command, using the busy_timeout pragma.
You may also wish to consider starting your transaction using BEGIN IMMEDIATE TRANSACTION or BEGIN EXCLUSIVE TRANSACTION so that the transaction can be guaranteed to complete without blocking. See the documentation on transactions.
Please check these two Stack Overflow posts. They seem to be related to your issue.
Can different connections of the same sqlite's database begin transactions concurrently?
If you read the SQLite documentation, you will see that it supports
multiple connections for reading only, you cannot write to the
database from mulitple connections, because it's not designed for
that.
Read and Write Sqlite database data concurrently from multiple connections
Multiple processes can have the same sqlite database open at the same
time, several read accesses can be satisfied in parallel.
In case of write, a single write to the database does lock the
database for a short time, nothing, even reading, can access the
database file at all.
Beginning with version 3.7.0, a new “Write Ahead Logging” (WAL) option
is available. In which Reading and writing can proceed concurrently.
By default, WAL is not enabled. To turn WAL on, please refer to Sqlite
documentation.

modbus communication decrease wait time to timeout

I'm writing a C++ app that will need to connect to various PLCs over modbus, the IPs of these PLCs is given by user input. Currently when the user puts in an IP that cannot be connected to, my program hangs for around 2 minutes in an attempt to connect and having my app hang for 2 minutes is not an option.
An example program illustrates the issue and my attempted fixes:
#include <modbus/modbus.h>
#include <string>
#include <errno.h>
#include <iostream>
#define PRINT_TIMEVAL(timeval) std::cout << "timeval sec: " << timeval.tv_sec << " usec: " << timeval.tv_usec << std::endl;
int main()
{
std::string ip = "192.168.2.5";
int port = 502;
int slaveNum = 1;
int address = 1;
int nb = 1;
struct timeval currentTimeout;
struct timeval responseTimeout;
responseTimeout.tv_sec = 1;
responseTimeout.tv_usec = 0;
struct timeval byteTimeout;
byteTimeout.tv_sec = 1;
byteTimeout.tv_usec = 0;
modbus_t *mb = modbus_new_tcp(ip.c_str(), port);
modbus_set_debug(mb, true);
modbus_set_error_recovery(mb, MODBUS_ERROR_RECOVERY_NONE);
modbus_flush(mb);
modbus_set_slave(mb, slaveNum);
modbus_get_response_timeout(mb, &currentTimeout);
PRINT_TIMEVAL(currentTimeout);
modbus_set_response_timeout(mb, &responseTimeout);
modbus_get_response_timeout(mb, &currentTimeout);
PRINT_TIMEVAL(currentTimeout);
modbus_get_byte_timeout(mb, &currentTimeout);
PRINT_TIMEVAL(currentTimeout);
modbus_set_byte_timeout(mb, &byteTimeout);
modbus_get_byte_timeout(mb, &currentTimeout);
PRINT_TIMEVAL(currentTimeout);
std::cout << "About to connect to " << ip << std::endl;
int errno;
if((errno = modbus_connect(mb)))
{
std::cout << "Error when connecting: " << modbus_strerror(errno) << std::endl;
}
std::cout << "Done connecting to " << ip << std::endl;
modbus_close(mb);
modbus_free(mb);
return 0;
}
As you can see I've tried setting both the response and byte timeout variables to 1 second (I've also tried 500 and 5000 microseconds). When I read the timeout values they have been set properly so I'm assuming that they don't have anything to do with the initial connection attempt. I've also tried explicitly setting the error recovery mode to none in case it was trying to reconnect on its own.
I would like something that will either stop modbus_connect after x amount of time or another command that will allow me to check to see if the IP is valid before attempting to connect through modbus, this would also need to timeout after a short amount of time.
I'm using libmodbus version 3.0.1-2
The issue was with my version of libmodbus (3.0.1), which is the current release version. In that version they were using the linux connect command but they weren't passing a NONBLOCKING flag, thus connect would become blocked for 2m7s. We resolved this issue by upgrading to libmodbus v3.1.1 which is marked as unstable but is not under active development (they're developing on v3.1.2). Unfortunately that version of libmodbus does not work for windows.
Use threads to listen for each device and push those messages into a queue that can be processed without holding up the other threads.

libmysqlclient.18.dylib memory leak

PROBLEM: What's the cause of the memory leaks?
SITUATION:
I've build a simple command line program using C++ together with MySQL using the MySQL C API
The problem is, the program has many "minor" memory leaks from the object malloc xx bytes" with xx ranging from a few bytes to 8 kb. All of the leaks links to the library libmysqlclient.18.dylib.
I've already removed all the mysql_free_result() from the code to see if that was the problem, but its still the same.
My MySQL code mainly consists of simple code like:
to connect:
MYSQL *databaseConnection()
{
// declarations
MYSQL *connection = mysql_init(NULL);
// connecting to database
if(!mysql_real_connect(connection,SERVER,USER,PASSWORD,DATABASE,0,NULL,0))
{
std::cout << "Connection error: " << mysql_error(connection) << std::endl;
}
return connection;
}
executing a query:
MYSQL_RES *getQuery(MYSQL *connection, std::string query)
{
// send the query to the database
if (mysql_query(connection, query.c_str()))
{
std::cout << "MySQL query error: " << mysql_error(connection);
exit(1);
}
return mysql_store_result(connection);
}
example of a query:
void resetTable(std::string table)
{
MYSQL *connection = databaseConnection();
MYSQL_RES *result;
std::string query = "truncate table " + table;
result = getQuery(connection, query);
mysql_close(connection);
}
First of all: Opening a new connection for every query (like you're doing in resetTable()) is incredibly wasteful. What you really want to do is open a single connection when the application starts, use that for everything (possibly by storing the connection in a global), and close it when you're done.
To answer your question, though: You need to call mysql_free_result() on result sets once you're done with them.

MySQL Segfault, Intermittent

I am trying to run through a series of checks/inserts into a MySQL 5.5 db, but I am having frequent yet intermittent issues with SIGSEGV errors. Over the course of many queries being executed, the SELECT statements run just fine. However, after some variable amount of time or number of executed queries (sometimes thousands of checks, sometimes 1 or 2, sometimes not at all and the program exits normally), I inexplicably get a segfault...
Program received signal SIGSEGV, Segmentation fault.
0x100188a8 in mysql_send_query () from K:\Programming\C\Test\libmysql.dll
(gdb) bt full
#0 0x100188a8 in mysql_send_query () from K:\Programming\C\Test\libmysql.dll
No symbol table info available.
#1 0x100188e5 in mysql_real_query () from K:\Programming\C\Test\libmysql.dll
No symbol table info available.
#2 0x00000000 in ?? ()
No symbol table info available.
(gdb)
This is from my heavily reduced code:
int main() {
for (int i = 0; i < 5000; i++) {
int iNewX = GenerateRandomInt(1, 50);
int iNewY = GenerateRandomInt(1, 50);
std::string str = "SELECT * FROM Resources WHERE XPOS = ";
str = str +
StatToString(iNewX) + " AND YPOS = " +
StatToString(iNewY) + ";";
const char * Query = str.c_str();
MYSQL *connect;
connect=mysql_init(NULL);
connect=mysql_real_connect(connect,SERVER,USER,PASSWORD,DATABASE,0,NULL,0);
// Print SQL statment for debugging only...
// This appears to always be good, even in the case of the segfault.
std::cout << Query << std::endl;
if (mysql_query(connect, Query)) {
// Supposed to log an error; I don't get this far...
// This does work when I intentionally break the statement.
std::cout << printf("Failed to SELECT, Error: %s", mysql_error(connect));
std::cout << printf("Query: %s", Query) << std::endl;
mysql_close(connect);
return 0;
}
mysql_close(connect);
}
return 1;
}
I have been unsuccessful in searching online for a case that really matches what I have going on here (though there are lots of MySQL/segfault related forum/Q+A topics/threads). Since this appears to be happening within the .dll itself, how can I fix this?
Can anyone explain why the issue seems to come and go?
I have not yet tried to reinstall MySQL, as that will likely be a very big headache that I would rather avoid. If I must, then I must.
If I am missing any details in my question or any pertinent code, please let me know and I will add.
After following Christian.K's advice, I was able to see that this was error 23 (as returned by mysql_error(connect)) after connect=mysql_init(NULL).
This led me to a few resources, most clearly, this one. This says that this is a know problem when working within Windows, and there's not much I can do about this.
You might get around the open file limit (error 23) by not opening a connection for every loop iteration (which is questionable anyway), but rather use one connection for all loop iterations.
Together with my comments about error handling, and the strange cout << printf use you end up with something like this:
int main() {
MYSQL *connect;
connect=mysql_init(NULL);
if (connect == NULL)
{
printf("Insufficient memory to initialize.\n");
return 1;
}
connect=mysql_real_connect(connect,SERVER,USER,PASSWORD,DATABASE,0,NULL,0);
if (connect == NULL)
{
printf("Could not connect: %s\n", mysql_error(connect);
return 1;
}
for (int i = 0; i < 5000; i++) {
int iNewX = GenerateRandomInt(1, 50);
int iNewY = GenerateRandomInt(1, 50);
std::string str = "SELECT * FROM Resources WHERE XPOS = ";
str = str +
StatToString(iNewX) + " AND YPOS = " +
StatToString(iNewY) + ";";
const char * Query = str.c_str();
if (mysql_query(connect, Query)) {
// Supposed to log an error; I don't get this far...
// This does work when I intentionally break the statement.
printf("Failed to SELECT, Error: %s", mysql_error(connect));
printf("Query: %s", Query);
mysql_close(connect);
return 1;
}
}
mysql_close(connect);
return 0;
}
Note that I also changed the return values. Per convention main() should return 0 on success and something else (mostly 1) otherwise.

Increasing memory usage in sqlite3?

I've written a console app which receives events via boost::interprocess memory and dumps the info into an sqlite3 database. While running the app I've noticed that, in the Windows task manager, the memory usage was cyclically increasing every... 30s-1min. This led me to believe that the problem lies within the main loop in which I execute my SQL. I've added some monitoring and apparently the sqlite3_memory_usage returns increasing results every couple loop iterations.
Can somebody tell me what am I doing wrong? Am I missing something I should de-allocate ?
Here are 2 strings I use to generate SQL
const std::string sql_insert =
"INSERT INTO EventsLog "
"(Sec, uSec, DeviceId, PmuId, EventId, Error, Msg) "
"VALUES (%ld, %ld, %ld, %d, %ld, %d, %Q)";
const std::string sql_create =
"CREATE TABLE IF NOT EXISTS EventsLog("
"Id INTEGER PRIMARY KEY AUTOINCREMENT, "
"Sec INTEGER NOT NULL, "
"uSec INTEGER NOT NULL, "
"DeviceId INTEGER NOT NULL, "
"PmuId INTEGER NOT NULL, "
"EventId INTEGER NOT NULL, "
"Error INTEGER NOT NULL, "
"Msg TEXT"
")";
In here, I generate the SQL INSERT command
std::string construct_sql_query
(const ELMessageData & data)
{
std::string query = "";
ptime jan1st1970 = ptime(date(1970,1,1));
ptime now = boost::posix_time::microsec_clock::universal_time();
time_duration delta = now - jan1st1970;
TimeVal time((uint32)delta.total_seconds(),
(uint32)now.time_of_day().fractional_seconds());
char * const sql = sqlite3_mprintf(sql_insert.c_str(),
time.tv_sec,
time.tv_usec,
data.getDeviceId(),
data.getPmuId(),
data.getEventId(),
(data.getIsError() ? 1 : 0),
data.getExMsg().c_str());
if(sql == NULL)
post_event(EvIOError("Failed to create the SQL command",
"StLoggingEvents::_construct_sql_query"));
query = std::string(sql);
sqlite3_free(sql);
return query;
} // construct_sql_query
Here's the main loop in which I execute the INSERT commands
while(true)
{
m_exchange_obj->wait(); // wait for the semaphore to be raised
const std::string sql = construct_sql_query
(m_exchange_obj->receive());
char ** err = NULL;
const int rc = sqlite3_exec(m_db_handle,
sql.c_str(),
NULL,
NULL,
err);
sqlite3_free(err);
if(rc != SQLITE_OK)
{
LERR_ << "Error while inserting into the database";
LERR_ << "Last SQL Query : ";
LERR_ << sql;
}
else
{
LDBG_ << "Event logged...";
LDBG_ << "Sqlite3 memory usage : "
<< sqlite3_memory_used();
}
}
I second the suggestion of trying this under valgrind. You may also want to look at google's tcmalloc replacement... It can print pretty graphs showing you all your leaks... That said, I hope you get the answer for this... I plan on using SQLite in an upcoming project...
How are you determining your memory usage? You may not have a real leak.
If you are on a windows system and using visual studio, compile in debug mode and use the memory debugging macros to find your leaks.
If you are on a unix based system, try valgrind / memcheck.
I think OS X's xcode also detects leaks too.