Determine whether SQLite database is locked - c++

I've read other answers on how to detect whether the SQLite database is locked, and it suggests to use sqlite3_busy_handler/sqlite3_busy_timeout. I'm using Qt which has support of SQLite databases but that doesn't matter here.
Because in the use case of my application another application might access the same database, I need to handle this case.
Here's the options I set to the database when opening it:
auto db = QSqlDatabase::database();
db.setDatabaseName(m_sDatabasePath);
db.setConnectOptions("QSQLITE_BUSY_TIMEOUT=0");
if (!db.open())
return false;
I have a function which should determine whether database is locked or not:
int callback(void *data, int)
{
std::cout << "callback" << std::endl;
return 0;
}
bool isDatabaseLocked(const QSqlDatabase &db)
{
if (auto driver = db.driver())
{
// get driver handler
QVariant v = driver->handle();
if (v.isValid() && qstrcmp(v.typeName(), "sqlite3*") == 0)
{
// v.data() returns a pointer to the handle
auto handle = *static_cast<sqlite3 **>(v.data());
if (handle)
{
std::cout << sqlite3_busy_handler(handle, callback, nullptr) << std::endl;
std::cout << sqlite3_busy_timeout(handle, 0) << std::endl;
}
}
}
return true;
}
When the database is lcoked, I execute this function and get printed two 0s (SQLITE_OK) while I'd expect to get 5 (SQLITE_BUSY). And the callback function isn't called either.
What am I doing wrong?

As clearly stated in the documentation, the sqlite3_busy_handler function sets a callback function as the busy handler for the connection. Such callback might be invoked whenever a lock is met. In your code, the call to sqlite3_busy_handler returns SQLITE_OK, which is perfectly fine: it means the attempt to set the callback succeeded, no reason to return SQLITE_BUSY at this time, since the call only affects the connection, not the database.
About the same applies to the sqlite3_busy_timeout function, which in turn sets a different kind of busy handler (which replace the callback you just set, by the way) which simply sleeps for the specified amount of milliseconds until the lock is released.
Now, generally speaking, how to detect if a resource is locked? The usual way is: try to lock it.
So you could use a function like this:
bool isDatabaseLocked(const QSqlDatabase & db)
{
QSqlQuery q(db);
if(q.exec("BEGIN EXCLUSIVE")) //tries to acquire the lock
{
q.exec("COMMIT"); //releases the lock immediately
return false; //db is not locked
}
return true; //failed to acquire the lock: returns true (db is locked)
}
To make it return immediately when the lock cannot be acquired, clear the busy handler, setting the busy timeout to zero for this connection (before opening it):
QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");
db.setConnectOptions("QSQLITE_BUSY_TIMEOUT=0");
To test the function, open a command line sqlite3 session on the database, and lock it by typing:
sqlite> BEGIN EXCLUSIVE;
When you want to release the lock, just type
sqlite> COMMIT;

Related

Second thread is never triggered

I've been struggling with a multithreading issue for a bit. I've written some simple code to try and isolate the issue and I'm not finding it. What's happening is that the first thread is being woken up with data being sent to it, but second one never does. They each have their own condition_variable yet it doesn't seem to matter. Ultimately, what I'm trying to do is have a few long running threads that do a single dedicated task when needed, and staying in a wait state when not needed. And running them each in their own thread is important and a requirement.
Here's the code:
#include <glib.h>
#include <string>
#include <mutex>
#include <condition_variable>
#include <unistd.h>
#define NUM_THREADS 2
bool DEBUG = true;
pthread_t threads[NUM_THREADS];
std::mutex m_0;
std::mutex m_1;
std::condition_variable cov_0;
std::condition_variable cov_1;
bool dataReady_0 = false;
bool dataReady_1 = false;
bool keepRunning[NUM_THREADS] = { true };
void date_update (guint source_id, const char *json_data) {
if (DEBUG) {
start_threads(2);
sleep(2);
DEBUG = false;
}
g_print("From source id=%d\n", source_id);
switch (source_id) {
case 0:
dataReady_0 = true;
cov_0.notify_one();
break;
case 1:
dataReady_1 = true;
cov_1.notify_one();
break;
}
}
void start_threads (int thread_count) {
int rc;
switch (thread_count) {
case 2:
rc = pthread_create(&threads[1], nullptr, custom_thread_1, nullptr);
if (rc) {
g_print("Error:unable to create thread(1), return code(%d)\n", rc);
}
case 1:
rc = pthread_create(&threads[0], nullptr, custom_thread_0, nullptr);
if (rc) {
g_print("Error:unable to create thread(0), return code(%d)\n", rc);
}
}
}
void *custom_thread_0 (void *pVoid) {
g_print("Created thread for source id=0\n");
while (keepRunning[0]) {
// Wait until date_update() sends data
std::unique_lock<std::mutex> lck(m_0);
cov_0.wait(lck, [&]{return dataReady_0;});
dataReady_0 = false;
g_print("THREAD=0, DATA RECEIVED\n");
lck.unlock();
}
pthread_exit(nullptr);
}
void *custom_thread_1 (void *pVoid) {
g_print("Created thread for source id=1\n");
while (keepRunning[1]) {
// Wait until date_update() sends data
std::unique_lock<std::mutex> lck(m_1);
cov_1.wait(lck, [&]{return dataReady_1;});
dataReady_1 = false;
g_print("THREAD=1, DATA RECEIVED\n");
lck.unlock();
}
pthread_exit(nullptr);
}
Here's the output. As you can see the data_update function gets the "data" from the calling function for both source 0 and source 1, but only thread 0 ever seems to process anything. I'm at a bit of a loss as to the source of the problem.
Sending data for source id=1
From source id=1
Sending data for source id=0
From source id=0
THREAD=0, DATA RECEIVED
Sending data for source id=1
From source id=1
Sending data for source id=0
From source id=0
THREAD=0, DATA RECEIVED
Sending data for source id=1
From source id=1
Sending data for source id=0
From source id=0
THREAD=0, DATA RECEIVED
I'm sure I'm just missing a minor detail somewhere, but I'm fully willing to accept that perhaps I do not understand C/C++ threading correctly.
The 2nd thread is exiting because the keepRunning state flag is false. It's usually a good first step in debugging threads to log the start and exit of all threads.
But you have a much less obvious problem.
It does not appear that the appropriate mutex is held when the value of the condition variable's predicate is changed in date_update().
I'll break that down a bit more.
When cov_0.wait() is called, the predicate used is [&]{return dataReady_0;} (*), and the unique_lock passed is holding the mutex m_0. This means that whenever the value of the predicate might change, the mutex m_0 must be held.
This predicate is quite simple and will change value whenever the global variable dataReady_0 changes value.
In date_update() there is code to change the value of dataReady_0 and the mutex m_0 is not held when doing this. There should be a scoped_lock or unique_lock in the block that changes the global variable's state.
It will still mostly work without this, but you have a race! It will fail eventually!
The condition variable may check and see that the predicate is false, then another thread changes the predicate's value and does a notify, and then the first thread waits on the condition variable. It misses the notify because it was not yet waiting when it was sent. The use of the mutex to prevent the predicate from changing in a way that races with the notification is a critical component of what makes this work.
(*) You don't need the capture [&] here. This lambda could be stateless.
You should initialize all elements of the built-in array:
bool keepRunning[2] = { true, true };

How to use "pessimistic" Couchbase locking with the C API?

This is incredibly frustrating... I'm using the Couchbase v3 C API, and I ran into a problem that would be solved perfectly by using their "pessimistic" locking. After much effort I believe I've finally figured out how to lock records using it (with lcb_cmdget_locktime(), which presumably takes a parameter in microseconds), but I can't figure out how to unlock the record once I'm done with it, other than by letting it time out, which isn't an acceptable solution.
The documentation provides one example, here, but for some unknown reason it's in Javascript instead of C (!!!), and the concepts don't map to the C API.
Does anyone know how to unlock a pessimistic lock, or have any example C/C++ code using that API? Barring that, does anyone know where to find the source code for any of the non-C APIs that use the C one, since I should be able to work it out from there? (I haven't been able to locate any of those either, not sure they're open-source.)
Thank you for question
Time in seconds, note that the server might reset time to default, if it larger than maximum time (both durations are configurable). The following command will help to discover effective values for the feature.
$ cbstats -u Administrator -p password localhost all | grep ep_getl
ep_getl_default_timeout: 15
ep_getl_max_timeout: 30
To lock the key, one must use get operation with and set lock time using lcb_cmdget_locktime and capture CAS value in case of successful lock, for instance like this
struct my_result {
lcb_STATUS status{LCB_SUCCESS};
uint64_t cas{0};
};
static void get_callback(lcb_INSTANCE *instance, lcb_CALLBACK_TYPE, const lcb_RESPGET *resp)
{
my_result *res = nullptr;
lcb_respget_cookie(resp, (void **)&res);
res->status = lcb_respget_status(resp);
if (res->status == LCB_SUCCESS) {
lcb_respget_cas(resp, &res->cas);
}
}
It is good idea to put get with lock code into loop
uint64_t locked_cas{0};
int retries = 3;
while (retries > 0) {
std::string document_id{"foo"};
my_result result{};
lcb_CMDGET* cmd = nullptr;
lcb_cmdget_create(&cmd);
lcb_cmdget_key(cmd, document_id.c_str(), document_id.size());
lcb_cmdget_locktime(cmd, 5);
lcb_get(instance, &result, cmd);
lcb_cmdget_destroy(cmd);
lcb_wait(instance, LCB_WAIT_DEFAULT);
if (result.rc == LCB_SUCCESS) {
locked_cas = result.cas;
break;
} else if (result.rc == LCB_ERR_DOCUMENT_LOCKED || result.rc == LCB_ERR_TEMPORARY_FAILURE) {
std::this_thread::sleep_for(std::chrono::milliseconds(100));
--retries;
continue;
} else {
std::cerr << "Unexpected issue during get with lock: " << lcb_strerror_short(result.rc) << "\n";
break;
}
}
Once the key got locked, the update operation might be performed, but keep in mind that it MUST use locked_cas, otherwise mutation operation will fail.
std::string document_id{"foo"};
std::string new_value{"new value"};
lcb_CMDSTORE* cmd = nullptr;
lcb_cmdstore_create(&cmd, LCB_STORE_REPLACE);
lcb_cmdstore_key(cmd, document_id.c_str(), document_id.size());
lcb_cmdstore_value(cmd, new_value.c_str(), new_value.size());
lcb_cmdstore_cas(cmd, locked_cas);
lcb_store(instance, nullptr, cmd);
lcb_cmdstore_destroy(cmd);
lcb_wait(instance, LCB_WAIT_DEFAULT);
To unlock key you also need locked_cas, or just wait until server will unlock the document automatically.
std::string document_id{"foo"};
lcb_CMDUNLOCK *cmd = nullptr;
lcb_cmdunlock_create(&cmd);
lcb_cmdunlock_key(cmd, document_id.c_str(), document_id.size());
lcb_cmdunlock_cas(cmd, locked_cas);
lcb_unlock(instance, nullptr, cmd);
lcb_cmdunlock_destroy(cmd);
lcb_wait(instance, LCB_WAIT_DEFAULT);

How to prevent name collisions when creating QSqlDatabase connection in multiple threads

I have multi-threaded QTcpServer and for each database request, it creates new Thread to keep server Responsive. So in each thread I have to creating new QSqlDatabase connection. But I keep getting name collisions between connections.
here is my sample code to recreate issue.:-
#include <QSqlDatabase>
class DBTask
{
public:
DBTask(ClientSocket *socket,ConnectionWorker *connectionWorker);
~DBTask();
static void initStatic();
private:
static QThreadPool *pool; // all addConnection() call be be called in QtConcurrent::run with this pool
static QString host, user, type, password, name;
static quint64 dbConnectionNumber;
QSqlDatabase db;
ClientSocket *socket;
ConnectionWorker *connectionWorker;
bool addDatabase() ;
};
quint64 DBTask::dbConnectionNumber=0;
DBTask::DBTask(ClientSocket *socket, ConnectionWorker *connectionWorker):
socket(socket),
connectionWorker(connectionWorker)
{
dbConnectionNumber++;
}
bool DBTask::addDatabase() {
QSqlDatabase db = QSqlDatabase::addDatabase(type,QString::number(dbConnectionNumber));
db.setHostName(host);
db.setDatabaseName(name);
db.setUserName(user);
db.setPassword(password);
if(!db.open()){
qWarning() << "Error while opening database for socket " << socket << '\n' << db.lastError();
return false;
}
else {
return true;
}
}
this works fine when I manually check my application with GUI with human speed But when I run a c++ test code which simulates thousands of requests like this:-
void connectionTest(){
QThreadPool pool;
pool.setMaxThreadCount(10);
for(int i=0;i<10;i++){
QtConcurrent::run(&pool,[this](){
for(int i=0;i<1000;i++){
login(i%2); // login function sends request to QTcpServer
}
});
}
}
I get multiple errors like this:-
QSqlDatabasePrivate::removeDatabase: connection '10' is still in use, all queries will cease to work.
QSqlDatabasePrivate::addDatabase: duplicate connection name '10', old connection removed.
QSqlDatabasePrivate::removeDatabase: connection '10' is still in use, all queries will cease to work.
QSqlDatabasePrivate::addDatabase: duplicate connection name '10', old connection removed.
and Server crashes with segfault
Even if you make the counter atomic, a thread can still get interrupted in the DBTask::addDatabase method (before creating the connection), another one can increment the counter and then they both continue and create 2 connections with the same id. You need to make both operations (increment the counter and the connection creation) in one transaction: inside the DBTask::addDatabase, by making use of a mutex lock.
After adding QMutex to addDatabase, it works:-
bool DBTask::addDatabase() {
mutex.lock();
dbConnectionNumber++;
db = QSqlDatabase::addDatabase(type,QString::number(dbConnectionNumber));
mutex.unlock();
...
}

Object pointer randomly points to 0x00000

I'm running a Visual C++ MFC application in release mode. I'm compiling everything using Visual Studio 2010.
My app runs a mini CNC mill through USB VCP communication.
I have a XML file that stores the app's settings.
My problem is this: ocassionaly (and this is repeatable) the pointer to the tinyxml2::XMLDocument I'm using gets set to 0x000.
Info:
Occasionally, the XML file get written to while the mill is running.
Before the error happens, the mill I'm running siezes for almost 30 seconds.
I'm using mutex locks to make sure the xmldoc doesn't get written to file twice at once.
The mutex locks are working, and the mutex error never occurs. I know the mutex code isn't perfect, but that isn't the issue. Honest.
I never write to the xmldoc pointer except when the parent class is booting up.
And then, all of a sudden, the xmlDoc pointer gets set to zero.
Any thoughts anyone?
Here is my saving code, although the problem may lie elsewhere:
void XMLSettings::SaveToXML()
{
HANDLE g_Mutex = CreateMutex( NULL, TRUE, "XMLSavingMutex");
DWORD wait_success = WaitForSingleObject( g_Mutex, 30000L);
if(wait_success == WAIT_OBJECT_0){
CIsoProApp* pApp = (CIsoProApp*)AfxGetApp();
if(PathFileExists(pApp->DrivePath + "IsoPro\\temp.xml"))
{
DeleteFile(pApp->DrivePath + "IsoPro\\temp.xml");
}
if(0==&xmlDoc)
{
OutputDebugString("xmlDoc == NULL");
}
int errorcode = xmlDoc->SaveFile(pApp->DrivePath + "IsoPro\\temp.xml");
if(errorcode != 0)
{
OutputDebugString("xmlDoc == errorcode");
}
if(0==&xmlDoc)
{
OutputDebugString("xmlDoc == NULL2");
}
if(0==xmlDoc)
{
OutputDebugString("xmlDoc == NULL");
}
if(PathFileExists(pApp->DrivePath + "IsoPro\\Settings.xml"))
{
DeleteFile(pApp->DrivePath + "IsoPro\\Settings.xml");
}
MoveFile(pApp->DrivePath + "IsoPro\\temp.xml",pApp->DrivePath + "IsoPro\\Settings.xml");
ReleaseMutex(g_Mutex);
}
else
{
int errorInt = GetLastError();
CString error;
error.Format("%d",errorInt);
if(errorInt != ERROR_ALREADY_EXISTS)
{
AfxMessageBox("XMLSavingMutex Error. WaitSuccess = " + wait_success);
AfxMessageBox("XMLSavingMutex Error. GetLastError = " + error);
}
}
CloseHandle(g_Mutex);
}
Since it seems that you are creating a Mutex each time SaveToXML is called, you should change your call to
HANDLE g_Mutex = CreateMutex( NULL, FALSE, "XMLSavingMutex");
Doing this will create a named mutex that allows the implementation to dictate who the owner is; other threads will receive the same mutex.
From the doc:
Two or more processes can call CreateMutex to create the same named mutex. The first process actually creates the mutex, and subsequent processes with sufficient access rights simply open a handle to the existing mutex. This enables multiple processes to get handles of the same mutex, while relieving the user of the responsibility of ensuring that the creating process is started first. When using this technique, you should set the bInitialOwner flag to FALSE; otherwise, it can be difficult to be certain which process has initial ownership.
(Credit to WhozCraig for pointing out named mutexes)
It appears that I was accessing the xml getter while writing the xml to a file. I put a single mutex lock in place for all xml actions and things seem to be functioning properly. Thanks to everyone for their help. I'll be in touch with more info if it becomes available.

How check fast, if database reachable? (Qt, QML, C++)- Linux

I use qt with qml and c++. On my application i use a database.
It all works, if the database is reachable.
My problem is, that i would like to check, if database is reachable (like ping).
I tried
db.setDatabaseName(dsn);
if(db.isValid())
{
if(db.open())
{
//std::cout <<"Offene Datenbank";
connected=true;
}
else
{
connected=false;
}
}
else
{
connected=false;
}
and give the connected value as result. But that takes very long (maybe 30 seconds), if there is no connection. How i can check fast, if i have a database connection?
Is there maybe a way to break the command .open after 5 seconds not connected?
I think one easy solution is to just check the ping of the database sever. You can use platform specific ways for pinging.
This would work on Linux :
int exitCode = QProcess::execute("ping", QStringList() << "-c 2" << serverIp);
if (exitCode==0)
{
// is reachable
} else
{
// is not reachable
}
I have studied this question a bit. Here is what I found out.
The problem is in default db connection timeout - it is too long. Each db allows you to change it to an acceptable value, using their own API. In Qt there is one common db interface - QSqlDatabase. And it does not have such method. You can set connection settings by calling it's QSqlDatabase::setConnectOptions method, but it accepts only predefined list of options (which you can read in Qt's help).
For PostgreSQL there is an option connect_timeout, so you can write:
db.setConnectOptions("connect_timeout=5"); // Set to 5 seconds
For other databases there is no such parameter. Connection options of each db are parsed in it's 'driver' class, which derives QSqlDriver and is stored in a 'driver' library.
So, what you can do:
You can rewrite database's driver in order it to accept timeout option.
You can write separate code for each db, using it's native API.
UPDATE
Turns out, that ODBC has SQL_ATTR_CONNECTION_TIMEOUT option.
UPDATE 2
qsql_odbc.cpp:713
} else if (opt.toUpper() == QLatin1String("SQL_ATTR_CONNECTION_TIMEOUT")) {
v = val.toUInt();
r = SQLSetConnectAttr(hDbc, SQL_ATTR_CONNECTION_TIMEOUT, (SQLPOINTER) v, 0);
https://msdn.microsoft.com/en-us/library/ms713605(v=vs.85).aspx
SQL_ATTR_CONNECTION_TIMEOUT (ODBC 3.0)
An SQLUINTEGER value
corresponding to the number of seconds to wait for any request on the
connection to complete before returning to the application. The driver
should return SQLSTATE HYT00 (Timeout expired) anytime that it is
possible to time out in a situation not associated with query
execution or login.
If ValuePtr is equal to 0 (the default), there is no timeout.
Should work fine...
I suggest having some separate thread/class where you check connection and emit signal after some timeout if nothing happens (with a check - knowConnection - if we found out already if its connected).
This code is not tested and written from scratch on top of my head.. may contain some errors.
/// db connection validator in separate thread
void validator::doValidate() {
this->knowConnection = false;
db.setDatabaseName(dsn);
if(db.isValid())
{
QTimer::singleShot(1000, [this]() {
if (!this->knowConnection) {
emit connected(false);dm->connected=false;
}
});
if(db.open())
{
//std::cout <<"Offene Datenbank";
this->knowConnection = true;
dm->connected=true;
emit connected(true);
}
else
{
dm->connected=false;
this->knowConnection = true;
emit connected(false);
}
}
else
{
dm->connected=false;
this->knowConnection = true;
emit connected(false);
}
}
/// db manager in different thread
void dm::someDbFunction() {
if (connected) {
/// db logic
}
}
/// in gui or whatever
MainWindow::MainWindow() : whatever, val(new validator(..), .. {
connect(val, SIGNAL(connected(bool)), this, SLOT(statusSlot(bool));
....
}
void MainWindow::statusSlot(bool connected) {
ui->statusBar->setText((connected?"Connected":"Disconnected"));
}