Reading the SQLite documentation here, when a process wants to write to a SQLite database it obtains a reserved lock. Then once the process is ready to write to disk it obtains a pending lock, during which no new processes can obtain a shared lock, but existing shared locks are allowed to finish their business. Once the remaining shared locks clear, the process can write.
However...when I try to write a database while other processes are reading from that database, I just get an immediate "Error: dataset is locked".
sqlite> insert into meta_models (model) values ("hello_world");
Error: database is locked
Why doesn't SQLite it go through the steps I outlined above of waiting for the shared locks to clear?
SQLite does go through these steps, but it timed out while waiting for the remaining shared locks to clear.
You can adjust this timeout with PRAGMA busy_timeout.
Related
I'm developing a program (using C++ running on a Linux machine) that uses SQLite as a back-end.
It has 2 threads which carry out the following tasks:
Thread 1
Waits for a piece of data to arrive (in this case, via a radio module)
Immediately inserts it into the database
Returns to waiting for new data
It is important this thread is "listening" for as much of the time as possible and isn't blocked waiting to insert into the database
Thread 2
Every 2 minutes, runs a SELECT on the database to find un-processed data
Processes the data
UPDATEs the rows fetched with a flag to show they have been processed
The key thing is to make sure that Thread 1 can always INSERT into the database, even if this means that Thread 2 is unable to SELECT or UPDATE (as this can just take place at a future point, the timing isn't critical).
I was hoping to find a way to prioritise INSERTs somehow using SQLite, but have failed to find a way so far. Another thought was for Thread 1 to push it's the data into a basic queue (held in memory) and then bulk INSERT it every so often (as this wouldn't be blocking the receiving of data and could do a simple check to see if the database was locked, if so, wait a few milliseconds and try again).
However, what is the "proper" way to do this with SQLite and C++ threads?
SQlite database can be opened with or without multi-threading support. Both threads should open the database separately.
If you want to do the hard way, you can use a priority queue and process the queries.
I have multiple processes (c++, Windows 8) that use the same SQLite database. I configured connections with SQLITE_CONFIG_SERIALIZED and PRAGMA busy_timeout = 60000;. Used journaling mode - DELETE.
Test scenario:
process #1 opens connection, makes reads/writes, sleeps for 5 sec
process #2 opens connection, makes reads/writes
After that process #1 failed to write to the database - it receives SQLITE_BUSY immediate after call to SQLite API (sqlite3_step, sqlite3_finalize). Process #2 still uses the connection without any problems.
I do not have any not closed transactions, I do not have any long operations on the database. What else can lead to this?
I use the same SQLite connection from the multiple threads inside the process. SQLite docs says that this is OK with config option SQLITE_CONFIG_SERIALIZED. Any exception for this rule?
SQLite acquires lock on database/table in sqlite3_prepare and releases it in sqlite3_finalize. Type of lock depends on your SQL expression.
If you created an STMT - you need to execute it and finalize as soon as possible. Otherwise you block different connections.
My applications created list of prepared STMT and kept it until the end. Generally this is misuse of SQLite.
Links:
SQLite locking
How much overhead of keeping sqlite3 database opened VS. open database only when need.
The application is high load.
1) But it's hard to write version that will use one handler per thread, but I can write something like driver that will keep ie. 3-5 handlers opened and ready for reading and 1 for writing. Drive them for threads by request, keep mutexes etc. ( not easy solution to implement )
VS.
2) open sqlite database only when I need it by some thread and give sqlite to do all job, but here is additional overhead to open database each time. (easy to implement)
UPDATE:
3) there are other option: I can keep one handler opened per database and use simple mutex to lock access to the database. The disadvantages of this is that I loose concurrency reads. So, only one thread will be able to read or write, while by option 3 there is concurrency free reading (more then 1 reader can read at the time)
You should keep it open.
Open and close file is more expensive then keep one file handler opened.
You can simulate the cost by running 1000 same queries in loop, 1st when open and close are inside the loop and then when you move them out.
Usually a multi-threaded application should use connection pool. The size of the pool should be calculated.
EDIT: synchronizing writes to DB can be done by TRANSACTION. in sqlite you use BEGIN TRANSACTION and END TRANSACTION sqls (or just BEGIN & END). BEGIN can be use as mutex lock in a loop, END can be use as unlock. it can protect you from altering the DB from other process.
EDIT2: more solution is connection per thread.
EDIT3: You can also implement or use a message queue to write to the DB.
EDIT4:
I think separating read & write is not so good idea, because write should be in higher priority than read. the problem is that in sqlite you can't lock a single table, you lock the entire DB.
When I used sqlite I used a wrapper class with a single handle to the DB, all the read and write from/to the DB by high level functions, I had a write queue, and also kept track for each table if it had unwritten change pending, so for every read function I could test if I have the updated data or should wait.
I'm opening the sqlite database file with sqlite3_open and inserting data with a sqlite3_exec.
The file is a global log file and many users are writting to it.
Now I wonder, what happens if two different users with two different program instances try to insert data at the same time... Is the opening failing for the second user? Or the inserting?
What will happen in this case?
Is there a way to handle the problem, if this scenario is not working? Without a server side database?
In most cases yes. It uses file locking, but it is broken on some systems, see http://www.sqlite.org/faq.html#q5
In short, the lock is created when you start a transaction, and released immediately after. While locked, other instances can neither read nor write to the db (in "big" db, they can still read). However, you can connect sqlite in exclusive mode.
When you want to write to db, which is locked by another process, the execution halts for a specified timeout, by default 5 seconds. If lock is released, it proceeds with writing, if not it raises error.
I have a problem with an sqlite3 db which remains locked/unaccessible after a certain access.
Behaviour occurs so far on Ubuntu 10.4 and on custom (OpenEmbedded) Linux.
The sqlite version is 3.7.7.1). Db is a local file.
One C++-applications accesses the db periodically (5s). Each time several insert statements are done wrapped in a deferred transaction. This happens in one thread only. The connection to the db is held over the whole lifetime of the application. The statements used are also persistent and reused via sqlite3_reset. sqlite_threadsafe is set to 1 (serialized), journaling is set to WAL.
Then I open in parellel the sqlite db with the sqlite command line tool. I enter BEGIN IMMEDIATE;, wait >5s, and commit with END;.
after this the db access of the application fails: the BEGIN TRANSACTION returns return code 1 ("SQL error or missing database"). If I execute an ROLLBACK TRANSACTION right before the begin, just to be sure there is not already an active transaction, it fails with return code 5 ("The database file is locked").
Has anyone an idea how to approach this problem or has an idea what may cause it?
EDIT: There is a workaround: If the described error occures, I close and reopen the db connection. This fixes the problem, but I'm currently at a loss at to why this is so.
Sqlite is a server less database. As far as I know it does not support concurrent access from multiple source by design. You are trying to access the same backing file from both your application and the command tool - so you attempt to perform concurrent access. This is why it is failing.
SQLite connections should only be used from a single thread, as among other things they contain mutexes that are used to ensure correct concurrent access. (Be aware that SQLite also only ever supports a single updating thread at once anyway, and with no concurrent reads at the time; that's a limitation of being a server-less DB.)
Luckily, SQLite connections are relatively cheap when they're not doing anything and the cost of things like cached prepared statements is actually fairly small; open up as many as you need.
[EDIT]:
Moreover, this would explain closing and reopening the connection works: it builds the connection in the new thread (and finalizes all the locks etc. in the old one).