I'm writing a desktop application to do vector drawing in C++, and considering using sqlite to back my undo/redo feature.
Has anybody used sqlite for undo/redo features? How does it work out for you?
Clarification:
I was aware of the stack approach, I have even implemented one application with that approach. The problem I encountered was that it becomes hard to maintain after a while.
What I meant by utilizing sqlite is that I will map my entire in-memory data structure into a sqlite database, and let the sqlite do the diff and revision for me.
Speed should not be a issue if I create a in-memory database.
That was the idea and I was wondering if that could work.
It makes sense to use SQLite to back undo/redo when an SQLite database is the application's data file format. See the SQLite website for an explanation of how to do this with SQLite triggers.
Basically a undo/redo feature can be implemented using a stack: when the user does an operation, you push on the stack an object that represents the delta between the state before and after the operation, and when you undo, you "unroll" the delta. Because every operation the user does creates a new delta object on the stack, it might by that sqlite is not your choice of technology because it might be too slow. I would recommend considering the possibility of just storing the undo/redo information in memory, and linearizing it on the disk only if you want to actually save the undo/redo history.
Take a look at Memento Design Pattern.
Here is what worked for me in sqlite3
BEGIN;
-- enter your INSERT, UPDATE or DELETE command
-- if you want to keep it then run command below
COMMIT;
-- to undo - run command below
ROLLBACK;
Source SQLite Transactions (Begin, Commit, Rollback)
Related
I am musing about the possibility of switching from a C++ runtime relational database (std::maps for indexes, std::vectors for columns, etc) to using an in-memory sqlite database. This would mainly be for code maintenance. (Going back and looking at my code months later took my some time to re-acquaint myself with the schema I designed using C++ components; using SQL statements would at least be more self-documenting.)
Here's the hurdle, though: The data cannot touch secondary storage without being encrypted. My current solution lends itself quite well to encryption before flushing. Sqlite, however, appears to be its own opaque, self-contained ecosystem that isn't very conducive to manipulating the underlying database binary data as anything but a database.
So, I'm curious if anybody knows of a way, or has developed one, to put an encrypting hook in the sqlite backup system such that: memory -> encryption -> disc, and then in reverse. Alternatively, I would also be fine just grabbing the in-memory database as a binary "blob" and do my own thing to it before I write it out myself (and reverse the process when loading it back in). I don't see any such opportunities explicitly available in the sqlite backup API.
Note: I am not interested in using the SEE stuff, only the "public domain" sqlite offering. I have sufficient cryptographic code in use already (via Qt) within my application, so I don't want competing systems.
Thanks for any suggestions.
My C++ app is using RocksDB to store in-memory key-value sets.
At some points, I want my app to be able to keep the DB values until its next run. Meaning, the program will shut down, start again and read the same values from the DB as it had before it shut down.
What would be the quickest and simplest way to achieve this?
I found the following article for backup & restore routine - https://github.com/facebook/rocksdb/wiki/How-to-backup-RocksDB%3F, but maybe its an overkill?
Adding to what yinqiwen said, RocksDB was not meant to be just an in memory data store. It works really well with a variety of storage types. And it is especially good in terms of performance when it comes to flash storage. You may use a variety of RocksDB Options to experiment with what configuration is best to use for your workload, but for most cases, even with the default settings for the persistent storage types, rocks db should work just fine.
rocksdb already provide some ways to persist in-memory RocksDB database. u can see this link to conigure your rocksdb. http://rocksdb.org/blog/245/how-to-persist-in-memory-rocksdb-database/
I'm developing optimization algorithms which operate on data stored in a postgres django database. My algorithms have to repeatedly modify the objects in the database and sometimes revert the change done (it is metaheuristic algorithms, for those who knows).
The problem is that I don't want to save the modification on the postgres database during the process. I would like to save the modifications at the end of the process, when i'm satisfied with the results of the optimization. I think that the solution is to load all concerned objects in memory, work on them, and save the objects in memory to the database at the end.
However it seems to be more difficult than I thought...
Indeed, when I will make a django query (ie. model1.objects.get or model.objects.filter), I fear that django call the objects sometimes in database and sometimes in it's cache, but I'm pretty sure that in some case it will not be the same than the instances I manually loaded in memory (which are the ones on which I want to work because they may have changed since the load from the database) ...
Is there a way to bypass such problems ?
I implemented a kind of custom mini-database which works but it's becoming too difficult to maintain and over all, I think it's not the most simple and elegant way to proceed. I thought to dump the concerned model of the postgres database into an in-memory one (for performance), work on this in-memory db and when finishing my algorithm, update the data of the original database from the data in the in-memory one (it would imply that django keeps a link, perhaps through the pk, of the original objects with those in the in-memory database to identify which are the same and I don't know if it's possible).
Does someone has an insight?
Thank you in advance.
What you are looking for is transactions. One of the most powerfull features of an RDBS. Simply use START TRANSACTION before you start playing around with the data. At the end if you are happy with it use COMMIT. If you don't want your django app to see the changes use ROLLBACK.
Due to the default transaction isolation level of postgresql, your django app will not see whatever changes you are doing elsewhere until it's committed. At the same time what ever changes you do in your sql console or with other code will be visible to that code even though it's not committed.
Read Committed is the default isolation level in PostgreSQL. When a
transaction uses this isolation level, a SELECT query (without a FOR
UPDATE/SHARE clause) sees only data committed before the query began;
it never sees either uncommitted data or changes committed during
query execution by concurrent transactions. In effect, a SELECT query
sees a snapshot of the database as of the instant the query begins to
run. However, SELECT does see the effects of previous updates executed
within its own transaction, even though they are not yet committed
I am looking for a database library that can be used within an editor to replace a custom document format. In my case the document would contain a functional program.
I want application data to be persistent even while editing, so that when the program crashes, no data is lost. I know that all databases offer that.
On top of that, I want to access and edit the document from multiple threads, processes, possibly even multiple computers.
Format: a simple key/value database would totally suffice. SQL usually needs to be wrapped, and if I can avoid pulling in a heavy ORM dependency, that would be splendid.
Revisions: I want to be able to roll back changes up to the first change to the document that has ever been made, not only in one session, but also between sessions/program runs.
I need notifications: each process must be able to be notified of changes to the document so it can update its view accordingly.
I see these requirements as rather basic, a foundation to solve the usual tough problems of an editing application: undo/redo, multiple views on the same data. Thus, the database system should be lightweight and undemanding.
Thank you for your insights in advance :)
Berkeley DB is an undemanding, light-weight key-value database that supports locking and transactions. There are bindings for it in a lot of programming languages, including C++ and python. You'll have to implement revisions and notifications yourself, but that's actually not all that difficult.
It might be a bit more power than what you ask for, but You should definitely look at CouchDB.
It is a document database with "document" being defined as a JSON record.
It stores all the changes to the documents as revisions, so you instantly get revisions.
It has powerful javascript based view engine to aggregate all the data you need from the database.
All the commits to the database are written to the end of the repository file and the writes are atomic, meaning that unsuccessful writes do not corrupt the database.
Another nice bonus You'll get is easy and flexible replication and of your database.
See the full feature list on their homepage
On the minus side (depending on Your point of view) is the fact that it is written in Erlang and (as far as I know) runs as an external process...
I don't know anything about notifications though - it seems that if you are working with replicated databases, the changes are instantly replicated/synchronized between databases. Other than that I suppose you should be able to roll your own notification schema...
Check out ZODB. It doesn't have notifications built in, so you would need a messaging system there (since you may use separate computers). But it has transactions, you can roll back forever (unless you pack the database, which removes earlier revisions), you can access it directly as an integrated part of the application, or it can run as client/server (with multiple clients of course), you can have automatic persistency, there is no ORM, etc.
It's pretty much Python-only though (it's based on Pickles).
http://en.wikipedia.org/wiki/Zope_Object_Database
http://pypi.python.org/pypi/ZODB3
http://wiki.zope.org/ZODB/guide/index.html
http://wiki.zope.org/ZODB/Documentation
On an official sqlite3 web page there is written that I should think about sqlite as a replacement of fopen() function.
What do you think about it? Is it always good solution to replece application internal data storage with sqlite? What are the pluses and the minuses of such solution?
Do you have some experience in it?
EDIT:
How about your experience? Is it easy to use? Was it painful or rather joyful? Do you like it?
It depends. There are some contra-indications:
for configuration files, use of plain text or XML is much easier to debug or to alter than using a relational database, even one as lightweight as SQLite.
tree structures are easier to describe using (for example) XML than by using relational tables
the SQLite API is quite badly documented - there are not enough examples, and the hyperlinking is poor. OTOH, the information is all there if you care to dig for it.
use of app-specific binary formats directly will be faster than storing same format as a BLOB in a database
database corruption can mean the los of all your data rather than that in a single bad file
OTOH, if your internal data fits in well with the relational model and if there is a a lot of it, I'd recommend SQLite - I use it myself for one of my projects.
Regarding experience - I use it, it works well and is easy to integrate with existing code. If the documentation were easier to navigate I'd give it 5 stars - as it is I'd give it four.
As always it depends, there are no "one size fits all" solutions
If you need to store data in a stand-alone file and you can take advantage of relational database capabilities of an SQL database than SQLite is great.
If your data is not a good fit for a relational model (hierarchical data for example) or you want your data to be humanly readable (config files) or you need to interoperate with another system than SQLite won't be very helpful and XML might be better.
If on the other hand you need to access the data from multiple programs or computers at the same time than again SQLite is not an optimal choice and you need a "real" database server (MS SQL, Oracle, MySQL, PosgreSQL ...).
The atomicity of SQLite is a plus. Knowing that if you half-way write some data(maybe crash in the middle), that it won't corrupt your data file. I normally accomplish something similar with xml config files by backing up the file on a successful load, and any future failed load(indicating corruption) automatically restores the last backup. Of course it's not as granular nor is it atomic, but it is sufficient for my desires.
I find SQLite a pleasure to work with, but I would not consider it a one-size-fits-all replacement for fopen().
As an example, I just wrote a piece of software that's downloading images from a web server and caching them locally. Storing them as individual files, I can watch them in Windows Explorer, which certainly has benefits. But I need to keep an index that maps between a URL and the image file in order to use the cache.
Storing them in a SQLite database, they all sit in one neat little file, and I can access them by URL (select imgdata from cache where url='http://foo.bar.jpg') with little effort.