I'm learning SQLite and C++ within the Qt framework. As a learning project I am doing a simple image viewer which enables the user to tag images with keywords, categories, comments and ROI (for some later OpenCV functionality). It's a pretty simple database with some primary tables and some relational tables.
I think I've got the basics down and early tests show that my record data is being stored but in writing the methods to manage the database it looks like I am going to end up with a great many methods, particularly when I start to add in searching, sorting, deleting, etc.
This leads me to ask if I am going about this the correct way. I can see how some of my query logic (for search, sort) is probably better off being in a different "controller" type class and that all this manager class needs to do is handle the basic creation and deletion tasks, and just return a query object in response to a SELECT statement passed in as a string.
So, am I going about this in a reasonable way?
Manager methods so far:
bool openDatabase(QString name);
void closeDatabase();
bool createTables();
bool addKeyword(QString keyword);
bool addCategory(QString category);
bool deleteKeyword(QString keyword);
bool deleteCategory(QString category);
bool addROI(int x, int y, int width, int height);
bool deleteROI(int id);
bool addImage(QString name, QString path, QByteArray image, QByteArray thumb);
You probably should be using Qt's Model View Framework. The important classes there are QSqlQueryModel, QSqlTableModel and QSqlRelationalTableModel. To keep your UI isolated from the database structure, a reasonable approach would be to add view models for the particular use cases you have. You can then easily link those to, say, a Qt Quick based user interface.
There's nothing particularly wrong with function-oriented interface that you propose, except that it requires a lot of boring glue code to use it for user interfaces. It's best to factor such glue code as a proxy view model, since you're working to a documented API that can be then easily picked up by coworkers.
I have seen a DatabaseManager class somewhere and modified it. This is the insert function :
bool DatabaseManager::insert(const QString &tableName, const QString& columns, const QString &value)
{
bool ret = false;
if(db.isOpen()){
QSqlQuery query;
ret = query.exec( QString("INSERT INTO %1 (%2) VALUES(%3)").arg(tableName, columns, value) );
}
return ret;
}
You give it your table's name, the columns you want to fill and then the values. This is an example where I call it :
if( !dm.insert("Product", "ID, Name, Price, Notes, Category",
"'1', 'A DVD','10€', 'Some Notes', 'DVD'") )
{
//Note that each value is surrounded by an apostrophe
//Now whatever you want to
}
Related
From what I have seen, Qt documentation and majority of examples online assume that we are happy with (column, row)-based lookup in data(). But what If my table is based on a custom structure? For instance let's have:
struct MyDrive
{
QString serialNo;
QString user;
QString pc;
QString ipAddress;
QString category;
};
where serialNo is the key. So any operation from outside (imagine the model having implemented a listener) uses it for removing/modifying an item, making QMap as an ideal candidate.
But how to connect this structure with QModelIndex's data? QAbstractTableModel::data asks for data with (column,row) as key, making it more suitable for QVector<QVector>> or something similar (somewhere I read I should avoid using containers with non-constant access time (like map) in data()).
I can imagine using, well, a map with QModelIndex as key and serialNo as value, which would be used as key to my (serialNo-based) map but this looks very inefficient -- QModelIndex addresses concrete entry (serialNo, user, pc, ...) after all so we'd be duplicating the same item over and over again. I was also thinking about having a <serialNo, MyDrive*> map but this is just a workaround to an ugly design decision.
I can't believe I'm the first one with this scenario, so how is it usually solved?
You can use QAbstractItemModel::match to find items by serial
qt help
And enter all the necessary data into the table. This will allow you not to use containers, but how efficient is this a question...
Second solution is subclass AbstractItemModel Reference. Now you can do what you want and use any container by implementing data() function.
http://www.sqlite.org/inmemorydb.html
Mentions the ability to create a temporary database that will be stored in memory until a file is necessary (if at all). It will also remove that file when finished automatically. This is achieved by providing a blank database name "".
rc = sqlite3_open("", &db);
I'm trying to do this within a QT4 based application using the QSQLDatabase
QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");
db.setDatabaseName("");
bool ok = db.open();
Results in a failure to open the database.
I'm aware of the :memory: option and using it is much faster in our application for small datasets. I would prefer something that will drop back to a file when necessary since we may have some large datasets.
I am making the assumption that allowing the database engine to cache to file when necessary will be more efficient that just letting the OS page the memory in and out.
I'm open to alternatives with the following requirements:
Fast insert and lookup
No file once application is closed
Update
After going through some of the suggested SQLite performance suggestions I now have acceptable performance when using a file (TRANSACTIONS!).
I have not been able to figure out how to use sqlite3's built-in temporary file functionality.
I'm trying to use a QTemporaryFile, but for some reason they won't auto-delete the way the documentation implies they should. I have some more experimenting to do.
TL;DR - NO, you cannot give an empty string as a database name to sqlite3 using Qt. (see Edit 3).
Original answer
One possibility is to use the backup option in SQLite for an in-memory database.
But since it sounds like an optimization issue, you should read this SO answer which goes into a fair amount of detail about how you can speed up your database and disconnect it from the disk temporarily using PRAGMA synchronous = OFF and/or PRAGMA journal_mode = MEMORY. You might also want to limit the size of your in-memory pages by using the PRAGMA cache_size.
Edit 1: After reading your question again I realize now that you are asking to store overflow data to disk and want SQLite to just manage that so my first paragraph is not useful.
Edit 2: Added the PRAGMA cache_size suggestion.
Edit 3: Answering the meat of the question:
Your link to the other SO answer about SQLite optimization was very helpful, but still does not answer the meat of the question. ie how to use the sqlite3 built-in temp file functionality through the QT interface. Gavin S.
Ok, but the answer to that is no, and here's why.
If you take a look at the SQLite driver in Qt source and in particular the QSQliteDriver::open function it looks like this:
/*
SQLite dbs have no user name, passwords, hosts or ports.
just file names.
*/
bool QSQLiteDriver::open(const QString & db, const QString &, const QString &, const QString &, int, const QString &conOpts)
{
if (isOpen())
close();
-> if (db.isEmpty())
-> return false;
bool sharedCache = false;
int openMode = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, timeOut=5000;
QStringList opts=QString(conOpts).remove(QLatin1Char(' ')).split(QLatin1Char(';'));
foreach(const QString &option, opts) {
if (option.startsWith(QLatin1String("QSQLITE_BUSY_TIMEOUT="))) {
bool ok;
int nt = option.mid(21).toInt(&ok);
if (ok)
timeOut = nt;
}
if (option == QLatin1String("QSQLITE_OPEN_READONLY"))
openMode = SQLITE_OPEN_READONLY;
if (option == QLatin1String("QSQLITE_ENABLE_SHARED_CACHE"))
sharedCache = true;
}
sqlite3_enable_shared_cache(sharedCache);
-> if (sqlite3_open_v2(db.toUtf8().constData(), &d->access, openMode, NULL) == SQLITE_OK) {
sqlite3_busy_timeout(d->access, timeOut);
setOpen(true);
setOpenError(false);
return true;
} else {
setLastError(qMakeError(d->access, tr("Error opening database"),
QSqlError::ConnectionError));
setOpenError(true);
return false;
}
}
The function sqlite3_open you are trying to call will never be called with an empty string as an argument because of the early return condition on lines 530 and 531 unless you patch the Qt driver on that particular line of code.
The question is more a Best way of doing question than a real problem.
Context and what I'm doing
I have an application that must access to the database. Therefore I have a class DatabaseManager that is called every time I need to access the database and which does for instance:
DatabaseManager *db = new DatabaseManager;
std::vector<Element> elementVector = db->getAElementsById(id);
And you the same for all insert, update or remove requests.
What eventually happened...
...is that my file has already 1200 lines and is growing with every new feature.
The question
So how do pretty people do ?
One header several splitted files? (This told that it's not a good idea)
Splitting the class in several class? The databaseManager is one class, it would be a nonsense...
Is there any other option? (I'm not familiar with the notion of namespace, maybe is it a way?)
I read also other post but didn't find either the good question to ask google, or people answering my question.
I have my database module split into more than one file. For example, one file handles insertions, another file handles extractions and a third handles searches.
Mine is actually more complex, but the practice should be to split up the code into separate themed files, preferably no more than 300 LOC (Lines Of Code).
The solution I chose for now is the following (because I was too lazy to reconstruct new classes and so on) : I splitted the class in several cpp files
// DatabaseManager.h
class DatabaseManager
{
//// Getters - in DatabaseManager_getters.cpp
public:
// Movies
Movie getOneMovieById(const int id);
QList<Movie> getAllMovies(const QString fieldOrder = "title");
....
private:
// Other functions for getters
Movie hydrateMovie(QSqlQuery &query);
//// Inserts - in DatabaseManager_insert.cpp
public:
bool insertNewMovie(Movie &movie);
....
private:
bool insertNewTag(Tag &tag);
....
//// Updates - in DatabaseManager_update.cpp
public:
bool updateMovie(Movie &movie);
....
//// Delete - in DatabaseManager_delete.cpp
public:
bool deleteMovie(Movie &movie);
private:
bool deletePeople(const People &people);
};
And obviously the .cpp files are following what announced. It's easier for me to find where a function is and to maintain the code.
Another time, I'll think twice if it's not more relevant to follow the advices you gave.
Thx again !
In our current project, we need some high level DBI for different databases. It should provide the following features:
in memory cache - the DBI should be able to cache all reads, and update the cache on writing calls (the application we are coding on is heavy threaded, and needs fast access to the current data all the time). The memory cache will be based on boost::multi_index
automatic sql building - We don't want to parse a sql statement to lookup in the memory cache
As we need to provide functions for: defining a table layout, do selects, do inserts, do updates, joins, ..., the interface will get very complex.
We need a good way to invoke the interface function.
There are many styles around, but we could not find any useful for our usage.
Here a few examples:
SOCI
sql << "select name, salary from persons where id = " << id, into(name), into(salary);
We don't want some SQL statements, so we would have to define what and from a different way.
pqxx
Conn.prepare("select_salary",
"select name, salary from persons where id = $1")
((string)"integer",prepare::treat_direct);
The heavy usage of the overloaded operator() is just ugly, but it could work for us too.
Any suggestions how to design the interface?
How about using object relational mapping? Here's some code fragment ideas off the top of my head - I've only done this in Python, never in C++, and only for fairly simple databases. There's a list of frameworks on Wikipedia that should avoid too much wheel-related R&D.
class people: public dbi_table
{
// id column handled by dbi_table.
name: string_column;
salary: money_column;
};
class cost_center: public dbi_table
{
name: string_column;
office: foreign_key<offices>;
};
class people_cost_center_link: public link_table
{
// Many-many relationships.
};
Then you can manipulate records as objects, all the relational stuff is handled by the framework. Querying is done by defining a query object and then getting an iterator to the results (see the ODB wikipedia page for a code example).
I would do it like this (and it'd be good in c++ point of view, dunno if it's correct database stuff):
struct Handle { int id; }
class DBI
{
public:
virtual Handle select(int column_id)=0;
virtual Handle select(int column1, int column2)=0;
virtual Handle id(int id)=0;
virtual Handle join(Handle i1, Handle i2)=0;
virtual void execute_query(Handle i)=0;
};
Usually these functions would be implemented like this:
Handle select(int column_id) {
return new_handle(new SelectNode(column_id));
}
where new_handle function would just insert SelectNode to std::vector or std::map and create an handle for it.
First a little back story on what I am trying to accomplish.
I am in the process of creating a custom HTTP Module whose purposes is to intercept messages to multiple (15+) different ArcGIS REST web services. The intercepted requests and/or responses will be stripped of any restricted information based on the current user.
For instance, a call that returns multiple layers might have certain layers stripped out.
Unmodified Response:
"layers" : [
{
"id" : 0,
"name" : "Facilities",
"parentLayerId" : -1,
"defaultVisibility" : true,
"subLayerIds" : [1, 2, 3]
},
{
"id" : 1,
"name" : "Hazardous Sites",
"parentLayerId" : 0,
"defaultVisibility" : true,
"subLayerIds" : null
},
]
Modified Response:
"layers" : [
{
"id" : 0,
"name" : "Facilities",
"parentLayerId" : -1,
"defaultVisibility" : true,
"subLayerIds" : [1, 2, 3]
}
]
There are numerous services available, all uniquely identified via a URL. Each service returns very different information and so needs to be filtered different. Additionally, each service may return the data in a variety of formats (HTML, JSON, etc).
As such, I will need to create a multitude of different filters to apply to HttpRequest.Filters and/or HttpResponse.Filters.
Example:
// Request for layers and the format is JSON
IPolicy policy = GetPolicy(userContext);
Filter filter = new LayerJsonResponseFilter(Response.Filter, policy);
Response.Filter = filter;
Request and response filters are implemented by inheriting from Stream (or another class that inherits from Stream such as MemoryStream). I want to be able to easily create new filters without reimplementing Stream for each filter.
A potential solution is described in here: http://www.west-wind.com/weblog/posts/72596.aspx
However, I want to simplify the solution without losing the flexibility of specifying many different transformations without reimplementing the stream. I think that I can accomplish this by:
Inherit from MemoryStream so as to reduce the reimplementation of methods.
Always operate on full content, rather than chunked content.
Replace the events with an abstract method (e.g., Filter())
I have considered two potential solutions.
Solution 1: Create Multiple Filters Inheriting from ResponseFilter
In this scenario each filter contains the logic for performing the filtration. There would be 15+ filters created all inheriting from a common ResponseFilter abstract base class like so:
// All filters will inherit from ResponseFilter
public abstract class ResponseFilter : MemoryStream
{
public ResponseFilter(Stream stream, Policy policy) { }
// Must be overridden in a derived class with specific Filter logic.
public abstract string Filter(string content);
// Overridden to cache content.
public override void Write(byte[] buffer, int offset, int count) { }
// Overridden to perform the filter/transformation before the content is written.
public override void Flush()
{
// Get stream content as a string
string content = Filter(content);
// Write new content to stream
}
}
This would be used in the following way.
// Example
var policy = GetPolicy();
var filter = new MapServiceJsonResponseFilter(response.Filter, policy);
response.Filter = filter;
The advantage to this option is that the number of classes is kept to a minimum. However, it becomes difficult to reuse any filter logic anywhere else in the application should it become necessary. Additionally, unit testing the filters would require mocking the Stream, another disadvantage.
Solution 2: Create Multiple Filters, Inject into a Common ResponseFilter
In this scenario, a single response filter is created. The actual filter logic or algorithm is injected into the filter. All filters inherit from an abstract base class FilterBase.
// Represents an HttpResponse Filter. Renamed to avoid confusion with
// the filter algorithm.
public class ResponseFilterStream : MemoryStream
{
public ResponseFilterStream(Stream stream, FilterBase filter) { }
// Overridden to cache content.
public override void Write(byte[] buffer, int offset, int count) { }
// Overridden to perform the filter/transformation before the content is written.
public override void Flush()
{
// Get stream content as a string
string content = _filter.Filter(content);
// Write new content to stream
}
}
// All filter algorithms inherit from FilterBase and must implement
// the filter method.
public abstract class FilterBase
{
protected TransformBase(Policy policy) { }
// Overridden to perform the filter/transformation.
public abstract string Filter(string content);
}
This would be used in the following way.
// Example
var policy = GetPolicy();
var filter = new MapServiceJsonResponseFilter(policy);
ResponseFilter responseFilter = new ResponseFilter(response.Filter, filter);
response.Filter = filter;
The advantage to this solution is that the filtration logic is completely independent of any classes that implement stream. The logic can be more easily reused if necessary. Unit testing is a little simpler as well as I do not need to mock the stream.
However, there are more classes (exactly 1) and the usage is a little more complex, though not terribly so.
Note: I'll probably want to rename FilterBase as to avoid confusion with ResponseFilter. Perhaps TransformBase.
I have several goals that I want to meet with either solution.
The solution must be highly testable. Unit testing will be used to check for correctness of the filters. It is imperative that testing is as simple as possible.
The solution must easily support the creation of multiple filters (15+).
The solution should be readable (i.e., easy to maintain).
I think that solution 2 is the best solution for this given scenario. I can test the filtration logic completely independently of Stream with minimal additional complexity. Either solution will support #2 and #3, so testing gets the edge.
What other considerations might there be? Are there better alternatives?
Solution 2 is obviously preferable. However, it seems that the major crux of the problem lies in the construction of the Filters themselves. Hopefully there is a lot of reusable composition within the Filter implementations. Can a new filter be "configured" from composite parts?