I am embedding Monetdbe into a multi-threaded C++ application.
I have several threads running on the server-side of my application and each thread opens its own instance of the same Monetdb database, i.e. each thread runs the following code:
monetdbe_database db = NULL;
if (monetdbe_open(&db, url /* inmemory database */, NULL /* no options */)) {
fprintf(stderr, "Failed to open database\n");
return -1;
}
Each thread runs MonetDB queries sent by clients connected to the server. therefore, there can be several clients connected to the db at the same time, and they may send requests to access/update the same underlying tables at the same time.
I just want to make sure that Monetdb has been designed to deal with this scenario.
I understand that Monetdb is not designed to be a high transaction db, and my use case is more analytical, but I do have several clients that will be connected to the server, and sometimes they may run queries against the same db tables at the same time. Is this the correct way to run multi-threaded applications with Monetdbe?
According to this issue on github (https://github.com/MonetDBSolutions/monetdbe-examples/issues/11) and this example (https://github.com/MonetDBSolutions/monetdbe-examples/blob/master/C/concurrent.c).
So long as you aren't using the same monetdbe_database for each connection, you should be okay to go.
Related
I need to create a server in Qt C++ with QTcpServer which can handle so many requests at the same time. nearly more than 1000 connections and all these connection will constantly need to use database which is MariaDB.
Before it can be deployed on main servers, It needs be able to handle 1000 connections with each connection Querying data as fast it can on 4 core 1 Ghz CPU with 2GB RAM Ubuntu virtual machine running on cloud. MySQL database is hosted on some other server which more powerful
So how can I implement this ? after googling around, I've come up with following options
1. Create A new QThread for each SQL Query
2. Use QThreadPool for new SQL Query
For the fist one, it might will create so many Threads and it might slow down system cause of so many context switches.
For second one,after pool becomes full, Other connections have to wait while MariaDB is doing its work. So what is the best strategy ?
Sorry for bad english.
1) Exclude.
2) Exclude.
3) Here first always doing work qt. Yes, connections (tasks for connections) have to wait for available threads, but you easy can add 10000 tasks to qt threadpool. If you want, configure max number of threads in pool, timeouts for tasks and other. Ofcourse your must sync shared data of different threads with semaphore/futex/mutex and/or atomics.
Mysql (maria) it's server, and this server can accept many connections same time. This behaviour equally what you want for your qt application. And mysql it's just backend with data for your application.
So your application it's server. For simple, you must listen socket for new connections and save this clients connections to vector/array and work with each client connection. Always when you need something (get data from mysql backend for client (yeah, with new, separated for each client, onced lazy connection to mysql), read/write data from/to client, close connection, etc.) - you create new task and add this task to threadpool.
This is very simple explanation but hope i'm helped you.
Consider for my.cnf [mysqld] section
thread_handling=pool-of-threads
Good luck.
I have an API which opens an access database for read and write. The API opens the connection when it's constructed and closes the connection when it's destructed. When the db is opened an .ldb file is created and when it closes it's removed (or disappears).
There are multiple applications using the API to read and write to the access db. I want to know:
Is ldb file used to track multiple connections
Does calling an db.close() closes all connections or just one instance.
Will there be any sync issues with the above approach.
db.Close() closes one connecton. The .ldb is automatically removed when all connections are closed.
Keep in mind that while Jet databases (i.e. Access) do support mutiple simultaneous users, they're not extremely well-suited for a very large concurrent user base; for one thing, they are easily corrupted when there are network issues. I'm actually dealing with that right now. If it comes to that, you will want to use a database server.
That said, I've used Jet databases in that way many times.
Not sure what you mean when you say "sync issues".
Yes, it's required to open database in shared mode by multiple users. Seems it stands for "Lock Database". See more info in MSDN: Introduction to .ldb files in Access 2000.
Close() closes only one connection, others are unaffected.
Yes, it's possible if you try to write records that another user has locked. However data will remain consistent, you will just receive error about write conflict.
Actually MS Access is not best solution for multi-connection usage scenario.
You may take a look at SQL Server Compact which is light version of MS SQL Server. It runs in-process, supports multiple connections and multithreading, most of robust T-SQL features (excluding stored procs) etc.
As an additional note to otherwise good answers, I would strongly recommend keeping a connection to a dummy table open for the lifetime of the client application.
Closing connections too often and allowing the lock file to be created/deleted every time is a huge performance bottleneck and, in some cases of rapid access to the database, can actually cause queries and inserts to fail.
You can read a bit more in this answer I gave a while ago.
When it comes to performance and reliability, you can get quite a lot out of Access databases providing that you keep some things in mind:
Keep a connection open to a dummy table for the duration of the life of the client (or at least use some timeout that would close the connection after like 20 seconds of inactivity if you don't want to keep it open all the time).
Engineer your clients apps to properly close all connections (including the dummy one when i'ts time to do it), whatever happens (eg crash, user shutdown, etc).
Leaving locks in place is not good, as it could mean that the client has left the database in an unknown state, and could increase the likelihood of corruption if other clients keep leaving stale locks.
Compact and repair the database regularly. Make it a nightly task.
This will ensure that the database is optimised, and that any stale data is removed and open locks properly closed.
Good, stable network connectivity is paramount to data integrity for a file-based database: avoid WiFi like the plague.
Have a way to kick out all clients from the database server itself.
For instance, have a table with for instance a MaintenanceLock field that clients poll regularly. If the field is set, the client should disconnect, after giving an opportunity for the user to save his work.
Similarly, when a client app starts, check this field in the database to allow or disallow the client to connect to it.
Now, you can quick out clients at any time without having to go to each user and ask them to close the app. It's also very useful to ensure that no client left open at night are still connected to the database when you run Compact & Repair maintenance on it.
I have an application interacts with Access database using DAO class, recently I converted the database to a sqlite database.
I do not know which connection method is better for the design as following:
Create only one database connection using a public variable when open the application, any queries use the only connection object for interaction during the run time, the connection is then closed when close the application
Create database connection every time before running a query, then close the database connection instantly after loading the resultset to the memory.
I recommend that you encapsulate your db access, so that the decision on whether to keep a persistent connection or not open can be changed at a later point.
Since you are using SqlLite I am assuming that it is a single user DB, so concurrency , connection contention, locking etc. are not likely to be issues.
Typically the main reasons to reuse short running connections is usually on a multi user web or service oriented system, where scalability and licensing considerations are important. This doesn't seem to be applicable in your case.
.
In short, there doesn't seem any reason not to keep a connection open for the entire duration of your app / user's login session based on the above assumptions.
If you use transactions however, I would suggest that you commit these after each successful atomic activity
You know your two options have + and -. For your special case I think to create database connection every time is not so bad idea, because creating connection to sqlite is very fast and no time consuming. Also this way you may create/close more than one connection at once, which is a good benefit, maybe you don't do it now, but in the future maybe you will have to.
I have a server that communicates to a lot of devices (>1000). Each connection has its own thread. Now, I realized that I would have to set my mysql config to allow >1000 open concurrent connections what seems to be a very bad idea in my opinion.
Qt docs say that every thread needs its own connection: http://qt-project.org/doc/qt-4.8/threads-modules.html#threads-and-the-sql-module
So, I have to call
QSqlDatabase::addDatabase("QMYSQL", "thread specific string");
in every thread.
What is the best practice here?
I would think some sort of resource pooling would be appropriate here.
Depending on the database workload from the >1000 device threads a single database thread could maybe manage it or then you will need several database threads.
Then setup a queuing system from the device threads to the database thread(s), where the devices push the work and the database thread(s) pulls work units off and perform the query.
I just realised that I was thinking of writing to database only like some sort of logging, and this idea may not work without modification if what you are doing is reading from the database and writing to devices.
Honestly speaking I don't know much about QT but if I take your problem in general then I would advise you to create a "Connection Pool"
If you don't want or can't implement a Connection Pool then its fine to increase Max_Connections in MySQL configuration and leave the pooling on MySQL, it has its own Connection Pooling mechanism.
I'm building an application which uses Mysql, I was wondering what would be the best way to manage the connection to the actual Mysql server?
I'm still in the design phase, but currently I have it Connecting (or aborting if error) before every query and disconnecting after which is just for testing as right now I'm only running 1 query to see if the code I've setup so far works.
My App might be performing a few queries every 5/10/20/30 minutes depending on settings and doesn't really need to do anything with SQL until that time.
So I'm wondering if its more beneficial to use a continuous connection that exists for the lifetime of the application (if possible) or to simply connect to sql before I intend to use it, do what the app needs to do then disconnect?
Connecting once and performing many queries will naturally be more efficient.
However, if performance isn't a major concern for your project, maybe aiming for simplicity in your code might be a better option (especially if you are the only connection to the database).
If you want to get clever, then maybe connect as and when you need to, then keep the connection alive until you stop making queries. Eg, drop the connection if there have been no queries for 30 seconds or something like that.
How many instances of this app will be connecting to MySQL? If it's just one, keeping a MySQL connection open for convenience shouldn't cause any problems, but remember there's a (configurable) limit to the number of MySQL connections you can have open to the server. In this case, I would recommend opening a connection, running whatever queries you need to run, and then closing it. Connecting per query adds more overhead as you add queries to your application.