I have an API which opens an access database for read and write. The API opens the connection when it's constructed and closes the connection when it's destructed. When the db is opened an .ldb file is created and when it closes it's removed (or disappears).
There are multiple applications using the API to read and write to the access db. I want to know:
Is ldb file used to track multiple connections
Does calling an db.close() closes all connections or just one instance.
Will there be any sync issues with the above approach.
db.Close() closes one connecton. The .ldb is automatically removed when all connections are closed.
Keep in mind that while Jet databases (i.e. Access) do support mutiple simultaneous users, they're not extremely well-suited for a very large concurrent user base; for one thing, they are easily corrupted when there are network issues. I'm actually dealing with that right now. If it comes to that, you will want to use a database server.
That said, I've used Jet databases in that way many times.
Not sure what you mean when you say "sync issues".
Yes, it's required to open database in shared mode by multiple users. Seems it stands for "Lock Database". See more info in MSDN: Introduction to .ldb files in Access 2000.
Close() closes only one connection, others are unaffected.
Yes, it's possible if you try to write records that another user has locked. However data will remain consistent, you will just receive error about write conflict.
Actually MS Access is not best solution for multi-connection usage scenario.
You may take a look at SQL Server Compact which is light version of MS SQL Server. It runs in-process, supports multiple connections and multithreading, most of robust T-SQL features (excluding stored procs) etc.
As an additional note to otherwise good answers, I would strongly recommend keeping a connection to a dummy table open for the lifetime of the client application.
Closing connections too often and allowing the lock file to be created/deleted every time is a huge performance bottleneck and, in some cases of rapid access to the database, can actually cause queries and inserts to fail.
You can read a bit more in this answer I gave a while ago.
When it comes to performance and reliability, you can get quite a lot out of Access databases providing that you keep some things in mind:
Keep a connection open to a dummy table for the duration of the life of the client (or at least use some timeout that would close the connection after like 20 seconds of inactivity if you don't want to keep it open all the time).
Engineer your clients apps to properly close all connections (including the dummy one when i'ts time to do it), whatever happens (eg crash, user shutdown, etc).
Leaving locks in place is not good, as it could mean that the client has left the database in an unknown state, and could increase the likelihood of corruption if other clients keep leaving stale locks.
Compact and repair the database regularly. Make it a nightly task.
This will ensure that the database is optimised, and that any stale data is removed and open locks properly closed.
Good, stable network connectivity is paramount to data integrity for a file-based database: avoid WiFi like the plague.
Have a way to kick out all clients from the database server itself.
For instance, have a table with for instance a MaintenanceLock field that clients poll regularly. If the field is set, the client should disconnect, after giving an opportunity for the user to save his work.
Similarly, when a client app starts, check this field in the database to allow or disallow the client to connect to it.
Now, you can quick out clients at any time without having to go to each user and ask them to close the app. It's also very useful to ensure that no client left open at night are still connected to the database when you run Compact & Repair maintenance on it.
Related
I'm trying to modify a game engine so it records events (like key presses), and store these in a MySQL database on a remote server. The game engine is written in C++, and I currently have the following straightforward architecture, using mysql++ to directly INSERTrecords into appropriate databases:
Unfortunately, there's a very large overhead when connecting to the MySQL server, and the game stops for a significant amount of time. Pushing a batch of Xs worth of events to the server causes a significant delay in gameplay (60s worth of events can take 12s to synchronise). There are also apparently security concerns with leaving the MySQL port accessible publicly.
I was considering an alternative option, instead sending commands to the server, which can interact with the database in its own time:
Here the game would only send the necessary information (e.g. the table to update and the data to insert). I'm not sure whether the speed increase would be sufficient, or what system would be appropriate for managing the commands sent from the game.
Someone else suggested Log4j, but obviously I need a C++ solution. Is there an appropriate existing framework for accomplishing what I want?
Most applications gathering user-interface interaction data (in your case keystrokes) put it into a local file of some sort.
Then at an appropriate time (for example at the end of the game, or the beginning of another game), they POST that file, often in compressed form, to a publicly accessible web server. The software on the web server decompresses the data and loads it into the analytics system (the MySQL server in your case) for processing.
So, I suggest the following.
stop making your MySQL server's port available to people you don't know and trust.
get your game to gather keystrokes locally somehow.
get it to upload that data in big bunches when your game is not in realtime mode.
write a web service to receive and interpret these files.
That way you'll build a more secure analytics system and a more responsive game.
The question is a little general, so to help narrow the focus, I'll share my current setup that is motivating this question. I have a LAMP web service running a RESTful API. We have two client implementations: one browser-based javascript client (local storage store) and one iOS-based client (core data store). Obviously these two clients store data very differently, but the data itself needs to be kept in two-way sync with the remote server as often as possible.
Currently, our "sync" process is a little dumb (as in, non-smart). Conceptually, it looks like:
Client periodically asks the server for ALL of the most-recent data.
Server sends down the remote data, which overwrites the current set of local data in the client's store.
Any local creates/updates/deletes after this point are treated as gold, and immediately sent to the server.
The data itself is stored relationally, and updated occasionally by client users. The clients in my specific case don't care too much about the relationships themselves (which is why we can get away with local storage in the browser client for now).
Obviously this isn't true synchronization. I want to move to a system where, conceptually, a "diff" of the most recent changes are sent to the server periodically, and the server sends back a "diff" of the most recent changes it knows about. It seems very difficult to get to this point, but maybe I just don't understand the problem very well.
REST feels like a good start, but REST only talks about the way two data stores talk to each other, not how the data itself is synchronized between them. (This sync process is left up to the implementer of each store.) What is the best way to implement this process? Is there a modern set of programming design patterns that apply to inform a specific solution to this problem? I'm mostly interested in a general (technology agnostic) approach if possible... but specific frameworks would be useful to look at too, if they exist.
Multi-master replication is always (and will always be) difficult and bespoke, because how conflicts are handled will be specific to your application.
IMO A more robust approach is to use Master-slave replication, with your web service as the master and the clients as slaves. To keep the clients in sync, use an archived atom feed of the changes (see event sourcing) as per RFC5005. This is the closest you'll get to a modern standard for this type of replication and it's RESTful.
When the clients are online, they do not update their replica directly, instead they send commands to the server and have their replica updated via the atom feed.
When the clients are offline things get difficult. Your clients will need to have a model of how your web service behaves. It will need to have an offline copy of your replica, which should be copied on write from the online replica (the online replica is the one that is updated by the atom feed). When the client executes commands that modify the data, it should store the command (for later replay against the web service), the expected result (for verification during replay) and update the offline replica.
When the client goes back online, it should replay the commands, compare the result with the expected result and notify the client of any variances. How these variances are handled will vary based on your application. The offline replica can then be discarded.
CouchDB replication works over HTTP and does what you are looking to do. Once databases are synced on either end it will send diffs for adds/updates/deletes.
Couch can do this with other Couch machines or with a mobile framework like TouchDB.
https://github.com/couchbaselabs/TouchDB-iOS
I've done a fair amount of it, but you can always set up CouchDB on one machine, set up TouchDB on a mobile device and then watch the HTTP traffic go back and forth to get an idea of how they do it.
Or read this: http://guide.couchdb.org/draft/replication.html
Maybe something from the link above will help you get an idea of how to do your own diffs for your REST service. (Since they are both over HTTP thought it could be useful.)
You may want to look into the Dropbox Datastore API:
https://www.dropbox.com/developers/datastore
It sounds like it might be a very good fit for your purposes. They have iOS and javascript clients.
Lately, I've been interested in Meteor.
The platform sets up Mongo on the server and minimongo in the browser. The client subscribes to some data and when that data changes, the platform automatically sends down the new data to the client.
It's a clever solution to the syncing problem, and it solves several other problems as well. It will be interesting to see if more platforms do this in the future.
I have an application interacts with Access database using DAO class, recently I converted the database to a sqlite database.
I do not know which connection method is better for the design as following:
Create only one database connection using a public variable when open the application, any queries use the only connection object for interaction during the run time, the connection is then closed when close the application
Create database connection every time before running a query, then close the database connection instantly after loading the resultset to the memory.
I recommend that you encapsulate your db access, so that the decision on whether to keep a persistent connection or not open can be changed at a later point.
Since you are using SqlLite I am assuming that it is a single user DB, so concurrency , connection contention, locking etc. are not likely to be issues.
Typically the main reasons to reuse short running connections is usually on a multi user web or service oriented system, where scalability and licensing considerations are important. This doesn't seem to be applicable in your case.
.
In short, there doesn't seem any reason not to keep a connection open for the entire duration of your app / user's login session based on the above assumptions.
If you use transactions however, I would suggest that you commit these after each successful atomic activity
You know your two options have + and -. For your special case I think to create database connection every time is not so bad idea, because creating connection to sqlite is very fast and no time consuming. Also this way you may create/close more than one connection at once, which is a good benefit, maybe you don't do it now, but in the future maybe you will have to.
My website is using Django, and now I want to port part of the logic to a Redis, so I need a Redis connection for my views.py code, obvious I can't write connect to redis code in views.py because it might be called multiple times, so I need to put the connect somewhere in the django, perhaps middleware?
But I don't want to make this complicated, just the same place where the MySQL database connected, I want to add a global object for Redis connection. Perhaps later for XMPP conenction and ZeroMQ.
How to do this?
ANy idea is appreciated. Thanks in advance :)
in typical Django server settings multiple requests will be handled by the same worker process.
You can simply put a global variable to hold the connection on top of views.py and use the conenction in each view function/class, the connection will be established when the worker process starts and closes when the worker process got recycled. It's semi-permanent connection but good enough.
MySQL connection works the same way in Django. It's not each db connection per request but per worker process life-span
It isn't obvious you want to do that. It isn't obvious why you would want to do that.
So why not connect in views.py? To use a single "global connection" will mean adding locking/serialization code to ensure that your connection is safe to use amongst many calls to your views. I actually create and connect right in the method in my various and sundry views.py files. Sometimes I connect to one instance or another. I've seen no performance issues and also don't have to worry about concurrency safety. I let Redis figure that out.
Another aspect of a global shared connection is degraded performance - you'll have one page view waiting on another's to finish before it can run. By allowing each to have it's own connection you avoid one view slowing down another while waiting on access to a global connector.
Consider this: if your queries are so small and fast that you don't expect to see a performance hit from serializing every page that accesses Redis, then you won't see any performance degradation from a connection per page as you connect, query, and close. I highly doubt that the cost of setting up the connection is significantly more than serializing all page accesses that connect to Redis.
So my suggestion is to just try it. If and only if you see an issue should you worry about implementing something you will probably not need.
There is a great piece of code for this already. http://github.com/andymccurdy/redis-py
I'm building an application which uses Mysql, I was wondering what would be the best way to manage the connection to the actual Mysql server?
I'm still in the design phase, but currently I have it Connecting (or aborting if error) before every query and disconnecting after which is just for testing as right now I'm only running 1 query to see if the code I've setup so far works.
My App might be performing a few queries every 5/10/20/30 minutes depending on settings and doesn't really need to do anything with SQL until that time.
So I'm wondering if its more beneficial to use a continuous connection that exists for the lifetime of the application (if possible) or to simply connect to sql before I intend to use it, do what the app needs to do then disconnect?
Connecting once and performing many queries will naturally be more efficient.
However, if performance isn't a major concern for your project, maybe aiming for simplicity in your code might be a better option (especially if you are the only connection to the database).
If you want to get clever, then maybe connect as and when you need to, then keep the connection alive until you stop making queries. Eg, drop the connection if there have been no queries for 30 seconds or something like that.
How many instances of this app will be connecting to MySQL? If it's just one, keeping a MySQL connection open for convenience shouldn't cause any problems, but remember there's a (configurable) limit to the number of MySQL connections you can have open to the server. In this case, I would recommend opening a connection, running whatever queries you need to run, and then closing it. Connecting per query adds more overhead as you add queries to your application.