I am currently writting a .dll that uses sqlite3 to talk to a database on a windows system.
Right now, it is very simple and I have just written a test program that does inserts and selects from the database.
This issue however, is that my dll will be used by multiple programs on the same system, each writing to the same database file. This of course brings mutex issues with it.
I want to know the extent of which the sqlite3 library supports this before I go ahead and write a complicated connection pool for it. According to the doco there are things like "Shared Caches" and "No Mutex" Modes as specified by a paramater given to the sqlite3_open_v2 function: http://www.sqlite.org/c3ref/open.html
I have also hunted around for a few sqlite3 wrappers but I cannot find anything that suits my multi-access needs.
Does anyone have any experience with writing to sqlite3 db files from multiple processes that use the same .dll? Or has anyone had any luck with using their SHARED_CACHES - I seem to run into all sorts of issues.
Any help on the matter would be great, thanks!
~ Dan
As long as you're accessing the same database file through SQLite3, it should be fine. Indeed, SQLite3 is designed specifically to handle multiple processes accessing the same DB file.
SQL, as a specification, already deals with multiple accesses, and it provides controls for atomic accessing and so forth (transactions, that sort of thing). As long as you properly use those controls, you should be file.
It's multiple threads that SQLite3 has an issue with.
Related
I need some help for interrogating an xlsb database (file) from a VCL C++ program; (using C++Builder XE2) mainly, i would like to query the data inside for simple viewing (its on the same machine, no networking involved), and i don't need to modify it ! so i would like to know :
Which control to use for connecting to an xlsb database
(TDatabase, dbExpress ?)
How to use that control for the purpose
of reading the data, (same as SQL?)
The difference between the regular xsl and xslb, and are they used in the same manner ?
I did some research before asking, but the information about that is pretty scarce !
(i'm on Win 7 64bit)
Thank you all !
Never tested it with these files but i believe you can use a TADOConnection and then user Microsoft OLE DB Provider for ODBC Drivers as the data provider, i know it can use access and excel files as a source, personally i have only ever used it for access files, but it might do what you want.
I switch to using sqlite3 instead of MySQL because I had to run many jobs on a PBS system which doesn't not have mysql. Of course on my machine I do not have a NFS while there exists one on the PBS. After spending lots of time switching to sqlite3, I go to run many jobs and I corrupt my database.
Of course down in the sqlite3 FAQ it is mentioned about NFS, but I didn't even think about this when I started.
I can copy the database at the beginning of the job but it will turn into a merging nightmare!
I would never recommend sqlite to any of my colleagues for this simple reason: "sqlite doesn't work (on the machines that matter)"
I have read rants about NFS not being up to par and it being their fault.
I have tried a few workarounds, but as this post suggests, it is not possible.
Isn't there a workaround which sacrifices performance?
So what do I do? Try some other db software? Which one?
You are using the wrong tool. Saying "I would never recommend sqlite ..." based on this experience is a bit like saying "I would never recommend glass bottles" after they keep breaking when you use them to hammer in a nail.
You need to specify your problem more precisely. My attempt to read between the lines of your question gives me something like this:
You have many nodes that get work through some unspecified path, and produce output. The jobs do not interact because you say you can copy the database. The output from all the jobs can be merged after they are finished. How do you effectively produce the merged output?
Given that as the question, this is my advice:
Have each job produce its output in a structured file, unique to each job. After the jobs are finished, write a program to parse each file and insert it into an sqlite3 database. This uses NFS in a way it can handle (single process writing sequentially to a file) and uses sqlite3 in a way that is also sensible (single process writing to a database on a local filesystem). This avoid NFS locking issues while running the job, and should improve throughput because you don't have contention on the sqlite3 database.
I'm writing a C++ MFC program, and I see programs that remember the last input values for some fields from one program run to the next. I could do this by saving it to a file and loading that and then repopulating, but is there another, quicker way of doing this, as I think I remember reading somewhere that these values could be stored in the registry? If anyone has an example or personal experiences, I would be very interested.
Thanks,
James
There are many options out there
I personally don't like the registery and prefer to keep my program portable.
so what I can think of right now are
1- ini files
2- property files
3- SQL server
4- you can also synchronize the user settings via web server, but as #Jeeva mentioned there are security considerations
Cheers
It all depends on what kind of values you want remember. If it is an configuration data you can use config files. If it is small application data you can use flat files with proprietary format. If it is huge data you can use a database like SQL Server.
Usually there is a debate about using ini file vs registry.
Other things you want to consider is whether the data will be used across network. Whether you need to encrypt.
I've recalled using little 'filesystems' before that basically provided an interface to something else. For example, I believe there was a GMail filesystem that created an entry in My Computer and could be used like any other drive on your local computer. How can I go about implementing something like this in C++?
Thank you!
Try Dokan. It's like FUSE, except for Windows. I think there are certain limitations to namespace extensions, like they cannot be accessed from the command line, but I'm really not sure as of now.
Writing an actual file-system involves writing a driver; which means kernel-mode code (scary stuff) and paying for getting the IFS DDK. (edit: looks like they don't charge for it anymore)
What you probably want is a “namespace extension”.
Try this: The Complete Idiot's Guide to Writing Namespace Extensions - CodeProject
This may be a starting point to extending NTFS in the way that the GMail filesystem used to do: Windows NT reparse points.
The GMail Filesystem is just the name given; it is not any filesystem as such. It is just a namespace extension for Windows Explorer that links with the GMail account of yours!
I dont know exactly what you are trying to do, but in anyway, I believe, the following link will be of some use to you:
http://msdn.microsoft.com/en-us/magazine/cc188741.aspx
Just as a reference: virtual drives can be created using our Callback File System product, which is a supported, documented and maintained solution.
I was thinking of this too, perhaps some example code ? (email me if i forget plz ;p doin sdk now)
I'm thinking of a similar filesystem that would plug in as a driver and allow dynamic 'soft raid' on larger files mostly by putting them on more than one disk, perhaps some compression options and 'smart' filters to toggle usage in high disk space low usage and other situations more effectively, with status controls and indicators as a normalish program too
Seems like I would load the driver kit,
then i want the file writing event, and am mostly replacing fopen and similar functions automatically as an intermediate driver with a little windows network driver experience
I also heard good things about developing on a virtual machine for less crashing and more debugging
Also perhaps more metainfo on some or all files, including files in special folders with options too, including maybe both fast and simple (obfuscated and/or symmetic key) encryption options on folder, specified, all, letter, etc, or whatever, or the slower version and maybe integrated and optional (also profitable) online cvs-like diff style backups that mostly target changes to hot files for online backup at intervals and prices, mostly perhaps with matching keyboard events and might even be useful as simply a keylogging online backup service that is reasonably secure too
while avoiding common files like windows files or the normal stuff in the 'programs' directory that can be copied easily with pirate tools, unlike all of your documents.
I'm just learning C++, just started to mess around with QT, and I am sitting here wondering how most applications save data? Is there an industry standard? Do they store it in a XML file, text file, SQLite? What about sensitive data that say accounting software would need to save? I'm just interested in learning what the best practices for this are.
Thanks
This question is way too broad. The only answer is it depends on the nature of the particular application and the data, and whether or not it is written in C++ has very little to do with it.
For example, user-configurable application settings are often stored in text files, but on Windows they are typically stored in the Registry. Accounting applications typically keep their data in a database of some sort.
There are many good ways to store application data (call it serialization).
Personally, I think for larger datasets, using an open format is much, much easier for debugging. If you go with XML, for example, you can store your data in an open form so that if you have file corruption issues (i.e. a client can't open your file for some reason), it's easier to find. If you have sensitive data in there, you can always encrypt it before writing it to file using key encryption. Microsoft, for instance, has gone from using a proprietary format to open xml in their office docs. They use .*x extension (.docx, .xlsx, etc). It's really just a compressed folder with xml files.
Using binary serialization is, of course, the industry standard at the moment for most standalone applications. Most likely that is because of the application framework they are using (such as MFC, which is old). If you take a look at most of the serialization techniques in modern application frameworks, XML serialization is very well supported.
First you need to clarify what kind of data you would like to save.
If you just want to save some application settings, use QSettings to save your settings to an INI file or registry.
If it is much more than just some application settings, go for XML files or SQL.
There is no standard practice, however if you want to use complex structured data, consider using an embedded database engine such as SQLite or Metakit, or Berkeley DB files. XML files would also do the job and be human readable/writable. Preferences can use INI files or the Windows registry, and so on. In short, it really depends on your usage pattern.
This is a general question. Like many things, the right answer depends on your application and its needs.
Most desktop applications save end-user data to a file (think Word and Excel). The format is up to you, XML, binary, etc. And if you can serialize/deserialize objects to file it will probably make your life easier.
Internal application data such as configuration files or temporary data might be saved to an XML file or an lightweight, local database such as SQLite
Often, "enterprise" applications used internally by a business will save their data to a back-end database such as SQL Server or Oracle. This is so all of the enterprise's data is saved to a single central location. And then it is available for reporting, etc.
For accounting software, you would need to consider the business domain and end users. For example, if the software is to be sold to large businesses you would probably use some form of a database to save data. Otherwise a binary file would be fine, perhaps with some form of encryption if you are really paranoid.
When you say "the best way", then you have to define what you mean by "good".
The problem is that various requirements conflict with each other, therefore so you can't satisfy all of them simultaneously.
For example, if one requirement is "concurrent multi-user access to the data" then this suggests using a database engine, but that conflicts with "as small as possible" and "minimize dependencies on 3rd-party software".
If a requirement is "portable data format" then this suggests XML, but that conflicts with "compact" and "indexed".
Do they store it in a XML file, text file, SQLite?
Yes.
Also, Binary files and relational databases.
Anything else?