Are there a set of commands used to perform CRUD on an XML file?
I'm envisioning a component (myData.cfc) that has a Save function (Insert/Update), a Delete function and a Read function.
Are you asking about a way to use a XML file to store data? While you certainly can do that, I'm not sure I'd recommend it. You would have to be extremely careful to single thread the updates and you would not get good performance out of it. I mean it would probably work fine for a few hundred rows, but after that your going to suffer.
Can you elaborate on how you want to use the XML file? If you just mean settings, or a static set of data, I think it's fine. If you mean CRUD as in "N rows of data", you really want a database.
Related
I am still learning QT, and this thing come to my mind: If I design a program and I want some data to be saved and protected, all the ways I know about saving data from QT Creator applications are writing into files which can be easily altered and deleted. Is there any way that I can safely store some application data and no one else can access it without breaking the application?
This might be a stupid question, but I am actually looking forward to some idea.
Thanks a lot.
It is not possible to make it impossible to rewrite data for someone who has root/administrator access. You can however make the application refuse unauthorized changes and exit.
Assuming you want an external data file that can only be changed from within the application you can do something lie this:
Each time when the data file is changed by the application, calculate a hash value of the data, encrypt that and store it along the data file. Each time when reading the data, first hash the data, then read in the stored hash, decrypt it and compare it with the data hash. If it is not the same, exit the application.
This way if someone makes any changes from the outside, he will not be able to provide the accurate encrypted hash value for the data file, so the hash check will fail. Only someone who has the secret key will be able to make changes that the application will accept.
I have a server-client application where clients are able to edit data in a file stored on the server side. The problem is that the file is too large in order to load it into the memory (8gb+). There could be around 50 string replacements per second invoked by the connected clients. So copying the whole file and replacing the specified string with the new one is out of question.
I was thinking about saving all changes in a cache on the server side and perform all the replacements after reaching a certain amount of data. After reaching that amount of data I would perform the update by copying the file in small chunks and replace the specified parts.
This is the only idea I came up with but I was wondering if there might be another way or what problems I could encounter with this method.
When you have more than 8GB of data which is edited by many users simultaneously, you are far beyond what can be handled with a flatfile.
You seriously need to move this data to a database. Regarding your comment that "the file content is no fit for a database": sorry, but I don't believe you. Especially regarding your remark that "many people can edit it" - that's one more reason to use a database. On a filesystem, only one user at a time can have write access to a file. But a database allows concurrent write access for multiple users.
We could help you to come up with a database schema, when you open a new question telling us how your data is structured exactly and what your use-cases are.
You could use some form of indexing on your data (in a separate file) to allow quick access to the relevant parts of this gigantic file (we've been doing this with large files successfully (~200-400gb), but as Phillipp mentioned you should move that data to a database, especially for the read/write access. Some frameworks (like OSG) already come with a database back-end for 3d terrain data, so you can peek there, how they do it.
I have a huge text file (~5GB) which is the database for my program. During run this database is read completely many times with string functions like string::find(), string::at(), string::substr()...
The problem is that this text file cannot be loaded in one string, because string::max_size is definitely too small.
How would you implement this? I had the idea of loading a part to string->reading->closing->loading another part to same string->reading->closing->...
Is there a better/more efficient way?
How would you implement this?
With a real database, for instance SQLite. The performance improvement from having indexes is more than going to make up for your time learning another API.
Since this is a database, I'm assuming it'd have many records. That to me implies best idea would be to implement a data class for each records and populate a list/vector/etc depending upon how you plan to use it. I'd also look into persistent cache as the file is big.
And within in your container class of all records, you could implement search etc functions as you see fit. But as suggested for a db of this size, you're probably best of using a database.
I'm currently brainstorming a financial program that will deal with (over time) fairly large amounts of data. It will be a C++/Qt GUI app.
I figure reading all the data into memory at runtime is out of the question because given enough data, it might hog too much memory.
I'm trying to come up with a way to read into memory only what I need, for example, if I have an account displayed, only the data that is actually being displayed (and anything else that is absolutely necessary). That way the memory footprint could remain small even if the data file is 4gb or so.
I thought about some sort of searching function that would slowly read the file line by line and find a 'tag' or something identifying the specific data I want, and then load that, but considering this could theoretically happen every time there's a gui update that seems like a terrible way to go.
Essentially I want to be able to efficiently locate specific data in a file, read only that into memory, and possibly change it and write it back without reading and writing the whole file every time. I'm not an experienced programmer and my googling for ideas hasn't been very successful.
Edit: I should probably mention I intend to use Qt's fancy QDataStream related classes to store the data. In other words the file will likely be binary and not easily searchable line by line like a text file.
Okay based on your comments.
Start simple. Forget about your fiscal application for now, except as background. So suitable example for your file system
One data type e.g accounts.
Start with fixed width columns giving you a fixed width record.
One file for data
Have another file for the index of account number
Do Insert, Update and Delete, you'll learn a lot.
For instance.
Delete, you could find the index and the data, move them out and rebuild both files.
You could have a an internal field on the account record, that indicated it had been deleted, set that in data, and just remove the index. The latter is also rewrite the entire file though. You could put the delete flag in the index file instead...
When inserting do you want to append, do you want to find a deleted record and reuse that slot?
Is your index just going to be a straight list of accounts and position, or dovyouvwant to hash it, use a tree. You could spend a weeks if not months just looking at indexing strategies alone.
Happy learning anyway. It will be interesting to help with your future questions.
I'm working on a project that requires me to load some of the data from an XML file on to a GUI. The GUI allows the user to make some changes to the data. What I want to be able to do is to save these changes back onto the XML file.
I know it is possible to rewrite the whole file but the file is pretty huge, and not all the data in the file is being changed or even being used in my program.
This is my first project working with TinyXML and C++ Builder. I am just looking for some suggestions as to how I should approach this.
Unless you are certain that the new text will be exactly the same size as the old, rewriting only part of a text file is not a good idea in general. There are file formats where piecemeal replacement is possible. XML is not one of them. Not in the general case, at least.
Inserting data in the middle of a file, thus moving the rest down, is basically equivalent to loading the rest of the file, making the file bigger, and writing it back. So you may as well just load the entire file, make your modifications, and save it again. Your code will be simpler and likely not much slower.
And no, a SAX parser isn't going to help you here. It allows you to stream reading (though I would suggest a pull parser rather than a push one), but that's not going to allow you to insert data into the file. That's generally not supported by most XML parsers I know of. They can write data, but writing and non-destructively inserting are two different things.
TinyXml will let you do what you want without damaging the file contents (as long as its valid xml). I just checked this so I am quite certain. Obviously you have to know and precisely what attributes and tags you want to edit, but you can add/edit tags without affecting existing attributes/tags/comments even within the tags you edit. It will take a while until you get used to the structure, but it is definitely possible.
You have to know the structure of the xml!
TiXmlDocument doc("filepath"); //will open your document
if (!doc.LoadFile()) //you do have to open the whole file
{
cout<<"No XML structure found"<<endl;
return; // exit function don't load anything
}
TiXmlElement *root = doc.RootElement(); //pointer to root element
Now you can use this pointer and commands like:
root->FirstChild("tageone")->ToElement();
tageone->SetDoubleAttribute("attribute", value);
to change stuff.
Sorry for the rushed explanation, but you'll need to read through the documentation a bit to get the hang of it.
cheers
Update
As I said in the comment, I don't think that you are better off if you insert into the middle of a file. However, if you need/want additional security I suggest two additional steps:
perform a sanity check of the xml file at all the important steps. This can be anything where you make sure that the file you are reading is really what you need.
calculate a checksum over the content of the whole file before saving and check it afterwards. This does not necessarily need to be a CRC, I just named the function calculate_crc(). Anything that lets you verify the integrity of the data is good.
I would do this approximately as follows (pseudocode):
TiXmlDocument doc( "demo.xml" );
doc.LoadFile();
perform_sanitycheck(doc);
// do whatever you need to change
perform_sanitycheck(doc);
unsigned int crc = calculate_crc(doc);
doc.SaveFile("temp_name.xml"); // save the file under another name
TiXmlDocument doc2( "temp_name.xml" );
perform_sanitycheck(doc2);
if(verify_crc(doc, crc))
{
delete_file("demo.xml");
rename_file("temp_name.xml", "demo.xml");
}
The sanity check would take the appropriate action if necessary. You need to substitute the two function delete_file() and rename_file() with an API or library function for your environment.
The functions calculate_crc() and verify_crc() could be specifically crafted to check only the parts that you need to have unchanged.