I have an xml file which holds a set of "game" nodes (which contain details about saved gameplay, as you'd save your game on any console game). All of this is contained within a "games" root node. I'm implementing save functionality to this xml file and wish to be able to append or overwrite a "game" node and its child nodes within the "games" root node.
How can this be accomplished with xmllite.dll?
You can't physicaly "rewrite in place" any text file (including an XML file) except in the rare case you can guarantee that you're overwriting exactly as many bytes as were there. What you always need to do is to write a new file (which has parts from the old one and parts that are new), then rename the old file (e.g. add a .bak extension to it, after removing any older .bak that might have been left hanging around), rename the new file to the old name, and only at this point remove the old file. This approach guarantees that a computer or disk crash in the middle of your work won't be a disaster -- either the old or the new data will be around (at worst you'll need a rename if the crash is just between the two renames).
To write a new file, with mods and lots from the old one, in xmlfile, use the reader functionality documented here and the writer functionality documented here. For a small file, you can first build a tree of objects in-memory via the reader, then write it all out via the writer; but that can take a lot of memory. The alternative is an incremental parsing approach such as the one the MSDN docs call a "pull programming model".
Related
I'm currently in the idea phase for a small application that requires keeping track of specific, user chosen files.
I want the system to be intuitive, such that a user can change the file name, directory name, or move the file to a new location, and the application would still be able to keep track of the file.
Now, I know that I can monitor directories already, for the majority of these kinds of changes, with windows system calls.
The problem I'd have is finding if/when a file is cut, copied, or pasted somewhere. I've read a little about shell extensions, but they have actually rather confused me (since I haven't had the time to actually read good and long about them yet). Is there a way I could monitor a file for being cut/pasted/copied, so that my application could continue to track the file in it's new location? Would I have to do this through a shell extension?
You can use shortcut objects to track files even if they are renamed or moved.
I use the FileDataModel as the DataModel for Recommendations in Mahout. I first generate the base file (e.g. prefs.txt). From time to time, there are some changes, which are written to update files (prefs.1.txt, prefs.2.txt, ...).
Am I allowed to delete the old update-Files after loading them to the model? When I try deleting them (in Windows), the Explorer say's that the file is currently used by Java. Why is it the case that it's not allowed to delete the original file? I believe that the data is now stored in memory and thus Mahout doesn't need the file anymore.
It doesn't re-read the old files, only the new update data. However it does expect that the "main" data file is always there since it keeps looking for its last modified time.
The general idea is that you sometimes push a full copy of the data file, and in between, more frequently, push small update files. If you do that it should work as expected, and, yes, you can delete update files once they've been read.
(OF course, if you rebooted your server, it would have to start over from the last main data file and whatever update files were left, which could be incomplete or inconsistent. I'd only delete updates after you've pushed a new main data file.)
I don't know why you can't delete them as they are never held open. Maybe it's a weird Windows thing.
I'm working on a project that requires me to load some of the data from an XML file on to a GUI. The GUI allows the user to make some changes to the data. What I want to be able to do is to save these changes back onto the XML file.
I know it is possible to rewrite the whole file but the file is pretty huge, and not all the data in the file is being changed or even being used in my program.
This is my first project working with TinyXML and C++ Builder. I am just looking for some suggestions as to how I should approach this.
Unless you are certain that the new text will be exactly the same size as the old, rewriting only part of a text file is not a good idea in general. There are file formats where piecemeal replacement is possible. XML is not one of them. Not in the general case, at least.
Inserting data in the middle of a file, thus moving the rest down, is basically equivalent to loading the rest of the file, making the file bigger, and writing it back. So you may as well just load the entire file, make your modifications, and save it again. Your code will be simpler and likely not much slower.
And no, a SAX parser isn't going to help you here. It allows you to stream reading (though I would suggest a pull parser rather than a push one), but that's not going to allow you to insert data into the file. That's generally not supported by most XML parsers I know of. They can write data, but writing and non-destructively inserting are two different things.
TinyXml will let you do what you want without damaging the file contents (as long as its valid xml). I just checked this so I am quite certain. Obviously you have to know and precisely what attributes and tags you want to edit, but you can add/edit tags without affecting existing attributes/tags/comments even within the tags you edit. It will take a while until you get used to the structure, but it is definitely possible.
You have to know the structure of the xml!
TiXmlDocument doc("filepath"); //will open your document
if (!doc.LoadFile()) //you do have to open the whole file
{
cout<<"No XML structure found"<<endl;
return; // exit function don't load anything
}
TiXmlElement *root = doc.RootElement(); //pointer to root element
Now you can use this pointer and commands like:
root->FirstChild("tageone")->ToElement();
tageone->SetDoubleAttribute("attribute", value);
to change stuff.
Sorry for the rushed explanation, but you'll need to read through the documentation a bit to get the hang of it.
cheers
Update
As I said in the comment, I don't think that you are better off if you insert into the middle of a file. However, if you need/want additional security I suggest two additional steps:
perform a sanity check of the xml file at all the important steps. This can be anything where you make sure that the file you are reading is really what you need.
calculate a checksum over the content of the whole file before saving and check it afterwards. This does not necessarily need to be a CRC, I just named the function calculate_crc(). Anything that lets you verify the integrity of the data is good.
I would do this approximately as follows (pseudocode):
TiXmlDocument doc( "demo.xml" );
doc.LoadFile();
perform_sanitycheck(doc);
// do whatever you need to change
perform_sanitycheck(doc);
unsigned int crc = calculate_crc(doc);
doc.SaveFile("temp_name.xml"); // save the file under another name
TiXmlDocument doc2( "temp_name.xml" );
perform_sanitycheck(doc2);
if(verify_crc(doc, crc))
{
delete_file("demo.xml");
rename_file("temp_name.xml", "demo.xml");
}
The sanity check would take the appropriate action if necessary. You need to substitute the two function delete_file() and rename_file() with an API or library function for your environment.
The functions calculate_crc() and verify_crc() could be specifically crafted to check only the parts that you need to have unchanged.
I wonder which is the fastest way to erase part of a file in c++.
I know the way of write a second file and skip the part you want. But i think is slow when you work with big files.
What about database system, how they remove records so fast?
A database keeps an index, with metadata listing which parts of the file are valid and which aren't. To delete data, just the index is updated to mark that section invalid, and the main file content doesn't have to be changed at all.
Database systems typically just mark deleted records as deleted, without physically recovering the unused space. They may later reuse the space occupied by deleted records. That's why they can delete parts of a database quickly.
The ability to quickly delete a portion of a file depends on the portion of the file you wish to delete. If the portion of the file that you are deleting is at the end of the file, you can simply truncate the file, using OS calls.
Deleting a portion of a file from the middle is potentially time consuming. Your choice is to either move the remainder of the file forward, or to copy the entire file to a new location, skipping the deleted portion. Either way could be time consuming for a large file.
The fastest way I know is to open data file as a Persisted memory-mapped file and simple move over the part you don't need. Would be faster than moving to second file but still not too fast with big files.
In windows is it possible through an API to write to the middle of a file without overwriting any data and without having to rewrite everything after that?
If it's possible then I believe it will obviously fragment the file; how many times can I do it before it becomes a serious problem?
If it's not possible what approach/workaround is usually taken? Re-writing everything after the insertion point becomes prohibitive really quickly with big (ie, gigabytes) files.
Note: I can't avoid having to write to the middle. Think of the application as a text editor for huge files where the user types stuff and then saves. I also can't split the files in several smaller ones.
I'm unaware of any way to do this if the interim result you need is a flat file that can be used by other applications other than the editor. If you want a flat file to be produced, you will have to update it from the change point to the end of file, since it's really just a sequential file.
But the italics are there for good reason. If you can control the file format, you have some options. Some versions of MS Word had a quick-save feature where they didn't rewrite the entire document, rather they appended a delta record to the end of the file. Then, when re-reading the file, it applied all the deltas in order so that what you ended up with was the right file. This obviously won't work if the saved file has to be usable immediately to another application that doesn't understand the file format.
What I'm proposing there is to not store the file as text. Use an intermediate form that you can efficiently edit and save, then have a step which converts that to a usable text file infrequently (e.g., on editor exit). That way, the user can save as much as they want but the time-expensive operation won't have as much of an impact.
Beyond that, there are some other possibilities.
Memory-mapping (rather than loading) the file may provide efficiences which would speed things up. You'd probably still have to rewrite to the end of the file but it would be happening at a lower level in the OS.
If the primary reason you want fast save is to start letting the user keep working (rather than having the file available to another application), you could farm the save operation out to a separate thread and return control to the user immediately. Then you would need synchronisation between the two threads to prevent the user modifying data yet to be saved to disk.
The realistic answer is no. Your only real choices are to rewrite from the point of the modification, or build a more complex format that uses something like an index to tell how to arrange records into their intended order.
From a purely theoretical viewpoint, you could sort of do it under just the right circumstances. Using FAT (for example, but most other file systems have at least some degree of similarity) you could go in and directly manipulate the FAT. The FAT is basically a linked list of clusters that make up a file. You could modify that linked list to add a new cluster in the middle of a file, and then write your new data to that cluster you added.
Please note that I said purely theoretical. Doing this kind of manipulation under a complete unprotected system like MS-DOS would have been difficult but bordering on reasonable. With most newer systems, doing the modification at all would generally be pretty difficult. Most modern file systems are also (considerably) more complex than FAT, which would add further difficulty to the implementation. In theory it's still possible -- in fact, it's now thoroughly insane to even contemplate, where it was once almost reasonable.
I'm not sure about the format of your file but you could make it 'record' based.
Write your data in chunks and give each chunk an id.
Id could be data offset in file.
At the start of the file you could
have a header with a list of ids so
that you can read records in
order.
At the end of 'list of ids' you could point to another location in the file (and id/offset) that stores another list of ids
Something similar to filesystem.
To add new data you append them at the end and update index (add id to the list).
You have to figure out how to handle delete record and update.
If records are of the same size then to delete you can just mark it empty and next time reuse it with appropriate updates to index table.
Probably the most efficient way to do this (if you really want to do it) is to call ReadFileScatter() to read the chunks before and after the insertion point, insert the new data in the middle of the FILE_SEGMENT_ELEMENT[3] list, and call WriteFileGather(). Yes, this involves moving bytes on disk. But you leave the hard parts to the OS.
If using .NET 4 try a memory-mapped file if you have an editor-like application - might jsut be the ticket. Something like this (I didn't type it into VS so not sure if I got the syntax right):
MemoryMappedFile bigFile = MemoryMappedFile.CreateFromFile(
new FileStream(#"C:\bigfile.dat", FileMode.Create),
"BigFileMemMapped",
1024 * 1024,
MemoryMappedFileAccess.ReadWrite);
MemoryMappedViewAccessor view = MemoryMapped.CreateViewAccessor();
int offset = 1000000000;
view.Write<ObjectType>(offset, ref MyObject);
I noted both paxdiablo's answer on dealing with other applications, and Matteo Italia's comment on Installable File Systems. That made me realize there's another non-trivial solution.
Using reparse points, you can create a "virtual" file from a base file plus deltas. Any application unaware of this method will see a continuous range of bytes, as the deltas are applied on the fly by a file system filter. For small deltas (total <16 KB), the delta information can be stored in the reparse point itself; larger deltas can be placed in an alternative data stream. Non-trivial of course.
I know that this question is marked "Windows", but I'll still add my $0.05 and say that on Linux it is possible to both insert or remove a lump of data to/from the middle of a file without either leaving a hole or copying the second half forward/backward:
fallocate(fd, FALLOC_FL_COLLAPSE_RANGE, offset, len)
fallocate(fd, FALLOC_FL_INSERT_RANGE, offset, len)
Again, I know that this probably won't help the OP but I personally landed here searching for a Linix-specific answer. (There is no "Windows" word in the question, so web search engine saw no problem with sending me here.