I have a binary file, when I opend it, I used ::StgOpenStorage with STGM_READWRITE | STGM_SHARE_DENY_WRITE | STGM_TRANSACTED mode to get a root storage named rootStorage. And then, I used rootStorage.OpenStream with STGM_READWRITE | STGM_SHARE_EXCLUSIVE mode to get a substream named subStream.
Next, I wrote some data with subStream.Wirte(...), and called subStream.Commit(STGC_DEFAULT), but it just couldn't write the data in the file.
And I tried rootStorage.Commit(STGC_DEFAULT) also, the data can be written.
But when I used UltraCompare Professional - Binary Compare to compare the original file with the file I opend, a lot of extra data had been written at the end of the file. The extra data seems to be from the beginning of the file.
I just want to write a little data into the file while opening it. What should I do?
Binary file comparison will probably not work for structured storage files. The issue is that structured storage files often have extra space allocated in them--to handle transacted mode and to grow the file. If you want to do a file comparison, it will take more work. You will have to open the root storage in each file, then open the stream, and do a binary comparison on the streams.
I had found out why there are extra data on my file.
1. Why should I use IStorage.Commit()
I used STGM_READWRITE mode to create a storage. It's called transacted mode. In transacted mode, changes are accumulated and are not reflected in the storage object until an explicit commit operation is done. So I need to call rootStorage.Commit().
2. Why there are extra data after calling IStorage.Commit(STGC_DEFAULT)
According to this website:
The OLE-provided compound files use a two phase commit process unless STGC_OVERWRITE is specified in the grfCommitFlags parameter. This two-phase process ensures the robustness of data in case the commit operation fails. First, all new data is written to unused space in the underlying file. If necessary, new space is allocated to the file. Once this step has been successfully completed, a table in the file is updated using a single sector write to indicate that the new data is to be used in place of the old. The old data becomes free space to be used at the next commit. Thus, the old data is available and can be restored in case an error occurs when committing changes. If STGC_OVERWRITE is specified, a single phase commit operation is used.
Related
When I write fout.open("file.dat",ios::out|ios::trunc|ios::binary);
does the file loose all its data at that instance
or it will wait for something to be written and then data will be lost?
(I hope you get my point, all I'm asking is whether just writting the above statement, i.e fout.write() will invoke removal of records from a binary file or we need to pass some data to the file and then the previous data already stored in the file would be lost)
The trunc flag will zero the file out at open().
Using Windows API's WriteFileGather, I am writing a file to the disk.
I want to append new buffers to the existing file.
What is the way to prevent WriteFileGather from overwriting the existing file?
WriteFileGather will never overwrite the file unless you ask it to - theres no implied overwrite/append option, theres ONLY a 'please write data at file position X option'.
You should open the file handle normally (making sure you've got GENERIC_WRITE access and specifying flags at least flags FILE_FLAG_OVERLAPPED and FILE_FLAG_NO_BUFFERING by using CreateFile
Then you set the position the file writes at by using the Offset and OffsetHigh members of the OVERLAPPED you pass in as the 5th parameter.
This is similar to the way WriteFile works when its running in asynchronous mode - you must specify the position to write at. Its probably easier to learn how to do positional asyncronous writes using WriteFile first then move onto WriteFileGather if you need its additional power.
See here for docs.
EDIT: To answer the comment from Harry, to get the end of file you can either remember how much you've written before (assuming this is a new file you created) or get the current file size from a HANDLE using SetFilePointerEx with distance 0 and method FILE_END which will return to you the end of the file. There are other ways of getting a file size but beware you may get a cached answer (e.g. if iterating over a directory) and so the above is recommended.
Say, I have a file of an arbitrary length S and I need to remove first of its N bytes (where N is much less than S.) What is the most efficient way to do it on Windows?
I'm looking for a WinAPI to do this, if one is available.
Otherwise, what are my options -- to load it into RAM and then re-write the existing file with the remainder of data? (In this case I cannot be sure that the PC has enough RAM?) Or write the remainder of file data into a new file, erase the old one, and rename the new file into the old one. (In this case what to do if any of these steps fail? Plus how about defragmentation that this method causes on disk?)
There is no general way to do this built into the OS. There are theoretical ways to edit the file system's data structures underneath the operating system on sector or cluster boundaries, but this is different for each file system, and would need to violate any security model.
To accomplish this you can read in the data starting at byte N in chunks of say 4k, and then write them back out starting at byte zero, and then use the file truncate command (setendoffile) to set the new smaller end of file when you are finished copying the data.
The most efficient method to delete data at the beginning of the file is to modify the directory entry, on the hard drive, that tells where the data starts. Again, this is the most efficient method.
Note: This may not be possible, if the data must start on a new boundary. If this is the case, you may have to write the remainder data on the sector(s) to new sector(s), essentially moving the data.
The preferred method is to write a new file that starts with data copied after the deleted area.
Moving files on same drive is faster than copying files since data isn't duplicated; only the file pointer, (symbolic)links & file allocation/index table is updated.
The move command in CMD could be modified to allow user to set file start & end markers, effecting file truncation without copying file data, saving valuable time & RAM/Disk overheads.
Alternative would be to send the commands direct to the device/disk driver bypassing the Operating System as long as OS knows where to find the file & file properties eg. file size, name & sectors occupied on disk.
This is my first post here, so please bear with me.
I have searched high and low on the internet for an answer, but I've not been able to resolve my issue, so I have decided to write a post here.
I am trying to write(append) to a JSON array on file using C++ and JZON, at intervals of 1 write each second. The JSON file is initially written by a “Prepare” function. Another function is then called each second to a add an array to the JSON file and append an new object to the array every second.
I have tried many things, most of which resulted in all sorts of issues. My latest attempt gave me the best results and this is the code that I have included below. However, the approach I took is very inefficient as I am writing an entire array every second. This is having a massive hit on CPU utilisation as the array grows, but not so much on memory as I had first anticipated.
What I really would like to be able to do is to append to an existing array contained in a JSON file on disk, line by line, rather than having to clear the entire array from the JSON object and rewriting the entire file, each and every second.
I am hoping that some of the geniuses on this website will be able to point me in the right direction.
Thank you very much in advance.
Here is my code:
//Create some object somewhere at the top of the cpp file
Jzon::Object jsonFlight;
Jzon::Array jsonFlightPath;
Jzon::Object jsonCoordinates;
int PrepareFlight(const char* jsonfilename) {
//...SOME PREPARE FUNCTION STUFF GOES HERE...
//Add the Flight Information to the jsonFlight root JSON Object
jsonFlight.Add("Flight Number", flightnum);
jsonFlight.Add("Origin", originicao);
jsonFlight.Add("Destination", desticao);
jsonFlight.Add("Pilot in Command", pic);
//Write the jsonFlight object to a .json file on disk. Filename is passed in as a param of the function.
Jzon::FileWriter::WriteFile(jsonfilename, jsonFlight, Jzon::NoFormat);
return 0;
}
int UpdateJSON_FlightPath(ACFT_PARAM* pS, const char* jsonfilename) {
//Add the current returned coordinates to the jsonCoordinates jzon object
jsonCoordinates.Add("altitude", pS-> altitude);
jsonCoordinates.Add("latitude", pS-> latitude);
jsonCoordinates.Add("longitude", pS-> longitude);
//Add the Coordinates to the FlightPath then clear the coordinates.
jsonFlightPath.Add(jsonCoordinates);
jsonCoordinates.Clear();
//Now add the entire flightpath array to the jsonFlight object.
jsonFlight.Add("Flightpath", jsonFlightPath);
//write the jsonFlight object to a JSON file on disk.
Jzon::FileWriter::WriteFile(jsonfilename, jsonFlight, Jzon::NoFormat);
//Remove the entire jsonFlighPath array from the jsonFlight object to avoid duplicaiton next time the function executes.
jsonFlight.Remove("Flightpath");
return 0;
}
For sure you can do "flat file" storage yourself.. but this is a symptom of needing a database. Something very light like SQLite, or mid-weight & open-source like MySQL, FireBird, or PostgreSQL.
But as to your question:
1) Leave the closing ] bracket off, and just keep the file open & appending -- but if you don't close the file correctly, it will be damaged & need repair to be readable.
2) Your current option -- writing a complete file each time -- isn't safe from data loss either, as the moment you "open to overwrite" you lose all data previously stored in the file. The workaround here, is to rename the old file as a backup before you start writing.
You should also make backup copies of your file, with the first option. (Say at daily intervals). Otherwise data loss is likely to occur eventually -- on Ctrl-C, power loss, program error or system crash.
Of course if you use any of SQLlite, MySQL, Firebird or PostgreSQL all the data-integrity problems will be handled for you.
My file(sample.txt) has 10 chars. I opened the file in write mode, have done a createfilemapping("mymapping"). Then I Hold the process by getchar.
Then opened another process which does the below
openfilemapping("mymapping"),
char*k = (char*)mapviewoffile
Now I can access the value of sample.txt and change it via k. However, how to insert/append a strip of another 10 characters into the file.
Shared-memory mappings are fixed in size upon creation. You will need to call CreateFileMapping again with a larger mapping size. This will automatically extend the file, at which point you can MapViewOfFile again, and write in whatever data you want. Note that you will need to change the name of the file mapping, unless you first close all handles and mappings to it so the file mapping is destroyed.
As an aside, it's unusual (but not illegal, of course) to use named file mappings backed by disk files. Generally, if you're mapping a real file, you'd pass NULL to the lpName parameter of CreateFileMapping. You'd pass in a name when you want to create a shared memory mapping with no backing file - that is, hFile would be NULL and lpName be the name of the shared memory segment.