In the QFile::copy documentation it says
If a file with the name newName already exists, copy() returns false
(i.e., QFile will not overwrite it).
But I need to copy a file even if the destination exists. Any workaround available in Qt for that?
Deleting the file is the obvious solution but it invites a race condition...
if (QFile::exists("/home/user/dst.txt"))
{
QFile::remove("/home/user/dst.txt");
}
QFile::copy("/home/user/src.txt", "/home/user/dst.txt");
The obvious solution is of course to delete the file if it exists, before doing the copy.
Note however that doing so opens up the code to a classic race condition, since on a typical multitasking operating system a different process could re-create the file between your applications' delete and copy calls. That would cause the copy to still fail, so you need to be prepared (and perhaps re-try the delete, but that might introduce a need for count so you don't spend forever attempting, and on and on).
The simplest retrying I can think of is:
while !QFile::copy("/home/user/src.txt", "/home/user/dst.txt")
{
QFile::remove("/home/user/dst.txt");
}
But this still isn't a real solution as some of the race conditions are things that don't block remove.
I'm currently hunting for a way to handle writing a web page as an output but without the auto refresh ever catching between the remove and the copy.
Just call remove() before calling copy()
Related
I'm creating a file format where I'd like to write an explicit message into the file indicating that the writer ran to completion. I've had problems in the past with generating files where the generating program crashed and the file was truncated without me realizing, since without an explicit marker there's no way for reading programs to detect a file is incomplete.
So I have a class that is used for writing these files. Now usually if you have an "open" operation and a "close" operation you want to use RAII, so I would put the code to write the end of file marker in the destructor. This way the user can't forget. But in a situation where writing doesn't complete because an exception is thrown the destructor will still be run -- in which case we don't want to write the message so readers will know the file is incomplete.
This seems like something that could happen any time there's a "commit" sort of operation. You want RAII so you can't forget to commit, but you also don't want to commit when an exception occurs. The temptation here is to use std::uncaught_exceptions, but I think that's a code smell.
What's the usual solution to this? Just require that people remember? I'm concerned this will be a stumbling block everytime someone tries to use my API.
One way to tackle this problem is to implement a simple framing system where you can define a header that is only filled in completely at the end of the write. Include a SHA256 hash to make the header useful for verifying the contents of the file. This is usually a lot more convenient than having to read bytes at the end of a file.
In terms of implementation you write out a header with some fields deliberately zeroed out, write the contents of the payload while feeding that data through your hashing method, and then seek back to the header and re-write that with the final values. The file starts out in an obviously invalid state and ends up valid only if everything ran to completion.
You could wrap up all of this in a stream handle that handles the implementation details so as far as the calling code is concerned it's just opening a regular file. Your reading version would throw an exception if the header is incomplete or invalid.
For your example, it seems like RAII would work fine if you add a commit method which the user of your class calls when they are done writing to a file.
class MyFileFormat {
public:
MyFileFormat() : committed_(false) {}
~MyFileFormat() {
if (committed_) {
// write the completion footer (I hope this doesn't throw!)
}
// close the underlying stream...
}
bool open(const std::string& path) {
committed_ = false;
// open the underlying stream...
}
bool commit() {
committed_ = true;
}
};
The onus is on the user to call commit when they're done, but at least you can be sure that resources get closed.
For a more general pattern for cases like this, take a look at ScopeGuards.
ScopeGuards would move the responsibility for cleanup out of your class, and can be used to specify an arbitrary "cleanup" callback in the event that the ScopeGuard goes out of scope and is destroyed before being explicitly dismissed. In your case, you might extend the idea to support callback for both failure-cleanup (e.g. close file handles), and success-cleanup (e.g. write completion footer and close file handles).
I've handled situations like that by writing to a temporary file. Even if you're appending to a file, append to a temporary copy of the file.
In your destructor, you can check std::uncaught_exception() to decide whether your temporary file should be moved to its intended location.
While writing a new vst-plugin using VSTGUI I'm really struggling with how to use the library, and most progress is made from guessing and debugging after (because there really is no documentation besides the million lines and ygrabit, which states little more than the obvious).
So far it's going good, but my last contribution to the project involved threads which made the design a little bit more problematic. Specifically, I'm working on a set of textlabels in a container (doing non-atomic operations) and these may (and obviously does) get destructed without my knowledge, when a user closes the window.
Even adding checks right before changing elements might still be a problem. So I actually need to control the lifetime of these objects (which is fine) except when they are shown in a CViewContainer, it automatically assumes ownership.
I have no idea how to write the backbone of the editor, so i used a program called VSTGUIBuilder for this, and appended (and basically rewrote) what i needed. However, since all 'views' you can work with requires either a parent or a systemwindow, you cannot instantiate any views/controls before reaching the AEffEditor::Open() function, which is called whenever your window is popped up.
And the AEffEditor::close() method is called whenever the window is closed. Now, the vstguibuilder put a
delete frame;
inside the AEffEditor::close() method which suggests you rebuild and dispense all resources on every open and close. Can this really be true? And if it is, is there no way i can protect my container's contents (which for details is a vector< CTextLabel *>) from getting deleted mid-function? It's no problem to dispose of it afterwards, I'm just worrying about segfaults while changing it.
Using mutexes and the such is really the last resort (if the call is coming from the host), I don't want to hang the host in any case if my code faults and never releases.
Edit:
I ended up finding a solution which is not so elegant, but works safely. Here's the code in the worker function:
while(bLock) {
Sleep(0);
}
bLock = true;
if(msgs.empty())
return;
/*
Prevent someone deletes our lines in close().
we create a copy of the container to be 100% sure
and increase the reference count, so we can safely
work with our own container and we 'forget' them
afterwards, so they will be deleted if needed.
This ensures that close AND open can be called
meanwhile we are working with the lines
*/
bDeleteLock = true;
// also the copy constructor should work as expected here
// since we are working with pointers, we still reference the same content.
auto copy_lines = lines;
for each(auto line in copy_lines) {
line->remember();
}
bDeleteLock = false;
...
for each(auto line in copy_lines) {
line->forget();
}
cont->setDirty();
bLock is another 'mutex' that protects a message queue, which this function will print out. bDeleteLock protects the process of copying the line container and 'remembering' them, and instantly releases if afterwards. Both are declared as volatile bools, shouldn't that be enough? Here's the close() method btw.
void CConsole::Close() {
// locking lines while copying them over in a container we can work with
while(bDeleteLock)
Sleep(0);
//waiting for bLock is not needed because it wont get deleted.
if(!visible) //if we are not visible it's our responsibility to remove the view
delete cont;
lines.clear();
}
Ahh, VSTGUI, that brings back some dark memories. ;) But seriously, yes, you will probably have to use a mutex to prevent the host from hanging. Having to instantiate everything when the window reopens seems kind of silly, but you can see many plugins do just that.
One potential workaround is to use a shared memory segment for cached view data, and then pass a reference to the location back to your plugin
I have multiple goroutines in my program, each of which makes calls to fmt.Println without any explicit synchronization. Is this safe (i.e., will each line appear separately without data corruption), or do I need to create another goroutine with synchronization specifically to handle printing?
No it's not safe even though you may not sometimes observe any troubles. IIRC, the fmt package tries to be on the safe side, so probably intermixing of some sort may occur but no process crash, hopefully.
This is an instance of a more universal Go documentation rule: Things are not safe for concurrent access unless specified otherwise or where obvious from context.
One can have a safe version of a nice subset of fmt.Print* functionality using the log package with some small initial setup.
Everything fmt does falls back to w.Write() as can be seen here. Because there's no locking around it, everything falls back to the implementation of Write(). As there is still no locking (for Stdout at least), there is no guarantee your output will not be mixed.
I'd recommend using a global log routine.
Furthermore, if you simply want to log data, use the log package, which locks access to the output properly.
See the implementation for reference.
The common methods (fmt.printLine) are not safe. However, there are methods that are.
log.Logger is "goroutine safe": https://golang.org/pkg/log/#Logger
Something like this will create a stdout logger that can be used from any go routine safely.
logger := log.New(os.Stdout, "", 0)
I have a situation where 2 different processes(mine C++, other done by other people in JAVA) are a writer and a reader from some shared data file. So I was trying to avoid race condition by writing a class like this(EDIT:this code is broken, it was just an example)
class ReadStatus
{
bool canRead;
public:
ReadStatus()
{
if (filesystem::exists(noReadFileName))
{
canRead = false;
return;
}
ofstream noWriteFile;
noWriteFile.open (noWriteFileName.c_str());
if ( ! noWriteFile.is_open())
{
canRead = false;
return;
}
boost::this_thread::sleep(boost::posix_time::seconds(1));
if (filesystem::exists(noReadFileName))
{
filesystem::remove(noWriteFileName);
canRead= false;
return;
}
canRead= true;
}
~ReadStatus()
{
if (filesystem::exists(noWriteFileName))
filesystem::remove(noWriteFileName);
}
inline bool OKToRead()
{
return canRead;
}
};
usage:
ReadStatus readStatus; //RAII FTW
if ( ! readStatus.OKToRead())
return;
This is for one program ofc, other will have analogous class.
Idea is:
1. check if other program created his "I'm owner file", if it has break else go to 2.
2. create my "I'm the owner" file, check again if other program created his own, if it has delete my file and break else go to 3.
3. do my reading, then delete mine "I'm the owner file".
Please note that rare occurences when they both dont read or write are OK, but the problem is that I still see a small chance of race conditions because theoretically other program can check for the existence of my lock file, see that there isnt one, then I create mine, other program creates his own, but before FS creates his file I check again, and it isnt there, then disaster occurs. This is why I added the one sec delay, but as a CS nerd I find it unnerving to have code like that running.
Ofc I don't expect anybody here to write me a solution, but I would be happy if someone does know a link to a reliable code that I can use.
P.S. It has to be files, cuz I'm not writing entire project and that is how it is arranged to be done.
P.P.S.: access to data file isn't reader,writer,reader,writer.... it can be reader,reader,writer,writer,writer,reader,writer....
P.P.S: other process is not written in C++ :(, so boost is out of the question.
On Unices the traditional way of doing pure filesystem based locking is to use dedicated lockfiles with mkdir() and rmdir(), which can be created and removed atomically via single system calls. You avoid races by never explicitly testing for the existence of the lock --- instead you always try to take the lock. So:
lock:
while mkdir(lockfile) fails
sleep
unlock:
rmdir(lockfile)
I believe this even works over NFS (which usually sucks for this sort of thing).
However, you probably also want to look into proper file locking, which is loads better; I use F_SETLK/F_UNLCK fcntl locks for this on Linux (note that these are different from flock locks, despite the name of the structure). This allows you to properly block until the lock is released. These locks also get automatically released if the app dies, which is usually a good thing. Plus, these will let you lock your shared file directly without having to have a separate lockfile. This, too, work on NFS.
Windows has very similar file locking functions, and it also has easy to use global named semaphores that are very convenient for synchronisation between processes.
As far as I've seen it, you can't reliably use files as locks for multiple processes. The problem is, while you create the file in one thread, you might get an interrupt and the OS switches to another process because I/O is taking so long. The same holds true for deletion of the lock file.
If you can, take a look at Boost.Interprocess, under the synchronization mechanisms part.
While I'm generally against making API calls which can throw from a constructor/destructor (see docs on boost::filesystem::remove) or making throwing calls without a catch block in general that's not really what you were asking about.
You could check out the Overlapped IO library if this is for windows. Otherwise have you considered using shared memory between the processes instead?
Edit: Just saw the other process was Java. You may still be able to create a named mutex that can be shared between processes and used that to create locks around the file IO bits so they have to take turns writing. Sorry I don't know Java so no I idea if that's more feasible than shared memory.
I'm working with two independent c/c++ applications on Windows where one of them constantly updates an image on disk (from a webcam) and the other reads that image for processing. This works fine and dandy 99.99% of the time, but every once in a while the reader app is in the middle of reading the image when the writer deletes it to refresh it with a new one.
The obvious solution to me seems to be to have the reader put some sort of a lock on the file so that the writer can see that it can't delete it and thus spin-lock on it until it can delete and update. Is there anyway to do this? Or is there another simple design pattern I can use to get the same sort of constant image refreshing between two programs?
Thanks,
-Robert
Try using a synchronization object, probably a mutex will do. Whenever a process wants to read or write to a file it should first acquire the mutex lock.
Yes, a locking mechanism would help. There are, unfortunately, several to choose from. Linux/Unix e.g. has flock (2), Windows has a similar (but different) mechanism.
Another (somewhat hacky) solution is to just write the file under a temporary name, then rename it. Many filesystems guarantee that a rename is atomic, so this may work. This however depends on the fs, so it's a bit hacky.
If you are willing to go with the Windows API, opening the file with CreateFile and passing in 0 for the dwShareMode will not allow any other application to open the file.
From the documentation:
Prevents other processes from opening a file or device if they
request delete, read, or write access.
Then you'd have to use ReadFile, WriteFile, CloseFile, etc rather than the C standard library functions.
Or, as a really simple kludge, the reader creates a temp file (says, .lock) before starting reading and deletes it afterwards. The write doesn't manipulate the file so long as .lock exists.
That's how Open Office does it (and others) and it's probably the simplest to implement, no matter which platform.
Joe, many solutions have been proposed; I commented on some of them but I'd like to chime in with an overall view and some specifics and recommendations:
You have the following options:
use filesystem locking: under Windows have both the reader and writer open (and create with the CREATE_ALWAYS disposition, respectively) the shared file in OF_SHARE_EXCLUSIVE mode; have both the reader and writer ready to handle ERROR_SHARING_VIOLATION and retry after some predefined period of time (e.g. 250ms)
use file renaming to essentially transfer file ownership: have the writer create a writer-private file (e.g. shared_file.tmpwrite), write to it, close it, then make it publicly available to the reader by renaming it to an agreed-upon "public" name (e.g. simply shared-file); have the reader periodically test for the existence of a file with the agreed-upon "public" name (e.g. shared-file) and, when one is found, attempt to first rename it to a reader-private name (e.g. shared_file.tmpread) before having the reader open it (under the reader-private name); under Windows use MOVEFILE_REPLACE_EXISTING; the rename operation does not have to be atomic for this to work
use other forms of interprocess communication (IPC): under Windows you can create a named mutex, and have both the reader and writer attempt to create (the existing mutex will be returned if it already exists) then acquire the named mutex before opening the shared file for reading or writing
implement your own filesystem-backed locking: take advantage of open(O_CREAT|O_EXCL) or, under Windows, of the CREATE_NEW disposition to atomically create an application lock file; unlike OF_SHARE_EXCLUSIVE approach above, it would be up to you to deal with stale lock files (i.e. lock files left by a process which did not shut down gracefully such as after a crash.)
I would implement method 1.
Method 2 would also work, but it is in a sense reinventing the wheel.
Method 3 arguably has the advantage of allowing your reader process to wait on the writer process and vice-versa, eliminating the need for the arbitrary sleep delays between the retries of methods 1 and 2 (polling); however, if you are OK with polling then you should still use method 1
Method 4 is listed for completeness only, as it is complex to implement (when the lock file is detected to be stale, e.g. by checking whether the PID contained therein still exists, multiple processes can potentially be competing for its removal, which introduces a race condition requiring a second lock, which in turn can become stale etc. etc., e.g.:
process A creates the lock file but dies without removing the lock file
process A restarts and tries to acquire the lock file but realizes it is stale
process B comes out of a sleep delay and also tries to acquire the lock file but realizes it is stale
process A removes the lock file, which it knew to be stale, and recreates it essentially reacquiring the lock
process B removes the lock file, which it (still) thinks is stale (although at this point it is no longer stale and owned by process A) -- violation
Instead of deleting images, what about appending them to the end of the file? This would allow you to keep adding to the file while the reader is still operating without destroying the file. The reader can then delete the image when it's done with it (provided it is necessary) and move onto the next image. Or, the other option would be store the image in a buffer, for writing, and you test the file pointer. If it's set to the head of the file then you can go ahead and write from the buffer to the file. Otherwise, wait until reader finishes and puts the pointer back at the head of the file.
couldn't you store a few images? ('n' sounds like a good number :-)
Not too many to fill your disk, but surely 3 would be enough? if not, you are writing faster than you can process and have a fundamental problem anyhoo (tune to discover 'n').
Cyclically overwrite.