Here is the problem: I monitor a directory using Win32 API ReadDirectoryChangesW function. And I need to distinguish between newly created files and modified files. But there are problems... as always :(
Cases:
I monitor directory for new/modify (FILE_NOTIFY_CHANGE_FILE_NAME | FILE_NOTIFY_CHANGE_SIZE). Problem: After file creation, new file event + modify file event is triggered. But i need only one. How can I avoid that? When file is modified I get what I want :).
I monitor directory only for new file (FILE_NOTIFY_CHANGE_FILE_NAME) - NO PROBLEM.
I monitor directory only for modify file (FILE_NOTIFY_CHANGE_SIZE). Problem: When a new file is, modify action is fired along with file creation event. How can I avoid that?
Of course, I implemented some workarounds. But, I want to know if there any elegant way of handling the problems I described.
You should be catching FILE_NOTIFY_CHANGE_LAST_WRITE, not FILE_NOTIFY_CHANGE_SIZE, for a modified file. Files may be modified without the size changing.
You should also keep a queue of changes and the time they happened and only process the queue after there have been no changes in the past 1-2 seconds. Some applications can do very strange things when creating or modifying files, and you'll most likely want to special case for popular applications if you plan on using this code in the wild.
ReadDirectoryChanges isn't one of the friendliest winapi functions. You probably can't get around receiving two events on file creation; I'm not completely sure whether you'll get an extra modify for FILE_NOTIFY_CHANGE_LAST_WRITE on creation, but I think you probably will. Using the queue approach will allow you to easily throw out the extra event if it has the same time stamp as the creation event.
Related
I want to make an C++ application that will monitor file changes in its directory or its sub-directories. I know inotify can monitor single level directory changes and i need to manually add a watch for each sub-directory to detect changes in the sub-directory.
I need to know if there is any other way to recursively monitor changes in a directory in Linux other than inotify.
To monitor recursively a directory you have to:
Create an inotify(7) object with inotify_init(2) or inotify_init2(2).
Descend recursively on your directory, using inotify_add_watch(2) for all the nodes you want to be notified on (add the watch for the directory itself before scanning, or you'll lose events ---se below).
Wait for events to come on the inotify descriptor you have received.
Take into account that directory creation forces you to possible rescan of directory contents, as you can get an event for a new subdirectory on a watched directory, but files can be created on it before you had a chance to add watches for the recently created directory, so you will not be informed of that files creation.
For this last reason, you'll need to consider putting all things in a subroutine and to call it each time a new directory gets created. This way perhaps you'll scan files for which you'll receive creation events, but the other side you can lost events.
Also, you must prepare yourshelf to do a complete tree rescan in case you lose some events (in a full queue overflow).
And believe me, this is by far more efficient than to do it the classic way. And you can bypass short lived files between rescans.
The reason there's not a recursive solution to this problem has been pointed above in one comment (You'll need the kernel to do the search for you, even if you are not interested on it)
Yes there is the old classic way: Simply check the directory (recursively) in intervals. Store a list of all files and directories and their modification dates. Then it's easy to see if you have a new file/directory, if one has been removed, or if otherwise modified.
It is time consuming though, and you need to store data about every file and directory, so if you try to do it on the root directory you will use a lot of memory or storage.
Probably you want to mix inotify and dnotify to achieve that.
I wish to know when another application finishes editing a file.
I am aware of FindFirstChangeNotification, ReadDirectoryChangesW and the FileSystemWatcher class, however I think these are only able to detect file creation and changes. Which won't allow me to know whether the file still has content which will be written to it in future or not. One solution would be to wait to ensure that all data has been written, but I feel that there should be a better way.
I wish to know when the process writing to the file closes the handle it has, I think writing a File System Filter Driver may allow for this, can anyone confirm? Or provide another method I which I could use. Also I would prefer not to rely on something like .net but I wouldn't rule it out.
Thanks very much, in advance.
You can poll this by trying to open the file in exclusive mode.
On windows you can use OpenFile with OF_SHARE_EXCLUSIVE.
Alright so to start this is strictly for Windows and I'd prefer to use C++ over .NET but I'm not opposed to boost::filesystem although if it can be avoided in favor of straight Windows API I'd prefer that.
Now the scenario is an application on another machine I can't change is going to create files in a particular directory on the machine that I need to make backups of and do some extra processing. Currently I've made a little application which will sit and listen for change notifications in a target directory using FindFirstChangeNotification and FindNextChangeNotification windows APIs.
The problem is that while I can get notified when new files are created in the directory, modified, size changes, etc it only notifies once and does not specifically tell me which files. I've looked at ReadDirectoryChangesW as well but it's the same story there except that I can get slightly more specific information.
Now I can scan the directory and try to acquire locks or open the files to determine what specifically changed from the last notification and whether they are available for further use but in the case of copying a large file I've found this isn't good enough as the file won't be ready to be manipulated and I won't get any other notifications after the first so there is no way to tell when it's actually done copying unless after the first notification I continually try to acquire locks until it succeeds.
The only other thing I can think of that would be less hackish would be to have some kind of end token file but since I don't have control over the application creating the files in the first place I don't see how I'd go about doing that and it's still not ideal.
Any suggestions?
This is a fairly common problem and one that doesn't have an easy answer. Acquiring locks is one of the best options when you cannot change the thing at the remote end. Another I have seen is to watch the file at intervals until the size doesn't change for an interval or two.
Other strategies include writing a no-byte file as a trigger when the main file is complete and writing to a temp directory then moving the complete file to the real destination. But to be reliable, it must be the sender who controls this. As the receiver, you are constrained to watching the directory and waiting for the file to settle.
It looks like ReadDirectoryChangesW is going to be your best bet. For each file copy operation, you should be receiving FILE_ACTION_ADDED followed by a bunch of FILE_ACTION_MODIFIED notifications. On the last FILE_ACTION_MODIFIED notification, the file should no longer be locked by the copying process. So, if you try to acquire a lock after each FILE_ACTION_MODIFIED of the copy, it should fail until the copy completes. It's not a particularly elegant solution, but there doesn't seem to be any notifications available for when a file copy completes.
You can process the data once the file is closed, right? So the task is to track when the file is closed. This can be done using file system filter driver. You can write your own or you can use our CallbackFilter product.
I'm trying to fulfill a client request here, and I'm not entirely sure I can actually do it. I have an MFC application that relies upon ShellExecute to open files in their appropriate viewer, which spawns multiple viewers if you try to open multiple files one after the other. If you open one .txt document, and then open another, two copies of notepad appear as expected.
The client wants us to change this functionality - Windows' functionality - to just pass file locations to any viewers that might already be opening. The first image clicked should open Image Viewer, but any other should just be opened in that existing process.
Is that kind of inter-application control/communication possible? Can I generically "pass" files to existing processes? I don't think I can. Executing a program with a file as a parameter is one thing, but passing a file to a running process is something else altogether. I'm not sure you can do that generically, I don't think that kind of functionality is anywhere in the Windows APIs.
I could be wrong, though.
This isn't possible if the viewer don't support multiple open files in same instance.
in your example: notepad will launch a new version with each file, while Notepad++ (a free editor) will open in same instance in a new tab.
The first thing you should try is calling the program again with the new parameters. If the program is written in such a way it will delegate the new parameter to the existing instance. Notepad doesn't do this, image viewer may though.
The next thing you can try is managing the life of the application by keeping track of the handle yourself. You call CreateProcess, so you create and own the handle to this process. On the next call to CreateProcess, enumerate the open windows and try to find your last handle. If the handle is found, close it and continue with your open process. You should only get one open application. For the most reliable solution, put this in a separate thread and wait for the handle (as well as a new request event) to avoid any race conditions.
I'm a newbie C++ developer and I'm working on an application which needs to write out a log file every so often, and we've noticed that the log file has been corrupted a few times when running the app. The main scenarios seems to be when the program is shutting down, or crashes, but I'm concerned that this isn't the only time that something may go wrong, as the application was born out of a fairly "quick and dirty" project.
It's not critical to have to the most absolute up-to-date data saved, so one idea that someone mentioned was to alternatively write to two log files, and then if the program crashes at least one will still have proper integrity. But this doesn't smell right to me as I haven't really seen any other application use this method.
Are there any "best practises" or standard "patterns" or frameworks to deal with this problem?
At the moment I'm thinking of doing something like this -
Write data to a temp file
Check the data was written correctly with a hash
Rename the original file, and put the temp file in place.
Delete the original
Then if anything fails I can just roll back by just deleting the temp, and the original be untouched.
You must find the reason why the file gets corrupted. If the app crashes unexpectedly, it can't corrupt the file. The only thing that can happen is that the file is truncated (i.e. the last log messages are missing). But the app can't really jump around in the file and modify something elsewhere (unless you call seek in the logging code which would surprise me).
My guess is that the app is multi threaded and the logging code is being called from several threads which can easily lead to data corrupted before the data is written to the log.
You probably forgot to call fsync() every so often, or the data comes in from different threads without proper synchronization among them. Hard to tell without more information (platform, form of corruption you see).
A workaround would be to use logfile rollover, ie. starting a new file every so often.
I really think that you (and others) are wasting your time when you start adding complexity to log files. The whole point of a log is that it should be simple to use and implement, and should work most of the time. To that end, just write the log to an unbuffered stream (l;ike cerr in a C++ program) and live with any, very occasional in my experience, snafus.
OTOH, if you really need an audit trail of everything your app does, for legal reasons, then you should be using some form of transactional storage such as a SQL database.
Not sure if your app is multi-threaded -- if so, consider using Active Object Pattern (PDF) to put a queue in front of the log and make all writes within a single thread. That thread can commit the log in the background. All logs writes will be asynchronous, and in order, but not necessarily written immediately.
The active object can also batch writes.