In Linux to read File System Events, most of us use Inotify which is really a Good One. But it has it's limitations.
I have to keep track of all the directories i have to watch and get a inotify descriptor for each and every directory. Let's say i want to monitor my Whole System and it has about 10 million directories. Listing and queuing them for inotify API itself would take hours. It would eat a lot of memory also.
And i certainly can't watch directories more than that specified in max_user_watches in /etc/syscntl.conf or /proc/sys/fs/inotify/max_user_watches .
But we know that each and every file system event is monitored by Kernel and we use inotify API to register for a certain directory modifications. But how do i read fs events directly from kernel without API ..? [ In Mac OS X, i can read from /dev/fsevents , likewise if i can read from /proc or something like that, it would be great]. I'm ready to write a Kernel Module too, but please share some light on this.
Did you look at the virus scanner stuff? Ie, Fanotify, also fsnotify and hooked to dnotify and inotify. This depends on kernel version; present in some form since approximately 2.6.36 (early 2011).
Some kernel files,
proc documentation
fsnotify.h
fsnotify_backend.h
fanotify.h
Lwn.net links,
API overview
Main Commit notice
Various commits
Related
Using the Windows API, I'm trying to write a program to read data from a disk. I managed to get access to the content of the drive using CreateFile and I'm able to search through it. Let's say there are some files on that disk and I know their paths, but I'm actually interested in their physical location.
My question is:
Is it possible to retrieve the physical location or address of the files (or sector they're located in) and where are they stored on the drive without searching the whole drive? If so, what functions should I use? Using SetFilePointer or FindFirstFile don't seem to solve the solution either.
The whole point of any file system is to abstract the physical disk sectors and provide you a higher level abstraction (called files). So the answer to "Is it possible to retrieve the physical location" should be no! (in general); some code might even move the sectors of a file (e.g. a disk defragmenter and you could imagine it is running concurrently with your program, even if that is not recommended..)
For more, read wikipages on file systems and files, then read a good book such as Operating systems: Three Easy Pieces
Notice that by using files, you are expecting that your program behave similarly after having moved a file system into a different disk, provided the file paths, contents, and metadata remain the same. In particular, you could have two external USB disks enclosures with different geometries or capacities having the same file contents (perhaps even in different file systems, e.g. VFAT on one and NTFS on another), and you then expect your program to behave identically when accessing such files (in the first box or the second one). Whatever box is plugged, your program would (for example) access the same F:\MyDir\MyFile.dat file. As file systems, both boxes would appear identical. At the physical sector level, data would be organized very differently.
BTW, the physical organization of files inside a file system varies greatly from one file system to another one. You could use some Ext3 file system on your machine (since there are Ext3 drivers for Windows) - and that is actually useful to share some data between Linux & Windows on a dual boot PC -, and the file organization is different from a FAT one or a NTFS one.
You might get some way to query the kernel to get the actual physical sector location. But I am not sure it works for all file systems (what would be the meaning of a sector location for some remote NFS one). And that information could be stale before your program get it (e.g. if some defragmenter is working in parallel). Also, other processes could access and modify the same file system at the same time (so that meta data -e.g. the sector location- would be obsolete by the time your process is scheduled to run again).
On Windows and on Unix like systems, file system code runs in the kernel. And other processes could use that same code (and the same file system) while your process is not running. Both Windows and Unix have preemptive scheduling, so you have no guarantee that your process runs again in user mode before some other process is using the same file system.
Remember that in practice, your file data often stays in the page cache. And that is why you might not hear your disk working -if you still have a rotating hard disk- when accessing the same file several times in a row (e.g. running the same program on the same file twice, a few seconds apart; usually the second run is keeping the disk silent, because the file data is already in RAM).
In a comment you mention that you want
To watch the data of the file and for example see what happens to the data when it gets deleted or modified.
but that should work at the file system level. Linux has inotify(7) facilities for that (they work on most local file systems, e.g. Ext4 or BTRFS, but not on remote file systems à la nfs(5), and neither on pseudo file systems à la proc(5)). I don't know if Windows has something similar to Linux inotify (but probably yes, at least in some cases).
You probably should consider using some database (maybe as simple as sqlite), and perhaps you want ACID properties (then use some real RDBMS like PostGreSQL). With PostGreSQL you might use TRIGGERs to be aware that some data changed, even if some other program changes the same database.
You could also do some file locking, and adopt the convention that every program accessing your particular file should lock it appropriately.
I'm trying to make a software that backups my entire hard drive.
I've managed to write a code for reading the raw data from hard disk sectors. However, i want to have incremental backups. For that i need to know the changed made to OS settings, file changes, everything.
My question is -
Using FileSystemWatcher and Inotify, will i be able to know every change made to every sector in the hard drive ? (OS settings etc)
I'm coding it in C++ for linux and windows.
(Saw this question on Stackoverflow which gave me some idea)
Inotify is to detect changes while your program is running, I'm guessing that FilySystemWatches is similar.
One way to solve this is to have a checksum on each sector or multiple of sectors, and when making a backup you compare the checksums to the list you have and only backup blocks that have been changed.
The MS Windows FileSystemWatcher mechanism is more limited than Linux's Inotify, but both probably will do what you need. The Linux mechanism provides (optional) notification for file reads, which causes the "access timestamp" to be updated.
However, the weakness from your application's perspective is that all file modifications made from system boot up to your program getting loaded (and unload to shutdown) will not be monitored. Your application might need to look through file modification timestamps of many files to identify changed files, depending on the level of monitoring you are targeting.
Both architectures maintain a timestamp for each file tracking when the file was last accessed. If that being updated is a trigger for a backup notification, the Windows mechanism lacking such notification will cause mismatched behavior on the platforms. Windows' mechanism can also drop notifications due to buffer size limitations. Here is a real gem from the documentation:
Note that a FileSystemWatcher does not raise an Error event when an event is missed or when the buffer size is exceeded, due to dependencies with the Windows operating system. To keep from missing events, follow these guidelines:
Increasing the buffer size with the InternalBufferSize property can prevent missing file system change events.
Avoid watching files with long file names. Consider renaming using shorter names.
Keep your event handling code as short as possible.
At least you can control two out of three of these....
I am writing (yet another) file manager (to learn stuff:) and have a silly/stupid block.
On Linux, to enumerate the storage devices which can contain files, I believe the best approach is to parse the contents of the /proc/partitions file and extract the /dev/sda* entries. (right?) However, how can I map the /dev/sda* to something that I can explore programmatically to get directory contents? I am planning on using boost/filesystem, but since I cannot ls /dev/sda I assume I cannot use boost to iterate over it.
Synopsis: how can I convert /dev/sda* to something that I can 'ls'
I think you're mis-understanding exactly what /dev/sd* actually are to a program. They are devices not directories. You use the mount command to tell the operating system to "interpret" the device as a filesystem, and to attach it somewhere (root, or otherwise). It's this step that makes it into "a directory" somewhere on your filesystem. So other than raw I/O commands (which you don't want to do), get the filesystem mounted, and THEN try and explore it.
It's kind of like opening a file really. When you do this, the operating system gives your program a stream of bytes that you can randomly access the file through. But on the disk, that file could actually be scattered all over the hard drive (or whatever device). But the OS is "making" it into a "nice" format for you to deal with transparently. The same is true of the disk itself when accessing directory/file listings.
I hope my example made it clearer as to why what you're trying to do isn't as simple as you think it is.
Off the cuff, I'd say that the output of mount with no arguments might be a faster choice. That should show you the mounted filesystems and devices while /proc would show you all the devices and partitions.
the device /dev/sda* is a block device and needs to be mounted. To be able to ls it you need to have something that can interpret the file system type. First step: identify the file system type, in raw code there is usually a header code in the partition table in the first segment of the harddrive which would just be /dev/sda On a Linux system it would be something like ext3
Next you need to either write or use a library for interfacing with that filesystem, if you get the Kernel source code for Linux it has a LOT of API code for interfacing with common filesystems, and wrappers for standard POSIX calls which is exactly what you're looking for. Things like ls and cwd use system calls to retrieve information about a mounted filesystem, the disk is a block (or sometimes a character) device and you need the ability to talk to it and speak the same language.
From kernel mode in Windows I'm able to intercept and monitor virtually all actions performed on a particular disk. When a file is opened for any purpose I get an event.
Now I want to trace which application that opened it. I think this should be possible but don't know how.
I'm using the standard file management functions in Windows Win32 API.
Thanks in advance.
/Robert
Sysinternals Filemon (free) does this, and better yet they describe how they did it:
For the Windows 9x driver, the heart
of FileMon is in the virtual device
driver, Filevxd.vxd. It is dynamically
loaded, and in its initialization it
installs a file system filter via the
VxD service,
IFSMGR_InstallFileSystemApiHook, to
insert itself onto the call chain of
all file system requests. On Windows
NT the heart of FileMon is a file
system driver that creates and
attaches filter device objects to
target file system device objects so
that FileMon will see all IRPs and
FastIO requests directed at drives.
When FileMon sees an open, create or
close call, it updates an internal
hash table that serves as the mapping
between internal file handles and file
path names. Whenever it sees calls
that are handle based, it looks up the
handle in the hash table to obtain the
full name for display. If a
handle-based access references a file
opened before FileMon started, FileMon
will fail to find the mapping in its
hash table and will simply present the
handle's value instead.
-Adam
Sysinternals did a so good job at doing it and explaining it, that some source code of old version are still available here for instance, and the code is well documented (imho). It could be a good start as well.
I would use the "handle.exe" app from Sysinternals.
Or, are you actually trying to do this programmactically?
Just use Win32 N.API to get the pid from the File handle.
It's a FAQ for 15 years...
The subject says it all - normally easy and cross platform way is to poll, intelligently. But every OS has some means to notify without polling. Is it possible in a reasonably cross platform way? (I only really care about Windows and Linux, but I use mac, so I thought posix may help?)
Linux users can use inotify
inotify is a Linux kernel subsystem
that provides file system event
notification.
Some goodies for Windows fellows:
File Change Notification on MSDN
"When Folders Change" article
File System Notification on Change
The Qt library has a QFileSystemWatcher class which provides cross platform notifications when a file changes. Even if you are not using Qt, because the source is available you could have a look at it as a sample for your own implementation. Qt has separate implementations for Windows, Linux and Mac.
There's File System Events API as of Leopard.
I don't think POSIX itself has facilities for that. The closest to cross-platform I've seen is FAM, which seems to work for Linux, BSD, and Irix, but I'm not how easy it would be to port it to Windows and MacOS.
I've actually built this system before for use in a commercial C++ code base- as long as you don't need every weird thing under the sun, the Windows and POSIX systems have a lot of overlap you can abstract.
POSIX: Use inotify- it is a whole system literally built for this job
Windows: Use "change events". You have to build more of the glue and reporting yourself (all the APIs you need are available, there's just not the 1-stop-shopping inotify gives you).
The common things you can detect in your "notification thread" for forwarding events are:
1) Basically any invasive operation boost::filesystem supports, with the (possible) exception of modifying permissions. This is things like moving, creating, deleting, copying folders and files.
2) Reads and writes to files (esp. writes). Be aware that if you're using async I/O the notifications can show up out-of-order.
3) When a new volume comes in, such as somebody connecting a flash drive.
inotify especially gives you an insane level of fine-grained control, Windows less so. With inotify you can literally monitor everything the filesystem is doing in near-real time if you really want to. I know #3 is possible with both without polling, but be aware that it can be really tricky to get it working correctly- on either system.
I believe OS X now has appropriate hooks/callbacks because they were needed for Spotlight indexing.
On linux you'll have the additional trouble that there are multiple file systems commonly used. If you need the functionality for only a limited amount of files/directories, I'd try about actively looking for modifications at regular intervals.
libevent or libev seem to be what you want, though I haven't used them.