I know how to register and mount a file system using a kernel module. Now I want to do the same from a normal c/c++ program. Just overriding file and inode operations for a single folder would be even better. I know there are libraries like FUSE but i think all they do is add a kernel which somehow communicates with the normal program. Is there any way to do this?
Yes, the code for the ext2/3/4 file system is also available as a library.
Not that it's really necessary. If you're just toying around, ext2 is simple enough to implement yourself. On Linux you can just unmount a filesystem and then access the partition as if it was one big file. (Got to be root for this, of course). I'd recommend getting a USB stick for this, though.
The idea of "overriding file and inode operations for one folder" suggests that you want to share access with the OS, though. That does not work. There is one party in control, and that's either your program or the OS, not both.
You can get somewhat close by replacing that one folder with a symlink to your own filesystem, though, but that would have to be a FUSE filesystem. (Assuming you want to stick to user mode - FUSE is how the kernel talks to user mode file systems)
Related
Using the Windows API, I'm trying to write a program to read data from a disk. I managed to get access to the content of the drive using CreateFile and I'm able to search through it. Let's say there are some files on that disk and I know their paths, but I'm actually interested in their physical location.
My question is:
Is it possible to retrieve the physical location or address of the files (or sector they're located in) and where are they stored on the drive without searching the whole drive? If so, what functions should I use? Using SetFilePointer or FindFirstFile don't seem to solve the solution either.
The whole point of any file system is to abstract the physical disk sectors and provide you a higher level abstraction (called files). So the answer to "Is it possible to retrieve the physical location" should be no! (in general); some code might even move the sectors of a file (e.g. a disk defragmenter and you could imagine it is running concurrently with your program, even if that is not recommended..)
For more, read wikipages on file systems and files, then read a good book such as Operating systems: Three Easy Pieces
Notice that by using files, you are expecting that your program behave similarly after having moved a file system into a different disk, provided the file paths, contents, and metadata remain the same. In particular, you could have two external USB disks enclosures with different geometries or capacities having the same file contents (perhaps even in different file systems, e.g. VFAT on one and NTFS on another), and you then expect your program to behave identically when accessing such files (in the first box or the second one). Whatever box is plugged, your program would (for example) access the same F:\MyDir\MyFile.dat file. As file systems, both boxes would appear identical. At the physical sector level, data would be organized very differently.
BTW, the physical organization of files inside a file system varies greatly from one file system to another one. You could use some Ext3 file system on your machine (since there are Ext3 drivers for Windows) - and that is actually useful to share some data between Linux & Windows on a dual boot PC -, and the file organization is different from a FAT one or a NTFS one.
You might get some way to query the kernel to get the actual physical sector location. But I am not sure it works for all file systems (what would be the meaning of a sector location for some remote NFS one). And that information could be stale before your program get it (e.g. if some defragmenter is working in parallel). Also, other processes could access and modify the same file system at the same time (so that meta data -e.g. the sector location- would be obsolete by the time your process is scheduled to run again).
On Windows and on Unix like systems, file system code runs in the kernel. And other processes could use that same code (and the same file system) while your process is not running. Both Windows and Unix have preemptive scheduling, so you have no guarantee that your process runs again in user mode before some other process is using the same file system.
Remember that in practice, your file data often stays in the page cache. And that is why you might not hear your disk working -if you still have a rotating hard disk- when accessing the same file several times in a row (e.g. running the same program on the same file twice, a few seconds apart; usually the second run is keeping the disk silent, because the file data is already in RAM).
In a comment you mention that you want
To watch the data of the file and for example see what happens to the data when it gets deleted or modified.
but that should work at the file system level. Linux has inotify(7) facilities for that (they work on most local file systems, e.g. Ext4 or BTRFS, but not on remote file systems à la nfs(5), and neither on pseudo file systems à la proc(5)). I don't know if Windows has something similar to Linux inotify (but probably yes, at least in some cases).
You probably should consider using some database (maybe as simple as sqlite), and perhaps you want ACID properties (then use some real RDBMS like PostGreSQL). With PostGreSQL you might use TRIGGERs to be aware that some data changed, even if some other program changes the same database.
You could also do some file locking, and adopt the convention that every program accessing your particular file should lock it appropriately.
We manage an legacy system from the early 80s. It is awkward to access the file system on it, and we're therefore thinking on methods to work with the files from Windows.
We have looked at two methods for doing this:
Create a user space file system (eg with Callback File System).
Looks to me like the best solution, but it is expensive
Create a CIFS/SMB server to share the files on the network and mount using "net use"
I'm not sure if this option even is possible
I have not found any good examples of how I write an SMB server. Maybe there's a reason for that?
My questions are:
Is the method two possible?
Is it possible to mount a CIFS/SMB share locally if there is no network connection?
What are the pros and cons of the two methods?
Edit
The legacy system runs under Windows in a house developed emulator (which works in much the same way as VMware). It is in this emulator we want to expand the functionality to distribute the file system in the legacy system to Windows.
The legacy system is totally unique. Both the operating system and file system is house developed (and very odd).
I need to merge content of multiple NTFS folders into one based on some rules.
These rules will ensure that there is no conflicting names. The goal is to do all that programatically (c# and/or c++).
The simplest solution I had is to create NTFS Link (or Junction) into the target folder pointing to each file in the source folder.
This would work so far, but the problem I have is that additional files will not occur and removed files will not disappear.
Well, of course I can run a background process (service) listening at the file system's source folders and performing appropriate modifications of NTFS Links on thy fly, but is this the way to go?
The first question is:
1. What is the most elegant way of doing that?
2. I have seen multiple programs which for instance can mount content of a ZIP or ISO file to some NTFS folder. How do they work?
Here is one project that enables you to create a usermode file system in Window: Dokan:
When you want to create a new file system on Windows, for example to
improve FAT or NTFS, you need to develop a file system driver.
Developing a device driver that works in kernel mode on windows is
extremely difficult. By using Dokan library, you can create your own
file systems very easily without writing device driver.
Mapping something to an existing folder is possible in two ways:
Filesystem filter driver filters FS requests to the existing folder and shows modified content of the folder.
Filesystem driver creates a virtual filesystem and mounts it as a junction point on NTFS drive.
Both methods require a corresponding kernel-mode driver.
Applications that expose a ZIP (or other similar file) use one of the above methods. ISO files, being images of the disk, can be exposed directly as virtual disks and also mounted to NTFS folder. In the latter case kernel-mode disk driver is needed.
Your particular task can be accomplished using our Callback File System (CBFS) product. CBFS offers a pre-created kernel-mode driver and lets you write your filesystem-related code in user mode. CBFS includes Mapper sample that does almost what you are looking for - it shows contents of an existing folder as a new virtual filesystem, which you can mount to NTFS folder (CBFS supports this). Non-commercial licenses are available for public non-commercial projects. APIs are offered for .NET, C++, Java and Delphi.
I am writing (yet another) file manager (to learn stuff:) and have a silly/stupid block.
On Linux, to enumerate the storage devices which can contain files, I believe the best approach is to parse the contents of the /proc/partitions file and extract the /dev/sda* entries. (right?) However, how can I map the /dev/sda* to something that I can explore programmatically to get directory contents? I am planning on using boost/filesystem, but since I cannot ls /dev/sda I assume I cannot use boost to iterate over it.
Synopsis: how can I convert /dev/sda* to something that I can 'ls'
I think you're mis-understanding exactly what /dev/sd* actually are to a program. They are devices not directories. You use the mount command to tell the operating system to "interpret" the device as a filesystem, and to attach it somewhere (root, or otherwise). It's this step that makes it into "a directory" somewhere on your filesystem. So other than raw I/O commands (which you don't want to do), get the filesystem mounted, and THEN try and explore it.
It's kind of like opening a file really. When you do this, the operating system gives your program a stream of bytes that you can randomly access the file through. But on the disk, that file could actually be scattered all over the hard drive (or whatever device). But the OS is "making" it into a "nice" format for you to deal with transparently. The same is true of the disk itself when accessing directory/file listings.
I hope my example made it clearer as to why what you're trying to do isn't as simple as you think it is.
Off the cuff, I'd say that the output of mount with no arguments might be a faster choice. That should show you the mounted filesystems and devices while /proc would show you all the devices and partitions.
the device /dev/sda* is a block device and needs to be mounted. To be able to ls it you need to have something that can interpret the file system type. First step: identify the file system type, in raw code there is usually a header code in the partition table in the first segment of the harddrive which would just be /dev/sda On a Linux system it would be something like ext3
Next you need to either write or use a library for interfacing with that filesystem, if you get the Kernel source code for Linux it has a LOT of API code for interfacing with common filesystems, and wrappers for standard POSIX calls which is exactly what you're looking for. Things like ls and cwd use system calls to retrieve information about a mounted filesystem, the disk is a block (or sometimes a character) device and you need the ability to talk to it and speak the same language.
I want to make an application in qt 4.7.4 where I need to use shared folders in Windows.
I have searched around and found that I can use QDir(hostName), but there is a problem:
I can't control the bandwidth used. I'd like to be able to set the max transfer rate when
accessing a host in the network.
More about the app: I have a folder on my computer whose content I wish to be identical on all the hosts(who's name will be provided from a file) in my network . For example if I add a new file in that folder and run the app, it should start copying the file in the hosts with the max bandwidth used specified by me.
Thank you in advance for your help.
I suppose that bandwidth manages by operating system drivers or services like QoS or firewalls. QDir use only interface provided by operating system to access directory. It works fine with shared dir only because of Windows shared dir behaves the same way as usual dir.
By the way you may try control bandwidth manually by implementing your own file copy procedure. Sorry if my solution too naive or doesn't fit. You may open file (see QFile) in shared directory for read-only and open file on local machine in write-only. Next you start reading portions of data from shared file by timer (see QTimer). So you can specify how often you want to copy portions of data (QTimer parameter) and how many bytes of data you copy per time (QIODevice::read function parameter).