I have a working Linux image stored in NOR flash. Every time I edit some file on this filesystem changes are stored permanently in NOR flash thus causing wearing out memory with time. I'm wondering is that possible to store image on NOR flash that will be read only and during boot will be copied to RAM, like e.g fitImage does? Even if I modify rootfs stored in RAM, after system reboot everything backs to its original form. How to achieve such a behavior?
Thanks.
Related
Using the Windows API, I'm trying to write a program to read data from a disk. I managed to get access to the content of the drive using CreateFile and I'm able to search through it. Let's say there are some files on that disk and I know their paths, but I'm actually interested in their physical location.
My question is:
Is it possible to retrieve the physical location or address of the files (or sector they're located in) and where are they stored on the drive without searching the whole drive? If so, what functions should I use? Using SetFilePointer or FindFirstFile don't seem to solve the solution either.
The whole point of any file system is to abstract the physical disk sectors and provide you a higher level abstraction (called files). So the answer to "Is it possible to retrieve the physical location" should be no! (in general); some code might even move the sectors of a file (e.g. a disk defragmenter and you could imagine it is running concurrently with your program, even if that is not recommended..)
For more, read wikipages on file systems and files, then read a good book such as Operating systems: Three Easy Pieces
Notice that by using files, you are expecting that your program behave similarly after having moved a file system into a different disk, provided the file paths, contents, and metadata remain the same. In particular, you could have two external USB disks enclosures with different geometries or capacities having the same file contents (perhaps even in different file systems, e.g. VFAT on one and NTFS on another), and you then expect your program to behave identically when accessing such files (in the first box or the second one). Whatever box is plugged, your program would (for example) access the same F:\MyDir\MyFile.dat file. As file systems, both boxes would appear identical. At the physical sector level, data would be organized very differently.
BTW, the physical organization of files inside a file system varies greatly from one file system to another one. You could use some Ext3 file system on your machine (since there are Ext3 drivers for Windows) - and that is actually useful to share some data between Linux & Windows on a dual boot PC -, and the file organization is different from a FAT one or a NTFS one.
You might get some way to query the kernel to get the actual physical sector location. But I am not sure it works for all file systems (what would be the meaning of a sector location for some remote NFS one). And that information could be stale before your program get it (e.g. if some defragmenter is working in parallel). Also, other processes could access and modify the same file system at the same time (so that meta data -e.g. the sector location- would be obsolete by the time your process is scheduled to run again).
On Windows and on Unix like systems, file system code runs in the kernel. And other processes could use that same code (and the same file system) while your process is not running. Both Windows and Unix have preemptive scheduling, so you have no guarantee that your process runs again in user mode before some other process is using the same file system.
Remember that in practice, your file data often stays in the page cache. And that is why you might not hear your disk working -if you still have a rotating hard disk- when accessing the same file several times in a row (e.g. running the same program on the same file twice, a few seconds apart; usually the second run is keeping the disk silent, because the file data is already in RAM).
In a comment you mention that you want
To watch the data of the file and for example see what happens to the data when it gets deleted or modified.
but that should work at the file system level. Linux has inotify(7) facilities for that (they work on most local file systems, e.g. Ext4 or BTRFS, but not on remote file systems à la nfs(5), and neither on pseudo file systems à la proc(5)). I don't know if Windows has something similar to Linux inotify (but probably yes, at least in some cases).
You probably should consider using some database (maybe as simple as sqlite), and perhaps you want ACID properties (then use some real RDBMS like PostGreSQL). With PostGreSQL you might use TRIGGERs to be aware that some data changed, even if some other program changes the same database.
You could also do some file locking, and adopt the convention that every program accessing your particular file should lock it appropriately.
I'm developing c++ WinAPI program to write data on my USB drive directly with CreateFile by the Volume{GUID} and WriteFile functions as dozens of examples do. As mentioned here in Remarks:
A write on a volume handle will succeed if the volume does not have a
mounted file system, or if one of the following conditions is true:
The sectors to be written to are boot sectors.
The sectors to be written to reside outside of file system space.
You have explicitly locked or dismounted the volume by using FSCTL_LOCK_VOLUME or FSCTL_DISMOUNT_VOLUME.
The volume has no actual file system. (In other words, it has a RAW file system mounted.)
I want write 100Mb of data on USB smoothly without any unmounting. So I've tried two of the cases from above.
The second case: writing outside of the file system (I've extended the number of sectors per partition without extending FAT32 table), but it doesn't work without unmounting!
The fourth case: write on the volume without any file system (unformatted). But it also doesn't work without unmounting!!!
Also I've tried to create the second partition (that is invisible for Windows) with/without file system and write directly there by the offset from the end of first partition, but also unsuccessfully: I can not read or write there.
So, if anybody knows ANSWER TO THE ONE OF THE QUESTIONS at least:
How can I create sectors inside the partition outside
the file system?
Is there an ability to write smoothly on USB directly?
Also can I write in Windows to the second partition on USB without
playing around with drivers, NTCreateFile and other deep functions?
I'm pretty sure it actually worked, but you don't realize that you're bypassing Windows. That means Windows will not have noticed what you did. So if you use Windows to check what you did, it will not report a change.
To address the individual sub-questions: A normal file system fills the entire partition, so you can't. Your functions write smoothly to USB, that's not the problem. And no, Windows normally treats USB as unpartitioned storage.
I'm testing my Windows local service in a situation when the system was previously suspended (or, put into Sleep mode) and then if it suddenly loses power (without going out from suspension.)
For simplicity, at the moment, I'm testing it using a virtual machine (VMWare Workstation 10) and their "Reset" option that is supposed to simulate it:
And what I am observing is somewhat strange. For instance, when I log the data, that was supposed to be saved into system registry before the system is suspended, and then check that same registry key after the system is booted back up (after the power reset) the data in the system registry seems to be missing. Or the value of the key is just not there. While my log (which is just a text file) has everything saved correctly.
So I'm curious if it's something specific to Windows System Registry, or some bug in the VMWare software?
PS. The OS that I'm testing it on is Windows 7.
It's not specific to the Registry. You have to understand that "the Registry" is actually an abstraction. It's a shared database, backed by multiple files with non-trivial structure. The shared abstraction lives in RAM and doesn't mirror the disk structure directly.
On the other hand, your file is almost certainly not shared. File access, the file cache and virtual memory are pretty well integrated. Your write initially ends in the file cache (RAM). When you suspend your PC, Windows isn't going to copy the **file* cache to the hibernation file. That's a bit pointless - the dirty file cache contents can be written out to disk, and the clean part can be discarded outright.
I have a large storage device (flash memory) plugged onto my computer via the PCIe bus, I want to access such device directly, i.e., without any file system (e.g., NTFS or ext4) on it.
How can I do this using C/C++? (on both Windows 7 and Linux)
I am wondering if I can 1) open the device just as a file, and then read and write binary data to it, or 2) allocate the whole device using some function like malloc, then each byte on the device have an address so that I can access them based on the addresses.
I prefer the second way if it possible, but I don't know if the OS supports this since it seems the address space needs to be shared with the main memory.
According to Microsoft documentation:
On Windows you can open a physical drive using CreateFile using a path of the form
\\.\PhysicalDriveN
where N is the device number or a logical drive using a path of the form
\\.\X:
You will need to seek, read and write in multiples of the sector size which can be retrieved using DeviceIoControl() with IOCTL_DISK_GET_DRIVE_GEOMETRY.
On Linux each storage device ends up getting a device entry in /dev. The first storage device is typically /dev/sda, the second storage device, if one is present, is /dev/sdb. Note that an optical disk is a storage device, so a CD-ROM or a DVD-ROM drive, if one is present, would get a device node entry.
Some Linux distributions may use a different naming convention, but this is what it usually is. So, you'll need to figure out which device corresponds to your flash disk, and just open the /dev/sdX device, and simply read and write from it. Your reads and writes must be for even block (sector) sizes, and seeking the opened file governs which disk blocks/sectors the subsequent read or write will affect.
Generally, /dev/sdX will be owned by root, but there are usually some Linux distribution-specific ways to fiddle the userid that owns a particular device node.
G'Day!
I have an executable (Unix or Windows - it should be cross-compiling). If one opens this executable by any editor and write some stuff to the end - the application would still run perfect.
On execution, the application with all its data loads to the RAM. So, the user-written part of file is also loaded into memory.
Is there any chance to read this data?
I need this data in fast access. Other workarounds are not OK, because it takes too much time:
Reading directly from file (on hard disk) or mapping it is not fine, because the application have to read this file on each run, but this application has lots of launches per sec.
Using shared memory with another process (something like server, which holds data) is not cross-compiling
Using pipes between app and so-called server is not fast enough, imho.
That's why I decided to write some stuff to the end of application.
Thanks in advance!
Are you re-inventing
exe packers (see http://en.wikipedia.org/wiki/Executable_compression)
embedded resources? A portable approach was described here Is there any standard way of embedding resources into Linux executable image?
I also think you're might be optimizing the wrong things.
Reading directly from file (on hard disk) or mapping it is not fine, because the application have to read this file on each run, but this application has lots of launches per sec.
The kernel[1] is way smarter than we are and is perfectly capable of caching the mapped stuff. Heck, if you map it READ-ONLY there will be no difference with directly accessing data from your program's base image.
[1]: this goes for both WIndows and Unix