We are continually receiving OutOfMemory exceptions when trying to download documents via a web service. We are storing the documents byte array data as part of a serialized mesage object and the original documents are all ~500kb in size. The only other thing in the message's object graph are two string properties for correlation.
We have tried retrieving the document compressed and uncompressed with the same issues. Now I believe that the exception is been raised when trying to deserialize the message and not while downloading the stream.
I know this is a vague question, but do any of you have any idea what could be causing this? The amount of data transferred here seems very small and there is little else happening on the device at the time of transfer.
Memory is VERY tight on Windows Mobile devices -- it really is a world of pain to work with. First of all, your total memory for a process is (if I remember right) 32 MB. Now that gets filled up not just with in-code memory allocations, but also loaded code from DLLs, the EXE and such.
Now when you're downloading the 500KB into a byte array, it doesn't just need 500KB. It needs 500KB of CONTIGUOUS MEMORY, something it's very unlikely to find once the app has been running for a short while.
My recommendation would be to download the files directly with an HttpRequest rather than the web service. Maybe the web service can pass back the URL that you need or something like that. But download the file directly and you can process it in chunks -- read 8K, then write that 8K to disk, then read another 8K, etc.
In order to get OutOfMemory exception the code either has a bug that consumes memory until the device has no more, or the device generally doesn't have enough memory to handle even 500Kb variations.
You say "while trying to download documents" maybe you keep them in memory and you get the exception.
Related
I'm trying to make a software that backups my entire hard drive.
I've managed to write a code for reading the raw data from hard disk sectors. However, i want to have incremental backups. For that i need to know the changed made to OS settings, file changes, everything.
My question is -
Using FileSystemWatcher and Inotify, will i be able to know every change made to every sector in the hard drive ? (OS settings etc)
I'm coding it in C++ for linux and windows.
(Saw this question on Stackoverflow which gave me some idea)
Inotify is to detect changes while your program is running, I'm guessing that FilySystemWatches is similar.
One way to solve this is to have a checksum on each sector or multiple of sectors, and when making a backup you compare the checksums to the list you have and only backup blocks that have been changed.
The MS Windows FileSystemWatcher mechanism is more limited than Linux's Inotify, but both probably will do what you need. The Linux mechanism provides (optional) notification for file reads, which causes the "access timestamp" to be updated.
However, the weakness from your application's perspective is that all file modifications made from system boot up to your program getting loaded (and unload to shutdown) will not be monitored. Your application might need to look through file modification timestamps of many files to identify changed files, depending on the level of monitoring you are targeting.
Both architectures maintain a timestamp for each file tracking when the file was last accessed. If that being updated is a trigger for a backup notification, the Windows mechanism lacking such notification will cause mismatched behavior on the platforms. Windows' mechanism can also drop notifications due to buffer size limitations. Here is a real gem from the documentation:
Note that a FileSystemWatcher does not raise an Error event when an event is missed or when the buffer size is exceeded, due to dependencies with the Windows operating system. To keep from missing events, follow these guidelines:
Increasing the buffer size with the InternalBufferSize property can prevent missing file system change events.
Avoid watching files with long file names. Consider renaming using shorter names.
Keep your event handling code as short as possible.
At least you can control two out of three of these....
I downloaded some flash games some time ago to play offline at school if I find myself with no work to do and have a spare. The only problem is that they always attempt to save/load from the application data, and I don't believe the files are created at all, let alone loaded next time I would play it again.
Instead of decompiling each one and finding/changing the save and load location I thought about the Windows API, hooks in particular. Perhaps it may be possible to monitor system requests to create such a file, or open it if it's trying to load, and then tell it to look elsewhere (i.e. my flash drive)?
I don't know if the idea is actually plausible or not, but any solutions, or a solid "It's impossible" would be great.
I hope some of these help:
WinAPIOverride32 is an advanced api monitoring software.
Deviare API
Deconstructing the Poker Client, Part 1
G'Day!
I have an executable (Unix or Windows - it should be cross-compiling). If one opens this executable by any editor and write some stuff to the end - the application would still run perfect.
On execution, the application with all its data loads to the RAM. So, the user-written part of file is also loaded into memory.
Is there any chance to read this data?
I need this data in fast access. Other workarounds are not OK, because it takes too much time:
Reading directly from file (on hard disk) or mapping it is not fine, because the application have to read this file on each run, but this application has lots of launches per sec.
Using shared memory with another process (something like server, which holds data) is not cross-compiling
Using pipes between app and so-called server is not fast enough, imho.
That's why I decided to write some stuff to the end of application.
Thanks in advance!
Are you re-inventing
exe packers (see http://en.wikipedia.org/wiki/Executable_compression)
embedded resources? A portable approach was described here Is there any standard way of embedding resources into Linux executable image?
I also think you're might be optimizing the wrong things.
Reading directly from file (on hard disk) or mapping it is not fine, because the application have to read this file on each run, but this application has lots of launches per sec.
The kernel[1] is way smarter than we are and is perfectly capable of caching the mapped stuff. Heck, if you map it READ-ONLY there will be no difference with directly accessing data from your program's base image.
[1]: this goes for both WIndows and Unix
can I change the ECC code for a block of a file stored on a flash drive by any means ?
of a file stored on a HDD (though I don't think there would be a difference between the two)
Maybe , through some hardware interrupts or anything like that?
Also I need the solution to be in C/C++.
A NAND flash drive is composed of a number of data pages and a flash controller. The ECC code on the NAND flash is used by the flash controller. The controller uses it to determine if the associated data page has any errors. A filesystem (like fat32, NTFS, or ext3) is usually implemented on top of the raw data page structure. A file may be spread across one or more flash pages. Please note that there is an error code for each flash page, meaning that a file larger than one data page will have an error code for each page it uses. A hard disk drive is composed of one or more platters, heads which read and write data on the platters, and a disk controller. Each platter is divided into sectors. Hard disk drives also have error codes to check the integrity of sectors on the platters. Again, a filesystem, is typically implemented on top of the raw disk sectors. If a file is larger than a sector then there will be multiple sectors associated with the file, each sector having its own error code. To access a data page or sector error code you will need to interface with the flash or hard dive controller directly. This will require interfacing with the device driver for the device. You will need to read the documentation for the device driver in order to discover what functions it offers to allow you access to error codes. In either instance, unless a file occupies only one data page or sector it will have more than one error code associated with it. Some filesystems create error codes for files, regardless of there length, but accessing a filesystem level error code typically does not require hardware access.
I'm developing a kind of filesystem driver. All of read requests that windows makes to my filesystem goes by the driver implementation.
I would like to distinguish between "normal" read requests and those who want to get only the metadata from the file. ( Windows reads first 4K of the file and then stop reading ).
Does Windows mark this metadata reads in some way? It would be very useful in order to treat that two kind of operations in a different way.
In a typical CreateFile call, we have AccessMode, ShareMode, CreationDisposition and FlagsAndAttributes parameters ( being DWORD ), i'm not sure if it's possible to extract some clue of the operation requested.
Thanks for reading :)
I'd advise you to get the SysInternals file monitoring tool. It captures stacktraces for each call, and since it understands PDBs can even show you function names. That should allow you to figure out many details of this particular call.
On rereading, it appears that the question is looking at the wrong place for an optimization. Why not treat every request for the first 4KB as a request for metadata? There is very little harm in that assumption.
An assumption the other way around would be harmful, if you're doing 100 MB of real I/O when you really only needed 4KB. But if you need 100 MB, a small optimization for the first 4KB causes at most a one-time small hickup for an inherently lengthy operation.
It's not Windows, but Windows Explorer that performs scanning of files to extract metadata. Moreover, you will also face the reads to create thumbnails.
Reporting the drive as a remote / network to Windows will make Explorer read less information and reduce load on the file system, but unfortunately there seems to be no way to block such reading completely.