In our Windows 8 application, we are using the IXMLHTTPRequest2 method to stream files over HTTP, files whose size can reach gigabytes. This all works perfectly, except for the fact that internally, WinRT has a caching system which stores all that is streamed over the call to IXMLHTTPRequest2 in the temporary internet cache. As we stream more and more files, the cache is never emptied and it just starts taking more and more space on disk, until the disk is full.
Optimally, we would like to disable this caching functionality entirely. Another option we could live with would be that the cached files would be removed after a short while (although we'd like to avoid having to browse the temporary internet cache and removing files manually).
We've tried adding the "Expires: 0" header to the server response, as well as disabling the caching directly inside IE (we thought this might have an influence on the call to IXMLHTTPRequest2), but to no avail.
Anyone has any thoughts on this?
I realize this question is similar to another one posted here, however, our problem has more to do with the space that is taken by the cache rather than by the "freshness" of the files.
EDIT:
We have also found this post on the MSDN forums, where, according to a MSFT Moderator, "The system will also periodically cleans up the cache so you will not have to worry about running out of disk space", but that is not the case in our scenario.
According to this post on the MSDN forums, this isn't possible and is a known limitation with WinRT.
Sometimes the only answer is bad news. :-[
As ildjarn noted, this seems to be unavoidable on Windows 8. But it looks like there might be a way to fix this for clients running Windows 8.1.
I haven't tried it myself, but I just noticed that there is now "IXMLHTTPRequest3" which extends "IXMLHTTPRequest2" with some new features:
http://msdn.microsoft.com/en-us/library/windows/desktop/dn376398%28v=vs.85%29.aspx
The relevant feature is:
XHR_PROP_NO_CACHE – Suppresses cache reads and writes for the HTTP request.
That sounds promising.
Related
These two links reference the ability, in ColdFusion, to get the name of an uploaded file using form.getPartsArray(). However I can not find ColdFusion documentation on it. I would like to use this but not if it has been deprecated or will be. Does anyone have more information on the origin and fate of this function?
ColdFusion: get the name of a file before uploading
http://www.stillnetstudios.com/get-filename-before-calling-cffile/
ColdFusion: get the name of a file before uploading
Ignoring the main question for a moment, can you elaborate on why you want to use it? Reason for asking is the title of the first question might give you a mistaken impression about what that method actually does. Form.getPartsArray() does not provide access to file information before the file is uploaded. The file is already on the server at that point, so in later versions of CF (with additional functionality) it does not necessarily buy you much over just using cffile action=upload.
Does anyone have more information on the origin and fate of this
function?
However, to answer your other question - it is an undocumented feature last I checked. (It was more useful in earlier versions of CF, which lacked some of the newer features relating to form fields and uploads.)
Internally, most form data can be handled using standard request objects, ie HttpServletRequest. However, those do not support multipart requests, ie file uploads. So a special handler is needed. Macromedia/Adobe chose to use the com.oreilly.servlet library for their internal implementation. That is what you are accessing when using FORM.getPartsArray().
The O'Reilly stuff has been bundled with CF since (at least) CF8, which is a good indicator. However, using any internal feature always comes with the risk the implementation will change and break your application. Also, if you move to another engine, the code may not be supported/compatible. So "You pays your money, you takes your choice".
CF8 / Form Scope
I'm writing a game that will have a lot of information (configuration, some content, etc) inside of some xml documents, as well as resource files. This will make it easier for myself and others to edit the program without having to edit the actual C++ files, and without having to recompile.
However, as the program is starting to grow there is an increase of files in the same directory as the program. So I thought of putting them inside a file archive (since they are mostly text, it goes great with compression).
My question is this: Will it be easier to compress all the files and:
Set a password to it (like a password-protected ZIP), then provide the password when the program needs it
Encrypt the archive with Crypto++ or similar
Modify the file header slightly as a "makeshift" encryption, and fix the file's headers while the file is loaded
I think numbers 1 and 2 are similar, but I couldn't find any information on whether zlib could handle password-protected archives.
Also note that I don't want the files inside the archive to be "extracted" into the folder while the program is using it. It should only be in the system's memory.
I think you misunderstands the possibilities brought up by encryption.
As long as the program is executed on an untrusted host, it's impossible to guarantee anything.
At most, you can make it difficult (encryption, code obfuscation), or extremely difficult (self-modifying code, debug/hooks detection), for someone to reverse engineer the code, but you cannot prevent cracking. And with Internet, it'll be available for all as soon as it's cracked by a single individual.
The same goes, truly, for preventing an individual to tamper with the configuration. Whatever the method (CRC, Hash --> by the way encryption is not meant to prevent tampering) it is still possible to reverse engineer it given sufficient time and means (and motivation).
The only way to guarantee an untampered with configuration would be to store it somewhere YOU control (a server), sign it (Asymmetric) and have the program checks the signature. But it would not, even then, prevent someone from coming with a patch that let's your program run with a user-supplied (unsigned) configuration file...
And you know the worst of it ? People will probably prefer the cracked version because freed from the burden of all those "security" measures it'll run faster...
Note: yes it is illegal, but let's be pragmatic...
Note: regarding motivation, the more clever you are with protecting the program, the more attractive it is to hackers --> it's like a brain teaser to them!
So how do you provide a secured service ?
You need to trust the person who executes the program
You need to trust the person who stores the configuration
It can only be done if you offer a thin client and executes everything on a server you trust... and even then you'll have trouble making sure that no-one finds doors in your server that you didn't thought about.
In your shoes, I'd simply make sure to detect light tampering with the configuration (treat it as hostile and make sure to validate the data before running anything). After all file corruption is equally likely, and if a corrupted configuration file meant a ruined client's machine, there would be hell to pay :)
If I had to choose among your three options, I'd go for Crypto++, as it fits in nicely with C++ iostreams.
But: you are
serializing your data to XML
compressing it
encrypting it
all in memory, and back again. I'd really reconsider this choice. Why not use eg. SQLite to store all your data in a file-based database (SQLite doesn't require any external database process)?
Encryption can be added through various extensions (SEE or SQLCipher). It's safe, quick, and completely transparent.
You don't get compression, but then again, by using SQLite instead of XML, this won't be an issue anyway (or so I think).
Set a password to it (like a password-protected ZIP), then provide the password when the program needs it
Firstly, you can't do this unless you are going to ask a user for the password. If that encryption key is stored in the code, don't bet on a determined reverse engineer from finding it and decrypting the archive.
The one big rule is: you cannot store encryption keys in your software, because if you do, what is the point of using encryption? I can find your key.
Now, onto other points. zlib does not support encryption and as they point out, PKZip is rather broken anyway. I suspect if you were so inclined to find one, you'd probably find a zip/compression library capable of handling encryption. (ZipArchive I believe handles Zip+AES but you need to pay for that).
But I second Daniel's answer that's just displayed on my screen. Why? Encryption/compression isn't going to give you any benefit unless the user presents some form of token (password, smartcard etc) not present in your compiled binary or related files. Similarly, if you're not using up masses of disk space, why compress?
I'm looking for a good efficient method for scanning a directory structure for changed files in Windows XP+. Something like how git does it is exactly what I'm looking for, when running a git status it displays all modified files, all new (untracked) files and deleted files very quickly which is exactly what I would like to do.
I have a basic model up and running which performs an initial scan and stores all filenames, size, dates and attributes.
On a subsequent scan it checks if the size, attributes or date have changed and marks as a changed file.
My issue now comes in detecting moved and deleted files. Is there a tried and tested method for this sort of thing? I'm struggling to come up with a good method.
I should mention that it will eventually use ReadDirectoryChangesW to monitor files and alert the user when something changes so a full scan is really a last resort after the initial scan.
Thanks,
J
EDIT: I think I may have described the problem badly. The issue I'm facing is not so much detecting the changes - I have ReadDirectoryChangesW() using IOCP on multiple threads to detected when a change happens, the issue is more what to do with the information. For example, a moved file is reported as a delete followed by a create and a rename comes in 2 parts, old name, followed by new name. So what I'm asking is how to differentiate between the delete as part of a move and an actual delete. I'm guessing buffering the changes and processing batches would be an option but feels messy.
In native code FileSystemWatcher is replaced by ReadDirectoryChangesW. Using this properly is not simple, there is a good baseline to build off here.
I have used this code in a previous job and it worked pretty well. The Win32 API itself (and FileSystemWatcher) are prone to problems that are described in the docs and also discussed in various places online, but impact of those will depending on your use cases.
EDIT: the exact change is indicated in the FILE_NOTIFY_INFORMATION structure that you get back - adds, removals, rename data including old and new name.
I voted Liviu M. up. However, another option if you don't want to use the .NET framework for some reason, would be to use the basic Win32 API call FindFirstChangeNotification.
You can use USN journaling if you are up to it, that is pretty low level (NTFS level) stuff.
Here you can find detailed information and source code included. It is written in C# but most of it is PInvoking C/C++ functions.
I've recalled using little 'filesystems' before that basically provided an interface to something else. For example, I believe there was a GMail filesystem that created an entry in My Computer and could be used like any other drive on your local computer. How can I go about implementing something like this in C++?
Thank you!
Try Dokan. It's like FUSE, except for Windows. I think there are certain limitations to namespace extensions, like they cannot be accessed from the command line, but I'm really not sure as of now.
Writing an actual file-system involves writing a driver; which means kernel-mode code (scary stuff) and paying for getting the IFS DDK. (edit: looks like they don't charge for it anymore)
What you probably want is a “namespace extension”.
Try this: The Complete Idiot's Guide to Writing Namespace Extensions - CodeProject
This may be a starting point to extending NTFS in the way that the GMail filesystem used to do: Windows NT reparse points.
The GMail Filesystem is just the name given; it is not any filesystem as such. It is just a namespace extension for Windows Explorer that links with the GMail account of yours!
I dont know exactly what you are trying to do, but in anyway, I believe, the following link will be of some use to you:
http://msdn.microsoft.com/en-us/magazine/cc188741.aspx
Just as a reference: virtual drives can be created using our Callback File System product, which is a supported, documented and maintained solution.
I was thinking of this too, perhaps some example code ? (email me if i forget plz ;p doin sdk now)
I'm thinking of a similar filesystem that would plug in as a driver and allow dynamic 'soft raid' on larger files mostly by putting them on more than one disk, perhaps some compression options and 'smart' filters to toggle usage in high disk space low usage and other situations more effectively, with status controls and indicators as a normalish program too
Seems like I would load the driver kit,
then i want the file writing event, and am mostly replacing fopen and similar functions automatically as an intermediate driver with a little windows network driver experience
I also heard good things about developing on a virtual machine for less crashing and more debugging
Also perhaps more metainfo on some or all files, including files in special folders with options too, including maybe both fast and simple (obfuscated and/or symmetic key) encryption options on folder, specified, all, letter, etc, or whatever, or the slower version and maybe integrated and optional (also profitable) online cvs-like diff style backups that mostly target changes to hot files for online backup at intervals and prices, mostly perhaps with matching keyboard events and might even be useful as simply a keylogging online backup service that is reasonably secure too
while avoiding common files like windows files or the normal stuff in the 'programs' directory that can be copied easily with pirate tools, unlike all of your documents.
I'm trying to write a chat client for a popular network. The original client is proprietary, and is about 15 GB larger than I would like. (To be fair, others call it a game.)
There is absolutely no documentation available for the protocol on the internet, and most search results only come back with the client's scripting interface. I can understand that, since used in the wrong way, it could lead to ruining other people's experience.
I've downloaded the source code of a couple of alternative servers, including the one I want to connect to, but those
contain no documentation other than install instructions
are poorly commented (I did a superficial browsing)
are HUGE (the src folder of the target server contains 12 MB worth of .cpp and .h files), and grep didn't find anything related
I've also tried searching their forums and contacting the maintainers of the server, but so far, no luck.
Packet sniffing isn't likely to help, as the protocol relies heavily on encryption.
At this point, all my hope is my ability to chew through an ungodly amount of code. How do I start?
Edit: A related question.
If your original code is encrypted with some well known library like OpenSSL or Ctypto++ it might be useful to write your wrapper for the main entry points of these libraries, then delagating the call to the actual library. If you make such substitution and build the project successfully, you will be able to trace everything which goes out in the plain text way.
If your project is not using third party encryption libs, hopefully it is still possible to substitute the encryption routines with some wrappers which trace their input and then delegate encryption to the actual code.
Your bet is that usually enctyption is implemented in separate, relatively small number of source files so that should be easier for you to track input/output in these files.
Good luck!
I'd say
find the command that is used to send data through the socket (the call depends on the network library)
find references of this command and unroll from there. If you can modify-recompile the server code, it might help.
On the way, you will be able to log decrypted (or, more likely, not yet encrypted) network activity.
IMO, the best answer is to read the source code of the alternative server. Try using a good C++ IDE to help you. It will make a lot of difference.
It is likely that the protocol related material you need to understand will be limited to a subset of the files. These will contain references to network sockets and things. Start from there and work outwards as far as you need to.
A viable approach is to tackle this as a crypto challenge. That makes it easy, because you control so much.
For instance, you can use a current client to send a known message to the server, and then check server memory for that string. Once you've found out in which object the string ends, it also becomes possible to trace its ancestry through the code. Set a breakpoint on any non-const method of the object, and find the stacktraces. This gives you a live view of how messages arrive at the server, and a list of core functions essential to message processing. You can next find related functions (caller/callee of the functions on your list).