Trying to create something similar to SFX (self extract installer), I was including the setup files in one exe using custom resource in visual c++. When the size of the setup files reached over ~900 MB, the linker cannot generate an exe with the resources embedded. Strange thing is that the linker does generate an exe without any warnings. The exe generated is an statically linked MFC App (main app of the exectuable). However, non of the resource is included. I have watched the exe size during the linking from a console. It was showing 900 MB for awhile befored dropped to the 2MB exe in the end. If I removed some of the files (say total size will be 500MB), the exe is generated all OK with all the resources embedded. Is there a pratical size limit for the custom resource? Note that the final size of the executable is about 1GB which is way less that 2GB win32 app can address. So I am wondering is this a vc++ 2010 bug or feature? BTW, I also used /expectedoutputsize trick. But it does not help except the linking is much faster with a bigger expected output size.
Belated answer. Might be helpful to others. May be not an answer to your question, but this may be a solution to your problem. Check for 'Bitrock installer'. This also gives Self Extract Installer. I used it successfully in many of my projects for deployment.
According to the PECOFF Specification, the size of the Resource Data entries is 4 bytes ( IMAGE_RESOURCE_DATA_ENTRY.Size - see winnt.h )
The size, in bytes, of the resource data that is pointed to by the Data RVA field.
I've answered a similar answer here.
Basically append the binary + magic number (unique separator) to the end of the executable. At runtime open the executable for reading, find the offset via a magic number or another method and the rest of the file is your big data.
Related
Question concerns only Windows for now - other OS's are not so relevant right now.
Just by quick googling - it's possible to load native .dll from RAM, there are for example following libraries:
https://www.joachim-bauch.de/tutorials/loading-a-dll-from-memory/
=>
https://github.com/fancycode/MemoryModule
https://forum.nim-lang.org/t/7943
But all of them requires:
in-depth knowledge of PE file format
mostly those approaches are not debugger friendly.
What I have checked - windows's LoadLibraryA / LoadLibraryW are directed to ntdll.dll / LdrLoadDll - and best picture of how things works can be found from here: https://github.com/hlldz/RefleXXion
And even thus I don't have windows source code - I've checked same functionality from Wine:
LdrLoadDll: https://source.winehq.org/source/dlls/ntdll/loader.c#3169
load_dll: https://source.winehq.org/source/dlls/ntdll/loader.c#3083
load_native_dll:
https://source.winehq.org/source/dlls/ntdll/loader.c#2564
NtMapViewOfSection: https://source.winehq.org/source/dlls/ntdll/unix/virtual.c#4469
find_dll_file: https://source.winehq.org/source/dlls/ntdll/loader.c#3021
open_dll_file: https://source.winehq.org/source/dlls/ntdll/loader.c#2467
Suspect loading dll happens via following function calls:
NtOpenFile, NtQueryAttributesFile, NtCreateSection/NtOpenSection, NtMapViewOfSection (*)
(More information could be found in
https://github.com/Hagrid29/PELoader
https://gist.github.com/bats3c/59932dfa1f5bb23dd36071119b91af0f
https://www.octawian.ro/fisiere/situri/asor/build/html/_downloads/122f95f9a032396603a837c53b125bb8/Russinovich_M_WinInternals_part1_7th_ed.pdf
)
I was also thinking if I could just override NtOpenFile and just redirect file open (in
https://github.com/SegaraRai/PathRedirector manner)
to different path - but main question what is the alternative location where to store file?
I was thinking if NtOpenFile can open even device, then maybe just replace file
with some sort of named pipe (https://learn.microsoft.com/en-us/windows/win32/ipc/named-pipe-client) - but then in maps on how well this will work with NtMapViewOfSection.
Since I was not able to find any working example of such hook or operation (E.g. LoadLibary("\\.\pipe\mynamedpipe_as_dll")) - there is always a risk that such combination is not simply supported.
Is it possible to load native .dll purely from RAM:
Without using file system (not to store .dll e.g. in temporary folder)
Without involving custom drivers (like Dokan) ?
So loaded .dll would be still debugger friendly ?
Not tightly bound to PE file format structures (or use PE structures as less as possible)
If you miss bit more information, check also my own experiments with native dll loading (maybe can give some hints on solving the issue):
https://github.com/tapika/test_native_dll_loading
https://github.com/tapika/test_native_dll_loading/discussions/2
Distinguish between debug and release use cases. In debug, save the DLL in a temp file and load with LoadLibrary, which will enable debugging. In release, run from memory with no capability for debugging.
Here's another idea, from considering the linked Guthub issue. If the purpose is to let the users provide their own compression/decompression logic while building a ReadyToRun executable, let them provide that as a static library (object) as opposed to a DLL. The larger project is already about packaging stuff into a single executable, might do some linking while at it.
Yet another idea would be to let the users provide the codec in some kind of interpreted language and optionally plug in the interpreter that supports debugging. Windows comes with a built-in JavaScript interpreter, look up Active Scripting, and debugging those is a free bonus. The performance probably won't be on par with a native code implementation, though.
I think you could probably do something similar with Frida. Hook the functions LoadLibraryA / LoadLibraryW and reimplement them in Frida. but I don't believe this is something that would be stable for production.
For some reference
By analyzing existing approaches (like PE Loader https://github.com/Hagrid29/PELoader) and using minhook library - I've managed to load .dll from RAM.
I've created git repository with example code on github:
https://github.com/tapika/dllloader
Could you create a ramdisk to put your DLL there? What exactly is the use-case for this? There are a couple ways to spin up a file in RAM, C#'s MemoryMappedFile for example. I'm not sure if this would be debugger friendly.
One of our application (windows form application C++, MSVS 2010 )crashes after few minutes of usage. Task manager tells that the memory usage grows to 60% of total system RAM in just few seconds of the run.
I used Intel inspector to get any idea of memory leaks. I was expecting I will get a a list of functions that are creating problem. But what I got is only the dlls as can be seen in the following screenshot.
The application is using a couple of third party libraries such as those starting with Pv, OpenCv cdio, CAIO etc. As you can see the last one is an opencv library, and is occupying close to 400MB. (How is this possible ? )
Also the right panel shows different types of leaks which have occurred.
I want to pin point the memory leak code and correct it. What should be my strategy, what functions should I start looking into? Why the inspector is not giving me correct source code and just giving me dlls? I am sure dlls are perfect as these are used by millions of people.
Please advice,
Thanks
Update
I think I have done something wrong in various compiler setting while generating the exe. .
As can be seen above, no symbol information is loaded. That was the reason I was unable to get the source code where memory leaks were happening. Pressing F1 reaches me to the following instructions:
Troubleshooting No Symbolic Information Symptoms
In the Sources window, the Intel Inspector displays no source code for any code locations in the problem set.
Details Intel Inspector cannot display source code for viewing and editing unless the application has symbols available.
If the Intel Inspector detects no symbols for a location, it sets the call stack and code pane to the first location where it can find symbols.
If the Intel Inspector cannot find any location in the call stack with symbols, it provides the module name and relative virtual address (RVA) for the location.
Possible Correction Strategies
1- When you compile and link an application on Windows* systems:
a) Enable the debug information compiler option.
b) Set the linker option to generate debug information.
2- Configure the project to search non-standard directories.
3- Copy the symbol files, such as *.pdb files, to a location where the Intel Inspector can find them.
So now I am focusing on the above correction strategies. My latest question are:
1- how do I set the above three strategies in MSVS 2010.
2- Do I need to use debgug exe or a release exe while using Intel inspector ?
If this is your source code, and you are sure your code is causing the leaks, you can use Visual Leak Detector.
You just need very minimal changes in project - I would say just #include<vld.h> (which you can make conditional). It will report all memory leaks on Debug Output window. This differs from VC++ standard leaking staticitics - it shows where memory was allocated.
Probally it couldn't load symbols for some module / modules and thus the information is a bit incorrect. Is symbol file (like opencv_core240.pdb) opencv_core240.dll available? Check it!
Also I would suggest to try another memory leak detectors to compare their results to each other.
In general when using Inspector the recommendation is to use a debug build of the code. Release builds may optimize away some important pieces of code.
You can also enable just debugging symbols in a release build, which is important when using Amplifier and Advisor. You can do this by going to Project -> [project name] properties... -> Linker -> Debugging -> Generate Debug Info -> Yes and Project->[project name] properties...->C/C++->General->Debug Information Format -> Program Database. Even if you are in debug configuration make sure these settings are set correctly, because they may have been accidentally modified.
WRT what you are seeing in the report:
OpenCV (and others) isn't occupying 350MB, rather there has been a leak of that size (which means the last pointer to dynamically allocated memory was overwritten without releasing that memory). Is it possible that you're misusing the library APIs?
Also, you may find it useful to look at the call stack at the location of the leak. You will possibly find the API leading to the memory leak which can help you pinpoint the issue.
I need to read information, code, flags, address, etc from a memory.dmp file generated from a windows BSOD through C++. The basic idea is that status info can be requested from a remote site and one of the requested pieces of information is some basic info from the last BSOD that occured on the machine thus I need to open the kernel/memory dump file through C++ (Im using MSVC 2005).
Start here, then realize using scripted commands in WinDBG is much easier.
Note: you only need WinDBG on the analysis machine, not the crashing one. You retrieve the minidump and analyse it externally. The only difficulty you will have is getting the right symbols - for Windows, Microsoft makes them available via their symbol servers, but applications that caused the crash may not supply the right symbols you need. IF they are you own applications causing the crash, get a symbol server and use it.
I would configure Windows to create small kernel memory dumps which will include the parameter of the bugcheck you are after.
On XP it was 64KB on my Win8.1 x64 it is 256KB. These files compress well. You should be able to get away with a zip file of 10-60KB size depending on the bitness of the OS. If bandwidth is of of utmost importance to you, you can use 7z which compresses about 50% better than the plain zip algo at the expense of much longer compression times (5-6 longer) but for such small files the CPU time difference should be irrelevant.
If you do not want your users to configure dump reporting you need to set the DWORD
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl
to 3 for a small kernel dump programatically.
For an explanation of the values see http://technet.microsoft.com/en-us/library/cc976050.aspx
0 Debugging information is not written to a file.
1 Complete crash dump is written to a file.
2 Kernel memory dump is written to a file.
3 Small memory dump is written to a file.
You will then get by default a small kernel dump in %SystemRoot%\MEMORY.DMP.
Is it possible to modify an executable file on runtime (I'm asking about Windows XP/Vista/7/Server)? I've just evaluated SmartUtils Portable Storage application. It can create so called "managed executable storage files" that modify them-self at runtime... Such storage file is like standard self-extracting archive (the data is apended to an executable module) but the main difference it that you are able to view and modify its content without the main program. How is it possible? I need similar functionality in my project (C++): I want to be able to create executable that can modify data attached to it.
If all you're really asking is how SmartUtils Portable Storage does it's magic, then I would suggest that it is a self-executing zip archive. The EXE of the archive (just as WinZip or 7-Zip create) auto-extracts and executes your application exe from a temp folder, and gives you an API that boils down to ways to extract, manipulate, and then modify that original self-executing archive.
So Windows is never trying to modify a running .exe. Rather, your .exe (temp file extracted & run) is what is executing (and the libraries bound to it), which manipulates the source .exe (really a self-executing archive - possibly .zip).
The next time the user "runs" the modified "exe", again your .exe is extracted & run, and it can again manipulate the self-extracting .exe.
I hope that makes sense to you.
And this is just a best guess!
Yes - a common technique is to append data files at the end of an executable.
Typical scheme is to write a 0x00000000 integer to the end of the executable and then append each file followed by it's size in bytes.
Then when the executable needs to read the data it checks the last 4bytes in it's own file, uses that as the file length and copies that number of bytes form it's own file, it then checks the next 4 bytes as another length and copies that as a file , until it gets a length of 0000. If you also need to code the file names - that adds a little complexity but it's basically the same idea.
You can append a TOC pointer to an EXE (and probably a magic ID cookie) so you can verify that it is a TOC pointer, and then use that to back up to the start of each appended record.
As long as you don't mess up the file's header & main contents, it should still be loadable by the OS.
However, you sacrifice any signing your EXE had - and you probably have various permissions issues to contend with...
I have written tools for my development environment that opens a Windows EXE, extrapolates the resources in it, modifies various ones, and repackages the whole thing. We use this to mark a beta as release (so it modifies the version records).
You can do anything you want to an EXE file if you know the structure of it and rebuild it correctly.
Since this is tagged as Windows, you might also consider "Alternate Data Streams". That allows you to treat a single file almost as a directory. You can add a stream called Program.EXE:ExtraData to your program and write to that with the normal file functions.
Then again, your executable most likely will be in Program Files\, which isn't writeable for normal (non-elevated) users.
What I'm trying to do is to sign my compiled executable's first 32 bytes with a version signature, say "1.2.0" and I need to modify this signature in runtime, keeping in mind that:
this will be done by the executable itself
the executable resides on the client side, meaning no recompilation is possible
using an external file to track the version instead of encoding it in the binary itself is also not an option
the solution has to be platform-independent; I'm aware that Windows/VC allows you to version an executable using a .rc resource, but I'm unaware of an equivalent for Mac (maybe Info.plist?) and Linux
The solution in my head was to write the version signature in the first or last 32 bytes of the binary (which I didn't figure out how to do yet) and then I'll modify those bytes when I need to. Sadly it's not that simple as I'm trying to modify the same binary that I'm executing.
If you know of how I can do this, or of a cleaner/mainstream solution for this problem, I'd be very grateful. FWIW, the application is a patcher/launcher for a game; I chose to encode the version in the patcher itself instead of the game executable as I'd like it to be self-contained and target-independent.
Update: from your helpful answers and comments, I see that messing with the header/footer of the binary is not the way to go. But regarding the write permission for the running users, the game has to be patched one way or another and the game files need to be modified, there's no way to circumvent that: to update the game, you'll need admin privileges.
I would opt for using an external file to hold the signature, and modify that with every update, but I can't see how I can guard against the user spoofing with that file: if they mess up the version numbers, how can I detect which version I'm running?
Update2: Thanks for all your answers and comments, in truth there are 2 ways to do this: either use an external resource to track the version or embed it in the main application's binary itself. I could choose only 1 answer on SO so I did the one I'm going with, although it's not the only one. :-)
Modern Windows versions will not allow you to update an installed program file unless you're running with administrator privileges. I believe all versions of Windows block modifications to a running file altogether; this is why you're forced to reboot after an update. I think you're asking for the impossible.
This is going to be a bit of a challenge, for a number of reasons. First, writing to the first N bytes of the binary is likely to step on the binary file's header information, which is used by the program loader to determine where the code & data segments, etc. are located within the file. This will be different on different platforms (see the ELF format and executable format comparison)--there are a lot of different binary format standards.
Assuming you can overcome that one, you're likely to run afoul of security/antivirus systems if you start modifying a program's code at runtime. I don't believe most current operating systems will allow you to overwrite a currently-running executable. At the very least, they might allow you to do so with elevated permissions--not likely to be present while gaming.
If your application is meant to patch a game, why not embed the version in there while you're at it? You can use a string like #Juliano shows and modify that from the patcher while the game is not running - which should be the case if you're currently patching anyways. :P
Edit: If you're working with Visual Studio, it's really easy to embed such a string in the executable with a #pragma comment, according to this MSDN page:
#pragma comment(user, "Version: 1.4.1")
Since the second argument is a simple string literal, it can be concatenated, and I'd have the version in a simple #define:
// somehwere
#define MY_EXE_VERSION "1.4.1"
// somewhere else
#pragma comment(user, "Version: " MY_EXE_VERSION)
I'll give just some ideas on how to do this.
I think it's not possible to change some arbitrary bytes in the executable without side effects. To overcome this, I would create some string in your source code, like:
char *Version = "Version: AA.BB.CC";
I don't know if this is a rule, but you can look for this string in your binary code (open it in a text editor and you will see). So, you search and change this bytes for your version number in the binary file. Probably, their position will vary each time you compile the application, so this it is possible only if that location is not a problem for you.
Because the file is being used (it's running), you have to launch an external program that would do this. After modifying the file, this external program could relaunch the original application.
The version will be stored in your binary code in some part. Is that useful? How will you retrieve the version number?