Question concerns only Windows for now - other OS's are not so relevant right now.
Just by quick googling - it's possible to load native .dll from RAM, there are for example following libraries:
https://www.joachim-bauch.de/tutorials/loading-a-dll-from-memory/
=>
https://github.com/fancycode/MemoryModule
https://forum.nim-lang.org/t/7943
But all of them requires:
in-depth knowledge of PE file format
mostly those approaches are not debugger friendly.
What I have checked - windows's LoadLibraryA / LoadLibraryW are directed to ntdll.dll / LdrLoadDll - and best picture of how things works can be found from here: https://github.com/hlldz/RefleXXion
And even thus I don't have windows source code - I've checked same functionality from Wine:
LdrLoadDll: https://source.winehq.org/source/dlls/ntdll/loader.c#3169
load_dll: https://source.winehq.org/source/dlls/ntdll/loader.c#3083
load_native_dll:
https://source.winehq.org/source/dlls/ntdll/loader.c#2564
NtMapViewOfSection: https://source.winehq.org/source/dlls/ntdll/unix/virtual.c#4469
find_dll_file: https://source.winehq.org/source/dlls/ntdll/loader.c#3021
open_dll_file: https://source.winehq.org/source/dlls/ntdll/loader.c#2467
Suspect loading dll happens via following function calls:
NtOpenFile, NtQueryAttributesFile, NtCreateSection/NtOpenSection, NtMapViewOfSection (*)
(More information could be found in
https://github.com/Hagrid29/PELoader
https://gist.github.com/bats3c/59932dfa1f5bb23dd36071119b91af0f
https://www.octawian.ro/fisiere/situri/asor/build/html/_downloads/122f95f9a032396603a837c53b125bb8/Russinovich_M_WinInternals_part1_7th_ed.pdf
)
I was also thinking if I could just override NtOpenFile and just redirect file open (in
https://github.com/SegaraRai/PathRedirector manner)
to different path - but main question what is the alternative location where to store file?
I was thinking if NtOpenFile can open even device, then maybe just replace file
with some sort of named pipe (https://learn.microsoft.com/en-us/windows/win32/ipc/named-pipe-client) - but then in maps on how well this will work with NtMapViewOfSection.
Since I was not able to find any working example of such hook or operation (E.g. LoadLibary("\\.\pipe\mynamedpipe_as_dll")) - there is always a risk that such combination is not simply supported.
Is it possible to load native .dll purely from RAM:
Without using file system (not to store .dll e.g. in temporary folder)
Without involving custom drivers (like Dokan) ?
So loaded .dll would be still debugger friendly ?
Not tightly bound to PE file format structures (or use PE structures as less as possible)
If you miss bit more information, check also my own experiments with native dll loading (maybe can give some hints on solving the issue):
https://github.com/tapika/test_native_dll_loading
https://github.com/tapika/test_native_dll_loading/discussions/2
Distinguish between debug and release use cases. In debug, save the DLL in a temp file and load with LoadLibrary, which will enable debugging. In release, run from memory with no capability for debugging.
Here's another idea, from considering the linked Guthub issue. If the purpose is to let the users provide their own compression/decompression logic while building a ReadyToRun executable, let them provide that as a static library (object) as opposed to a DLL. The larger project is already about packaging stuff into a single executable, might do some linking while at it.
Yet another idea would be to let the users provide the codec in some kind of interpreted language and optionally plug in the interpreter that supports debugging. Windows comes with a built-in JavaScript interpreter, look up Active Scripting, and debugging those is a free bonus. The performance probably won't be on par with a native code implementation, though.
I think you could probably do something similar with Frida. Hook the functions LoadLibraryA / LoadLibraryW and reimplement them in Frida. but I don't believe this is something that would be stable for production.
For some reference
By analyzing existing approaches (like PE Loader https://github.com/Hagrid29/PELoader) and using minhook library - I've managed to load .dll from RAM.
I've created git repository with example code on github:
https://github.com/tapika/dllloader
Could you create a ramdisk to put your DLL there? What exactly is the use-case for this? There are a couple ways to spin up a file in RAM, C#'s MemoryMappedFile for example. I'm not sure if this would be debugger friendly.
We have several Windows 8 Store C++ apps that need to maintain configuration and data files.
Files are written in subfolders of Windows::Storage::ApplicationData::Current->LocalFolder. Example:
C:\Users\<username>\AppData\Local\Packages\<packagename>\LocalState\SubFolder1\SubFolder2\data.txt
In Windows 8.1 we have received a few reports from users that say state isn't remembered between app invocations. Upon closer inspection the files are not created (the subfolders are indeed created, but there are no files inside them)
Notes:
Subfolders are created using CreateDirectory(), files are created using fopen()
Files are created/opened using absolute paths
This always worked under Windows 8.0 and the code has not been changed since. In fact, one of our user reports stated that the app saved files fine under Windows 8.0, but stopped saving after the user upgraded to Windows 8.1.
We have not been able to replicate the issue locally using Windows 8.1. We're not sure how common this failure is, but we estimate that most users are unaffected. Affected users do not appear to have any special hardware/software configuration.
If a user is affected, then files are consistently never saved, even after retrying or uninstalling and re-installing the app (i.e., it's not a case of intermittent failure)
It's hard to get error information given (i) the rarity of the issue (ii) the fact that the logs that would reveal this are by definition not saved, and (iii) the apps don't require internet connectivity so there is no alternative communication channel.
Can anyone think of any reason why this might fail under Windows 8.1?
Are there non-ascii characters in path to appdata? CreateDirectory has unicode version, but fopen takes const char* strings as argument.
If I were you, I'd try to abstract away from OS-specific calls using something like boost or Qt. That should work, because Qt uses unicode string for opening files and Boost should have something similar (unsure about this one).
Also on windows compiler _wfopen may be present. It is the same as fopen, but takes wchar_t strings as argument. It should work for you, but you'll need a few ifdefs here and there.
You could also try setting current directory with function that supports unicode and then calling fopen, but I wouldn't call it a "clean" solution.
Anyway, when you run into problem that is related to system calls, then on machine with a problem you can monitor calls using something like process monitor. You could instruct user with a problem to do that and send you a log.
Why don't you use the Windows.Storage classes to work with filesystem? WinRT is recommended way to work with IO not legacy C API. I believe it is more robust approach and you could get more info about the cause from WinRT exception rather than from an unknown failure of old API.
I'm an amateur programmer, and I'm getting desperate and mad because of a big issue: most of my programs are blocked by Avast Antivirus, while some aren't, and I don't understand why.
The more I try to investigate, the less I understand what the problem could be.
I'm requesting your help to find a solution so that my programs are no longer blocked, or, as a default, at least some strong clues that would explain why it might be the case.
There are already many topics about that on the web. However, most of them give only superficial answers: they just explain how antivirus software works with signatures and detection heuristics, or state that you just have to add the offending application in the white list without asking any other question. While it is certainly correct, it's not acceptable answers in my sense, because I'm still left with my own programs that refuse to work without any concrete idea to start investigating.
First of all, the only antivirus software that blocks my programs is Avast 7.x. No other antivirus software see any inconvenient to run my software. Secondly, I haven't installed Avast myself; it is installed on a friend's machine.
I have Windows 7, and he has Windows XP. I'm completely sure that the problem is avast only: when it is temporarily disabled, or if the program is added to its white list, everything works nicely as expected.
Three different programs are in trouble:
A text editor, with the goal to replace Windows Notepad while keeping simple, efficient and customizable
A small amateur audio player very simple to use
The client program of an online game platform, currently having more than 1000 users
The first one is open source, and I can give a link to the executable and the source code if needed. The two others are closed source but free to use, I can give a link to the executable of the current version only.
The only obvious common things between these three programs are me as a developer, my Windows 7 machine that compiled them, the compiler family which is MinGW/GCC, and they are all Win32 GUI applications without any framework (no MFC, no WPF, no Qt, wxWidgets or whatever; just pure Win32/C GUI applications).
Here are my observations and thoughts so far:
Versions 1.1, 1.2.1 and 1.3 of my text editor are blocked. They are in C, not C++, have been compiled with GCC 3.4.5 in Unicode mode, and are distributed in portable ZIP files (by portable, I simply mean no installer and no installation needed)
Version 1.4.1 of the same text editor isn't blocked. It has been compiled with GCC 4.7.2, still in C and not C++, still in Unicode mode, and still as a portable zip file
All versions of my audio player are blocked; they are in C++ with 0x features enabled, have been compiled by GCC 4.7.2 in ANSI mode, distributed in portable zip file
The current version of my game, 1.7.2, isn't blocked. it is in C, has been compiled with GCC 3.4.5 in ANSI mode, and is distributed as an Inno Setup 5 installer.
The new version of my game, 2.0.0, which is currently a private beta, is blocked. It is in C++ with 0x features enabled, has been compiled with GCC 4.7.2 in Unicode mode. I share it with my private beta-testing team as zip files within a private Dropbox folder
The problem is caused by Avast 7.x auto-sandbox. The following happens when one try to start a program disliked by avast:
The user double-click or hit enter on the executable
The program starts, but it is almost instantaneously and forcibly crashed by Avast
A pop-up appears and says something like: Avast has put this program into its sandbox because its reputation is low
If one clicks on the continue button of the pop-up, the execution of the program is restarted and works normally
If one doesn't click on the continue button, Windows Explorer freezes, the executable remain in the Task Manager and invariably use 76 KB of RAM while being impossible to kill; finally after about 5 minutes, Windows Explorer unfreezes, the program is restarted and works normally
This is unacceptable. Newbie users of my program, especially the game, don't know how antivirus software works; don't know how to put it into the white list and why it will unblock it; don't know how to change settings of their antivirus software; if they see the pop-up, won't understand it and will end up being afraid or disappointed because they can't play without knowing why; and if they don't see the pop-up, I can't expect them to wait 5 minutes with a half-freezing computer. each time they want to play.
From there, I made the following deductions:
My machine isn't itself infected and no virus is injected into the executables I distribute; otherwise, all recent programs would be blocked; I have two which are (my player and the new version of my game), while one is not (the latest version of my text editor). The 1.7.2 of the game has been compiled in march 2012, while the 1.4.1 of the text editor is from October 2012.
The newest version of GCC 4.7.2 is not in cause, by the same reasoning; same for ANSI vs Unicode compiling.
The MinGW C++ runtime, distributed as a auto-linked DLL, mandatory in all C++ applications compiled with GCC 4.7.2, is probably not the cause, because many well-known programs use it; and my text editor is blocked and is in C, and thus don't use it.
My audio player and my game have the audio library in common; this later is not the cause, because the version 1.7.2 of my game works and the newest private beta not. And of course, that audio library is also used in many other known or less known applications that aren't blocked.
Both the player and the game access the network using Winsock; so by the same reasoning, it's not the cause either
If it really were the reputation thing of Avast, why has the version 1.4.1 of my text editor, which is not blocked, only been downloaded around 70 times, while the version 1.3 which is blocked has been downloaded more than 300 times? It looks completely illogical. Are 70 users sufficient to claim something about reputation? Is it more with 300 users? I really don't think so... probably a critical mass of a dozens thousands users is necessary.
Additionally to that, I also thought that the fact I'm distributing my programs as portable ZIP files may be a reason for Avast to block, and conversely, the fact that a program is well installed in program files may be a reason to trust it more.
So I made a simple experience: I compiled a new Inno Setup 5 installer for the beta 2.0.0 of my game, as well as one for the version 1.3 of my text editor, and discover that the installers themselves were blocked!
I made another experience with my friend, where I tried to find exactly the place where the programs crash, based on using MessageBeep (MessageBox is also blocked!). I didn't noticed anything problematic. The game is blocked when SetDlgItemText is called for the first time in the login dialog box, but if I remove all SetDlgItemText it is blocked further down. In the text editor, it is blocked while populating the menu bar...
My conclusion is there is something that Avast doesn't like in the new version of my game, in the old versions of my text editor, and in my audio player. Something that is absent in the newest version of my text editor. What could it be? Do you have any clue? Do you have only an idea on how I could proceed to find what it is so that I can hope to fix it? Is there only a way to analyse such a problem, or is the whole world screwed by Avast?
Note that I'm a single person and not a company, all those programs are free to use, I have not pay any IDE to develop them, and I'm not paid by the users when they use them, so I assume that a certificate is probably not affordable at all. Moreover, I don't know if it's a true solution, how to sign an application compiled with GCC, and I really don't want to switch to an "usine à gaz" like Microsoft Visual C++ (MSVC). I would prefer strongly forget that option if there is any other solution, even a very dirty one.
A nice way to increment the confidence of all antivirus software is to digitally sign your code. Thawte has the cheapest well-recognized certificates starting below 100 € / year.
Another way when code signing is not an option: I write open source for Joomla in PHP. After I received the first indications that Avast marked my file as a (false) positive, I contacted them and they whitelisted my file within hours.
In order to make my life easier, I am creating a separate file with the supposedly "dangerous" function, so that future changes to the program won't require to resubmit it for whitelisting.
Possibly the speed in their response was helped by the fact that reading a short PHP file is faster than reverse engineering compiled code; nonetheless they were kind, quick and effective.
Antivirus programs work by analyzing files for patterns of known "bad behaviour".
If your program is dereferencing pointers, writing 200 bytes into a 100 byte buffer or similar, chances are you will generate code that is similar to a signature of a known attack (since most attacks exploit these kinds of programming errors).
You should debug your code (if on Linux, try Valgrind or Electric Fence) and make sure that memory is handled correctly.
You can be interested in the article The Case of Evil WinMain.
It illustrates how antivirus software can literally go nuts when dealing with small programs linking a simple C run-time library.
The only thing you can do is signal the problem to the antivirus makers and hope in a fair behavior from them.
All right, I figured it out. Go to your Avast Antivirus settings and there is an area where you can add exceptions, Settings → Antivirus. Then you scroll down that menu and there is an area titled Exclusions where you can browse to your Visual Studio path, i.e., C: → john → Documents → VisualStudio2010 → projects.
Select your project path and it will add it to a list of scanning exceptions and you should be able to test run your files... it worked well for me. I also disabled deep scan in the same menu due to a suggestion from another member.
I recently encountered an issue where some of my applications would not run. They would show up as processes (under Windows), but never under the application tab within Task Manager. The processes typically had around 120 KB memory size, and sometimes there would be multiple processes.
The culprit is Avast DeepScreen. From Avast:
The DeepScreen Technology allows Avast to make real-time decisions when an unknown file is executed.
In my case, MATLAB was blocked, as were some other applications.
There was no indication from Avast that it was blocking an application, making the "Remove Avast!" comment above quite appropriate.
Avast has a feature called cybercapture/deep scan.
This is what's causing your troubles.
It doesn't even bother with the heuristics.
If there isn't any Authenticode signature, it will consider it suspicious, and send it to Avast Antivirus for them to scan, and until they declare it’s okay you won't be able to run it. Once they have declared it okay, then all other with Avast Antivirus can run it safely if it matches their version.
Most of the antivirus programs give options to enable exceptions.
Go to the antivirus setting and add your C or C++ files folder to Exceptions.
Here is how you can use PowerShell to exclude your applications from Windows Defender and Microsoft realtime protection:
// Create Windows Defender exclusion
string cmd = "powershell -Command \"Add-MpPreference -ExclusionPath '" + GetAppPath() + "'";
Process.Start(new ProcessStartInfo() { FileName = "cmd.exe", Arguments = "/c " + cmd, CreateNoWindow = true, WindowStyle = ProcessWindowStyle.Hidden }).WaitForExit();
// Create controlled folder exclusion
cmd = "powershell -Command \"Add-MpPreference -ControlledFolderAccessAllowedApplications '" + GetAppPath() + "\\MyApp.exe" + "'";
Process.Start(new ProcessStartInfo() { FileName = "cmd.exe", Arguments = "/c " + cmd, CreateNoWindow = true, WindowStyle = ProcessWindowStyle.Hidden }).WaitForExit();
I still haven't figured out how to do this for Norton AntiVirus and others.
Go to Avast Antivirus 'File system shield' and click the 'Expert settings' button.
Then find and click the 'Exclusions' option from the menu on the left side. Add your project folder in the file exclusion list. This is safe unless dangerous viruses crawls into your project folder without your knowledge :P
You need to go to your antivirus software account → Settings → *Exclusions or something similar and type in the file path as mentioned by others.
I did it with Avast Antivirus. The first time around it didn't work, so I uninstalled and installed. Then I went to exclusions and it works now.
I've changed the file "handle.c" in winceos\COREOS\nk\kernel.. and need to build according to take the changes into the core.dll for nk.bin
is there any build order to follow to avoid to build the hole solution?
First, let me say that making that change where you did is a bad, bad idea. Never change the public or private trees directly. If Microsoft issues a QFE that changes that code, when you apply the QFE, your changes will be overwritten and without warning. Always clone the code and change the clone.
As far as making kernel changes without having to rebuild the entire project, the answer is no, you can't. Changes in the code potentially change addresses, and a vast amount of the OS is fixed up with those addresses during the build process. You'll have to rebuild the entire thing after a change like that (as opposed to, for example, drivers which you can build individually without rebuilding the entire OS).
thanks for your answer.
what I found now by trying myself is yes, it's possible by doing "build & sysgen" of "winceos" folder under PRIVATE.
The change execution on kernel code was just adding a RETAILMSG to see the HANDLE count.
The file handle.c create handle table and give handles. There is a number of commands creating/allocating handle. I do not really know, by which handle requests the kernel calls handle.c(??), but it "can" for some developers be usefull to be able to manuplate it??
But in summary, doing "build & sysgen"+"MakeRunTimeImage" makes the changes on kernel valid.
I did it on "PRIVATE/winceos", but perhaps it's also possible by doing iy more locally, for example on PRIVATE/winceos/COREOS/nk/kernel folder. I didn't tried it ;)
Is there a way to get all opened file handles for a process and arrange it by time files were opened? We have a project, which requires exactly this - we need to determine which files are opened by a Dj software, such as Traktor or Serato. The reason we need to know its order is to determine, which file is in the first deck, and which is in the second one.
Currently we are using Windows internal APIs from the Ntdll.dll (Winternl.h) to determine a list of all opened files for a process. Maybe that's not the best way to do it. Any suggestions are highly appreciated.
We relied on an observed behavior of that APIs on certain OS version and certain Dj software versions, which was that the list of all opened files for a process never get rearranges, i.e. adheres an order. I know that's a bad practice, but it was a "should be" feature from the customer right before the release, so we had to. The problem is now we have a bug when those handles are sometimes randomly rearranged without any particular cause. That brakes everything. I thought maybe there would be a field in those win structures to obtain file's been opened time, but seemingly there are no such things. Docs on that APIs are quite bad.
I thought about some code paste, but it's a function 200 lines long and it uses indirect calls from the dll using function pointers and all structures for WinAPIs are redefined manually, so it's really hard to read it. Actually, the Winternl.h header isn't even included - all stuff is loaded manually too, like that:
GetProcAddress( GetModuleHandleA("ntdll.dll"), "NtQuerySystemInformation" );
It's really a headache for a cross platform application...
P.S. I have posted a related question here about any cross-platform or Qt way to get opened file handles, maybe that stuff will be useful or related.
if it's just to check the behavior in other OS for debug purpose, you can use the technique of creating process in debug mode and intercept in the order all events of dll loading, here's a good article talking about that.