virtual filesystem design - c++

Question

Im starting a protector/packer/binder like project.
the goal is when you have a full app directory with
/images/
/music/
base *.ini files
dlls
exes
you just use packer.exe on it and all these files are packed, encrypted, and stored in the resulting exe.
the resulting exe then creates a transparent virtual filesystem that falls back to the "real" one if a file is not found.
i allready can handle (not very accurately) loading dlls from memory, etc but i have a problem with the hmm hooks..
for now, as a ProofOfConcept im attaching a debbuger (written in c++) to a target.exe
it looks somewhat like
======= Started [target.exe] =======
> Placing breakpoint on EP : 0x401130
Process started
Loaded module : [target.exe]
Loaded module : [ntdll.dll]
Loaded module : [kernel32.dll]
[...]
Break point at [0x401130]
> Restored EP byte.
Loaded module : [bass.dll]
Break point at [0x760fcc4e]
Found set bp : kernel32!CreateFileW
[!] CreateFileW Callback Function :
FileName : C:\Users\user\Desktop\cppve\loader\bin\Debug\target.exe
Access : 0x80000000
Return Addr: 0x741b91e6
> Re-setting bp at [0x760fcc4e]
Break point at [0x760fcc4e]
Found set bp : kernel32!CreateFileW
[!] CreateFileW Callback Function :
FileName : .\beyond_v.mod
Access : 0x80000000
Return Addr: 0x760fcfa0
i am handling breakpoints in the debugger for things like CreateFileW ReadFile etc
im having problems in supplying the target with useable data.
should i create a fake handle and then catch it and process it ? or are there too many things that can go very wrong with that approach ?
here is a sample callback function for CreateFileW
void callback_createfilew(CONTEXT* ct){
//stub
cout<<"[!] CreateFileW Callback Function :"<<endl;
void* returnaddr=MemReadDwordPtr(hProcess,(void*)ct->Esp);
string fn=MemReadCString(hProcess,MemReadDwordPtr(hProcess,(void*)ct->Esp+4),true);
void* access=MemReadDwordPtr(hProcess,(void*)ct->Esp+8);
void* sharemode=MemReadDwordPtr(hProcess,(void*)ct->Esp+12);
void* dwCreationDisposition=MemReadDwordPtr(hProcess,(void*)ct->Esp+20);
void* dwFlagsAndAttributes=MemReadDwordPtr(hProcess,(void*)ct->Esp+24);
cout<<" FileName : "<<fn<<endl;
cout<<" Access : "<<(void*)access<<endl;
cout<<" Return Addr: "<<(void*)returnaddr<<endl;
if(fn.compare(".\\beyond_v.mod")==0){
// this is wrong, we need to call it from the target process...
HANDLE ret=CreateFileA(".\\_beyond_v.mod",(DWORD)access,(DWORD)sharemode,NULL,(DWORD)dwCreationDisposition,(DWORD)dwFlagsAndAttributes,NULL);
ct->Esp+=0x20;
ct->Eax=(DWORD)ret;
ct->Eip=(DWORD)returnaddr;
}
should i make a codecave in the process and push shellcodes [ Edit: sorry, i use many of these words to describe different things but i think you will catch what i ment :) ] there to execute my faking code ?
or maybe inject a dll that will handle int3s and pass control to it via exception handlers set up by the loader ? however that can proove to be tricky... that dll would have to be in the virtual filesystem ! so i would have to hand-load it before any other initialisation takes place.
i would like, in the final version, to completly drop the debugger. it will only cause problems and seriously comprimise the protector part of the project.

Answer 1

If you want your "packer" to operate transparently on precompiled binaries and want everything to be inside the resultant single binary, the packer needs to add the hooking code to the binary, perhaps making it execute as the very first thing and only then pass control to the original entry point of the binary. This isn't very trivial, although is certainly doable.
But you have another problem here. This hooking code will contain the decryption code and probably the key as well and this all thing is breakable by a good programmer with a debugger and some other tools.
As for fake handles, I'd see if it's possible to open a file multiple times and get different handles. If it is, just open any existing file for reading in a shared mode, remember the handle and use it for an in-memory file. Need another handle? Open the file again to get one. That will guarantee no collision with other real handles.