Calling API from a x86 DLL loaded into a x64 application - c++

Question

This is an experiment to exploit the availability of a 32-bit "compatibility" mode selector in GDT in Windows x64 which has the value of 0x23.
I 've successfully loaded a x86 DLL into the address space of a x64 application. This is done by first taking all imports of the x86 dll and saving it into a XML, then using a modified MemoryModule to load the DLL from memory and patch the Import Address Table with those pointers.
Then, in the x64 host I jump to a compatibility code segment:
push 0x23
xor rcx,rcx
mov ecx,Start32
push rcx
retf
Start32 is a 32-bit entry point that would call a configured function:
USE32
push eax
push dword [WhatToPass]
mov eax, [WhatToCall]
call eax
pop eax
Then go back to 64-bit
USE64
db 0eah
ret_64:
dd 0
dw 0x33
nop
This works, as long as the configured exp1 function does practically nothing. The concept is fully discussed in my CodeProject article and the source is on GitHub.
It doesn't work when run in Visual Studio, but it does work when run in WinDbg; WinDbg shows a 'x86' running mode when switching.
Now the ambitious project is to call the API from the x86 module. This fails even with a simple MessageBeep(0);. I don't of course expect it at this point to go fully to the function but it doesn't even enter; I get an invalid execution instruction on the call. The address is valid.
What could be wrong? Yes this is too much perhaps but I'm still curious.
Patching the IAT works correctly. Sample code:
void PatchIAT(HINSTANCE h,std::vector<FANDP>* CustomLoading = 0)
{
PCHAR codeBase = (PCHAR)h;
XML3::XML x(xResult);
PIMAGE_NT_HEADERS32 ntheaders = (PIMAGE_NT_HEADERS32)(PCHAR(h) + PIMAGE_DOS_HEADER(h)->e_lfanew);
PIMAGE_SECTION_HEADER pSech = IMAGE_FIRST_SECTION(ntheaders);//Pointer to first section header
DWORD ulsize = 0;
PIMAGE_IMPORT_DESCRIPTOR importDesc = (PIMAGE_IMPORT_DESCRIPTOR)ImageDirectoryEntryToData(h, TRUE, IMAGE_DIRECTORY_ENTRY_IMPORT, &ulsize);
if (!importDesc)
return;
for (; importDesc && importDesc->Name; importDesc++) {
PSTR pszModName = (PSTR)((PBYTE)h + importDesc->Name);
if (!pszModName)
break;
XML3::XMLElement* module = ... ; // find the XML entry
if (!module)
continue;
DWORD* thunkRef;
DWORD* funcRef = 0;
if (importDesc->OriginalFirstThunk) {
thunkRef = (DWORD*)(codeBase + importDesc->OriginalFirstThunk);
funcRef = (DWORD*)(codeBase + importDesc->FirstThunk);
}
else {
// no hint table
thunkRef = (DWORD*)(codeBase + importDesc->FirstThunk);
funcRef = (DWORD *)(codeBase + importDesc->FirstThunk);
}
DWORD V = 0;
for (; *thunkRef; thunkRef++, funcRef++) {
DWORD* wr = (DWORD*)funcRef;
if (IMAGE_SNAP_BY_ORDINAL(*thunkRef)) {
// *funcRef = module->getProcAddress(handle, (LPCSTR)IMAGE_ORDINAL(*thunkRef), module->userdata);
const char* fe = (LPCSTR)IMAGE_ORDINAL(*thunkRef);
// find V from xml
...
}
else {
PIMAGE_IMPORT_BY_NAME thunkData = (PIMAGE_IMPORT_BY_NAME)(codeBase + (*thunkRef));
// *wr = module->getProcAddress(handle, (LPCSTR)&thunkData->Name, module->userdata);
... // Find V from XML
}
// Patch it now...
DWORD dwOldProtect = 0;
if (VirtualProtect((LPVOID)wr, 4, PAGE_READWRITE, &dwOldProtect))
{
memcpy((void*)wr, &V, 4);
VirtualProtect((LPVOID)wr, 4, dwOldProtect, &dwOldProtect);
}
VirtualProtect((LPVOID)V, 4, PAGE_EXECUTE_READWRITE, &dwOldProtect);
}
}
}