C++ SYSENTER x86 calls in inline assembly - c++

Question

I'm about to learning how sysenter on x86 works. and i created a simple console application on x86 platform, which should call the NtWriteVirtualMemory function manually in inline assembly.
i started with this code here, but it seems that the compiler dont understand the opcode "sysenter" so i decided to _emit them with the bytes for sysenter.(maybe i need to change something in my project settings?) it compiles but when its about to calling the function visual studio gives me an error that my ret is an illegal instruction while executing, and the program stops.
someone have knowledge how to do that correctly?
#include <windows.h>
#include <iostream>
__declspec(naked) void __KiFastSystemCall()
{
__asm
{
mov edx, esp
// need to emit "sysenter" because of syntaxerrors, "Opcode"; "newline"
_emit 0x0F
_emit 0x34
ret // illegal instructiona after execute?
}
}
void Test_NtWriteVirtualMemory(HANDLE hProcess, PVOID BaseAddress, PVOID Buffer, SIZE_T sizeToWrite, SIZE_T* NumberOfBytesWritten)
{
__asm
{
push NumberOfBytesWritten
push sizeToWrite
push Buffer
push BaseAddress
push hProcess
mov eax, 0x3A // Syscall ID NtWriteVirtualMemory in Windows10
mov edx, __KiFastSystemCall
call edx
add esp, 0x14 // 5 push * 4 bytes 20 dec
retn
}
}
void Test_NtWriteVirtualMemory(HANDLE hProcess, PVOID BaseAddress, PVOID Buffer, SIZE_T sizeToWrite, SIZE_T* NumberOfBytesWritten)
{
__asm
{
push NumberOfBytesWritten
push sizeToWrite
push Buffer
push BaseAddress
push hProcess
mov eax, 0x3a // Syscall ID NtWriteVirtualMemory in Windows10
mov edx, 0x76F88E00
call edx
ret 0x14
}
}
int main()
{
std::cout << "Test Hello World\n";
HANDLE hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, GetProcessId("MyGame.exe"));
if (hProcess == NULL)
return false;
DWORD TestAddress = 0x87A0B4; // harcoded
DWORD TestValue = 4;
Test_NtWriteVirtualMemory(hProcess, (PVOID)TestAddress, (PVOID)TestValue, sizeof(DWORD), NULL);
CloseHandle(hProcess);
return 0;
}

Answer 1

Do you have a 32-bit only version of Windows?
sysenter was the "successor" of int 2eh and introduced during the Windows XP era.
The 64-bit versions of Windows don't use it, in fact it was removed since:
sysenter and sysret are illegal in long mode in an AMD CPU (irrespective of the compatibility mode).
The IA32_SYSENTER_CS MSR is left to zero by a 64-bit version of Windows1.
This will cause a #GP fault when executing sysenter.
If you single-step through your __KiFastSystemCall you should see the debugger catch an exception with code 0xc0000005 when executing sysenter.
So, in order to use sysenter you must have a real 32-bit version of Windows.
Running a 32-bit program on a 64-bit version of Windows won't work, that's compatibility mode (done through the WOW64 machinery).
If, besides having a 64-bit version of Windows, you also have an AMD CPU then it won't work double time.
Windows 64-bit uses syscall for 64-bit program or an indirect call to the WOW32Reserved field of the TEB2, you should use those.
Beware that the 64-bit system call convention is slightly different from the usual one: particularly it assumes the syscall is in a function of its own, thus it expects the parameters on the stack to be shifted up by 8.
Plus, the first parameter must be in r10, not rcx.
For example, if you inline the syscall instruction, the first parameter on the stack (if any) must be at rsp + 28h and not at rsp + 20h.
The 32-bit compatibility mode syscall convention is also different, you need to set both eax and ecx to specific values.
I didn't dig what exactly ecx is used for but it may be related to an optimization called Turbo thunks and must be set to a specific value.
Note that while syscall numbers are very volatile, turbo thunks are even more because they can be disabled by the admin.
1I don't have a definitive source for this, it is just zero on my version of Windows and it makes sysenter fault.
2I.e. a call DWORD [fs:0c0h], this will point to a code that will jump to a gate descriptor for a 64-bit code segment that in turn will execute a syscall

Answer 2

Since you got illegal instruction on the ret instruction rather than the sysenter instruction, you know the sysenter instruction was decoded correctly by the CPU. Your call got into kernel mode, but the kernel didn't like your system call invocation.
Probably it was depending on user-space to help save some registers because sysenter is very minimal. Check the stack pointer after returning from the kernel as you single-step before letting ret execute.
I'd be only speculating as the the problem, but wrapping the syscall gate in another function call looks wrong to my eyes. As I said in comments, do not do this because the syscall numbers can change on you.
Under Linux, 32-bit processes call through the VDSO (a library injected into their address space by the kernel) to get the optimal system-call instruction, used in a way that matches what the kernel wants. (sysenter doesn't preserve the stack pointer so user-space has to help.)
Perhaps if you want to play with this instruction you're better off writing a toy OS.
Sorry it's not a ton of answer, but it's not completely unreasonable.

Answer 3

Making system calls in x86 Windows is different from x64. You need to specify the correct arguments length in ret otherwise you will get illegal instruction and/or runtime esp.
Furthermore I don't recommend you to use inline assembly, instead use it inside an .asm file or as shellcode.
To make a correct x86 system call on x86 Windows:
mov eax, SYSCALL_INDEX
call sysentry
ret ARGUMENTS_LENGTH_SIZE
mov edx,esp
sysenter
retn
To make a correct x64 system call on x64 Windows:
mov eax, SYSCALL_INDEX
mov r10,rcx
syscall
retn
The above will work 100% correctly on any x86 and x64 Windows (tested). Can't help you with inline assembly though, because I never used it that way.
Enjoy.