How to call a JNI Java function from MASM - java-native-interface

I'm writing simple program in assembler which will be called from Java by JNI.
I want to execute function from JNI, which according to the documentation is under 187 index. While i'm performing this line:
call qword ptr[rax + 187*8]
I'm reciving access violation error. What am I possible doing wrong? (I'm multipling index by 8 because of x64 architecture).
All code which i'm performing:
push rdx ; save Java array pointer for later use
push rdi ; save JNIEnv pointer for later use
push rcx ; save array length for later use
mov rsi, rdx ; set array parameter for GetIntArrayElements
mov rax, [rdi] ; get location of JNI function table
xor edx, edx ; set isCopy parameter to false for GetIntArrayElements
call qword ptr[rax + 187*8]

Related

Implementation of Stackful Coroutine in C++

Now I am trying to implement stackful coroutine in C++17 on Windows x64 OS, but, unfortunately, I have encountered the problem: I can't throw exception in my coroutine, if I do so, the program is immediately terminated with a bad exit code.
Implementation
At the begining, I allocate a stack for a new coroutine, the code looks something like that:
void* Allocate() {
static constexpr std::size_t kStackSize{524'288};
auto new_stack{::operator new(kStackSize)};
return static_cast<std::byte *>(new_stack) + kStackSize;
}
The next step is setting a trampoline function on the recently allocated stack. The code is written using MASM, since I utilize MVSC (I would like to use GCC and NASM but I have the problem with thread_local variables, see question, if it is interesting):
SetTrampoline PROC
mov rax, rsp ; saves the current stack pointer
mov rsp, [rcx] ; sets the new stack pointer
sub rsp, 20h ; shadow stack
push rdx ; saves the function pointer
; place for nonvolatile registers
sub rsp, 0e0h
mov [rcx], rsp ; saves the moved stack pointer
mov rsp, rax ; returns the initial stack pointer
ret
SetTrampoline ENDP
Then I switch machine context with this assembly function (I read this calling convetion):
SwitchContext PROC
; saves all nonvolatile registers to the caller stack
push rbx
push rbp
push rdi
push rsi
push r12
push r13
push r14
push r15
sub rsp, 10h
movdqu [rsp], xmm6
; ... pushes xmm7 - xmm14 in here, removed for brevity
sub rsp, 10h
movdqu [rsp], xmm15
mov [rdx], rsp ; saves the caller stack pointer
SwitchContextFinally PROC
mov rsp, [rcx] ; sets the callee stack pointer
; takes out the callee registers
movdqu xmm15, [rsp]
add rsp, 10h
; ... pops xmm7 - xmm14 in here, removed for brevity
movdqu xmm6, [rsp]
add rsp, 10h
pop r15
pop r14
pop r13
pop r12
pop rsi
pop rdi
pop rbp
pop rbx
ret
SwitchContextFinally ENDP
SwitchContext ENDP
Problem
Inside the trampoline I just invoke any passed function and within these functions I can't throw exceptions and catch them instantly in the same fucntion. What have I done wrong? Is it possible to throw exceptions in my case? Should I have shadow stack in SetTrampoline?
Also, I guarantee that the exception thrown don't go outside the trampoline function.

Why are parameters allocated below the frame pointer instead of above?

I have tried to understand this basing on a square function in c++ at godbolt.org . Clearly, return, parameters and local variables use “rbp - alignment” for this function.
Could someone please explain how this is possible?
What then would rbp + alignment do in this case?
int square(int num){
int n = 5;// just to test how locals are treated with frame pointer
return num * num;
}
Compiler (x86-64 gcc 11.1)
Generated Assembly:
square(int):
push rbp
mov rbp, rsp
mov DWORD PTR [rbp-20], edi. ;\\Both param and local var use rbp-*
mov DWORD PTR[rbp-4], 5. ;//
mov eax, DWORD PTR [rbp-20]
imul eax, eax
pop rbp
ret
This is one of those cases where it’s handy to distinguish between parameters and arguments. In short: arguments are the values given by the caller, while parameters are the variables holding them.
When square is called, the caller places the argument in the rdi register, in accordance with the standard x86-64 calling convention. square then allocates a local variable, the parameter, and places the argument in the parameter. This allows the parameter to be used like any other variable: be read, written into, having its address taken, and so on. Since in this case it’s the callee that allocated the memory for the parameter, it necessarily has to reside below the frame pointer.
With an ABI where arguments are passed on the stack, the callee would be able to reuse the stack slot containing the argument as the parameter. This is exactly what happens on x86-32 (pass -m32 to see yourself):
square(int): # #square(int)
push ebp
mov ebp, esp
push eax
mov eax, dword ptr [ebp + 8]
mov dword ptr [ebp - 4], 5
mov eax, dword ptr [ebp + 8]
imul eax, dword ptr [ebp + 8]
add esp, 4
pop ebp
ret
Of course, if you enabled optimisations, the compiler would not bother with allocating a parameter on the stack in the callee; it would just use the value in the register directly:
square(int): # #square(int)
mov eax, edi
imul eax, edi
ret
GCC allows "leaf" functions, those that don't call other functions, to not bother creating a stack frame. The free stack is fair game to do so as these fns wish.

Decoding assembly from MSVC 32-bit release (homework). What does no-op do?

Hi heads up this is a homework. I'm given an assembly generated by MSVC 32-bit Release with optimizations on, and I'm supposed to decode it back into C++. I've included the top of the function to the line I'm having problems with. The comments are mine, which I'm wrote while trying to understand this.
Note: Code is supposedly generated from C++. Not traditional ASM.
Note 2: There is one area of undefined behavior in the code.
Here are the lines I'm stuck with
TheFunction: ; TheFunction(int* a, int s);
0F2D4670 push ebp ; Push/clear/save ebp
0F2D4671 mov ebp,esp ; ebp now points to top of stack
0F2D4673 push ecx ; Push/clear/save ecx
0F2D4674 push ebx ; Push/clear/save ebs
0F2D4675 push esi ; Push/clear/save esi
0F2D4676 mov ebx,edx ; ebx = int s
0F2D4678 mov esi,1 ; esi = 1
0F2D467D push edi ; calling convention ; Push/clear/save edi
0F2D467E mov edi,dword ptr [a (0F2D95E8h)] ; edi = a[0]
0F2D4684 cmp ebx,esi ; if(s < 1)
0F2D4686 jl SomeFunction+3Ch (0F2D46ACh) ; Jump to return
0F2D4688 nop dword ptr [eax+eax] ; !! <-- No op involving dereferencing? What does this do?
0F2D4690 mov eax,dword ptr [edi+esi*4-4] ; !! <-- edi is *a, while esi is 1. There is no address
here!
..... More code but I've figured these out ....
I've more or less got the gist of the function. Its a function that takes a pointer to an int, with an underlying array, and a size. It then goes through each element in the array from last to first, adding to each subsequent one and printing it out. However, I still haven't got the details down and need help
Two questions, both at the end of the code snippet. What does no op on a dereference pointer do, and am I reading the last line in that its attempting to dereference something not in memory?
The nop dword ptr [eax+eax] instruciton does nothing. It doesn't even access the memory location given by the operand. It literally performs no operation.
It's just there so the next instruction is aligned to a 16-byte boundary. You'll notice that next instruction address is 0F2D4690 which ends with 0 which means it's 16-byte aligned. This can improve the performance of loops. Somewhere there will be an instruction that jumps back to 0F2D4690 as part of a loop. This particular form of a NOP instruction is used because it encodes a single NOP instruction in 8 bytes.
There is no corresponding C++ code for this instruction. You shouldn't try to represent it in your C++ code, just ignore it.
Also note that your comment for mov edi,dword ptr [a (0F2D95E8h)] is incorrect. Instead of being edi = a[0] it's simply edi = a. The variable a isn't a parameter at all, instead it's a global (or file level static) variable located at memory location 0F2D95E8h. This instruction just loads the value from memory.

C++ inline ASM exception on return

Due to a WPO patch the way a function I called through an injected DLL changed.
The function is a __fastcall
The original function looked like
PUSH EAX
MOV EAX,DWORD PTR SS:[ESP]
PUSH EAX
LEA EBX,[ARG.22]
LEA EDI,[ARG.23]
CALL Function
So I could call it via:
Push ebx
Push edi
Push 0
Push 0
lea ebx,dword ptr ss:[ecx]
lea edi,dword ptr ss:[edx]
call Function
Pop edi
Pop ebx
retn
The function only needed 2 ascii strings.
Now after the WPO the function changed to
PUSH 0
LEA EDX,[LOCAL.22]
PUSH EDX
LEA EDX,[LOCAL.23]
XOR ECX,ECX
CALL Function
A common fastcall, which looks simpler. But the issue started that the ebp register carried a number while esi and edi the same strings but in Unicode.
While the call still needed only 2 arguments the registers contained additional which was required.
So instead of calling the function via 2 Ascii on ecx and edx I wrote a struct which contained the strings as ascii and unicode.
My attempt to solve it looked like
pushad
push 0
lea edi,dword ptr ss:[ecx+0x20]
lea esi,dword ptr ss:[ecx]
mov ebp, 100
lea edx,dword ptr ss:[ecx+0x50]
push edx
lea edx,dword ptr ss:[ecx+0x40]
xor ecx, ecx
call Function
pop edx
popad
retn
I followed it in the debugger and the call is processed as it should be, but after the the function returns to my asmstub and returns to my c++ code my code creates an exception on write.
Did I make a fundamental asm mistake such as messing up the order which causes the exception?

MSVC Assembly function arguments C++ vs _asm

I have a function which takes 3 arguments, dest, src0, src1, each a pointer to data of size 12. I made two versions. One is written in C and optimized by the compiler, the other one is fully written in _asm. So yeah. 3 arguments? I naturally do something like:
mov ecx, [src0]
mov edx, [src1]
mov eax, [dest]
I am a bit confused by the compiler, as it saw fit to add the following:
_src0$ = -8 ; size = 4
_dest$ = -4 ; size = 4
_src1$ = 8 ; size = 4
?vm_vec_add_scalar_asm##YAXPAUvec3d##PBU1#1#Z PROC ; vm_vec_add_scalar_asm
; _dest$ = ecx
; _src0$ = edx
; 20 : {
sub esp, 8
mov DWORD PTR _src0$[esp+8], edx
mov DWORD PTR _dest$[esp+8], ecx
; 21 : _asm
; 22 : {
; 23 : mov ecx, [src0]
mov ecx, DWORD PTR _src0$[esp+8]
; 24 : mov edx, [src1]
mov edx, DWORD PTR _src1$[esp+4]
; 25 : mov eax, [dest]
mov eax, DWORD PTR _dest$[esp+8]
Function body etc.
add esp, 8
ret 0
What does the _src0$[esp+8] etc. even means? Why does it do all this stuff before my code? Why does it try to [apparently]stack anything so badly?
In comparison, the C++ version has only the following before its body, which is pretty similar:
_src1$ = 8 ; size = 4
?vm_vec_add##YAXPAUvec3d##PBU1#1#Z PROC ; vm_vec_add
; _dest$ = ecx
; _src0$ = edx
mov eax, DWORD PTR _src1$[esp-4]
Why is this little sufficient?
The answer of Mats Petersson explained __fastcall. But I guess that is not exactly what you're asking ...
Actually _src0$[esp+8] just means [_src0$ + esp + 8], and _src0$ is defined above:
_src0$ = -8 ; size = 4
So, the whole expression _src0$[esp+8] is nothing but [esp] ...
To see why it does all these stuff, you should probably first understand what Mats Petersson said in his post, the __fastcall, or more generally, what is a calling convention. See the link in his post for detailed informations.
Assuming that you have understood __fastcall, now let's see what happens to your codes. The compiler is using __fastcall. Your callee function is f(dst, src0, src1), which requires 3 parameters, so according to the calling convention, when a caller calls f, it does the following:
Move dst to ecx and src0 to edx
Push src1 onto the stack
Push the 4 bytes return address onto the stack
Go to the starting address of the function f
And the callee f, when its code begins, then knows where the parameters are: dst and src0 are in the registers ecx and edx, respectively; esp is pointing to the 4 bytes return address, but the 4 bytes below it (i.e. DWORD PTR[esp+4]) is exactly src1.
So, in your "C++ version", the function f just does what it should do:
mov eax, DWORD PTR _src1$[esp-4]
Here _src1$ = 8, so _src1$[esp-4] is exactly [esp+4]. See, it just retrieves the parameter src1 and stores it in eax.
There is however a tricky point here. In the code of f, if you want to use the parameter src1 multiple times, you can certainly do that, because it's always stored in the stack, right below the return address; but what if you want to use dst and src0 multiple times? They are in the registers, and can be destroyed at any time.
So in that case, the compiler should do the following: right after entering the function f, it should remember the current values of ecx and edx (by pushing them onto the stack). These 8 bytes are the so-called "shadow space". It is not done in your "C++ version", probably because the compiler knows for sure that these two parameters will not be used multiple times, or that it can handle it properly some other way.
Now, what happens to your _asm version? The problem here is that you are using inline assembly. The compiler then loses its control to the registers, and it cannot assume that the registers ecx and edx are safe in your _asm block (they are actually not, since you used them in the _asm block). Thus it is forced to save them at the beginning of the function.
The saving goes as follows: it first raises esp by 8 bytes (sub esp, 8), then move edx and ecx to [esp] and [esp+4] respectively.
And then it can enter safely your _asm block. Now in its mind (if it has one), the picture is that [esp] is src0, [esp+4] is dst, [esp+8] is the 4 byte return address, and [esp+12] is src1. It no longer thinks about ecx and edx.
Thus your first instruction in the _asm block, mov ecx, [src0], should be interpreted as mov ecx, [esp], which is the same as
mov ecx, DWORD PTR _src0$[esp+8]
and the same for the other two instructions.
At this point, you might say, aha it's doing stupid things, I don't want it to waste time and space on that, is there a way?
Well there is a way - do not use inline assembly... it's convenient, but there is a compromise.
You can write the assembly function f in a .asm source file and public it. In the C/C++ code, declare it as extern 'C' f(...). Then, when you begin your assembly function f, you can play directly with your ecx and edx.
The compiler has decided to use a calling convention that uses "pass arguments in registers" aka __fastcall. This allows the compiler to pass some of the arguments in registers, instead of pushing onto stack, and this can reduce the overhead in the call, because moving from a variable to a register is faster than pushing onto the stack, and it's now already in a register when we get to the callee function, so no need to read it from the stack.
There is a lot more information about how calling conventions work on the web. The wikipedia article on x86 calling conventions is a good starting point.