MASM Fixing 64 bit Truncation in a DLL - c++

Question

I am working with the Adobe Flash ocx by loading it into my C++ program. The ocx is supposed to be 64 bit but for some reason it has issues when I compile with the x64 platform. I have read up on this and found that it is likely that some function receives DWORD userData instead of void* userData through some structure and then casts it to an object pointer. This works ok in a 32-bit environment, but crashes in 64-bit.
The disassembly of the function calls inside the ocx that cause the crash are the following lines:
mov ecx,r8d
The first operation copies only low 32-bits from R8D to ECX (ECX is 32-bit).
cmp dword ptr [rcx+11BCh],0
The second operation accesses 64-bit register, where low 32-bits contains correct address and high 32-bits contains some junk. Leading to a crash, of course.
Solution
I have read that one possible solution is to do the following:
Create an asm file containing the following code:
nop
nop
nop
mov ecx,r8d
cmp dword ptr [rcx+11BCh],0
nop
nop
nop
mov rcx,r8d // I've replaced ecx with rcx here
cmp dword ptr [rcx+11BCh],0
Build an obj file using this asm file and MASM.exe
Open the obj file with a hex editor and locate the 90's that represent the nop
In the Flash ocx locate the first string of bytes between the nops and replace it with the new string of bytes that comes after the nops. This will change it from 32 bit to 64 bit function calls.
Problem
I have attempted this by making the following asm file and building it with ml64.exe (I do not have masm.exe but I think that ml.exe is the new 32 bit version of it, and this code would only build with the ml64.exe, probably because of the 64-bit only operators?):
TITLE: Print String Assembly Program (test.asm)
.Code
main Proc
nop
nop
nop
mov ecx,r8d
cmp dword ptr [rcx+11BCh],0
nop
nop
nop
mov rcx,r8
cmp dword ptr [rcx+11BCh],0
main ENDP
END
I had trouble getting it to build (I kept getting errors about instruction length matching) until I changed r8d to r8 in the second section.
I got this obj to build, and opened it with a hex editor and was able to locate the two byte strings. But where my problem comes is that when I search for the first byte string that should be in the flash ocx, I cannot find it. It is not there, so I cannot replace it with the second one.
What am I doing wrong?
Thanks!

Answer 1

Create an asm file containing the following code:
nop
nop
nop
mov ecx,r8d
cmp dword ptr [rcx+11BCh],0
nop
nop
nop
mov rcx,r8d // I've replaced ecx with rcx here
cmp dword ptr [rcx+11BCh],0
Build an obj file using this asm file and MASM.exe
Open the obj file with a hex editor and locate the 90's that represent the nop
In the Flash ocx locate the first string of bytes between the nops and replace it with the new string of bytes that comes after the nops. This will change it from 32 bit to 64 bit function calls.
I made the following asm file and built it with ml64.exe
TITLE: Print String Assembly Program (test.asm)
.Code
main Proc
nop
nop
nop
mov ecx,r8d
cmp dword ptr [rcx+11BCh],0
nop
nop
nop
mov rcx,r8
cmp dword ptr [rcx+11BCh],0
main ENDP
END
I got this obj to build, and opened it with a hex editor and was able to locate the two byte strings. I found the first byte string in the Flash OCX and changed it to the second one. (The only actual change was a 41 to a 49 in the strings)