X86 assembly cdecl confusion - c++

Morning, I've implemented a cdecl call method into the following 'encryption' routine. However whilst my method works, it's not following the recommended (uni and other sources) exactly.
Advise appreciated (Some comments may be 'wrong' cdecl related functs have be marked for)
My confusion is that I've been told that for each parameter pushed (in this case 2) I must push before the call, push again inside the funct, then do mov [ebp+n] for each additional push - before the main body.
However if I follow the logic of : do a mov ebp shift for every push - without adding additional pushes, all works fine. But this could well be an error.
If required I can post some of reference docs.
code extract:
void encrypt_chars (int length, char EKey)
{char temp_char; // Character temporary store
for (int i = 0; i < length; i++) // Encrypt characters one at a time
{
temp_char = OChars [i]; // Get the next char from Original Chars array
// Note the lamentable lack of comments below!
__asm { //
push eax //making a copy of char to be encrypted index
push ecx //making a copy of the char to be encrypted
//
movzx ecx,temp_char //padding out temp_char
lea eax,EKey //moving Ekey adress to eax, for function to use, eax contents acting as perameter
call encrypt6 //Doing the encryption
add esp, 8 //cdecl added stack pointer baxk
mov temp_char,al //Move the encrypted result into temp_char
pop ecx //resetting registers to before call
pop eax //
}
EChars [i] = temp_char; // Store encrypted char in the Encrypted Chars array
}
return;
// Encrypt subroutine. You should paste in the encryption routine you've been allocated from Bb and
// overwrite this initial, simple, version. Ensure you change the ‘call’ above to use the
// correct 'encryptnn' label where nn is your encryption routine number.
// Inputs: register EAX = 32-bit address of Ekey,
// ECX = the character to be encrypted (in the low 8-bit field, CL).
// Output: register EAX = the encrypted value of the source character (in the low 8-bit field, AL).
__asm {
encrypt6:
push ebp //cdecl Making a copy of base pointer
mov ebp, esp //cdecl storing the point at which the base pointer has shifted the stack
// push ecx //cdecl
// push eax //cdecl
// mov ecx, [ebp + 12] // cdecl storing Ekey adress
// mov eax, [ebp + 8] //cdecl storing temp_char data
ror byte ptr[eax], 1 //Rotating the EKey data right 6 times
ror byte ptr[eax], 1 //
ror byte ptr[eax], 1 //
ror byte ptr[eax], 1 //
ror byte ptr[eax], 1 //
push ecx //Making a copy of ecx
mov ecx, [ebp + 8] // cdecl ,
not byte ptr[eax] //Inverting the Ekey data
movzx edx, byte ptr[eax] //making a copy of the ekey data with leading zero's
pop eax //restoring register to thet temp_char ascii value
xor eax, edx //XORing original temp_char with encrypted temp_Char
ror al, 1 //Rotating the Ekey adreses 8 bit component right twice (To edit)
ror al, 1 //
not al // Inverting the result
add eax, 0x20 // adding a space (32) to result
pop ebp //cdecl ,pop ebp
ret //cdecl
//----
}

Related

Run-Time Check Failure #0. Lost as to how to use CDECL

I've been asked with changing some __asm code so that it implements a C++ calling convention. I've tried using cdecl but I keep getting this error
Run-Time Check Failure #0: The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.
The code is used to encrypt a string one character at a time using a for loop. The function encrypt_1 encodes the character that is passing through the for loop at the time.
I've attempted to use cdecl by assigning different values to the base pointer (ebp) and the stack pointer (esp) for both the for loop and the function but I'm continuously getting Run-Time Check Failure #0, and every attempt I've made to tweak it results in the program breaking. I've been stressing over this for a couple of days, can anybody give me a hint as to where I may be going wrong?
The Body:
void encrypt_chars (int length, char EKey)
{
char temp_char; // Character temporary store
for (int i = 0; i < length; i++) // Encrypt characters one at a time
{
temp_char = OChars[i]; // Get the next char from Original Chars array
// Each character in the string will be encrypted individually
__asm
{ //
push eax // Stores a backup of the location of eax to be used later
push ecx // Stores a backup of the charcter to be encrypted in the stack for later use
push edx // Stores a backup of the location for the encrypted character in the stack for later use
//
movzx ecx, temp_char // Places the character (8 bits) to be encrypted in the ecx register (32 bits) and replaces any extraneous bits with 0 as they are not being used.
lea eax, EKey // places the Ekey in the eax register for later use. Registers are fast than calling variables?
push ebp
mov ebp, esp
// sub esp, 4
push ecx // Parameter for encrypt1. - Temporary Character
push eax // Parameter for encrypt1. - Address for the key.
call encrypt_1 // Begins the Encryption Function
mov temp_char, dl // Stores the encrypted character in the temp_char variable to be reconstruced later
add esp, 8
pop eax // Restoring eax to previous location before parameter call
pop ecx // Restores Temporary Character location before parameter call
pop edx // Restores the edx register to its original value, ready for the next character
pop ecx // Restores the ecx register to its original value, ready for the next character
pop eax // Restores the eax register to its original value, ready for the next character
mov esp, ebp
pop ebp
// ret
}
EChars[i] = temp_char; // Store encrypted char in the Encrypted Chars array
}
return;
The Function:
__asm
{
encrypt_1:
push ebp //Saves the present value of ebp on the stack for later use
mov ebp, esp //Places the stack pointer where the base pointer is so variables can be stored
mov ecx, [ebp +08h]
mov edx, [ebp +0Ch]
push ecx // Stores the original character on the top of the stack
movzx ecx, byte ptr[eax] // Moves the Ekey (8-bit) into the ecx register (32-bit), changing all extraneous bits to 0 so it can be rotated
add cl, 0x01 // Adds 1 to Ekey
rol cl, 1 // Rotates the binary value of Ekey to the left 6 times for further security
rol cl, 1 // Could have just used "rol cl, 6"
rol cl, 1
rol cl, 1
rol cl, 1
rol cl, 1
mov edx, ecx // Move the encrypted Ekey to edx, freeing the ecx register for the original character to be used later.
mov byte ptr[eax], dl // Move byte pointer to the left side of eax
pop ecx // Retrieves the original character to be encrypted
x1 : rol cl, 1 // Rotate the original character one place to the left , encrypting it
dec edx // Decrease the rotated Ekey by one, acting as a counter to continue decrypting the original character
jnz x1 // Jump back to x1 unless the Ekey = 0, which dictates that the encryption is complete
add ecx, 0x20 // Add 32 bits to the encrypted character, filling the register
mov edx, ecx // Place the encrypted character in the edx register, ready to be returned
mov esp, ebp
pop ebp
//add esp, 8
ret // Return the encrypted character
}
//--- End of Assembly code
}
PS. Sorry for some of the messy commented out lines, I've been trying various things trying to get it to work.
There's no reason to write the call in assembly. You should change the calling code to:
EChars[i] = encrypt_1(&EKey, OChars[i]);
If for some reason you feel that you must write it in assembly, then you could write:
temp_char = OChars[i];
__asm
{
movzx ecx, temp_char
lea eax, EKey
push ecx // Second param: character to encrypt
push eax // First param: address of EKey.
call encrypt_1
mov temp_char, al
}
EChars[i] = temp_char;
The Function:
__asm
{
encrypt_1:
push ebp
mov ebp, esp
mov eax, [ebp+08h] // first param: address of EKey
mov dl, [ebp+0Ch] // second param: character to encrypt
mov cl, byte ptr [eax] // get value of EKey
add cl, 0x01
rol cl, 6
mov byte ptr [eax], cl // store updated value of EKey
rol dl, cl
add dl, 0x20
movzx eax, dl // Return encrypted character in eax
mov esp, ebp
pop ebp
ret
}
Other than simplification, the only things that I fixed were:
1. Use of eax without loading it within the function.
2. Reversed order of the function parameters.
3. The EKey parameter to the function wasn't used at all.

High Level to ASM conversion

I'm learning assembly programming and I've been tasked with converting a for loop (along with any array use) into assembly within my program.
The program simply takes an encryption key (EKey) and uses it to encrypt an array of letters (hello for example).
Here is the loop in C++:
void encrypt_chars(int length, char EKey)
{
char temp_char; // char temporary store
for (int i = 0; i < length; i++) // encrypt characters one at a time
{
temp_char = OChars[i]; // temp_char now contains the address values of the individual character
__asm
{
push eax // Save values contained within register to stack
push ecx
movzx ecx, temp_char
push ecx // Push argument #2
lea eax, EKey
push eax // Push argument #1
call encrypt
add esp, 8 // Clean parameters of stack
mov temp_char, al // Move the temp character into a register
pop ecx
pop eax
}
EChars[i] = temp_char; // Store encrypted char in the encrypted chars array
}
return;
// Inputs: register EAX = 32-bit address of Ekey,
// ECX = the character to be encrypted (in the low 8-bit field, CL).
// Output: register EAX = the encrypted value of the source character (in the low 8-bit field, AL).
__asm
{
encrypt:
push ebp // Set stack
mov ebp, esp // Set up the base pointer
mov eax, [ebp + 8] // Move value of parameter 1 into EAX
mov ecx, [ebp + 12] // Move value of parameter 2 into ECX
push edi // Used for string and memory array copying
push ecx // Loop counter for pushing character onto stack
not byte ptr[eax] // Negation
add byte ptr[eax], 0x04 // Adds hex 4 to EKey
movzx edi, byte ptr[eax] // Moves value of EKey into EDI using zeroes
pop eax // Pop the character value from stack
xor eax, edi // XOR character to give encrypted value of source
pop edi // Pop original address of EDI from the stack
rol al, 1 // Rotates the encrypted value of source by 1 bit (left)
rol al, 1 // Rotates the encrypted value of source by 1 bit (left) again
add al, 0x04 // Adds hex 4 to encrypted value of source
mov esp, ebp // Deallocate values
pop ebp // Restore the base pointer
ret
}
//--- End of Assembly code
}
I've looked at the disassembly code and used that to rewrite the ASM code. I typed in the disassembly code as follows however it gives the following errors: http://gyazo.com/3b6875c9e1207df61df4e95506af7ed6
Here is my code:
void encrypt_chars(int length, char EKey)
{
char temp_char;
__asm
{
mov DWORD PTR[rbp - 4], 0
jmp L2
L3:
mov eax, DWORD PTR[rbp - 4]
cdqe
cltq
// ERROR HERE ^ above: error C2400: inline assembler syntax error in 'opcode'; found 'newline'
movzx eax, BYTE PTR EChars[rax]
// ERROR HERE ^ above: error C2400: inline assembler syntax error in 'opcode'; found 'newline'
// another error ^ above: error C2424: '[' : improper expression in 'second operand'
mov BYTE PTR[rbp - 5], al
// OG code
push eax // Save values contained within register to stack
push ecx
movzx ecx, temp_char
push ecx // Push argument #2
lea eax, EKey
push eax // Push argument #1
call encrypt4
add esp, 8 // Clean parameters of stack
mov temp_char, al // Move the temp character into a register
pop ecx
pop eax
// end of OG code
mov eax, DWORD PTR[rbp - 4]
cdqe
movzx edx, BYTE PTR[rbp - 5] // ERROR HERE: error C2400: inline assembler syntax error in 'opcode'; found 'newline'
mov BYTE PTR DChars[rax], dl // ERROR HERE: error C2424: '[' : improper expression in 'first operand'
L2:
mov eax, DWORD PTR[rbp - 4]
cmp eax, DWORD PTR[rbp - 20]
jl L3
}
return;
__asm
{
encrypt4:
push ebp // Set stack
mov ebp, esp // Set up the base pointer
mov eax, [ebp + 8] // Move value of parameter 1 into EAX
mov ecx, [ebp + 12] // Move value of parameter 2 into ECX
push edi // Used for string and memory array copying
push ecx // Loop counter for pushing character onto stack
not byte ptr[eax] // Negation
add byte ptr[eax], 0x04 // Adds hex 4 to EKey
movzx edi, byte ptr[eax] // Moves value of EKey into EDI using zeroes
pop eax // Pop the character value from stack
xor eax, edi // XOR character to give encrypted value of source
pop edi // Pop original address of EDI from the stack
rol al, 1 // Rotates the encrypted value of source by 1 bit (left)
rol al, 1 // Rotates the encrypted value of source by 1 bit (left) again
add al, 0x04 // Adds hex 4 to encrypted value of source
mov esp, ebp // Deallocate values
pop ebp // Restore the base pointer
ret
}
//--- End of Assembly code
}
Would anyone be able to point out where I'm going wrong please? I'm finding this relatively difficult so step-by-step explanations are most welcome. Please let me know where I'm going wrong. Thanks x
EDIT:
My full code: http://pastebin.com/tP9Dgvpn
Try this:
void encrypt_chars(int length, char EKey, char *Msg)
{
int InLength = length;
int counter;
__asm
{
push eax // Counter
mov eax, 0 // Zero counter
push ebx // Value
mov ebx, InLength
jmp L2
L3:
mov counter, EAX
call encrypt4
inc eax // Increment counter.
L2:
cmp eax, ebx
jl L3 // Jump if we haven't reached our count.
pop ebx
pop eax
}
return;
__asm
{
encrypt4:
push eax
push ebx
push edi
mov ebx, counter
add ebx, Msg
mov al, [ebx] // Move character into al
CBW // Make word.
CWDE // Make Dword.
not byte ptr[EKey] // Negation
add byte ptr[EKey], 0x04 // Adds hex 4 to EKey
movzx edi, byte ptr[EKey] // Moves value of EKey into EDI using zeroes
xor eax, edi // XOR character to give encrypted value of source
pop edi // Pop original address of EDI from the stack
rol al, 1 // Rotates the encrypted value of source by 1 bit (left)
rol al, 1 // Rotates the encrypted value of source by 1 bit (left) again
add al, 0x04 // Adds hex 4 to encrypted value of source
mov [ebx], al
pop ebx
pop eax
ret
}
//--- End of Assembly code
}
I call it with this snippet:
EKey = 'i';
sprintf(OChars, "hello");
printf("%s\n", OChars);
encrypt_chars(sizeof(OChars), EKey, OChars);
printf("%s\n", OChars);
My output looks like this:
hello
╧4▀↑█⌐
This matches your output but all except the first character. I am convinced the algorithm looks correct and I can't spend any more time on it. I believe it can be differences in character sets. Try it out.
Good luck!

Assembly asm x86 Encryption/Decryption Program

I've been creating this program for a while now and I can't work out how to solve the decryption routine, any help would be greatly appreciated.
The encryption part of the code works correctly as of now.
#include <conio.h> // for kbhit
#include <iostream> // for cin >> and cout <<
#include <iomanip> // for fancy output
using namespace std;
#define MAXCHARS 6 // feel free to alter this, but 6 is the minimum
#define dollarchar '$' // string terminator
char OChars[MAXCHARS],
EChars[MAXCHARS],
DChars[MAXCHARS] = "Soon!"; // Global Original, Encrypted, Decrypted character strings
//----------------------------- C++ Functions ----------------------------------------------------------
void get_char (char& a_character)
{
cin >> a_character;
while (((a_character < '0') | (a_character > 'z')) && (a_character != dollarchar))
{ cout << "Alphanumeric characters only, please try again > ";
cin >> a_character;
}
}
//-------------------------------------------------------------------------------------------------------------
void get_original_chars (int& length)
{ char next_char;
length = 0;
get_char (next_char);
while ((length < MAXCHARS) && (next_char != dollarchar))
{
OChars [length++] = next_char;
get_char (next_char);
}
}
//---------------------------------------------------------------------------------------------------------------
//----------------- ENCRYPTION ROUTINES -------------------------------------------------------------------------
void encrypt_chars (int length, char EKey)
{ char temp_char; // char temporary store
for (int i = 0; i < length; i++) // encrypt characters one at a time
{
temp_char = OChars [i]; //
__asm { //
push eax // save register values on stack to be safe
push ecx // Push last parameter first
lea eax,EKey
push temp_char
push eax
call encrypt3 // encrypt the character
mov temp_char, al
add esp, 8 // Clean parameters from stack
pop ecx // restore original register values from stack
pop eax //
}
EChars [i] = temp_char; // Store encrypted char in the encrypted chars array
}
return;
__asm {
encrypt3:
push ebp // Save the old base pointer value
mov ebp, esp // Set the new base pointer value
push edx // Save EDX to the first unused empty stack
push ecx //ecx register containing the temp_char is pushed to the stack
push eax // Save EAX to the first unused empty stack
mov edx, [ebp + 8] // Accessing the last value of ebp
movzx eax, byte ptr[eax] // Move 4 bytes to the EAX register
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
mov edx, eax // Move 4 bytes from EAX into edx
pop eax // Restore original EAX
mov byte ptr[eax], dl //moves the Ekey value into the EAX register as an 8-bit value
pop ecx //stores the current letter being encrypted within the ECX register (it was pushed to the stack earlier in the assembly code).
xor ecx, edx //clears the EDX register of all values
mov eax, ecx // Move 4 bytes from ECX into EAX
ror al, 1 // Rotate AL register one position to the left
ror al, 1 // Rotate AL register one position to the left
ror al, 1 // Rotate AL register one position to the left
pop edx // Restore the value of EDX
pop ebx // Restore original EBX
mov esp, ebp // Dellocate local variables
pop ebp // Restore the original value of EBP
ret // Return EAX value
}
//--- End of Assembly code
}
// end of encrypt_chars function
//---------------------------------------------------------------------------------------------------------------
//---------------------------------------------------------------------------------------------------------------
//----------------- DECRYPTION ROUTINES -------------------------------------------------------------------------
//
void decrypt_chars(int length, char EKey)
{
return;
}
// end of decrypt_chars function
//---------------------------------------------------------------------------------------------------------------
Definitely, write your code in high language before placing into assembly language.
Here are some reasons:
Clobbering edx register
mov edx, [ebp + 8] // Accessing the last value of ebp
movzx eax, byte ptr[eax] // Move 4 bytes to the EAX register
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
mov edx, eax // Move 4 bytes from EAX into edx
In the code above, you move [ebp + 8] into edx. Then you copy eax into edx four instructions later. Why bother with the first instruction here?
Duplicate instructions
One of the common reasons for coding in assembly language is efficiency. Supposedly you can code better than the compiler or use special instructions better than the compiler. You don't, as this example shows:
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
The above should be coded as rol al, 3.
Also, is there a reason you are using al register instead of eax?
Clearing edx register is wrong
The operation does not match the comments.
xor ecx, edx //clears the EDX register of all values
The edx register is on the wrong side.
The statement xor edx, edx actually clears the edx register.
Restart with high level language.
I suggest trashing your assembly language and rewriting the functions in a high level language. Get it working first. Examine the compiler's assembly language. If you can code the algorithm more effectively than the compiler, then do so.

Creating loop in x86 assembly and use of arrays? [duplicate]

I am currently learning assembly programming as part of one of my university modules. I have a program written in C++ with inline x86 assembly which takes a string of 6 characters and encrypts them based on the encryption key.
Here's the full program: https://gist.github.com/anonymous/1bb0c3be77566d9b791d
My code fo the encrypt_chars function:
void encrypt_chars (int length, char EKey)
{ char temp_char; // char temporary store
for (int i = 0; i < length; i++) // encrypt characters one at a time
{
temp_char = OChars [i]; // temp_char now contains the address values of the individual character
__asm
{
push eax // Save values contained within register to stack
push ecx
movzx ecx, temp_char
push ecx // Push argument #2
lea eax, EKey
push eax // Push argument #1
call encrypt
add esp, 8 // Clean parameters of stack
mov temp_char, al // Move the temp character into a register
pop ecx
pop eax
}
EChars [i] = temp_char; // Store encrypted char in the encrypted chars array
}
return;
// Inputs: register EAX = 32-bit address of Ekey,
// ECX = the character to be encrypted (in the low 8-bit field, CL).
// Output: register EAX = the encrypted value of the source character (in the low 8-bit field, AL).
__asm
{
encrypt:
push ebp // Set stack
mov ebp, esp // Set up the base pointer
mov eax, [ebp + 8] // Move value of parameter 1 into EAX
mov ecx, [ebp + 12] // Move value of parameter 2 into ECX
push edi // Used for string and memory array copying
push ecx // Loop counter for pushing character onto stack
not byte ptr[eax] // Negation
add byte ptr[eax], 0x04 // Adds hex 4 to EKey
movzx edi, byte ptr[eax] // Moves value of EKey into EDI using zeroes
pop eax // Pop the character value from stack
xor eax, edi // XOR character to give encrypted value of source
pop edi // Pop original address of EDI from the stack
rol al, 1 // Rotates the encrypted value of source by 1 bit (left)
rol al, 1 // Rotates the encrypted value of source by 1 bit (left) again
add al, 0x04 // Adds hex 4 to encrypted value of source
mov esp, ebp // Deallocate values
pop ebp // Restore the base pointer
ret
}
//--- End of Assembly code
}
My questions are:
What is the best/ most efficient way to convert this for loop into assembly?
Is there a way to remove the call for encrypt and place the code directly in its place?
How can I optimise/minimise the use of registers and instructions to make the code smaller and potentially faster?
Is there a way for me to convert the OChars and EChars arrays into assembly?
If possible, would you be able to provide me with an explanation of how the solution works as I am eager to learn.
I can't help with optimization or the cryptography but i can show you a way to go about making a loop, if you look at the loop in this function:
void f()
{
int a, b ;
for(a = 10, b = 1; a != 0; --a)
{
b = b << 2 ;
}
}
The loop is essentially:
for(/*initialize*/; /*condition*/; /*modify*/)
{
// run code
}
So the function in assembly would look something along these lines:
_f:
push ebp
mov ebp, esp
sub esp, 8 ; int a,b
initialize: ; for
mov dword ptr [ebp-4], 10 ; a = 10,
mov dword ptr [ebp-8], 1 ; b = 1
mov eax, [ebp-4]
condition:
test eax, eax ; tests if a == 0
je exit
runCode:
mov eax, [ebp-8]
shl eax, 2 ; b = b << 2
mov dword ptr [ebp-8], eax
modify:
mov eax, [ebp-4]
sub eax, 1 ; --a
mov dword ptr [ebp-4], eax
jmp condition
exit:
mov esp, ebp
pop ebp
ret
Plus I show in the source how you make local variables;
subtract the space from the stack pointer.
and access them through the base pointer.
I tried to make the source as generic intel x86 assembly syntax as i could so my apologies if anything needs changing for your specific environment i was more aiming to give a general idea about how to construct a loop in assembly then giving you something you can copy, paste and run.
I would suggest to look into assembly code which is generated by compiler. You can change and optimize it later.
How do you get assembler output from C/C++ source in gcc?

Simple encryption Assembly Program - Access violation writing to memory location

I had to implement a cdecl calling convention into this program which originally used a non standardized convention. As far as I can tell it looks right, but I get a unhandled exception error saying "Accress violation writing location 0x00000066, which seems to hit when the program gets down to the line "not byte ptr[eax]" or atleast that is where the arrow points after breaking the program.
Could anyone tell me what is wrong with my program and how I may fix it? Thank you.
void encrypt_chars (int length, char EKey)
{ char temp_char;
for (int i = 0; i < length; i++)
{
temp_char = OChars [i];
__asm {
push eax
movzx eax, temp_char
push eax
lea eax, EKey
push eax
call encrypt
mov temp_char, al
pop eax
}
EChars[i] = temp_char;
return;
// Inputs: register EAX = 32-bit address of Ekey,
// ECX = the character to be encrypted (in the low 8-bit field, CL).
// Output: register EAX = the encrypted value of the source character (in the low 8-bit field, AL).
__asm {
encrypt:
push ebp
mov ebp, esp
mov ecx, 8[ebp]
mov eax, 12[ebp]
push edi
push ecx
not byte ptr[eax]
add byte ptr[eax], 0x04
movzx edi, byte ptr[eax]
pop eax
xor eax, edi
pop edi
rol al, 1
rol al, 1
add al, 0x04
mov esp, ebp
pop ebp
ret
}
By inspection, the comment on the encrypt function is wrong. Remember: the stack grows down, so when the arguments are pushed onto the stack, the ones pushed first have the higher address and, therefore, the higher offset from the base pointer in the stack frame.
The comment to encrypt says:
// Inputs: register EAX = 32-bit address of Ekey,
// ECX = the character to be encrypted
However, your calling sequence is:
movzx eax, temp_char ; push the char to encrypt FIRST
push eax
lea eax, EKey ; push the encryption key SECOND
push eax
call encrypt
So the character is push first. So the character to encrypt But encrypt is loading them this way:
; On function entry, the old Instruction Pointer (4 bytes) is pushed onto the stack
; so now the EKey is +4 bytes from the stack pointer
; and the character is +8 bytes from the stack pointer
;
push ebp
mov ebp, esp
; We just pushed another 4 bytes onto the stack (the esp register)
; and THEN we put the stack pointer (esp) into ebp as base pointer
; to the stack frame.
;
; That means EKey is now +8 bytes off of the base pointer
; and the char to encrypt is +12 off of the base pointer
;
mov ecx, 8[ebp] ; This loads EKey pointer to ECX
mov eax, 12[ebp] ; This loads char-to-encrypt to EAX
The code then proceeds to try to reference EAX as a pointer (since it thinks that's EKey), which is going to cause an access violation since it's your character to encrypt the first time it tries to reference EAX as a pointer, which is here:
not byte ptr[eax]
So your debugger pointer was right! :)
You can fix it just by swapping these two registers:
mov eax, 8[ebp] ; This loads EKey pointer to EAX
mov ecx, 12[ebp] ; This loads char-to-encrypt to ECX
Finally, your call to encrypt doesn't clean up the stack pointer when it's done. Since you pushed 8 bytes of data onto the stack before calling encrypt, and since encrypt does a standard ret with no stack clean-up, you need to clean up after the call:
...
call encrypt
add esp, 8
...