Assembly asm x86 Encryption/Decryption Program - c++

Question

I've been creating this program for a while now and I can't work out how to solve the decryption routine, any help would be greatly appreciated.
The encryption part of the code works correctly as of now.
#include <conio.h> // for kbhit
#include <iostream> // for cin >> and cout <<
#include <iomanip> // for fancy output
using namespace std;
#define MAXCHARS 6 // feel free to alter this, but 6 is the minimum
#define dollarchar '$' // string terminator
char OChars[MAXCHARS],
EChars[MAXCHARS],
DChars[MAXCHARS] = "Soon!"; // Global Original, Encrypted, Decrypted character strings
//----------------------------- C++ Functions ----------------------------------------------------------
void get_char (char& a_character)
{
cin >> a_character;
while (((a_character < '0') | (a_character > 'z')) && (a_character != dollarchar))
{ cout << "Alphanumeric characters only, please try again > ";
cin >> a_character;
}
}
//-------------------------------------------------------------------------------------------------------------
void get_original_chars (int& length)
{ char next_char;
length = 0;
get_char (next_char);
while ((length < MAXCHARS) && (next_char != dollarchar))
{
OChars [length++] = next_char;
get_char (next_char);
}
}
//---------------------------------------------------------------------------------------------------------------
//----------------- ENCRYPTION ROUTINES -------------------------------------------------------------------------
void encrypt_chars (int length, char EKey)
{ char temp_char; // char temporary store
for (int i = 0; i < length; i++) // encrypt characters one at a time
{
temp_char = OChars [i]; //
__asm { //
push eax // save register values on stack to be safe
push ecx // Push last parameter first
lea eax,EKey
push temp_char
push eax
call encrypt3 // encrypt the character
mov temp_char, al
add esp, 8 // Clean parameters from stack
pop ecx // restore original register values from stack
pop eax //
}
EChars [i] = temp_char; // Store encrypted char in the encrypted chars array
}
return;
__asm {
encrypt3:
push ebp // Save the old base pointer value
mov ebp, esp // Set the new base pointer value
push edx // Save EDX to the first unused empty stack
push ecx //ecx register containing the temp_char is pushed to the stack
push eax // Save EAX to the first unused empty stack
mov edx, [ebp + 8] // Accessing the last value of ebp
movzx eax, byte ptr[eax] // Move 4 bytes to the EAX register
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
mov edx, eax // Move 4 bytes from EAX into edx
pop eax // Restore original EAX
mov byte ptr[eax], dl //moves the Ekey value into the EAX register as an 8-bit value
pop ecx //stores the current letter being encrypted within the ECX register (it was pushed to the stack earlier in the assembly code).
xor ecx, edx //clears the EDX register of all values
mov eax, ecx // Move 4 bytes from ECX into EAX
ror al, 1 // Rotate AL register one position to the left
ror al, 1 // Rotate AL register one position to the left
ror al, 1 // Rotate AL register one position to the left
pop edx // Restore the value of EDX
pop ebx // Restore original EBX
mov esp, ebp // Dellocate local variables
pop ebp // Restore the original value of EBP
ret // Return EAX value
}
//--- End of Assembly code
}
// end of encrypt_chars function
//---------------------------------------------------------------------------------------------------------------
//---------------------------------------------------------------------------------------------------------------
//----------------- DECRYPTION ROUTINES -------------------------------------------------------------------------
//
void decrypt_chars(int length, char EKey)
{
return;
}
// end of decrypt_chars function
//---------------------------------------------------------------------------------------------------------------

Answer 1

Definitely, write your code in high language before placing into assembly language.
Here are some reasons:
Clobbering edx register
mov edx, [ebp + 8] // Accessing the last value of ebp
movzx eax, byte ptr[eax] // Move 4 bytes to the EAX register
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
mov edx, eax // Move 4 bytes from EAX into edx
In the code above, you move [ebp + 8] into edx. Then you copy eax into edx four instructions later. Why bother with the first instruction here?
Duplicate instructions
One of the common reasons for coding in assembly language is efficiency. Supposedly you can code better than the compiler or use special instructions better than the compiler. You don't, as this example shows:
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
rol al, 1 // Rotate AL register one position to the left
The above should be coded as rol al, 3.
Also, is there a reason you are using al register instead of eax?
Clearing edx register is wrong
The operation does not match the comments.
xor ecx, edx //clears the EDX register of all values
The edx register is on the wrong side.
The statement xor edx, edx actually clears the edx register.
Restart with high level language.
I suggest trashing your assembly language and rewriting the functions in a high level language. Get it working first. Examine the compiler's assembly language. If you can code the algorithm more effectively than the compiler, then do so.