SIMPLE OpenSSL RSA Encryption in C/C++ is causing me headaches - c++

Question

SOLVED: I was dumb. First argument of encrypt should have been key.size() and first argument of decrypt should have been RSA_size(myKey).
ORIGINAL QUESTION
Hey guys, I'm having some trouble figuring out how to do this.
Basically I just want a client and server to be able to send each other encrypted messages.
This is going to be incredibly insecure because I'm trying to figure this all out so I might as well start at the ground floor.
So far I've got all the keys working but encryption/decryption is giving me hell.
I'll start by saying I am using C++ but most of these functions require C strings so whatever I'm doing may be causing problems.
Note that on the client side I receive the following error in regards to decryption.
error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed
I don't really understand how padding works so I don't know how to fix it.
Anywho here are the relevant variables on each side followed by the code.
Client:
RSA *myKey; // Loaded with private key
// The below will hold the decrypted message
unsigned char* decrypted = (unsigned char*) malloc(RSA_size(myKey));
/* The below holds the encrypted string received over the network.
Originally held in a C-string but C strings never work for me and scare me
so I put it in a C++ string */
string encrypted;
// The reinterpret_cast line was to get rid of an error message.
// Maybe the cause of one of my problems?
if(RSA_private_decrypt(sizeof(encrypted.c_str()), reinterpret_cast<const unsigned char*>(encrypted.c_str()), decrypted, myKey, RSA_PKCS1_OAEP_PADDING)==-1)
{
cout << "Private decryption failed" << endl;
ERR_error_string(ERR_peek_last_error(), errBuf);
printf("Error: %s\n", errBuf);
free(decrypted);
exit(1);
}
Server:
RSA *pkey; // Holds the client's public key
string key; // Holds a session key I want to encrypt and send
//The below will hold the encrypted message
unsigned char *encrypted = (unsigned char*)malloc(RSA_size(pkey));
// The reinterpret_cast line was to get rid of an error message.
// Maybe the cause of one of my problems?
if(RSA_public_encrypt(sizeof(key.c_str()), reinterpret_cast<const unsigned char*>(key.c_str()), encrypted, pkey, RSA_PKCS1_OAEP_PADDING)==-1)
{
cout << "Public encryption failed" << endl;
ERR_error_string(ERR_peek_last_error(), errBuf);
printf("Error: %s\n", errBuf);
free(encrypted);
exit(1);
}
Let me once again state, in case I didn't before, that I know my code sucks but I'm just trying to establish a framework for understanding this.
I'm sorry if this offends you veteran coders.
Thanks in advance for any help you guys can provide!

Answer 1

Maybe not the only problem but: The first argument to RAS_xxxcrypt functions is the number of bytes of the buffers. sizeof(key.c_str()) does not yield the number of bytes in key, it yields the size of the type of key.c_str()'s result type, i.e. sizeof(const char*). You probably want to pass the number of chars in the string instead, which can be obtained with the size() member function.