OpenSSL RSA Encryption returning gibberish. C++ - c++

Question

I am new to socket programming, so be kind :)
I am writing a client-server application in C++ and using OpenSSL. Till now I have generated the public-private keys for the client and server and have exchanged it over the network. Now is the part where I want to encrypt my client's message using the server's public key. But my public_encrypt function returns gibberish. I know the methods which I am using are deprecated and there are better methods but the purpose is to get the hands dirty only.
Below is the function that invokes the encryption API. (Ignore the if part, it's for sending the clients public key)
#define RSA_SIZE 256
void sendMessage(int clientFD, uint16_t type, char *data, serverState *server){
uint16_t length = strlen(data);
unsigned char message[MESSAGE_SIZE];
if (server->state == 0)
{
memcpy(message, (char *)&length, sizeof(length));
memcpy(message + 2, (char *)&type, sizeof(type));
memcpy(message + 4, data, length);
send(clientFD, message, 4 + length, 0);
server->state = 1;
}
else
{
unsigned char encrypted[RSA_SIZE] = {0};
length = public_encrypt(reinterpret_cast<unsigned char *>(data), length, server->key, encrypted);
assert(length != -1);
printf("%s\n", encrypted);
memcpy(message, (char *)&length, sizeof(length));
memcpy(message + 2, (char *)&type, sizeof(type));
memcpy(message + 4, encrypted, length);
send(clientFD, message, 4 + length, 0);
}}
This is the code for the encryption
int padding = RSA_PKCS1_OAEP_PADDING;
RSA *createRSA(unsigned char *key, int pub){
RSA *rsa = NULL;
BIO *keybio;
keybio = BIO_new_mem_buf(key, -1);
if (keybio == NULL)
{
printf("Failed to create key BIO");
return 0;
}
if (pub)
{
rsa = PEM_read_bio_RSA_PUBKEY(keybio, &rsa, NULL, NULL);
}
else
{
rsa = PEM_read_bio_RSAPrivateKey(keybio, &rsa, NULL, NULL);
}
if (rsa == NULL)
{
printf("Failed to create RSA");
}
return rsa;}
int public_encrypt(unsigned char *data, int data_len, unsigned char *key, unsigned char *encrypted){
printf("Data:%s\n:", data);
printf("Data Length:%d\n:", data_len);
printf("Server's Key:\n%s\n:", key);
RSA *rsa = createRSA(key, 1);
int result = RSA_public_encrypt(data_len, data, encrypted, rsa, padding);
return result;}
Please check out the link https://i.stack.imgur.com/WJn7e.png to see my output.
PS: Sorry for such a long post.

Answer 1

The output of RSA is a random value between 0 and the modulus of the RSA private key, encoded as an unsigned big endian octet string (octet string is just another name for byte array, a char[] in C / C++). It contains bytes with any value, and it is therefore certainly not ASCII. If you want ASCII you have to base 64 encode the ciphertext.
However, quite often ciphertext is "stringified" for no good reason at all, so only do this if this is necessary within your protocol / system. Python strings are made somewhat readable for you by the Python runtime. I'm not sure if that's a good thing or not - it's certainly not a good idea to copy that string as it is only Python proprietary.
C is not as forgiving, if you treat the binary array as text you'll run into trouble, as it can contain any character, including control characters and the NUL character (00), which can play merry hell with functions such as strlen and many others that expect a textual string instead of an array of bytes (both are usually based on char in C/C++).