Errors decrypting message using OpenSSL RSA functions in C++ - c++

Question

I've been struggling with this for a while now. When I run my program, sometimes I see these errors:
bad decrypt
140380701197976:error:0606506D:digital envelope
routines:EVP_DecryptFinal_ex:wrong final block length:evp_enc.c:518:
Sometimes, I see these errors:
RSA operation error
139986632922776:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:rsa_pk1.c:273:
139986632922776:error:04065072:rsa
routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:602:
Error reading password from BIO
Error getting password
And sometimes, I don't see any errors at all! I'm running on GalliumOS, a flavor of Ubuntu made for Chrome hardware.
What am I missing? I've looked all over, and I can't find anything that's very relevant. For your reference, I've attached my code below. I think I've narrowed down the error to one of the last two commands, but I'm not positive.
#include <cstdlib>
#include <string>
using namespace std;
int main()
{
string bob_keys, alice_plaintext, alice_encrypted, bob_decrypted;
// ----- USER INPUT -----
// TODO: switch to user input
bob_keys = "bob_keys.pem";
bob_decrypted = "bob_decrypted.txt";
alice_encrypted = "alice_encrypted.txt";
alice_plaintext = "alice_plaintext.txt";
// ----- CONFIDENTIALITY: MESSAGE ENCRYPTION -----
// generate session key
system("openssl rand -base64 64 -out key.bin");
// encrypt message using session key
system(("openssl enc -aes-128-cbc -salt -in " + alice_plaintext
+ " -out alice_plaintext.txt.enc -pass file:./key.bin").c_str());
// encrypt session key using Bob's public key
system(("openssl rsautl -encrypt -inkey " + bob_keys
+ " -pubin -in key.bin -out key.bin.enc").c_str());
// write encrypted message and encrypted session key to file
system(("cat alice_plaintext.txt.enc > " + alice_encrypted).c_str());
system(("echo >> " + alice_encrypted).c_str());
system(("cat key.bin.enc >> " + alice_encrypted).c_str());
// ----- CONFIDENTIALITY: MESSAGE DECRYPTION -----
// get encrypted message and encrypted session key from file (and remove newlines)
system(("head -1 " + alice_encrypted + " > message.bin.enc").c_str());
system("tr -d '\n' < message.bin.enc > temp.bin && mv temp.bin message.bin.enc");
system(("tail -1 " + alice_encrypted + " > key.bin.enc").c_str());
// decrypt the key using Bob's private key
system(("openssl rsautl -decrypt -inkey " + bob_keys
+ " -in key.bin.enc -out key.bin").c_str());
// decrypt the message using the decrypted key
system(("openssl enc -d -aes-128-cbc -in message.bin.enc -out "
+ bob_decrypted + " -pass file:./key.bin").c_str());
return 0;
}

Answer 1

You are using ˋheadˋ and ˋtailˋ on binary files. Before that you'e managed to escape the many traps where you could have interpreted binary as text, but here it finally fails. It is perfectly possible that your ciphertext contains newline characters, and if those are found then your ciphertext will be incorrect. You will either have to use just binary or you should base 64 encode your ciphertext.
Note that it is great that you manage to circumvent the use of C/C++ the way that you do, but the last laugh is on you. You are using passwords instead of keys and the result will not be identical to just using an AES key. Be smart and use the C library instead. Or, if you like C++ more, use Crypto++ or Bothan. And learn how to actually program instead of hacking your way out of evertything. Because you - in the end - will be the person who suffers the most from your own hacking.