How to decrypt using openssl EVP? - c++

Question

I'm trying to decrypt a file using C (but I can change to C++), but I don't know how to use the EVP library correctly.
The console command that I want to replicate is:
openssl enc -rc2-ecb -d -in myfile.bin -iter 1 -md sha1 -pbkdf2 -pass pass:'Yumi'
My actual C code using EVP is:
unsigned char salt[8] = "Salted__";
ctx=EVP_CIPHER_CTX_new();
PKCS5_PBKDF2_HMAC_SHA1("Yumi", -1,
salt, 8, 1,
EVP_MAX_KEY_LENGTH, key);
EVP_DecryptInit(ctx, EVP_rc2_ecb(), key, iv);
pt = (unsigned char *)malloc(sz + EVP_CIPHER_CTX_block_size(ctx) + 1);
EVP_DecryptUpdate(ctx, pt, &ptlen, ciphertext, sz);
if (!EVP_DecryptFinal(ctx,&pt[ptlen],&tmplen)) {
printf("Error decrypting on padding \n");
} else {
printf("Succesful decryption\n");
}
I suppose that's not working because I have to declare something to use SHA1 instead of SHA256 (default on rc2-ecb), but I'm not seeing how to do it.
Any help will be appreciated

Answer 1

The following hex encoded ciphertext could be decrypted with the posted OpenSSL statement, if this data would be contained hex decoded in myfile.bin:
53616C7465645F5F2DC4C4867D3B9268C82E23A672D6698FB51D41EA8601367A9112623EC27CDEB18FD1444BDB8D8DE16F1A35706EC7FED266CB909D28BF6BEC
The decryption would result in:
The quick brown fox jumps over the lazy dog
For simplicity, the ciphertext is assigned directly and the loading of the ciphertext from a file is skipped:
unsigned char data[] = {
0x53, 0x61, 0x6C, 0x74, 0x65, 0x64, 0x5F, 0x5F, // Salted__
0x2D, 0xC4, 0xC4, 0x86, 0x7D, 0x3B, 0x92, 0x68, // Salt
0xC8, 0x2E, 0x23, 0xA6, 0x72, 0xD6, 0x69, 0x8F, 0xB5, 0x1D, 0x41, 0xEA, 0x86, 0x01, 0x36, 0x7A, // Ciphertext...
0x91, 0x12, 0x62, 0x3E, 0xC2, 0x7C, 0xDE, 0xB1, 0x8F, 0xD1, 0x44, 0x4B, 0xDB, 0x8D, 0x8D, 0xE1,
0x6F, 0x1A, 0x35, 0x70, 0x6E, 0xC7, 0xFE, 0xD2, 0x66, 0xCB, 0x90, 0x9D, 0x28, 0xBF, 0x6B, 0xEC };
int dataLength = sizeof(data) / sizeof(unsigned char);
For decryption, salt and ciphertext must first be separated, e.g.:
int ciphertextLength = dataLength - 16;
unsigned char* salt = (unsigned char*)malloc(sizeof(unsigned char) * 8);
unsigned char* ciphertext = (unsigned char*)malloc(sizeof(unsigned char) * ciphertextLength);
memcpy(salt, data + 8, 8); // Get 8 bytes salt (starts at index 8)
memcpy(ciphertext, data + 16, ciphertextLength); // Get ciphertext (starts at index 16)
The next step is to derive the key, see PKCS5_PBKDF2_HMAC, e.g.:
#define KEYSIZE 16
unsigned char key[KEYSIZE];
const char* password = "'Yumi'"; // The quotation marks (') in the openssl-statement are part of the password.
int passwordLen = strlen(password);
PKCS5_PBKDF2_HMAC(password, passwordLen, salt, 8, 1, EVP_sha1(), KEYSIZE, key);
Finally the decryption can be performed, see here and here, e.g.:
EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
EVP_DecryptInit_ex(ctx, EVP_rc2_ecb(), NULL, NULL, NULL);
EVP_CIPHER_CTX_set_key_length(ctx, KEYSIZE); // RC2 is an algorithm with variable key size. Therefore the key size must generally be set.
EVP_DecryptInit_ex(ctx, NULL, NULL, key, NULL);
unsigned char* plaintext = (unsigned char*)malloc(sizeof(unsigned char) * ciphertextLength);
int length;
EVP_DecryptUpdate(ctx, plaintext, &length, ciphertext, ciphertextLength);
int plaintextLength = length;
EVP_DecryptFinal_ex(ctx, plaintext + plaintextLength, &length);
plaintextLength += length;
printf("Plaintext: "); for (int i = 0; i < plaintextLength; i++) { printf("%c", plaintext[i]); } printf("\n");
For simplicity, the code doesn't include exception handling and memory release.
Note the following:
Some of the parameters used in the posted OpenSSL statement are insecure, e.g. the ECB mode and an iteration count of 1. Instead, a mode with an IV should be used and an iteration count of 10,000 or larger if performance allows.
OpenSSL generates a random 8 bytes salt during encryption, which is used to derive the key. The ciphertext consists of the ASCII encoding of Salted__, followed by the 8 bytes salt, followed by the actual ciphertext, here. Just this format is expected by the posted OpenSSL statement for decryption.
RC2 defines a variable key size. The posted OpenSSL statement uses a size of 16 bytes.
The -pass pass: option specifies the password without quotation marks, i.e. in the posted OpenSSL statement the quotation marks are part of the password.