Why does my OpenSSL C++ code create binary encryption output? - c++

Question

I'm trying to encrypt a file using AES from OpenSSL and then write the output to a file. But I'm getting messy outputs, sometimes decipherable and sometimes not.
The main code is based from here: https://github.com/shanet/Crypto-Example/blob/master/crypto-example.cpp
Here's the code:
int Crypt::__aesEncrypt(const unsigned char *msg, size_t msgLen, unsigned char **encMsg) {
EVP_CIPHER_CTX *aesEncryptCtx = (EVP_CIPHER_CTX*)malloc(sizeof(EVP_CIPHER_CTX));
EVP_CIPHER_CTX_init(aesEncryptCtx);
unsigned char *aesKey = (unsigned char*)malloc(AES_KEYLEN/8);
unsigned char *aesIV = (unsigned char*)malloc(AES_KEYLEN/8);
unsigned char *aesPass = (unsigned char*)malloc(AES_KEYLEN/8);
unsigned char *aesSalt = (unsigned char*)malloc(8);
if(RAND_bytes(aesPass, AES_KEYLEN/8) == 0) {
return FAILURE;
}
if(RAND_bytes(aesSalt, 8) == 0) {
return FAILURE;
}
if(EVP_BytesToKey(EVP_aes_256_cbc(), EVP_sha1(), aesSalt, aesPass, AES_KEYLEN/8, AES_ROUNDS, aesKey, aesIV) == 0) {
return FAILURE;
}
strncpy((char*)aesKey, (const char*)"B374A26A71490437AA024E4FADD5B4AA", AES_KEYLEN/8);
strncpy((char*)aesIV, (const char*)"7E892875A52C59A3B588306B13C31FBD", AES_KEYLEN/16);
size_t blockLen = 0;
size_t encMsgLen = 0;
*encMsg = (unsigned char*)malloc(msgLen + AES_BLOCK_SIZE);
if(encMsg == NULL) return FAILURE;
if(!EVP_EncryptInit_ex(aesEncryptCtx, EVP_aes_256_cbc(), NULL, aesKey, aesIV)) {
return FAILURE;
}
if(!EVP_EncryptUpdate(aesEncryptCtx, *encMsg, (int*)&blockLen, (unsigned char*)msg, msgLen)) {
return FAILURE;
}
encMsgLen += blockLen;
if(!EVP_EncryptFinal_ex(aesEncryptCtx, *encMsg + encMsgLen, (int*)&blockLen)) {
return FAILURE;
}
EVP_CIPHER_CTX_cleanup(aesEncryptCtx);
free(aesEncryptCtx);
free(aesKey);
free(aesIV);
return encMsgLen + blockLen;
}
Im calling like this:
unsigned char *encMsg = NULL;
__aesEncrypt((const unsigned char*)decrypted_string.c_str(), decrypted_string.size(), &encMsg);
std::stringstream ss;
ss << encMsg;
//write ss to file...
Thanks.

Answer 1

I'm actually the author of the example you've based your code off of. As WhozCraig pointed out in the comments above, you are using a stringstream to write the encrypted message to a file. The problem with this is that encrypted messages are not regular ASCII strings. They are binary data (values greater than 127, hence the need for an unsigned char array) and binary data cannot be treated the same as ASCII strings.
I'm not much of a C++ person, so I would write the data to a file the C way with fwrite, but if you want to do it the C++ way, I think you're looking for ifstream rather than stringstream.
Side note, I'm betting this is just for debugging, but I'll point it out anyway just to make sure: Hardcoding your AES key and IV (strncpy((char*)aesKey, (const char*)"B374A26A71490437AA024E4FADD5B4AA", AES_KEYLEN/8)) completely defeats the purpose of encryption. If you want to avoid the PBKDF (EVP_BytesToKey) you can just use RAND_Bytes to get random data for your AES key.