OpenSSL Decryption - EVP_DecryptFinal_ex fails - c++

Question

I'm using this decryption function to get the plain text value of a cipher which was encrypted using EVP AES 265 GCM; I can see data in rawOut but ret = EVP_DecryptFinal_ex(ctx, rawOut, &len); returns 0; can you provide any insight as to why? I've also seen sources which do rawOut + len in the EVP_DecryptFinal_ex code, I'm not sure why this would be needed as it would move the pointer to the end of the buffer.
unsigned char* keyDecrypter(unsigned char* pszMasterKey)
{
ERR_load_crypto_strings();
int ret, len;
EVP_CIPHER_CTX* ctx;
unsigned char* rawOut = new unsigned char[48]; // ToDo Remove Hardcoded Value
Info info = m_header.processKeyInfo();
if (NULL == info.nonce)
return NULL;
if (!(ctx = EVP_CIPHER_CTX_new()))
return NULL;
if (!EVP_DecryptInit_ex(ctx, EVP_aes_256_gcm(), NULL, pszMasterKey, info.nonce))
return NULL;
if (!EVP_DecryptUpdate(ctx, NULL, &len, m_header.aad, m_header.aad_len))
return NULL;
if (!EVP_DecryptUpdate(ctx, rawOut, &len, m_header.encryptedValue, m_header.encryptedValueLen))
return NULL;
// Finalise the decryption. A positive return value indicates success,
// anything else is a failure - the plain text is not trustworthy.
ret = EVP_DecryptFinal_ex(ctx, rawOut, &len);
ERR_print_errors_fp(stderr);
EVP_CIPHER_CTX_free(ctx);
if (ret > 0)
{
return rawOut;
}
else
{
return NULL;
}
}

Answer 1

From the OpenSSL doc:
"EVP_DecryptFinal() will return an error code if padding is enabled and the final block is not correctly formatted."
Apparently, the padding scheme between encryption and decryption do not match, or perhaps the size of ciphertext fed into the decryption engine did not exactly match the size of the ciphertext that was output from the encryption engine. Note that the ciphertext must include the result of a corresponding call to EVP_EncryptFinal_ex.
It is unfortunate that the original poster did not provide sufficient information to make an exact determination.

Answer 2

You need to pass rawOut + len to EVP_DecryptFinal_ex. See in the example at the end of the documentation:
/* Buffer passed to EVP_EncryptFinal() must be after data just
* encrypted to avoid overwriting it.
*/
if(!EVP_EncryptFinal_ex(ctx, outbuf + outlen, &tmplen))
{
/* Error */
return 0;
}
outlen += tmplen;
Also note that rawOut must have enough room for (m_header.aad_len + cipher_block_size) bytes. You can get the block size with EVP_CIPHER_block_size().