Bad Data error at CryptDecrypt when using AES 256 (MS CryptoAPI) - c++

Question

I'm trying to decrypt - using the microsoft's CryptoAPI in C++ - a short message encrypted using mcrypt_encrypt in PHP. The php line is:
mcrypt_encrypt( MCRYPT_RIJNDAEL_256, $key, $msg, MCRYPT_MODE_CBC);
where $key and $msg are strings.
In C++ I have the key, and my decryption function looks like this:
bool decrypt( const unsigned char* input_buffer,
const size_t& input_size,
const unsigned char* key,
const size_t& key_size,
unsigned char* output_buffer,
DWORD* out_size)
{
Log(L"START init_crypto");
bool ret = false;
HCRYPTKEY hKey = NULL;
HCRYPTPROV hCryptProv = NULL;
DWORD dwDataLen = 0;
// Attempt to acquire a handle to the crypto provider for AES
if(0 == CryptAcquireContext(&hCryptProv, NULL, NULL, PROV_RSA_AES, CRYPT_VERIFYCONTEXT) ){//PROV_RSA_AES
Log(L"CryptAcquireContext failed with code %ld", GetLastError());
goto end;
}
// key creation based on
// http://mirror.leaseweb.com/NetBSD/NetBSD-release-5-0/src/dist/wpa/src/crypto/crypto_cryptoapi.c
struct {
BLOBHEADER hdr;
DWORD len;
BYTE key[32];
} key_blob;
key_blob.hdr.bType = PLAINTEXTKEYBLOB;
key_blob.hdr.bVersion = CUR_BLOB_VERSION;
key_blob.hdr.reserved = 0;
key_blob.hdr.aiKeyAlg = CALG_AES_256;
key_blob.len = 32;//key_size;
memset(key_blob.key, '\0', sizeof(key_blob.key));
assert(key_size <= sizeof(key_blob.key));
CopyMemory(key_blob.key, key, min(key_size, sizeof(key_blob.key)));
if (!CryptImportKey( hCryptProv,
(BYTE *)&key_blob,
sizeof(key_blob),
0,
CRYPT_EXPORTABLE,
&hKey)){
Log(L"Error in CryptImportKey 0x%08x \n", GetLastError());
goto free_context;
}
else{
//---------------------------
// Set Mode
DWORD dwMode = CRYPT_MODE_CBC;
if(!CryptSetKeyParam( hKey, KP_MODE, (BYTE*)&dwMode, 0 )){
// Handle error
Log(L"Cannot set the cbc mode for decryption 0x%08x \n", GetLastError());
goto free_key;
}
//----------------------------
// Set IV
DWORD dwBlockLen = 0;
DWORD dwDataLen = sizeof(dwBlockLen);
if (!CryptGetKeyParam(hKey, KP_BLOCKLEN, (BYTE *)&dwBlockLen, &dwDataLen, 0)){
// Handle error
Log(_USTR("Cannot get the block length 0x%08x \n"), GetLastError());
goto free_key;
}
dwBlockLen /= 8;
BYTE *pbTemp = NULL;
if (!(pbTemp = (BYTE *)LocalAlloc(LMEM_FIXED, dwBlockLen))){
// Handle error
Log(L"Cannot allcoate the IV block 0x%08x \n", GetLastError());
goto free_key;
}
memset(pbTemp, '\0', dwBlockLen);
if (!CryptSetKeyParam(hKey, KP_IV, pbTemp, 0)){
// Handle error
Log(L"Cannot set the IV block 0x%08x \n", GetLastError());
LocalFree(pbTemp);
goto free_key;
}
LocalFree(pbTemp);
}
CopyMemory(output_buffer, input_buffer, min(*out_size, input_size));
*out_size = input_size;
if (!CryptDecrypt(hKey, NULL, TRUE, 0, output_buffer, out_size)){
Log(L"CryptDecrypt failed with code %ld", GetLastError());
goto free_key;
}
else{
Log(L"Decryption...");
ret = true;
}
free_key:
if (hKey)
CryptDestroyKey( hKey );
free_context:
if (hCryptProv)
CryptReleaseContext(hCryptProv, 0);
end:
return ret;
}
I consistently get the "bad data" error at CryptDecrypt()... I may be missing something obvious - if so, please help.
EDIT - To isolate the cause of the problem I replaced the two lines before CryptDecrypt (the CopyMemory stuff) with the following code:
....
strcpy((char*)output_buffer, "stuff");
DWORD plain_txt_len = strlen((char*)output_buffer);
if (!CryptEncrypt(hKey, NULL, TRUE, 0, output_buffer, &plain_txt_len, *out_size)){
Log(L"CryptEncrypt failed with code 0x%08x", GetLastError());
goto free_key;
}
...
and the CryptDecrypt is working -- which makes me believe that the problem is the key/and or message transmission from php to C++ ... If you agree can you give me a hint on how to make sure that the strings I use in PHP are the same with the ones in C++ (at byte level?)
EDIT2 --
After I changed my strings in binary streams (using pack) in php, and after I implemented the workaround(?) for AES vs Rijndael from here : http://kix.in/2008/07/22/aes-256-using-php-mcrypt/ I finaly have CryptDecrypt decrypting my PHP message... the problem is that it also still fails - even if the output contains the decrypted text. Any ideas about why could this happen?

Answer 1

Try passing NULL instead of CRYPT_VERIFYCONTEXT when acquiring the context.

Answer 2

With block encryption algorithms such as AES you need to add padding to the data being encrypted up to a multiple of the block length. Using your code example that already retrieves the block size you can calculate the padding required for encryption:
dwPadding = dwBlockLen - dwDataLen % dwBlockLen
Then append "dwPadding" number of characters (NULL works fine) to the data being encrypted and you will no longer get "Bad Data" errors in decryption.
You can also find out the size of the required encryption buffer directly by making an additional call to "CryptEncrypt" with a NULL buffer before the actual encryption:
dwBuffSize = dwDataLen
CryptEncrypt(hKey, NULL, TRUE, 0, NULL, &dwBuffSize, 0)
dwPadding = dwBuffsize - dwDataLen
Both methods are equivalent and produce the same desired result. The "CryptDecrypt" function already knows about padding and will return the actual size of the decryption buffer in one go so you will know exactly where your decrypted data ends.