OpenSSL RSA_private_decrypt() fails with "oaep decoding error" - c++

Question

I'm trying to implement RSA encryption/decryption using OpenSSL. Unfortunately my code fails during decryption.
I'm using Qt. So here is my code:
QByteArray CryptRSA::rsaEncrypt(QByteArray input)
{
QByteArray result(RSA_size(rsaKey), '\0');
int encryptedBytes = RSA_public_encrypt(RSA_size(rsaKey) - 42, (unsigned char *)input.data(), (unsigned char *) result.data(), rsaKey, RSA_PKCS1_OAEP_PADDING);
if (encryptedBytes== -1)
{
qDebug() << "Error encrypting RSA Key:";
handleErrors();
return QByteArray();
}
else
{
return result;
}
}
QByteArray CryptRSA::rsaDecrypt(QByteArray input)
{
QByteArray result(RSA_size(rsaKey), '\0');
int decryptedBytes = RSA_private_decrypt(RSA_size(rsaKey) - 42, (unsigned char *)input.data(), (unsigned char *)result.data(), rsaKey, RSA_PKCS1_OAEP_PADDING);
if (decryptedBytes == -1)
{
qDebug() << "Error decrypting RSA Key.";
handleErrors();
return QByteArray();
}
else
{
result.resize(decryptedBytes);
return result;
}
}
Here is the error:
error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error
error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed
It fails in:
RSA_private_decrypt(RSA_size(rsaKey) - 42, (unsigned char *)input.data(),
(unsigned char *)result.data(), rsaKey, RSA_PKCS1_OAEP_PADDING);
I have tried several things, but i can't find my mistakes.

Answer 1

error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error
error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed
If RSA_public_encrypt succeeds, then set the size of the result array to encryptedBytes. Do similar for RSA_private_decrypt.
Also, its not clear what you are trying to do with RSA_size(rsaKey) - 42. That looks very odd to me. I would expect it to be input.size(). But I'm guessing you know what you are doing with you array.
There could be other problems (like the public and private keys don't match), but we'd need to see more code and parameters to tell.
Also, you should use the EVP_* interfaces. See EVP Asymmetric Encryption and Decryption of an Envelope on the OpenSSL wiki.