error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095) during RSA decyption - c++

Question

I'm trying to create a hybrid cryptography tool in C++ with Qt Gui.
(The data will be encrypted with AES 256-CBC, the AES Key RSA encrypted and saved then.)
But the RSA part of this tool doesn't work.
I wrote the sourcecode several times but I always get the same error on decrypt.
error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095)
I hope someone could help me get a working RSA encrypt + decrypt implementation.
You can see the sourcecode here or download a testing Qt Project from my dropbox..
Dropbox Download: https://db.tt/6HKsYRTa
Sourcecode 1. Implementation:
void MainWindow::rsaEncrypt()
{
EVP_PKEY *pk = NULL;
EVP_PKEY_CTX *ctx = NULL;
QByteArray encrypted = QByteArray();
//------------------------------------------------
//--- READ PUBLIC KEY ----------------------------
FILE *pkFile = fopen(ui->publicKeyPath->text().toStdString().c_str(), "r");
if(pkFile == NULL) throw NULL;
pk = PEM_read_PUBKEY(pkFile, NULL, NULL, NULL);
if(pk == NULL) throw NULL;
fclose(pkFile);
//------------------------------------------------
ctx = EVP_PKEY_CTX_new(pk, NULL);
//------------------------------------------------
//--- ENCRYPT DATA -------------------------------
int err;
err = EVP_PKEY_encrypt_init(ctx);
if(err <= 0) throw NULL;
err = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
if(err <= 0) throw NULL;
size_t outLen = 0;
err = EVP_PKEY_encrypt(
ctx,
NULL,
&outLen,
(uchar*) ui->plainTextEdit->document()->toPlainText().toStdString().c_str(),
ui->plainTextEdit->document()->toPlainText().size()
);
if(err <= 0) throw NULL;
encrypted.resize(outLen);
err = EVP_PKEY_encrypt(
ctx,
(uchar*) encrypted.data(),
&outLen,
(uchar*) ui->plainTextEdit->document()->toPlainText().toStdString().c_str(),
ui->plainTextEdit->document()->toPlainText().size()
);
//------------------------------------------------
EVP_PKEY_CTX_free(ctx);
EVP_PKEY_free(pk);
if(err > 0) ui->encryptedTextEdit->document()->setPlainText(QString(encrypted));
else {
QByteArray errStr = QByteArray();
errStr.resize(256);
ERR_load_ERR_strings();
ERR_error_string(err, errStr.data());
ui->encryptedTextEdit->document()->setPlainText( QString(errStr) );
}
}
void MainWindow::rsaDecrypt()
{
EVP_PKEY *pk = NULL;
EVP_PKEY_CTX *ctx = NULL;
QByteArray decrypted = QByteArray();
//------------------------------------------------
//--- READ PRIVATE KEY ---------------------------
FILE *pkFile = fopen(ui->privateKeyPath->text().toStdString().c_str(), "r");
if(pkFile == NULL) throw NULL;
pk = PEM_read_PrivateKey(pkFile, NULL, NULL, NULL);
if(pk == NULL) throw NULL;
fclose(pkFile);
//------------------------------------------------
ctx = EVP_PKEY_CTX_new(pk, NULL);
//------------------------------------------------
//--- DECRYPT DATA -------------------------------
int err;
err = EVP_PKEY_decrypt_init(ctx);
if(err <= 0) throw NULL;
err = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
if(err <= 0) throw NULL;
size_t outLen = 0;
err = EVP_PKEY_decrypt(
ctx,
NULL,
&outLen,
(uchar*) ui->encryptedTextEdit->document()->toPlainText().toStdString().c_str(),
ui->encryptedTextEdit->document()->toPlainText().size()
);
if(err <= 0) throw NULL;
decrypted.resize(outLen);
err = EVP_PKEY_decrypt(
ctx,
(uchar*) decrypted.data(),
&outLen,
(uchar*) ui->encryptedTextEdit->document()->toPlainText().toStdString().c_str(),
ui->encryptedTextEdit->document()->toPlainText().size()
);
//------------------------------------------------
EVP_PKEY_CTX_free(ctx);
EVP_PKEY_free(pk);
if(err > 0) ui->decryptedTextEdit->document()->setPlainText(QString(decrypted));
else {
QByteArray errStr = QByteArray();
errStr.resize(256);
ERR_load_ERR_strings();
ERR_error_string(err, errStr.data());
ui->decryptedTextEdit->document()->setPlainText( QString(errStr) );
}
}
Sourcecode 2. Implementation:
void MainWindow::rsaEncrypt()
{
RSA *rsa = createRSAFromFile(ui->publicKeyPath->text().toStdString().c_str(), 1);
QByteArray encrypted = QByteArray();
encrypted.resize(2048);
int err = RSA_public_encrypt(
ui->plainTextEdit->document()->toPlainText().size(),
(uchar*) ui->plainTextEdit->document()->toPlainText().toStdString().c_str(),
(uchar*) encrypted.data(),
rsa,
RSA_PADDING
);
RSA_free(rsa);
if(err > 0) ui->encryptedTextEdit->document()->setPlainText( QString(encrypted) );
else {
QByteArray errStr = QByteArray();
errStr.resize(256);
ERR_load_ERR_strings();
ERR_error_string(err, errStr.data());
ui->encryptedTextEdit->document()->setPlainText( QString(errStr) );
}
}
void MainWindow::rsaDecrypt()
{
RSA *rsa = createRSAFromFile(ui->privateKeyPath->text().toStdString().c_str(), 0);
QByteArray decrypted = QByteArray();
decrypted.resize(2048);
int err = RSA_private_decrypt(
ui->encryptedTextEdit->document()->toPlainText().size(),
(uchar*) ui->encryptedTextEdit->document()->toPlainText().toStdString().c_str(),
(uchar*) decrypted.data(),
rsa,
RSA_PADDING
);
RSA_free(rsa);
if(err > 0) ui->decryptedTextEdit->document()->setPlainText( QString(decrypted) );
else {
QByteArray errStr = QByteArray();
errStr.resize(256);
ERR_load_ERR_strings();
ERR_error_string(err, errStr.data());
ui->decryptedTextEdit->document()->setPlainText( QString(errStr) );
}
}
RSA *MainWindow::createRSAFromFile(const char *keyPath, int pub)
{
FILE *keyFile = fopen(keyPath, "rb");
if(keyFile==NULL)
{
return 0;
}
RSA *rsa = RSA_new();
if(pub)
{
rsa = PEM_read_RSA_PUBKEY(keyFile, &rsa, NULL, NULL);
}
else
{
rsa = PEM_read_RSAPrivateKey(keyFile, &rsa, NULL, NULL);
}
fclose(keyFile);
return rsa;
}
Includes and defines for both implementations:
#include "mainwindow.h"
#include "ui_mainwindow.h"
#include <QDebug>
#include <QByteArray>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#define RSA_PADDING RSA_PKCS1_OAEP_PADDING

Answer 1

Try to use this Code for creating a RSA-object. It definetly works. You should read the .pem file before and then call this function:
RSA *CryptClassRSA::createRSA(unsigned char *key, int isPublic){
RSA *rsa = NULL;
BIO *keybio;
keybio = BIO_new_mem_buf(key, -1);
if (keybio==NULL){
printf( "Failed to create key BIO");
return NULL;
}
if(isPublic){
rsa = PEM_read_bio_RSA_PUBKEY(keybio, &rsa,NULL, NULL);
}
else{
rsa = PEM_read_bio_RSAPrivateKey(keybio, &rsa,NULL, NULL);
}
if(rsa == NULL){
printf( "Failed to create RSA");
}
return rsa;
}

Answer 2

C++ RSA decrypt error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095)
And
int err;
...
err = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
if(err <= 0) throw NULL;
...
Its not clear to me where the error is happening or how you are getting the errstr output of error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095). I picked EVP_PKEY_CTX_set_rsa_padding as the snippet because I think the return value is -2 (and it has significance as explained below).
err is just a return code. To get the actual error, you need to call ERR_get_error. Maybe something like:
int rc;
unsigned long err;
...
rc = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
err = ERR_get_error();
if(rc <= 0)
{
// err is valid
}
You should also visit the EVP_PKEY_CTX_set_rsa_padding and ERR_get_error man pages.
Often, OpenSSL will return 0 for success, so I'm not sure about err <= 0 in some places. Also, because your error is 0xffff...4095 (and not 4096), I think you are getting the -2 return value discussed in the man page:
EVP_PKEY_CTX_ctrl() and its macros return a positive value for success and 0 or a negative value for failure. In particular a return value of -2 indicates the operation is not supported by the public key algorithm.
Also note... If you gather your error and print it in hex:
rc = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
err = ERR_get_error();
if(rc <= 0)
{
// err is valid
std::cerr << std::hex << err << std::endl;
}
Then you can use the openssl errstr 0xNNNN to print it.