Strange behavior from decrypt EVP envelope_open() - c++

I have a simple program that encrypts files in a directory. I can iterate through and everything works perfectly. This is using pub/priv key pair. When decrypting one file at a time, it works as it should. However, if there are multiple files in a directory, or even if I put the filenames in a vector and fopen them for reading/writing respectively, it ONLY decrypts the LAST file in the vector/directory. How is this even possible? It fails on everyone of them on OpenFinal(). Here is the function and heart of the program. Everything else is solid. As stated, it works as a standalone program if I just decrypt one file manually or if there is just ONE file in the directory or vector.
Any help would be appreciated. This makes no sense at all. It seems like an implementation issue on their end.
void handleErrors(void)
{
// perror("Error: ");
ERR_print_errors_fp(stderr);
abort();
}
int envelope_open(EVP_PKEY *priv_key, unsigned char *ciphertext,
int ciphertext_len, unsigned char *encrypted_key,
int encrypted_key_len, unsigned char *iv,
unsigned char **plaintext, int *plaintext_len)
{
EVP_CIPHER_CTX *ctx;
int len = 0, ret = 0;
unsigned char *tmpptxt = NULL;
if((ctx = EVP_CIPHER_CTX_new()) == NULL)
return 0;
if ((tmpptxt = (unsigned char*)malloc(ciphertext_len)) == NULL)
{
printf("tmptxt error!\n");
handleErrors();
}
if(EVP_OpenInit(ctx, EVP_aes_256_cbc(), encrypted_key, encrypted_key_len,
iv, priv_key) != 1)
{
printf("OpenInit error\n");
handleErrors();
}
if(EVP_OpenUpdate(ctx, tmpptxt, &len, ciphertext, ciphertext_len) != 1)
{
printf("OpenUpdate error\n");
handleErrors();
}
*plaintext_len = len;
if(EVP_OpenFinal(ctx, tmpptxt + len, &len) != 1)
{
printf("OpenFinal error\n");
handleErrors();
}
*plaintext_len += len;
*plaintext = tmpptxt;
tmpptxt = NULL;
ret = 1;
err:
EVP_CIPHER_CTX_free(ctx);
free(tmpptxt);
return ret;
}

I was overwriting the key and iv returned from envelope_seal(). I looked over the fact it was unique. Each file must have this key and iv along with the private key and passphrase on the key to be able to decrypt a file. So this is definitely secure ... until someone cracks AES 256 of course.

Related

Unexpected invalid padding error with RSA_private_encrypt() and RSA_public_decrypt()

I'm trying to encrypt with private key and decrypt with public key, with RSA_PKCS1_PADDING as padding. The encryption works fine, but when I do the decryption I got an invalid padding error:
processed 9 of 256 bytes, RSA_public_decrypt() error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
Anyone know what's wrong here? Attached full source code:
void encrypt_stdout(const char *from_file, int to_base64, int padding)
{
unsigned char *input = NULL, *output = NULL, *output2 = NULL;
int output_fd = -1, input_len = 0, output_len = 0, output2_len = 0;
if (readfile(from_file, &input, &input_len) != 0)
{
goto end;
}
output = malloc(RSA_size(rsa));
if (output == NULL)
{
fprintf(stderr, "malloc() on output: %s\n", strerror(errno));
goto end;
}
output_len = RSA_private_encrypt(input_len, input, output, rsa, padding);
if (output_len == -1)
{
fprintf(stderr, "RSA_private_encrypt() %s\n",ERR_error_string(ERR_get_error(), errbuf));
}
else
{
if (to_base64)
{
}
write(1, output, output_len);
}
end:
if (input != NULL)
{
free(input);
}
if (output != NULL)
{
free(output);
}
}
void decrypt_stdout(const char *from_file, int is_base64, int skip_bytes, int padding)
{
unsigned char *input = NULL, *output = NULL, *output2 = NULL;
int output_fd = -1, input_len = 0, output_len = 0, output2_len = 0, total_read = skip_bytes;
if (readfile(from_file, &input, &input_len) != 0)
{
goto end;
}
if (is_base64)
{
if (base64_decode(input, input_len, &output2, &output2_len) != 0)
{
if (output2 != NULL)
free(output2);
goto end;
}
free(input);
input = output2;
input_len = output2_len;
}
output = malloc(RSA_size(rsa));
if (output == NULL)
{
fprintf(stderr, "malloc() on output: %s\n", strerror(errno));
goto end;
}
while (total_read < input_len)
{
memset(output, 0, RSA_size(rsa));
output_len = RSA_public_decrypt(RSA_size(rsa), input + total_read, output, rsa, padding);
if (output_len == -1)
{
fprintf(stderr, "\nprocessed %d of %d bytes, RSA_public_decrypt() %s\n", total_read, input_len, ERR_error_string(ERR_get_error(), errbuf));
break;
}
else
{
write(STDOUT_FILENO, output, output_len);
}
total_read += output_len;
}
end:
if (input != NULL)
{
free(input);
}
if (output != NULL)
{
free(output);
}
}
The problem is caused by the while loop in decrypt_stdout().
To me the sense of this loop is not clear. Instead of the loop it should be quite analogous to encrypt_stdout():
output_len = RSA_public_decrypt(input_len, input, output, rsa, padding);
if (output_len == -1)
{
fprintf(stderr, "RSA_public_decrypt() %s\n", ERR_error_string(ERR_get_error(), errbuf));
}
else
{
_write(1, output, output_len);
}
With this, decryption works on my machine.
However, if the loop is used, the first decryption succeeds and returns the plaintext, while the subsequent decryption fails and returns a -1, resulting in the output of the error message.
A few notes on the logic used in the code:
Since you talk about encryption and decryption: RSA_private_encrypt() and RSA_public_decrypt() are not really meant for encryption and decryption, but for low level signing and verification.
Typically, when signing with RSA_private_encrypt(), it is not the data itself that is passed, but a hash of the data prefixed with a digest ID (more precisely, the DER encoding of the DigestInfo value). In the posted code, there is neither hashing nor prepending a digest ID. Of course, the loaded file might already contain the concatenation of digest ID and hash, but that cannot be said without sample data. If this is not the case, the generated signature is not compliant with PKCS#1 v1.5 padding.
If no hashing takes place, care must be taken that the length criterion is met, i.e. the data to be signed must be smaller than the key size minus the minimum space required by the padding (11 bytes for PKCS#1 v1.5 padding or flag_padding = 1).
For completeness: PKCS#1 v1.5 padding is determinstic (in the context of signing). The padded data looks like this: 0x00 || 0x01 || PS || 0x00 || T, where PS consists of so many 0xff values that the size is equal to the key size. T is the data or, if compliant with the specification, the concatenation of digest ID and hashed data, s. RFC8017.

OpenSSL EVP_DecryptFinal_ex returns "wrong final block length" error when decrypting a file

I am using the EVP Symmetric Encryption and Decryption algorithm to encrypt and decrypt a file of text.
The encryption works file, a new encrypted file is generated, but when I try to decrypt the file back it always crashes when EVP_DecryptFinal_ex is called the first time.
I am using two Visual Studio projects: one for encryption and one for decryption.
The libraries I am using I presumed they are build in DEBUG mode (because they have the .pdb files), so that is how my project is also build. (if I choose release mode, the compiler cannot find the openssl include header anymore).
This is the error I get:
digital envelope routines:EVP_DecryptFinal_ex:wrong final block length
I am using the C++11 version, here is my code:
void Cipher::Encrypt(const byte key[KEY_SIZE], const byte iv[BLOCK_SIZE], const secure_string& ptext, secure_string& ctext) {
EVP_CIPHER_CTX_free_ptr ctx(EVP_CIPHER_CTX_new(), ::EVP_CIPHER_CTX_free);
int rc = EVP_EncryptInit_ex(ctx.get(), EVP_aes_256_cbc(), NULL, key, iv);
if (rc != 1)
throw std::runtime_error("EVP_EncryptInit_ex failed");
// Recovered text expands upto BLOCK_SIZE
ctext.resize(ptext.size() + BLOCK_SIZE);
int out_len1 = (int)ctext.size();
rc = EVP_EncryptUpdate(ctx.get(), (byte*)&ctext[0], &out_len1, (const byte*)&ptext[0], (int)ptext.size());
if (rc != 1)
throw std::runtime_error("EVP_EncryptUpdate failed");
int out_len2 = (int)ctext.size() - out_len1;
rc = EVP_EncryptFinal_ex(ctx.get(), (byte*)&ctext[0] + out_len1, &out_len2);
if (rc != 1)
throw std::runtime_error("EVP_EncryptFinal_ex failed");
// Set cipher text size now that we know it
ctext.resize(out_len1 + out_len2);
}
void Cipher::Decrypt(const byte key[KEY_SIZE], const byte iv[BLOCK_SIZE], const secure_string& ctext, secure_string& rtext) {
EVP_CIPHER_CTX_free_ptr ctx(EVP_CIPHER_CTX_new(), ::EVP_CIPHER_CTX_free);
int rc = EVP_DecryptInit_ex(ctx.get(), EVP_aes_256_cbc(), NULL, key, iv);
if (rc != 1)
throw std::runtime_error("EVP_DecryptInit_ex failed");
// Recovered text contracts upto BLOCK_SIZEB
rtext.resize(ctext.size());
int out_len1 = (int)rtext.size();
rc = EVP_DecryptUpdate(ctx.get(), (byte*)&rtext[0], &out_len1, (const byte*)&ctext[0], (int)ctext.size());
if (rc != 1)
throw std::runtime_error("EVP_DecryptUpdate failed");
int out_len2 = (int)rtext.size() - out_len1;
rc = EVP_DecryptFinal_ex(ctx.get(), (byte*)&rtext[0] + out_len1, &out_len2);
if (rc != 1) {
ERR_print_errors_fp(stderr);
throw std::runtime_error("EVP_DecryptFinal_ex failed");
}
// Set recovered text size now that we know it
rtext.resize(out_len1 + out_len2);
}
int main(int argc, char* argv[])
{
// Load the necessary cipher
EVP_add_cipher(EVP_aes_256_cbc());
// create Cipher object
Cipher cipher;
ifstream f("d:/temp.YML");
ofstream out("d:/tempDecrypt.YML");
byte key[KEY_SIZE] = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2};
byte iv[BLOCK_SIZE] = {1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6};
//cipher.gen_params(key, iv);
secure_string line;
secure_string temp;
while (getline(f, line)) {
cipher.Decrypt(key, iv, line, temp);
std::cout << temp << std::endl;
out << temp;
}
OPENSSL_cleanse(key, KEY_SIZE);
OPENSSL_cleanse(iv, BLOCK_SIZE);
return 0;
}
I also read that it could be a padding issue, not sure if that is the case and what should I do. I am not that good with encryption.
Any pointers on how to proceed further would be welcomed. If you need more information let me know.

OpenSSL sha256 signing and verifying with CPP

My code should read a PKCS12 to get the private key to sign some files and check its signature.
Reading and parse the PKCS12 seems to work fine. the problem happend specially at verifying. Here follows the signing and verifying.
int sign (EVP_PKEY *pkey, char* msg,unsigned char** sig, size_t* slen){
EVP_MD_CTX *mdctx = NULL;
int ret = 0;
size_t req = 0;
*sig = NULL;
/* Create the Message Digest Context */
if(!(mdctx = EVP_MD_CTX_create())) {
qDebug()<<"0";
return 0;
}
/* Initialise the DigestSign operation - SHA-256 has been selected as the message digest function in this example */
if(1 != EVP_DigestSignInit(mdctx, NULL, EVP_sha256(), NULL,pkey)) {
qDebug()<<"A";
return 0;
}
/* Call update with the message */
if(1 != EVP_DigestSignUpdate(mdctx, msg, strlen(msg))) {
qDebug()<<"B";
return 0;
}
/* Finalise the DigestSign operation */
/* First call EVP_DigestSignFinal with a NULL sig parameter to obtain the length of the
* signature. Length is returned in slen */
if(1 != EVP_DigestSignFinal(mdctx, NULL, slen)) {
qDebug()<<"C";
return 0;
}
/* Allocate memory for the signature based on size in slen */
if(!(*sig =(unsigned char *) OPENSSL_malloc(sizeof(unsigned char) * (*slen)))){
qDebug()<<"D";
return 0;
}
/* Obtain the signature */
if(1 != EVP_DigestSignFinal(mdctx, *sig, slen)){
qDebug()<<"E";
return 0;
}
//qDebug()<<(char*)sig;
/* Success */
ret = 1;
/* Clean up */
if(*sig && !ret) OPENSSL_free(*sig);
if(mdctx) EVP_MD_CTX_destroy(mdctx);
qDebug()<<"SIGNED";
return 1;
}
/*------------------------------*/
int verify(EVP_PKEY *pubkey, const char* msg,unsigned char* sig, size_t slen){
EVP_MD_CTX *mdctx;
ERR_get_error();
if(!(mdctx = EVP_MD_CTX_create())) {
qDebug()<<"DEU RUIM";
}
EVP_MD_CTX_init(mdctx);
if(1 != EVP_DigestVerifyInit(mdctx, NULL, EVP_sha256(), NULL, pubkey)){
qDebug()<<"Err1";
return 0;
}
if(1 != EVP_DigestVerifyUpdate(mdctx, msg, strlen(msg))){
qDebug()<<"Err2";
return 0;
}
ERR_print_errors_fp(stderr);
if(1 == EVP_DigestVerifyFinal(mdctx, sig, slen)) {
qDebug()<<"THE FILE IS ORIGINAL";
return 1;
} else{
qDebug()<<"THE FILE IS NOT ORIGINAL";
return 0;
}
}
The idea is to write the certificate to a file and read it from the file when I want to verify. The problem is the it only verifies correctly(prints 'THE FILE IS ORIGINAL') when I call the verify() after sign() passing the 'sig' and 'slen' parameters altered in the sign() to the verify(), otherwise I just receive 'THE FILE IS NOT ORIGINAL' without any error.
Printed out the parameters used in verify() when it works and when it doesn't, but they look exactly the same the same.
Here is how I call the functions
/*---(START)IF COMMENT THIS PART, VERIFIES WRONGLY---*/
sign(prkey, msg, &sig, &len);
/*write file certificate*/
ofstream outfile("outCert",ios::out | ios::binary);
outfile.write((reinterpret_cast<const char*>(sig)),len);
/*---(END)---*/
//set public key
EVP_PKEY* pubkey=X509_get_pubkey(cert);
FILE* file2;
unsigned char *msg2;
size_t sz;
file2=fopen ("outCert","rb");
if(file2!=NULL){
fseek(file2, 0L, SEEK_END);
sz = ftell(file2);
fseek(file2, 0, SEEK_SET);
msg2 = new unsigned char [sz];
fread(msg2,1,sz,file2);
fclose(file2);
}else cout << "Unable to open file";
//change 'msg2' and 'sz' for 'sig' and 'len' to work respectively
verify(pubkey,msg,msg2,sz);
One problem is that it can't read from the file right after writing in it. So I have to run once to write, comment the writing part and run to read. << SOLVED this part. Just added outfile.close() after writing
I don't know if OpenSSL functions are affected by time, but tried put some delay between the functions and got same result.
Any help is welcome!

Why does my OpenSSL C++ code create binary encryption output?

I'm trying to encrypt a file using AES from OpenSSL and then write the output to a file. But I'm getting messy outputs, sometimes decipherable and sometimes not.
The main code is based from here: https://github.com/shanet/Crypto-Example/blob/master/crypto-example.cpp
Here's the code:
int Crypt::__aesEncrypt(const unsigned char *msg, size_t msgLen, unsigned char **encMsg) {
EVP_CIPHER_CTX *aesEncryptCtx = (EVP_CIPHER_CTX*)malloc(sizeof(EVP_CIPHER_CTX));
EVP_CIPHER_CTX_init(aesEncryptCtx);
unsigned char *aesKey = (unsigned char*)malloc(AES_KEYLEN/8);
unsigned char *aesIV = (unsigned char*)malloc(AES_KEYLEN/8);
unsigned char *aesPass = (unsigned char*)malloc(AES_KEYLEN/8);
unsigned char *aesSalt = (unsigned char*)malloc(8);
if(RAND_bytes(aesPass, AES_KEYLEN/8) == 0) {
return FAILURE;
}
if(RAND_bytes(aesSalt, 8) == 0) {
return FAILURE;
}
if(EVP_BytesToKey(EVP_aes_256_cbc(), EVP_sha1(), aesSalt, aesPass, AES_KEYLEN/8, AES_ROUNDS, aesKey, aesIV) == 0) {
return FAILURE;
}
strncpy((char*)aesKey, (const char*)"B374A26A71490437AA024E4FADD5B4AA", AES_KEYLEN/8);
strncpy((char*)aesIV, (const char*)"7E892875A52C59A3B588306B13C31FBD", AES_KEYLEN/16);
size_t blockLen = 0;
size_t encMsgLen = 0;
*encMsg = (unsigned char*)malloc(msgLen + AES_BLOCK_SIZE);
if(encMsg == NULL) return FAILURE;
if(!EVP_EncryptInit_ex(aesEncryptCtx, EVP_aes_256_cbc(), NULL, aesKey, aesIV)) {
return FAILURE;
}
if(!EVP_EncryptUpdate(aesEncryptCtx, *encMsg, (int*)&blockLen, (unsigned char*)msg, msgLen)) {
return FAILURE;
}
encMsgLen += blockLen;
if(!EVP_EncryptFinal_ex(aesEncryptCtx, *encMsg + encMsgLen, (int*)&blockLen)) {
return FAILURE;
}
EVP_CIPHER_CTX_cleanup(aesEncryptCtx);
free(aesEncryptCtx);
free(aesKey);
free(aesIV);
return encMsgLen + blockLen;
}
Im calling like this:
unsigned char *encMsg = NULL;
__aesEncrypt((const unsigned char*)decrypted_string.c_str(), decrypted_string.size(), &encMsg);
std::stringstream ss;
ss << encMsg;
//write ss to file...
Thanks.
I'm actually the author of the example you've based your code off of. As WhozCraig pointed out in the comments above, you are using a stringstream to write the encrypted message to a file. The problem with this is that encrypted messages are not regular ASCII strings. They are binary data (values greater than 127, hence the need for an unsigned char array) and binary data cannot be treated the same as ASCII strings.
I'm not much of a C++ person, so I would write the data to a file the C way with fwrite, but if you want to do it the C++ way, I think you're looking for ifstream rather than stringstream.
Side note, I'm betting this is just for debugging, but I'll point it out anyway just to make sure: Hardcoding your AES key and IV (strncpy((char*)aesKey, (const char*)"B374A26A71490437AA024E4FADD5B4AA", AES_KEYLEN/8)) completely defeats the purpose of encryption. If you want to avoid the PBKDF (EVP_BytesToKey) you can just use RAND_Bytes to get random data for your AES key.

Can you help me get my head around openssl public key encryption with rsa.h in c++?

I am trying to get my head around public key encryption using the openssl implementation of rsa in C++. Can you help? So far these are my thoughts (please do correct if necessary)
Alice is connected to Bob over a network
Alice and Bob want secure communications
Alice generates a public / private key pair and sends public key to Bob
Bob receives public key and encrypts a randomly generated symmetric cypher key (e.g. blowfish) with the public key and sends the result to Alice
Alice decrypts the ciphertext with the originally generated private key and obtains the symmetric blowfish key
Alice and Bob now both have knowledge of symmetric blowfish key and can establish a secure communication channel
Now, I have looked at the openssl/rsa.h rsa implementation (since I already have practical experience with openssl/blowfish.h), and I see these two functions:
int RSA_public_encrypt(int flen, unsigned char *from,
unsigned char *to, RSA *rsa, int padding);
int RSA_private_decrypt(int flen, unsigned char *from,
unsigned char *to, RSA *rsa, int padding);
If Alice is to generate *rsa, how does this yield the rsa key pair? Is there something like rsa_public and rsa_private which are derived from rsa? Does *rsa contain both public and private key and the above function automatically strips out the necessary key depending on whether it requires the public or private part? Should two unique *rsa pointers be generated so that actually, we have the following:
int RSA_public_encrypt(int flen, unsigned char *from,
unsigned char *to, RSA *rsa_public, int padding);
int RSA_private_decrypt(int flen, unsigned char *from,
unsigned char *to, RSA *rsa_private, int padding);
Secondly, in what format should the *rsa public key be sent to Bob? Must it be reinterpreted in to a character array and then sent the standard way? I've heard something about certificates -- are they anything to do with it?
Sorry for all the questions,
Best Wishes,
Ben.
EDIT: Coe I am currently employing:
/*
* theEncryptor.cpp
*
*
* Created by ben on 14/01/2010.
* Copyright 2010 __MyCompanyName__. All rights reserved.
*
*/
#include "theEncryptor.h"
#include <iostream>
#include <sys/socket.h>
#include <sstream>
theEncryptor::theEncryptor()
{
}
void
theEncryptor::blowfish(unsigned char *data, int data_len, unsigned char* key, int enc)
{
// hash the key first!
unsigned char obuf[20];
bzero(obuf,20);
SHA1((const unsigned char*)key, 64, obuf);
BF_KEY bfkey;
int keySize = 16;//strlen((char*)key);
BF_set_key(&bfkey, keySize, obuf);
unsigned char ivec[16];
memset(ivec, 0, 16);
unsigned char* out=(unsigned char*) malloc(data_len);
bzero(out,data_len);
int num = 0;
BF_cfb64_encrypt(data, out, data_len, &bfkey, ivec, &num, enc);
//for(int i = 0;i<data_len;i++)data[i]=out[i];
memcpy(data, out, data_len);
free(out);
}
void
theEncryptor::generateRSAKeyPair(int bits)
{
rsa = RSA_generate_key(bits, 65537, NULL, NULL);
}
int
theEncryptor::publicEncrypt(unsigned char* data, unsigned char* dataEncrypted,int dataLen)
{
return RSA_public_encrypt(dataLen, data, dataEncrypted, rsa, RSA_PKCS1_OAEP_PADDING);
}
int
theEncryptor::privateDecrypt(unsigned char* dataEncrypted,
unsigned char* dataDecrypted)
{
return RSA_private_decrypt(RSA_size(rsa), dataEncrypted,
dataDecrypted, rsa, RSA_PKCS1_OAEP_PADDING);
}
void
theEncryptor::receivePublicKeyAndSetRSA(int sock, int bits)
{
int max_hex_size = (bits / 4) + 1;
char keybufA[max_hex_size];
bzero(keybufA,max_hex_size);
char keybufB[max_hex_size];
bzero(keybufB,max_hex_size);
int n = recv(sock,keybufA,max_hex_size,0);
n = send(sock,"OK",2,0);
n = recv(sock,keybufB,max_hex_size,0);
n = send(sock,"OK",2,0);
rsa = RSA_new();
BN_hex2bn(&rsa->n, keybufA);
BN_hex2bn(&rsa->e, keybufB);
}
void
theEncryptor::transmitPublicKey(int sock, int bits)
{
const int max_hex_size = (bits / 4) + 1;
long size = max_hex_size;
char keyBufferA[size];
char keyBufferB[size];
bzero(keyBufferA,size);
bzero(keyBufferB,size);
sprintf(keyBufferA,"%s\r\n",BN_bn2hex(rsa->n));
sprintf(keyBufferB,"%s\r\n",BN_bn2hex(rsa->e));
int n = send(sock,keyBufferA,size,0);
char recBuf[2];
n = recv(sock,recBuf,2,0);
n = send(sock,keyBufferB,size,0);
n = recv(sock,recBuf,2,0);
}
void
theEncryptor::generateRandomBlowfishKey(unsigned char* key, int bytes)
{
/*
srand( (unsigned)time( NULL ) );
std::ostringstream stm;
for(int i = 0;i<bytes;i++){
int randomValue = 65 + rand()% 26;
stm << (char)((int)randomValue);
}
std::string str(stm.str());
const char* strs = str.c_str();
for(int i = 0;bytes;i++)key[i]=strs[i];
*/
int n = RAND_bytes(key, bytes);
if(n==0)std::cout<<"Warning key was generated with bad entropy. You should not consider communication to be secure"<<std::endl;
}
theEncryptor::~theEncryptor(){}
You should actually be using the higher-level "Envelope Encryption" functions from openssl/evp.h, rather than the low-level RSA functions directly. These do most of the work for you and mean you don't have to reinvent the wheel.
In this case, you'd use the EVP_SealInit(), EVP_SealUpdate() and EVP_SealFinal() functions. The corresponding decryption functions are EVP_OpenInit(), EVP_OpenUpdate() and EVP_OpenFinal(). I would suggest using EVP_aes_128_cbc() as the value of the cipher type parameter.
Once you've got the public key loaded into an RSA * handle, you use EVP_PKEY_assign_RSA() to put it into an EVP_PKEY * handle for the EVP functions.
Once you've got this going, to solve the authentication problem I mentioned in my comment, you'll need to established a trusted authority ("Trent"). Trent's public key is known to all users (distributed with the application or similar - just load it from a PEM file). Instead of exchanging bare RSA parameters, Alice and Bob exchange x509 certificates that contain their RSA public keys together with their name, and are signed by Trent. Alice and Bob then each verify the certificate they recieved from the other (using Trent's public key, which they already know), including checking that the associated name is the right one, before continuing the protocol. OpenSSL includes functions for loading and verifying certificates in the x509.h header.
Here's an example of how to use EVP_Seal*() to encrypt a file given the recipient's public key. It takes the PEM RSA Public Key file (ie as generated by openssl rsa -pubout) as a command line argument, reads the source data from stdin and writes the encrypted data to stdout. To decrypt, use EVP_Open*() instead, and PEM_read_RSAPrivateKey() to read a private key rather than public key.
It's not really that hard - and certainly less error prone than messing about generating padding, IVs and so on yourself (the Seal function does both the RSA and AES parts of the deal). Anyway, the code:
#include <stdio.h>
#include <stdlib.h>
#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/err.h>
#include <arpa/inet.h> /* For htonl() */
int do_evp_seal(FILE *rsa_pkey_file, FILE *in_file, FILE *out_file)
{
int retval = 0;
RSA *rsa_pkey = NULL;
EVP_PKEY *pkey = EVP_PKEY_new();
EVP_CIPHER_CTX ctx;
unsigned char buffer[4096];
unsigned char buffer_out[4096 + EVP_MAX_IV_LENGTH];
size_t len;
int len_out;
unsigned char *ek;
int eklen;
uint32_t eklen_n;
unsigned char iv[EVP_MAX_IV_LENGTH];
if (!PEM_read_RSA_PUBKEY(rsa_pkey_file, &rsa_pkey, NULL, NULL))
{
fprintf(stderr, "Error loading RSA Public Key File.\n");
ERR_print_errors_fp(stderr);
retval = 2;
goto out;
}
if (!EVP_PKEY_assign_RSA(pkey, rsa_pkey))
{
fprintf(stderr, "EVP_PKEY_assign_RSA: failed.\n");
retval = 3;
goto out;
}
EVP_CIPHER_CTX_init(&ctx);
ek = malloc(EVP_PKEY_size(pkey));
if (!EVP_SealInit(&ctx, EVP_aes_128_cbc(), &ek, &eklen, iv, &pkey, 1))
{
fprintf(stderr, "EVP_SealInit: failed.\n");
retval = 3;
goto out_free;
}
/* First we write out the encrypted key length, then the encrypted key,
* then the iv (the IV length is fixed by the cipher we have chosen).
*/
eklen_n = htonl(eklen);
if (fwrite(&eklen_n, sizeof eklen_n, 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
if (fwrite(ek, eklen, 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
if (fwrite(iv, EVP_CIPHER_iv_length(EVP_aes_128_cbc()), 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
/* Now we process the input file and write the encrypted data to the
* output file. */
while ((len = fread(buffer, 1, sizeof buffer, in_file)) > 0)
{
if (!EVP_SealUpdate(&ctx, buffer_out, &len_out, buffer, len))
{
fprintf(stderr, "EVP_SealUpdate: failed.\n");
retval = 3;
goto out_free;
}
if (fwrite(buffer_out, len_out, 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
}
if (ferror(in_file))
{
perror("input file");
retval = 4;
goto out_free;
}
if (!EVP_SealFinal(&ctx, buffer_out, &len_out))
{
fprintf(stderr, "EVP_SealFinal: failed.\n");
retval = 3;
goto out_free;
}
if (fwrite(buffer_out, len_out, 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
out_free:
EVP_PKEY_free(pkey);
free(ek);
out:
return retval;
}
int main(int argc, char *argv[])
{
FILE *rsa_pkey_file;
int rv;
if (argc < 2)
{
fprintf(stderr, "Usage: %s <PEM RSA Public Key File>\n", argv[0]);
exit(1);
}
rsa_pkey_file = fopen(argv[1], "rb");
if (!rsa_pkey_file)
{
perror(argv[1]);
fprintf(stderr, "Error loading PEM RSA Public Key File.\n");
exit(2);
}
rv = do_evp_seal(rsa_pkey_file, stdin, stdout);
fclose(rsa_pkey_file);
return rv;
}
The code you've posted illustrates nicely why you should use the higher-level functions - you've fallen into a couple of pitfalls:
rand() is emphatically not a cryptographically strong random number generator! Generating your symmetric key using rand() is enough to make the entire system completely insecure. (The EVP_*() functions generate the necessary random numbers themselves, using a cryptographically strong RNG, seeded from an appropriate entropy source).
You are setting the IV for CFB mode to a fixed value (zero). This negates any advantage of using CFB mode in the first place (allowing attackers to trivially perform block-replacement attacks and worse). (The EVP_*() functions generate an appropriate IV for you, when required).
RSA_PKCS1_OAEP_PADDING should be used if you're defining a new protocol, rather than interoperating with an existing protocol.
The corresponding decryption code, for posterity:
#include <stdio.h>
#include <stdlib.h>
#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/err.h>
#include <arpa/inet.h> /* For htonl() */
int do_evp_unseal(FILE *rsa_pkey_file, FILE *in_file, FILE *out_file)
{
int retval = 0;
RSA *rsa_pkey = NULL;
EVP_PKEY *pkey = EVP_PKEY_new();
EVP_CIPHER_CTX ctx;
unsigned char buffer[4096];
unsigned char buffer_out[4096 + EVP_MAX_IV_LENGTH];
size_t len;
int len_out;
unsigned char *ek;
unsigned int eklen;
uint32_t eklen_n;
unsigned char iv[EVP_MAX_IV_LENGTH];
if (!PEM_read_RSAPrivateKey(rsa_pkey_file, &rsa_pkey, NULL, NULL))
{
fprintf(stderr, "Error loading RSA Private Key File.\n");
ERR_print_errors_fp(stderr);
retval = 2;
goto out;
}
if (!EVP_PKEY_assign_RSA(pkey, rsa_pkey))
{
fprintf(stderr, "EVP_PKEY_assign_RSA: failed.\n");
retval = 3;
goto out;
}
EVP_CIPHER_CTX_init(&ctx);
ek = malloc(EVP_PKEY_size(pkey));
/* First need to fetch the encrypted key length, encrypted key and IV */
if (fread(&eklen_n, sizeof eklen_n, 1, in_file) != 1)
{
perror("input file");
retval = 4;
goto out_free;
}
eklen = ntohl(eklen_n);
if (eklen > EVP_PKEY_size(pkey))
{
fprintf(stderr, "Bad encrypted key length (%u > %d)\n", eklen,
EVP_PKEY_size(pkey));
retval = 4;
goto out_free;
}
if (fread(ek, eklen, 1, in_file) != 1)
{
perror("input file");
retval = 4;
goto out_free;
}
if (fread(iv, EVP_CIPHER_iv_length(EVP_aes_128_cbc()), 1, in_file) != 1)
{
perror("input file");
retval = 4;
goto out_free;
}
if (!EVP_OpenInit(&ctx, EVP_aes_128_cbc(), ek, eklen, iv, pkey))
{
fprintf(stderr, "EVP_OpenInit: failed.\n");
retval = 3;
goto out_free;
}
while ((len = fread(buffer, 1, sizeof buffer, in_file)) > 0)
{
if (!EVP_OpenUpdate(&ctx, buffer_out, &len_out, buffer, len))
{
fprintf(stderr, "EVP_OpenUpdate: failed.\n");
retval = 3;
goto out_free;
}
if (fwrite(buffer_out, len_out, 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
}
if (ferror(in_file))
{
perror("input file");
retval = 4;
goto out_free;
}
if (!EVP_OpenFinal(&ctx, buffer_out, &len_out))
{
fprintf(stderr, "EVP_OpenFinal: failed.\n");
retval = 3;
goto out_free;
}
if (fwrite(buffer_out, len_out, 1, out_file) != 1)
{
perror("output file");
retval = 5;
goto out_free;
}
out_free:
EVP_PKEY_free(pkey);
free(ek);
out:
return retval;
}
int main(int argc, char *argv[])
{
FILE *rsa_pkey_file;
int rv;
if (argc < 2)
{
fprintf(stderr, "Usage: %s <PEM RSA Private Key File>\n", argv[0]);
exit(1);
}
rsa_pkey_file = fopen(argv[1], "rb");
if (!rsa_pkey_file)
{
perror(argv[1]);
fprintf(stderr, "Error loading PEM RSA Private Key File.\n");
exit(2);
}
rv = do_evp_unseal(rsa_pkey_file, stdin, stdout);
fclose(rsa_pkey_file);
return rv;
}
Actually, no problem, I have just read that basically, the RSA object is a structure that contains both public and private fields. One can extract the public field data and only send that to Bob.
I.e. basically, to extract the public fields from rsa and store each in two different buffers (which are char arrays and can then be sent to Bob), you do:
sprintf(keyBufferA,"%s\r\n",BN_bn2hex(rsa->n));
sprintf(keyBufferB,"%s\r\n",BN_bn2hex(rsa->e));
And then Bob, on the receiving end, reconstructs as follows:
rsa = RSA_new();
BN_hex2bn(&rsa->n, keybufA);
BN_hex2bn(&rsa->e, keybufB);
Bob can then use rsa* to publicly encrypt the symmetric cypher key which can then be sent to Alice. Alice can then decrypt with the private key
Ben.
I write two examples around CAF's code. They are heavily modifed and uses OpenSSL's BIO container for more abstraction.
One example uses a file as input and the other example uses a string buffer. It uses RSA and DES, however you can easily change it from the code. Compile instructions are inside the code. I needed a working example, I hope someone find this useful. I also commented the code. You can get it from here:
Take file as input:
https://github.com/farslan/snippets/blob/master/hybrid_file.c
Take string buffer as input:
https://github.com/farslan/snippets/blob/master/hybrid_data.c
Thanks #Caf. Your post helped. However I got
The program '[7056] Encryption2.exe: Native' has exited with code -1073741811 (0xc000000d) for the line
PEM_read_RSA_PUBKEY(rsa_pkey_file, &rsa_pkey, NULL, NULL)
I changed to
BIO *bio;
X509 *certificate;
bio = BIO_new(BIO_s_mem());
BIO_puts(bio, (const char*)data);
certificate = PEM_read_bio_X509(bio, NULL, NULL, NULL);
EVP_PKEY *pubkey = X509_get_pubkey (certificate);
rsa_pkey = EVP_PKEY_get1_RSA(pubkey);
Where data has the PEM file with only public key. My challenge was to encrypt in C++ and decrypt in java. I transmitted the base64 encoded ek of size eklen (i did not use eklen_n) and decrypted to get the AES key using the RSA private key. Then I decrypted the cipher file using this AES key. It worked fine.