Segmentation fault when trying to use EVP functions in OpenSSL - c++

Question

I am trying to do public encryption with OpenSSL using RSA and its high-level envelope functions. However I cannot seem to get my head around them and I'm getting a segmentation fault. This condensed code from my project reproduces the problem:
#include <iostream>
#include <string>
#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <openssl/rand.h>
int main()
{
EVP_CIPHER_CTX *rsaCtx;
rsaCtx = new EVP_CIPHER_CTX;
unsigned char *ek;
size_t ekl;
unsigned char *iv;
size_t ivl;
EVP_PKEY *keypair;
keypair = NULL;
EVP_CIPHER_CTX_init(rsaCtx);
EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL);
EVP_PKEY_keygen_init(ctx);
EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, 2048);
EVP_PKEY_keygen(ctx, &keypair);
EVP_PKEY_CTX_free(ctx);
ek = new unsigned char[EVP_PKEY_size(keypair)];
iv = new unsigned char[EVP_MAX_IV_LENGTH];
ivl = EVP_MAX_IV_LENGTH;
std::string cipherText;
std::string plainText = "A STRING";
size_t encMsgLen = 0;
size_t blockLen = 0;
EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, (int*)ekl, iv, &keypair, 1);
EVP_SealUpdate(rsaCtx, (unsigned char*)cipherText.c_str() + encMsgLen, (int*)&blockLen, (const unsigned char*)plainText.c_str(), (int)plainText.size() + 1);
encMsgLen += blockLen;
EVP_SealFinal(rsaCtx, (unsigned char*)cipherText.c_str() + encMsgLen, (int*)&blockLen);
encMsgLen += blockLen;
EVP_CIPHER_CTX_cleanup(rsaCtx);
EVP_PKEY_free(keypair);
delete[] ek;
delete[] iv;
delete rsaCtx;
std::cout << cipherText;
return 0;
}
I get a segmentation fault at the line EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, (int*)ekl, iv, &keypair, 1);
What am I doing wrong?

Answer 1

ekl is a size_t, and you are casting it to an (int*).
The docs for EVP_SealInit say:
The actual size of each encrypted secret key is written to the array
ekl.
You're just passing one key, so passing the address of a single integer is sufficient, but you should be passing the address of that integer, e.g.:
EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, reinterpret_cast<int*>(&ekl), iv, &keypair, 1);
Alternatively, just declare ekl as an int in the first place, and you can avoid the cast:
int ekl;
//...
EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, &ekl, iv, &keypair, 1);
I'm surprised your compiler didn't warn you about using an uninitialized local variable.
UPDATE: There are some more problems with this code besides the segmentation fault.
You're passing the buffer from an empty std::string (cipherText) into EVP_SealUpdate and EVP_SealFinal. This isn't going to work in general, and may crash or corrupt memory if there isn't enough room in the buffer.
You should declare a buffer of suitable size for the output, perhaps as std::vector<unsigned char> cipherText(bufferSize);, and pass &cipherText[0] to get the pointer to the first element.
The data in cipherText isn't a human-readable string, it's binary data, and std::cout isn't suitable for displaying it.
Some more general notes:
Avoid C-style casts in C++, and where possible write the code so you don't need to cast at all (e.g. declare the integers as int rather than size_t if that's what the APIs are expecting).
Avoid explicit memory-management with new and delete where you can, e.g. by using std::vector<unsigned char> for the buffers.
I suggest having a look at the documentation for these functions again, or some other examples on the web for using them. Also, write some code which does the decryption step so you can test that the plain-text is round-tripping correctly.

Answer 2

Using a too small buffer with EVP_SealX functions may create maddening consequences in seemingly unrelated parts of your code. This was my experience.
Putting in guards that ensure the cipher buffer is as big as the total plaintext input plus possible padding overhead will reduce risk.