OpenSSL EVP_DigestSignFinal segfault - c++

Question

I'm trying to sign a message using the OpenSSL C api. The following code segfaults because of EXC_BAD_ACCESS during either of the calls to EVP_DigestSignFinal. I'm using OpenSSL 1.0.1g. I tried switching from the newer DigestSign* functions to the older Sign* functions, and it still segfaults.
private_key is set with EVP_PKEY_set1_RSA from an RSA key loaded from a PEM file. The first call to EVP_DigestSignFinal fills s_len with the maximum possible length of the signature for the signing algorithm, so signature not being big enough shouldn't be the issue, and the second call writes to the signature buffer and fills s_len with the length of the signature.
I would appreciate any help I could get with this.
vector<unsigned char> rsa_sha512_sign(
const vector<unsigned char>& document,
shared_ptr<EVP_PKEY> private_key) {
EVP_MD_CTX* md;
if (!(md = EVP_MD_CTX_create())) {
throw runtime_error("Error initializing ENV_MD_CTX.");
}
if (EVP_DigestSignInit(md, NULL, EVP_sha512(), NULL, private_key.get())
!= 1) {
throw runtime_error("Error in EVP_DigestSignInit.");
}
if (EVP_DigestSignUpdate(md, document.data(), document.size()) != 1) {
throw runtime_error("Error computing hash on document.");
}
size_t s_len;
if (EVP_DigestSignFinal(md, NULL, &s_len) != 1) { // Segfault here
throw runtime_error("Error determining maximum signature size.");
}
vector<unsigned char> signature(s_len);
if (EVP_DigestSignFinal(md, signature.data(), &s_len) != 1) { // or here (or both)
throw runtime_error("Error signing document.");
}
signature.resize(s_len);
EVP_MD_CTX_destroy(md);
return move(signature);
}

Answer 1

The problem is likely with how you are initializing private_key. Probably, you are mixing malloc() with delete, and corrupting the heap in the process. You need to provide shared_ptr the proper deleter for the pointer you feed to it if the pointer was not created with new.
shared_ptr<RSA> r(RSA_new(), RSA_free);
shared_ptr<EVP_PKEY> p(EVP_PKEY_new(), EVP_PKEY_free);
shared_ptr<BIGNUM> bn(BN_new(), BN_free);
vector<unsigned char> doc(0, 100);
BN_set_word(bn.get(), RSA_F4);
RSA_generate_key_ex(r.get(), 2048, bn.get(), 0);
EVP_PKEY_set1_RSA(p.get(), r.get());
rsa_sha512_sign(doc, p);