How to avoid SIGABRT when generating RSA Signature at EVP_SignFinal - c++

Question

I'm trying to generate a RSA Signature with libopenssl for c++:
But when I run my code, I get a SIGABRT. I did some deep debugging into libopenssl internal stuff to see where the Segfault comes from. I'll come to this later on.
First I want to make clear, that the RSA PrivateKey was successfully loaded from a .pem file. So Im pretty sure that's not the problem's origin.
So my question is: How to avoid the SIGABRT and what is the cause of it ?
I'm doing this for my B.Sc. Thesis so I really appreciate your help :)
Signature Generation Function:
DocumentSignature* RSASignatureGenerator::generateSignature(ContentHash* ch, CryptographicKey* pK) throw(PDVSException) {
OpenSSL_add_all_algorithms();
OpenSSL_add_all_ciphers();
OpenSSL_add_all_digests();
if(pK == nullptr)
throw MissingPrivateKeyException();
if(pK->getKeyType() != CryptographicKey::KeyType::RSA_PRIVATE || !dynamic_cast<RSAPrivateKey*>(pK))
throw KeyTypeMissmatchException(pK->getPem()->getPath().string(), "Generate RSA Signature");
//get msg to encrypt
const char* msg = ch->getStringHash().c_str();
//get openssl rsa key
RSA* rsaPK = dynamic_cast<RSAPrivateKey*>(pK)->createOpenSSLRSAKeyObject();
//create openssl signing context
EVP_MD_CTX* rsaSignCtx = EVP_MD_CTX_create();
EVP_PKEY* priKey = EVP_PKEY_new();
EVP_PKEY_assign_RSA(priKey, rsaPK);
//init ctxt
if (EVP_SignInit(rsaSignCtx, EVP_sha256()) <=0)
throw RSASignatureGenerationException();
//add data to sign
if (EVP_SignUpdate(rsaSignCtx, msg, std::strlen(msg)) <= 0) {
throw RSASignatureGenerationException();
}
//create result byte signature struct
DocumentSignature::ByteSignature* byteSig = new DocumentSignature::ByteSignature();
//set size to max possible
byteSig->size = EVP_MAX_MD_SIZE;
//alloc buffer memory
byteSig->data = (unsigned char*)malloc(byteSig->size);
//do signing
if (EVP_SignFinal(rsaSignCtx, byteSig->data, (unsigned int*) &byteSig->size, priKey) <= 0)
throw RSASignatureGenerationException();
DocumentSignature* res = new DocumentSignature(ch);
res->setByteSignature(byteSig);
EVP_MD_CTX_destroy(rsaSignCtx);
//TODO open SSL Memory leaks -> where to free open ssl stuff?!
return res;
}
RSA* rsaPK = dynamic_cast(pK)->createOpenSSLRSAKeyObject();
virtual RSA* createOpenSSLRSAKeyObject() throw (PDVSException) override {
RSA* rsa = NULL;
const char* c_string = _pem->getContent().c_str();
BIO * keybio = BIO_new_mem_buf((void*)c_string, -1);
if (keybio==NULL)
throw OpenSSLRSAPrivateKeyObjectCreationException(_pem->getPath());
rsa = PEM_read_bio_RSAPrivateKey(keybio, &rsa, NULL, NULL);
if(rsa == nullptr)
throw OpenSSLRSAPrivateKeyObjectCreationException(_pem->getPath());
//BIO_free(keybio);
return rsa;
}
SigAbrt origin in file openssl/crypto/mem.c
void CRYPTO_free(void *str, const char *file, int line)
{
if (free_impl != NULL && free_impl != &CRYPTO_free) {
free_impl(str, file, line);
return;
}
#ifndef OPENSSL_NO_CRYPTO_MDEBUG
if (call_malloc_debug) {
CRYPTO_mem_debug_free(str, 0, file, line);
free(str);
CRYPTO_mem_debug_free(str, 1, file, line);
} else {
free(str);
}
#else
free(str); // <<<<<<< HERE
#endif
}
the stacktrace
stacktrace screenshot from debugger (clion - gdb based)

Answer 1

I just found the Bug (and Im really not sure if this is a libopenssl bug..)
//set size to max possible
byteSig->size = EVP_MAX_MD_SIZE;
//alloc buffer memory
byteSig->data = (unsigned char*)malloc(byteSig->size);
The problem was when I set the buffer size to EVP_MAX_MD_SIZE!
The (in my opinion) very very strange thing is, that you have to keep the size uninitialized! (not even set to 0 - just "size_t size;" ).
Strange thing here is that then you also HAVE TO allocate memory just like I did. I dont understand this because then an undefined size of memory gets allocated..
What the really weird is that libopenssl internally sets the size back to 0 and allocates the memory itself.. (I detected this by browsing the libopenssl source code)