Crash in d2i_RSAPrivateKey_fp() - c++

Question

I have a small VC++ code to load an ASN.1 .der file and read its private key. It compiles and runs fine, but it crashes at function d2i_RSAPrivateKey_fp.
Tools Used:
Visual Studio 2008 [V90]
OpenSSL 1.0.2d 9 Jul 2015
I have also tested the certificate to check if it was corrupted
Testing certificate:
C:\OpenSSL-Win32\bin>openssl rsa -in private.der -check -inform DER
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQC/4V4jxRYeFBDh8XBNq2EMs3hXWW5IIN51lM/Oyz0U/Bw8HF0m
/VBJU3SCy2FzoYPa2o3HHYWDMnjmOlXb9aXR3hyLHnvgvE/0YkMXlxh58H1srjw4
FL7cLXe+lwFbZYtxRaHyn/3U3NIkZkCzR74oxHwyWJ1/zz+TzpmNx8AfyQIDAQAB
AoGAMn+9puxXxdLCHrTMOaTBBfa11UdUHueHpKplhqc2jC9NvwQ3/+rrFmFAaKve
GfCIIzEh3yWF3eGKsAzqS9l6qiyAT+LaCPcn7FbAsaMdVGPc4G290/3maASjQMj6
YfNTcmvPaqfq2+B+aNunS0pi6XGv3917KdGo4hTa2xFXwAECQQDfm5Rh0CjDwhjh
urghbvBgGVRJiAFowgp9xdJj8Hm/U+zXK9Tz2SwVcbEKbcqSVZqB8Keb11TEfF3W
JGIHGdOJAkEA260wKNAdWxgEXiSCEx/tSV3bcRQg4ypTqODc0cWtM295/lhV2Hgx
3zWQ5NiBYZsU104DLWlxnqulJKyBECmaQQJAFAjskpcEEAYkFJWWSeiWwQWD99Kn
zasVJY/D+hBh2DK81cqnmfGrcYBuTHDp5ZKl9V6Kpfv1LGW4Qqef4OL/gQJAb1Mp
IMW22r8lF4Bw2rhHS/LgjkGhGP4OP3sU7Mm8qGBJ9ndVqcnfnDpNH2wIxSoOOb4z
JgRVrA9YNWmmTkaHAQJBANiY3RDyEouYoNKN0oWXsZ/N6BwSFOYAAknmdY5uRwu2
tUQHFiW7u2wZCFmIZVqSBZc5XBdZcmffZeEiO1nVYH4=
-----END RSA PRIVATE KEY-----
C:\OpenSSL-Win32\bin>
Below is the code for the project.
Thing I have tried:
Win32OpenSSL-0_9_8zg (but same issue), OpenSSL 1.0.1
Tried linking libeay to MD, MDd, MT
Set flags to Generate /MAP, /MAPINFO:EXPORTS ,/ASSEMBLYDEBUG
exported to VS2010 project and compiled with openssl32 for VS2010
#include <stdio.h>
#include <conio.h>
#include <string.h>
#include <openssl/rsa.h>
#include <openssl/x509.h>
#include <openssl/err.h>
#include <stdio.h>
int main()
{
int ret;
RSA *pkey=NULL;//RSA_new();
RSA *rsa =NULL;//RSA_new();
unsigned char text[2048/8] = "this is a test string";
unsigned char encrypted[4098]={};
unsigned char decrypted[4098]={};
OpenSSL_add_all_algorithms();
OpenSSL_add_all_ciphers();
OpenSSL_add_all_digests();
ERR_load_ERR_strings();
FILE *fp;
fp = fopen("C:/Users/Public/private.der","rb");
if (fp != NULL)
{
rsa = d2i_RSAPrivateKey_fp(fp, &pkey); // <<< CRASH
}
else
{
//return "Error::Unable to read private key file";
}
if (rsa)
{
// RSA is good
ret = RSA_private_encrypt(RSA_size(pkey), text, encrypted, pkey, RSA_NO_PADDING);
if (ret == -1)
{
fclose(fp);
//return "Error::Failed to encrypt the data";
}
}
else
{
fclose(fp);
//return "Error::Failed to get *RSA Handle1";
}
fclose(fp);
}
Second attempt
I am facing issue with importing private key from ASN.1 .der file to a .pem file.
C:\OpenSSL-Win32\bin>openssl rsa -in private.der -inform DER -out privatepem.pem -outform PEM
writing RSA key
Now, I have .PEM file which have private key in Base64 Form which i will convert it to RSA *rsa; structure.
int main()
{
int ret;
RSA *pkey=NULL;//RSA_new();
RSA *rsa =NULL;//RSA_new();
FILE *fp;
fp = fopen("C:/Users/Public/privatepem.pem","r");
if(PEM_read_RSAPrivateKey(fp, &rsa, NULL, NULL) == NULL) // <<<CRASH
{
printf("\n%s\n", "Error Reading public key");
}
else
{
printf("\n%s\n", "Private key Imported");
}
return 0;
}
But the new code operating on .pem files has a crash while copying to RSA structure at PEM_read_RSAPrivateKey.
Why am I getting the crash? Is there something I am missing to do, may be initialization of some sort or something wrong with the key?

Answer 1

See the warnings at d2i_X509. This code will attempt to free (or reuse parts of) the pkey object:
if (fp != NULL)
rsa = d2i_RSAPrivateKey_fp(fp, &pkey); /* CRASH */
OpenSSL is not exactly known for validating parameters before taking actions on them :)
Instead, use:
if (fp != NULL)
rsa = d2i_RSAPrivateKey_fp(fp, NULL);
Or:
RSA* pkey = RSA_new();
...
if (fp != NULL)
rsa = d2i_RSAPrivateKey_fp(fp, &pkey);
Also see Seg fault from d2i_RSAPrivateKey_fp on the OpenSSL Users mailing list.
As for the crash after using the API as directed, then it sounds like you have other issues. Since OpenSSL is cross-platofrm, its "write once, run everywhere". Port the OpenSSL-based TLS logic to Linux, and get a memory checker like Valgrind on it.

Answer 2

Problem was solved by replacing the OpenSSL binaries.
Code worked without any changes.
Project had prebuild dependencies of OpenSSL which was from unknown source.
So just by replacing dependencies from "https://slproweb.com/products/Win32OpenSSL.html" it worked like a charm.
:)
Thanks guys