RSA_private_decrypt crashes when called for the second time - c++

Question

Please take a look at the following method:
int BCVirtualCard::decrypt(std::string from, std::string *to, int keyId, bool padding)
{
if (to == nullptr)
{
NSCAssert(NO, #"Invalid params");
return 0;
}
NSString* privateKey = [m_storage privateKeyForSlot:keyId];
NSArray<NSString*>* components = [privateKey componentsSeparatedByString:#"_"];
const NSInteger componentsCount = 4;
if (components.count != componentsCount)
{
*to = "";
return 0;
}
const char* d = [components[0] UTF8String];
const char* n = [components[1] UTF8String];
const char* p = [components[2] UTF8String];
const char* q = [components[3] UTF8String];
RSA* rsa = RSA_new();
BN_hex2bn(&rsa->d, d);
BN_hex2bn(&rsa->n, n);
BN_hex2bn(&rsa->p, p);
BN_hex2bn(&rsa->q, q);
unsigned char* _to = (unsigned char *)calloc(1, sizeof(unsigned char));
int decryptedSize = RSA_private_decrypt((int)from.length(), (unsigned char *)from.c_str(), _to, rsa, RSA_NO_PADDING);
free(_to);
if (decryptedSize <= 0)
{
ERR_print_errors_cb(test, NULL);
*to = "";
return 0;
}
_to = (unsigned char *)calloc(decryptedSize, sizeof(unsigned char));
RSA_private_decrypt((int)from.length(), (unsigned char *)from.c_str(), _to, rsa, RSA_NO_PADDING);
*to = std::string((char *)_to, strlen((char *)_to));
free(_to);
RSA_free(rsa);
return 1;
}
Here the string from is supposed to be decrypted and written to the string to. For decryption I use RSA_private_decrypt function. I call it two times. First time in order to determine the size of the decrypted text and second time in order to write the decrypted text into _to buffer. And when I call it second time it usually crashes like this:
malloc: Heap corruption detected, free list is damaged at 0x280ff3d70
*** Incorrect guard value: 0
No1BCmail(2171,0x170efb000) malloc: *** set a breakpoint in malloc_error_break to debug
The breakpoint is on and this let me find the place of the crash. However I can't understand the reason why it crashes. I tried recreating the RSA structure and playing with the size allocated for _to for the second time, but nothing helps. Can you see what is wrong here? Thanks

Answer 1

RSA_private_decrypt requires that the to parameter points to a buffer of an adequate size. Your first call only uses a buffer of size 1, which is too small and is undefined behavior. What you need to do is get the size from rsa using RSA_size(rsa), and then you can use that to allocate space for _to. That means you don't need to call the function twice, since you'll already have the size the first time around
You should also be using decryptedSize for the length if the string to build instead of using strlen as _to may not be null terminated.
Putting all that together you could should look something like
int BCVirtualCard::decrypt(std::string from, std::string *to, int keyId, bool padding)
{
if (to == nullptr)
{
NSCAssert(NO, #"Invalid params");
return 0;
}
NSString* privateKey = [m_storage privateKeyForSlot:keyId];
NSArray<NSString*>* components = [privateKey componentsSeparatedByString:#"_"];
const NSInteger componentsCount = 4;
if (components.count != componentsCount)
{
*to = "";
return 0;
}
const char* d = [components[0] UTF8String];
const char* n = [components[1] UTF8String];
const char* p = [components[2] UTF8String];
const char* q = [components[3] UTF8String];
RSA* rsa = RSA_new();
BN_hex2bn(&rsa->d, d);
BN_hex2bn(&rsa->n, n);
BN_hex2bn(&rsa->p, p);
BN_hex2bn(&rsa->q, q);
auto _to = std::make_unique<unsigned char[]>(RSA_size(rsa)); // use smart pointers so you don't have to worry about releasing the memory
int decryptedSize = RSA_private_decrypt((int)from.length(), (unsigned char *)from.c_str(), _to.get(), rsa, RSA_NO_PADDING);
if (decryptedSize <= 0)
{
ERR_print_errors_cb(test, NULL);
*to = "";
return 0;
}
*to = std::string((char *)_to.get(), decryptedSize);
return 1;
}