RSA encryption between c++ and node.js - c++

Question

I have to send some encrypted data throught the network (websocket)
I generated a key pair with the the following node.js module :
https://github.com/juliangruber/keypair
My public key looks like this:
-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEAlUiMDQsBgj5P/T86w/eg9MXUj8M4WMVihP8YzmDxMqCFb7D+w4N/1XcxWxQT
....
Wo+SRCsr6npfp1ctDhMtkXIeNT4lKf3qUGhP5tbx/TreaNF/d8zCeinGR/KeBGadMwIDAQAB
-----END RSA PUBLIC KEY-----
In the C++ code, I generated a RSA class with read the public key via a char*
const char rsaKey1[] = "-----BEGIN RSA PUBLIC KEY-----\n"
"MIIBCgKCAQEAlUiMDQsBgj5P/T86w/eg9MXUj8M4WMVihP8YzmDxMqCFb7D+w4N/1XcxWxQT\n"
....
"Wo+SRCsr6npfp1ctDhMtkXIeNT4lKf3qUGhP5tbx/TreaNF/d8zCeinGR/KeBGadMwIDAQAB\n"
"-----END RSA PUBLIC KEY-----\n";
BIO* bio = BIO_new_mem_buf( rsaKey1, strlen(rsaKey1));
m_rsaPubKey = PEM_read_bio_RSAPublicKey(bio, NULL, NULL, NULL);
usigned the m_rsaPubKey , I have able to generate a std::vector of unsigned char with encrypted data
std::vector<u8> Rsa::encrypt(std::string & msg)
{
std::vector<u8> encryptedData;
char *encrypt = new char[RSA_size(m_rsaPubKey)];
int encryptLen;
if (encryptLen = RSA_public_encrypt(msg.size() + 1, (unsigned
char*)msg.c_str(), (unsigned char*)encrypt, m_rsaPubKey,
RSA_PKCS1_OAEP_PADDING) == -1)
{
LogOutSys("error encoding string");
}
for (u32 i = 0; i < strlen(encrypt); i++)
{
encryptedData.push_back(encrypt[i]);
}
delete encrypt;
return encryptedData;
}
I don't get any errors while reading the public key or encrypting my data so I assume the encryption went ok.
then the data went throught a websocket and is received with node.js
the private key is read like this:
var rsa = new RSA(fs.readFileSync("./rsa-keys/sj_private_1.pem"),
{encryptionScheme :'pkcs8'})
and decoding
var decrypted = rsa.decrypt(data)
where data is a buffer of same length and content (no corruption while sending via the websocket)
c++ side:
encrypted len 256, first bytes 117 125 58 109
node size :
Buffer(256) [117, 125, 58, 109, 38, 229, 7, 189, …]
the rsa.decrypt generated an exception :
TypeError: Cannot read property 'length' of null
I tried several encryptionScheme option (including the default , but always getting the same error or Incorrect key or data

Answer 1

Because of the random padding in OAEP, troubleshooting encryption issues with it can sometimes be a bit tricky.
For further troubleshooting use the following checklist to shoot down potential issues:
Make sure you use the same crypto mechanism on both ends. In your C++ code you are using RSA_PKCS1_OAEP_PADDING but the JavaScript lines in your question does not tell what mechanism you use there.
Make sure that the mechanisms are implemented the same ways in both C++ and Node libraries. It is crucial you have the same hashing method and MGF1 (mask generation function) in both implementations. This is one of the most typical failing points that I've seen in my career.
Since you are working with byte arrays, make sure you are not having any issues in the byte order. In other words, make sure both ends talks the same language in regard of endianness (For self-study: https://www.cs.umd.edu/class/sum2003/cmsc311/Notes/Data/endian.html).