Unable to do RSA Encryption/Decryption using Crypto++ (isValidCoding is false) - c++

Question

I am using Crypto++ to encrypt an array of bytes using RSA. I have followed Crypto++ wiki's samples with no luck getting them to work. Encryption and Decryption in all the samples are done within a single process but I am trying to decrypt the content which is already encrypted in another process.
Here is my code:
class FixedRNG : public CryptoPP::RandomNumberGenerator
{
public:
FixedRNG(CryptoPP::BufferedTransformation &source) : m_source(source) {}
void GenerateBlock(byte *output, size_t size)
{
m_source.Get(output, size);
}
private:
CryptoPP::BufferedTransformation &m_source;
};
uint16_t Encrypt()
{
byte *oaepSeed = new byte[2048];
for (int i = 0; i < 2048; i++)
{
oaepSeed[i] = (byte)i;
}
CryptoPP::ByteQueue bq;
bq.Put(oaepSeed, 2048);
FixedRNG prng(bq);
Integer n("Value of N"),
e("11H"),
d("Value of D");
RSA::PrivateKey privKey;
privKey.Initialize(n, e, d);
RSA::PublicKey pubKey(privKey);
CryptoPP::RSAES_OAEP_SHA_Encryptor encryptor( pubKey );
assert( 0 != encryptor.FixedMaxPlaintextLength() );
byte blockSize = encryptor.FixedMaxPlaintextLength();
int divisionCount = fileSize / blockSize;
int proccessedBytes = 0;
// Create cipher text space
uint16_t cipherSize = encryptor.CiphertextLength( blockSize );
assert( 0 != cipherSize );
encryptor.Encrypt(prng, (byte*)plaintext, blockSize, (byte*)output);
return cipherSize;
}
void Decrypt(uint16_t cipherSize)
{
byte *oaepSeed = new byte[2048];
for (int i = 0; i < 2048; i++)
{
oaepSeed[i] = (byte)i;
}
CryptoPP::ByteQueue bq;
bq.Put(oaepSeed, 2048);
FixedRNG prng(bq);
Integer n("Value of N"),
e("11H"),
d("Value of D");
RSA::PrivateKey privKey;
privKey.Initialize(n, e, d);
//RSA::PublicKey pubKey(privKey);
CryptoPP::RSAES_OAEP_SHA_Decryptor decryptor( privKey );
byte blockSize = decryptor.FixedMaxPlaintextLength();
assert(blockSize != 0);
size_t maxPlainTextSize = decryptor.MaxPlaintextLength( cipherSize );
assert( 0 != maxPlainTextSize );
void* subBuffer = malloc(maxPlainTextSize);
CryptoPP::DecodingResult result = decryptor.Decrypt(prng, (byte*)cipherText, cipherSize, (byte*)subBuffer);
assert( result.isValidCoding );
assert( result.messageLength <= maxPlainTextSize );
}
Unfortunately, value of isValidCoding is false. I think I am misunderstanding something about RSA encryption/decryption!!
Note that, privKey and pubKey have been validated using KEY.Validate(prng, 3).
I have also tried to use RAW RSA instead of OAEP and SHA with no luck. I have tried to debug through crypto++ code, what I am suspicious about is prng variable. I think there is something wrong with it. I have also used AutoSeededRandomPool instead of FixedRNG but it didn't help. Worth to know that, if I copy the decryption code right after encryption code and execute it in Encrypt() method, everything is fine and isValidCoding is true!!

Answer 1

This is probably not be correct:
byte blockSize = encryptor.FixedMaxPlaintextLength();
...
encryptor.Encrypt(prng, (byte*)plaintext, blockSize, (byte*)output);
return cipherSize;
Try:
size_t maxLength = encryptor.FixedMaxPlaintextLength();
size_t cipherLength = encryptor.CiphertextLength( blockSize );
...
SecureByteBlock secBlock(cipherLength);
cipherLength = encryptor.Encrypt(prng, (byte*)plaintext, blockSize, secBlock);
secBlock.resize(cipherLength);
FixedMaxPlaintextLength returns a size_t, not a byte.
You should probably be calling CiphertextLength on plaintext.
I'm not really sure how you are just returning an uint_t from encrypt().
You might do better by starting fresh, and using an example from the Crypto++ as a starting point. I'm not sure this design is worth pursuing.
If you start over, then Shoup's Elliptic Curve Integrated Encryption Scheme (ECIES) would be a good choice since it combines public key with symmetric ciphers and authentication tags.