Crash when trying to encrypt a file with Crypto++ RSA Scheme - c++

Question

I have successfully used this some lines ago in my program:
string tmp;
StringSource(msg, true, new PK_EncryptorFilter(*rng, *encryptor, new CryptoPP::HexEncoder(new StringSink(tmp))));
return tmp;
So you know that the Crypto++ objects are well created and so.
Now I want to encrypt a whole binary file and save it to an adjacent file:
FileSource(file.c_str(), true, new PK_EncryptorFilter(*rng, *encryptor, new FileSink((file+".xx").c_str(), true)),true);
But this last line crashes with a debug error stating that abort() has been called.
Hunting down the error, I tried to change the second argument to the FileSource call to false, leading to the following code:
FileSource(file.c_str(), false, new PK_EncryptorFilter(*rng, *encryptor, new FileSink((file+".xx").c_str(), true)),true);
And then the error gone, but the destination file weights 0 bytes, nothing was read/wrote.
I do not know what can can the key to the problem, so, I hope someone can help a little bit.
EDIT: I am using Visual Studio 2013 Pro.
EDIT2: I hunted the error further.
This works and the file binary content is correctly printed on screen:
string s;
FileSource file2("C:\\test.jpg", true, new StringSink(s));
std::cout << s << std::endl;
But this don't work and ends with the mentioned crash.
string s;
FileSource file2("C:\\test.jpg", true, new PK_EncryptorFilter(*rng, *encryptor, new StringSink (s)));
std::cout << s << std::endl;
This is so strange since the same PK_EncryptorFilter filter is used in another method without trouble, as I stated at the beginning of the post.
Anyway, I post here my entire class, so as to get a clear idea of what is going on:
RSASystem::RSASystem()
{
std::string pubkey = "...OMITED...";
rng = new AutoSeededRandomPool;
CryptoPP::HexDecoder decoder;
decoder.Put((byte*)pubkey.c_str(), pubkey.size());
decoder.MessageEnd();
CryptoPP::HexDecoder decoder2;
decoder2.Put((byte*)pubkey.c_str(), pubkey.size());
decoder2.MessageEnd();
verifier = new RSASSA_PKCS1v15_SHA_Verifier;
encryptor = new RSAES_OAEP_SHA_Encryptor;
verifier->AccessKey().Load(decoder);
encryptor->AccessKey().Load(decoder2);
}
string RSASystem::encrypt(string msg)
{
string tmp;
StringSource(msg, true, new PK_EncryptorFilter(*rng, *encryptor, new CryptoPP::HexEncoder(new StringSink(tmp))));
return tmp;
}
void RSASystem::encryptFile(string file)
{
FileSource(file.c_str(), true, new PK_EncryptorFilter(*rng, *encryptor, new FileSink((file+".xx").c_str(), true)),true);
}
EDIT 3: After surrounding the code with try..catch() I got this error:
RSA/OAEP-MGF1(SHA-1): message length of 490986 exceeds the maximum of 214 for this public key
Which now I think can be easily solved.

Answer 1

FileSource(file.c_str(), false,
new PK_EncryptorFilter(*rng, *encryptor,
new FileSink((file+".xx").c_str(), true)
),
true);
This does not look right. new FileSink((file+".xx").c_str() returns a char*, and you need a pointer to a Sink. Plus, there's an extra false in there I'm not used to seeing. Something like:
FileSource fs1(filename, true,
new PK_EncryptorFilter(rng, encryptor,
new FileSink(filename, true)
) // PK_EncryptorFilter
); // StringSource
There's a couple of examples on the Crypto++ wiki. See RSA Cryptography and RSA Encryption Schemes.
The following is an example from the Crypto++ wiki using RSA. But you can use the code for any cryptosystem that adheres to PK_Encryptor and PK_Decryptor (Sources (like StringSource and FileSource) and Sinks (like StringSink and FileSink) are also interchangeable):
////////////////////////////////////////////////
// Generate keys
AutoSeededRandomPool rng;
InvertibleRSAFunction params;
params.GenerateRandomWithKeySize( rng, 1536 );
RSA::PrivateKey privateKey( params );
RSA::PublicKey publicKey( params );
string plain="RSA Encryption", cipher, recovered;
////////////////////////////////////////////////
// Encryption
RSAES_OAEP_SHA_Encryptor e( publicKey );
StringSource ss1( plain, true,
new PK_EncryptorFilter( rng, e,
new StringSink( cipher )
) // PK_EncryptorFilter
); // StringSource
////////////////////////////////////////////////
// Decryption
RSAES_OAEP_SHA_Decryptor d( privateKey );
StringSource ss2( cipher, true,
new PK_DecryptorFilter( rng, d,
new StringSink( recovered )
) // PK_DecryptorFilter
); // StringSource
assert( plain == recovered );
Also, don't use anonymous declarations. Some versions of GCC has problems with them. That is, use:
StringSource ss1( plain, true,
...
rather than:
StringSource( plain, true,
...

Answer 2

I had already pending the encryption and security subject so I wasn't aware of the limitation on the length of the message of the RSA scheme.
https://security.stackexchange.com/questions/44702/whats-the-limit-on-the-size-of-the-data-that-public-key-cryptos-can-handle
So the solution passes by implementing an Integrated or Hybrid Encryption Scheme, like ECIES.
I've done this successfully with Crypto++ using: http://www.cryptopp.com/wiki/Elliptic_Curve_Integrated_Encryption_Scheme
Thanks to jww to point to the correct decision.

Answer 3

OK, I think I know where you might be having problems. But I'd need to see all your code and not just the encryption.
I could coax a BER Decode error by omitting encoder1.MessageEnd and encoder2.MessageEnd. Apparently, I was able to read the key before it was fully written. I assume it was fully written after leaving main (and the destructors ran) because the file sizes looked OK with ls.
In the code below, the message was encrypted under publicKey1 and then decrypted with privateKey2 to ensure the keys were round-tripping.
try {
////////////////////////////////////////////////
// Generate keys
AutoSeededRandomPool rng;
InvertibleRSAFunction params;
params.GenerateRandomWithKeySize(rng, 1024);
RSA::PrivateKey privateKey1(params);
RSA::PublicKey publicKey1(privateKey1);
////////////////////////////////////////////////
// Save/Load keys
HexEncoder encoder1(new FileSink("private-key-der.txt", true));
HexEncoder encoder2(new FileSink("public-key-der.txt", true));
privateKey1.Save(encoder1);
publicKey1.Save(encoder2);
// Must have these. Otherwise, the full key (hex encoded)
// is not written until destructors are run
encoder1.MessageEnd();
encoder2.MessageEnd();
FileSource fs1("private-key-der.txt", true, new HexDecoder);
FileSource fs2("public-key-der.txt", true, new HexDecoder);
RSA::PrivateKey privateKey2;
RSA::PublicKey publicKey2;
privateKey2.Load(fs1);
bool valid = privateKey2.Validate(rng, 3);
if(!valid)
throw Exception(Exception::OTHER_ERROR, "Failed to validate key 1");
publicKey2.Load(fs2);
valid = publicKey2.Validate(rng, 3);
if(!valid)
throw Exception(Exception::OTHER_ERROR, "Failed to validate key 2");
////////////////////////////////////////////////
// Scratch
string plain="RSA Encryption", cipher, recovered;
////////////////////////////////////////////////
// Encryption
RSAES_OAEP_SHA_Encryptor encryptor(publicKey1);
StringSource ss1(plain, true,
new PK_EncryptorFilter(rng, encryptor,
new StringSink(cipher)
) // PK_EncryptorFilter
); // StringSource
////////////////////////////////////////////////
// Decryption
RSAES_OAEP_SHA_Decryptor decryptor(privateKey2);
StringSource ss2(cipher, true,
new PK_DecryptorFilter(rng, decryptor,
new StringSink(recovered)
) // PK_DecryptorFilter
); // StringSource
cout << "Recovered plain text: " << recovered << endl;
} catch (const Exception& ex) {
cerr << ex.what() << endl;
}