I'm trying to implement AES symmetric encryption using the OpenSSL library. In the encryption examples, after calling the EVP_EncryptUpdate function, the EVP_EncryptFinal_ex function is immediately called. What is the purpose of calling the EVP_EncryptFinal_ex function?
This is strangely only explained in the API documentation of EVP_EncryptUpdate():
If padding is enabled (the default) then EVP_EncryptFinal_ex() encrypts the "final" data, that is any data that remains in a partial block. It uses standard block padding (aka PKCS padding) as described in the NOTES section, below. The encrypted final data is written to out which should have sufficient space for one cipher block. The number of bytes written is placed in outl. After this function is called the encryption operation is finished and no further calls to EVP_EncryptUpdate() should be made.
In general, it will perform any final calculations that cannot be performed without knowing that the last part of the message are being encrypted. Most (lower level) cryptographic libraries contain an update / final notion. Note that in case of OpenSSL the authentication tag is not considered part of the ciphertext.
Although it is considered good practice to call EVP_EncryptFinal_ex, the function doesn't really do much for modes that don't require full blocks of plaintext or padding. Most cryptographic libraries - including OpenSSL - try and encrypt the delivered plaintext as soon as possible; you should however not take that for granted.
Related
I'm trying to encrypt a message with a public key and decrypt the cipher with the private key using crypto++ like this in the shell:
openssl rsautl -encrypt -inkey id_rsa.pub.pem -pubin -in message -out message.enc
and
openssl rsautl -decrypt -inkey id_rsa.pem -in message.enc -out message.dec
Encryption/Decryption is done in separate applications. I started with the example from https://www.cryptopp.com/wiki/RSA_Cryptography.
My code:
std::string publicEncrypt(std::string const& plain) {
auto cipher = std::string{};
CryptoPP::RSAES_OAEP_SHA_Encryptor e(getPublicKey());
CryptoPP::StringSource(plain, true,
new CryptoPP::PK_EncryptorFilter(CryptoPP::NullRNG(), e,
new CryptoPP::StringSink(cipher)));
return cipher;
}
std::string privateDecrypt(std::string const& cipher) {
auto decrypted = std::string{};
CryptoPP::RSAES_OAEP_SHA_Decryptor d(getPrivateKey());
CryptoPP::StringSource(cipher, true,
new CryptoPP::PK_DecryptorFilter(CryptoPP::NullRNG(), d,
new CryptoPP::StringSink(decrypted)));
return decrypted;
}
My questions:
Why is a random number generator (RNG) needed for EncryptorFilter/DecryptorFilter?
The RNG has to be the same for encryption/decription, right? So, how to share between processes?
Using the NullRNG() as recommended by https://stackoverflow.com/users/608639/jww in Unable to do RSA Encrption/Decryption using Crypto++ (isValidCoding is false) leads to
std::exception NullRNG: NullRNG should only be passed to functions that don't need to generate random bytes.
I guess I fundamentally miss something. Thanks for hints and advices.
If I use this code in a unit test with a global RNG, everything works fine.
Why is a random number generator (RNG) needed for EncryptorFilter/DecryptorFilter?
The signing and verification classes are abstract interfaces setup in cryptlib.h. Some cryptosystems use them, others do not. A class will specialize and can forgo using a generator. Sometimes a class does not need a generator for one of the operations. NullRNG can be used if not needed.
The reason RSA needs a RNG during public key operations is message padding. Padding is often part of the message formatting function. As #PuzzlePalace pointed out, OAEP padding is randomized and not deterministic.
The reason RSA needs a RNG during private key operations is blinding. For RSA and other RSA-like schemes (like Rabin-Williams), blinding is just a multiplication by a random value to mask the inversion by the priavte key to recover the original value. Later, after signing or decryption, the blinding value is removed and the result of the operation remains.
Related, a reason DSA or ECDSA would not need a RNG during private key operations is RFC 6979, Deterministic Usage of the Digital Signature Algorithm (DSA) and Elliptic Curve Digital Signature Algorithm (ECDSA). Deterministic signatures don't use randomized formatting or randomized k's.
Another reason a RNG is needed for public key and private key operations is validation checks on the key. For example, a key might be checked to ensure a particular constraint holds, like its prime or it has a particular Jacobi symbol.
The RNG has to be the same for encryption/decryption, right? So, how to share between processes?
No, the generators can be different. The only requirements is they produce a random stream for some reasonable definition of what it means to be "random". Without splitting too many hairs, it means the generator produces a uniform distribution.
You can find more reading on Crypto++ generators at RandomNumberGenerator on the wiki.
If I use this code in a unit test with a global RNG, everything works fine.
One quick word of caution... GlobalRNG is part of the Test namespace. It is defined in test.cpp : 115:
NAMESPACE_BEGIN(CryptoPP)
NAMESPACE_BEGIN(Test)
ANONYMOUS_NAMESPACE_BEGIN
OFB_Mode<AES>::Encryption s_globalRNG;
NAMESPACE_END
RandomNumberGenerator & GlobalRNG()
{
return dynamic_cast<RandomNumberGenerator&>(s_globalRNG);
}
NAMESPACE_END // Test
NAMESPACE_END // CryptoPP
GlobalRNG is a deterministic generator and its not part of the library proper. Your code will fail to compile in the field if you depend on it.
Use one of the other generators discussed at RandomNumberGenerator on the wiki. AutoSeededRandomPool is a good choice.
Using the NullRNG() as recommended by https://stackoverflow.com/users/608639/jww in Unable to do RSA
Encrption/Decryption using Crypto++ (isValidCoding is false) leads to
std::exception NullRNG: NullRNG should only be passed to functions that don't need to generate random bytes.
That information is incorrect. I need to fix it. Thanks.
Interestingly (in a morbid sort of way), Crypto++ took CVE-2015-2141 due to blinding in Rabin-Williams. The blinding value needed to be a quadratic residue; otherwise an attacker could prepare special messages to reveal the private key.
The full paper by Evgeny Sidorov is available at Breaking the Rabin-Williams digital signature system implementation in the Crypto++ library. Here's what the new and improved inverse function looks like after fixing Sidorov's attack (from rw.cpp):
ModularArithmetic modn(m_n), modp(m_p), modq(m_q);
Integer r, rInv;
do
{
// Do this in a loop for people using small numbers for testing
r.Randomize(rng, Integer::One(), m_n - Integer::One());
// Fix for CVE-2015-2141. Thanks to Evgeny Sidorov for reporting.
// Squaring to satisfy Jacobi requirements suggested by Jean-Pierre Munch.
r = modn.Square(r);
rInv = modn.MultiplicativeInverse(r);
} while (rInv.IsZero());
If you read Section 6 of Sidorov's paper, he suggests generating a random r, and then checking the Jacobi symbol of r to ensure its a quadratic residue. If it was not a QR, then try a new random r. The triage used the method, but it showed the scheme slowed down considerably because a random r satisfies the condition with probability 1/16.
However, we knew squaring r ensured we satisfied Jacobi on the first try because r2 mod n was always a quadratic residue. The squaring/multiplication only takes log (exp) (not n log (n)), so it turned out to be a significant speedup over trial and error. Before we released the next version of the library, we switched to the squaring method.
I am wondering if there is a way to generate a Cryptographic Nonce using OpenSSL or Crypto++ libraries. Is there anything more to it than just generating a set of random bytes using autoseeded pools?
I am wondering if there is a way to generate a cryptographic nonce using OpenSSL or Crypto++ libraries.
Crypto++:
SecByteBlock nonce(16);
AutoSeededRandomPool prng;
prng.GenerateBlock(nonce, nonce.size());
OpenSSL:
unsigned char nonce[16];
int rc = RAND_bytes(nonce, sizeof(nonce));
unsigned long err = ERR_get_error();
if(rc != 1) {
/* RAND_bytes failed */
/* `err` is valid */
}
/* OK to proceed */
Is there anything more to it than just generating a set of random bytes using autoseeded pools?
A nonce is basically an IV. Its usually considered a public parameter, like an IV or a Salt.
A nonce must be unique within a security context. You may need a nonce to be unpredictable, too.
Uniqueness and unpredictability are two different properties. For example, a counter starting at 0000000000000000 is unique, but its also predictable.
When you need both uniqueness and unpredictability, you can partition the nonce into a random value and a counter. The random value will take up 8 bytes of a 16 byte nonce; while the counter will take up the remaining 8 bytes of a 16 byte nonce. Then you use an increment function to basically perform i++ each time you need a value.
You don't need an 8-8 split. 12-4 works, as does 4-12. It depends on the application and the number of nonces required before rekeying. Rekeying is usually driven by plain text byte counts.
16-0 also works. In this case, you're using random values, avoiding the counter, and avoiding the increment function. (The increment function is basically a cascading add).
NIST SP800-38C and SP800-38D offer a couple of methods for creating nonces because CCM and GCM uses them.
Also see What are the requirements of a nonce? on the Crypto Stack Exchange.
You need a unique number for each nonce. You can use either a serial number or a random number. To help ensure uniqueness, it is common, though not required, to add a timestamp to the nonce. Either passing the timestamp as a separate field or concatenating it with the nonce. Sometimes information such as IP addresses and process IDs are also added.
When you use a serial number, you don't need to worry about skipping numbers. That's fine. Just make sure you never repeat. It must be unique across restarts of your software. This is one place where adding a timestamp can help. Because time-in-millis+serial-number is almost certainly unique across restarts of the server.
For the pseudo random number generator, anyone should be fine. Just make sure that you use a sufficiently large space to make the chances of getting a duplicate effectively impossible. Again, adding time will reduce the likelihood of you getting duplicates as you'll need to get the same random number twice in the same millisecond.
You may wish to hash the nonce to obscure the data in it (eg: process ID) though the hash will only be secure if you include a secure random number in the nonce. Otherwise it may be possible for a viewer of the nonce to guess the components and validate by redoing the hash (ie: they guess the time and try all possible proc IDs).
No. If the nonce is large enough then an autoseeded DRBG (deterministic random bit generator - NIST nomenclature) is just fine. I would suggest a nonce of about 12 bytes. If the nonce needs to be 16 bytes then you can leave the least significant bits - most often the rightmost bytes - set to zero for maximum compatibility.
Just using the cryptographically secure random number generators provided by the API should be fine - they should be seeded using information obtained from the operating system (possibly among other data). It never hurts to add the system time to the seed data just to be sure.
Alternatively you could use a serial number, but that would require you to keep some kind of state which may be hard across invocations. Beware that there are many pitfalls that may allow a clock to repeat itself (daylight saving, OS changes, dead battery etc. etc.).
It never hurts to double check that the random number generator doesn't repeat for a large enough output. There have been issues just with programming or system configuration mistakes, e.g. when a fix after a static code analysis for Debian caused the OpenSSL RNG not to be seeded at all.
Using the aes_cfb_encrypt and aes_cfb_decrypt functions, I have the following questions.
What is unsigned char *iv (Initialization Vector) in an encryption.
Is it required to preserve the *iv for decryption.
Each time i encrypt a block of data the *iv is modified, What i have to do with this modified *iv.
I am encrypting a large file around 100mb, and passing a random *iv for the very first time, do i have to use the same *iv for the rest of the loop, or i have to use the updated *iv from the last call of encrypt block.
Lastly, I am dealing with a structured file, so do i had to use Sizeof(struct) as the length of the buffer or have to use sizeof(struct)*8 as length of the buffer for encryption or decryption.
Please guide..
AES_RETURN aes_cfb_encrypt(const unsigned char *ibuf, unsigned char *obuf, int len, unsigned char *iv, aes_encrypt_ctx cx[1]);
AES_RETURN aes_cfb_decrypt(const unsigned char *ibuf, unsigned char *obuf, int len, unsigned char *iv, aes_encrypt_ctx cx[1]);
In answering your questions, please note the following:
PT(x) = Plain Text representation of 'x'
CT(x) = Cipher Text representation of 'x'
Bn = Logical Data Block 'n' in a sequence of multiple blocks.
1. What is an IV?
IV is short notation for Initialization Vector. It is used in symmetric block-encryption algorithms that perform their encryption in what are called chained or feedback modes. In either, the previous block of encrypted data is used as a piece of functional data "goo" to alter the next block of data to be encrypted. Each successive block of data that is encrypted is fed the prior already-encrypted data block as their blob of goo to use. But what about the first block of plaintext? What does it use for its special sauce? Answer: the IV provided to the function. Pictorially, it looks like the following:
CT(B1) = Encrypt(IV + PT(B1))
CT(B2) = Encrypt(CT(B1) + PT(B2))
CT(B3) = Encrypt(CT(B2) + PT(B3))
...
CT(Bn) = Encrypt(CT(Bn-1) + PT(Bn))
Note: '+' in the above denotes the application of the prior cipher block to the next plaintext block. It is not to be thought of as mathematical-addition. Think of it as "combined with".
The size of the IV must be the same as the block size of the symmetric algorithm being used. Both AES-128-CFB and AES-256-CFB use a 128-bit block size (16 bytes). Therefore, your IV should be 16 bytes of random goo for your purposes in this question, and should be generated on the encryption-side using a secure FIPS-compliant random-source algorithm.
2. Is it required to preserve the IV for decryption?
Yes, but not necessarily in the fashion you may first think. The first IV (provided by you) must be retained somehow. Traditionally, it is sent right where you would think it should be; as the first block of encrypted data. This often freaks people out, they think "But if I send the IV with the data, it isn't as secure, is it?" Think about it this way. How many "IV's" are you sending, anyway? Remember, each data block is encrypted using the prior block of encrypted data as its IV. Therefore, you're actually sending an entire stream of IVs, each encrypted block the IV for the next encrypted block, etc. Where the initial IV is in your output ciphertext is a data-representation question, but where it goes is ultimately irrelevant to the question. It must be preserved. It is possible your API does this for you as part of its output stream (it is not uncommon at all, in fact).
3. Each time I encrypt a block of data the *iv is modified, What i have to do with this modified *iv?
I'm not familiar with the API you're using, but it sounds like you're given the IV to use for the next encryption, which makes perfect sense when you consider how feedback or chaining works for block-mode encryption. You should NOT use the same IV repeatedly. Use the one returned last as the next one. Since your API is modifying the IV in place, it appears the only thing you may need to do is preserve the initial IV somewhere else before sending. I would compare the first ciphertext block against your IV. if they are not the same, you probably need to send your IV, then the cipher text chain in your data stream, and have the receiver aware that the first block is the IV of the decryption.
4. I am encrypting a large file around 100mb, and passing a random *iv for the very first time, do i have to use the same *iv for the rest of the loop, or i have to use the updated *iv from the last call of encrypt block.
See (3). Use the updated IV for each successive block.
5. Lastly, I am dealing with a structured file, so do i Sizeof(struct) as the length of the buffer or have to use sizeof(struct)*8 as length of the buffer for encryption or decryption.?
Use the size of your structure in bytes (not bits). The C/C++ sizeof(yourstruct) should compute this for you, but note if you're encrypting each structure as an independent entity (and not the entire file in one mass), each encryption will carry with it a minimum amount of data added to account for (a) the IV used for that structure, and (b) padding the last block out to an even block boundary, assuming you're using PKCS5 padding. The exact size of an encrypted structure, therefore, would be:
IV + ((sizeof(struct) + 15)/16)*16) bytes.
Again, this is if you're independently encrypting, and storing, each structure as a singular encryption, and again, your API may account for some of this for you.
For more information on symmetric AES, see the AES entry on Wiki. For information on the CFB block cipher mode, see the Block Cipher Modes of Operation article on the same site.
I hope this helps. Do some homework and above all, learn exactly how your API works, which is something I cannot, unfortunately, help you with.
The initialization vector (IV) in a cryptographic system is a random value that is included as part of the encryption system's initialization to ensure that if the same data is encrypted multiple times, it always comes back looking different. This is a requirement of secure cryptographic systems to ensure that an attacker looking at multiple different encrypted messages cannot easily determine whether any two of those messages are the same. Ideally, the IV should be chosen completely randomly.
You should not need to preserve the IV for decryption. Typically, the IV is sent in plaintext along with the encrypted data. That's not a security concern - it's by design.
The IV is changed on each iteration of encryption because internally the cryptographic system is iteratively applying a block cipher to the data, then using the output of that cipher, combined with some extra data, as the new IV for the next application of the block cipher. This process is then iterated as many times as necessary. I suspect (but am not sure) that the IV is handed back to you so that you can encrypt more data where you left off from before. You should definitely double-check this!
As for whether to use the size of your structure or eight times that - I can't say without seeing more of your code. However, you should probably be providing the total number of bytes to encrypt, so if you're encrypting eight copies of the struct, pass in eight times the sizeof the struct.
Hope this helps!
I have an application that uses an opensource "libgcrypt" to encrypt/decrypt a data block (32 bytes). Now I am going to use Microsoft CryptAPI to replace it. My problem is that the libgcrypt and cryptApi approaches generate different ciphertext contents as I use the same AES-256 algoritjm in CFB mode, same key, and same IV, although the ciphertext can be decrypted by their own correspndingly.
Could some tell me what is the problem? Thanks.
Do the two assume different endianness, or assign the bytes in the key/IV in different orders?
If the endianness assumptions are different, you may need to re-order the bytes in the key, IV and/or plaintext to get matching results. For example, if you are supplying bytes in the order abcdefgh, you may need to switch this to 'dcbahgfe' to get things to work.
There is an additional parameter for CFB, namely the "shift amount" at each iteration. The Wikipedia page on CFB has some information. Namely, you encrypt x bits for every block encryption, where x is any value between 1 and the block size (128 for AES). I suspect that in your code, the Microsoft CryptoAPI and libgcrypt do not use the same value for x.
As explained in the documentation for CryptSetKeyParam(), Windows defaults to x=8 (i.e. one byte at a time). This is the KP_MODE_BITS parameter. On the other hand, libgcrypt defaults to x=n for a n-bit block cipher (i.e. x=128 for AES). I am not sure libgcrypt can be convinced to use another value.
i think the problem is with block size .as you said you are using 32 byte as block size make sure if block size of both are same and supports as well .because some of library block size is fixed for Aes as 16 byte .
What is the length of your key and IV?
Are ciphertexts different if the length of opentext is exactly 256 bit?
I have same problem, but with a different library. I noticed one thing in this library; If I pass input byte less than 32 bytes, in that case it's showing me both are the same encrypted data.
Is that what's happening in your case? If so, it means the problem is with the padding mechanism.
Okay so i have a packed a proprietary binary format. That is basically a loose packing of several different raster datasets. Anyways in the past just reading this and unpacking was an easy task. But now in the next version the raster xml data is now to be encrypted using AES-256(Not my choice nor do we have a choice).
Now we basically were sent the AES Key along with the SALT they are using so we can modify our unpackager.
NOTE THESE ARE NOT THE KEYS JUST AN EXAMPLE:
They are each 63 byte long ASCII characters:
Key: "QS;x||COdn'YQ#vs-`X\/xf}6T7Fe)[qnr^U*HkLv(yF~n~E23DwA5^#-YK|]v."
Salt: "|$-3C]IWo%g6,!K~FvL0Fy`1s&N<|1fg24Eg#{)lO=o;xXY6o%ux42AvB][j#/&"
We basically want to use the C++ CryptoAPI to decrypt this(I also am the only programmer here this week, and this is going live tomorrow. Not our fault). I've looked around for a simple tutorial of implementing this. Unfortunately i cannot even find a tutorial where they have both the salt and key separately. Basically all i have really right now is a small function that takes in an array of BYTE. Along with its length. How can i do this?
I've spent most of the morning trying to make heads/tails of the cryptoAPI. But its not going well period :(
EDIT
So i asked for how they encrypt it. They use C#, and use RijndaelManaged, which from my knowledge is not equivalent to AES.
EDIT2
Okay finally got exactly what was going on, and they sent us the wrong keys.
They are doing the following:
Padding = PKCS7
CipherMode = CBC
The Key is defined as a set of 32 Bytes in hex.
The IV is defined as a set of 32 bytes in hex too.
They took away the salt when i asked them.
How hard is it to set these things in CryptoAPI using the wincrypt.h header file.?
AES-256 uses 256 bit keys. Ideally, each key in your system should be equally likely. A 63 byte string would be 504 bits. You first need to figure out how the string of 63 characters needs to be converted to 256 bits (The sample ones you gave are not base64 encoded). Next, "salt" isn't an intrinsic part of AES. You might be referring to either an initialization vector (IV) in Cipher-Block-Chaining mode or you could be referring to somehow updating the key.
If I were to guess, I'm assuming that by "SALT" you mean IV and specifically CBC mode.
You will need to know all of this when using CAPI functions (e.g. decrypt).
If all of this sounds confusing, then it might be best to change your design so that you don't have to worry about getting all of this right. Crypto is hard. One bad step could invalidate all the security. Consider looking at this comment on my Stick Figure Guide to AES.
UPDATE: You can look at this for a rough starting point for C++ CAPI. You'll need a 64 character hex string to get 256 bits ( 256 bits / (4 bits / char) == 64 chars). You can convert the chars to bits yourself.
Again, I must caution that playing fast and loose with IV's and keys can have disastrous consequences. I've studied AES/Rijndael in depth down to the math and gate level and have even written my own implementation. However, in my production code, I stick to using a well-tested TLS implementation if at all possible for data in transit. Even for data at rest, it'd be better to use a higher level library.
Rijndael is the algorithm name for AES