When using Botan encryption with botansqlite3, what are the optimal configuration settings for performance?
OR
How can I configure Botansqlite3 to use CAST5?
I am currently using AES and it is too slow. My use case is a game.
I am looking for weak or moderate encryption to protect my game's data (not end user data) so security is less of a consideration than performance.
Here is my current BotanSqlite3 codec.h
/*These constants can be used to tweak the codec behavior as follows */
//BLOCK_CIPHER_STR: Cipher and mode used for encrypting the database
//make sure to add "/NoPadding" for modes that use padding schemes
const string BLOCK_CIPHER_STR = "Twofish/XTS";
//PBKDF_STR: Key derivation function used to derive both the encryption
//and IV derivation keys from the given database passphrase
const string PBKDF_STR = "PBKDF2(SHA-160)";
//SALT_STR: Hard coded salt used to derive the key from the passphrase.
const string SALT_STR = "&g#nB'9]";
//SALT_SIZE: Size of the salt in bytes (as given in SALT_STR)
const int SALT_SIZE = 64/8; //64 bit, 8 byte salt
//MAC_STR: CMAC used to derive the IV that is used for db page
//encryption
const string MAC_STR = "CMAC(Twofish)";
//PBKDF_ITERATIONS: Number of hash iterations used in the key derivation
//process.
const int PBKDF_ITERATIONS = 10000;
//KEY_SIZE: Size of the encryption key. Note that XTS splits the key
//between two ciphers, so if you're using XTS, double the intended key
//size. (ie, "AES-128/XTS" should have a 256 bit KEY_SIZE)
const int KEY_SIZE = 512/8; //512 bit, 64 byte key. (256 bit XTS key)
//IV_DERIVATION_KEY_SIZE: Size of the key used with the CMAC (MAC_STR)
//above.
const int IV_DERIVATION_KEY_SIZE = 256/8; //256 bit, 32 byte key
//This is definited in sqlite.h and very unlikely to change
#define SQLITE_MAX_PAGE_SIZE 32768
I believe that I need to find replacements for BLOCK_CIPHER_STR, PBKDF_STR, MAC_STR, KEY_SIZE and IV_DERIVATION_KEY_SIZE to reconfigure BotanSqlite3 to use a different codec.
I found a extensive comparison test of Botan codec performance here:
http://panthema.net/2008/0714-cryptography-speedtest-comparison/crypto-speedtest-0.1/results/cpu-sidebyside-comparison-3x2.pdf#page=5
However, the testing was done with Botan directly, not botansqlite3 as I intend to use it. Looking at the charts, a good candidate appears to be CAST5 from a performance perspective.
The database in question is 300KB, mostly INTEGER fields with some text blobs.
I am configuring Botan as suggested by OlivierJG of botansqlite3 fame, using the amalgamation
'./configure.py --no-autoload --enable-modules=twofish,xts,pbkdf2,cmac,sha1 --gen-amalgamation --cc=msvc --os=win32 --cpu=x86 --disable-shared --disable-asm'
References:
http://github.com/OlivierJG/botansqlite3 - botansqlite3 is an encryption codec for SQLite3 that can use any algorithms in Botan for encryption
http://www.sqlite.org - sqlite3 is a cross-platform SQL database
http://botan.randombit.net/ - botan is a C++ encryption library with support for a number of codecs
You can get CAST-128 (or as I was calling it, CAST5) to work, it is a block cipher.
The best bet is the above with different configuration of key size.
Twofish is pretty fast.
Thank you to 'Olivier JG' for all the excellent code.
Related
I am implementing Azure DPS (device provisioning service) for my ESP32-based firmware.
The bash script I use so far is as follows (where KEY is the primary key of the DPS enrolment group and REG_ID is the registration device Id for the given ESP it runs on):
#!/bin/sh
KEY=KKKKKKKKK
REG_ID=RRRRRRRRRRR
keybytes=$(echo $KEY | base64 --decode | xxd -p -u -c 1000)
echo -n $REG_ID | openssl sha256 -mac HMAC -macopt hexkey:$keybytes -binary | base64
I use the Arduino platform in platformIO.
How to translate the script in C/C++?
[UPDATE] The reason why I can't run openSSL: I need to generate the symmetric key from the actual device MAC address in order to obtain the credential from DPS and then be granted to connect to IoT Hub - I run on an EPS32-based custom PCB. No shell. No OS.
I manage to do it by using bed library (which is available from both ESP32/Arduino platforms).
Here is my implementation for the Arduino platform:
#include <mbedtls/md.h> // mbed tls lib used to sign SHA-256
#include <base64.hpp> // Densaugeo Base64 version 1.2.0 or 1.2.1
/// Returns the SHA-256 signature of [dataToSign] with the key [enrollmentPrimaryKey]
/// params[in]: dataToSign The data to sign (for our purpose, it is the registration ID (or the device ID if it is different)
/// params[in]: enrollmentPrimaryKey The group enrollment primary key.
/// returns The SHA-256 base-64 signature to present to DPS.
/// Note: I use mbed to SHA-256 sign.
String Sha256Sign(String dataToSign, String enrollmentPrimaryKey){
/// Length of the dataToSign string
const unsigned dataToSignLength = dataToSign.length();
/// Buffer to hold the dataToSign as a char[] buffer from String.
char dataToSignChar[dataToSignLength + 1];
/// String to c-style string (char[])
dataToSign.toCharArray(dataToSignChar, dataToSignLength + 1);
/// The binary decoded key (from the base 64 definition)
unsigned char decodedPSK[32];
/// Encrypted binary signature
unsigned char encryptedSignature[32];
/// Base 64 encoded signature
unsigned char encodedSignature[100];
Serial.printf("Sha256Sign(): Registration Id to sign is: (%d bytes) %s\n", dataToSignLength, dataToSignChar);
Serial.printf("Sha256Sign(): DPS group enrollment primary key is: (%d bytes) %s\n", enrollmentPrimaryKey.length(), enrollmentPrimaryKey.c_str());
// Need to base64 decode the Preshared key and the length
const unsigned base64DecodedDeviceLength = decode_base64((unsigned char*)enrollmentPrimaryKey.c_str(), decodedPSK);
Serial.printf("Sha256Sign(): Decoded primary key is: (%d bytes) ", base64DecodedDeviceLength);
for(int i= 0; i<base64DecodedDeviceLength; i++) {
Serial.printf("%02x ", (int)decodedPSK[i]);
}
Serial.println();
// Use mbed to sign
mbedtls_md_type_t mdType = MBEDTLS_MD_SHA256;
mbedtls_md_context_t hmacKeyContext;
mbedtls_md_init(&hmacKeyContext);
mbedtls_md_setup(&hmacKeyContext, mbedtls_md_info_from_type(mdType), 1);
mbedtls_md_hmac_starts(&hmacKeyContext, (const unsigned char *) decodedPSK, base64DecodedDeviceLength);
mbedtls_md_hmac_update(&hmacKeyContext, (const unsigned char *) dataToSignChar, dataToSignLength);
mbedtls_md_hmac_finish(&hmacKeyContext, encryptedSignature);
mbedtls_md_free(&hmacKeyContext);
Serial.print("Sha256Sign(): Computed hash is: ");
for(int i= 0; i<sizeof(encryptedSignature); i++) {
Serial.printf("%02x ", (int)encryptedSignature[i]);
}
Serial.println();
// base64 decode the HMAC to a char
encode_base64(encryptedSignature, sizeof(encryptedSignature), encodedSignature);
Serial.printf("Sha256Sign(): Computed hash as base64: %s\n", encodedSignature);
// creating the real SAS Token
return String((char*)encodedSignature);
}
You have a very interesting question from mathematical/algorithmical point of view. So just for fun decided to implement ALL sub-algorithms of it from scratch, without almost NO dependacy on standard C++ library.
All algorithms of me are based on Wikipedia and described well in its articles SHA-256, HMAC, Base64 (and StackOverflow), Hex.
I made whole my code specifically from scratch and without almost NO dependency on std C++ library. Only two headers used right now <cstdint> for implementing all sized integers u8, u16, u32, i32, u64, i64.
And <string> is used only to implement Heap allocations. Also you can easily implement this heap allocations inside my HeapMem class, or by removing using String = std::string; (and #include <string>) on first lines of my code and using built-in heap-allocated String of Arduino if it has built-in one.
Header <iostream> is used only in few last lines of code snippet, only to output result to console, so that StackOverflow visitors my run program without external dependencies. This console output may be removed of course.
Besides main algorithms I had to implement my own classes Array, Vector, Str, Tuple, HeapMem to re-implement basic concepts of standard C++ library. Also standard library function like MemSet(), MemCpy(), MemCmp(), StrLen(), Move() had to be implemented.
You may notice too that I never used exceptions in code, specifically if you have disabled/non-supporting them. Instead of exceptions I implemented special Result<T> template that resembles Result from Rust language. This template is used to return/check correct and error results from whole stack of functions.
All algorithms (Sha256, Hmac, Base64) are tested by simple test cases with reference vectors taken from internet. Final SignSha256() function that you desired is also tested by several test cases against your reference bash OpenSSL script.
Important!. Don't use this code snippet directly inside production code, because it is not very well tested and might contain some errors. Use it Only for educational purposes or test it thourughly before using.
Code snippet is very large, around 32 KiB, bigger that limit of StackOverflow post size (which is 30 000 symbols), so I'm sharing code snippet through two external services - GodBolt (click Try it online! link), where you can also test it online, and GitHub Gist service for download/view only.
SOURCE CODE HERE
Try it online on GodBolt!
GitHub Gist
I've generated a random 256 bit symmetric key, in a file, to use for encrypting some data using the OpenSSL command line which I need to decrypt later programmatically using the OpenSSL library. I'm not having success, and I think the problem might be in the initialization vector I'm using (or not using).
I encrypt the data using this command:
/usr/bin/openssl enc -aes-256-cbc -salt -in input_filename -out output_filename -pass file:keyfile
I'm using the following call to initialize the decrypting of the data:
EVP_DecryptInit_ex(ctx, EVP_aes_256_cbc(), nullptr, keyfile.data(), nullptr))
keyfile is a vector<unsigned char> that holds the 32 bytes of the key. My question is regarding that last parameter. It's supposed to be an initialization vector to the cipher algorithm. I didn't specify an IV when encrypting, so some default must have been used.
Does passing nullptr for that parameter mean "use the default"? Is the default null, and nothing is added to the first cipher block?
I should mention that I'm able to decrypt from the command line without supplying an IV.
What is the default IV when encrypting with EVP_aes_256_cbc() [sic] cipher...
Does passing nullptr for that parameter mean "use the default"? Is the default null, and nothing is added to the first cipher block?
There is none. You have to supply it. For completeness, the IV should be non-predictable.
Non-Predictable is slightly different than both Unique and Random. For example, SSLv3 used to use the last block of ciphertext for the next block's IV. It was Unique, but it was neither Random nor Non-Predictable, and it made SSLv3 vulnerable to chosen plaintext attacks.
Other libraries do clever things like provide a null vector (a string of 0's). Their attackers thank them for it. Also see Why is using a Non-Random IV with CBC Mode a vulnerability? on Stack Overflow and Is AES in CBC mode secure if a known and/or fixed IV is used? on Crypto.SE.
/usr/bin/openssl enc -aes-256-cbc...
I should mention that I'm able to decrypt from the command line without supplying an IV.
OpenSSL uses an internal mashup/key derivation function which takes the password, and derives a key and iv. Its called EVP_BytesToKey, and you can read about it in the man pages. The man pages also say:
If the total key and IV length is less than the digest length and MD5 is used then the derivation algorithm is compatible with PKCS#5 v1.5 otherwise a non standard extension is used to derive the extra data.
There are plenty of examples of EVP_BytesToKey once you know what to look for. Openssl password to key is one in C. How to decrypt file in Java encrypted with openssl command using AES in one in Java.
EVP_DecryptInit_ex(ctx, EVP_aes_256_cbc(), nullptr, keyfile.data(), nullptr))
I didn't specify an IV when encrypting, so some default must have been used.
Check your return values. A call should have failed somewhere along the path. Maybe not at EVP_DecryptInit_ex, but surely before EVP_DecryptFinal.
If its not failing, then please file a bug report.
EVP_DecryptInit_ex is an interface to the AES decryption primitive. That is just one piece of what you need to decrypt the OpenSSL encryption format. The OpenSSL encryption format is not well documented, but you can work it backwards from the code and some of the docs. The key and IV computation is explained in the EVP_BytesToKey documentation:
The key and IV is derived by concatenating D_1, D_2, etc until enough
data is available for the key and IV. D_i is defined as:
D_i = HASH^count(D_(i-1) || data || salt)
where || denotes concatentaion, D_0 is empty, HASH is the digest
algorithm in use, HASH^1(data) is simply HASH(data), HASH^2(data) is
HASH(HASH(data)) and so on.
The initial bytes are used for the key and the subsequent bytes for the
IV.
"HASH" here is MD5. In practice, this means you compute hashes like this:
Hash0 = ''
Hash1 = MD5(Hash0 + Password + Salt)
Hash2 = MD5(Hash1 + Password + Salt)
Hash3 = MD5(Hash2 + Password + Salt)
...
Then you pull of the bytes you need for the key, and then pull the bytes you need for the IV. For AES-128 that means Hash1 is the key and Hash2 is the IV. For AES-256, the key is Hash1+Hash2 (concatenated, not added) and Hash3 is the IV.
You need to strip off the leading Salted___ header, then use the salt to compute the key and IV. Then you'll have the pieces to feed into EVP_DecryptInit_ex.
Since you're doing this in C++, though, you can probably just dig through the enc code and reuse it (after verifying its license is compatible with your use).
Note that the OpenSSL IV is randomly generated, since it's the output of a hashing process involving a random salt. The security of the first block doesn't depend on the IV being random per se; it just requires that a particular IV+Key pair never be repeated. The OpenSSL process ensures that as long as the random salt is never repeated.
It is possible that using MD5 this way entangles the key and IV in a way that leaks information, but I've never seen an analysis that claims that. If you have to use the OpenSSL format, I wouldn't have any hesitations over its IV generation. The big problems with the OpenSSL format is that it's fast to brute force (4 rounds of MD5 is not enough stretching) and it lacks any authentication.
Okay so i'm looking for a way to encrypt/decrypt larger files using RSA and AES.
I don't quite understand what I need to do.
The scenario is that i've got larger files (anywhere from 200kb - 50mb). I want to be able to encrypt specific files leaving a key (private key) in the current directory as well as the encrypted file. The user can then save the key, take it with them and come back to decrypt the file at a later time.
I just don't quite understand how to use AES/RSA together to achieve this. I have some code to do simple RSA encryption/decryption and some working AES code. I got this code from other SO questions.
I'm using Openssl with C++.
Current AES program: (from online)
int main() {
int bytes_read, bytes_written;
unsigned char indata[AES_BLOCK_SIZE];
unsigned char outdata[AES_BLOCK_SIZE];
/* ckey and ivec are the two 128-bits keys necesary to
en- and recrypt your data. Note that ckey can be
192 or 256 bits as well */
unsigned char ckey[] = "thiskeyisverybad";
unsigned char ivec[] = "dontusethisinput";
/* data structure that contains the key itself */
AES_KEY key;
/* set the encryption key */
AES_set_encrypt_key(ckey, 128, &key);
/* set where on the 128 bit encrypted block to begin encryption*/
int num = 0;
FILE *ifp = fopen("out.txt", "rb");
FILE *ofp = fopen("outORIG.txt", "wb");
while (true) {
bytes_read = fread(indata, 1, AES_BLOCK_SIZE, ifp);
AES_cfb128_encrypt(indata, outdata, bytes_read, &key, ivec, &num,
AES_DECRYPT); //or AES_DECRYPT
bytes_written = fwrite(outdata, 1, bytes_read, ofp);
if (bytes_read < AES_BLOCK_SIZE)
break;
}
Okay so i'm looking for a way to encrypt/decrypt larger files using RSA and AES. I don't quite understand what I need to do...
All you need to do is:
Generate random AES key
Encrypt large file under AES key
Encrypt AES key under RSA key
Also, encryption alone is usually not enough. That means your choice of AES/CFB could be improved. That's because CFB (and other modes like CBC) provide confidentiality only. You cannot detect accidental and malicious tampering.
To improve upon the mode, you should select a mode that provides confidentiality and authenticity. AES/GCM would be a good choice. There's an example of it on the OpenSSL wiki at EVP Authenticated Encryption and Decryption.
You can read more about Authenticated Encryption on the Crypto++ wiki at Authenticated Encryption. Its a different library and different wiki, but it provides the information on authenticated encryption.
In an ideal world, OpenSSL would provide an Integrated Encryption Scheme like Shoup's Elliptic Curve Integrated Encryption Scheme (ECIES) or Abdalla, Bellare and Rogaway's Diffie-Hellman Authenticated Encryption Scheme (DHAES). An integrated encryption schemes does it all for you.
By the way, Crypto++ is a C++ crypto library that provides both integrated encryption schemes. Maybe you should consider switching security libraries. Here's the documentation with sample code on ECIES.
So I am using the Crypto++ Library to encrypt a file. I need to save the key and iv for future use. I am following this tutorial. Here is my function :
void AESUtil::encrypt(string filename,bool savekeys,string savefilename){
AutoSeededRandomPool rnd;
// Generate a random key
byte key[AES::DEFAULT_KEYLENGTH];
rnd.GenerateBlock(key, AES::DEFAULT_KEYLENGTH);
// Generate a random IV
byte iv[AES::BLOCKSIZE];
rnd.GenerateBlock(iv, AES::BLOCKSIZE);
Binary b;
string plaintext = b.decoder(filename);
unsigned char *ciphertext= new unsigned char[plaintext.size()+1];
ciphertext[plaintext.size()]='\0';
if(savekeys){
ofstream("key.bin", ios::binary).write((char*)key, sizeof(key));
}
CFB_Mode<AES>::Encryption cfbEncryption(key, AES::DEFAULT_KEYLENGTH, iv);
cfbEncryption.ProcessData(ciphertext,reinterpret_cast<const unsigned char*>(plaintext.c_str()),plaintext.size()+1);
ofstream outfile(savefilename.c_str());
outfile.write((char*)ciphertext,sizeof(ciphertext));
}
The files contain data in �/���� format. I want to know the best method to save the key and iv programmatically which are a byte array to a file and the ciphertext which is a unsigned char* to a separate file.
The key could be saved in a separate file. Normally the key is established between sender / receiver in advance, or it is encrypted using a public key of the receiver. Note that it doesn't make sense to save the key next to the ciphertext, as it would provide no protection whatsoever. The handling of keys is called key management and entire books have been written about it.
The IV is a different animal. The IV should be randomly generated. For CFB it should at least be unique and identical at both sides. Usually the IV is simply prefixed to the ciphertext, it doesn't have to be kept secret.
Your key and iv variables are the key and IV used to encrypt the plain text.
You didn't fill either; you're actually using an array filled with 0 bytes as both the key and IV for your encryption.
The IV is public information. You don't need to hide it. Save it the way you want.
The KEY is what you must keep safe. To do that you may decide how much effort you want to put on it to hide it from the external factors.
I have some keys that I don't care to leave them as a "plain text" in the binary code. (NO SECURITY, but my mom can't figure out what to do, but a beginner in reverse engineer will laugh at it.)
Some keys I do a play with the bytes, like inverting parts, separating them, XOR with something. (Very unsafe, but better than nothing, a programmer with decent knowledge in reverse engineering will be able to spend some time and eventually break the security)
Some other cases I use 3rd party advanced obfuscation... If possible, depending on what you want, you may even replace your encryption engine with some "white-box" cryptography. Then you will have your keys very well protected. But this is usually hard/expensive. It doesn't seem to be your case. (Yes, even a very advanced assembly guru will not be happy to start reverse engineer this case.)
Another solution, if you don't need the key on your binary, is to give it to the system's password manager. On Windows, it's called "Data Protection API", and on Mac, it's called "Keychain". Take a look at these, and then you will understand why this is considered security. But it's because all the passwords here are encrypted with the "user password" so nothing is stored "on disk". A turned-off device in this scenario is considered very secure.
Resolved (look for my answer). I need help encrypting a password with RSA in C++. I'm working for Blackberry10 in C++ and I've been looking for a RSA tutorial or something that walk me through the encrypting process but sadly I haven't found anything consistant.
I've been told that the process consist basicaly in three steps:
1)Generate the RSA public key with the module and exponent.
2)Encrypt the text with the public key.
3)Encrypt the text again in base640.
If you are wondering why I need to follow this three steps is because I'm working in a banking app, and those are their requirements.
I've been searching all over and I haven't found even how to make the first step, generate the public key with the module and the exponent (I already have both).
Any help will be appreciated.
Thanks.
Is not that I'm building the whole app by myself. We are a team and I need to encrypt just the client password with RSA and send it to the next step of the process. I do have experiencie working with Blackberry and RSA but in Java, where the process is a little bit easier, e.g: In java the first step once you got the module and public exponent is create the RSA Public Key Object through a very simple syntaxis: RSAPublicKey publicKey = new RSAPublicKey(new RSACryptoSystem(2048),ebytes, mbytes);. In this part is where I'm little bit lost because I haven't found the proper BB10 documentation. I'm sorry if I created the illusion that I'm creating a whole encryption API by myself. BB10 has this security API based in hursa.h (https://developer.blackberry.com/native/beta/reference/com.qnx.doc.crypto/topic/about_rsa_raw.html) I'm trying to implement it, but I haven't been successful. #owlstead #dajames #bta
I highly recommend that you take advantage of an existing encryption library to handle all of this for you. OpenSSL is widely used, and the LibTom libraries include an encryption lib as well. Encryption is not an easy thing to implement from scratch, and you will save a considerable amount of time and frustration by using an existing implementation. OpenSSL in particular is a good choice, as it has passed FIPS certification tests multiple times. Since you're working on a banking app, you (and your customers) will most likely want to use an implementation that has been certified.
Even if you insist on implementing your own encryption libraries from scratch, I encourage you to check out the aforementioned libraries as examples.
It seems to me that you know even less than you think you do about how this cryptography works.
One doesn't usually use RSA encryption to encrypt data. It can be done for something short like a PIN or password, but what is usually done is to generate a symmetric key and to encrypt the data with the symmetric key and then use RSA to encrypt the symmetric key.
If you are sending a password to a bank then what you are probably supposed to be doing is to use the RSA key supplied by the bank in their own key certificate. Only the bank has the private key, so only they can decrypt the password. If that's right then you don't need to generate an RSA key, but you do need to verify that the certificate is trusted.
I don't program Blackberries, so I don't know what cryptography APIs they support, but I would expect everything you need to be built-in. This is all pretty standard stuff.
I recommend that you start by reading up on public key cryptography (e.g. on Wikipedia here and here) before starting to design your solution.
Resolved. After doing some research and getting to know better with BB10 coding and RSA, I finally came out with the solution to successfuly encrypt a plain text with RSA and Base64 in C/C++ for BB10 or any other platform. Please take in consideration that I have the modulus and the public exponent of the rsa object from the services that I'm working with.
The code:
QByteArray answer;
RSA* rsa = RSA_new();
BIGNUM *modulus = BN_new();
BIGNUM *exponent = BN_new();
const char *modulusString = rsaObj->getM(); //My Modulus
const char *exponentString = rsaObj->getE(); //My exponent
BN_hex2bn(&modulus, modulusString);
BN_hex2bn(&exponent, exponentString);
rsa->n = BN_new();
BN_copy(rsa->n, modulus);
rsa->e = BN_new();
BN_copy(rsa->e, exponent);
int maxSize = RSA_size(rsa);
qDebug() << "maxSize:" << maxSize;
const char *inn = "1234";
unsigned char *encrypted = (unsigned char*) malloc(maxSize);
int bufferSize = RSA_public_encrypt(strlen(inn), (unsigned char *) inn,
encrypted, rsa, RSA_PKCS1_PADDING);
if (bufferSize == -1) {
RSA_free(rsa);
qDebug() << "Error";
}
QByteArray enc = QByteArray::fromRawData((const char*) encrypted, 256);
answer = enc.toBase64();
return answer;
Thanks and I expect this helps to the new BB10 developers