In my application I need to use encryption algorithm that allows me to decrypt single byte at requested offset in encrypted buffer, without reading surrounding blocks. My choice is AES with CTR mode using Crypto++ library. Since I couldn't find any good example, I have wrote it on my own:
unique_ptr<vector<byte>> GetIV(int counter)
{
byte* counterPtr = (byte*)&counter;
unique_ptr<vector<byte>> iv(new vector<byte>());
for (int j = 0; j < 4; j++)
{
iv->push_back(counterPtr[j]);
}
return move(iv);
}
unique_ptr<vector<uint8_t>> Encrypt(const vector<uint8_t>& plainInput)
{
unique_ptr<vector<uint8_t>> encryptedOutput(new vector<uint8_t>(plainInput.size()));
for (int i = 0; i < plainInput.size(); i++)
{
auto iv = GetIV(i);
CTR_Mode<AES>::Encryption encryptor(_key->data(), _key->size(), iv->data());
byte encryptedValue = encryptor.ProcessByte(plainInput.at(i));
encryptedOutput->at(i) = encryptedValue;
}
return move(encryptedOutput);
}
unique_ptr<vector<uint8_t>> Decrypt(const vector<uint8_t>& encryptedInput, int position)
{
unique_ptr<vector<uint8_t>> decryptedOutput(new vector<uint8_t>(encryptedInput.size()));
for (int i = 0; i < encryptedInput.size(); i++)
{
auto iv = GetIV(position + i);
CTR_Mode<AES>::Decryption decryptor(_key->data(), _key->size(), iv->data());
byte decryptedValue = decryptor.ProcessByte(encryptedInput.at(i));
decryptedOutput->at(i) = decryptedValue;
}
return move(decryptedOutput);
}
As you can see, I iterate through all bytes in my input buffer, and encrypt\decrypt each of them separately, because it is necessary to have unique counter for each block (in CTR mode). Since I need to be able to decrypt random byte, I need to have as much blocks as buffer size is, is that correct? My solution works, but it is very very slow... Am I doing it right? Or maybe there is much more efficient way to do this?
There are several major problems with your code:
You are using unauthenticated encryption, which is insecure in most application domains. Please use AES-GCM instead, which looks a lot like AES-CTR anyway. This is in fact mentioned right on the documentation of Crypto++.
The IV of CTR mode is 16 bytes long, yet you use only 4 bytes. Your codes not only calculate it wrong, but also exhibit undefined behavior.
IV is per message, not per byte.
Because you choose the IV wrong, your algorithm basically reduces to the one-time pad, except not as secure. If you ever encrypt two messages with the same key, the system is broken.
The performance issue is your least concern. This whole implementation is simply incorrect and insecure. You must study cryptography systematically before trying to utilize it, for it is not a field you can learn just by trial and error. It is easy to design a system that passes all the unit tests and looks fine to your own eyes, but completely broken to the trained ones.
I recommend cryptography on coursera.
No, you are not doing this right. You don't need to iterate through the input of the decrypt method at all.
You only have to calculate the right counter for the block that contains the byte to decrypt. Then you can use that counter as IV value. Now you can encrypt or decrypt a block of ciphertext and retrieve the right byte. There is no need to decrypt specific bytes separately.
So if the block size of the cipher is 16, the IV/nonce is F000000000000000F000000000000000h and the offset of the byte is 260 then the counter/IV needs to be advanced with 260 / 16 = 16 = 10h. Then F000000000000000F000000000000000h + 10h = F000000000000000F000000000000010. Then you decrypt the 16th block and take the 4th byte at offset 3 (as 260 % 16 = 4).
Related
I am implementing a custom iostream (i.e., with read, write, seek and close) which uses the RC4 stream cipher for encryption and decryption. One of the contracts of this stream is that it is bidirectional and calling code needs to be able to arbitrarily seek to any position in the stream before doing any actual reading or writing.
Now because RC4 utilizes a key that relies on all previous swap operations up to a given 'tell' position, how can I incorporate an ability to arbitrarily seek to any position?
Obviously I could seek up to the position of the given seek offset (marked by THIS BIT in the following example), before doing the actual xor-ing transformation process, something like,:
/**
* #brief called from a stream's read or write function
* #param in the input buffer
* #param out the output buffer
* #param startPosition the current stream position (obtained via the streams
* tellg or tellp functions for read and write respectively)
* #param length the number of bytes to transform
*/
void transform(char *in, char *out,
std::ios_base::streamoff startPosition,
long length)
{
// need to reset sbox from member s_box each time this
// function is called
long sbox[256];
for (int i = 0; i<256; ++i) {
sbox[i]=m_sbox[i];
}
// ***THIS BIT***
// need to run the swap operation startPosition times
// to get sbox integer sequence in order
int i = 0, j = 0, k = 0;
for (int a=0; a < startPosition; ++a) {
i = (i + 1) % 256;
j = (j + sbox[i]) % 256;
swapints(sbox, i, j);
}
// now do the actual xoring process up to the length
// of how many bytes are being read or written
for (int a=0; a < length; ++a) {
i = (i + 1) % 256;
j = (j + sbox[i]) % 256;
swapints(sbox, i, j);
k = sbox[(sbox[i] + sbox[j]) % 256];
out[a] = in[a] ^ k;
}
}
and then the transform would be called from the read or write of the stream implementation, something like:
MyStream&
MyStream::read(char * const buf, std::streamsize const n)
{
std::ios_base::streamoff start = m_stream.tellg();
std::vector<char> in;
in.resize(n);
(void)m_stream.read(&in.front(), n);
m_byteTransformer->transform(&in.front(), buf, start, n);
return *this;
}
EDIT: the stream should have no knowledge of how the transformation function works. The transformation function is completely independent and I should be able to freely swap in different transformation implementations.
EDIT: the function swapints looks like this:
void swapints(long *array, long ndx1, long ndx2)
{
int temp = array[ndx1];
array[ndx1] = array[ndx2];
array[ndx2] = temp;
}
The real problem with the above transform function is in its slowness because it has to perform startPosition initial swap operations before the xor transformation-proper is performed. This is very problematic when many seek operations are performed. Now I've heard that RC4 is meant to be quick but my (probably bad implementation) suggests otherwise given the initial set of swap operations.
So my real question is: how can the above code be optimized to reduce the number of required operations? Ideally I would like to eliminate the initial ("THIS BIT") set of swap operations
EDIT: optimizing out the initial sbox setting is probably trivial (e.g. using memcpy as suggested by egur). The important optimization I think is how I can optimize out the loop marked by THIS BIT. Perhaps all those swap ints can be programmed more concisely without the need for a for-loop.
Thanks,
Ben
Change all % 255 to & 0xff, much faster:
i = (i + 1) % 256;
To:
i = (i + 1) & 0xFF;
Edit:
You're wasting a lot of time initializing sbox. You should pass sbox as a parameter to the transform function so the original copy is updated between calls. What you're doing now is initializing it again and again and every time it will take longer since startPosition grows.
void transform(char *in, char *out,
long length,
unsigned char* sbox)
The temporary sbox should be a member of the MyStream class. The read function should be:
MyStream&
MyStream::read(char * const buf, std::streamsize const n)
{
std::ios_base::streamoff start = m_stream.tellg();
std::vector<char> in;
in.resize(n);
(void)m_stream.read(&in.front(), n);
// init m_TempSbox on first call
if (m_FirstCall) {
initTempSbox();
}
m_byteTransformer->transform(&in.front(), buf, n, m_TempSbox);
return *this;
}
After some research, it turns out that random access of RC4's key-stream is not possible. See discussion at this link: crypto.stackexchange. A better alternative (as pointed out by Rossum in his comment) is to use a block cipher in counter mode.
What you do in counter mode is to encrypt a sequence of numbers. This sequence is incremental and is the length of the entire stream of data. So, say you want to encrypt 8 bytes of data starting at position '16' of the original data stream using a 64 bit (8 bytes) block cipher.
8 bytes need to be enciphered since you operate over 8-bytes of plain text at a time. Since the position we want to randomly offset to is 16, we essentially encrypt 'block 3' of this number sequence (bytes 0 to 7 == block 1, bytes 8 to 15 == block 2, bytes 16 to 23 == block 3 and so on...)
For example using the XTEA algorithm which encrypts blocks of 8 bytes using a 128 bit key, we'd do something like:
Block 3:
// create a plain text number sequence
uint8_t plainText[8];
plainText[0] = 16;
plainText[1] = 17;
.
.
.
plainText[7] = 23;
// encrypt the number sequence
uint8_t cipherText[8];
applyXTEATransformation(plainText, cipherText, keyOfLength128Bit);
// use the encrypted number sequence as a
// key stream on the data to be encrypted
transformedData[16] = dataToBeEncrypted[16] ^ cipherText[0];
transformedData[17] = dataToBeEncrypted[17] ^ cipherText[1];
.
.
.
transformedData[23] = dataToBeEncrypted[23] ^ cipherText[7];
tldr: I wanted to do random access on RC4 but discovered it isn't possible so used counter mode on an XTEA block cipher instead.
Ben
I'm having trouble reading in a 16bit .wav file. I have read in the header information, however, the conversion does not seem to work.
For example, in Matlab if I read in wave file I get the following type of data:
-0.0064, -0.0047, -0.0051, -0.0036, -0.0046, -0.0059, -0.0051
However, in my C++ program the following is returned:
0.960938, -0.00390625, -0.949219, -0.00390625, -0.996094, -0.00390625
I need the data to be represented the same way. Now, for 8 bit .wav files I did the following:
uint8_t c;
for(unsigned i=0; (i < size); i++)
{
c = (unsigned)(unsigned char)(data[i]);
double t = (c-128)/128.0;
rawSignal.push_back(t);
}
This worked, however, when I did this for 16bit:
uint16_t c;
for(unsigned i=0; (i < size); i++)
{
c = (signed)(signed char)(data[i]);
double t = (c-256)/256.0;
rawSignal.push_back(t);
}
Does not work and shows the output (above).
I'm following the standards found Here
Where data is a char array and rawSignal is a std::vector<double> I'm probably just handing the conversion wrong but cannot seem to find out where. Anyone have any suggestions?
Thanks
EDIT:
This is what is now displaying (In a graph):
This is what it should be displaying:
There are a few problems here:
8 bit wavs are unsigned, but 16 bit wavs are signed. Therefore, the subtraction step given in the answers by Carl and Jay are unnecessary. I presume they just copied from your code, but they are wrong.
16 bit waves have a range from -32,768 to 32,767, not from -256 to 255, making the multiplication you are using incorrect anyway.
16-bit wavs are 2 bytes, thus you must read two bytes to make one sample, not one. You appear to be reading one character at a time. When you read the bytes, you may have to swap them if your native endianness is not little-endian.
Assuming a little-endian architecture, your code would look more like this (very close to Carl's answer):
for (int i = 0; i < size; i += 2)
{
int c = (data[i + 1] << 8) | data[i];
double t = c/32768.0;
rawSignal.push_back(t);
}
for a big-endian architecture:
for (int i = 0; i < size; i += 2)
{
int c = (data[i] << 8) | data[i+1];
double t = c/32768.0;
rawSignal.push_back(t);
}
That code is untested, so please LMK if it doesn't work.
(First of all about little-endian/big-endian-ness. WAV is just a container format, the data encoded in it can be in countless format. Most of the codecs are lossless (MPEG Layer-3 aka MP3, yes, the stream can be "packaged" into a WAV, various CCITT and other codecs). You assume that you deal with some kind of PCM format, where you see the actual wave in RAW format, no lossless transformation was done on it. The endianness depends on the codec, which produced the stream.
Is the endianness of format params guaranteed in RIFF WAV files?)
It's also a question if the one PCM sample is in linear scale sampled integer or there some scaling, log scale or other transformation behind it. Regular PCM wav files I encountered were simple linear scale samples, but I'm not working in the audio recording or producing industry.
So a path to your solution:
Make sure that you are dealing with regular 16 bit PCM encoded RIFF WAV file.
While reading the stream, always read two bytes (char) at a time and convert the two chars into a 16 bit short. People showed this before me.
The wave form you show clearly suggest that you either not estimated the frequency well (or you just have one mono channel instead of a stereo). Because the sampling rate (44.1kHz, 22KHz, 11KHz, 8kHz, etc) is just as important as the resolution (8 bit, 16 bit, 24 bit, etc). Maybe in the first case you had a stereo data. You can read it in as mono, you may not notice it. In the second case if you have mono data, then you'll run out of samples half way into reading the data. That's what it seems to happen according to your graphs. Talking about the other cause: the lower sampling resolutions (and 16 bit is also lower) often paired with lower sampling rates. So if your input data is the recording time, and you think you have a 22kHz data, but it's actually just 11kHz, then again you'll run out half way through from the actual samples and read in memory garbage. So either one of these.
Make sure that you interpret and treat your loop iterator variable and the size well. It seems that size tells how many bytes you have. You'll have exactly half as much short integer samples. Notice, that Bjorn's solution correctly increments i by 2 because of that.
My working code is
int8_t* buffer = new int8_t[size];
/*
HERE buffer IS FILLED
*/
for (int i = 0; i < size; i += 2)
{
int16_t c = ((unsigned char)buffer[i + 1] << 8) | (unsigned char)buffer[i];
double t = c/32768.0;
rawSignal.push_back(t);
}
A 16-bit quantity gives you a range from -32,768 to 32,767, not from -256 to 255 (that's just 9 bits). Use:
for (int i = 0; i < size; i += 2)
{
c = (data[i + 1] << 8) + data[i]; // WAV files are little-endian
double t = (c - 32768)/32768.0;
rawSignal.push_back(t);
}
You might want something more like this:
uint16_t c;
for(unsigned i=0; (i < size); i++)
{
// get a 16 bit pointer to the array
uint16_t* p = (uint16_t*)data;
// get the i-th element
c = *( p + i );
// convert to signed? I'm guessing this is what you want
int16_t cs = (int16_t)c;
double t = (cs-256)/256.0;
rawSignal.push_back(t);
}
Your code converts the 8 bit value to a signed value then writes it into an unsigned variable. You should look at that and see if it's what you want.
I need to extract an 8 byte digest from a variable length string so I'm looking for such an algorithm that I will implement in c/c++. That will be part of a digital signature procedure on a microcontroller, so it has to be:
writable in few lines of code, since the firmware has to be kept as little as possible;
low in resource consumption, expecially ram (preferably less than 100 bytes);
strong enough that changing a single character at any point of the string would change the overall digest.
I took a look at existing algorithms such as crc64 but they seems to be too heavy for my platform.
There is no chance to do a secure hash in 64 bits. Even SHA-1 at 160 bit is considered theoretically broken. You should use SHA2-256 if you really care about secure digital signing. If you don't care about security and just want a hash function that avoids non-adversarial collisions just use the following, it is fine:
constexpr uint64 P1 = 7;
constexpr uint64 P2 = 31;
uint64 hash = P1;
for (const char* p = s; *p != 0; p++) {
hash = hash * P2 + *p;
}
As AndrewTomazos-Fathomling said, it's impossible to do a secure hash in 64 bits, so if that's your intention then my advice is STOP, pick up a book and read about cryptographically secure hashing.
If you don't plan on using this as a secure hash and you do not care about collisions or attacks, then the answer he gave you works just fine and you can tweak the primes P1 and P2 as necessary. I will give you another alternative which allows you to do tagged hashing and mixes things up more.
// Disclaimer: I make no claims about the quality of this particular hash - it's
// certainly not a cryptographically secure hash, nor should it *ever* be
// construed as such.
unsigned long long quickhash64(const char *str, unsigned long long mix = 0)
{ // set 'mix' to some value other than zero if you want a tagged hash
const unsigned long long mulp = 2654435789;
mix ^= 104395301;
while(*str)
mix += (*str++ * mulp) ^ (mix >> 23);
return mix ^ (mix << 37);
}
Here is a modified version of a 32 bit version I found in my old source files
static unsigned long long llhash(const char *str)
{
unsigned long long hash = 5381;
int c;
while (c = *str++)
hash = ((hash << 5) + hash) + c;
return hash;
}
But hashing will always result in collisions. Of course some algorithms are better than others.
Edit:
I found the source of the 32 bit version: http://www.cse.yorku.ca/~oz/hash.html
I had the exact same requirement, and I settled for FNV-1A, after dismissing SIP hash (implemented by bloomberg here).
I found an FNV implementation here:
https://github.com/foonathan/string_id/blob/master/hash.hpp
which is:
constexpr uint64_t fnv_basis = 14695981039346656037ull;
constexpr uint64_t fnv_prime = 1099511628211ull;
// FNV-1a 64 bit hash of null terminated buffer
uint64_t fnv1a_hash(const char* str, uint64_t hash = fnv_basis)
{
return *str ? fnv1a_hash(str + 1, (hash ^ *str) * fnv_prime) : hash;
}
It appears he is looping using tail recursion. And stop condition is the null byte.
(boost uses hash_range which is hash_combining each element in chain I guess.)
License is zlib and copyright is Jonathan Müller. Though I'm not convinced a oneliner can be legally licensed if it implements research by other persons (Fowler-Noll-Vo).
i have an two arrays : char data1[length] where length is a multiple of 8 i.e length can be 8, 16,24 ... The array contains binary data read from a file that is open in binary mode. I will keep reading from the file and everytime i read i will store the read value in a hash table. The disterbution of this binary data has a random distribution. I would like to hash each array and store them in a hash table in order to be able to look for the char with the specific data again. What would be a good hashing function to achive this task. Thanks
Please note that i am writing this in c++ and c so any language you choose to provide a solution for would be great.
If the data that you read is 8 bytes long and really distributed randomly, and your hashcode needs to be 32 bits, what about this:
uint32_t hashcode(const unsigned char *data) {
uint32_t hash = 0;
hash ^= get_uint32_le(data + 0);
hash ^= get_uint32_le(data + 4);
return hash;
}
uint32_t get_uint32_le(const unsigned char *data) {
uint32_t value = 0;
value |= data[0] << 0;
value |= data[1] << 8;
value |= data[2] << 16;
value |= data[3] << 24;
return value;
}
If you need more speed, this code can probably made a lot faster if you can guarantee that data is always properly aligned to be interpreted as an const uint32_t *.
I have successfully used MurmurHash3 in one of my projects.
Pros:
It is fast. Very fast.
It supposedly has a low collision rate.
Cons:
It's not suitable for cryptography applications.
It's not standardized in any shape or form.
It's not portable to non-x86 platforms. However, it's small enough that you should be able to port it if you really need to - I was able to port it to Java, although that's not nearly the same thing.
It's a good possibility for use in e.g. a fast hash-table implementation...
So in my code I have a series of chars which I want to replace with random data. Since rand can replace ints, I figured I could save some time by replacing four chars at once instead of one at a time. So basically instead of this:
unsigned char TXT[] = { data1,data2,data3,data4,data4,data5....
for (i = 34; i < flenght; i++) // generating the data to send.
TXT[i] = rand() % 255;
I'd like to do something like:
unsigned char TXT[] = { data1,data2,data3,data4,data4,data5....
for (i = 34; i < flenght; i+4) // generating the data to send.
TXT[i] = rand() % 4294967295;
Something that effect, but I'm not sure how to do the latter part. Any help you can give me is greatly appreciated, thanks!
That won't work. The compiler will take the result from rand() % big_number and chop off the extra data to fit it in an unsigned char.
Speed-wise, your initial approach was fine. The optimization you contemplated is valid, but most likely unneeded. It probably wouldn't make a noticeable difference.
What you wanted to do is possible, of course, but given your mistake, I'd say the effort to understand how right now far outweights the benefits. Keep learning, and the next time you run across code like this, you'll know what to do (and judge if it's necessary), look back on this moment and smile :).
You'll have to access memory directly, and do some transformations on your data. You probably want something like this:
unsigned char TXT[] = { data1,data2,data3,data4,data4,data5....
for (i = 34; i < flenght/sizeof(int); i+=sizeof(int)) // generating the data to send.
{
int *temp = (int*)&TXT[i]; // very ugly
*temp = rand() % 4294967295;
}
It can be problematic though because of alignment issues, so be careful. Alignment issues can cause your program to crash unexpectedly, and are hard to debug. I wouldn't do this if I were you, your initial code is just fine.
TXT[i] = rand() % 4294967295;
Will not work the way you expect it to. Perhaps you are expecting that rand()%4294967295 will generate a 4 byte integer(which you maybe interpreting as 4 different characters). The value that rand()%4294967295, produces will be type cast into a single char and will get assigned to only one of the index of TXT[i].
Though it's not quire clear as to why you need to make 4 assigning at the same time, one approach would be to use bit operators to obtain 4 different significant bytes of the number generated and those can then be assigned to the four different index.
There are valid answers just so much C does not care very much about what type it stores at which address. So you can get away with something like:
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
char *arr;
int *iArr;
int main (void){
int i;
arr = malloc(100);
/* Error handling ommitted, yes that's evil */
iArr = (int*) arr;
for (i = 0; i < 25; i++) {
iArr[i] = rand() % INT_MAX;
}
for (i = 0; i < 25; i++) {
printf("iArr[%d] = %d\n", i, iArr[i]);
}
for (i = 0; i < 100; i++) {
printf("arr[%d] = %c\n", i, arr[i]);
}
free(arr);
return 0;
}
In the end an array is just some contiguous block in memory. And you can interpret it as you like (if you want). If you know that sizeof(int) = 4 * sizeof(char) then the above code will work.
I do not say I recommend it. And the others have pointed out whatever happened the first loop through all the chars in TXT will yield the same result. One could think for example of unrolling a loop but really I'd not care about that.
The (int*) just alone is warning enough. It means to the compiler, do not think about what you think the type is just "believe" he programmer that he knows better.
Well this "know better" is probably the root of all evil in C programming....
unsigned char TXT[] = { data1,data2,data3,data4,data4,data5....
for (i = 34; i < flenght; i+4)
// generating the data to send.
TXT[i] = rand() % 4294967295;
This has a few issues:
TXT is not guaranteed to be memory-aligned as needed for the CPU to write int data (whether it works - perhaps relatively slowly - or not - e.g. SIGBUS on Solaris - is hardware specific)
the last 1-3 characters may be missed (even if you change i + 4 to i += 4 ;-P)
rand() returns an int anyway - you don't need to mod it with anything
you need to write your random data via an int* so you're accessing 4 bytes at a time and not simply slicing a byte off the end of the random data and overwriting every fourth single character
for stuff like this where you're dependent on the size of int, you should really write it in terms of sizeof(int) so it'll work even if int isn't 32 bits, or use a (currently sadly) non-Standard but common typedef such as int32_t (or on Windows I think it's __int32, or you can use a boost or other library header to get int32_t, or write your own typedef).
It's actually pretty tricky to align your text data: your code suggests you want int-sized slices from the 35th character... even if the overall character array is aligned properly for ints, the 35th character will not be.
If it really is always the 35th, then you can pad the data with a leading character so you're accessing the 36th (being a multiple of presumably 32-bit int size), then align the text to an 32-bit address (with a compiler-specific #pragma or using a union with int32_t). If the real code varies the character you start overwriting from, such that you can't simply align the data once, then you're stuck with:
your original character-at-a-time overwrites
non-portable unaligned overwrites (if that's possible and better on your system), OR
implementing code that overwrites up to three leading unaligned characters, then switches to 32-bit integer overwrite mode for aligned addresses, then back to character-by-character overwrites for up to three trailing characters.
That does not work because the generated value is converted to type of array element - char in this particular case. But you are free to interpret allocated memory in the manner you like. For example, you could convert it into array int:
unsigned char TXT[] = { data1,data2,data3,data4,data4,data5....
for (i = 34; i < flenght-sizeof(int); i+=sizeof(int)) // generating the data to send.
*(int*)(TXT+i) = rand(); // There is no need in modulo operator
for (; i < flenght; ++i) // generating the data to send.
TXT[i] = rand(); // There is no need in modulo operator either
I just want to complete solution with the remarks about modulo operator and handling of arrays not multiple of sizeof(int).
1) % means "the remainder when divided by", so you want rand() % 256 for a char, or else you will never get chars with a value of 255. Similarly for the int case, although here there is no point in doing a modulus operation anyway, since you want the entire range of output values.
2) rand usually only generates two bytes at a time; check the value of RAND_MAX.
3) 34 isn't divisible by 4 anyway, so you will have to handle the end case specially.
4) You will want to cast the pointer, and it won't work if it isn't already aligned. Once you have the cast, though, there is no need to account for the sizeof(int) in your iteration: pointer arithmetic automatically takes care of the element size.
5) Chances are very good that it won't make a noticeable difference. If scribbling random data into an array is really the bottleneck in your program, then it isn't really doing anything significiant anyway.