In practice, I select an executable. Size 20Mb.
I read the content using file.read(size=16).
If length of the returned byte string is less than 16 I fill the rest with \0 (NULL).
f = open("./installer.exe","rb")
obj = AES.new(b"0123456789012345",AES.MODE_CBC, b"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0")
bs = b""
t = f.read(16)
while t != b"":
if len(t) < 16:
t = t + b"\0" * (16 - len(t)) # if < 16 bytes using padding
bs = bs + obj.encrypt(t)
else:
bs = bs + obj.encrypt(t)
t = f.read(16)
then, bs contents the byte string of ALL content encrypted with 0123456789012345
I realise the mechanism of reading file first, then I encrypt the content as seen in the above piece of code (using obj.encrypt()). Then I write a new file with the content encrypted. The I read the data of encrypted file and by a similar procedure decrypt the data using obj.decrypt in intervals of 16 bytes and then I write a new file with the decrypted data.
This takes approximately 3 minutes.
¿It's fast, slow, or expected?
According to what I saw, the module is written in C. ¿Maybe should I use Cython embedded to make it faster?
How PGP can supposedly decrypt higher amounts of data in real time, for example, in an encrypted virtual disk?
edit:
This take almost same:
obj = AES.new(b"0123456789012345",AES.MODE_CBC, b"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0")
bs = b""
t = f.read(16)
while t != b"":
if len(t) < 16:
t = t + b"\0" * (16 - len(t))
bs = bs + t
else:
bs = bs + t
t = f.read(16)
bse = obj.encrypt(bs)
Ok. The problem was the size of the buffers encrypted. I decided to use 64000 bytes strings.
The procedure its simple. total size / string segments -> encrypt. And in the last segment, if the size of the segment is inferior to 64000 AND NOT multiple of 16, finds the nearest multiple and the remaining space is filled
bs = b""
dt = f.read()
dtl = len(dt)
dtr = ( dtl / 64000 ) + 1
for x in range(0, dtr):
if x == dtr-1:
i1 = 64000 * x
dst = dtl - i1
i = math.ceil(dst / 16.0) * 16
dst = i - dst
buf = dt[i1:] + (b"\0" * int(dst))
bs = bs + obj.encrypt(buf)
else:
i1, i2 = 64000 * x , 64000 * (x+1)
bs = bs + obj.encrypt(dt[i1:i2])
Now takes 10 seconds.
Thanks for all.
Related
I could use your support on this. Here is my issue:
I've got a 2D buffer of floats (in a data object) in a C++ code, that I write in a binary file using:
ptrToFile.write(reinterpret_cast<char *>(&data->array[0][0]), nbOfEltsInArray * sizeof(float));
The data contains 8192 floats, and I (correctly ?) get a 32 kbytes (8192 * 4 bytes) file out of this line of code.
Now I want to read that binary file using MATLAB. The code is:
hdr_binaryfile = fopen(str_binaryfile_path,'r');
res2_raw = fread(hdr_binaryfile, 'float');
res2 = reshape(res2_raw, int_sizel, int_sizec);
But it's not happening as I expect it to happen. If I print the array of data in the C++ code using std::cout, I get:
pCarte_bin->m_size = 8192
pCarte_bin->m_sizel = 64
pCarte_bin->m_sizec = 128
pCarte_bin->m_p[0][0] = 1014.97
pCarte_bin->m_p[0][1] = 566946
pCarte_bin->m_p[0][2] = 423177
pCarte_bin->m_p[0][3] = 497375
pCarte_bin->m_p[0][4] = 624860
pCarte_bin->m_p[0][5] = 478834
pCarte_bin->m_p[1][0] = 2652.25
pCarte_bin->m_p[2][0] = 642077
pCarte_bin->m_p[3][0] = 5.33649e+006
pCarte_bin->m_p[4][0] = 3.80922e+006
pCarte_bin->m_p[5][0] = 568725
And on the MATLAB side, after I read the file using the little block of code above:
size(res2) = 64 128
res2(1,1) = 1014.9659
res2(1,2) = 323288.4063
res2(1,3) = 2652.2515
res2(1,4) = 457593.375
res2(1,5) = 642076.6875
res2(1,6) = 581674.625
res2(2,1) = 566946.1875
res2(3,1) = 423177.1563
res2(4,1) = 497374.6563
res2(5,1) = 624860.0625
res2(6,1) = 478833.7188
The size (lines, columns) is OK, as well as the very first item ([0][0] in C++ == [1][1] in MATLAB). But:
I'm reading the C++ line elements along the columns: [0][1] in C++ == [1][2] in MATLAB (remember that indexing starts at 1 in MATLAB), etc.
I'm reading one correct element out of two along the other dimension: [1][0] in C++ == [1][3] in MATLAB, [2][0] == [1][5], etc.
Any idea about this ?
Thanks!
bye
Leaving aside the fact there seems to be some precision difference (likely the display settings in MATLAB) the issue here is likely the difference between row major and column major ordering of data. Without more details it will be hard to be certain. In particular MATLAB is column major meaning that contiguous memory on disk is interpreted as detailing sequential elements in a column rather than a row.
The likely solution is to reverse the two sizes in your reshape, and access the elements with indices reversed. That is, swap the int_size1 and int_size2, and then read elements expecting
pCarte_bin->m_p[0][0] = res2(1,1)
pCarte_bin->m_p[0][1] = res2(2,1)
pCarte_bin->m_p[0][2] = res2(3,1)
pCarte_bin->m_p[0][3] = res2(4,1)
pCarte_bin->m_p[1][0] = res2(1,2)
etc.
You could also transpose the array in MATLAB after read, but for a large array that could be costly in itself
I want to send a message to the server as per the AIS 140 standard. Please explain how to calculate the checksum. Find below the sample message format.
$Header,iTriangle,KA01I2000,861693034634154,1_37T02B0164MAIS,AIS140,12.976545,N,77.5497 59,E*50
As per AIS140 Standard Checksum is calculated by performing xor to all bytes received from packet.
Note: You have to remove '$'.
Caution: Use data from device to verify this code (Example provided from doc doesnt have valid checksum)
This Javascript code will help your job done.
function checksum(packet) {
const charArray = packet.split('');
let xor = 0;
const n = charArray.length;
for (let i = 1; i < n - 3; i++) {
xor = xor ^ charArray[i].charCodeAt(0);
}
const cs = parseInt("0x" + charArray[n - 2] + charArray[n - 1]);
return xor === cs;
}
checksum('$Header,iTriangle,KA01I2000,861693034634154,1_37T02B0164MAIS,AIS140,12.976545,N,77.549759,E*50')
I have a original allocated SGL array of structures that contain buffer address and lengths. We need to prepare a temporary SGL array based off original SGL structure array meeting few requirements and later use the temporary SGL array for crypto operations.
Requirement: Ignore the first 8 bytes and last 10 bytes
// Buffer data structure
typedef struct
{
void *data_addr;
int data_len;
}T_BUF_DATA;
Final Array:
T_BUF_DATA final_array[100];
Case1:
Original array: T_BUF_DATA[0] = buf_addr = 0xabc, buf_len = 1
T_BUF_DATA[1] = buf_addr = 0xdef, buf_len = 10
T_BUF_DATA[2] = buf_addr = 0x123, buf_len = 23
final_array[0] = buf_addr =( 0xdef + 7 ), buf_len = ( 10 - 7 ) // since we need to ignore the first 8 bytes from original list, adjust the buf_Addr by offset of 7 bytes
final_array[1] = buf_addr = 0x123, buf_len = ( 23 - 10 ) .. since we need to ignore the last 10 bytes
Case2:
Original array: T_BUF_DATA[0] = buf_addr = 0xabc, buf_len = 100
final_array[0] = buf_addr = ( 0xabc + 8 ), buf_len = 100 - ( 8 + 10 );
We need to implement a generic solution that can handle all the varying buffer length original array structures while preparing the final array. Can someone please assist me here ? I am able to handle the case 2, but stuck on attempting case 1 and any few other unknown corner cases.
void adjust_my_original_buffer ( T_BUF_DATA *data, int num_bufs )
{
T_BUF_DATA final_array[100];
int idx = 0;
for ( int i = 0 ; i < num_bufs; i++ )
{
// prepare the final array
}
}
Something like the following should work. The idea is to skip whole SG entries at the start (keeping track of remaining initial bytes to be skipped in initial_skip) and skip whole SG entries at the end (keeping track of remaining final bytes to be skipped in final_skip) in order to simplify the problem.
After the simplification, there may be 0, 1 or more SG entries remaining, indicated by the adjusted num_bufs, and the adjusted orig points to the first remaining entry. If there is at least one remaining SG entry, the first entry's data_len will be greater than initial_skip and the last entry's data_len will be greater than final_skip. If there is exactly one remaining SG entry, an additional test is required to check that its data_len is greater than initial_skip + final_skip, reducing the number of remaining SG entries to zero if that is not the case.
A loop copies the remaining SG entries from orig to final and the if statements within the loop adjust the first and last SG entries (which might be a single SG entry).
Finally, the function returns the length of the final SG list, which could be 0 if everything was skipped.
int adjust_my_original_buffer ( const T_BUF_DATA * restrict orig, T_BUF_DATA * restrict final, int num_bufs )
{
int initial_skip;
int final_skip;
// Skip initial bytes.
initial_skip = 8;
while ( num_bufs && orig[0].data_len <= initial_skip )
{
initial_skip -= orig[0].data_len;
orig++;
num_bufs--;
}
// Skip final bytes.
final_skip = 10;
while ( num_bufs && orig[num_bufs - 1].data_len <= final_skip )
{
final_skip -= orig[num_bufs - 1].data_len;
num_bufs--;
}
// If single SG entry remaining, check its length.
if ( num_bufs == 1 && data[0].data_len <= initial_skip + final_skip )
{
// Singleton SG entry is too short.
num_bufs = 0;
}
// Copy SG entries to final list, adjusting first and last entry.
for ( int i = 0; i < num_bufs; i++ )
{
final[i] = orig[i]; // Copy SG entry.
if ( i == 0 )
{
// Adjust first SG entry.
final[i].data_addr = (char *)final[i].data_addr + initial_skip;
final[i].data_len -= initial_skip;
}
if ( i == num_bufs - 1 )
{
// Adjust last SG entry.
final[i].data_len -= final_skip;
}
}
return num_bufs;
}
i have a key with type AcccAA where A-[A...Z] (capital letters), and c is [1..9]. i have 1500 segments.
Now my temp hash function
int HashFunc(string key){
int Adress = ((key[0] + key[1] + key[2] + key[3] + key[4] + key[5]) - 339) * 14;
return Adress;
}
and Excel show a lot of collision in center (from 400 to 900)
Please tell me the hash function to be more evenly.
A common way to build a hash function in this case is to evaluate some polynomial with prime coefficients, like this one:
int address = key[0] +
31 * key[1] +
137 * key[2] +
1571 * key[3] +
11047 * key[4] +
77813 * key[5];
return address % kNumBuckets;
This gives a much larger dispersion over the key space. Right now, you get a lot of collisions because anagrams like AB000A and BA000A will collide, but with the above hash function the hash is much more sensitive to small changes in the input.
For a more complex but (probably) way better hash function, consider using a string hash function like the shift-add-XOR hash, which also gets good dispersion but is less intuitive.
Hope this helps!
One way is to construct a guaranteed collision-free number (which will not make your hash table collision free of course), as long as the possible keys fit in an integral type (e.g. int):
int number = (key[0] - 'A') + 26 * (
(key[1] - '0') + 10 * (
(key[2] - '0') + 10 * (
(key[3] - '0') + 10 * (
(key[4] - 'A') + 26 * (
(key[5] - 'A')
)))));
This works since 26 * 10 * 10 * 10 * 26 * 26 = 17576000 which fits into an int fine.
Finally simply hash this integer.
Consider you have this message (ab,cd,ef) and you have the ROHC (Robust header compression) CRC8 polynomial e0.
C(x) = x^0 + x^1 + x^2 + x^8
Is there any way that I can calculate the CRC on the message backward starting from the last byte and get the same results as if I am calculating it on the original message?
No this is generally not possible for your polynomial (100000111).
EG: 110100111/100000111 = 011010011
but: 111001011/xxxxxxxxx != 011010011 (in general)
However, you can still check for the validity of your message if you know the CRC beforehand.
EG: 110100111/100000111 = 01101001
=> message transmitted = 11010011 01101001
=> message received (reversed) = 10010110 11001011
then: 10010110 11001011/111000001 == 0
(where: 111000001 = reversed(100000111))
=> crc(reversed(11001011)) = crc(11010011) == reversed(10010110) = 01101001
Note that this is only true if the message is reversed BITEWISE.
IE: reversed(ABC) = reversed(101010111100) = 001111010101
= 3D5 = reversed(ABC) != CBA = 110010111010 != reversed(101010111100)
So be careful when implementing your algorithm ;-)