I have simple question? I have Data of 256 bytes including 32 parity bytes.
If I receive only 223 bytes including the 32 parity bytes. Am I still able to recover the 256 Block? If not, what if i know the location of the missing bytes?
Even if the locations of the missing bytes are known, the received message has 33 missing bytes and can't be recovered. If the received message was only missing 32 bytes, and assuming there are no errors in the received bytes, then the 32 missing bytes can be treated as erasures and corrected. It doesn't make any difference if the missing bytes are data bytes or parity bytes.
Note that there are two different encoding schemes named Reed Solomon, "original view" and "BCH view". The question mentions a total size of 256 bytes, which would require "original view" Reed Solomon encoding (see link to Wiki article below). If using "BCH view" Reed Solomon encoding, the maximum message size (data + parity) is 255 bytes (assuming that the Galois Field is GF(2^8)).
https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction#Constructions
Raid 6 uses yet another encoding scheme. The parity bytes are the calculated "BCH view" syndromes of the data bytes, and can allow up to 255 data bytes plus 255 parity bytes (for a total of 510 bytes). It is intended to be an erasure only code where location of the error or missing bytes are known.
Related
In describing and ID3v2 header and the frames within, https://id3.org/id3v2.3.0#ID3v2_frame_overview states:
The frame ID is followed by a size descriptor, making a total header size of ten bytes in every frame.
Yet when I use a hex editor to look through the frames of an ID3 tag, the frame seems to be 12 bytes. I have looked at numerous songs and they seem to have the tag, followed by a 4 byte size descriptor and then 4 additional bytes (the description says this should be two flag bytes).
I admit to being a little of of my depth here but I'm trying to write ID3v2 tags using PHP and I'm a bit stumped.
You haven't read 3.3.1. Frame header flags:
Some flags indicates that the frame header is extended with additional information. This information will be added to the frame header in the same order as the flags indicating the additions. I.e. the four bytes of decompressed size will precede the encryption method byte.
Which means the following in addition to 3.3. ID3v2 frame overview:
The layout of the frame header:
Frame ID $xx xx xx xx (four characters)
Size $xx xx xx xx
Flags $xx xx
If looking bit wise at those 2 bytes "Flags" of the 10 bytes frame header then you have to expect additional bytes as per set flag:
bit 7 ("i") = 4 more bytes "decompressed size"
bit 6 ("j") = 1 more byte "encryption method"
bit 5 ("k") = 1 more byte "group identifier"
So your observation might be correct. If you link to the first 4096 bytes of such a file then I can tell you if they're still correct as per the standard although they "look" like having 12 bytes per header.
I am using openssl to encrypt the string and i get null terminated string. I have now encrypted string and I want to send it over network with base64 encoding. I just need to send the encrypted data, how can I calculate the length of the string on the other side before decryption?
unsigned char *plaintext = (unsigned char*) "The quick brown fox jumps sover thes lazy dog";
unsigned char ciphertext[256] = {};
// After using openssl envelope_seal(), with EVP_aes_128_cbc()
strlen((const char*)ciphertext) // length is different each time due to null terminated binary string
sizeof(ciphertext) // lenght is equal to 256
int envelope_seal( ) // gives me the length of that string as 48.
Kindly help me to calculate the length of the ciphertext which is 48 but none of the method in my knowledge gives me the correct output.
AES is a block cipher. It has block size of 16 bytes - which means if you want to encrypt some data with it the length of the data in bytes must be multiple of 16 (if it is not you might need to use padding such as say PKCS7, more details).
Now after you encrypt a string with AES (say length of string is 32 bytes) - you can't use strlen anymore to get the length of the result, because the result, isn't a string anymore it is some byte array which represents the results of encryption. Actually you don't need to get the length anyway, it will be same size as plaintext - 32 bytes as we said in our case.
So I think you don't have issues with calculating length anymore - now if the other side should know length of the ciphertext you can send the length (say 32 in our case) in advance in packet. The other side should reconstruct the plain text now (and also remove padding bytes if one was used).
note: After you performed encryption and have the ciphertext you can apply base64 encoding to it and send it over, but you could as well send the byte array representing the ciphertext too.
In regard to comments, I will try to briefly highlight how this process goes. Say you have string char * str = "hello" - which is 5 bytes and you need to encrypt it. Like I said you can't encrypt it directly, you need to pad it to make multiple of 16. For this you can use PKCS7 padding (it is similar to PKCS5 but is for 16 bytes blocks). Now when you apply padding e.g., char * paddedText = PKCS7(str), you will end up with byte array which is 16 bytes.
Now, there is no more problem. You can encrypt this 16 bytes of plaintext. Result will also be 16 bytes cipher text.
Now if you want to decrypt this 16 bytes of cipher text, you can decrypt it directly (because it is multiple of 16). Now, due to the way PKCS7 padding works you will easily be able to tell from the result that only first 5 bytes were original string and you will remove 11 redundant bytes (how PKCS5 works, see in my link-PKCS7 is similar just for 16 byte block lengths), and obtain "hello".
Finally, I don't see where is the problem, when you send from client to server, you can just encode message length, e.g., 16 in packet, so that receiver knows 16 bytes represent cipher text. But again as I said, using this 16 bytes, the receiver, after decrypting will be able to determine that only first 5 bytes were original text (due to used padding PKCS7). So there is no need to send anything except 16; with help of PKCS padding scheme you will be able to tell that only first 5 bytes were plain text.
ps. After some discussions with OP it seems padding was not his main issue, because openssl method that was used seems to take care of that.
If the valid data in the array isn't terminated, then there's no way to tell its length by looking at the array.
If envelope_seal told you the length, then use that, passing it wherever the length is needed.
AES is a block cipher. Therefore the length of ciphertext will be the length of your plaintext modulo blocksize, rounded up to nearest blocksize.
I am trying to build a fast 8-bit greyscale PNG encoder. Unfortunately I must be misunderstanding part of the spec. Smaller image sizes seem to work, but the larger ones will only open in some image viewers. This image (with multiple DEFLATE Blocks) gives a
"Decompression error in IDAT" error in my image viewer but opens fine in my browser:
This image has just one DEFLATE block but also gives an error:
Below I will outline what I put in my IDAT chunk in case you can easily spot any mistakes (note, images and steps have been modified based on answers, but there is still a problem):
IDAT length
"IDAT" in ascii (literally the bytes 0x49 0x44 0x41 0x54)
Zlib header 0x78 0x01
Steps 4-7 are for every deflate block, as the data may need to be broken up:
The byte 0x00 or 0x01, depending on if it is a middle or the last block.
Number of bytes in block (up to 2^16-1) stored as a little endian 16-bit integer
The 1's complement of this integer representation.
Image data (each scan-line is starts with a zero-byte for the no filter option in PNG, and is followed by width bytes of greyscale pixel data)
An adler-32 checksum of all the image data
A CRC of all the IDAT data
I've tried pngcheck on linux, an it does not spot any errors. If nobody can see what is wrong, can you point me in the right direction for a debugging tool?
My last resort is to use the libpng library to make my own decoder, and debug from there.
Some people have suggested it may be my adler-32 function calculation:
static uint32_t adler32(uint32_t height, uint32_t width, char** pixel_array)
{
uint32_t a=1,b=0,w,h;
for(h=0;h<height;h++)
{
b+=a;
for(w=0;w<width;w++)
{
a+=pixel_array[h][w];
b+=a;
}
}
return (uint32_t)(((b%65521)*65536)|(a%65521));
}
Note that because the pixel_array passed to the function does not contain the zero-byte at the beginning of each scanline (needed for PNG) there is an extra b+=a (and implicit a+=0) at the beginning of each iteration of the outer loop.
I do get an error with pngcheck: "zlib: inflate error = -3 (data error)". As your PNG scaffolding structure looks okay, it's time to take a low-level look into the IDAT block with a hex viewer. (I'm going to type this up while working through it.)
The header looks alright; IDAT length is okay. Your zlib flags are 78 01 ("No/low compression", see also What does a zlib header look like?), where one of my own tools use 78 9C ("Default compression"), but then again, these flags are only informative.
Next: zlib's internal blocks (per RFC1950).
Directly after the compression flags (CMF in RFC1950) it expects FLATE compressed data, which is the only compression scheme zlib supports. And that is in another castle RFC: RFC1951.
Each separately compression block is prepended by a byte:
3.2.3. Details of block format
Each block of compressed data begins with 3 header bits
containing the following data:
first bit BFINAL
next 2 bits BTYPE
...
BFINAL is set if and only if this is the last block of the data
set.
BTYPE specifies how the data are compressed, as follows:
00 - no compression
01 - compressed with fixed Huffman codes
10 - compressed with dynamic Huffman codes
11 - reserved (error)
So this value can be set to 00 for 'not last block, uncompressed' and to 01 for 'last block, uncompressed', immediately followed by the length (2 bytes) and its bitwise inverse, per 3.2.4. Non-compressed blocks (BTYPE=00):
3.2.4. Non-compressed blocks (BTYPE=00)
Any bits of input up to the next byte boundary are ignored.
The rest of the block consists of the following information:
0 1 2 3 4...
+---+---+---+---+================================+
| LEN | NLEN |... LEN bytes of literal data...|
+---+---+---+---+================================+
LEN is the number of data bytes in the block. NLEN is the
one's complement of LEN.
They are the final 4 bytes in your IDAT segment. Why do small images work, and larger not? Because you only have 2 bytes for the length.1 You need to break up your image into blocks no larger than 65,535 bytes (in my own PNG creator I seem to have used 32,768, probably "for safety"). If the last block, write out 01, else 00. Then add the two times two LEN bytes, properly encoded, followed by exactly LEN data bytes. Repeat until done.
The Adler-32 checksum is not part of this Flate-compressed data, and should not be counted in the blocks of LEN data. (It is still part of the IDAT block, though.)
After re-reading your question to verify I addressed all of your issues (and confirming I spelled "Adler-32" correctly), I realized you describe all of the steps right -- except that the 'last block' indicator is 01, not 80 (later edit: uh, perhaps you are right about that!) -- but that it does not show in this sample PNG. See if you can get it to work following all of the steps by the letter.
Kudos for doing this 'by hand'. It's a nice exercise in 'following the specs', and if you get this to work, it may be worthwhile to try and add proper compression. I shun pre-made libraries as much as possible; the only allowance I made for my own PNG encoder/decoder was to use Rich Geldreich's miniz.c, because implementing proper Flate encoding/decoding is beyond my ken.
1 That's not the whole story. Browsers are particularly forgiving in HTML errors; it seems they are as forgiving for PNG errors as well. Safari displays your image just fine, and so does Preview. But they may just all be sharing OS X's PNG decoder, because Photoshop rejects the file.
The byte 0x00 or 0x80, depending on if it is a middle or the last block.
Change the 0x80 to 0x01 and all will be well.
The 0x80 is appearing as a stored block that is not the last block. All that's being looked at is the low bit, which is zero, indicating a middle block. All of the data is in that "middle" block, so a decoder will recover the full image. Some liberal PNG decoders may then ignore the errors it gets when it tries to decode the next block, which isn't there, and then ignore the missing check values (Adler-32 and CRC-32), etc. That's why it shows up ok in browsers, even though it is an invalid PNG file.
There are two things wrong with your Adler-32 code. First, you are accessing the data from a char array. char is signed, so your 0xff bytes are being added not as 255, but rather as -127. You need to make the array unsigned char or cast it to that before extracting byte values from it.
Second, you are doing the modulo operation too late. You must do the % 65521 before the uint32_t overflows. Otherwise you don't get the modulo of the sum as required by the algorithm. A simple fix would be to do the % 65521 to a and b right after the width loop, inside the height loop. This will work so long as you can guarantee that the width will be less than 5551 bytes. (Why 5551 is left as an exercise for the reader.) If you cannot guarantee that, then you will need to embed a another loop to consume bytes from the line until you get to 5551 of them, do the modulo, and then continue with the line. Or, a smidge slower, just run a counter and do the modulo when it gets to the limit.
Here is an example of a version that works for any width:
static uint32_t adler32(uint32_t height, uint32_t width, unsigned char ** pixel_array)
{
uint32_t a = 1, b = 0, w, h, k;
for (h = 0; h < height; h++)
{
b += a;
w = k = 0;
while (k < width) {
k += 5551;
if (k > width)
k = width;
while (w < k) {
a += pixel_array[h][w++];
b += a;
}
a %= 65521;
b %= 65521;
}
}
return (b << 16) | a;
}
I'm using a DES algorithm I found on the web. It works fine but I have a problem.
As you know, DES encrypts/decrypts blocks of 64 bytes each. But what happens if in a big file the last block doesn't end at a 64 byte block boundary? I know, there would be errors.
I'm using the following code:
Des d1;
char *plaintext=new char[64];
char *chyphertext=new char[64];
h.open("requisiti.txt",ios::in|ios::binary);.
k.open("requisiti2.txt",ios::out|ios::binary);
while(!h.eof())
{
h.read(plaintext,64);
chyphertext=d1.Encrypt(plaintext);
//decryption is the same.just change Encrypt to Decrypt
k.write(chyphertext,64);
}
h.close();
k.close();
remove("requisiti.txt");
rename("requisiti2.txt","requisiti.txt");
So I need a solution like "padding", but I don't know a simple algorithm for it. Please help me to encrypt/decrypt file in a good way.
First, I'd like to point that DES works on 64bits chunks (making it 8bytes, not 64), as you can see in http://en.wikipedia.org/wiki/Data_Encryption_Standard (check data block size).
Now you're looking for some padding (and unpadding when deciphering). You can look at http://en.wikipedia.org/wiki/Padding_(cryptography)
I personnally like PKCS#7 because it's easy and usually adds a little overhead compared to standard size.
For encryption:
check the size of the chunk you just read from file
if it's 64bits, add a new chunk [8,8,8,...8], otherwise, pad it with the number of missing bytes (see example below)
encrypt
note that LAST packet is always containing padding with that algorithm (worst case is 8 bytes of padding)
Example:
read 0a 0b 0c, missing 5 bytes to fit in 8 bytes
padded packet :0a 0b 0c 05 05 05 05 05
For decryption :
read packet
decrypt
if it's the last packet, check value of the last byte (say it's n)
remove n bytes at the end of your packet
Hope this makes it more clear and helps you
EDIT
If your input file is pure text, you can pad with 0, if it's binary (and it must be since you're opening it as binary), PKCS#7 is better
Think about a file created like that : dd if=/dev/zero of=temp.zero count=100
a few of hundred bytes of zeros, what is padding and what ain't ?
Implementation is really easy :
think memset
don't forget to add last chunk if ile is a multiple of 8
By the way, DES is nowadays seriously broken, you should think about using a decent cipher if concerned with security (thinking AES at least, check http://en.wikipedia.org/wiki/Data_Encryption_Standard#Replacement_algorithms )
First of all: never use eof() to check whether the end of file is reached since it doesn't predict the end of file.
while(h.read(plaintext,64))
{
if (std::h.gcount() < 64) // gcount returns the number of characters extracted by the last unformatted input operation.
for (int i=std::h.gcount(); i<64; i++)
paintext[i] = 0; // pad the last block
chyphertext=d1.Encrypt(plaintext);
//decryption is the same.just change Encrypt to Decrypt
k.write(chyphertext,64);
}
I'm not sure what you are using DES for, but you should really use something else if you are actually trying to protect the data you are encrypting. DES is not secure anymore.
Also, I would imagine that a good library would do the padding for you.
First of all do not use DES! DES is broken and can be brute forced quite fast. Secondly you are using ECB mode you can read on wiki why you should avoid this one. Your data can be tampered with and you will not know about this - use AE mode like GCM. Like someone mentioned earlier DES have 64 bits not bytes block size which is 8 bytes.
I have some confusion regarding how the frame size bytes should be coded/decoded for ID3 v2.3.0. According to the (informal) ID3 v2.3.0 specification, the size of each frame should be coded into 4 bytes, where the most significant bit of each byte is unused. To calculate the size, it would take the formula below:
byte MASK = (byte)0x7F;
int size = 0;
for (int = 0; i < 4; i++) {
size = size * 128 + (b[i] & MASK);
}
But when I used my parser to parse some MP3 files, quite a few files had GEOB (general encapsulated object tag) frames whose size bytes were coded as if it were a Big Endian 32-bit Integer.
After I fixed these bytes by re-coding them using the proper algorithm, commercial software such as Windows 7 and Winamp were not able proper display the subsequent tags (in several instances, TIT2 was right after GEOB, so the song's title was not displayed although it was in the file).
I also found similar problems for MCDI (music cd identifier), and TALB ('Album/Movie/Show title') tags.
I read through the v2.3 spec, and also Googled, but wasn't able to find any information regarding the use of a 32-bit integer as size metadata for these frames. Yet the common behavior in different commercial software seems to suggest for such fields, a 32-bit integer should be used as size instead of 4 bytes masked by 0x7F.
So I am just wondering if anyone here has worked on ID3 v2.3 and could clarify this for me.
Yes. However, I consider the docs to be explicit enough, given the conventions of % (binary) and $ (hexadecimal) which are explained right away:
Header size:
4 * %0xxxxxxx as per v2.2.0 (§3.1.) header
4 * %0xxxxxxx as per v2.3.0 header
4 * %0xxxxxxx as per v2.4.0 (§3.1.) header
Frame size:
$xx xx xx as per v2.2.0 (i.e. §4.1.) frame
$xx xx xx xx as per v2.3.0 frame
4 * %0xxxxxxx as per v2.4.0 (§4.) frame
Summary:
For all 3 versions in ID3v2 the header size is stored in the same way: using 4 bytes, but for each only 7 bits are valid.
Only for ID3v2.2 frames the size consists of 3 (full) bytes.
Only for ID3v2.3 frames the size consists of 4 (full) bytes.
Only for ID3v2.4 frames the size finally is stored just like the header's size: 4 bytes, but only 28 bits are valid.
ID3v2.4.0 changes §3 also lines out the frame size change from v2.3.0. The whole issue comes from MPEG Audio (and AAC) stream which synchronizes with 9 (or 12) bits set - any decoder might then misinterpret the ID3 metadata as audio data.
I believe I have found the answer. ID3 v2.3, despite its being the more commonly supported (as opposed to v2.4) has not to well-written (and informal) spec. Its header size uses the 4 0x7F bytes, but the frame sizes are in fact 32-bit integers, only they are never clearly spelled out.
the reason I usually encountered the problem when dealing with GEOB is because the problem won't crop up until the frame size is larger than 0x7F, and GEOB usually is.