I'm studying a working program that between others verifies the validity of an incoming message. The CRC used is CCITT 16, the initial value of the CRC is set to 0xFFFF, and the message is sent in bytes that are received LSb first and the bytes the MSB first. The message forms a long bitstream up to 1024 bits long, including the 16 CRC bits at the end.
I fully understand the calculation of the CRC value. What puzzles me, and I can find no reference justifying it, is the validity check performed after all message has been received. Namely, the CRC value is checked against 0xF0B8 instead of 0x0000. Am I missing something?
// crcValue is the 16 bit CRC variable under construction
// dataBit is a byte containing one bit of the data bitstream at LSb
...
// CRC calculation bit by bit
if(dataBit ^ (crcValue & 0x0001)) crcValue = (crcValue >> 1) ^ 0x8408;
else crcValue >>=1;
//After running over all the bitstream the crcValue is checked
if(crcValue != 0xF0B8) ... // reject data
else ... // accept data
The program works. As said I cannot understand where the 0xF0B8 value comes from.
Thank You in advance
You left out the fact that are also inverting the CRC before appending to the message. Checking against 0x0000 would work if there were no final invert or other final exclusive-or. 0xf08b is that CRC (initializing with 0xffff, but not inverting the result) of two zero bytes, which is your appended CRC in the case of an empty message.
Related
Working on a SPI communication bus between an array of SAMD MCUs.
I have an incoming packet which is something like { 0x00, 0xFF, 0x00, 0xFF }.
The receiver chip performs CRC16 check on the incoming packet.
Since I am expecting the exact same packet every time, I want to have zero CRC checksum when the packet is valid and not zero checksum when there is a transfer error.
I know that I can add the calculated CRC16 to the end of the packet when sending it and on the receiver side the CRC check will output 0, but in this case it is impossible to add a CRC16 checksum to the packet since the packet is constructed by multiple sender chips on the SPI line and each chip only fills its own two bytes from the entire packet.
I need to load an initial CRC checksum on the receiver side, so after the incoming packet is checked, the resulting CRC equals to zero (if packet is intact).
The answer here on SO is actually what I am looking for, but it is for CRC32 format and I don't actually understand the principle of the code, so I can't rewrite if for CRC16 format.
Any help would be greatly appreciated!
Regards,
Niko
The solution is simply to use a look-up table based CRC. If you can't append the checksum (aka the Frame Check Sequence, FCS) to the package, then do the table look-up first and then simply compare that one against the expected sequence for your fixed data.
Please note that "CRC 16" could mean anything, there are multiple versions and (non)standards. The most common one is perhaps the one called "CRC-16-CCITT" with 1021h poly and initial value FFFFh, but even for that one there's multiple algorithms out there - some are correct, some are broken. Your biggest challenge will be to find a trustworthy CRC algorithm.
However, I actually think SAMD specifically uses hardware-generated CRC-16-CCITT on-chip, for DMA purposes. Since this is SPI, it should be DMA-able, so perhaps investigate if you can use that one somehow.
I found a solution, thanks to the advice of Bastian Molkenthin, who did this great online CRC calculator.
He advised trying a brute force calculation of all the 2^16 values of a CRC16 initial value. Indeed, after a few lines of code and few microseconds later the SAMD51 found an initial value, which matches a zero CRC value for the given buffer.
I'm very curious about observation I've made during development of my app.
Long story short, I was making app which reads data from smart electric meters. They have request frame like:
7E [hex-address] [crc1] [cmd] [crc2] 7E
CRC's algoritm is CRC-16/X-25, and they should calculated of whole left part of frame without 7e, so crc2 includes crc1. I've noticed that all the time crc2 is the same for any cmd, even having different hex-address.
I've used crccalc tool to ensure. Steps to reproduce:
put any (even any size) hex array
calc CRC-16/X-25
swap CRC bytes
put this swapped CRC in end of array
calc CRC again
This CRC would be same for any byte array. Why so? Is there any name for this phenomenon?
calc CRC again
If you append a CRC to data and calculate it again, the result is a constant value. If the CRC isn't post complemented (xorout = 0), the result is zero, but in this case it is post complemented (xorout = 0xffff), so the result is a non-zero constant, in this case 0x0f47 (assuming no errors occurred).
Can anyone tell what flavor CRC-CCITT is implemented in this code?
When I use it to compute a CRC over "123456789", I get 0x6E90. But it doesn't match any of the CRCs here: https://www.lammertbies.nl/comm/info/crc-calculation.html
But it does exhibit the property of a CRC where including the CRC in the data that is CRC'd results in a known constant value.
The code shows a right shifting CRC16 based on 0x11021, which when bit reversed and shifted right 1 bit becomes 0x8408. This is a variation of CRC-CCITT (Kermit), but unlike Kermit, it starts with CRC = 0xFFFF (Kermit starts with CRC = 0x0000), and ones complement the CRC at the end, so it won't match any of the CRCs shown at the lammertbies web site. You can verify that it's similar to Kermit by going to the lammertbies web site, switching to hex mode and entering a single byte 80, which will result in 0x0884, which is a byte swapped 0x8408 due to little endian.
If it wasn't for the ones complement at the end, then appending a CRC to data and recalculating a CRC for the data + CRC would return a CRC of 0x0000, for any of the normal CRC methods. With the post ones complement, you get a constant depending on which CRC implementation is used, in this case 0x470F (byte swapped again due to little endian).
To test this, go to the lammertbies web site, prefix the string with hex 21 F3, which will cause the Kermit CRC to be 0xFFFF after processing the 21 F3. Switch to hex input and enter the string:
21F3313233343536373839
this will result in the Kermit CRC returning
0x916F
which when one's complemented is
0x6e90
You can change the Dr Dobb's example code to match Kermit with these changes:
crc = 0; /* instead of crc = 0xffff; */
/* ... */
/* crc = ~crc; comment out this line */
I am trying to build a fast 8-bit greyscale PNG encoder. Unfortunately I must be misunderstanding part of the spec. Smaller image sizes seem to work, but the larger ones will only open in some image viewers. This image (with multiple DEFLATE Blocks) gives a
"Decompression error in IDAT" error in my image viewer but opens fine in my browser:
This image has just one DEFLATE block but also gives an error:
Below I will outline what I put in my IDAT chunk in case you can easily spot any mistakes (note, images and steps have been modified based on answers, but there is still a problem):
IDAT length
"IDAT" in ascii (literally the bytes 0x49 0x44 0x41 0x54)
Zlib header 0x78 0x01
Steps 4-7 are for every deflate block, as the data may need to be broken up:
The byte 0x00 or 0x01, depending on if it is a middle or the last block.
Number of bytes in block (up to 2^16-1) stored as a little endian 16-bit integer
The 1's complement of this integer representation.
Image data (each scan-line is starts with a zero-byte for the no filter option in PNG, and is followed by width bytes of greyscale pixel data)
An adler-32 checksum of all the image data
A CRC of all the IDAT data
I've tried pngcheck on linux, an it does not spot any errors. If nobody can see what is wrong, can you point me in the right direction for a debugging tool?
My last resort is to use the libpng library to make my own decoder, and debug from there.
Some people have suggested it may be my adler-32 function calculation:
static uint32_t adler32(uint32_t height, uint32_t width, char** pixel_array)
{
uint32_t a=1,b=0,w,h;
for(h=0;h<height;h++)
{
b+=a;
for(w=0;w<width;w++)
{
a+=pixel_array[h][w];
b+=a;
}
}
return (uint32_t)(((b%65521)*65536)|(a%65521));
}
Note that because the pixel_array passed to the function does not contain the zero-byte at the beginning of each scanline (needed for PNG) there is an extra b+=a (and implicit a+=0) at the beginning of each iteration of the outer loop.
I do get an error with pngcheck: "zlib: inflate error = -3 (data error)". As your PNG scaffolding structure looks okay, it's time to take a low-level look into the IDAT block with a hex viewer. (I'm going to type this up while working through it.)
The header looks alright; IDAT length is okay. Your zlib flags are 78 01 ("No/low compression", see also What does a zlib header look like?), where one of my own tools use 78 9C ("Default compression"), but then again, these flags are only informative.
Next: zlib's internal blocks (per RFC1950).
Directly after the compression flags (CMF in RFC1950) it expects FLATE compressed data, which is the only compression scheme zlib supports. And that is in another castle RFC: RFC1951.
Each separately compression block is prepended by a byte:
3.2.3. Details of block format
Each block of compressed data begins with 3 header bits
containing the following data:
first bit BFINAL
next 2 bits BTYPE
...
BFINAL is set if and only if this is the last block of the data
set.
BTYPE specifies how the data are compressed, as follows:
00 - no compression
01 - compressed with fixed Huffman codes
10 - compressed with dynamic Huffman codes
11 - reserved (error)
So this value can be set to 00 for 'not last block, uncompressed' and to 01 for 'last block, uncompressed', immediately followed by the length (2 bytes) and its bitwise inverse, per 3.2.4. Non-compressed blocks (BTYPE=00):
3.2.4. Non-compressed blocks (BTYPE=00)
Any bits of input up to the next byte boundary are ignored.
The rest of the block consists of the following information:
0 1 2 3 4...
+---+---+---+---+================================+
| LEN | NLEN |... LEN bytes of literal data...|
+---+---+---+---+================================+
LEN is the number of data bytes in the block. NLEN is the
one's complement of LEN.
They are the final 4 bytes in your IDAT segment. Why do small images work, and larger not? Because you only have 2 bytes for the length.1 You need to break up your image into blocks no larger than 65,535 bytes (in my own PNG creator I seem to have used 32,768, probably "for safety"). If the last block, write out 01, else 00. Then add the two times two LEN bytes, properly encoded, followed by exactly LEN data bytes. Repeat until done.
The Adler-32 checksum is not part of this Flate-compressed data, and should not be counted in the blocks of LEN data. (It is still part of the IDAT block, though.)
After re-reading your question to verify I addressed all of your issues (and confirming I spelled "Adler-32" correctly), I realized you describe all of the steps right -- except that the 'last block' indicator is 01, not 80 (later edit: uh, perhaps you are right about that!) -- but that it does not show in this sample PNG. See if you can get it to work following all of the steps by the letter.
Kudos for doing this 'by hand'. It's a nice exercise in 'following the specs', and if you get this to work, it may be worthwhile to try and add proper compression. I shun pre-made libraries as much as possible; the only allowance I made for my own PNG encoder/decoder was to use Rich Geldreich's miniz.c, because implementing proper Flate encoding/decoding is beyond my ken.
1 That's not the whole story. Browsers are particularly forgiving in HTML errors; it seems they are as forgiving for PNG errors as well. Safari displays your image just fine, and so does Preview. But they may just all be sharing OS X's PNG decoder, because Photoshop rejects the file.
The byte 0x00 or 0x80, depending on if it is a middle or the last block.
Change the 0x80 to 0x01 and all will be well.
The 0x80 is appearing as a stored block that is not the last block. All that's being looked at is the low bit, which is zero, indicating a middle block. All of the data is in that "middle" block, so a decoder will recover the full image. Some liberal PNG decoders may then ignore the errors it gets when it tries to decode the next block, which isn't there, and then ignore the missing check values (Adler-32 and CRC-32), etc. That's why it shows up ok in browsers, even though it is an invalid PNG file.
There are two things wrong with your Adler-32 code. First, you are accessing the data from a char array. char is signed, so your 0xff bytes are being added not as 255, but rather as -127. You need to make the array unsigned char or cast it to that before extracting byte values from it.
Second, you are doing the modulo operation too late. You must do the % 65521 before the uint32_t overflows. Otherwise you don't get the modulo of the sum as required by the algorithm. A simple fix would be to do the % 65521 to a and b right after the width loop, inside the height loop. This will work so long as you can guarantee that the width will be less than 5551 bytes. (Why 5551 is left as an exercise for the reader.) If you cannot guarantee that, then you will need to embed a another loop to consume bytes from the line until you get to 5551 of them, do the modulo, and then continue with the line. Or, a smidge slower, just run a counter and do the modulo when it gets to the limit.
Here is an example of a version that works for any width:
static uint32_t adler32(uint32_t height, uint32_t width, unsigned char ** pixel_array)
{
uint32_t a = 1, b = 0, w, h, k;
for (h = 0; h < height; h++)
{
b += a;
w = k = 0;
while (k < width) {
k += 5551;
if (k > width)
k = width;
while (w < k) {
a += pixel_array[h][w++];
b += a;
}
a %= 65521;
b %= 65521;
}
}
return (b << 16) | a;
}
I'm trying to send data with a fixed-length header that tells the server how many bytes of data it's going to have to have available to read before it reads it. I'm having trouble doing this, though. The maximum number of bytes of data I want to be able to send at once is 65536, so I'm sending a uint16_t type variable as the header of my data because the maximum number it can represent is 65536.
The problem is, a uint16_t takes up two bytes, but numbers less than 255 only require one byte. So I have this code on the client side:
uint16_t messageSize = clientSendBuf.size(); //clientSendBuf is the data I want to send
char *bytes((char*)&messageSize);
clientSendBuf.prepend(bytes);
client.write(clientSendBuf);
And on the server, I handle receiving messages like this:
char serverReceiveBuf[65536];
uint16_t messageSize;
client->read((char*)&messageSize, sizeof(uint16_t));
client->read(serverReceiveBuf, messageSize);
I'm going to change this around a bit later because it's not the best solution (particularly for when all of the data isn't available yet), but I want to get this fixed first. My problem is that when clientSendBuf.size() is too small (in my test case it was 16 bytes, I assume this happens for every value under 255) reading data with
client->read((char*)&messageSize, sizeof(uint16_t));
reads a second byte that isn't part of the header, giving and incorrect value for messageSize and crashing the server. If I replace sizeof(uint16_t) with 1, then the server reads the data fine as I'd expect, although then I have a messageSize maximum of 255, which is much lower than I want. How do I make it so that the messageSize prepended to clientSendBuf is always two bytes, even for numbers <255?
Your
clientSendBuf.prepend(bytes);
Should also be told that it needs to send 2 bytes; now it treats the bytes as a zero-terminated string, which accidently works since on your platform the second byte of 0x0010 is zero (using little-endian numbers: 0x16, 0x00).
The prepend(char*, int) method will do the trick:
// use this instead:
cliendSendBuf.prepend(bytes, sizeof(messageSize));