Related
I need to implement CRC16-CCITT for HDLC, as part of developing some feature to control external HW (some type of LTE antenna).
The problem is that even though I have found many online implementations for CRC16-CCITT, the output values don't match the CRC values the HW vendor sent us.
For example, this is the first HDLC frame (named "Device Scan") which should be sent to the device, in hexadecimal format:
7EFFBF81F02A0113000000000000000000000000000000000000000313000000000000000000000000000000000000007F0F7E
As can be seen, the 7E in the start and end are the delimiters, and the last 4 hexadecimal characters are the CRC - which means 0x7F0F is the CRC result.
However, many online CRC calculators produce the result 0x6CC3
(take a look here for example: http://www.zorc.breitbandkatze.de/crc.html , https://www.lammertbies.nl/comm/info/crc-calculation)
Same goes for another command, which is the matching "receive" command for the above.
the RCV command is:
7E00BF81F018010F4B4D4B3032313941303035335A585806024B4D040130F1ED7E
which means that 0xF1ED is the CRC, and again it isn't what i see in other calculators, which produce 0xD814.
We have asked the vendor for clarifications, and he sent us this lookup table they are using:
crc table:
{
0x0000, 0x1189, 0x2312, 0x329b, 0x4624, 0x57ad, 0x6536, 0x74bf,
0x8c48, 0x9dc1, 0xaf5a, 0xbed3, 0xca6c, 0xdbe5, 0xe97e, 0xf8f7,
0x1081, 0x0108, 0x3393, 0x221a, 0x56a5, 0x472c, 0x75b7, 0x643e,
0x9cc9, 0x8d40, 0xbfdb, 0xae52, 0xdaed, 0xcb64, 0xf9ff, 0xe876,
0x2102, 0x308b, 0x0210, 0x1399, 0x6726, 0x76af, 0x4434, 0x55bd,
0xad4a, 0xbcc3, 0x8e58, 0x9fd1, 0xeb6e, 0xfae7, 0xc87c, 0xd9f5,
0x3183, 0x200a, 0x1291, 0x0318, 0x77a7, 0x662e, 0x54b5, 0x453c,
0xbdcb, 0xac42, 0x9ed9, 0x8f50, 0xfbef, 0xea66, 0xd8fd, 0xc974,
0x4204, 0x538d, 0x6116, 0x709f, 0x0420, 0x15a9, 0x2732, 0x36bb,
0xce4c, 0xdfc5, 0xed5e, 0xfcd7, 0x8868, 0x99e1, 0xab7a, 0xbaf3,
0x5285, 0x430c, 0x7197, 0x601e, 0x14a1, 0x0528, 0x37b3, 0x263a,
0xdecd, 0xcf44, 0xfddf, 0xec56, 0x98e9, 0x8960, 0xbbfb, 0xaa72,
0x6306, 0x728f, 0x4014, 0x519d, 0x2522, 0x34ab, 0x0630, 0x17b9,
0xef4e, 0xfec7, 0xcc5c, 0xddd5, 0xa96a, 0xb8e3, 0x8a78, 0x9bf1,
0x7387, 0x620e, 0x5095, 0x411c, 0x35a3, 0x242a, 0x16b1, 0x0738,
0xffcf, 0xee46, 0xdcdd, 0xcd54, 0xb9eb, 0xa862, 0x9af9, 0x8b70,
0x8408, 0x9581, 0xa71a, 0xb693, 0xc22c, 0xd3a5, 0xe13e, 0xf0b7,
0x0840, 0x19c9, 0x2b52, 0x3adb, 0x4e64, 0x5fed, 0x6d76, 0x7cff,
0x9489, 0x8500, 0xb79b, 0xa612, 0xd2ad, 0xc324, 0xf1bf, 0xe036,
0x18c1, 0x0948, 0x3bd3, 0x2a5a, 0x5ee5, 0x4f6c, 0x7df7, 0x6c7e,
0xa50a, 0xb483, 0x8618, 0x9791, 0xe32e, 0xf2a7, 0xc03c, 0xd1b5,
0x2942, 0x38cb, 0x0a50, 0x1bd9, 0x6f66, 0x7eef, 0x4c74, 0x5dfd,
0xb58b, 0xa402, 0x9699, 0x8710, 0xf3af, 0xe226, 0xd0bd, 0xc134,
0x39c3, 0x284a, 0x1ad1, 0x0b58, 0x7fe7, 0x6e6e, 0x5cf5, 0x4d7c,
0xc60c, 0xd785, 0xe51e, 0xf497, 0x8028, 0x91a1, 0xa33a, 0xb2b3,
0x4a44, 0x5bcd, 0x6956, 0x78df, 0x0c60, 0x1de9, 0x2f72, 0x3efb,
0xd68d, 0xc704, 0xf59f, 0xe416, 0x90a9, 0x8120, 0xb3bb, 0xa232,
0x5ac5, 0x4b4c, 0x79d7, 0x685e, 0x1ce1, 0x0d68, 0x3ff3, 0x2e7a,
0xe70e, 0xf687, 0xc41c, 0xd595, 0xa12a, 0xb0a3, 0x8238, 0x93b1,
0x6b46, 0x7acf, 0x4854, 0x59dd, 0x2d62, 0x3ceb, 0x0e70, 0x1ff9,
0xf78f, 0xe606, 0xd49d, 0xc514, 0xb1ab, 0xa022, 0x92b9, 0x8330,
0x7bc7, 0x6a4e, 0x58d5, 0x495c, 0x3de3, 0x2c6a, 0x1ef1, 0x0f78
}
This doesn't look like the regular table of CRC16-CCITT i found online, such as:
static const unsigned short CRC_CCITT_TABLE[256] =
{
0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50A5, 0x60C6, 0x70E7,
0x8108, 0x9129, 0xA14A, 0xB16B, 0xC18C, 0xD1AD, 0xE1CE, 0xF1EF,
0x1231, 0x0210, 0x3273, 0x2252, 0x52B5, 0x4294, 0x72F7, 0x62D6,
0x9339, 0x8318, 0xB37B, 0xA35A, 0xD3BD, 0xC39C, 0xF3FF, 0xE3DE,
0x2462, 0x3443, 0x0420, 0x1401, 0x64E6, 0x74C7, 0x44A4, 0x5485,
0xA56A, 0xB54B, 0x8528, 0x9509, 0xE5EE, 0xF5CF, 0xC5AC, 0xD58D,
0x3653, 0x2672, 0x1611, 0x0630, 0x76D7, 0x66F6, 0x5695, 0x46B4,
0xB75B, 0xA77A, 0x9719, 0x8738, 0xF7DF, 0xE7FE, 0xD79D, 0xC7BC,
0x48C4, 0x58E5, 0x6886, 0x78A7, 0x0840, 0x1861, 0x2802, 0x3823,
0xC9CC, 0xD9ED, 0xE98E, 0xF9AF, 0x8948, 0x9969, 0xA90A, 0xB92B,
0x5AF5, 0x4AD4, 0x7AB7, 0x6A96, 0x1A71, 0x0A50, 0x3A33, 0x2A12,
0xDBFD, 0xCBDC, 0xFBBF, 0xEB9E, 0x9B79, 0x8B58, 0xBB3B, 0xAB1A,
0x6CA6, 0x7C87, 0x4CE4, 0x5CC5, 0x2C22, 0x3C03, 0x0C60, 0x1C41,
0xEDAE, 0xFD8F, 0xCDEC, 0xDDCD, 0xAD2A, 0xBD0B, 0x8D68, 0x9D49,
0x7E97, 0x6EB6, 0x5ED5, 0x4EF4, 0x3E13, 0x2E32, 0x1E51, 0x0E70,
0xFF9F, 0xEFBE, 0xDFDD, 0xCFFC, 0xBF1B, 0xAF3A, 0x9F59, 0x8F78,
0x9188, 0x81A9, 0xB1CA, 0xA1EB, 0xD10C, 0xC12D, 0xF14E, 0xE16F,
0x1080, 0x00A1, 0x30C2, 0x20E3, 0x5004, 0x4025, 0x7046, 0x6067,
0x83B9, 0x9398, 0xA3FB, 0xB3DA, 0xC33D, 0xD31C, 0xE37F, 0xF35E,
0x02B1, 0x1290, 0x22F3, 0x32D2, 0x4235, 0x5214, 0x6277, 0x7256,
0xB5EA, 0xA5CB, 0x95A8, 0x8589, 0xF56E, 0xE54F, 0xD52C, 0xC50D,
0x34E2, 0x24C3, 0x14A0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405,
0xA7DB, 0xB7FA, 0x8799, 0x97B8, 0xE75F, 0xF77E, 0xC71D, 0xD73C,
0x26D3, 0x36F2, 0x0691, 0x16B0, 0x6657, 0x7676, 0x4615, 0x5634,
0xD94C, 0xC96D, 0xF90E, 0xE92F, 0x99C8, 0x89E9, 0xB98A, 0xA9AB,
0x5844, 0x4865, 0x7806, 0x6827, 0x18C0, 0x08E1, 0x3882, 0x28A3,
0xCB7D, 0xDB5C, 0xEB3F, 0xFB1E, 0x8BF9, 0x9BD8, 0xABBB, 0xBB9A,
0x4A75, 0x5A54, 0x6A37, 0x7A16, 0x0AF1, 0x1AD0, 0x2AB3, 0x3A92,
0xFD2E, 0xED0F, 0xDD6C, 0xCD4D, 0xBDAA, 0xAD8B, 0x9DE8, 0x8DC9,
0x7C26, 0x6C07, 0x5C64, 0x4C45, 0x3CA2, 0x2C83, 0x1CE0, 0x0CC1,
0xEF1F, 0xFF3E, 0xCF5D, 0xDF7C, 0xAF9B, 0xBFBA, 0x8FD9, 0x9FF8,
0x6E17, 0x7E36, 0x4E55, 0x5E74, 0x2E93, 0x3EB2, 0x0ED1, 0x1EF0
};
i tried to use their table instead of mine, in our CRC implementation, but we still get different results than theirs.
Does anyone understand what is going on here?
which CRC is the actual correct HDLC CRC16-CCITT?
What the the correct result of CRC16-CCITT for the above frames?
Thanks a lot guys!
I initially thought the poly was 0x1189, but instead it's a bit reversal of 0x1021 = 0x8408, which is seen at table[0x80]. Using the online CRC calculator linked to below, it's the predefined CRC_X_25, poly = 0x1021, input reflected, result reflected, initial value 0xFFFF, final xor 0xFFFF. Click on "show reflected lookup table", and you will get the same table as the vendor's table when you calculate a CRC using the calculator.
When appending the CRC to data, the CRC is appended as lower byte, upper byte. So a CRC of 0xEDF1 is appended as: 0xF1, 0xED, and a CRC of 0x0F7F, is appended as: 0x7F, 0x0F.
Because the CRC is post complemented (xorout = 0xFFFF), if checking the CRC by recalculating it on data + CRC, if there are no errors, the CRC will be a non-zero constant, 0x0F47.
http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
Example code:
#include <iostream>
#include <iomanip>
uint16_t crctbl[256];
void gentbl()
{
uint16_t crc, i, j;
for(j = 0; j < 0x100; j++){
crc = j;
for(i = 0; i < 8; i++)
crc = (crc>>1)^((0-(crc&1))&0x8408);
crctbl[j] = crc;
}
}
uint16_t crc16(uint8_t * bfr, size_t size)
{
uint16_t crc = 0xffff;
while (size--)
crc = (crc>>8)^crctbl[(crc^*bfr++)&0xff];
return(crc ^ 0xffff);
}
int main(int argc, char**argv)
{
uint16_t crc;
// bfr has 2 trailing zeroes to store crc into
uint8_t bfr[] = {0x00,0xBF,0x81,0xF0,0x18,0x01,0x0F,0x4B,
0x4D,0x4B,0x30,0x32,0x31,0x39,0x41,0x30,
0x30,0x35,0x33,0x5A,0x58,0x58,0x06,0x02,
0x4B,0x4D,0x04,0x01,0x30,0x00,0x00};
gentbl();
crc = crc16(bfr, sizeof(bfr)-2);
std::cout << std::hex << crc << std::endl;
bfr[sizeof(bfr)-2] = crc&0xff;
bfr[sizeof(bfr)-1] = crc>>8;
crc = crc16(bfr, sizeof(bfr));
std::cout << std::hex << crc << std::endl;
if(crc != 0xf47)
std::cout << "error" << std::endl;
return(0);
}
I am trying to convert c++ code into python.
int fd_vdma = open("/dev/mem", O_RDWR|O_SYNC); // open uiox device for vdma access
if (fd_vdma < 1) {
printf("Invalid mem device file\n");
}
// mmap the vdma device for vdma access
unsigned int *ptr_vdma;
ptr_vdma = (unsigned int*)mmap(NULL, VDMA_MAP_SIZE,PROT_READ|PROT_WRITE,
MAP_SHARED, fd_vdma, VDMA_ADDR);
printf("DMA 1 virtual address: 0x%08x \n",ptr_vdma);
*(ptr_vdma+5) = FRBUF_ADDR_0;
*(ptr_vdma+7) = 2; // use internal fifos to trigger xfer
*(ptr_vdma+8) = 20480;
*(ptr_vdma+6) = (75 << 16) + (HORIZ_PIXELS_SMALL+75);
*(ptr_vdma+0x0D) = 200; // no. FIFO threshhold .. max.. 240
I have done so far,
# vdm memory check
try:
fd_vdm_path = "/dev/mem"
mode = "rb+"
fd_vdm = open(fd_vdm_path, mode)
# print(fd_vdm.fileno())
print("[INFO] " + fd_vdm_path + " checked")
except Exception as error:
print("{}".format(error))
# mmap the VDMA device for VDM access
vdma_buf = mmap.mmap(fd_vdm.fileno(), int(VDMA_MAP_SIZE),
mmap.MAP_SHARED, mmap.PROT_READ|mmap.PROT_WRITE, 0)
ptr_vdm = ctypes.c_uint.from_buffer(vdma_buf)
print(type(ctypes.addressof(ptr_vdm)))
print("[INFO] " + fd_vdm_path + " has allocated virtual address : " +
hex(ctypes.addressof(ptr_vdm)))
but I have stuck here, I do not know how to perform below operations in python.
*(ptr_vdma+5) = FRBUF_ADDR_0;
*(ptr_vdma+7) = 2; // use internal fifos to trigger xfer
*(ptr_vdma+8) = 20480;
*(ptr_vdma+6) = (75 << 16) + (HORIZ_PIXELS_SMALL+75);
*(ptr_vdma+0x0D) = 200; // no. FIFO threshhold .. max.. 240
I have tried to vdma_buf.write(FRBUF_ADDR_0) but it is not working and I could not find anything on the internet (possibly I did not search it well)
can you please help with this problem or else suggest me some links or tutorials?
You can use memoryview and struct.pack:
...
mm = memoryview(vdma_buf)
mm[5*4:6*4] = struct.pack("I", FRBUF_ADDR_0)
mm[7*4:8*4] = struct.pack("I", 2)
mm[8*4:9*4] = struct.pack("I", 20480)
mm[6*4:7*4] = struct.pack("I", (75 << 16) + (HORIZ_PIXELS_SMALL+75))
mm[0x0D*4:0x0E*4] = struct.pack("I", 200)
memoryview represents sequence of bytes, and if you want to write some other type to memory, you must convert it to bytes using struct.pack.
So, let's say you want to to write an unsigned int at index N.
First, you call struct.pack("I", ...) to get int's byte representation. Then you must calculate destination memory address. Because unsigned int's size is 4 bytes, the address is equal to N*4.
memoryview supports slicing so you can just write:
mm[StartAddress:EndAddress] = BytesYouWantToWrite
where StartAddress is equal to N*4 and EndAddress is (N+1) * 4
I need to compute a hash value for a set of integer numbers which their order is the subject of interest. The integers might be a lot so I want to sum up them! Specifically, I am writing my code in c++. Any suggestion for a simple algorithm is appreciated.
Assuming that you want to output an integer, you could do worse (and better, but at least it's simple) than something like:
unsigned int hash = 0;
for each int i // not sure how you're storing them
{
hash = ((31 * hash) + (unsigned)i);
}
If your integers are i_1, ... i_n then this computes
31^n * i_1 + 31^(n-1) * i_2 + ... + 31 * i_(n-1) + i_n
i.e. order is significant.
Casting to an unsigned type provides defined overflow behaviour, as opposed to undefined behaviour (in C at any rate; not sure about C++ off the top of my head).
That's a fairly standard algorithm, I think described by Sedgewick.
(Nit-picking, but if order is important then you don't have a set.)
You can use a CRC function. Here is a 64bit implementation.
Header information:
typedef unsigned long long DWORD64 //maybe this is needed
inline void InitCRC64(DWORD64 &crc)
{
crc = 0xFFFFFFFFFFFFFFFFULL;
}
extern const DWORD64 crctab64[256];
/**
* CRC64 function
*
* #param[in] data pointer to the buffer
* #param[in] len length of buffer in bytes
* #param[in,out] hash in = initial value of the hash; out = returned hash
*/
inline void GetCRC64(DWORD64 &crc, const unsigned char *cp, unsigned long length)
{
while (length--)
crc = crctab64[(crc ^ *(cp++)) & 0xff] ^ (crc >> 8);
}
Source File:
/*
* poly 0x95AC9329AC4BC9B5ULL and init 0xFFFFFFFFFFFFFFFFULL
*/
const DWORD64 crctab64[256] = {
0x0000000000000000ULL, 0x7ad870c830358979ULL, 0xf5b0e190606b12f2ULL,
0x8f689158505e9b8bULL, 0xc038e5739841b68fULL, 0xbae095bba8743ff6ULL,
0x358804e3f82aa47dULL, 0x4f50742bc81f2d04ULL, 0xab28ecb46814fe75ULL,
0xd1f09c7c5821770cULL, 0x5e980d24087fec87ULL, 0x24407dec384a65feULL,
0x6b1009c7f05548faULL, 0x11c8790fc060c183ULL, 0x9ea0e857903e5a08ULL,
0xe478989fa00bd371ULL, 0x7d08ff3b88be6f81ULL, 0x07d08ff3b88be6f8ULL,
0x88b81eabe8d57d73ULL, 0xf2606e63d8e0f40aULL, 0xbd301a4810ffd90eULL,
0xc7e86a8020ca5077ULL, 0x4880fbd87094cbfcULL, 0x32588b1040a14285ULL,
0xd620138fe0aa91f4ULL, 0xacf86347d09f188dULL, 0x2390f21f80c18306ULL,
0x594882d7b0f40a7fULL, 0x1618f6fc78eb277bULL, 0x6cc0863448deae02ULL,
0xe3a8176c18803589ULL, 0x997067a428b5bcf0ULL, 0xfa11fe77117cdf02ULL,
0x80c98ebf2149567bULL, 0x0fa11fe77117cdf0ULL, 0x75796f2f41224489ULL,
0x3a291b04893d698dULL, 0x40f16bccb908e0f4ULL, 0xcf99fa94e9567b7fULL,
0xb5418a5cd963f206ULL, 0x513912c379682177ULL, 0x2be1620b495da80eULL,
0xa489f35319033385ULL, 0xde51839b2936bafcULL, 0x9101f7b0e12997f8ULL,
0xebd98778d11c1e81ULL, 0x64b116208142850aULL, 0x1e6966e8b1770c73ULL,
0x8719014c99c2b083ULL, 0xfdc17184a9f739faULL, 0x72a9e0dcf9a9a271ULL,
0x08719014c99c2b08ULL, 0x4721e43f0183060cULL, 0x3df994f731b68f75ULL,
0xb29105af61e814feULL, 0xc849756751dd9d87ULL, 0x2c31edf8f1d64ef6ULL,
0x56e99d30c1e3c78fULL, 0xd9810c6891bd5c04ULL, 0xa3597ca0a188d57dULL,
0xec09088b6997f879ULL, 0x96d1784359a27100ULL, 0x19b9e91b09fcea8bULL,
0x636199d339c963f2ULL, 0xdf7adabd7a6e2d6fULL, 0xa5a2aa754a5ba416ULL,
0x2aca3b2d1a053f9dULL, 0x50124be52a30b6e4ULL, 0x1f423fcee22f9be0ULL,
0x659a4f06d21a1299ULL, 0xeaf2de5e82448912ULL, 0x902aae96b271006bULL,
0x74523609127ad31aULL, 0x0e8a46c1224f5a63ULL, 0x81e2d7997211c1e8ULL,
0xfb3aa75142244891ULL, 0xb46ad37a8a3b6595ULL, 0xceb2a3b2ba0eececULL,
0x41da32eaea507767ULL, 0x3b024222da65fe1eULL, 0xa2722586f2d042eeULL,
0xd8aa554ec2e5cb97ULL, 0x57c2c41692bb501cULL, 0x2d1ab4dea28ed965ULL,
0x624ac0f56a91f461ULL, 0x1892b03d5aa47d18ULL, 0x97fa21650afae693ULL,
0xed2251ad3acf6feaULL, 0x095ac9329ac4bc9bULL, 0x7382b9faaaf135e2ULL,
0xfcea28a2faafae69ULL, 0x8632586aca9a2710ULL, 0xc9622c4102850a14ULL,
0xb3ba5c8932b0836dULL, 0x3cd2cdd162ee18e6ULL, 0x460abd1952db919fULL,
0x256b24ca6b12f26dULL, 0x5fb354025b277b14ULL, 0xd0dbc55a0b79e09fULL,
0xaa03b5923b4c69e6ULL, 0xe553c1b9f35344e2ULL, 0x9f8bb171c366cd9bULL,
0x10e3202993385610ULL, 0x6a3b50e1a30ddf69ULL, 0x8e43c87e03060c18ULL,
0xf49bb8b633338561ULL, 0x7bf329ee636d1eeaULL, 0x012b592653589793ULL,
0x4e7b2d0d9b47ba97ULL, 0x34a35dc5ab7233eeULL, 0xbbcbcc9dfb2ca865ULL,
0xc113bc55cb19211cULL, 0x5863dbf1e3ac9decULL, 0x22bbab39d3991495ULL,
0xadd33a6183c78f1eULL, 0xd70b4aa9b3f20667ULL, 0x985b3e827bed2b63ULL,
0xe2834e4a4bd8a21aULL, 0x6debdf121b863991ULL, 0x1733afda2bb3b0e8ULL,
0xf34b37458bb86399ULL, 0x8993478dbb8deae0ULL, 0x06fbd6d5ebd3716bULL,
0x7c23a61ddbe6f812ULL, 0x3373d23613f9d516ULL, 0x49aba2fe23cc5c6fULL,
0xc6c333a67392c7e4ULL, 0xbc1b436e43a74e9dULL, 0x95ac9329ac4bc9b5ULL,
0xef74e3e19c7e40ccULL, 0x601c72b9cc20db47ULL, 0x1ac40271fc15523eULL,
0x5594765a340a7f3aULL, 0x2f4c0692043ff643ULL, 0xa02497ca54616dc8ULL,
0xdafce7026454e4b1ULL, 0x3e847f9dc45f37c0ULL, 0x445c0f55f46abeb9ULL,
0xcb349e0da4342532ULL, 0xb1eceec59401ac4bULL, 0xfebc9aee5c1e814fULL,
0x8464ea266c2b0836ULL, 0x0b0c7b7e3c7593bdULL, 0x71d40bb60c401ac4ULL,
0xe8a46c1224f5a634ULL, 0x927c1cda14c02f4dULL, 0x1d148d82449eb4c6ULL,
0x67ccfd4a74ab3dbfULL, 0x289c8961bcb410bbULL, 0x5244f9a98c8199c2ULL,
0xdd2c68f1dcdf0249ULL, 0xa7f41839ecea8b30ULL, 0x438c80a64ce15841ULL,
0x3954f06e7cd4d138ULL, 0xb63c61362c8a4ab3ULL, 0xcce411fe1cbfc3caULL,
0x83b465d5d4a0eeceULL, 0xf96c151de49567b7ULL, 0x76048445b4cbfc3cULL,
0x0cdcf48d84fe7545ULL, 0x6fbd6d5ebd3716b7ULL, 0x15651d968d029fceULL,
0x9a0d8ccedd5c0445ULL, 0xe0d5fc06ed698d3cULL, 0xaf85882d2576a038ULL,
0xd55df8e515432941ULL, 0x5a3569bd451db2caULL, 0x20ed197575283bb3ULL,
0xc49581ead523e8c2ULL, 0xbe4df122e51661bbULL, 0x3125607ab548fa30ULL,
0x4bfd10b2857d7349ULL, 0x04ad64994d625e4dULL, 0x7e7514517d57d734ULL,
0xf11d85092d094cbfULL, 0x8bc5f5c11d3cc5c6ULL, 0x12b5926535897936ULL,
0x686de2ad05bcf04fULL, 0xe70573f555e26bc4ULL, 0x9ddd033d65d7e2bdULL,
0xd28d7716adc8cfb9ULL, 0xa85507de9dfd46c0ULL, 0x273d9686cda3dd4bULL,
0x5de5e64efd965432ULL, 0xb99d7ed15d9d8743ULL, 0xc3450e196da80e3aULL,
0x4c2d9f413df695b1ULL, 0x36f5ef890dc31cc8ULL, 0x79a59ba2c5dc31ccULL,
0x037deb6af5e9b8b5ULL, 0x8c157a32a5b7233eULL, 0xf6cd0afa9582aa47ULL,
0x4ad64994d625e4daULL, 0x300e395ce6106da3ULL, 0xbf66a804b64ef628ULL,
0xc5bed8cc867b7f51ULL, 0x8aeeace74e645255ULL, 0xf036dc2f7e51db2cULL,
0x7f5e4d772e0f40a7ULL, 0x05863dbf1e3ac9deULL, 0xe1fea520be311aafULL,
0x9b26d5e88e0493d6ULL, 0x144e44b0de5a085dULL, 0x6e963478ee6f8124ULL,
0x21c640532670ac20ULL, 0x5b1e309b16452559ULL, 0xd476a1c3461bbed2ULL,
0xaeaed10b762e37abULL, 0x37deb6af5e9b8b5bULL, 0x4d06c6676eae0222ULL,
0xc26e573f3ef099a9ULL, 0xb8b627f70ec510d0ULL, 0xf7e653dcc6da3dd4ULL,
0x8d3e2314f6efb4adULL, 0x0256b24ca6b12f26ULL, 0x788ec2849684a65fULL,
0x9cf65a1b368f752eULL, 0xe62e2ad306bafc57ULL, 0x6946bb8b56e467dcULL,
0x139ecb4366d1eea5ULL, 0x5ccebf68aecec3a1ULL, 0x2616cfa09efb4ad8ULL,
0xa97e5ef8cea5d153ULL, 0xd3a62e30fe90582aULL, 0xb0c7b7e3c7593bd8ULL,
0xca1fc72bf76cb2a1ULL, 0x45775673a732292aULL, 0x3faf26bb9707a053ULL,
0x70ff52905f188d57ULL, 0x0a2722586f2d042eULL, 0x854fb3003f739fa5ULL,
0xff97c3c80f4616dcULL, 0x1bef5b57af4dc5adULL, 0x61372b9f9f784cd4ULL,
0xee5fbac7cf26d75fULL, 0x9487ca0fff135e26ULL, 0xdbd7be24370c7322ULL,
0xa10fceec0739fa5bULL, 0x2e675fb4576761d0ULL, 0x54bf2f7c6752e8a9ULL,
0xcdcf48d84fe75459ULL, 0xb71738107fd2dd20ULL, 0x387fa9482f8c46abULL,
0x42a7d9801fb9cfd2ULL, 0x0df7adabd7a6e2d6ULL, 0x772fdd63e7936bafULL,
0xf8474c3bb7cdf024ULL, 0x829f3cf387f8795dULL, 0x66e7a46c27f3aa2cULL,
0x1c3fd4a417c62355ULL, 0x935745fc4798b8deULL, 0xe98f353477ad31a7ULL,
0xa6df411fbfb21ca3ULL, 0xdc0731d78f8795daULL, 0x536fa08fdfd90e51ULL,
0x29b7d047efec8728ULL
};
I'm working on a function that can uncompress the deflate compression, so i can read/draw png files in my c++ program. However, the deflate specification isn't very clear on some things.
So my main question is:
Paragraph 3.2.7. Compression with dynamic Huffman codes (BTYPE=10) of the specification state that
the distance code follows the literal/length
But it does not state how many bits the distance code occupy, is it an entire byte?
And how does the distance code relate?.. whats its use, really?
Any one have a general explanation? since the specification is kinda lacking in clarity.
The specification i found here:
http://www.ietf.org/rfc/rfc1951.txt
Edit (Here is my following code to use with puff inflate code.)
First the header (ConceptApp.h)
#include "resource.h"
#ifdef _WIN64
typedef unsigned long long SIZE_PTR;
#else
typedef unsigned long SIZE_PTR;
#endif
typedef struct _IMAGE {
DWORD Width; //Width in pixels.
DWORD Height; //Height in pixels.
DWORD BitsPerPixel; //24 (RGB), 32 (RGBA).
DWORD Planes; //Count of color planes
PBYTE Pixels; //Pointer to the first pixel of the image.
} IMAGE, *PIMAGE;
typedef DWORD LodePNGColorType;
typedef struct _LodePNGColorMode {
DWORD colortype;
DWORD bitdepth;
} LodePNGColorMode;
typedef struct LodePNGInfo
{
/*header (IHDR), palette (PLTE) and transparency (tRNS) chunks*/
unsigned compression_method;/*compression method of the original file. Always 0.*/
unsigned filter_method; /*filter method of the original file*/
unsigned interlace_method; /*interlace method of the original file*/
LodePNGColorMode color; /*color type and bits, palette and transparency of the PNG file*/
} LodePNGInfo;
typedef struct _ZLIB {
BYTE CMF;
BYTE FLG;
//DWORD DICTID; //if FLG.FDICT (Bit 5) is set, this variable follows.
//Compressed data here...
} ZLIB, *PZLIB;
typedef struct _PNG_IHDR {
DWORD Width;
DWORD Height;
BYTE BitDepth;
BYTE ColourType;
BYTE CompressionMethod;
BYTE FilterMethod;
BYTE InterlaceMethod;
} PNG_IHDR, *PPNG_IHDR;
typedef struct _PNG_CHUNK {
DWORD Length;
CHAR ChuckType[4];
} PNG_CHUNK, *PPNG_CHUNK;
typedef struct _PNG {
BYTE Signature[8];
PNG_CHUNK FirstChunk;
} PNG, *PPNG;
And the code .cpp file:
The main function can be found at the bottom of the file (LoadPng)
BYTE LoadPng(PPNG PngFile, PIMAGE ImageData)
{
PDWORD Pixel = 0;
DWORD ChunkSize = 0;
PPNG_IHDR PngIhdr = (PPNG_IHDR) ((SIZE_PTR) &PngFile->FirstChunk + sizeof(PNG_CHUNK));
DWORD Png_Width = Png_ReadDword((PBYTE)&PngIhdr->Width);
DWORD Png_Height = Png_ReadDword((PBYTE)&PngIhdr->Height);
DWORD BufferSize = (Png_Width*Png_Height) * 8; //This just a guess right now, havent done the math yet. !!!
ChunkSize = Png_ReadDword((PBYTE)&PngFile->FirstChunk.Length);
PPNG_CHUNK ThisChunk = (PPNG_CHUNK) ((SIZE_PTR)&PngFile->FirstChunk + ChunkSize + 12); //12 is the length var itself, Chunktype and CRC.
PPNG_CHUNK NextChunk;
PBYTE UncompressedData = (PBYTE) malloc(BufferSize);
INT RetValue = 0;
do
{
ChunkSize = Png_ReadDword((PBYTE)&ThisChunk->Length);
NextChunk = (PPNG_CHUNK) ((SIZE_PTR)ThisChunk + ChunkSize + 12); //12 is the length var itself, Chunktype and CRC.
if (Png_IsChunk(ThisChunk->ChuckType, "IDAT")) //Is IDAT ?
{
PZLIB iData = (PZLIB) ((SIZE_PTR)ThisChunk + 8); //8 is the length and chunkType.
PBYTE FirstBlock; //ponter to the first 3 bits of the deflate stuff.
if ((iData->CMF & 8) == 8) //deflate compression method.
{
if ((iData->FLG & 0x20) == 0x20)
{
FirstBlock = (PBYTE) ((SIZE_PTR)iData + 6); //DICTID Present.
}
else FirstBlock = (PBYTE) ((SIZE_PTR)iData + 2); //DICTID Not present.
RetValue = puff(UncompressedData, &BufferSize, FirstBlock, &ChunkSize); //I belive chunksize should be fine.
if (RetValue != 0)
{
WCHAR ErrorText[100];
swprintf_s(ErrorText, 100, L"%u", RetValue); //Convert data into string.
MessageBox(NULL, ErrorText, NULL, MB_OK);
}
}
}
ThisChunk = NextChunk;
} while (!Png_IsChunk(ThisChunk->ChuckType, "IEND"));
//LodePNGInfo ImageInfo;
//PBYTE Png_Real_Image = (PBYTE) malloc(BufferSize);
//ImageInfo.compression_method = PngIhdr->CompressionMethod;
//ImageInfo.filter_method = PngIhdr->FilterMethod;
//ImageInfo.interlace_method = PngIhdr->InterlaceMethod;
//ImageInfo.color.bitdepth = PngIhdr->BitDepth;
//ImageInfo.color.colortype = PngIhdr->ColourType;
//Remove Filter/crap blah blah.
//postProcessScanlines(Png_Real_Image, UncompressedData, Png_Width, Png_Height, &ImageInfo);
ImageData->Width = Png_Width;
ImageData->Height = Png_Height;
ImageData->Planes = 0; //Will need changed later.
ImageData->BitsPerPixel = 32; //Will need changed later.
ImageData->Pixels = 0;
//ImageData->Pixels = Png_Real_Image; //image not uncompressed yet.
return TRUE; //ret true for now. fix later.
}
I just hope to make clearer what is stated before--Huffman coding is a method for encoding values using a variable number of bits. In, say, ASCII coding, every letter gets the same number of bits no matter how frequently it is used. In Huffman coding, you could make "e" have fewer bits than an "X".
The trick in huffman coding is how the codes are prefixed. After reading each bit, the decoder knows, unambiguously, whether it has a value or needs to read another bit.
To comprehend the deflate process you need to understand LZ algorithm and Huffman coding.
On their own, both techniques are simple. The complexity comes from how they are put together.
LZ compresses by finding previous occurrences of a string. When a string has occurred previously, it is compressed by referencing the previous occurrence. The Distance is the offset to the previous occurrence. Distance and length specify that occurrence.
The problem is not with puff.
All the IDAT chunks in the png file need to be put together before calling puff.
It should look something like this:
BYTE LoadPng(PPNG PngFile, PIMAGE ImageData)
{
PDWORD Pixel = 0;
DWORD ChunkSize = 0;
PPNG_IHDR PngIhdr = (PPNG_IHDR) ((SIZE_PTR) &PngFile->FirstChunk + sizeof(PNG_CHUNK));
DWORD Png_Width = Png_ReadDword((PBYTE)&PngIhdr->Width);
DWORD Png_Height = Png_ReadDword((PBYTE)&PngIhdr->Height);
DWORD BufferSize = (Png_Width*Png_Height) * 8; //This just a guess right now, havent done the math yet. !!!
ChunkSize = Png_ReadDword((PBYTE)&PngFile->FirstChunk.Length);
PPNG_CHUNK ThisChunk = (PPNG_CHUNK) ((SIZE_PTR)&PngFile->FirstChunk + ChunkSize + 12); //12 is the length var itself, Chunktype and CRC.
PPNG_CHUNK NextChunk;
PBYTE UncompressedData = (PBYTE) malloc(BufferSize);
PBYTE TempBuffer = (PBYTE) malloc(BufferSize); //Put all idat chunks together befor uncompressing.
DWORD DeflateSize = 0; //All IDAT Chunks Added.
PZLIB iData = NULL;
PBYTE FirstBlock = NULL; //ponter to the first 3 bits of the deflate stuff.
INT RetValue = 0;
do
{
ChunkSize = Png_ReadDword((PBYTE)&ThisChunk->Length);
NextChunk = (PPNG_CHUNK) ((SIZE_PTR)ThisChunk + ChunkSize + 12); //12 is the length var itself, Chunktype and CRC.
if (Png_IsChunk(ThisChunk->ChuckType, "IDAT")) //Is IDAT ?
{
CopyMemory(&TempBuffer[DeflateSize], (PBYTE) ((SIZE_PTR)ThisChunk + 8), ChunkSize); //8 is the length and chunkType.
DeflateSize += ChunkSize;
}
ThisChunk = NextChunk;
} while (!Png_IsChunk(ThisChunk->ChuckType, "IEND"));
iData = (PZLIB) TempBuffer;
if ((iData->CMF & 8) == 8) //deflate compression method.
{
if ((iData->FLG & 0x20) == 0x20)
{
FirstBlock = (PBYTE) ((SIZE_PTR)iData + 6); //DICTID Present.
}
else FirstBlock = (PBYTE) ((SIZE_PTR)iData + 2); //DICTID Not present.
}
RetValue = puff(UncompressedData, &BufferSize, FirstBlock, &DeflateSize); //I belive chunksize should be fine.
if (RetValue != 0)
{
WCHAR ErrorText[100];
swprintf_s(ErrorText, 100, L"%u", RetValue);
MessageBox(NULL, ErrorText, NULL, MB_OK);
}
//LodePNGInfo ImageInfo;
//PBYTE Png_Real_Image = (PBYTE) malloc(BufferSize);
//ImageInfo.compression_method = PngIhdr->CompressionMethod;
//ImageInfo.filter_method = PngIhdr->FilterMethod;
//ImageInfo.interlace_method = PngIhdr->InterlaceMethod;
//ImageInfo.color.bitdepth = PngIhdr->BitDepth;
//ImageInfo.color.colortype = PngIhdr->ColourType;
//Remove Filter/crap blah blah.
//postProcessScanlines(Png_Real_Image, UncompressedData, Png_Width, Png_Height, &ImageInfo);
ImageData->Width = Png_Width;
ImageData->Height = Png_Height;
ImageData->Planes = 0; //Will need changed later.
ImageData->BitsPerPixel = 32; //Will need changed later.
ImageData->Pixels = 0;
//ImageData->Pixels = Png_Real_Image; //image not uncompressed yet.
return TRUE; //ret true for now. fix later.
}
You need to first read up on compression, since there is a lot of basic stuff that you're not getting. E.g. The Data Compression Book, by Nelson and Gailly.
Since it's a code, specifically a Huffman code, by definition the number of bits are variable.
If you don't know what the distance is for, then you need to first understand the LZ77 compression approach.
Lastly, aside from curiosity and self-education, there is no need for you to understand the deflate specification or to write your own inflate code. That's what zlib is for.
I am using the following function to generate a CRC sum and it doesn't appear to be returning the same checksum when compared to online CRC-CCITT calculators.
This function specifically uses the XMODEM CRC generation with a 0x8408 polynomial with an initial fcs of 0xFFFF.
uint16_t crc16(uint8_t byte, uint16_t fcs)
{
uint8_t bit;
for(bit=0; bit<8; bit++)
{
fcs ^= (byte & 0x01);
fcs = (fcs & 0x01) ? (fcs >> 1) ^ 0x8408 : (fcs >> 1);
byte = byte >> 1;
}
return fcs;
}
Am I doing something wrong? If I send 0xFF, or 0x00 I do not get the same checksum as I do on http://depa.usst.edu.cn/chenjq/www2/SDesign/JavaScript/CRCcalculation.htm
printf("%04X\n", crc16(0x31, 0xFFFF)); //returns 2F8D
I have read your questions and I also had the similar problem like you.
I have solved this issue for calculating CRC-CCITT in XMODEM. here I am attaching the sample program to calculate CRC-CCITT.
I have tried the data with the online converter and this program. Please use this if you wish.
unsigned short crc16(char *ptr, int count)
{
int crc;
char i;
crc = 0;
while (--count >= 0)
{
crc = crc ^ (int) *ptr++ << 8;
i = 8;
do
{
if (crc & 0x8000)
crc = crc << 1 ^ 0x1021;
else
crc = crc << 1;
} while(--i);
}
return (crc);
}
CRC should be defined as unsigned short since the crc16 function returns an unsigned short. CRC is defined as an int which on most systems is 4 bytes.
Take a look at Greg Cook's excellent catalog of CRCs. There is a variant often falsely identified as the CCITT CRC, which it isn't. That is what your code, with the 0xFFFF initialization, appears to be computing, though reflected. The Kermit CRC is the actual CCITT CRC. To get the CCITT CRC, you should start with zero, not 0xFFFF. The XMODEM CRC is different still, like the Kermit CRC, but unreflected (so bits go in the top, and you exclusive-or with 0x1021).
KERMIT
width=16 poly=0x1021 init=0x0000 refin=true refout=true xorout=0x0000 check=0x2189 name="KERMIT"
XMODEM
width=16 poly=0x1021 init=0x0000 refin=false refout=false xorout=0x0000 check=0x31c3 name="XMODEM"
CRC-16/CCITT-FALSE
width=16 poly=0x1021 init=0xffff refin=false refout=false xorout=0x0000 check=0x29b1 name="CRC-16/CCITT-FALSE"
static const unsigned short CRC_CCITT_TABLE[256] =
{
0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50A5, 0x60C6, 0x70E7,
0x8108, 0x9129, 0xA14A, 0xB16B, 0xC18C, 0xD1AD, 0xE1CE, 0xF1EF,
0x1231, 0x0210, 0x3273, 0x2252, 0x52B5, 0x4294, 0x72F7, 0x62D6,
0x9339, 0x8318, 0xB37B, 0xA35A, 0xD3BD, 0xC39C, 0xF3FF, 0xE3DE,
0x2462, 0x3443, 0x0420, 0x1401, 0x64E6, 0x74C7, 0x44A4, 0x5485,
0xA56A, 0xB54B, 0x8528, 0x9509, 0xE5EE, 0xF5CF, 0xC5AC, 0xD58D,
0x3653, 0x2672, 0x1611, 0x0630, 0x76D7, 0x66F6, 0x5695, 0x46B4,
0xB75B, 0xA77A, 0x9719, 0x8738, 0xF7DF, 0xE7FE, 0xD79D, 0xC7BC,
0x48C4, 0x58E5, 0x6886, 0x78A7, 0x0840, 0x1861, 0x2802, 0x3823,
0xC9CC, 0xD9ED, 0xE98E, 0xF9AF, 0x8948, 0x9969, 0xA90A, 0xB92B,
0x5AF5, 0x4AD4, 0x7AB7, 0x6A96, 0x1A71, 0x0A50, 0x3A33, 0x2A12,
0xDBFD, 0xCBDC, 0xFBBF, 0xEB9E, 0x9B79, 0x8B58, 0xBB3B, 0xAB1A,
0x6CA6, 0x7C87, 0x4CE4, 0x5CC5, 0x2C22, 0x3C03, 0x0C60, 0x1C41,
0xEDAE, 0xFD8F, 0xCDEC, 0xDDCD, 0xAD2A, 0xBD0B, 0x8D68, 0x9D49,
0x7E97, 0x6EB6, 0x5ED5, 0x4EF4, 0x3E13, 0x2E32, 0x1E51, 0x0E70,
0xFF9F, 0xEFBE, 0xDFDD, 0xCFFC, 0xBF1B, 0xAF3A, 0x9F59, 0x8F78,
0x9188, 0x81A9, 0xB1CA, 0xA1EB, 0xD10C, 0xC12D, 0xF14E, 0xE16F,
0x1080, 0x00A1, 0x30C2, 0x20E3, 0x5004, 0x4025, 0x7046, 0x6067,
0x83B9, 0x9398, 0xA3FB, 0xB3DA, 0xC33D, 0xD31C, 0xE37F, 0xF35E,
0x02B1, 0x1290, 0x22F3, 0x32D2, 0x4235, 0x5214, 0x6277, 0x7256,
0xB5EA, 0xA5CB, 0x95A8, 0x8589, 0xF56E, 0xE54F, 0xD52C, 0xC50D,
0x34E2, 0x24C3, 0x14A0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405,
0xA7DB, 0xB7FA, 0x8799, 0x97B8, 0xE75F, 0xF77E, 0xC71D, 0xD73C,
0x26D3, 0x36F2, 0x0691, 0x16B0, 0x6657, 0x7676, 0x4615, 0x5634,
0xD94C, 0xC96D, 0xF90E, 0xE92F, 0x99C8, 0x89E9, 0xB98A, 0xA9AB,
0x5844, 0x4865, 0x7806, 0x6827, 0x18C0, 0x08E1, 0x3882, 0x28A3,
0xCB7D, 0xDB5C, 0xEB3F, 0xFB1E, 0x8BF9, 0x9BD8, 0xABBB, 0xBB9A,
0x4A75, 0x5A54, 0x6A37, 0x7A16, 0x0AF1, 0x1AD0, 0x2AB3, 0x3A92,
0xFD2E, 0xED0F, 0xDD6C, 0xCD4D, 0xBDAA, 0xAD8B, 0x9DE8, 0x8DC9,
0x7C26, 0x6C07, 0x5C64, 0x4C45, 0x3CA2, 0x2C83, 0x1CE0, 0x0CC1,
0xEF1F, 0xFF3E, 0xCF5D, 0xDF7C, 0xAF9B, 0xBFBA, 0x8FD9, 0x9FF8,
0x6E17, 0x7E36, 0x4E55, 0x5E74, 0x2E93, 0x3EB2, 0x0ED1, 0x1EF0
};
I use the following code to calculate a CRC-CCITT (0xFFFF):
unsigned short Calculate_CRC_CCITT(const unsigned char* buffer, int size)
{
unsigned short tmp;
unsigned short crc = 0xffff;
for (int i=0; i < size ; i++)
{
tmp = (crc >> 8) ^ buffer[i];
crc = (crc << 8) ^ CRC_CCITT_TABLE[tmp];
}
return crc;
}
For Java
Install: https://mvnrepository.com/artifact/com.github.snksoft/crc/1.0.2
Long returnValue = CRC.calculateCRC(Parameters.CCITT, "yourStringValue");