I am using the boost crc library to calculate a 32-bit crc of an 112-bit (80-bit data + 32-bit crc) bitset. For testing, I reset all 80 data bits to 0. The calculation of the crc seems to work fine, but when I append the calculated crc to the data and calculate the crc again, I get a value > 0. But I expected a crc value of exactly 0. Here is the code:
// creating 112-bit bitset and reset all values to 0
bitset<112> b_data;
b_data.reset();
// create string containing data and blank crc
string crc_string = b_data.to_string<char,string::traits_type,string::allocator_type>();
// calculating crc using boost library
boost::crc_32_type crc32;
crc32 = for_each(crc_string.begin(), crc_string.end(), crc32);
bitset<32> b_crc(crc32());
// writing calculated crc to the last 32-bit of the 112-bit bitset.
for(int i = 0; i!= b_crc.size(); i++){
b_data[i+80] = b_crc[i];
}
// create string containing data and calculated crc
string crc_string_check = b_data.to_string<char,string::traits_type,string::allocator_type>();
// calculate crc again to check if the data is correct
boost::crc_32_type crc32_check;
crc32_check = std::for_each(crc_string_check.begin(), crc_string_check.end(), crc32_check);
// output the result
cout << crc32() << endl;
cout << crc32_check() << endl;
The output is:
1326744236
559431208
The output I have expected is:
1326744236
0
So something goes wrong, but what? Any ideas ?
Thanks in advance.
Your expectation of what is "satisfactory" is not correct. A CRC has the property you expect only if it has a zero initialization and no exclusive-or of the result. However the standard CRC-32 that you requested, crc_32_type initializes the CRC register with 0xffffffff, and exclusive-ors the result with 0xffffffff.
However you will always get the same constant when you take the CRC of the message concatenated with the message's CRC (assuming that you order the bytes of the CRC correctly). That constant is 0x2144df1c for this particular CRC, and is the CRC of four zero bytes.
It is common for CRC's to be defined in this way, so that the CRC of a string of zeros is not zero. Since the initialization and exclusive-or are the same, the CRC of the empty set is conveniently zero.
What you should be doing is simply computing the CRC on the message without the CRC, and compare that to the transmitted CRC. That is what is normally done, and applies to all message hashes in general.
First, like the commenter said, donot include the CRC bits once you check the sum. I suggest making a helper function to check n bits (80, in this case).
Secondly, consider refactoring the code to be readable and expressive of the intent:
Live On Coliru
#include <iostream>
#include <string>
#include <bitset>
#include <algorithm>
#include <boost/crc.hpp>
#include <cassert>
template <size_t N>
uint32_t crc32_n(std::bitset<N> const& bs, size_t n) {
assert(n <= N);
std::string const s = bs.template to_string<char>();
auto f = s.begin();
return std::for_each(f, f+n, boost::crc_32_type{})();
}
int main() {
std::bitset<112> b_data;
// set some random data (above the first 32 bits that are for CRC)
srand(time(0));
b_data.set(rand()%80 + 32);
auto const crc = crc32_n(b_data, 80);
std::cout << crc << ":\t" << b_data << "\n";
b_data |= crc;
std::cout << crc32_n(b_data, 80) << ":\t" << b_data << "\n";
}
Prints e.g.
1770803766: 0000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000
1770803766: 0000000000000000000000000000000000000000000000000000000000000000000000010000000001101001100011000101001000110110
or (on another run)
2436181323: 0000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000
2436181323: 0000000000000000000000000000000000000000000000000000000000000000000000100000000010010001001101010010110101001011
I would love to go on and remove the horrific to_string call but see How do I convert bitset to array of bytes/uint8?
If you're going to include the CRC in the CRC check, which in my view is the correct technique, the expected result is always zero. Not the CRC that was transmitted.
Related
I have just started to study the CRC and how to implement it in software. My information source is mainly following document. Here is mentioned some simple algorithm for calculating CRC for any generator polynomial. I have attempted to write this algorithm in C++ language. I have tested it for generator polynomial x^5 + x^4 + x^2 + 1 (CRC-5) (generator polynomial used by chip) with usage of this online calculator and it works.
#include <iostream>
using namespace std;
int main() {
uint8_t data_byte = 0x31;
// polynom x^5 + x^4 + x^2 + 1
uint16_t polynom = 0x35;
// register contains 0 at the beginning
uint32_t crc = 0;
uint32_t message = 0;
// shift the message byte to left by so many bits which is needed for generator polynomial
message = (data_byte << 5);
// now the message byte is 13 bits long
uint8_t processed_bit = 13;
while(processed_bit > 0) {
// prepare free space for new bit from the message byte
crc = crc << 1;
// find out value of current msb in the message byte
message = message << 1;
if(message & 0x2000) {
// msb in message byte is "1"
// lsb in register is set to "1"
crc |= 1U;
} else {
// msb in message byte is "0"
// lsb in register is set to "0"
crc &= ~1U;
}
// remove msb from message byte
message = message & ~0x2000;
if(crc & 0x20) {
// subtract current multiple of the generator polynomial
crc = crc ^ polynom;
}
// remove msb from the register
crc = crc & ~0x20;
processed_bit--;
}
cout << "CRC: " << (int)crc << endl;
return 0;
}
It is obvious that this program is uneffective as far as execution time. So I have been thinking about a possibility how to improve it in this perspective. I know that there is a variant to use the look-up table containing the precalculated reminders but I would like to avoid this method. Does anybody know how to improve the above mentioned algorithm from the execution time perspective? Thanks in advance for any suggestions.
Just a quick glance shows several unnecessary statements. You don't need crc &= ~1U;, since the crc = crc << 1; already put a zero there. You don't need message = message & ~0x2000;, since you are only ever looking at one bit in there. Just let the other bits shift up and away. You don't need the crc = crc & ~0x20;, since the exclusive-or with the polynomial already did that.
If you read the document you linked, you will find that you do not need to process five more bits (13 total). You only need to process the eight message bits. Also reading that document, you do not need to feed in the message bits one at a time. You can exclusive-or the message byte directly onto the CRC register, and then process the eight bits all in the register.
Finally, you can speed up the calculation significantly with a table look up, processing eight bits at a time instead of one bit at a time. This is also described beautifully in the document you linked. You can find code here to automatically generate the table and C code for the calculation.
In the end though, none of this matters if you're not calculating the right thing to begin with. You need to verify the calculation with data from the chip first. I found this document with details on the CRC calculation for that chip. You need to spend some time with it and understand it in detail.
To answer your question directly, here is some code that does what your code does, but is much simpler. Also it is extended to work on n bits, not just eight. It does n loops instead of n+5 loops:
// Return a CRC-5 of the low n bits of data. The remaining bits of data must be
// zero. n must be in [5..32].
uint8_t crc5(uint32_t data, int n) {
int shift = n - 5;
uint32_t poly = (uint32_t)0x15 << shift;
uint32_t top = (uint32_t)1 << (n - 1);
do {
data = data & top ? (data << 1) ^ poly : data << 1;
} while (--n);
return (data >> shift) & 0x1f;
}
Simpler and faster still is the equivalent of yours restricted to eight bits, unrolled:
uint8_t crc5_8(uint8_t data) {
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
data = data & 0x80 ? (data << 1) ^ 0xa8 : data << 1;
return data >> 3;
}
However neither can calculate what you need for your chip.
I am trying to fix this part of an abandonware program because I failed to find an alternative program.
As you can see the data of PUSH instructions are in the wrong order whereas Ethereum is a big endian machine (address are correctly represented because they use a smaller type).
An alternative is to run porosity.exe --code '0x61004b60026319e44e32' --disassm
Theu256 type is defined as
using u256 = boost::multiprecision::number<boost::multiprecision::cpp_int_backend<256, 256, boost::multiprecision::unsigned_magnitude, boost::multiprecision::unchecked, void>>;
Here’s a minimal example to reproduce the bug:
#include <sstream>
#include <iostream>
#include <iomanip>
#include <boost/multiprecision/cpp_int.hpp>
using u256 = boost::multiprecision::number<boost::multiprecision::cpp_int_backend<256, 256, boost::multiprecision::unsigned_magnitude, boost::multiprecision::unchecked, void>>;
int main() {
std::stringstream stream;
u256 data=0xFEDEFA;
for (int i = 0; i<5; ++i) { // print only the first 5 digits
uint8_t dataByte = int(data & 0xFF);
data >>= 8;
stream << std::setfill('0') << std::setw(sizeof(char) * 2) << std::hex << int(dataByte) << " ";
}
std::cout << stream.str();
}
So numbers are converted to string with a space between each byte (and only the first bytes).
But then I ran into an endianness problem: bytes were printed in the reverse order. I mean for example, 31722 is written 8a 02 02 on my machine and 02 02 8a when compiled for a big endian target.
So as I don’t which boost function to call, I modified the code:
#include <sstream>
#include <iostream>
#include <iomanip>
#include <boost/multiprecision/cpp_int.hpp>
using u256 = boost::multiprecision::number<boost::multiprecision::cpp_int_backend<256, 256, boost::multiprecision::unsigned_magnitude, boost::multiprecision::unchecked, void>>;
int main() {
std::stringstream stream;
u256 data=0xFEDEFA;
for (int i = 0; i<5; ++i) {
uint8_t dataByte = int(data >> ((32 - i - 1) * 8));
stream << std::setfill('0') << std::setw(sizeof(char) * 2) << std::hex << int(dataByte) << " ";
}
std::cout << stream.str();
}
Now, why are my 256 bits integers printed mostly as series of 00 00 00 00 00?
BTW, this is not an endianness issue; you aren't doing byte accesses to the object-representation. You're operating on it as a 256-bit integer and simply asking for the low 8 bits at a time with data & 0xFF.
If you did know the endianness of the target C implementation, and the data layout of the boost object, you could efficiently loop over it in descending address order with unsigned char*.
You're introducing the idea of endianness only because it's associated with byte-reversal, which is what you're trying to do. But that's really inefficient, just loop over the bytes of your bigint the other way.
I'm hesitant to recommend a specific solution because I don't know what will compile efficiently. But you might want something like this instead of byte-reversing ahead of time:
for (outer loop) {
uint64_t chunk = data >> (64*3); // grab the highest 64-bit chunk
data <<= 64; // and shift everything up
// alternative: maybe keep a shift-count in a variable instead of modifying `data`
// Then pick apart the chunk into its component bytes, in MSB first order
for (int = 0 ; i<8 ; i++) {
unsigned tmp = (chunk >> 56) & 0xFF;
// do something with it
chunk <<= 8; // bring the next byte to the top
}
}
In the inner loop, more efficient than using two shifts can be using a rotate to bring the high byte to the bottom (for & 0xFF) at the same time as shifting lower bytes upward.
Best practices for circular shift (rotate) operations in C++
In the outer loop, IDK if boost::multiprecision::number has any APIs for efficient indexing of chunks built in; if so using that is probably more efficient.
I used nested loops because I assume data <<= 8 doesn't compile particularly efficiently, and neither would (data >> (256-8)) & 0xFF. But that's how you'd grab bytes from the top instead of the bottom.
Another option is the standard trick for converting numbers to strings: store characters into a buffer in descending order. A 256-bit (32-byte) number will take 64 hex digits, and you want another 32 bytes of spaces between them.
For example:
// 97 = 32 * 2 + 32, plus 1 byte for an implicit-length C string terminator
// plus another 1 for an extra space
char buf[98]; // small enough to use automatic storage
char *outp = buf+96; // pointer to the end
*outp = 0; // terminator
const char *hex_lut = "0123456789abcdef";
for (int i=0 ; i<32 ; i++) {
uint8_t byte = data & 0xFF;
*--outp = hex_lut[byte >> 4];
*--outp = hex_lut[byte & 0xF];
*--outp = ' ';
data >>= 8;
}
// outp points at an extra ' '
outp++;
// outp points at the first byte of a string like "12 ab cd"
stream << outp;
If you want to break that up into chunks to put a line break in there, you can do that too.
If you're interested in efficient conversion to hex for 8, 16 or 32 bytes of data at once, see How to convert a number to hex? for some x86 SIMD ways. The asm should port easily to C++ intrinsics. (You can use SIMD shuffles to handle putting bytes into MSB-first printing order after loading from little-endian integers.)
You could also use a SIMD shuffle to space-separate your pairs of hex digits before storing to memory like you apparently want here.
Bug in the code you added:
So I added this code before the loop above:
for(unsigned int i=0,data,_data;i<33;++i)
unsigned i, data, _data declares new variables of type unsigned int that shadow the previous declarations of data and _data. That loop has zero effect on data or _data outside the scope of the loop. (And contains UB because you read _data and data without initializing them.)
If those vars are actually both still the u256 vars of the outer scope, I don't see an obvious problem other than efficiency, but maybe I'm missing the obvious too. I didn't look very hard because using 64x 256-bit shifts and 32x ORs seems like a horrible idea. It's possible it could optimize away completely, or into bswap byte-reverse instructions on ISAs that have them, but I doubt it. Especially not through the extra complication of the boost::multiprecision::number wrapper functions.
I need to create a program, that calculates CRC from file. It needs to be done bit by bit.
The way I would like to read a file:
unsigned char byte;
ifstream file;
bool result;
int number;
file.open("test.txt", ios::binary);
while(true)
{
byte = file.get();
number = (int)byte;
result = file.good();
if(!result)
break;
}
However, I don't know how to read it bit by bit.
My CRC's divisor (called a "polynomial") is 0x04C11DB7 and I need to import 1 new bit from file each time I calculate my buffer.
My idea is to add first 4 bytes to variable (for let's say "1234" it would be 0x31323334), then remove last bit (by moving the number 1 bit to the left), but I don't know how to add a new bit from the next char.
Do you mean something along these lines?
The CRC calculation may vary, but the focus here is on getting the file content "bit by bit".
#include <iostream>
#include <fstream>
int main(int argc, char* argv[])
{
unsigned char next;
unsigned long crc = 0;
if (argc < 2)
return -1;
std::fstream fs(argv[1], std::fstream::in);
while (!fs.bad() && !fs.eof())
{
fs >> next;
for (int i = 0; i < 8; i++)
{
crc += next & 1;
next >>= 1;
}
}
std::cout << "CRC " << crc << std::endl;
return 0;
}
The divisor is not just called a polynomial. It means that each bit is a coefficient of a polynomial (of degree 32) and thus the way of computing with polynomial differs significantly from working with integers. You can add (and substract, which is the same in this case) two polynomials with a simple XOR operation. Multiplying/Dividing with/by X means shifting. To the right or to the left depends on the order in which the coefficients of the polynomials are written. This is important to know because both directions (left to right and right to left) actually exist. In the case of 0x04C11DB7, the coefficient of X^0 is bit 0 and the coefficient of X^31 is bit 31. Be aware that the popular implementation of the IEEE802.3 CRC has the opposite bit order. So, just copying the implementation of an Ethernet CRC will not work.
This means the next bit to process is always bit 31. You must therefore check for 0x80000000. If the bit is set, XOR your polynomial. This means, you subtract the polynomical from your work register. In any case, shift the result to the left afterwards. Then a 0 bit is shifted in at the right. Replace it with the next bit to process by a binary or operation (| in C++). You obtain that bit in the same way: if you are reading byte by byte, your next bit is 1 or 0, depending on whether 0x80 is set in your input. Then shift your input to the left.
I'm trying to implement the simple POSIX cksum using Boost.CRC.
The code I'm using amounts to this:
for(int i = 1; i<argc; ++i)
{
support::file current(argv[i], support::file::access::read);
size_t octets = 0;
boost::crc_32_type crc;
while(true)
{
size_t bytes_read = current.read_some(buffer_size, buffer);
octets += bytes_read;
crc.process_bytes(&buffer[0], bytes_read);
if(bytes_read < buffer_size)
break;
}
if(i>1)
support::print("\n");
support::print(boost::lexical_cast<string>(crc.checksum()) + " " + boost::lexical_cast<string>(octets) + " " + argv[i]);
}
Where support::file is a simple fopen/fread binary file I/O wrapper, which I succesfully used for a cat implementation. support::print gives the same output as std::cout, but I need it for reliable non-ASCII output on Windows.
The Boost header has this:
typedef crc_optimal<32, 0x04C11DB7, 0xFFFFFFFF, 0xFFFFFFFF, true, true> crc_32_type;
as the only 32-bit CRC typedef. It gives the wrong answer (checked with GNUWin32 coreutils cksum) for an empty file (touch test && cksum test). I have tried using the above typedef and modifying one or both of the 0xFFFFFFFF values to 0, I get the correct result for an empty file, but any other file still gives different results.
How is Boost.CRC related to the POSIX cksum specification?
This isn't really an answer to your question, but the results of the investigation that I've carried out. I used to struggle with CRC when I was building a simple implementation of an Ethernet controller in VHDL, and I do realize that implementations can be sometimes very varying due to reasons unknown.
Okay, let's begin. The typedef that you found in boost/crc.hpp :
typedef crc_optimal<32, 0x04C11DB7, 0xFFFFFFFF, 0xFFFFFFFF, true, true> crc_32_type;
Is a simple declaration of a CRC generator that will produce the CRC for Ethernet. The parameters of the template are as follows : Bits (the number of bits output by the generator), TruncPoly (the polynomial used by the generator), InitRem (the initial remainder to be fed into the algorithm before processing the first byte of input), FinalXor (the value that the output value should be XORed with after processing all the bytes of the input), ReflectIn and ReflectRem (if the input bytes and/or the output should be bit-reflected, e.g. bit 0 becomes bit 7, and so on). Ethernet not only requires the output to be calculated with the given polynomial, but it also requires the constraints that you can read from that typedef.
According to the specification of cksum, the typedef for the CRC generator for it should look something like this :
typedef crc_optimal<32, 0x04C11DB7, 0, 0xFFFFFFFF, false, false> cksum_crc_type;
This is because :
The specification doesn't specify a starting value for the generator, thus 0.
The output value should be complemented, according to number 4 of the specification. The same result can be achieved by XORing the value with ones.
Bit reflecting is not mentioned anywhere, and thus will not be performed.
However, there is one significant difference when it comes to cksum as opposed to ordinary CRCs :
(...) followed by one or more octets representing the length of the
file as a binary value, least significant octet first. The smallest
number of octets capable of representing this integer shall be used.
Ordinary CRC generators don't take into account the number of octets that were processed. This would also explain why you got a good result when processing a zero-length file, but a bad one when processing larger files.
Unfortunately, I don't see an easy solution for that problem. I guess what you could do is modify the process_bytes method in a following way :
template < std::size_t Bits, BOOST_CRC_PARM_TYPE TruncPoly,
BOOST_CRC_PARM_TYPE InitRem, BOOST_CRC_PARM_TYPE FinalXor,
bool ReflectIn, bool ReflectRem >
inline
void
BOOST_CRC_OPTIMAL_NAME::process_bytes
(
void const * buffer,
std::size_t byte_count
)
{
unsigned char const * const b = static_cast<unsigned char const *>(
buffer );
process_block( b, b + byte_count );
for(; byte_count; byte_count >>= 8)
rem_ = (rem_ << 8) ^ crc_table_type::table_[((rem_ >> 24) ^ byte_count) & 0xFF];
}
With such an implementation, the method gives the same result as cksum. for loop courtesy of GNU coreutils.
Hope I helped.
I have a large mass of integers that I'm reading from a file. All of them will be either 0 or 1, so I have converted each read integer to a boolean.
What I need to do is take advantage of the space (8 bits) that a character provides by packing every 8 bits/booleans into a single character. How can I do this?
I have experimented with binary operations, but I'm not coming up with what I want.
int count = 7;
unsigned char compressedValue = 0x00;
while(/*Not end of file*/)
{
...
compressedValue |= booleanValue << count;
count--;
if (count == 0)
{
count = 7;
//write char to stream
compressedValue &= 0;
}
}
Update
I have updated the code to reflect some corrections suggested so far. My next question is, how should I initialize/clear the unsigned char?
Update
Reflected the changes to clear the character bits.
Thanks for the help, everyone.
Several notes:
while(!in.eof()) is wrong, you have to first try(!) to read something and if that succeeded, you can use the data.
Use an unsigned char to get an integer of at least eight bits. Alternatively, look into stdint.h and use uint8_t (or uint_least8_t).
The shift operation is in the wrong direction, use uint8_t(1) << count instead.
If you want to do something like that in memory, I'd use a bigger type, like 32 or 64 bits, because reading a byte is still a single RAM access even if much more than a byte could be read at once.
After writing a byte, don't forget to zero the temporary.
As Mooing Duck suggested, you can use a bitset.
The source code is only a proof of concept - especially the file-read has to be implemented.
#include <bitset>
#include <cstdint>
#include <iostream>
int main() {
char const fcontent[56] { "\0\001\0\001\0\001\0\001\0\001"
"\001\001\001\001\001\001\001\001\001\001\001\001"
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
"\0\001\0\001\0\001\0\001" };
for( int i { 0 }; i < 56; i += 8 ) {
std::bitset<8> const bs(fcontent+i, 8, '\0', '\001');
std::cout << bs.to_ulong() << " ";
}
std::cout << std::endl;
return 0;
}
Output:
85 127 252 0 0 1 84
The standard guaranties that vector<bool> is packed the way you want. Don't reinvent the wheel.more info here