I am trying to use the first bit of an unsigned int as a flag if the client as server should re-key for their encryption keys. I would like to use the rest of the unsigned int as the length for the remaining data that is being sent.
Currently I am trying this but it doesn't seem to work.
unsigned int payloadLength;
read(sock, &payloadLength, sizeof(payloadLength));
short bit = (payloadLength >> 0) & 1U; // get first bit
payloadLength &= 1UL << 0; // set first bit to 0
payloadLength = ntohl(payloadLength);
if (bit == 1)
//rekey
else
//read more data
The rekey flag seems to be set right, but when trying to get the length it always ends up as the wrong number.
Edit: I should have clarified I meant most significant bit, not first bit
Let's take a look at this code:
short bit = (payloadLength >> 0) & 1U; // get first bit
payloadLength &= 1UL << 0; // set first bit to 0
payloadLength = ntohl(payloadLength);
Although it's perfectly legal for you to order these statements this way, there's a risk that this won't work the way you want it to. Specifically, you probably want to decode the payload to use the host byte ordering before you start poking and prodding the bytes. Otherwise, if you send data from one system to another, there's a risk that you'll be reading the wrong bits back. But that's not too hard to fix - just move the last statement, which decodes payloadLength, to the top, like this:
/* CAUTION: This still has errors! */
payloadLength = ntohl(payloadLength);
short bit = (payloadLength >> 0) & 1U; // get first bit
payloadLength &= 1UL << 0; // set first bit to 0
Next, there's a question about which bit you are trying to read. It looks like you are trying to read the least-significant bit of the number. If that's the case, you don't need to include any bitshifts, as bitshifting by zero positions doesn't have any effect. Let's remove those, giving this:
/* CAUTION: This still has errors! */
payloadLength = ntohl(payloadLength);
short bit = payloadLength & 1U; // get first bit
payloadLength &= 1UL; // set first bit to 0
Your code to extract the final bit of the number is correct. It'll mask out everything except the lowest bit, then store the value in the variable bit. (Out of curiosity, is there any reason you're storing this as a short? If you're looking for a boolean "do I need to reencode this?," you might want to just go with bool and write something like
bool reencode = (payloadLength & 1U) != 0;
That's up to you to decide, though.)
However, your code to clear the last bit is incorrect. Right now, when you write
payloadLength &= 1UL; // set first bit to 0
you are actually doing the opposite of what you intended - you're clearing every bit except the first. That's because if you AND payloadLength with a value, you're zeroing every bit in payloadLength except for the bits that are equal to 1 in 1UL. However, 1UL only has a 1 bit in the last position. You probably meant to write something like
payloadLength &= ~1UL; // set first bit to 0
where the ~ operator flips the bits of your mask. Overall, this would give you the following:
payloadLength = ntohl(payloadLength);
short bit = payloadLength & 1U; // get first bit
payloadLength &= ~1UL; // set first bit to 0
I have one last question, though. By repurposing the lowest bit of the number this way, you are requiring that your payload length always be an even number, since you're harnessing the 1's bit of the payload length to encode whether to rekey. If you're okay with that, great! You don't need to do anything.
On the other hand, if that's an issue, you have a couple of options, which I'll leave to you to select from:
Instead of using the least-significant bit, use the most-significant bit. This will cause problems if you try to send payloads of size 231 or higher, though.
Use the least-significant bit, but have the upper 31 bits of the number represent the actual payload length. To extract the payload length, just shift everything over one position. This has the drawback of not supporting payloads of size 231 or greater, though.
Don't use the bits at all to encode this! Instead, send a payload length, then have the payload start with a header that tells you whether to rekey, etc. This uses more bytes per payload, but also gives you more flexibility in the future (what if you need to send other flags as well?)
Hope this helps!
payloadLength &= 1UL << 0; will zero all bits other than the first one.
payloadLength &= ~1UL; will zero the first bit.
Related
I have a 64-bit unsigned integer. I want to check the 6th bit of each byte and return a single byte representing those 6th bits.
The obvious, "brute force" solution is:
inline const unsigned char Get6thBits(unsigned long long num) {
unsigned char byte(0);
for (int i = 7; i >= 0; --i) {
byte <<= 1;
byte |= bool((0x20 << 8 * i) & num);
}
return byte;
}
I could unroll the loop into a bunch of concatenated | statements to avoid the int allocation, but that's still pretty ugly.
Is there a faster, more clever way to do it? Maybe use a bitmask to get the 6th bits, 0x2020202020202020 and then somehow convert that to a byte?
If _pext_u64 is a possibility (this will work on Haswell and newer, it's very slow on Ryzen though), you could write this:
int extracted = _pext_u64(num, 0x2020202020202020);
This is a really literal way to implement it. pext takes a value (first argument) and a mask (second argument), at every position that the mask has a set bit it takes the corresponding bit from the value, and all bits are concatenated.
_mm_movemask_epi8 is more widely usable, you could use it like this:
__m128i n = _mm_set_epi64x(0, num);
int extracted = _mm_movemask_epi8(_mm_slli_epi64(n, 2));
pmovmskb takes the high bit of every byte in its input vector and concatenates them. The bits we want are not the high bit of every byte, so I move them up two positions with psllq (of course you could shift num directly). The _mm_set_epi64x is just some way to get num into a vector.
Don't forget to #include <intrin.h>, and none of this was tested.
Codegen seems reasonable enough
A weirder option is gathering the bits with a multiplication: (only slightly tested)
int extracted = (num & 0x2020202020202020) * 0x08102040810204 >> 56;
The idea here is that num & 0x2020202020202020 only has very few bits set, so we can arrange a product that never carries into bits that we need (or indeed at all). The multiplier is constructed to do this:
a0000000b0000000c0000000d0000000e0000000f0000000g0000000h0000000 +
0b0000000c0000000d0000000e0000000f0000000g0000000h00000000000000 +
00c0000000d0000000e0000000f0000000g0000000h000000000000000000000 etc..
Then the top byte will have all the bits "compacted" together. The lower bytes actually have something like that too, but they're missing bits that would have to come from "higher" (bits can only move to the left in a multiplication).
I need a function to read n bits starting from bit x(bit index should start from zero), and if the result is not byte aligned, pad it with zeros. The function will receive uint8_t array on the input, and should return uint8_t array as well. For example, I have file with following contents:
1011 0011 0110 0000
Read three bits from the third bit(x=2,n=3); Result:
1100 0000
There's no (theoretical) limit on input and bit pattern lengths
Implementing such a bitfield extraction efficiently without beyond the direct bit-serial algorithm isn't precisely hard but a tad cumbersome.
Effectively it boils down to an innerloop reading a pair of bytes from the input for each output byte, shifting the resulting word into place based on the source bit-offset, and writing back the upper or lower byte. In addition the final output byte is masked based on the length.
Below is my (poorly-tested) attempt at an implementation:
void extract_bitfield(unsigned char *dstptr, const unsigned char *srcptr, size_t bitpos, size_t bitlen) {
// Skip to the source byte covering the first bit of the range
srcptr += bitpos / CHAR_BIT;
// Similarly work out the expected, inclusive, final output byte
unsigned char *endptr = &dstptr[bitlen / CHAR_BIT];
// Truncate the bit-positions to offsets within a byte
bitpos %= CHAR_BIT;
bitlen %= CHAR_BIT;
// Scan through and write out a correctly shifted version of every destination byte
// via an intermediate shifter register
unsigned long accum = *srcptr++;
while(dstptr <= endptr) {
accum = accum << CHAR_BIT | *srcptr++;
*dstptr++ = accum << bitpos >> CHAR_BIT;
}
// Mask out the unwanted LSB bits not covered by the length
*endptr &= ~(UCHAR_MAX >> bitlen);
}
Beware that the code above may read past the end of the source buffer and somewhat messy special handling is required if you can't set up the overhead to allow this. It also assumes sizeof(long) != 1.
Of course to get efficiency out of this you will want to use as wide of a native word as possible. However if the target buffer necessarily word-aligned then things get even messier. Furthermore little-endian systems will need byte swizzling fix-ups.
Another subtlety to take heed of is the potential inability to shift a whole word, that is shift counts are frequently interpreted modulo the word length.
Anyway, happy bit-hacking!
Basically it's still a bunch of shift and addition operations.
I'll use a slightly larger example to demonstrate this.
Suppose we are give an input of 4 characters, and x = 10, n = 18.
00101011 10001001 10101110 01011100
First we need to locate the character contains our first bit, by x / 8, which gives us 1 (the second character) in this case. We also need the offset in that character, by x % 8, which equals to 2.
Now we can get out first character of the solution in three operations.
Left shift the second character 10001001 with 2 bits, gives us 00100100.
Right shift the third character 10101110 with 6 (comes from 8 - 2) bits, gives us 00000010.
Add these two characters gives us the first character in your return string, gives 00100110.
Loop this routine for n / 8 rounds. And if n % 8 is not 0, extract that many bits from the next character, you can do it in many approaches.
So in this example, our second round will give us 10111001, and the last step we get 10, then pad the rest bits with 0s.
I've got to program a function that receives
a binary number like 10001, and
a decimal number that indicates how many shifts I should perform.
The problem is that if I use the C++ operator <<, the zeroes are pushed from behind but the first numbers aren't dropped... For example
shifLeftAddingZeroes(10001,1)
returns 100010 instead of 00010 that is what I want.
I hope I've made myself clear =P
I assume you are storing that information in int. Take into consideration, that this number actually has more leading zeroes than what you see, ergo your number is most likely 16 bits, meaning 00000000 00000001 . Maybe try AND-ing it with number having as many 1 as the number you want to have after shifting? (Assuming you want to stick to bitwise operations).
What you want is to bit shift and then limit the number of output bits which can be active (hold a value of 1). One way to do this is to create a mask for the number of bits you want, then AND the bitshifted value with that mask. Below is a code sample for doing that, just replace int_type with the type of value your using -- or make it a template type.
int_type shiftLeftLimitingBitSize(int_type value, int numshift, int_type numbits=some_default) {
int_type mask = 0;
for (unsigned int bit=0; bit < numbits; bit++) {
mask += 1 << bit;
}
return (value << numshift) & mask;
}
Your output for 10001,1 would now be shiftLeftLimitingBitSize(0b10001, 1, 5) == 0b00010.
Realize that unless your numbits is exactly the length of your integer type, you will always have excess 0 bits on the 'front' of your number.
Sorry about the clumsy title; I couldn't find a bit way of expressing what I'm trying to do.
I am getting an input from the user of multiple 32-bit integers. For example, the user may enter the following values (showing in hex for ease of explanation):
0x00001234
0x00005678
0x0000abcd
In this particular case, the first 2 bytes of each input is constant, and the last 2 bytes are variable. For efficiency purposes, I could store 0x0000 as a single constant, and create a vector of uint16_t values to store the variable portion of the input (0x1234, 0x5678, 0xabcd).
Now let's say the user enters the following:
0x00000234
0x56780000
0x00001000
In this case I would need a vector of uint32_t values to store the variable portion of the input as each value affects different bytes.
My current thought is to do the following:
uint32_t myVal = 0;
myVal |= input1;
myVal |= input2;
// ...
And then at the end find the distance between the first and last "toggled" (i.e. 1) bit in myVal. The distance will give me required field size for the variable portion of all of the inputs.
However, this doesn't sound like it would scale well for a large number of user inputs. Any recommendations about an elegant and efficient way of determining this?
Update:
I simplified the problem in my above explanation.
Just to be clear, I am not doing this to save memory (I have better things to do than to try and conserve a few bytes and this isn't for optimization purposes).
In summary, component A provides component B in my system with values. Sometimes these values are 128-bit, but component B only supports 32-bit values.
If the variable portion of the 128-bit value can be expressed with a 32-bit value, I can accept it. Otherwise I will need to reject it with an error.
I'm not in a position to modify component B to allow 128-bit values, or modify component A to prevent its use of 128-bit values (there are hardware limitations here too).
Though I can't see a reason for all that... Why just not to compare an input with the std::numeric_limits<uint16_t>::max()? If the input gives a larger value then you need to use uint32_t.
Answering your edit:
I suppose for for better performance you should use hardware specific low level instructions. You could iterate over 32-bit parts of the input 128-bit value and subsequently add each one to the some variable and check the difference between next value and current sum. If the difference isn't equal to the sum then you should skip this 128-bit value, otherwise you'll get the necessary result in the end. The sample follows:
uint32_t get_value( uint32_t v1, uint32_t v2, uint32_t v3, uint32_t v4)
{
uint32_t temp = v1;
if ( temp - v2 != temp ) throw exception;
temp += v2; if ( temp - v3 != temp ) throw exception;
temp += v3; if ( temp - v4 != temp ) throw exception;
temp = v4;
return temp;
}
In this C++ example it may be looks silly but I believe in the assembly code this should efficiently process the input stream.
Store the first full 128 bit number you encounter, then push the lower order 32 bits of it onto a vector, set bool reject_all = false. For each remaining number, if high-order (128-32=96) bits differ from the first number's then set reject_all = true, otherwise push their lower-order bits on the vector. At the end of the loop, use reject_all to decide whether to use the vector of values.
The most efficient way to store a series of unsigned integers in the range [0, (2^32)-1] is by just using uint32_t. Jumping through hoops to save 2 bytes from user input is not worth your time--the user cannot possibly, in his lifetime, enter enough integers that your code would have to start compressing them. He or she would die of old age long before memory constraints became apparent on any modern system.
It looks like you have to come up with a cumulative bitmask -- which you can then look at to see whether you have trailing or leading constant bits. An algorithm that operates on each input will be required (making it an O(n) algorithm, where n is the number of values to inspect).
The algorithm would be similar to something like what you've already done:
unsigned long long bitmask = 0uL;
std::size_t count = val.size();
for (std::size_t i = 0; i < count; ++i)
bitmask |= val[i];
You can then check to see how many bits/bytes leading/trailing can be made constant, and whether you're going to use the full 32 bits. If you have access to SSE instructions, you can vectorize this using OpenMP.
There's also a possible optimization by short-circuiting to see if the distance between the first 1 bit and the last 1 bit is already greater than 32, in which case you can stop.
For this algorithm to scale better, you're going to have to do it in parallel. Your friend would be vector processing (maybe using CUDA for Nvidia GPUs, or OpenCL if you're on the Mac or on platforms that already support OpenCL, or just OpenMP annotations).
Use
uint32_t ORVal = 0;
uint32_t ANDVal = 0xFFFFFFFF;
ORVal |= input1;
ANDVal &= input1;
ORVal |= input2;
ANDVal &= input2;
ORVal |= input3;
ANDVal &= input3; // etc.
// At end of input...
mask = ORVal ^ ANDVal;
// bit positions set to 0 were constant, bit positions set to 1 changed
A bit position in ORVal will be 1 if at least one input had 1 in that position and 0 if ALL inputs had 0 in that position. A bit position in ANDVal will be 0 if at least one input had 0 in that bit position and 1 if ALL inputs had 1 in that position.
If a bit position in inputs was always 1, then ORVal and ANDVal will both be set to 1.
If a bit position in inputs was always 0, then ORVal and ANDVal will both be set to 0.
If there was a mix of 0 and 1 in a bit position then ORVal will be set to 1 and ANDVal set to 0, hence the XOR at the end gives the mask for bit positions that changed.
I am new to working with bits & bytes in C++ and I'm looking at some previously developed code and I need some help in understanding what is going on with the code. There is a byte array and populating it with some data and I noticed that the data was being '&' with a 0x0F (Please see code snipped below). I don't really understand what is going on there....if somebody could please explain that, it would be greatly apperciated. Thanks!
//Message Definition
/*
Byte 1: Bit(s) 3:0 = Unused; set to zero
Bit(s) 7:4 = Message ID; set to 10
*/
/*
Byte 2: Bit(s) 3:0 = Unused; set to zero
Bit(s) 7:4 = Acknowledge Message ID; set to 11
*/
//Implementation
BYTE Msg_Arry[2];
int Msg_Id = 10;
int AckMsg_Id = 11;
Msg_Arry[0] = Msg_Id & 0x0F; //MsgID & Unused
Msg_Arry[1] = AckMsg_Id & 0x0F; //AckMsgID & Unused
0x0f is 00001111 in binary. When you perform a bitwise-and (&) with this, it has the effect of masking off the top four bits of the char (because 0 & anything is always 0).
x & 0xF
returns the low four bits of the data.
If you think of the binary representation of x, and use the and operator with 0x0f (00001111 in binary), the top four bits of x will always become zero, and the bottom four bits will become what they were before the operation.
In the given example, it actually does nothing. Msg_Id and AckMsg_Id are both less than 0x0F, and so masking them has no effect here.
However the use of the bitwise-and operator (&) on integer types performs a bit for bit AND between the given operands.