Bit expand byte array - c++
I have a situation where I need to upscale a dynamic sized byte array by 3.
Example:
10101010 11001100
to
11100011 10001110 00111000 11111100 00001111 11000000
I've used the algorithm here to generate a lookup table.
https://stackoverflow.com/a/9044057/280980
static const uint32_t bitExpandTable[256] = {
00000000, 0x000007, 0x000038, 0x00003f, 0x0001c0, 0x0001c7, 0x0001f8, 0x0001ff,
0x000e00, 0x000e07, 0x000e38, 0x000e3f, 0x000fc0, 0x000fc7, 0x000ff8, 0x000fff,
0x007000, 0x007007, 0x007038, 0x00703f, 0x0071c0, 0x0071c7, 0x0071f8, 0x0071ff,
0x007e00, 0x007e07, 0x007e38, 0x007e3f, 0x007fc0, 0x007fc7, 0x007ff8, 0x007fff,
0x038000, 0x038007, 0x038038, 0x03803f, 0x0381c0, 0x0381c7, 0x0381f8, 0x0381ff,
0x038e00, 0x038e07, 0x038e38, 0x038e3f, 0x038fc0, 0x038fc7, 0x038ff8, 0x038fff,
0x03f000, 0x03f007, 0x03f038, 0x03f03f, 0x03f1c0, 0x03f1c7, 0x03f1f8, 0x03f1ff,
0x03fe00, 0x03fe07, 0x03fe38, 0x03fe3f, 0x03ffc0, 0x03ffc7, 0x03fff8, 0x03ffff,
0x1c0000, 0x1c0007, 0x1c0038, 0x1c003f, 0x1c01c0, 0x1c01c7, 0x1c01f8, 0x1c01ff,
0x1c0e00, 0x1c0e07, 0x1c0e38, 0x1c0e3f, 0x1c0fc0, 0x1c0fc7, 0x1c0ff8, 0x1c0fff,
0x1c7000, 0x1c7007, 0x1c7038, 0x1c703f, 0x1c71c0, 0x1c71c7, 0x1c71f8, 0x1c71ff,
0x1c7e00, 0x1c7e07, 0x1c7e38, 0x1c7e3f, 0x1c7fc0, 0x1c7fc7, 0x1c7ff8, 0x1c7fff,
0x1f8000, 0x1f8007, 0x1f8038, 0x1f803f, 0x1f81c0, 0x1f81c7, 0x1f81f8, 0x1f81ff,
0x1f8e00, 0x1f8e07, 0x1f8e38, 0x1f8e3f, 0x1f8fc0, 0x1f8fc7, 0x1f8ff8, 0x1f8fff,
0x1ff000, 0x1ff007, 0x1ff038, 0x1ff03f, 0x1ff1c0, 0x1ff1c7, 0x1ff1f8, 0x1ff1ff,
0x1ffe00, 0x1ffe07, 0x1ffe38, 0x1ffe3f, 0x1fffc0, 0x1fffc7, 0x1ffff8, 0x1fffff,
0xe00000, 0xe00007, 0xe00038, 0xe0003f, 0xe001c0, 0xe001c7, 0xe001f8, 0xe001ff,
0xe00e00, 0xe00e07, 0xe00e38, 0xe00e3f, 0xe00fc0, 0xe00fc7, 0xe00ff8, 0xe00fff,
0xe07000, 0xe07007, 0xe07038, 0xe0703f, 0xe071c0, 0xe071c7, 0xe071f8, 0xe071ff,
0xe07e00, 0xe07e07, 0xe07e38, 0xe07e3f, 0xe07fc0, 0xe07fc7, 0xe07ff8, 0xe07fff,
0xe38000, 0xe38007, 0xe38038, 0xe3803f, 0xe381c0, 0xe381c7, 0xe381f8, 0xe381ff,
0xe38e00, 0xe38e07, 0xe38e38, 0xe38e3f, 0xe38fc0, 0xe38fc7, 0xe38ff8, 0xe38fff,
0xe3f000, 0xe3f007, 0xe3f038, 0xe3f03f, 0xe3f1c0, 0xe3f1c7, 0xe3f1f8, 0xe3f1ff,
0xe3fe00, 0xe3fe07, 0xe3fe38, 0xe3fe3f, 0xe3ffc0, 0xe3ffc7, 0xe3fff8, 0xe3ffff,
0xfc0000, 0xfc0007, 0xfc0038, 0xfc003f, 0xfc01c0, 0xfc01c7, 0xfc01f8, 0xfc01ff,
0xfc0e00, 0xfc0e07, 0xfc0e38, 0xfc0e3f, 0xfc0fc0, 0xfc0fc7, 0xfc0ff8, 0xfc0fff,
0xfc7000, 0xfc7007, 0xfc7038, 0xfc703f, 0xfc71c0, 0xfc71c7, 0xfc71f8, 0xfc71ff,
0xfc7e00, 0xfc7e07, 0xfc7e38, 0xfc7e3f, 0xfc7fc0, 0xfc7fc7, 0xfc7ff8, 0xfc7fff,
0xff8000, 0xff8007, 0xff8038, 0xff803f, 0xff81c0, 0xff81c7, 0xff81f8, 0xff81ff,
0xff8e00, 0xff8e07, 0xff8e38, 0xff8e3f, 0xff8fc0, 0xff8fc7, 0xff8ff8, 0xff8fff,
0xfff000, 0xfff007, 0xfff038, 0xfff03f, 0xfff1c0, 0xfff1c7, 0xfff1f8, 0xfff1ff,
0xfffe00, 0xfffe07, 0xfffe38, 0xfffe3f, 0xffffc0, 0xffffc7, 0xfffff8, 0xffffff,
};
I've tried looping through the byte array using the LUT to memcpy the first 3 bytes to the new array. However my output never looks correct.
Does anyone have any suggestions on how to efficiently implement this? This will be running on an embedded ARM processor.
Edit
LUT test
uint8_t msg[] = { 0xaa, 0x02, 0x43, 0x5a, 0x8d, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xd0, 0x84, 0xc6, 0x2d, 0x00, 0xb9 };
uint8_t expanded_msg[63] = { 0 };
uint8_t tmp_val = 0;
uint32_t lut_val;
for (int i = 0; i < 21; i++)
{
tmp_val = *(uint8_t*)(&msg + i);
lut_val = bitExpandTable[tmp_val];
memcpy(&expanded_msg[(i * 3)], &lut_val, 3);
}
print_binary(&msg, 21);
print_binary(&expanded_msg, sizeof(expanded_msg));
Output
[ 10101010 00000010 01000011 01011010 10001101 00000110 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 11010000 10000100 11000110 00101101 00000000 10111001 ]
[ 00111000 10001110 11100011 00000000 00000000 00000000 11000111 10001111 00000011 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000111 01110000 00011100 00000111 11110000 00000011 11111111 11111111 00011111 11000111 11110001 11100011 00000000 00000000 00000000 11111111 11111111 11111111 11000000 10000001 11100011 00000000 00000000 00000000 00000111 01110000 00011100 00111000 10000000 00011111 11111111 11111111 00011111 11000111 11110001 11100011 00000000 00000000 00000000 11111111 11111111 11111111 ]
There is a byte order issue with your table (or how you copy the data):
For the first input byte
10101010
your output is
a b c
00111000 10001110 11100011
but should be
c b a
11100011 10001110 00111000
So the 1st and 3rd byte need to be swapped (and so on)
Instead of changing the table to suit memcpy, I'd just do something like
int j = 0;
for (int i = 0; i < 21; i++) {
lut_val = bitExpandTable[msg[i]];
expanded_msg[j++] = (uint8_t) (lut_val >> 16);
expanded_msg[j++] = (uint8_t) (lut_val >> 8);
expanded_msg[j++] = (uint8_t) lut_val;
}
Related
Cast from double to size_t yields wrong result?
The following code works. My question is, should 2) not lead to a result very close to 1) ? Why is 2) casted to such a small amount ? Whereby, maybe worth to note 2) is exactly half of 1): std::cout << "1) " << std::pow(2, 8 * sizeof(size_t)) << std::endl; std::cout << "2) " << static_cast<size_t>(std::pow(2, 8 * sizeof(size_t))) << std::endl; The output is: 18446744073709551616 9223372036854775808
It is due to that part of the specification: 7.3.10 Floating-integral conversions [conv.fpint] A prvalue of a floating-point type can be converted to a prvalue of an integer type. The conversion truncates; that is, the fractional part is discarded. The behavior is undefined if the truncated value cannot be represented in the destination type. The value 18446744073709551616 (that's the truncated part) is larger than std::numberic_limit<size_t>::max() on your system, and due to that, the behavior of that cast is undefined.
If we want to calculate the amount of different values a certain unsigned integral datatype can represent we can calculate std::cout << "1) " << std::pow(2, 8 * sizeof(size_t)) << std::endl; // yields 18446744073709551616 This calculates 2 to the power of 64 and yields 18446744073709551616. Since sizeof(size_t) is 8 byte, on a 64 bit machine, and a byte has 8 bit, the width of the size_t data type is 64 bit hence 2^64. This is no surprise since usually it is the case that size_t on a system has the width of its underlying hardware bus system since we want to consume no more than one clock cycle to deliver an address or an index of an array or vector. The above number represents the amount of all different integral values that can be represented by an unsigned integral datatype of 64 bit like size_t or unsigned long long including 0 as one possibility. And since it does include 0, the highest value to be represented is exactly one less, so 18446744073709551615. This number can also be retrieved by std::cout << std::numeric_limits<size_t>::max() << std::endl; // yields 18446744073709551615 std::cout << std::numeric_limits<unsigned long long>::max() << std::endl; // yields the same Now an unsigned datatype stores its values like 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 is 0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 is 1 or 2^0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000010 is 2 or 2^1 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000011 is 3 or 2^1+2^0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000100 is 4 or 2^2 ... 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 is 18446744073709551615 and if you want to add another 1, you would need a 65th bit on the left which you dont have: 1 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 is 0 because there are no more bits on the left. Any amount higher than the highest possible value you would wish to represent will come down to amount modulo the largest possible value + 1. (amount % (max + 1)) which leads as we can see to zero in above sample. And since this comes so naturally the standard defines that if you convert any integral datatype signed or unsigned to another unsigned integral datatype it is to be converted amount modulo the largest possible value + 1. Beautiful. But this easy rule has a little surprise for us when we wish to convert a negative integral to an unsigned integral like -1 to unsigned long long for eaxample. You have a 0 value first and then you deduct 1. What happens is the oposite sequence of the above sample. Have a look: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 is 0 and now do -1 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 is 18446744073709551615 So yes, converting -1 to size_t leads to std::numeric_limits<size_t>::max(). Quite unbelievable at first but understandable after some thinking and playing around with it. Now for our second line of code std::cout << "2) " << static_cast<size_t>(std::pow(2, 8 * sizeof(size_t))) << std::endl; we would expect naively 18446744073709551616, the same result as line one, of course. But since we know now about modulo the largest + 1 and we know now that the largest plus one gives 0 we would also, again naively, accept 0 as an answer. Why naively? Because std::pow returns a double and not an integral datatype. The double datatype is again 64 bit but internally its representation is entirely different. 0XXXXXXX XXXX0000 00000000 00000000 00000000 00000000 00000000 00000000 Only those 11 X bits represent the exponent in 2^n form. That means only those 11 bits have to show 64 and the double will represent 2^64 * 1. So the representation of our big number is much more compact in double than in size_t. Would someone want to do modulo the largest plus 1 some more conversion would be needed before to change the representation of 2^64 into a 64 bit line. Some further reading about floating point representation can be found at https://learn.microsoft.com/en-us/cpp/build/ieee-floating-point-representation?view=msvc-160 for example. And the standard says that if you convert a floating value to an integral which cannot be represented by the target integral datatype the result is UB, undefined behaviour. See the C++17 Standard ISO/IEC14882: 7.10 Floating-integral conversions [conv.fpint] A prvalue of a floating-point type can be converted to a prvalue of an integer type. The conversion truncates ; that is, the fractional part is discarded. The behavior is undefined if the truncated value cannot be represented in the destination type. ... So double can easily hold 2^64 and thats the reason why line 1 could print out so easily. But it is 1 too much to be represented in size_t so the result is UB. So whatever is the outcome of our line 2 is simply irrelevant because it is UB. Ok, but if any random result will do, how come the UB outcome is exactly half? Well fist of all, the outcome is from MSVC. Clang or other compiler may deliver any other UB result. But lets look at the "half" outcome since it is easy. Trying to add 1 to the largest 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 is 18446744073709551615 would if only integrals would be involved lead to, 1 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 but thats not possible since the bit does not exist and it is not integral but double datatype and hence UB, so accidentially the result is 10000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 which is 9223372036854775808 so exactly half of the naively expected or 2^63.
C++ Only Store Specific Bytes of a double in a char*
Off the bat I am unfortunately using an older version of c++ (I believe 98) so c++11 goodies are unavailable to me.
That aside, I was wondering- is it possible to only store specific bytes of a double in a char* buffer? For example, if I have a double that has a low value and therefore only uses 3 bytes of data can I then copy just 3 bytes of data into a char* buffer?
I know it is possible to copy full doubles into a char* buffer. Currently I am doing so and printing out the binary of the char* buffer afterwards using this code:
char* buffer = new char[8]; // A double is 8 bytes
memset(buffer, 0, sizeof(buffer)); // Fill the buffer with 0's
double value = 243;
memcpy(&buffer[0], &value, 8); // copy all 8 bytes (sizeof(value) is better here, I'm just typing '8' for readability)
for (int i = sizeof(value); i > 0; i --)
{
std::bitset<8> x(buffer[i-1]); // 8 bits per byte
std::cout << x << " ";
}
The output of the above code is as expected:
01000000 01101110 01100000 00000000
00000000 00000000 00000000 00000000
If I try and only copy the first 3 bytes into the char* buffer, however, it appears that I don't end up copying over anything at all. Here is the code I'm attempting to use:
char* buffer = new char[8]; // A double is 8 bytes
memset(buffer, 0, sizeof(buffer)); // Fill the buffer with 0's
double value = 243;
memcpy(&buffer[0], &value, 3); // Only copy over 3 bytes
for (int i = sizeof(value); i > 0; i --)
{
std::bitset<8> x(buffer[i-1]); // 8 bits per byte
std::cout << x << " ";
}
The output of the above code is an empty buffer:
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Is there a way for me to only copy 3 bytes of this double over to a char* buffer that I am missing?
Thanks!
You are copying over the wrong bytes, you're computer is in little endian and so the 3 bytes you want to copy over will actually be the last three bytes of the double. If you change the copy line of your code to this memcpy(&buffer[0], (void*)(&value)+5, 3); // Copy the last three bytes you get a correct result of 00000000 00000000 00000000 00000000 00000000 01000000 01101110 01100000
How to use MPI_Win_allocate_shared without getting an error?
I have to replicate an algorithm in which I need two buffers (matrices (L+1)x(N+2)) that must be shared across processes (every process must be able to write in them and read what other processes wrote).
I found that a solution could be using MPI_Win_allocate_shared, however I think I didn't understand very well how to use it, because I get errors.
I'll report below my code with the two trials that I think are close to the solution (I avoid he whole algorithm to focus on the problem):
#include "Options.h"
#include <math.h>
#include <array>
#include <algorithm>
#include <memory>
#include <cmath>
#include <mpi.h>
std::pair <double, double> Options::BinomialPriceAmericanPut(void) {
int rank,size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
// shared buffers to save data for seller and buyer
MPI_Win win_seller, win_buyer;
// size of the local window in bytes
MPI_Aint buff_size;
///////////////// TRIAL 1 /////////////////////////
// pointers that will (locally) point to the shared memory
typedef std::array<PWL, N+2> row_type;
row_type *seller_buff;
row_type *buyer_buff;
///////////////// TRIAL 2 /////////////////////////
// pointers that will (locally) point to the shared memory
typedef std::array<PWL, N+2> row_type;
row_type seller_buff[L+1];
row_type buyer_buff[L+1];
// with this TRIAL 2 I'll remove "&"in front of seller_buff and buyer_buff
// in MPI_Win_allocate_shared and MPI_Win_shared_query
// allocate shared memory
if (rank == 0) {
buff_size = (N+2) * (L+1) * sizeof(PWL);
MPI_Win_allocate_shared(buff_size, sizeof(PWL), MPI_INFO_NULL,
MPI_COMM_WORLD, &seller_buff, &win_seller);
MPI_Win_allocate_shared(buff_size, sizeof(PWL), MPI_INFO_NULL,
MPI_COMM_WORLD, &buyer_buff, &win_buyer);
}
else {
int disp_unit;
MPI_Win_allocate_shared(0, sizeof(PWL), MPI_INFO_NULL,
MPI_COMM_WORLD, &seller_buff, &win_seller);
MPI_Win_allocate_shared(0, sizeof(PWL), MPI_INFO_NULL,
MPI_COMM_WORLD, &buyer_buff, &win_buyer);
MPI_Win_shared_query(win_seller, 0, &buff_size, &disp_unit,
&seller_buff);
MPI_Win_shared_query(win_buyer, 0, &buff_size, &disp_unit,
&buyer_buff);
}
// up- and down- move factors
double u = exp( sigma * sqrt(expiry/N) );
// cash accumulation factor
double r = exp( R*expiry / N );
// initialize algorithm
int p(size);
int n = N + 2; // number of nodes in the current base level
int s = rank * ( n/p );
int e = (rank==p-1)? n: (rank+1) * ( n/p );
// each core works on e-s nodes in the current level
// compute u and z for both seller and buyer: payoff (0,0) at time N+1
for (int l=s; l<e; l++) {
const double St = S0*pow (u, N+1-2*l);
const double Sa = St * (1+k);
const double Sb = St * (1-k);
// compute functions
PWL u_s( {Line(-Sa, 0), Line(-Sb,0)} );
PWL u_b( {Line(-Sa, 0), Line(-Sb, 0)} );
// fill buffers
seller_buff[0][l] = u_s;
buyer_buff[0][l] = u_b;
}
MPI_Barrier(MPI_COMM_WORLD);
if (rank == 0 ) {
std::cout << "Row: " << 11 << std::endl
<< "\tAsk = " << seller_buff[0][7].valueInPoint(0) << std::endl
<< "\tBid = " << -buyer_buff[0][7].valueInPoint(0) << std::endl;
}
int U = 0; // variable for the mapping from tree to buffers
int B=N+1; // current base level
while ( B>0 ) {
// DO stuffs with buffers
}
// compute ask and bid prices
double ask(0), bid(0);
// clear shared windows
MPI_Win_free(&win_seller);
MPI_Win_free(&win_buyer);
return std::make_pair(bid, ask);
}
I added the "if" after MPI_Barrier to see if buffers work, where column 7 (with N=10) is supposed to be computed by rank 1.
Actually TRIAL 1 worked when using another simpler class, but with PWL class not. Error in two trials are:
1) In TRIAL 1 I get a segmentation fault due to the call of valueInPoint() in the if: the problem is that rank 0 cannot see what rank 1 wrote in its columns but I do not understand why.
mpiexec -np 3 main
[localhost:09623] *** Process received signal ***
[localhost:09623] Signal: Segmentation fault (11)
[localhost:09623] Signal code: Address not mapped (1)
[localhost:09623] Failing at address: 0x26fd440
[localhost:09623] [ 0] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libpthread.so.0(+0x10e20)[0x7fa78a99de20]
[localhost:09623] [ 1] main[0x4048ac]
[localhost:09623] [ 2] main[0x4048f8]
[localhost:09623] [ 3] main[0x401e55]
[localhost:09623] [ 4] main[0x40178e]
[localhost:09623] [ 5] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(__libc_start_main+0xf0)[0x7fa78a60c6b0]
[localhost:09623] [ 6] main[0x401389]
[localhost:09623] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 9623 on node localhost exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
make: *** [Makefile:26: run] Error 139
2) In this case rank 0 is able to access and print what rank 1 did, but I get another error.
mpiexec -np 3 main
Row: 11
Ask = 0
Bid = -0
[localhost:09651] *** Process received signal ***
[localhost:09651] Signal: Segmentation fault (11)
[localhost:09651] Signal code: Address not mapped (1)
[localhost:09651] Failing at address: 0x7fe9777c90bc
[localhost:09651] [ 0] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libpthread.so.0(+0x10e20)[0x7fe976627e20]
[localhost:09651] [ 1] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(cfree+0x14)[0x7fe9762eef74]
[localhost:09651] [ 2] main[0x404724]
[localhost:09651] [ 3] main[0x404096]
[localhost:09651] [ 4] main[0x40362e]
[localhost:09651] [ 5] main[0x403127]
[localhost:09651] [ 6] main[0x40274f]
[localhost:09651] [ 7] main[0x402528]
[localhost:09651] [ 8] main[0x4025e6]
[localhost:09651] [ 9] main[0x402017]
[localhost:09651] [10] main[0x40178e]
[localhost:09651] [11] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(__libc_start_main+0xf0)[0x7fe9762966b0]
[localhost:09651] [12] main[0x401389]
[localhost:09651] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 1 with PID 9651 on node localhost exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
make: *** [Makefile:26: run] Error 139
Moreover, actually when I run the all algorithm (without commenting the while loop) with TRIAL 2 i get another error:
mpiexec -np 3 main
Row: 11
Ask = 0
Bid = -0
*** Error in `main': free(): invalid pointer: 0x00007f2ccac660c4 ***
======= Backtrace: =========
/u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(+0x6f2e4)[0x7f2cc977f2e4]
/u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(+0x74d16)[0x7f2cc9784d16]
/u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(+0x754fe)[0x7f2cc97854fe]
main[0x405dd6]
main[0x40556c]
main[0x404930]
main[0x404429]
main[0x40396f]
main[0x404eb7]
main[0x404334]
main[0x403833]
main[0x4023ee]
main[0x40178e]
/u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(__libc_start_main+0xf0)[0x7f2cc97306b0]
main[0x401389]
======= Memory map: ========
00400000-0040e000 r-xp 00000000 00:25 659 /vagrant/Google Drive/ACP/TENTATIVO4/main
0060d000-0060e000 r--p 0000d000 00:25 659 /vagrant/Google Drive/ACP/TENTATIVO4/main
0060e000-0060f000 rw-p 0000e000 00:25 659 /vagrant/Google Drive/ACP/TENTATIVO4/main
00fac000-01239000 rw-p 00000000 00:00 0 [heap]
7f2cb0000000-7f2cb0021000 rw-p 00000000 00:00 0
7f2cb0021000-7f2cb4000000 ---p 00000000 00:00 0
7f2cb7fff000-7f2cc0000000 rw-s 00000000 fd:00 202783851 /tmp/openmpi-sessions-vagrant#localhost_0/63096/1/shared_mem_pool.localhost (deleted)
7f2cc0000000-7f2cc0021000 rw-p 00000000 00:00 0
7f2cc0021000-7f2cc4000000 ---p 00000000 00:00 0
7f2cc48f1000-7f2cc4cf2000 rw-s 00000000 fd:00 135477240 /tmp/openmpi-sessions-vagrant#localhost_0/63096/1/2/vader_segment.localhost.2
7f2cc4cf2000-7f2cc50f3000 rw-s 00000000 fd:00 68300033 /tmp/openmpi-sessions-vagrant#localhost_0/63096/1/1/vader_segment.localhost.1
7f2cc50f3000-7f2cc54f4000 rw-s 00000000 fd:00 1474379 /tmp/openmpi-sessions-vagrant#localhost_0/63096/1/0/vader_segment.localhost.0
7f2cc54f4000-7f2cc54ff000 r-xp 00000000 fd:00 2626640 /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libnss_files-2.23.so
7f2cc54ff000-7f2cc56fe000 ---p 0000b000 fd:00 2626640 /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libnss_files-2.23.so
7f2cc56fe000-7f2cc56ff000 r--p 0000a000 fd:00 2626640 /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libnss_files-2.23.so
7f2cc56ff000-7f2cc5700000 rw-p 0000b000 fd:00 2626640 /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libnss_files-2.23.so
7f2cc5700000-7f2cc5706000 rw-p 00000000 00:00 0
7f2cc5706000-7f2cc5707000 ---p 00000000 00:00 0
7f2cc5707000-7f2cc5f07000 rw-p 00000000 00:00 0 [stack:9979]
7f2cc5f07000-7f2cc5f2b000 r-xp 00000000 fd:00 4240669 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/liblzma.so.5.2.2
7f2cc5f2b000-7f2cc612b000 ---p 00024000 fd:00 4240669 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/liblzma.so.5.2.2
7f2cc612b000-7f2cc612c000 r--p 00024000 fd:00 4240669 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/liblzma.so.5.2.2
7f2cc612c000-7f2cc612d000 rw-p 00025000 fd:00 4240669 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/liblzma.so.5.2.2
7f2cc612d000-7f2cc6142000 r-xp 00000000 fd:00 1363668 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libz.so.1.2.8
7f2cc6142000-7f2cc6341000 ---p 00015000 fd:00 1363668 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libz.so.1.2.8
7f2cc6341000-7f2cc6342000 r--p 00014000 fd:00 1363668 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libz.so.1.2.8
7f2cc6342000-7f2cc6343000 rw-p 00015000 fd:00 1363668 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libz.so.1.2.8
7f2cc6343000-7f2cc7bbf000 r--p 00000000 fd:00 1549735 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicudata.so.57.1
7f2cc7bbf000-7f2cc7dbe000 ---p 0187c000 fd:00 1549735 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicudata.so.57.1
7f2cc7dbe000-7f2cc7dbf000 r--p 0187b000 fd:00 1549735 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicudata.so.57.1
7f2cc7dbf000-7f2cc7f4d000 r-xp 00000000 fd:00 1549736 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicuuc.so.57.1
7f2cc7f4d000-7f2cc814d000 ---p 0018e000 fd:00 1549736 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicuuc.so.57.1
7f2cc814d000-7f2cc815f000 r--p 0018e000 fd:00 1549736 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicuuc.so.57.1
7f2cc815f000-7f2cc8160000 rw-p 001a0000 fd:00 1549736 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicuuc.so.57.1
7f2cc8160000-7f2cc8162000 rw-p 00000000 00:00 0
7f2cc8162000-7f2cc83c3000 r-xp 00000000 fd:00 1549762 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicui18n.so.57.1
7f2cc83c3000-7f2cc85c3000 ---p 00261000 fd:00 1549762 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicui18n.so.57.1
7f2cc85c3000-7f2cc85d0000 r--p 00261000 fd:00 1549762 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicui18n.so.57.1
7f2cc85d0000-7f2cc85d2000 rw-p 0026e000 fd:00 1549762 /u/sw/pkgs/toolchains/gcc-glibc/5/base/lib/libicui18n.so.57.1
7f2cc85d2000-7f2cc85d3000 rw-p 00000000 00:00 0
7f2cc85d3000-7f2cc85d5000 r-xp 00000000 fd:00 2590182 /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libdl-2.23.so[localhost:09977] *** Process received signal ***
[localhost:09977] Signal: Aborted (6)
[localhost:09977] Signal code: (-6)
[localhost:09977] [ 0] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libpthread.so.0(+0x10e20)[0x7f2cc9ac1e20]
[localhost:09977] [ 1] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(gsignal+0x38)[0x7f2cc9743228]
[localhost:09977] [ 2] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(abort+0x16a)[0x7f2cc97446aa]
[localhost:09977] [ 3] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(+0x6f2e9)[0x7f2cc977f2e9]
[localhost:09977] [ 4] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(+0x74d16)[0x7f2cc9784d16]
[localhost:09977] [ 5] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(+0x754fe)[0x7f2cc97854fe]
[localhost:09977] [ 6] main[0x405dd6]
[localhost:09977] [ 7] main[0x40556c]
[localhost:09977] [ 8] main[0x404930]
[localhost:09977] [ 9] main[0x404429]
[localhost:09977] [10] main[0x40396f]
[localhost:09977] [11] main[0x404eb7]
[localhost:09977] [12] main[0x404334]
[localhost:09977] [13] main[0x403833]
[localhost:09977] [14] main[0x4023ee]
[localhost:09977] [15] main[0x40178e]
[localhost:09977] [16] /u/sw/pkgs/toolchains/gcc-glibc/5/prefix/lib/libc.so.6(__libc_start_main+0xf0)[0x7f2cc97306b0]
[localhost:09977] [17] main[0x401389]
[localhost:09977] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 9977 on node localhost exited on signal 6 (Aborted).
--------------------------------------------------------------------------
make: *** [Makefile:26: run] Error 134
Please could someone help me to understand what's going on and how to fix it?? Thank you all.
How to read a binary file to calculate frequency of Huffman tree?
I have to calculate frequency of Huffman tree from a "binary file" as sole argument. I have a doubt that binary files are the files which contains "0" and "1" only.
Whereas frequency is the repetition of the number of alphabets (eg, abbacdd here freq of a=2, b=2 ,c=1, d=2).
And my structure must be like this:
struct Node
{
unsigned char symbol; /* the symbol or alphabets */
int freq; /* related frequency */
struct Node *left,*right; /* Left and right leafs */
};
But i not at all understand how can i get the symbol and from ".bin" file (which consists of only "0" and "1") ?
When i try to see the contents of a file i get:
hp#ubuntu:~/Desktop/Internship_Xav/Huf_pointer$ xxd -b out.bin
0000000: 00000000 00000000 00000000 00000000 00000000 00000000 ......
0000006: 00000000 00000000 00000000 00000000 00000000 00000000 ......
000000c: 00000000 00000000 00000000 00000000 00000000 00000000 ......
0000012: 00000000 00000000 00000000 00000000 00000000 00000000 ......
0000018: 00000000 00000000 00000000 00000000 00000000 00000000 ......
000001e: 00000000 00000000 00000000 00000000 00000000 00000000 ......
0000024: 00000000 00000000 00000000 00000000 00000000 00000000 ......
000002a: 00000000 00000000 00000000 00000000 00000000 00000000 ......
0000030: 00000000 00000000 00000000 00000000 00000000 00000000 ......
.........//Here also there is similar kind of data ................
00008ca: 00010011 00010011 00010011 00010011 00010011 00010011 ......
00008d0: 00010011 00010011 00010011 00010011 00010011 00010011 ......
00008d6: 00010011 00010011 00010011 00010011 00010011 00010011 .....
So , I not at all understand where are the frequencies and where are the symbols. How to store the symbols and how to calculate frequencies. Actually after having frequencies and symbols i will create HUffman tree using it.
First, you need to create some sort of frequency table.
You could use a std::map.
You would do something like this:
#include <algorithm>
#include <fstream>
#include <map>
#include <string>
std::map <unsigned char, int> CreateFrequencyTable (const std::string &strFile)
{
std::map <unsigned char, int> char_freqs ; // character frequencies
std::ifstream file (strFile) ;
int next = 0 ;
while ((next = file.get ()) != EOF) {
unsigned char uc = static_cast <unsigned char> (next) ;
std::map <unsigned char, int>::iterator iter ;
iter = char_freqs.find (uc) ;
// This character is in our map.
if (iter != char_freqs.end ()) {
iter->second += 1 ;
}
// This character is not in our map yet.
else {
char_freqs [uc] = 1 ;
}
}
return char_freqs ;
}
Then you could use this function like this:
std::map <unsigned char, int> char_freqs = CreateFrequencyTable ("file") ;
You can obtain the element with the highest frequency like this:
std::map <unsigned char, int>::iterator iter = std::max_element (
char_freqs.begin (),
char_freqs.end (),
std::map <unsigned char, int>::value_comp
) ;
Then you would need to build your Huffman tree.
Remember that the characters are all leaf nodes, so you need a way to differentiate the leaf nodes from the non-leaf nodes.
Update
If reading a single character from the file is too slow, you could always load all of the contents into a vector like this:
// Make sure to #include <iterator>
std::ifstream file ("test.txt") ;
std::istream_iterator <unsigned char> begin = file ;
std::vector<unsigned char> vecBuffer (begin, std::istream_iterator <unsigned char> ()) ;
You would still need to create a frequency table.
A symbol in a huffman tree could be anything, but as you have to use an unsigned char per symbol you should probably take a byte? So no, not only 0 or 1, but eight time 0 or 1 together. Like 00010011 somewhere in your output of xxd xxd -b will just give you eight 0/1 per byte. You could write a number between 0 and 255 as well, or two times one character of 0123456789abcdef There are lots of possibilies how to show a byte on the screen, but that does not matter at all. If you know how to read the content of a file in C/C++, just read unsigned char until the file ends and count which value is how often in there. That´s all. As you´re probably writing decimal numbers in your program code, there are 256 different values (0,1,2...255). So you will need 256 integers (in an array, or your Node struct...) to count how often each value appears.
Explanation of an RGB to BGR C++ macro
#define RGB2BGR(a_ulColor) (a_ulColor & 0xFF000000) | ((a_ulColor & 0xFF0000) >> 16) | (a_ulColor & 0x00FF00) | ((a_ulColor & 0x0000FF) << 16) Can you please explain to me the meaning of this macro?
Colors are usually represented by a 32-bit integer. 32-bit integers can hold four 8-bit bytes. Three of them are used to hold red, green, and blue color information. The remaining byte is either left unused or used to hold transparency information. Which byte represents which color is not standardized. Some APIs expect the bytes like this: (MSB) ******** rrrrrrrr gggggggg bbbbbbbb (LSB) Which is the "RGB" layout, perhaps the most common form. In the illlustration above, the most sigificant 8-bits are the "don't care" bits, that is, the bits there are not used. The least significant 8-bits store the information for the blue color. Some APIs expect the reverse for the 3 color bytes, like this: (MSB) ******** bbbbbbbb gggggggg rrrrrrrr (LSB) Which is the "BGR" layout. The macro helps interconvert the two layouts using the bitwise operators. Let's take a look at its definition: (a_ulColor & 0xFF000000) | ((a_ulColor & 0xFF0000) >> 16) | (a_ulColor & 0x00FF00) | ((a_ulColor & 0x0000FF) << 16) Let's say we have a color, Cornflower Blue, which has a value of 0x93CCEA. In the RGB layout, it has the following bit pattern: a_ulColor = 00000000 10010011 11001100 11101010 The following expressions give you the following patterns: 1. a_ulColor & 0xFF000000 --> 00000000 00000000 00000000 00000000 2. a_ulColor & 0xFF0000 --> 00000000 10010011 00000000 00000000 3. a_ulColor & 0x00FF00 --> 00000000 00000000 11001100 00000000 4. a_ulColor & 0x0000FF --> 00000000 00000000 00000000 11101010 Notices that we're just extracting the individual bytes. Expression #1 extracts the most significant 8-bits, and expression #4 extracts the least signficiant 8-bits. We were able to do this via the AND bitwise operation. Now, to convert RGB to BGR, we have to move some bits left or right, via bitshifts. Like this: 1. (a_ulColor & 0xFF000000) --> 00000000 00000000 00000000 00000000 2. (a_ulColor & 0xFF0000) >> 16 --> 00000000 00000000 00000000 10010011 3. (a_ulColor & 0x00FF00) --> 00000000 00000000 11001100 00000000 4. (a_ulColor & 0x0000FF) << 16 --> 00000000 11101010 00000000 00000000 The expression a >> 16 simply shifts the bits to the right by 16 bits. a << 16 shifts the bits to the left by 16 bits. Then, when you OR them all together, you get this: 00000000 11101010 11001100 10010011 Compare the result to the original bit pattern: 00000000 11101010 11001100 10010011 00000000 10010011 11001100 11101010 You can see that the 2nd and 4th bytes are swapped. That's all the macro does.
It takes a four-byte integral value, AA BB CC DD, and returns the value AA DD CC BB. You can see that the first and third byte are retained unchanged, while the second byte is moved down two bytes (>> 16) and the fourth is moved up by two (<< 16).
It swaps the order of the byte-sized RGB elements from RGB to BGR (and vice-versa, to be fair).
a_ulColor is a 32 bit RGB representation (e.g. of a pixel or bitmap). The macro converts it to BGR layout. It effectively produces a new value by swapping the Red and Blue component values.