I have implemented the Huffman coding algorithm in C++, and it's working fine. I want to create a text compression algorithm.
behind every file or data in the digital world, there is 0/1.
I want to persist the sequence of bits(0/1) that are generated by the Huffman encoding algorithm in the file.
my goal is to save the number of bits used in the file to store. I'm storing metadata for decoding in a separate file. I want to write bit by bit data to file, and then read the same bit by bit in c++.
the problem I'm facing with the binary mode is that it not allowing me to put data bit by bit.
I want to put "10101" as bit by bit to file but it put asci values or 8-bits of each character at a time.
code
#include "iostream"
#include "fstream"
using namespace std;
int main(){
ofstream f;
f.open("./one.bin", ios::out | ios::binary);
f<<"10101";
f.close();
return 0;
}
output
any help or pointer to help is appreciated. thank you.
"Binary mode" means only that you have requested that the actual bytes you write are not corrupted by end-of-line conversions. (This is only a problem on Windows. No other system has the need to deliberately corrupt your data.)
You are still writing a byte at a time in binary mode.
To write bits, you accumulate them in an integer. For convenience, in an unsigned integer. This is your bit buffer. You need to decide whether to accumulate them from the least to most or from the most to least significant positions. Once you have eight or more bits accumulated, you write out one byte to your file, and remove those eight bits from the buffer.
When you're done, if there are bits left in your buffer, you write out those last one to seven bits to one byte. You need to carefully consider how exactly you do that, and how to know how many bits there were, so that you can properly decode the bits on the other end.
The accumulation and extraction are done using the bit operations in your language. In C++ (and many other languages), those are & (and), | (or), >> (right shift), and << (left shift).
For example, to insert one bit, x, into your buffer, and later three bits in y, ending up with the earliest bits in the most significant positions:
unsigned buf = 0, bits = 0;
...
// some loop
{
...
// write one bit (don't need the & if you know x is 0 or 1)
buf = (buf << 1) | (x & 1);
bits++;
...
// write three bits
buf = (buf << 3) | (y & 7);
bits += 3;
...
// write bytes from the buffer before it fills the integer length
if (bits >= 8) { // the if could be a while if expect 16 or more
// out is an ostream -- must be in binary mode if on Windows
bits -= 8;
out.put(buf >> bits);
}
...
}
...
// write any leftover bits (it is assumed here that bits is in 0..7 --
// if not, first repeat if or while from above to clear out bytes)
if (bits) {
out.put(buf << (8 - bits));
bits = 0;
}
...
Related
I was working on a Huffman project to compress text files. I was able to generate the required codes. I read the whole file and accordingly stored the codes in a "vector char" variable. I also padded the encoded vector.
vector<char> padding(vector<char> text)
{
int num = text.size();
unsigned int pad_value = 32-(num%32);
for(int i=0;i<pad_value;i++){
text.push_back('0');
}
string pad_info = bitset<32>(pad_value).to_string();
for(int i=pad_info.length()-1;i>=0;i--){
text.insert(text.begin(),pad_info[i]);
}
return text;
}
I padded on the base of 32 bits, as I was thinking if using an array of "unsigned int" to directly store the integers in a binary file so that they occupy 4 bytes for every 32 characters. I used this function for that:
vector<unsigned int> build_byte_array(vector<char> padded_text)
{
vector<unsigned int> byte_arr;
for(int i=0;i<padded_text.size();i+=32)
{
string byte="";
for(int j=i;j<i+32;j++){
byte += padded_text[j];
}
unsigned int b = stoul(byte,nullptr,2);
//cout<<b<<":"<<byte<<endl;
byte_arr.push_back(b);
}
return byte_arr;
}
Now the problem is when I write this byte array to binary file using
ofstream output("compressed.bin",ios::binary);
for(int i=0;i<byte_array.size();i++){
unsigned int a = byte_array[i];
output.write((char*)(&a),sizeof(a));
}
I get a binary file which is bigger than the original text file. How do I solve that or what error am I making.
Edit : I tried to compress a file of about 2,493 KB (for testing purposes) and it generated a compressed.bin file of 3,431 KB. So, I don't think padding is the issue here.
I also tried with 15KB file but the size of always increases after using this algo.
I tried using:
for(int i=0;i<byte_array.size();i++){
unsigned int a = byte_array[i];
char b = (char)a;
output.write((char*)(&a),sizeof(b));
}
but after using this I am unable to recover the original byte array when decompressing the file.
unsigned int a = byte_array[i];
output.write((char*)(&a),sizeof(a));
The size of the write is sizeof(a) which is usually 4 bytes.
An unsigned int is not a byte. A more suitable type for a byte would be std::byte, uint8_t, or unsigned char.
You are expanding your data with padding, so if you're not getting much compression or there's not much data to begin with, the output could easily be larger.
You don't need to pad nearly as much as you do. First off, you are adding 32 bits when the data already ends on a word boundary (when num is a multiple of 32). Pad zero bits in that case. Second, you are inserting 32 bits at the start to record how many bits you padded, where five bits would suffice to encode 0..31. Third, you could write bytes instead of ints, so the padding on the end could be 0..7 bits, and you could prepend three bits instead of five. The padding overall could be reduced from your current 33..64 bits to 3..10 bits.
my task is to read metadata values from a unsigned char array, which contains the bytes of a binary .shp file (Shapefile)
unsigned char* bytes;
The header of the file which is stored in the array and the order of the information stored in it looks like this:
int32_t filecode // BigEndian
int32_t skip[5] // Uninteresting stuff
int32_t filelength // BigEndian
int32_t version // LitteEndian
int32_t shapetype // LitteEndian
// Rest of the header and of the filecontent which I don't need
So my question would be how can I extract this information (except the skip part of course) under consideration of the endianness and read it into the according variables.
I thought about using ifstream, but I couldnt figure out how to do it properly.
Example:
Read the first four bytes of the binary, ensure big endian byte order, store it in a int32_t. Then skip 5* 4 Bytes (5 * int32). Then read four bytes, ensure big endian byte order, and store it in a int32_t. Then read four bytes, ensure little endian byte order, and again store it in a int32_t and so on.
Thanks for your help guys!
So 'reading' a byte array just means extracting the bytes from the positions in the byte array where you know your data is stored. Then you just need to do the appropriate bit manipulations to handle the endianess. So for example, filecode would be this
filecode = (bytes[0] << 24) | (bytes[1] << 16) | (bytes[2] << 8) | bytes[3];
and version would be this
version = bytes[13] | (bytes[14] << 8) | (bytes[15] << 16) | (bytes[16] << 24);
(An offset of 13 for the version seems a bit odd, I'm just going on what you stated above).
Again I've got a little problem with my DLL:
I try to convert a number (in this case "20") to a char which I can write to the file.
It doesn't really matter in which way this is done (whether following the ascii-table or not), but I need a way to convert back as well.
This was my attempt:
file.write((char*)20,3);
But it's throwing an access violence error..
Could someone tell me how this is done and also how I can reverse the process?
I could also use a method which works with numbers larger than 255 so the result are for example two or three chars (two chars = 16-bit-number.
Anyone have an idea?
If you just want to write an arbitrary byte, you can do this:
file.put(20);
or
char ch = 20;
file.write(&ch, 1); // Note a higher digit than 1 here will mean "undefined behaviour".
To reverse the process, you'd use file.get() or file.read(&ch, 1).
For larger units than a single byte, you'll have to use file.write(...), but it gets less portable, since it now relies on the size of the value being the same between different plaforms, AND that the internal representation is the same. This is not a problem if you are always running this on the same type of machine (Windows on an x86 processor, for example), but it will be a problem if you start using the code on different types of machines (x86, Sparc, ARM, IBM mainframe, Mobile phone DSP, etc) and possibly also between different OS's.
Something like this will work with the above restrictions:
int value = 4711;
file.write((char *)&value, sizeof(value));
It is much more portable to write this value to a file in text-form, which can be read by any other computer than recognises the same character encoding.
This will convert an unsigned long long into multiple characters depending on how big the number is, and output them to a file.
#include <fstream>
int main() {
unsigned long long number = 2098798987879879999;
std::ofstream out("out.txt");
while (number) { // While number != 0.
unsigned long long n = number & 255; // Copy the 8 rightmost bits.
number >>= 8; // Shift the original number 8 bits right.
out << static_cast<unsigned char>(n); // Cast to char and output.
}
out << std::endl; // Append line break for every number.
}
You can read it back from a file using something like this
#include <iostream>
#include <fstream>
#include <algorithm>
#include <string>
int main() {
std::ifstream in("out.txt");
unsigned long long number = 0;
std::string s;
std::getline(in, s); // Read line of characters.
std::reverse(begin(s), end(s)); // To account for little-endian order.
for (unsigned char c : s) {
number <<= 8;
number |= c;
}
std::cout << number << std::endl;
}
This outputs
2098798987879879999
I am using huffman algorithm to develop a file compressor and right now I am facing a problem which is:
By using the algorithm to the word:
stackoverflow, i get the following result:
a,c,e,f,k,l,r,s,t,v,w = 1 time repeated
o = 2 times repeated
a,c,e,f,k,l,r,s,t,v,w = 7.69231%
and
o = 15.3846%
So I start inserting then into a Binary Tree, which will get me the results:
o=00
a=010
e=0110
c=0111
t=1000
s=1001
w=1010
v=1011
k=1100
f=1101
r=1110
l=1111
which means the path for the character in the tree, considering 0 to be left and 1 to right.
then the word "stackoverflow" will be:
100110000100111010011111000010110110111011011111001010
and well, I want to put that whole value into a binary file to be in bits, which will result in 47bits, which would happen to be 6bytes, but instead I can only make it 47bytes because the minimun to put into a file with fwrite or fprintf is 1byte, by using sizeof(something).
Than my question is: how can I print in my file only a single bit?
Just write the "header" to the file: the number of bits and then "pack" the bits into bytes padding the last one. Here's a sample.
#include <stdio.h>
FILE* f;
/* how many bits in current byte */
int bit_counter;
/* current byte */
unsigned char cur_byte;
/* write 1 or 0 bit */
void write_bit(unsigned char bit)
{
if(++bit_counter == 8)
{
fwrite(&cur_byte,1,1,f);
bit_counter = 0;
cur_byte = 0;
}
cur_byte <<= 1;
cur_byte |= bit;
}
int main()
{
f = fopen("test.bits", "w");
cur_byte = 0;
bit_counter = 0;
/* write the number of bits here to decode the bitstream later (47 in your case) */
/* int num = 47; */
/* fwrite(num, 1, 4, f); */
write_bit(1);
write_bit(0);
write_bit(0);
/* etc... - do this in a loop for each encoded character */
/* 100110000100111010011111000010110110111011011111001010 */
if(bit_counter > 0)
{
// pad the last byte with zeroes
cur_byte <<= 8 - bit_counter;
fwrite(&cur_byte, 1, 1, f);
}
fclose(f);
return 0;
}
To do the full Huffman encoder you'll have to write the bit codes at the beginning, of course.
This is sort of an encoding issue. The problem is that files can only contain bytes - so 1 and 0 can only be '1' and '0' in a file - the characters for 1 and 0, which are bytes.
What you'll have to do is to pack the bits into bytes, creating a file that contains the bits in a set of bytes. You won't be able to open the file in a text editor - it doesn't know you want to display each bit as a 1 or 0 char, it will display whatever each packed byte turns out to be. You could open it with an editor that understands how to work with binary files, though. For instance, vim can do this.
As far as extra trailing bytes or an end-of-file marker, you're going to have to create some sort of encoding convention. For example, you can pack and pad with extra zeros, like you mention in your comments, but then by convention have the first N bytes be metadata - e.g. the data length, how many bits are interesting in your file. This sort of thing is very common.
You'll need to manage this yourself, by buffering the bits to write and only actually writing data when you have a complete byte. Something like...
void writeBit(bool b)
{
static char buffer=0;
static int bitcount=0;
buffer = (buffer << 1) | (b ? 1:0);
if (++bitcount == 8)
{
fputc(buffer); // write out the byte
bitcount = 0;
buffer = 0;
}
}
The above isn't reentrant (and is likely to be pretty inefficient) - and you need to make sure you somehow flush any half-written byte at the end, (write an extra 7 zero bits, maybe) but you should get the general idea.
Can't exactly find a way on how to do the following in C/C++.
Input : hexdecimal values, for example: ffffffffff...
I've tried the following code in order to read the input :
uint16_t twoBytes;
scanf("%x",&twoBytes);
Thats works fine and all, but how do I split the 2bytes in 1bytes uint8_t values (or maybe even read the first byte only). Would like to read the first byte from the input, and store it in a byte matrix in a position of choosing.
uint8_t matrix[50][50]
Since I'm not very skilled in formating / reading from input in C/C++ (and have only used scanf so far) any other ideas on how to do this easily (and fast if it goes) is greatly appreciated .
Edit: Found even a better method by using the fread function as it lets one specify how many bytes it should read from the stream (stdin in this case) and save to a variable/array.
size_t fread ( void * ptr, size_t size, size_t count, FILE * stream );
Parameters
ptr - Pointer to a block of memory with a minimum size of (size*count) bytes.
size - Size in bytes of each element to be read.
count - Number of elements, each one with a size of size bytes.
stream - Pointer to a FILE object that specifies an input stream.
cplusplus ref
%x reads an unsigned int, not a uint16_t (thought they may be the same on your particular platform).
To read only one byte, try this:
uint32_t byteTmp;
scanf("%2x", &byteTmp);
uint8_t byte = byteTmp;
This reads an unsigned int, but stops after reading two characters (two hex characters equals eight bits, or one byte).
You should be able to split the variable like this:
uint8_t LowerByte=twoBytes & 256;
uint8_t HigherByte=twoBytes >> 8;
A couple of thoughts:
1) read it as characters and convert it manually - painful
2) If you know that there are a multiple of 4 hexits, you can just read in twobytes and then convert to one-byte values with high = twobytes << 8; low = twobyets & FF;
3) %2x