Concatenate binary bit-sized strings - c++

Question

I want to write to a file a series of binary strings whose length is expressed in bits rather than bytes. Take in consideration two strings s1 and s2 that in binary are respectively 011 and 01011. In this case the contents of the output file has to be: 01101011 (1 byte). I am trying to do this in the most efficient way possible since I have several million strings to concatenate for a total of several GB in output.

Answer 1

C++ has no way of working directly with bits because it aims at being a light layer
above the hardware and the hardware itself is not bit oriented. The very minimum
amount of bits you can read/write in one operation is a byte (normally 8 bits).
Also if you need to do disk i/o it's better to write your data in blocks instead of one byte at a time. The library has some buffering, but the earlier things are buffered the faster the code will be (less code is involved in passing data around).
A simple approach could be
unsigned char iobuffer[4096];
int bufsz; // how many bytes are present in the buffer
unsigned long long bit_accumulator;
int acc_bits; // how many bits are present in the accumulator
void writeCode(unsigned long long code, int bits) {
bit_accumulator |= code << acc_bits;
acc_bits += bits;
while (acc_bits >= 8) {
iobuffer[bufsz++] = bit_accumulator & 255;
bit_accumulator >>= 8;
acc_bits -= 8;
if (bufsz == sizeof(iobuffer)) {
// Write the buffer to disk
bufsz = 0;
}
}
}

Answer 2

There is no optimal way to solve your problem per se. But you can use a few pinches to speed things up:
Experiment with the file I/O sync flag. It might be that set/unset is significantly faster that the other, because of buffering and caching.
Try to use architecture sized variables so that they fit into the registers directly: uint32_t for 32 bit machines and uint64_t for 64 bit machines ...
"Volatile" might help to, keep things in registers
Use pointer and references for large data and copy small data blobs (to avoid unnecessary copy of large data and much lookups and page touching for small data)
Use mmap of the file for direct access and align your output to the page size of your architecture and hard disk (usually 4 KiB = 4096 Bytes)
Try to reduce branching (instructions like "if", "for", "while", "() ? :") and linearize your code.
And if that is not enough and when the going gets rough: Use assembler (but I would not recommend that for beginners)
I think multi threading would be contra productive in this case, because of the limited file writes that can be issued and the problem is not easy dividable into little tasks as each one needs to know how many bits after the other ones it has to start and then you would have to join all the results together in the end.

Answer 3

I've used the following in the past, it might help a bit...
FileWriter.h:
#ifndef FILE_WRITER_H
#define FILE_WRITER_H
#include <stdio.h>
class FileWriter
{
public:
FileWriter(const char* pFileName);
virtual ~FileWriter();
void AddBit(int iBit);
private:
FILE* m_pFile;
unsigned char m_iBitSeq;
unsigned char m_iBitSeqLen;
};
#endif
FileWriter.cpp:
#include "FileWriter.h"
#include <limits.h>
FileWriter::FileWriter(const char* pFileName)
{
m_pFile = fopen(pFileName,"wb");
m_iBitSeq = 0;
m_iBitSeqLen = 0;
}
FileWriter::~FileWriter()
{
while (m_iBitSeqLen > 0)
AddBit(0);
fclose(m_pFile);
}
void FileWriter::AddBit(int iBit)
{
m_iBitSeq |= iBit<<CHAR_BIT;
m_iBitSeq >>= 1;
m_iBitSeqLen++;
if (m_iBitSeqLen == CHAR_BIT)
{
fwrite(&m_iBitSeq,1,1,m_pFile);
m_iBitSeqLen = 0;
}
}
You can further improve it by accumulating the data up to a certain amount before writing it into the file.