lz4 compression c++ example [duplicate] - c++

This question already has answers here:
Question about seekg() function of ifstream in C++?
(3 answers)
Closed 8 years ago.
In the process of writing a lz4 csv to compressed binary file converter (high volume forex tick data csv) in the hope of reducing the storage/disk bandwidth requirements on my tiny vps.
self contained code to illustrate
#include <string>
#include <fstream>
#include <iostream>
#include "lz4.h"
using namespace std;
int main()
{
char szString[] = "2013-01-07 00:00:04,0.98644,0.98676 2013-01-07 00:01:19,0.98654,0.98676 2013-01-07 00:01:38,0.98644,0.98696";
const char* pchSource = szString;
int nInputSize = sizeof(szString);
cout <<"- pchSource -" << endl << pchSource << endl;
cout <<"nbytes = "<< nInputSize << endl << endl;
ofstream source("pchSource.txt");
source << pchSource;
int nbytesPassed = 0;
int nMaxCompressedSize = LZ4_compressBound(nInputSize);
char *pszDest = new char[nMaxCompressedSize];
nbytesPassed = LZ4_compress(pchSource, pszDest, nInputSize);
cout <<"- pszDest Compressed-" << endl;
cout <<"nbytesPassed = "<< nbytesPassed << endl;
cout << pszDest << endl << endl;
// pszDest garbage ?
char *pszDestUnCompressed = new char[nInputSize];
LZ4_uncompress(pszDest, pszDestUnCompressed, nInputSize);
cout <<"- pszDestUnCompressed -" << endl;
cout <<"nbytesPassed = "<< nbytesPassed << endl;
cout << pszDestUnCompressed << endl << endl;
//pszDestUnCompressed is correct ?
delete[] pszDestUnCompressed;
pszDestUnCompressed = 0;
// ok lets write compressed pszDest to pszDest.dat
ofstream outCompressedFile("pszDest.dat",std::ofstream::binary);
outCompressedFile.write (pszDest,nMaxCompressedSize);
delete[] pszDest;
pszDest = 0;
//read it back in and try to uncompress it
ifstream infile("pszDest.dat",std::ifstream::binary);
infile.seekg (0,infile.end);
int nCompressedInputSize = infile.tellg();
infile.seekg (0);
char* buffer = new char[nCompressedInputSize];
infile.read (buffer,nCompressedInputSize);
const char* pchbuffer = buffer;
char* pszUnCompressedFile = new char[nInputSize];
nbytesPassed = LZ4_uncompress(pchbuffer, pszUnCompressedFile, nInputSize);
cout <<"- pszUnCompressedFile -" << endl;
cout <<"nbytesPassed = "<< nbytesPassed << endl;
cout << pszUnCompressedFile << endl;
//write uncompressed pszDest.dat to pszUnCompressedFile.txt
ofstream outUncompressedSource("pszUnCompressedFile.txt");
outUncompressedSource << pszUnCompressedFile;
// On my system 32bit ArchLinux 3.7.10-1 - gcc 4.7.2-4
// file contains random Garbage
delete[] buffer;
buffer = 0;
delete[] pszUnCompressedFile;
pszUnCompressedFile = 0;
return 0;
}
CONSOLE OUTPUT :
- pchSource -
2013-01-07 00:00:04,0.98644 .....
nbytes = 108
- pszDest Compressed-
nbytesPassed = 63
�2013-01-07 00:
- pszDestUnCompressed -
nbytesPassed = 63
2013-01-07 00:00:04,0.98644 .....
- pszUnCompressedFile -
nbytesPassed = -17
�W��W�-07 q
Process returned 0 (0x0) execution time : 0.010 s
Press ENTER to continue.
I'm obviously missing something, apart form the samples included in the source are there any-other usage examples ?

All working now thanks, here is the code for anyone that is interested
#include <fstream>
#include <iostream>
#include "lz4.h"
using namespace std;
int main()
{
char szSource[] = "2013-01-07 00:00:04,0.98644,0.98676 2013-01-07 00:01:19,0.98654,0.98676 2013-01-07 00:01:38,0.98644,0.98696";
int nInputSize = sizeof(szSource);
// compress szSource into pchCompressed
char* pchCompressed = new char[nInputSize];
int nCompressedSize = LZ4_compress((const char *)(&szSource), pchCompressed, nInputSize);
// write pachCompressed to binary lz4.dat
ofstream outBinaryFile("lz4.dat",ofstream::binary);
outBinaryFile.write(pchCompressed, nCompressedSize);
outBinaryFile.close();
delete[] pchCompressed;
pchCompressed = 0;
//read compressed binary file (assume we pass/encode nInputSize but don't know nCompressedSize)
ifstream infCompressedBinaryFile( "lz4.dat", ifstream::binary );
//Get compressed file size for buffer
infCompressedBinaryFile.seekg (0,infCompressedBinaryFile.end);
int nCompressedInputSize = infCompressedBinaryFile.tellg();
infCompressedBinaryFile.clear();
infCompressedBinaryFile.seekg(0,ios::beg);
//Read file into buffer
char* pchCompressedInput = new char[nCompressedInputSize];
infCompressedBinaryFile.read(pchCompressedInput,nCompressedSize);
infCompressedBinaryFile.close();
// Decompress buffer
char* pchDeCompressed = new char[nInputSize]; //(nCompressedInputSize *2) +8
LZ4_uncompress(pchCompressedInput, pchDeCompressed, nInputSize);
delete[] pchCompressedInput;
pchCompressedInput = 0;
// write decompressed pachUnCompressed to
ofstream outFile("lz4.txt");
outFile.write(pchDeCompressed, nInputSize);
outFile.close();
delete[] pchDeCompressed;
pchDeCompressed = 0;
return 0;
}
I am also working on a a simple CLI csv to binary I/O example here

Related

How do I read the whole 64 bytes of a binary file?

I am writing a little program that reads a disk image file in binary and then checks its partition entry tables to display each partition, it's type, start sector and size.
So far it reads the first 16 bytes accurately but the rest of the partition entries are not recognized or have some kind of error.
The result looks like this:
EDIT: The first line of the output is supposed to look like this:
`Partition 0: Type: FAT-16 Start: 63 Size: 518760`
What am I missing? How do I fix the code so that all the partition entries give the appropriate result?
using namespace std;
#include <iostream>
#include <fstream>
struct Partition { char type; int start_sect; int size; } part_entry[4]; // 4 x partition table entry
int main(int argc, char *argv[])
{
//DECLARATIONS
int i, offset = 26, not_exist = 0;
char buf_part_table[64], vol_type[12];
char* diskdata;
int n;
streampos begin, end;
ifstream diskimage;
diskimage.open("Sample_1.dd", ios::in | ios::binary | ios::out);
diskdata = new char[begin];
begin = diskimage.tellg();
diskdata = new char[begin];
diskimage.seekg(446, ios::beg);
diskimage.read(buf_part_table, 64);
for (i = 0; i < 4; i++)
{
part_entry[i].type = *(char*)(buf_part_table + 0x04 + (i * offset));
if (part_entry[i].type == 0) not_exist++;
part_entry[i].start_sect = *(int*)(buf_part_table + 0x08 + (i * offset));
part_entry[i].size = *(int*)(buf_part_table + 0x0C + (i * offset));
switch (part_entry[i].type)
{
case 00: strcpy(vol_type, "NOT-VALID");
break;
case 06: strcpy(vol_type, "FAT-16");
break;
case 07: strcpy(vol_type, "NTFS");
break;
case 0x0B: strcpy(vol_type, "FAT-32");
break;
default: strcpy(vol_type, "NOT-DECODED");
break;
}
cout << "Partition " << i << ":" << " Type:" << vol_type << " Start: " << part_entry[i].start_sect << " Size: " << part_entry[i].size << endl;
}
return 0;
}
You unnecesary made program unreadable and harder to debug.
You can read whole boot sector at once and than display desired content.
Here is my quick example (it does not check if file exists, some may complain it should use memcpy for some fields etc.)
#include <iostream>
#include <fstream>
#include <cstdint>
#include <cstddef>
#include <iomanip>
using namespace std;
struct partition_t {
uint8_t status;
uint8_t start_CHS[3];
uint8_t type;
uint8_t end_CHS[3];
uint32_t start_LBA;
uint32_t size_LBA;
} __attribute__((packed));
struct mbr_t
{
uint8_t bootstrap[446];
partition_t partitions[4];
uint16_t signature;
} __attribute__((packed));
int main(int argc, char *argv[])
{
mbr_t mbr;
ifstream diskimage;
diskimage.open( "/tmp/mbr.dd", ios::in | ios::binary );
diskimage.read( reinterpret_cast<char*>(&mbr), sizeof(mbr) );
diskimage.close();
for( int idx = 0 ; idx < 4 ; idx++ )
{
string bootable = (mbr.partitions[idx].status == 128) ? "yes" : "no";
cout << " bootable : " << setw(5) << bootable <<
" type : " << setw(5) << (int)mbr.partitions[idx].type <<
" start LBA : " << setw(10) << mbr.partitions[idx].start_LBA <<
" size : " << setw(10) << mbr.partitions[idx].size_LBA << endl;
}
return 0;
}
It is easier to read, right?

C++ Save a Struct String into A Text File

In my program, I save high scores along with a time in minutes and seconds. In my code, I currently store this as two ints in a struct called highscore. However, this is a little tedious when it comes to formatting when I display the output. I want to display the times as 12:02 not 12:2. I have a variable already made called string clock throughout my game, it is already formatted with the colon, all I want to do is add that inside my text file.
How can I refactor my code to have a single variable for the timestamp, which will be correctly formatted? I want to be able to write my data into the file by directly calling the structure.
// Used for Highscores
struct highscore
{
char name[10];
int zombiesKilled;
// I would like these to be a single variable
int clockMin;
int clockSec;
char Date[10];
};
// I write the data like this:
highscore data;
// ...
data[playerScore].clockMin = clockData.minutes;
data[playerScore].clockSec = clockData.seconds;
streaming = fopen( "Highscores.dat", "wb" );
fwrite( data, sizeof(data), 1 , streaming);
// ...
It seems that you want to simply just write a C-string or std::string to a file using C's fwrite() function.
This should be quite easy, given that your C-string is in ASCII-conforming format (no Unicode funny business):
//It appears you want to use C-style file I/O
FILE* file = NULL;
fopen("Highscores.dat", "wb");
//std::string has an internal C-string that you can access
std::string str = "01:00";
fwrite(str.c_str(), sizeof(char), sizeof(str.c_str()), file);
//You can also do this with regular C strings if you know the size.
We can also choose to try and use C++-style file I/O for cleaner interfaces.
#include <iostream>
#include <string>
#include <iomanip>
#include <fstream>
int main() {
std::string str = "00:11";
std::ofstream file("example.txt");
if (file.good()) {
file << str;
std::cout << "Wrote line to file example.txt.\n";
}
file.close();
//Let's check if we actually wrote the file.
std::ifstream read("example.txt");
std::string buffer;
if (read.good())
std::cout << "Opened example.txt.\n";
while(std::getline(read, buffer)) {
std::cout << buffer;
}
return 0;
}
Additionally, there are data types in <chrono> that can prove quite helpful for times like there.
If you want to be able to do this:
file << data_struct;
then it would make sense for you to create an operator overload for std::ostream.
You can experiment with time functions. And reading/writing structures.
The right way however is to use c++ basic file storage instead of dumping binar data.
struct highscore
{
char name[10];
int n;
std::time_t dateTime;
};
int main()
{
int total_seconds = 61;
char buf[50];
sprintf(buf, "minutes:seconds=> %02d:%02d", total_seconds / 60, total_seconds % 60);
cout << buf << endl;
std::time_t timeNow = std::time(NULL);
std::tm timeFormat = *std::localtime(&timeNow);
cout << "Current date/time " << std::put_time(&timeFormat, "%c %Z") << endl;
highscore data;
//write data:
{
FILE *streaming = fopen("Highscores.dat", "wb");
strcpy(data.name, "name1");
data.n = 1;
data.dateTime = std::time(NULL);
fwrite(&data, sizeof(data), 1, streaming);
strcpy(data.name, "name2");
data.n = 2;
data.dateTime = std::time(NULL);
fwrite(&data, sizeof(data), 1, streaming);
fclose(streaming);
}
//read data:
{
FILE *streaming = fopen("Highscores.dat", "rb");
fread(&data, sizeof(data), 1, streaming);
cout << "reading:\n";
cout << data.name << endl;
cout << data.n << endl;
timeFormat = *std::localtime(&data.dateTime);
cout << std::put_time(&timeFormat, "%c %Z") << endl;
cout << endl;
fread(&data, sizeof(data), 1, streaming);
cout << "reading:\n";
cout << data.name << endl;
cout << data.n << endl;
timeFormat = *std::localtime(&data.dateTime);
cout << std::put_time(&timeFormat, "%c %Z") << endl;
cout << endl;
fclose(streaming);
}
return 0;
}

Reading a char* from a binary file

I have to store the name of a file into a binary file that I am writing, I currently have written it like this:
void write(map<char, bits> &bitstring,map<char,int> &ccount, string header,string fname,ofstream &outf)
{
ifstream inf(header+fname);
cout << "FName: " << fname << endl;
const char * pName = fname.c_str();
fname = header+ fname + ".mcp";
const char * c = fname.c_str();
FILE* pFile;
pFile = fopen(c, "w+b");
inf.seekg(ios::beg, ios::end);
int size = inf.tellg();
int length = 0;
string data = "";
int magicNum = 2262;
int fileNameSize = strlen(pName);
fwrite(&fileNameSize, sizeof(int), 1, pFile);
cout <<"FIle Name Size: "<< fileNameSize << endl;
fwrite(pName, fileNameSize, 1, pFile);
fclose(pFile);
}
And I also send the size of the file name, so that I know how much data I need to read to get the whole file name.
void read2(string fname, map<char, int> &charCounts, std::vector<bool> &bits,ofstream &outf)
{
string fname1 = fname + ".mcp", outfile = "binarycheck";
bool testDone = false, counts = false;
std::ifstream inf(fname1, std::ios::binary);
std::ofstream ouf("binarycheck.txt", std::ios::binary);
char character;
int count[1] = { 0 };
int checkcount = 0;
int mNum[1] = { 0 }, size[1] = { 0 };
int FNamesize = 0;
inf.read((char*)&FNamesize, sizeof(int));
char *name=new char[FNamesize+1];
inf.read(name, FNamesize);
name[FNamesize] = '\0';
string str(name);
cout << "File Name: ";
cout << std::string(name) << endl;
cout << "Magic Num: " << mNum[0] << endl;
cout << "File Name Size: " << FNamesize<< endl;
inf.close();
}
I get the Size correctly, but I have no idea how to iterate through name in order to save it back as a string. I tried using a vector but it didn't really help since inf.read uses a char* as its first parameter.
Any help would be great.
Well, in a fluke accident I ended up solving my own issue. For some reason when I declared
FILE* pFile;
pFile = fopen(c, "w+b");
Before the declaration of
const char * pName = fname.c_str();
The call corrupted the value of pName before it was written to the file, which is what caused the errors. Problem solved!
Seeing as you're using ifstream, why not also use ofstream? Then it would just be ofs << filename to store and ifs >> filename to read where filename is a string. No need to faff around with the length yourself.

Copy string until '.' And how to copy only numbers when i know the structure

I have a code that i want it to get input file from command line and create output file with XXX at the end - meanning if intput= "blabla.txt" or "/johny/first/blabla.txt" i till get "blablaXXX.txt" or "/johny/first/blablaXXX.txt"
The second question is that when i find a line i was looking for i want to copy only the numbers (keep in date mode) and the len
Line will be "IT IS HERE time 12:04:56.186, len 000120"
And i want to get in the new file line: 12:04:56.186 120
#include <iostream>
#include <fstream>
using namespace std;
int main( int argc, char* args[] )
{
string inputName=args[1];
ifstream inputName(inputFileName);
////// here i will need to get the output string name some thing like
// string outputFileName=EDITED_INPUT_NAME+"XXX"+".txt";
ofstream outpuName(outputFileName);
while( std::getline( inputName, line ) )
{
if(line.find("IT IS HERE") != string::npos)
// how to make it take only the parts i need??????
outpuName << line << endl;
cout << line << endl;
}
inputName.close();
outpuName.close();
return 0;
}
Does this solve your problem:
#include <iostream>
#include <fstream>
#include <cstdlib>
using namespace std;
int main(int argc, char* args[]) {
ifstream inputFile(args[1]);
// Your first problem
string outputFileName(args[1]);
outputFileName.insert(outputFileName.find("."), "XXX");
cout << "Writing to " << outputFileName << endl;
// End of first problem
ofstream outputFile(outputFileName.c_str());
string line;
while (getline(inputFile, line)) {
if (line.find("IT IS HERE") != string::npos) {
// Your second problem
string::size_type time_start = line.find("time ") + 5;
string::size_type time_end = line.find(",", time_start);
cout << time_start << " " << time_end << endl;
string time = line.substr(time_start, time_end - time_start);
string::size_type len_start = line.find("len ") + 4;
string::size_type len_end = line.find(" ", len_start);
if (len_end != string::npos)
len_end += 4;
int len = atoi(line.substr(len_start, len_end - len_start).c_str());
// End of second problem
outputFile << time << " " << len << endl;
cout << time << " " << len << endl;
}
}
inputFile.close();
outputFile.close();
return 0;
}
Example input:
sdfghjk sdfghjk fghjkl
IT IS HERE time 12:04:56.186, len 000120
usjvowv weovnwoivjw wvijwvjwv
IT IS HERE time 12:05:42.937, len 000140
Example output:
12:04:56.186 120
12:05:42.937 140
The code could look nicer with std::regex and auto, but as this wasn't tagged with C++11, I held back.

fstream.read() Not Reading Anything At All

I'm trying to read the first line of an MP3 file (I edited this mp3 file to contain the text "I'm an MP3" right at the beginning of the file).
This is what I'm trying to do:
#include <iostream>
#include <string>
#include <fstream>
using namespace std;
int main()
{
fstream mp3;
mp3.open("05 Imagine.mp3", ios::binary | ios::in | ios::out);
/*mp3.seekg(0, ios::end);
int lof = mp3.tellg();
cout << "Length of file: " << lof << endl;
mp3.seekg(0, ios::beg);*/
//char ch;
//cout << mp3.get(ch) << endl;
char* somebuf;
while(mp3.read(somebuf, 10)) //Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}
return 0;
}
For some reason, that is crashing. At some point it didn't crash, but it didn't print anything when I did cout << somebuf. Can someone help me with this?
You never allocated anything for somebuf:
char* somebuf;
therefore, it doesn't point anywhere.
char* somebuf = new char[11];
somebuf[10] = '\0'; // Not sure if it is necessary to null-terminate...
while(mp3.read(somebuf, 10)) // Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}
// and free it later
delete [] somebuf;
Alternatively:
char somebuf[11];
somebuf[10] = '\0'; // Not sure if it is necessary to null-terminate...
while(mp3.read(somebuf, 10)) // Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}
Initialize the buffer:
char somebuf[10];
while(mp3.read(somebuf, 10)) //Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}