I'm having trouble getting boost::iostreams's zlib filter to ignore gzip headers ... It seems that setting zlib_param's default_noheader to true and then calling zlib_decompressor() produces the 'data_error' error (incorrect header check). This tells me zlib is still expecting to find headers.
Has anyone gotten boost::iostreams::zlib to decompress data without headers? I need to be able to read and decompress files/streams that do not have the two-byte header. Any assistance will be greatly appreciated.
Here's a modified version of the sample program provided by the boost::iostreams::zlib documentation:
#include <fstream>
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/zlib.hpp>
int main(int argc, char** argv)
{
using namespace std;
using namespace boost::iostreams;
ifstream ifs(argv[1]);
ofstream ofs("out");
boost::iostreams::filtering_istreambuf in;
zlib_params p(
zlib::default_compression,
zlib::deflated,
zlib::default_window_bits,
zlib::default_mem_level,
zlib::default_strategy,
true
);
try
{
in.push(zlib_decompressor(p));
in.push(ifs);
boost::iostreams::copy(in, ofs);
ofs.close();
ifs.close();
}
catch(zlib_error& e)
{
cout << "zlib_error num: " << e.error() << endl;
}
return 0;
}
I know my test data is not bad; I wrote a small program to call gzread() on the test file; it is successfully decompressed ... so I'm confused as to why this does not work.
Thanks in advance.
-Ice
I think what you want to do is something that's described here which is to adjust the window bits parameter.
e.g
zlib_params p;
p.window_bits = 16 + MAX_WBITS;
in.push(zlib_decompressor(p));
in.push(ifs);
MAX_WBITS is defined in zlib.h I think.
Much simple, try this:
FILE* fp = fopen("abc.gz", "w+");
int dupfd = dup( fileno( fp ) );
int zfp = gzdopen( dupfd, "ab" )
gzwrite( zfp, YOUR_DATA, YOUR_DATA_LEN );
gzclose( zfp );
fclose( fp );
Link with zlib and include zlib.h
You can use STDOUT instead of a file by using fileno( stdout )
Just use the boost::iostreams::gzip_decompressor for decompressing gzip files.
For example:
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/device/file.hpp>
#include <boost/iostreams/filtering_stream.hpp>
// ...
boost::iostreams::filtering_istream stream;
stream.push(boost::iostreams::gzip_decompressor());
ifstream file(filename, std::ios_base::in | std::ios_base::binary);
stream.push(file);
Related
I need to have a way to check if there is data to read on a file (fifo) in a non-blocking way.
I have tried using peek; but it is blocking, I have tried to get and then unget a character in order to check the file without altering the contents; but once again get is blocking...
The only non-blocking solution I have found is to use std::getline(file, line_str) and check if the string is empty; however this does not suit my needs as it alters the data on the file. (The data is a serialized object I will read once I detect there is something to read).
Note: I need this to be non-blocking: I have multiple file streams and need to check all of them regularly to see if there is an object to read/deserialize.
Here is a simple example of what I am trying to achieve:
Sender.cpp:
#include <fstream>
#include <iostream>
#include <string>
extern "C"{
#include <sys/stat.h> // S_IRUSR, S_IWUSR, mkfifo
}
#include <cerrno> // errno
int main(int, char** argv) {
std::string pipe = "foobar";
if(mkfifo(pipe.c_str(), S_IRUSR | S_IWUSR) < 0){
if (errno != EEXIST){
std::cerr << errno;
}
}
std::ofstream file{pipe.c_str()};
file.write("boop", 4); // Simulated object serialization
}
Reader.cpp:
#include <fstream>
#include <iostream>
#include <string>
extern "C"{
#include <sys/stat.h> // S_IRUSR, S_IWUSR, mkfifo
}
#include <cerrno> // errno
int main(int, char** argv) {
std::string pipe = "foobar";
if(mkfifo(pipe.c_str(), S_IRUSR | S_IWUSR) < 0){
if (errno != EEXIST){
std::cerr << errno;
}
}
std::ifstream file{pipe.c_str()};
// ...
/* Do check for data and read/deserialize if any data */
// This is in some sort of loop that goes over the different
// filestreams and checks to see if they have data to treat
}
Any help is really appreciated...
EDIT:
Following Zoso's answer I tried using the file size to determine if the file had been changed; however attempeting to get the size of a fifo named pipe is not possible : filesystem error: cannot get file size: Operation not supported [myFilePath]
I'm not sure if this would work for your particular use case but you could use the filesystem APIs. A simple example is
#include <iostream>
#include <fstream>
#include <filesystem>
namespace fs = std::filesystem;
int main()
{
while (true) {
auto path = fs::current_path().append("test");
std::cout <<"Press enter to know file size of "<<path.c_str() <<'\n';
char c= getchar();
try {
std::cout<<"Size of "<<path.c_str()<<"is "<<fs::file_size(path)<<'\n';
} catch(fs::filesystem_error& e) {
std::cout << e.what() << '\n';
}
}
}
As and when the file gets data, that can be kept track of based on the increasing size and the data to be processed can be tracked as and when that data is consumed.
I tried to unzip a zip file with boost zlib, but it doesn't work ,my boost version is 1.75 which is binary which is have been built,I tried the code below with VS 2013 CE
#include<iostream>
#include<fstream>
#include<algorithm>
#include<iterator>
#include<sstream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/zlib.hpp>
using namespace std;
int _tmain(int argc, _TCHAR* argv[])
{
ifstream src;
ofstream dst;
stringstream ss;
try
{
src.open("d:\\202012303333629003180526491.txt.zip", ios::in | ios::binary);
dst.open("to.zip", ios::out | ios::binary);
copy(istreambuf_iterator<char>(src),
istreambuf_iterator<char>(),
ostreambuf_iterator<char>(ss));
cout << ss.str() << endl;
std::stringstream decompressed;
boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
in.push(boost::iostreams::zlib_decompressor());
in.push(ss);
boost::iostreams::copy(in, decompressed);
string decompressestr = decompressed.str();
src.close();
dst.close();
return 0;
}
catch (const std::exception& error)
{
std::cerr << error.what() << std::endl;
return (EXIT_FAILURE);
}
}
the exception was zlib error: iostream stream error,any idea what wrong I have done? thank you so much
zlib does not process zip files. zlib can process zlib, gzip, and raw deflate streams.
I am currently working on project where I need to add some message at the end of a file and then I want to change its extension.
I know how to add the message at the end of the file; my code:
_ofstream myfile;
_myfile.open("check.txt", std::ios_base::app);
_myfile << "Thanks for your help.\n";
How can I change the file's extension?
Actualy, it is very simple:
#include <iostream>
#include <fstream>
#include <cstdio>
using namespace std;
int main(int argc, char* argv[])
{
ofstream fout("test.txt", ios_base::app);
fout << "My cool string";
fout.close();
rename("test.txt", "test.txt1");
return 0;
}
To determine a size of a binary file seems to always involve read the whole file into memory. How do I determine the size of a very large binary file which is known way bigger than the memory can take?
On most systems, there's stat() and
fstat() functions (not part of ANSI-C, but part of POSIX). For Linux, look at the man page.
EDIT: For Windows, the documentation is here.
EDIT: For a more portable version, use the Boost library:
#include <iostream>
#include <boost/filesystem.hpp>
using namespace boost::filesystem;
int main(int argc, char* argv[])
{
if (argc < 2)
{
std::cout << "Usage: tut1 path\n";
return 1;
}
std::cout << argv[1] << " " << file_size(argv[1]) << '\n';
return 0;
}
#include <cstdio>
FILE *fp = std::fopen("filename", "rb");
std::fseek(fp, 0, SEEK_END);
long filesize = std::ftell(fp);
std::fclose(fp);
Or, use ifstream:
#include <fstream>
std::ifstream fstrm("filename", ios_base::in | ios_base::binary);
fstrm.seekg(0, ios_base::end);
long filesize = fstrm.tellg();
This should work:
uintmax_t file_size(std::string path) {
return std::ifstream(path, std::ios::binary|std::ios::ate).tellg();
}
Basically I need to open and read a list of files I get from another command.
For each line of output of popen
open a file usen ifstream.open
it compiles and if I put the file name directly it works fine, but it doesn't do anything when using popen output. I've seen questions like this but none of this particular way of giving filenames.
here's the code:
#include <iostream>
#include <sqlite3.h>
#include <stdio.h>
#include <fstream>
using namespace std;
int main () {
ifstream singlefile;
FILE *filelist;
char filename[512];
string progline;
if(!(filelist = popen("find `pwd` -name \"*.js\"", "r"))){
return 1;
}
while( fgets(filename, sizeof(filename), filelist)!=NULL)
{
cout << filename;
singlefile.open(filename, ifstream::in);
while ( singlefile.good() )
{
getline (singlefile,progline);
cout << progline << endl;
}
singlefile.close();
}
pclose(filelist);
return 0;
}
next step would be not open each file inside the loop but to store the file list and then open each file.
Thanks
fgets keeps the trailing newline, resulting in a filename of a non-existing file. Also the stream state is only updated after reading. If I replace the while body with the following code, it works for me:
cout << filename;
size_t len = strlen(filename);
// chop off trailing newline
if (len > 1 && filename[len - 1] == '\n') filename[len - 1] = 0;
singlefile.open(filename, ifstream::in);
while ( getline(singlefile, progline) )
{
cout << progline << endl;
}
singlefile.close();
If you actually want to iterate through a list of files, I'd use Boost.Filesystem, which has a nice C++ interface, works for all filenames (even for those with newlines), and is platform-independent.
If this actually is only an example and your actual command is not find, there is still some room for simplification. Here is a suggestion that uses Boost.Iostreams to get rid of most of the C function calls (it would be great to have a device source reading from a process's standard output, but Boost.Iostreams lacks that):
#include <cstdio>
#include <iostream>
#include <fstream>
#include <stdexcept>
#include <string>
#include <stdio.h>
#include <boost/noncopyable.hpp>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
using namespace std;
namespace io = boost::iostreams;
class Popen: private boost::noncopyable {
public:
explicit Popen(const char* command):
m_stream(popen(command, "r")) {
if (!m_stream) throw runtime_error("popen failed");
}
~Popen() {
pclose(m_stream);
}
FILE* stream() const {
return m_stream;
}
private:
FILE* m_stream;
};
int main() {
Popen pipe_wrapper("find `pwd` -name \"*.cpp\"");
io::file_descriptor_source pipe_device(fileno(pipe_wrapper.stream()), io::never_close_handle);
io::stream<io::file_descriptor_source> pipe_stream(pipe_device, 0x1000, 0x1000);
string filename;
while (getline(pipe_stream, filename)) {
cout << filename << endl;
ifstream file_stream(filename.c_str(), ifstream::in);
string progline;
while (getline(file_stream, progline)) {
cout << progline << endl;
}
}
}