Boost gzip how to output compressed string as text - c++

I'm using boost gzip example code here.
I am attempting to compress a simple string test and am expecting the compressed string H4sIAAAAAAAACitJLS4BAAx+f9gEAAAA as shown in this online compressor
static std::string compress(const std::string& data)
{
namespace bio = boost::iostreams;
std::stringstream compressed;
std::stringstream origin(data);
bio::filtering_streambuf<bio::input> out;
out.push(bio::gzip_compressor(bio::gzip_params(bio::gzip::best_compression)));
out.push(origin);
bio::copy(out, compressed);
return compressed.str();
}
int main(int argc, char* argv[]){
std::cout << compress("text") << std::endl;
// prints out garabage
return 0;
}
However when I print out the result of the conversion I get garbage values like +I-. ~
I know that it's a valid conversion because the decompression value returns the correct string. However I need the format of the string to be human readable i.e. H4sIAAAAAAAACitJLS4BAAx+f9gEAAAA.
How can I modify the code to output human readable text?
Thanks
Motivation
The garbage format is not compatible with my JSON library where I will send the compressed text through.

The example site completely fails to mention they also base64 encode the result:
base64 -d <<< 'H4sIAAAAAAAACitJLS4BAAx+f9gEAAAA' | gunzip -
Prints:
test
In short, you need to also do that:
Live On Coliru
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <iostream>
#include <sstream>
#include <boost/archive/iterators/binary_from_base64.hpp>
#include <boost/archive/iterators/base64_from_binary.hpp>
#include <boost/archive/iterators/transform_width.hpp>
std::string decode64(std::string const& val)
{
using namespace boost::archive::iterators;
return {
transform_width<binary_from_base64<std::string::const_iterator>, 8, 6>{
std::begin(val)},
{std::end(val)},
};
}
std::string encode64(std::string const& val)
{
using namespace boost::archive::iterators;
std::string r{
base64_from_binary<transform_width<std::string::const_iterator, 6, 8>>{
std::begin(val)},
{std::end(val)},
};
return r.append((3 - val.size() % 3) % 3, '=');
}
static std::string compress(const std::string& data)
{
namespace bio = boost::iostreams;
std::istringstream origin(data);
bio::filtering_istreambuf in;
in.push(
bio::gzip_compressor(bio::gzip_params(bio::gzip::best_compression)));
in.push(origin);
std::ostringstream compressed;
bio::copy(in, compressed);
return compressed.str();
}
static std::string decompress(const std::string& data)
{
namespace bio = boost::iostreams;
std::istringstream compressed(data);
bio::filtering_istreambuf in;
in.push(bio::gzip_decompressor());
in.push(compressed);
std::ostringstream origin;
bio::copy(in, origin);
return origin.str();
}
int main() {
auto msg = encode64(compress("test"));
std::cout << msg << std::endl;
std::cout << decompress(decode64(msg)) << std::endl;
}
Prints
H4sIAAAAAAAC/ytJLS4BAAx+f9gEAAAA
test

Related

Return boost streambuf from function

I am trying to wrap up a code that read gz files into a function, source code is taken from https://techoverflow.net/2013/11/03/c-iterating-lines-in-a-gz-file-using-boostiostreams/
My try
boost::iostreams::filtering_streambuf<boost::iostreams::input> func(std::string filename);
boost::iostreams::filtering_streambuf<boost::iostreams::input> func(std::string filename)
{
std::ifstream file(filename, std::ios_base::in | std::ios_base::binary);
boost::iostreams::filtering_streambuf<boost::iostreams::input> inbuf;
inbuf.push(boost::iostreams::gzip_decompressor());
inbuf.push(file);
return inbuf;
}
void mymainfunc(std::string filename)
{
//Convert streambuf to istream
std::istream ifstrm( func( filename));
std::string line;
while(std::getline(ifstrm, line)) {
std::cout << line << std::endl;
}
}
Code runs fine if not run through the function, i am doing something wrong in the return type I think.
Error: https://pastebin.com/kFpjYG0M
Streams are not copyable. In fact, this filteringstreambuf is not even movable.
So in this case, you will want to dynamically allocate and return by smart-pointer. However, even just returning the filtering streambuf will not work, because it would hold a reference to the ifstream. And that's a local.
So, maybe you need to package it up:
Live On Coliru
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <fstream>
namespace bio = boost::iostreams;
struct MySource {
using fisb = bio::filtering_istreambuf;
struct State {
State(std::string filename) : ifs(filename, std::ios::binary) {
buf.push(bio::gzip_decompressor());
buf.push(ifs);
}
fisb buf;
std::ifstream ifs;
std::istream is { &buf };
};
std::unique_ptr<State> _state;
operator std::istream&() const { return _state->is; }
};
MySource func(std::string filename) {
auto inbuf = std::make_unique<MySource::State>(filename);
return {std::move(inbuf)};
}
#include <iostream>
void mymainfunc(std::string filename)
{
auto source = func(filename);
std::istream& is = source;
std::string line;
while(std::getline(is, line)) {
std::cout << line << std::endl;
}
}
int main(){
mymainfunc("test.cpp.gz");
}
Alternative #1
You can simplify that:
Live On Coliru
struct MySource {
struct State {
State(std::string filename) : ifs(filename, std::ios::binary) {
is.push(bio::gzip_decompressor());
is.push(ifs);
}
std::ifstream ifs;
bio::filtering_istream is;
};
std::unique_ptr<State> _state;
operator std::istream&() const { return _state->is; }
};
Not dealing with the streambuffer separately makes it simpler.
Alternative #2
Not copying the whole thing has its own elegance:
Live On Coliru
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <fstream>
#include <iostream>
void mymainfunc(std::istream& is) {
std::string line;
while(std::getline(is, line)) {
std::cout << line << std::endl;
}
}
namespace bio = boost::iostreams;
int main(){
std::ifstream ifs("test.cpp.gz", std::ios::binary);
bio::filtering_istream is;
is.push(bio::gzip_decompressor());
is.push(ifs);
mymainfunc(is);
}

Json-cpp - how to initialize from string and get string value?

My code below crashes(Debug Error! R6010 abort() has been called). Can you help me? I'd also would like to know how to initialize the json object from a string value.
Json::Value obj;
obj["test"] = 5;
obj["testsd"] = 655;
string c = obj.asString();
Hello it is pretty simple:
1 - You need a CPP JSON value object (Json::Value) to store your data
2 - Use a Json Reader (Json::Reader) to read a JSON String and parse into a JSON Object
3 - Do your Stuff :)
Here is a simple code to make those steps:
#include <stdio.h>
#include <jsoncpp/json/json.h>
#include <jsoncpp/json/reader.h>
#include <jsoncpp/json/writer.h>
#include <jsoncpp/json/value.h>
#include <string>
int main( int argc, const char* argv[] )
{
std::string strJson = "{\"mykey\" : \"myvalue\"}"; // need escape the quotes
Json::Value root;
Json::Reader reader;
bool parsingSuccessful = reader.parse( strJson.c_str(), root ); //parse process
if ( !parsingSuccessful )
{
std::cout << "Failed to parse"
<< reader.getFormattedErrorMessages();
return 0;
}
std::cout << root.get("mykey", "A Default Value if not exists" ).asString() << std::endl;
return 0;
}
To compile: g++ YourMainFile.cpp -o main -l jsoncpp
I hope it helps ;)
Json::Reader is deprecated. Use Json::CharReader and Json::CharReaderBuilder instead:
std::string strJson = R"({"foo": "bar"})";
Json::CharReaderBuilder builder;
Json::CharReader* reader = builder.newCharReader();
Json::Value json;
std::string errors;
bool parsingSuccessful = reader->parse(
strJson.c_str(),
strJson.c_str() + strJson.size(),
&json,
&errors
);
delete reader;
if (!parsingSuccessful) {
std::cout << "Failed to parse the JSON, errors:" << std::endl;
std::cout << errors << std::endl;
return;
}
std::cout << json.get("foo", "default value").asString() << std::endl;
Kudos to p-a-o-l-o for their answer here: Parsing JSON string with jsoncpp
You can avoid using Json::CharReader and Json::CharReaderBuilder by using stringstream instead.
#include <string>
#include <sstream>
#include "jsoncpp/json/json.h"
int main() {
std::string strJson = "{\"mykey\" : \"myvalue\"}";
Json::Value obj;
// read a JSON String
stringstream(strJson) >> obj;
// get string value
std::string value1 = obj["mykey"].asString();
// or to get a default value if it isn't set
std::string value2 = obj.get("mykey", "...").asString();
return 0;
}

Simple Zlib C++ String Compression and Decompression

I need a simple compression and decompression of a std::string in C++. I looked at this site and the code is for Character array. What I want to implement are the two functions:
std::string original = "This is to be compressed!!!!";
std::string compressed = string_compress(original);
std::cout << compressed << std::endl;
std::string decompressed = string_decompress(compressed);
std::cout << decompressed << std::endl;
I had tried the boost compression as:
std::string CompressData(const std::string &data)
{
std::stringstream compressed;
std::stringstream decompressed;
decompressed << data;
boost::iostreams::filtering_streambuf<boost::iostreams::input> out;
out.push(boost::iostreams::zlib_compressor());
out.push(decompressed);
boost::iostreams::copy(out, compressed);
return compressed.str();
}
std::string DecompressData(const std::string &data)
{
std::stringstream compressed;
std::stringstream decompressed;
compressed << data;
boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
in.push(boost::iostreams::zlib_decompressor());
in.push(compressed);
boost::iostreams::copy(in, decompressed);
return decompressed.str();
}
but the code sometimes gives Null characters in string ie \u0000. How do I handle if the compressed data contains these null characters. Is the return type string correct? How can I implement function string_compress and string_decompress using zlib?
You can do as #LawfulEvil suggested. Here is the code snippet that works :)
std::string original = "This is to be compressed!!!!";
std::string compressed_encoded = string_compress_encode(original);
std::cout << compressed_encoded << std::endl;
std::string decompressed_decoded = string_decompress_decode(compressed_encoded);
std::cout << decompressed_decoded << std::endl;
Using this as the base64 encode/decode library.
#include <sstream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/zlib.hpp>
#include <cpp-base64/base64.h>
std::string string_compress_encode(const std::string &data)
{
std::stringstream compressed;
std::stringstream original;
original << data;
boost::iostreams::filtering_streambuf<boost::iostreams::input> out;
out.push(boost::iostreams::zlib_compressor());
out.push(original);
boost::iostreams::copy(out, compressed);
/**need to encode here **/
std::string compressed_encoded = base64_encode(reinterpret_cast<const unsigned char*>(compressed.c_str()), compressed.length());
return compressed_encoded;
}
std::string string_decompress_decode(const std::string &data)
{
std::stringstream compressed_encoded;
std::stringstream decompressed;
compressed_encoded << data;
/** first decode then decompress **/
std::string compressed = base64_decode(compressed_encoded);
boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
in.push(boost::iostreams::zlib_decompressor());
in.push(compressed);
boost::iostreams::copy(in, decompressed);
return decompressed.str();
}
Compression makes use of all the values available for each byte, so it will appear as 'garbage' or 'weird' characters when attempting to view as ascii. Its expected. You'll need to encode the data for transmission / json packing to avoid nulls. I suggest base 64. Code to do that is available at the link below(which I didn't author so I won't copy here).
http://www.adp-gmbh.ch/cpp/common/base64.html
Binary data JSONCPP

boost json serialization and message_queue segfault

i'm making some test with boost interprocess and ptree structure, i have a segfault when i try to read the message sent(or when i try to parse it in json).
i'm using boost1.49 on debian linux.
i'm serializing it in json for later uses, and because i didn't find any good doc for the direct serialization of the boost property threes.
this is the code i'm using to test(the commed say where the segfault is):
recv.cc
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <boost/interprocess/ipc/message_queue.hpp>
#include <sstream>
struct test_data{
std::string action;
std::string name;
int faceID;
uint32_t Flags;
uint32_t freshness;
};
test_data recvData()
{
boost::interprocess::message_queue::remove("queue");
boost::property_tree::ptree pt;
test_data data;
std::istringstream buffer;
boost::interprocess::message_queue mq(boost::interprocess::open_or_create,"queue", 1, sizeof(buffer)
boost::interprocess::message_queue::size_type recvd_size;
unsigned int pri;
mq.receive(&buffer,sizeof(buffer),recvd_size,pri);
std::cout << buffer.str() << std::endl; //the segfault is there
boost::property_tree::read_json(buffer,pt);
data.action = pt.get<std::string>("action");
data.name = pt.get<std::string>("name");
data.faceID = pt.get<int>("face");
data.Flags = pt.get<uint32_t>("flags");
data.freshness = pt.get<uint32_t>("freshness");
boost::interprocess::message_queue::remove("queue");
return data;
}
int main()
{
test_data test;
test = recvData();
std::cout << test.action << test.name << test.faceID << test.Flags << test.freshness << std::endl;
}
sender.cc
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <boost/interprocess/ipc/message_queue.hpp>
#include <sstream>
struct test_data{
std::string action;
std::string name;
int faceID;
uint32_t Flags;
uint32_t freshness;
};
int sendData(test_data data)
{
boost::property_tree::ptree pt;
pt.put("action",data.action);
pt.put("name",data.name);
pt.put("face",data.faceID);
pt.put("flags",data.Flags);
pt.put("freshness",data.freshness);
std::ostringstream buffer;
boost::property_tree::write_json(buffer,pt,false);
boost::interprocess::message_queue mq(boost::interprocess::open_only,"chiappen")
std::cout << sizeof(buffer) << std::endl;
mq.send(&buffer,sizeof(buffer),0);
return 0;
}
int main ()
{
test_data prova;
prova.action = "registration";
prova.name = "prefix";
prova.Flags = 0;
prova.freshness = 52;
sendData(prova);
}
I know it's a bit late to an answer right now, but anyway..
You can't pass an istringstream as a buffer for receive. Boost message queues only handle raw bytes and don't handle std like objects.
To make it work, you must use a char array or any buffer previously reserved with malloc.
Ex:
char buffer [1024];
mq.receive(buffer, sizeof(buffer), recvd_size, pri);
For sending it's the same, you can only send raw bytes, so you can't use ostringstream.
Hope it helps.

JSONCPP Not reading files correctly

So I recently installed JSONCPP and for some reason it gives me errors when I try this code:
#include <json.h>
#include <iostream>
#include <fstream>
int main(){
bool alive = true;
while (alive){
Json::Value root; // will contains the root value after parsing.
Json::Reader reader;
std::string test = "testis.json";
bool parsingSuccessful = reader.parse( test, root, false );
if ( !parsingSuccessful )
{
// report to the user the failure and their locations in the document.
std::cout << reader.getFormatedErrorMessages()
<< "\n";
}
std::string encoding = root.get("encoding", "UTF-8" ).asString();
std::cout << encoding << "\n";
alive = false;
}
return 0;
}
And here is the file:
{
"encoding" : "lab"
}
It says that there is a syntax error on Line 1, Column 1, and that there must be a value, object or array. Anyone know how to fix this?
EDIT: Changed to current code, from pastebin
See the Json::Reader::parse documentation. For that overload, the string needs to be the actual document, not the filename.
You can use the istream overload with a ifstream instead.
std::ifstream test("testis.json", std::ifstream::binary);
EDIT: I got it work with:
#include "json/json.h"
#include <iostream>
#include <fstream>
int main(){
bool alive = true;
while (alive){
Json::Value root; // will contains the root value after parsing.
Json::Reader reader;
std::ifstream test("testis.json", std::ifstream::binary);
bool parsingSuccessful = reader.parse( test, root, false );
if ( !parsingSuccessful )
{
// report to the user the failure and their locations in the document.
std::cout << reader.getFormatedErrorMessages()
<< "\n";
}
std::string encoding = root.get("encoding", "UTF-8" ).asString();
std::cout << encoding << "\n";
alive = false;
}
return 0;
}
#include "json/json.h"
#include <iostream>
#include <fstream>
int main(){
Json::Value root; // will contain the root value after parsing.
std::ifstream stream("testis.json", std::ifstream::binary);
stream >> root;
std::string encoding = root.get("encoding", "UTF-8" ).asString();
std::cout << encoding << "\n";
return 0;
}
Or more generally:
#include "json/json.h"
#include <iostream>
#include <fstream>
int main(){
Json::Value root; // will contain the root value after parsing.
Json::CharReaderBuilder builder;
std::ifstream test("testis.json", std::ifstream::binary);
std::string errs;
bool ok = Json::parseFromStream(builder, test, &root, &errs);
if ( !ok )
{
// report to the user the failure and their locations in the document.
std::cout << errs << "\n";
}
std::string encoding = root.get("encoding", "UTF-8" ).asString();
std::cout << encoding << "\n";
return 0;
}
http://open-source-parsers.github.io/jsoncpp-docs/doxygen/namespace_json.html
json cannot contain newlines. Try this instead:
{"encoding": "lab"}
You may need to ensure the file is saved without a final newline.
EDIT: Maybe your parser tolerates newlines, but some don't. Something to try if the other answers don't work