I'm using boost gzip example code here.
I am attempting to compress a simple string test and am expecting the compressed string H4sIAAAAAAAACitJLS4BAAx+f9gEAAAA as shown in this online compressor
static std::string compress(const std::string& data)
{
namespace bio = boost::iostreams;
std::stringstream compressed;
std::stringstream origin(data);
bio::filtering_streambuf<bio::input> out;
out.push(bio::gzip_compressor(bio::gzip_params(bio::gzip::best_compression)));
out.push(origin);
bio::copy(out, compressed);
return compressed.str();
}
int main(int argc, char* argv[]){
std::cout << compress("text") << std::endl;
// prints out garabage
return 0;
}
However when I print out the result of the conversion I get garbage values like +I-. ~
I know that it's a valid conversion because the decompression value returns the correct string. However I need the format of the string to be human readable i.e. H4sIAAAAAAAACitJLS4BAAx+f9gEAAAA.
How can I modify the code to output human readable text?
Thanks
Motivation
The garbage format is not compatible with my JSON library where I will send the compressed text through.
The example site completely fails to mention they also base64 encode the result:
base64 -d <<< 'H4sIAAAAAAAACitJLS4BAAx+f9gEAAAA' | gunzip -
Prints:
test
In short, you need to also do that:
Live On Coliru
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <iostream>
#include <sstream>
#include <boost/archive/iterators/binary_from_base64.hpp>
#include <boost/archive/iterators/base64_from_binary.hpp>
#include <boost/archive/iterators/transform_width.hpp>
std::string decode64(std::string const& val)
{
using namespace boost::archive::iterators;
return {
transform_width<binary_from_base64<std::string::const_iterator>, 8, 6>{
std::begin(val)},
{std::end(val)},
};
}
std::string encode64(std::string const& val)
{
using namespace boost::archive::iterators;
std::string r{
base64_from_binary<transform_width<std::string::const_iterator, 6, 8>>{
std::begin(val)},
{std::end(val)},
};
return r.append((3 - val.size() % 3) % 3, '=');
}
static std::string compress(const std::string& data)
{
namespace bio = boost::iostreams;
std::istringstream origin(data);
bio::filtering_istreambuf in;
in.push(
bio::gzip_compressor(bio::gzip_params(bio::gzip::best_compression)));
in.push(origin);
std::ostringstream compressed;
bio::copy(in, compressed);
return compressed.str();
}
static std::string decompress(const std::string& data)
{
namespace bio = boost::iostreams;
std::istringstream compressed(data);
bio::filtering_istreambuf in;
in.push(bio::gzip_decompressor());
in.push(compressed);
std::ostringstream origin;
bio::copy(in, origin);
return origin.str();
}
int main() {
auto msg = encode64(compress("test"));
std::cout << msg << std::endl;
std::cout << decompress(decode64(msg)) << std::endl;
}
Prints
H4sIAAAAAAAC/ytJLS4BAAx+f9gEAAAA
test
I am trying to wrap up a code that read gz files into a function, source code is taken from https://techoverflow.net/2013/11/03/c-iterating-lines-in-a-gz-file-using-boostiostreams/
My try
boost::iostreams::filtering_streambuf<boost::iostreams::input> func(std::string filename);
boost::iostreams::filtering_streambuf<boost::iostreams::input> func(std::string filename)
{
std::ifstream file(filename, std::ios_base::in | std::ios_base::binary);
boost::iostreams::filtering_streambuf<boost::iostreams::input> inbuf;
inbuf.push(boost::iostreams::gzip_decompressor());
inbuf.push(file);
return inbuf;
}
void mymainfunc(std::string filename)
{
//Convert streambuf to istream
std::istream ifstrm( func( filename));
std::string line;
while(std::getline(ifstrm, line)) {
std::cout << line << std::endl;
}
}
Code runs fine if not run through the function, i am doing something wrong in the return type I think.
Error: https://pastebin.com/kFpjYG0M
Streams are not copyable. In fact, this filteringstreambuf is not even movable.
So in this case, you will want to dynamically allocate and return by smart-pointer. However, even just returning the filtering streambuf will not work, because it would hold a reference to the ifstream. And that's a local.
So, maybe you need to package it up:
Live On Coliru
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <fstream>
namespace bio = boost::iostreams;
struct MySource {
using fisb = bio::filtering_istreambuf;
struct State {
State(std::string filename) : ifs(filename, std::ios::binary) {
buf.push(bio::gzip_decompressor());
buf.push(ifs);
}
fisb buf;
std::ifstream ifs;
std::istream is { &buf };
};
std::unique_ptr<State> _state;
operator std::istream&() const { return _state->is; }
};
MySource func(std::string filename) {
auto inbuf = std::make_unique<MySource::State>(filename);
return {std::move(inbuf)};
}
#include <iostream>
void mymainfunc(std::string filename)
{
auto source = func(filename);
std::istream& is = source;
std::string line;
while(std::getline(is, line)) {
std::cout << line << std::endl;
}
}
int main(){
mymainfunc("test.cpp.gz");
}
Alternative #1
You can simplify that:
Live On Coliru
struct MySource {
struct State {
State(std::string filename) : ifs(filename, std::ios::binary) {
is.push(bio::gzip_decompressor());
is.push(ifs);
}
std::ifstream ifs;
bio::filtering_istream is;
};
std::unique_ptr<State> _state;
operator std::istream&() const { return _state->is; }
};
Not dealing with the streambuffer separately makes it simpler.
Alternative #2
Not copying the whole thing has its own elegance:
Live On Coliru
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <fstream>
#include <iostream>
void mymainfunc(std::istream& is) {
std::string line;
while(std::getline(is, line)) {
std::cout << line << std::endl;
}
}
namespace bio = boost::iostreams;
int main(){
std::ifstream ifs("test.cpp.gz", std::ios::binary);
bio::filtering_istream is;
is.push(bio::gzip_decompressor());
is.push(ifs);
mymainfunc(is);
}
I wrote a simple code using cpprestsdk. I use a map (records) as a member of CommandHandler class, and manipulate it in a public method (has_record()).
it works before handler.open().wait() runs, but when I call it in a request, it crashes!
Here is my code:
#define BOOST_DATE_TIME_NO_LIB
#include <string>
#include <vector>
#include <cpprest/uri.h>
#include <cpprest/http_listener.h>
#include <cpprest/asyncrt_utils.h>
using namespace std;
using namespace web;
using namespace http;
using namespace utility;
using namespace http::experimental::listener;
class CommandHandler
{
public:
CommandHandler(utility::string_t url);
pplx::task<void> open() { return m_listener.open(); }
pplx::task<void> close() { return m_listener.close(); }
bool has_record();
private:
std::map< std::string, unsigned int > records;
void handle_get_or_post(http_request message);
http_listener m_listener;
};
bool CommandHandler::has_record()
{
return records.size() > 0 && records.find("1") != records.end();
}
CommandHandler::CommandHandler(utility::string_t url) : m_listener(url)
{
m_listener.support(methods::GET, std::bind(&CommandHandler::handle_get_or_post, this, std::placeholders::_1));
m_listener.support(methods::POST, std::bind(&CommandHandler::handle_get_or_post, this, std::placeholders::_1));
}
void CommandHandler::handle_get_or_post(http_request request)
{
if(this->has_record())
request.reply(status_codes::OK, 1);
else
request.reply(status_codes::OK, 0);
};
int main(int argc, char** argv)
{
try
{
utility::string_t address = U("http://127.0.0.1:9595");
uri_builder uri(address);
auto addr = uri.to_uri().to_string();
CommandHandler handler(addr);
if(handler.has_record())
std::cout<<"work!";
handler.open().wait();
ucout << utility::string_t(U("Listening for requests at: ")) << addr << std::endl;
ucout << U("Press ENTER key to quit...") << std::endl;
std::string line;
std::getline(std::cin, line);
handler.close().wait();
}
catch (std::exception& ex)
{
ucout << U("Exception: ") << ex.what() << std::endl;
ucout << U("Press ENTER key to quit...") << std::endl;
std::string line;
std::getline(std::cin, line);
}
return 0;
}
I found the problem but I don't understand why it throws. The problem is in this line of code:
request.reply(status_codes::OK, 0);
That 0 throws an exception
I'm trying to parse an XML file using the boost/propert_tree library. I can get the xml file correctly and everything, but when I look for childs, it doesn't find any.
I have an input.xml file:
<ax:hello someatribute:ax="dwadawfesfjsefs">
<something>523523</something>
<ax:whatever>
<ax:service_tree>
<ax:service>some</ax:service>
<ax:url>someulr</ax:url>
</ax:service_tree>
</ax:whatever>
</ax:hello>
The function where I try to parse the xml:
void parseXml(std::istream &stream)
{
using boost::property_tree::ptree;
ptree pt;
read_xml(stream, pt);
BOOST_FOREACH(ptree::value_type const &value, pt.get_child("ax:hello"))
{
std::cout << value.first;
}
}
And the main function:
int main()
{
std::ifstream stream("input.xml");
parseXml(stream);
return 0;
}
The error message I get is:
terminate called after throwing an instance of 'boost::exception_detail::clone_impl >'
what(): No such node (ax:hello)
Aborted (core dumped)`
As you can see, the ax:hello tag is properly opened and closed, so it should be able to find it despite the attributes, right?
Hope someone knows what's going on in here!
You're doing something else wrong/different:
Live On Coliru
#include <iostream>
#include <fstream>
#include <boost/property_tree/xml_parser.hpp>
#include <boost/foreach.hpp>
void parseXml(std::istream &stream)
{
using boost::property_tree::ptree;
ptree pt;
read_xml(stream, pt);
BOOST_FOREACH(ptree::value_type const &value, pt.get_child("ax:hello"))
{
std::cout << value.first << "\n";
}
}
int main()
{
std::istringstream stream(R"(
<ax:hello someatribute:ax="dwadawfesfjsefs">
<something>523523</something>
<ax:whatever>
<ax:service_tree>
<ax:service>some</ax:service>
<ax:url>someulr</ax:url>
</ax:service_tree>
</ax:whatever>
</ax:hello>
)");
parseXml(stream);
}
Prints
<xmlattr>
something
ax:whatever
Somewhat more elaborate dumping:
void dump(ptree const& pt, std::string const& indent = "") {
for (auto& node : pt) {
std::cout << indent << node.first;
auto value = boost::trim_copy(node.second.get_value(""));
if (!value.empty())
std::cout << ": '" << value << "'";
std::cout << "\n";
dump(node.second, indent + " ");
}
}
Prints Live On Coliru too
ax:hello
<xmlattr>
someatribute:ax: 'dwadawfesfjsefs'
something: '523523'
ax:whatever
ax:service_tree
ax:service: 'some'
ax:url: 'someulr'
okay, so I am banging my head on this for last couple of days but still I am unable to get it right. I have a std::list container and I want to serialize it into JSON string so that I can send it over network.
NOTE: I compile my code using following:
g++ -std=c++11 -o main main.cpp DBAccess11.cpp -lsqlite3 -lboost_serialization
I took help of this and this
Below is my DBAccess1.h file.
#ifndef DBAccess1_HH
#define DBAccess1_HH
#include <list> // I have deleted some header for sake of readability
#include <boost/serialization/list.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
using namespace std;
using boost::property_tree::ptree;
using boost::property_tree::read_json;
using boost::property_tree::write_json;
using boost::property_tree::basic_ptree;
//================================//
struct SiteCode
{
int siteID;
int siteCode;
};
inline ostream& operator<< (ostream &out, SiteCode &site)
{
out << "(" << site.siteID << "," << site.siteCode << ")";
return out;
}
//================================//
class sqliteDB {
list<SiteCode> Site_Code_list;
public:
list<SiteCode> GET_ALL_Site_Code();
void printList();
};
#endif**
Below is the DBAccess11.cpp file where all the functions are defined
#include <list> // I have deleted some header for sake of readability
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <boost/serialization/list.hpp>
#include "DBAccess1.h"
using boost::property_tree::ptree;
using boost::property_tree::read_json;
using boost::property_tree::write_json;
using boost::property_tree::basic_ptree;
list<SiteCode> sqliteDB::GET_ALL_Site_Code()
{
sqlite3 *db;
const char *sql;
sqlite3_stmt * stmt;
int rc = sqlite3_open("/path/to/database.db", &db);
sql = "SELECT * FROM SiteCode;";
rc = sqlite3_prepare_v2(db, sql, -1, &stmt, 0);
while(sqlite3_step(stmt)==SQLITE_ROW) {
int A = sqlite3_column_int(stmt, 0);
int B = sqlite3_column_int(stmt, 1);
SiteCode info;
info.siteID = A;
info.siteCode = B;
cout<<"Preparing to push data into List"<<endl;
Site_Code_list.push_back(info);
cout<<"Data was pushed successfully"<<endl;
ptree pt;
for (auto& entry: list<SiteCode> Site_Code_list) //<< ERROR LINE80
pt.put(entry.siteID, entry.siteCode);
std::ostringstream buf;
write_json (buf, pt, false);
cout<< buf.str() << endl;
}
sqlite3_finalize(stmt);
sqlite3_close(db);
return Site_Code_list;
}
//====================================================//
void sqliteDB::printList()
{
int s = Site_Code_list.size();
cout << "The size of List is :" << s << endl;
for( list<SiteCode> :: iterator it = Site_Code_list.begin(); it != Site_Code_list.end(); it++)
cout << *it << " ";
}
Below is main.cpp
#include <list> // I have deleted some header for sake of readability
#include <boost/serialization/list.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
using namespace std;
using boost::property_tree::ptree;
using boost::property_tree::read_json;
using boost::property_tree::write_json;
using boost::property_tree::basic_ptree;
int main()
{
sqliteDB object1;
object1.GET_ALL_Site_Code();
object1.printList();
cout << "\n\nAll the statement were executed properly\n\n";
return 0;
}
The error I get is as follows:
DBAccess11.cpp: In member function ‘std::list<SiteCode> sqliteDB::GET_ALL_Site_Code()’:
DBAccess11.cpp:80:38: error: expected primary-expression before ‘Site_Code_list’
for (auto& entry: list<SiteCode> Site_Code_list)
^
DBAccess11.cpp:80:38: error: expected ‘)’ before ‘Site_Code_list’
DBAccess11.cpp:80:52: error: expected ‘;’ before ‘)’ token
for (auto& entry: list<SiteCode> Site_Code_list)
^
My Questions:
(1)Is this the right way to convert std::list into JSON using boost ? if NO then how should it be done ?
(Note- I can only use boost and no other library )
(2) If my approach is right then what changes shall I make to correct it ?
Paths in your tree are always strings. The compiler will tell you this in the remainder of the message. Arguably the documentation is a more readable source:
Both key_type and data_type are configurable, but will usually be std::string here
The self_type & put(const path_type & path, const Type & value, Translator tr);
So the essence of the fix is
pt.put(std::to_string(entry.id), entry.code);
I got a little bit side-tracked cleaning up the code, so here goes:
Self Contained Sample
// FILE: some header
#include <ostream>
struct SiteCode {
int id;
int code;
SiteCode(int id, int code) : id(id), code(code)
{ }
friend inline std::ostream &operator<<(std::ostream &out, SiteCode const& site) {
return out << "(" << site.id << "," << site.code << ")";
}
};
#include <list> // I have deleted some header for sake of readability
// FILE: sqliteDB header
class sqliteDB {
using Records = std::list<SiteCode>;
Records _records;
public:
void load();
Records const& get() const { return _records; }
void printList() const;
void writeJson(std::ostream& os) const;
};
// FILE: some sqlpp.hpp utility header (inline implementations only)
#include <memory>
#include <sqlite3.h>
namespace sqlpp {
using database = std::shared_ptr<::sqlite3>;
void perror(int rc) {
if (rc != SQLITE_OK) throw std::runtime_error(::sqlite3_errstr(rc));
}
struct statement {
static statement prepare(database db, std::string const& sql) {
::sqlite3_stmt* stmt = nullptr;
perror(::sqlite3_prepare_v2(db.get(), sql.c_str(), -1, &stmt, 0));
return { handle(stmt, ::sqlite3_finalize), db };
}
int step() { return ::sqlite3_step(_stmt.get()); }
int column_int(int c) { return ::sqlite3_column_int(_stmt.get(), c); }
private:
using handle = std::shared_ptr<::sqlite3_stmt>;
database _db; // keeping it around for the lifetime of _stmt
handle _stmt;
statement(handle&& h, database& db) : _db(db), _stmt(std::move(h)) { }
};
database open(char const* path) {
::sqlite3* db = nullptr;
perror(::sqlite3_open(path, &db));
return database(db, ::sqlite3_close);
}
statement prepare(database db, std::string const& sql) {
return statement::prepare(db, sql);
}
}
// FILE: sqliteDB implementation file
#include <boost/property_tree/json_parser.hpp>
#include <boost/property_tree/ptree.hpp>
void sqliteDB::load() {
using namespace sqlpp;
auto stmt = prepare(open("/tmp/database.db"), "SELECT ID, CODE FROM SiteCode;");
while (stmt.step() == SQLITE_ROW)
_records.emplace_back(stmt.column_int(0), stmt.column_int(1));
}
void sqliteDB::writeJson(std::ostream& os) const {
using namespace boost::property_tree;
ptree pt;
for (auto &entry : _records)
pt.put(std::to_string(entry.id), entry.code);
write_json(os, pt, false);
}
// FILE: main program
template <typename List>
static void printList(List const& list) {
int s = list.size();
std::cout << "The number of Records is: " << s << "\n";
for (auto& r : list) std::cout << r << " ";
}
void dump(sqliteDB const& db) {
printList(db.get());
std::cout << "\n==============[ AS JSON ]===============\n";
db.writeJson(std::cout);
}
int main() {
sqliteDB db;
std::cout << "before loading: \n";
dump(db);
std::cout << "after loading: \n";
db.load();
dump(db);
}
Just compile as g++ -std=c++11 -g -Wall -Wextra -pedantic main.cpp -lsqlite3 and get:
sehe#desktop:/tmp$ sqlite3 database.db <<< "create table SiteCode (id int primary key, code int);"
sehe#desktop:/tmp$ for a in {1..10}; do echo "insert into SiteCode(ID,CODE) VALUES($a, $RANDOM);"; done | sqlite3 database.db
sehe#desktop:/tmp$ ./test
Output
before loading:
The number of Records is: 0
==============[ AS JSON ]===============
{}
after loading:
The number of Records is: 10
(1,5591) (2,31578) (3,30641) (4,4850) (5,1628) (6,5133) (7,8798) (8,20601) (9,21213) (10,18222)
==============[ AS JSON ]===============
{"1":"5591","2":"31578","3":"30641","4":"4850","5":"1628","6":"5133","7":"8798","8":"20601","9":"21213","10":"18222"}
ok so working on the solution provide above by sehe. i finally able to get the output right. I had to modify his solution as per my need. here the answer is as per below. All credit and thanks to sehe.
Below is my DBAccess1.h file:
#ifndef DBAccess1_HH
#define DBAccess1_HH
#include <sqlite3.h>
#include <list>
#include <boost/serialization/list.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
using namespace std;
using boost::property_tree::ptree;
using boost::property_tree::read_json;
using boost::property_tree::write_json;
using boost::property_tree::basic_ptree;
//=========================//
struct SiteCode
{
int siteID;
int siteCode;
friend inline std::ostream &operator<<(std::ostream &out, SiteCode const& site) {
return out << "(" << site.siteID << "," << site.siteCode << ")";
}
};
//=========================//
class sqliteDB {
using Records = std::list<SiteCode>;
Records Site_Code_list;
public:
list<SiteCode> GET_ALL_Site_Code();
Records const& get() const { return Site_Code_list; }
void writeJson(std::ostream& os) const;
void printList() const;
};
#endif
Below is DBAccess11.cpp
#include <ostream>
#include <boost/serialization/list.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include "DBAccess1.h"
using boost::property_tree::ptree;
using boost::property_tree::read_json;
using boost::property_tree::write_json;
using boost::property_tree::basic_ptree;
list<SiteCode> sqliteDB::GET_ALL_Site_Code()
{
sqlite3 *db;
const char *sql;
sqlite3_stmt * stmt;
int rc = sqlite3_open("/path.to/database.db", &db);
sql = "SELECT * FROM SiteCode;";
rc = sqlite3_prepare_v2(db, sql, -1, &stmt, 0);
while(sqlite3_step(stmt)==SQLITE_ROW) {
int A = sqlite3_column_int(stmt, 0);
int B = sqlite3_column_int(stmt, 1);
SiteCode info;
info.siteID = A;
info.siteCode = B;
cout<<"Preparing to push data into List"<<endl;
Site_Code_list.push_back(info);
cout<<"Data was pushed successfully"<<endl;
}
sqlite3_finalize(stmt);
sqlite3_close(db);
return Site_Code_list;
}
//==============================================================//
void sqliteDB::writeJson(std::ostream& os) const
{
using namespace boost::property_tree;
ptree pt;
for (auto &entry : Site_Code_list)
pt.put(std::to_string(entry.siteID), entry.siteCode);
write_json(os, pt, false);
}
//=========================================================//
Below is main.cpp
#include <ostream>
#include <boost/serialization/list.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include "DBAccess1.h"
using namespace std ;
using boost::property_tree::ptree;
using boost::property_tree::read_json;
using boost::property_tree::write_json;
using boost::property_tree::basic_ptree;
template <typename List>
static void printList(List const& list) {
int s = list.size();
std::cout << "The number of Records is: " << s << "\n";
for (auto& r : list) std::cout << r << " ";
}
void dump(sqliteDB const& db) {
printList(db.get());
std::cout << "\n==============[ AS JSON ]===============\n";
db.writeJson(std::cout);
}
int main()
{
sqliteDB object1;
std::cout << "before loading: \n";
dump(object1);
std::cout << "after loading: \n";
object1.GET_ALL_Site_Code();
dump(object1);
return 0;
}
compiled the above using:
g++ -std=c++11 -o main main.cpp DBAccess11.cpp -lsqlite3 -lboost_serialization
And the following is the output I get:
before loading:
The number of Records is: 0
==============[ AS JSON ]===============
{}
after loading:
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
Preparing to push data into List
Data was pushed successfully
The number of Records is: 10
(7,786) (8,78) (9,785) (10,998) (11,656) (13,23) (14,7) (15,74) (16,954) (17,752)
==============[ AS JSON ]===============
{"7":"786","8":"78","9":"785","10":"998","11":"656","13":"23","14":"7","15":"74","16":"954","17":"752"}