Is there a way to compress a JSON string in c++ , so that the overall size can be reduced ?
In my case mobile app which retreives XML create by CCUserDefault, then it converts that XML to JSON using rapidJson. Now I want to reduce its size or compress it using any cpp library.
Assuming you just want to minimise the size of the string (as opposed to general compression such as gzip), then a library such as rapidjson could be used.
There's an example in this unit test:
Roughly:
StringStream s("{ \"hello\" : \"world\" ");
StringBuffer buffer;
Writer<StringBuffer> writer(buffer);
Reader reader;
reader.Parse<0>(s, writer);
EXPECT_STREQ("{\"hello\":\"world\"}", buffer.GetString());
You could use zlib to compress the JSON string in memory and decompress it back. Perhaps using the ideas in here
Related
I'm trying to use a gzip c++ library to decompress some text that i compressed using this website that had a tool to do it, but when i try to decompress it in my project it says that its not compressed and fails to decompress. Am i just misunderstanding these compression formats because the names are the same or is this some other issue that i'm not aware of?
//'test message' compressed using the website
std::string test_string = R"(eJwrSS0uUchNLS5OTE8FAB8fBMY=)";
//returns false
bool is_compressed = gzip::is_compressed(test_string.data(), test_string.size());
//crashes
std::string decompressed = gzip::decompress(test_string.data(), test_string.size());
Website outputs a Base64 encoded string as ASCII, instead of the byte array. I need to decode the Base64 encoding before trying to decompress.
I'm trying to pass the content of a binary file from c++ to node using the node-gyp library. I have a process that creates a binary file using the .fit format and I need to pass the content of the file to js to process it. So, my first aproach was to extract the content of the file in a string and try to pass it to node like this.
char c;
std::string content="";
while (file.get(c)){
content+=c;
}
I'm using the following code to pass it to Node
v8::Local<v8::ArrayBuffer> ab = v8::ArrayBuffer::New(args.GetIsolate(), (void*)content.data(), content.size());
args.GetReturnValue().Set(ab);
In node a get an arrayBuffer but when I print the content to a file it is different to the one that show a c++ cout.
How can I pass the binary data succesfully?
Thanks.
Probably the best approach is to write your data to a binary disk file. Write to disk in C++; read from disk in NodeJS.
Very importantly, make sure you specify BINARY MODE.
For example:
myFile.open ("data2.bin", ios::out | ios::binary);
Do not use "strings" (at least not unless you want to uuencode). Use buffers. Here is a good example:
How to read binary files byte by byte in Node.js
var fs = require('fs');
fs.open('file.txt', 'r', function(status, fd) {
if (status) {
console.log(status.message);
return;
}
var buffer = new Buffer(100);
fs.read(fd, buffer, 0, 100, 0, function(err, num) {
...
});
});
You might also find these links helpful:
https://nodejs.org/api/buffer.html
<= Has good examples for specific Node APIs
http://blog.paracode.com/2013/04/24/parsing-binary-data-with-node-dot-js/
<= Good discussion of some of the issues you might face, including "endianness" and "interpreting numbers"
ADDENDUM:
The OP clarified that he's considering using C++ as a NodeJS Add-On (not a standalone C++ program.
Consequently, using buffers is definitely an option. Here is a good tutorial:
https://community.risingstack.com/using-buffers-node-js-c-plus-plus/
If you choose to go this route, I would DEFINITELY download the example code and play with it first, before implementing buffers in your own application.
It depends but for example using redis
Values can be strings (including binary data) of every kind, for
instance you can store a jpeg image inside a value. A value can't be
bigger than 512 MB.
If the file is bigger than 512MB, then you can store it in chunks.
But I wouldnt suggest since this is an in-memory data store
Its easy to implement in both c++ and node.js
I am using rapidjson to output some data for doing some statistic and plotting of a c++ programms algorithm like an internal runtime snapshots of the algorithm.
I output json like this:
string filename="output.json";
StringBuffer sb;
PrettyWriter<StringBuffer> writer(sb);
writer.StartArray();
for (std::vector<O_Class>::const_iterator netItr = O_Class_Array.begin(); netItr != O_Class_Array.end(); ++netItr)
netItr->Serialize(writer);
writer.EndArray();
ofstream out;
out.open(filename);
out << sb.GetString() ;
As files become quite big (~100MiB) i'd like to output minified json, but I didn't find a documented way of doing so.
With an external minifier I shrunk filesize from 100 to 18MB and like to have the same result as native in my application.
Any ideas?
Thanks for any suggestions!
Replace PrettyWriter for Writer.
And you could ZIP the content too. This will significantly reduce the size.
I am trying to get a function going to unzip a single text file compressed with .gz. It needs to uncompress the .gz file given its path and write the uncompressed text file given its destination. I am using C++ and what I have seen is that ZLIB does exactly what I need except I cannot find 1 single example anywhere on the net that shows it doing this. Can anyone show me an example or at least guide me in the right direction?
If you just want to inflate a file with raw deflated data (i.e. no archive) you can use something like this:
gzFile inFileZ = gzopen(fileName, "rb");
if (inFileZ == NULL) {
printf("Error: Failed to gzopen %s\n", filename);
exit(0);
}
unsigned char unzipBuffer[8192];
unsigned int unzippedBytes;
std::vector<unsigned char> unzippedData;
while (true) {
unzippedBytes = gzread(inFileZ, unzipBuffer, 8192);
if (unzippedBytes > 0) {
unzippedData.insert(unzippedData.end(), unzipBuffer, unzipBuffer + unzippedBytes);
} else {
break;
}
}
gzclose(inFileZ);
The unzippedData vector now holds your inflated data. There are probably more efficient ways to store the inflated data, especially if you know the uncompressed size in advance, but this approach works for me.
If you only want to save the inflated data to a file without any further processing you could skip the vector and just write the unzipBuffer contents to another file.
You can use the gzopen(), gzread(), and gzclose() functions of zlib, much like you would the corresponding stdio functions fopen(), etc. That will read the gzip file and decompress it. You can then use fopen(), fwrite(), etc. to write the uncompressed data back out.
You can use ZLibComplete to do this. There is a complete example in C++ on the front page of GZip decompression.
http://rudi-cilibrasi.github.io/zlibcomplete/
Ah, I'll assume http://zlib.net/zlib_how.html does what you want?
So we have some stringstream with somedata. With help of which cross platform library we can turn its contents into gziped format (you would save it into file with extention .tar.gz)
Lets get into code: so I have:
stringstream body;
body << std::ifstream( path.string().c_str(), ios::binary).rdbuf();
Which includes shall I add, which function should I call to encode stringstream and which finction to call to turn to put that encoded data into stringstream?
BTW: is there any boost library that can encode into gzip
boost.iostreams has optional built-in support for bzip2 and gzip compression/decompression.
It's not too clear what you're asking. If you mean that the
file (specified by path) is in gzipped format, the easiest
solution is probably to define a pipe input stream, and read it
from gunzip < file through that. If you want to unzip it
within your application, some sort of filtering input streambuf
would seem in order (see boost::iostream).
And I wouldn't use a temporary std::ifstream in any case: you
want to verify that the open succeeded before trying to read.