RapidJSON c++ efficient and scalable way to append json object to file - c++

Question

i have a json file that has array of json objects. i am using rapidjson c++.
i want to append new object to json array that is inside this file
currently what i do is that i read the whole file in a json object using fileread stream and the i add new member (new json object) using AddMember inside array of that document that i read previously. and now i overwrite this new object inside the file and repeat the process for new objects.
this solution is not scalable. Can someone pointout anyother solution using rapidjson or raw filestream. help will be appreciated, i've been looking all over the internet but no luck.
is there something like append to file incrementally using json.
or any other scalable solution because my file size will get very large with time and thus reading the whole file everytime and then appending a new object and then rewrite the whole file will be a waste to memory and cpu time.
help me with this one please

Answer 1

This question is from some years ago, but this answer is still relevant.
The goal is to append a json object with rapidjson to a potentially already existing file which contains a json array. The following is satisfied:
No reading or parsing of the already existing file.
The new object is added directly to the already existing file without document merging.
Time does not depend on what has been added previously.
Here is the code with comments:
bool appendToFile(const std::string& filename, const rapidjson::Document& document)
{
using namespace rapidjson;
// create file if it doesn't exist
if (FILE* fp = fopen(filename.c_str(), "r"); !fp)
{
if (fp = fopen(filename.c_str(), "w"); !fp)
return false;
fputs("[]", fp);
fclose(fp);
}
// add the document to the file
if (FILE* fp = fopen(filename.c_str(), "rb+"); fp)
{
// check if first is [
std::fseek(fp, 0, SEEK_SET);
if (getc(fp) != '[')
{
std::fclose(fp);
return false;
}
// is array empty?
bool isEmpty = false;
if (getc(fp) == ']')
isEmpty = true;
// check if last is ]
std::fseek(fp, -1, SEEK_END);
if (getc(fp) != ']')
{
std::fclose(fp);
return false;
}
// replace ] by ,
fseek(fp, -1, SEEK_END);
if (!isEmpty)
fputc(',', fp);
// append the document
char writeBuffer[65536];
FileWriteStream os(fp, writeBuffer, sizeof(writeBuffer));
Writer<FileWriteStream> writer(os);
document.Accept(writer);
// close the array
std::fputc(']', fp);
fclose(fp);
return true;
}
return false;
}

Answer 2

I do not know if there is a readymade library for that, but if you decide to do it yourself is not impossible.
In few steps you could:
1) Load the all JSON in ram.
2) Take every request to append JSON and save it to a log file
3) Update JSON In RAM after written request to log
4) Every x seconds block changes, write the all JSON to disk and clear the log file
5) Unblock changes
6) Goto 2
Further optimizations could b:
1) Check for log file on start (after a crash) and apply log requests
2) When you write the JSON file do not rewrite completely but check if there were only appends at the end and write only the new part.
How does this sound ?