I need to parse a file, get some data and write them in another file using RapidJson.
Right now I could retrieve values and put them in a document. My only problem is to insert that document in file:
FILE * pFile = fopen ("read.json" , "r");
FILE * wFile = fopen ("Test.json" , "w");
if (pFile != NULL)
{
rapidjson::FileStream is(pFile);
rapidjson::Document document;
document.ParseStream<0>(is);
string mMeshID = a.GetString();
//how to add that document to wfile
fclose (pFile);
}
Is there any way to write a RapidJson::Document in file ?
EDIT: the only way I found is:
// Convert JSON document to string
GenericStringBuffer< UTF8<> > buffer;
Writer<GenericStringBuffer< UTF8<> > > writer(buffer);
doc.Accept(writer);
const char* str = buffer.GetString();
fprintf(wFile, "%s", str);
fclose(wFile);
There is better documentation about FileWriteStream after this question was asked.
Using FileWriteStream instead of StringBuffer can reduce memory usage. FileWriteStream uses a fixed size of buffer (can be stored in the stack), while StringBuffer needs to store the whole JSON in (heap) memory. It becomes a big difference for big JSON.
You better use
fwrite (buffer.GetString(), buffer.GetSize(), 1, wFile);
it's safer (in case the buffer is not null-terminated) and faster (no strlen).
Other than that and the lack of error checking in your code, it's fine and should write the JSON to the file NP.
Related
i have a json file that has array of json objects. i am using rapidjson c++.
i want to append new object to json array that is inside this file
currently what i do is that i read the whole file in a json object using fileread stream and the i add new member (new json object) using AddMember inside array of that document that i read previously. and now i overwrite this new object inside the file and repeat the process for new objects.
this solution is not scalable. Can someone pointout anyother solution using rapidjson or raw filestream. help will be appreciated, i've been looking all over the internet but no luck.
is there something like append to file incrementally using json.
or any other scalable solution because my file size will get very large with time and thus reading the whole file everytime and then appending a new object and then rewrite the whole file will be a waste to memory and cpu time.
help me with this one please
This question is from some years ago, but this answer is still relevant.
The goal is to append a json object with rapidjson to a potentially already existing file which contains a json array. The following is satisfied:
No reading or parsing of the already existing file.
The new object is added directly to the already existing file without document merging.
Time does not depend on what has been added previously.
Here is the code with comments:
bool appendToFile(const std::string& filename, const rapidjson::Document& document)
{
using namespace rapidjson;
// create file if it doesn't exist
if (FILE* fp = fopen(filename.c_str(), "r"); !fp)
{
if (fp = fopen(filename.c_str(), "w"); !fp)
return false;
fputs("[]", fp);
fclose(fp);
}
// add the document to the file
if (FILE* fp = fopen(filename.c_str(), "rb+"); fp)
{
// check if first is [
std::fseek(fp, 0, SEEK_SET);
if (getc(fp) != '[')
{
std::fclose(fp);
return false;
}
// is array empty?
bool isEmpty = false;
if (getc(fp) == ']')
isEmpty = true;
// check if last is ]
std::fseek(fp, -1, SEEK_END);
if (getc(fp) != ']')
{
std::fclose(fp);
return false;
}
// replace ] by ,
fseek(fp, -1, SEEK_END);
if (!isEmpty)
fputc(',', fp);
// append the document
char writeBuffer[65536];
FileWriteStream os(fp, writeBuffer, sizeof(writeBuffer));
Writer<FileWriteStream> writer(os);
document.Accept(writer);
// close the array
std::fputc(']', fp);
fclose(fp);
return true;
}
return false;
}
I do not know if there is a readymade library for that, but if you decide to do it yourself is not impossible.
In few steps you could:
1) Load the all JSON in ram.
2) Take every request to append JSON and save it to a log file
3) Update JSON In RAM after written request to log
4) Every x seconds block changes, write the all JSON to disk and clear the log file
5) Unblock changes
6) Goto 2
Further optimizations could b:
1) Check for log file on start (after a crash) and apply log requests
2) When you write the JSON file do not rewrite completely but check if there were only appends at the end and write only the new part.
How does this sound ?
i have a big xml file( osm map data file to parse). the initial code to process is like this:
FILE* file = fopen(fileName.c_str(), "r");
size_t BUF_SIZE = 10 * 1024 * 1024;
char* buf = new char[BUF_SIZE];
string contents;
while (!feof(file))
{
int ret = fread(buf, BUF_SIZE, 1, file);
assert(ret != -1);
contents.append(buf);
}
size_t pos = 0;
while (true)
{
pos = contents.find('<', pos);
if (pos == string::npos) break;
// Case: found new node.
if (contents.substr(pos, 5) == "<node")
{
do something;
}
// Case: found new way.
else if (contents.substr(pos, 4) == "<way")
{
do something;
}
}
then here people tell me i should use memory mapping file to process those "big data file",
detail is here:
how to read to huge file into buffer,
i mean when it is a fixed size and not very large, may i could load one time into memory and append the content to a string object, then i could apply find(), method and other string method to extract the node content of a xml file. ( the code in the beginning of my question use this method and i test that will produce right result). Then if the file is very large, how apply those methods (not using xml library such as libxml)?
in one word, for small xml file, i could load the whole content to a std::string and apply the find(), substr() operation and got wanted information in the xml file. when the xml file is very large, when i need use the memory mapping file to cope with. then could append the whole content to a std::string, how could i parse the file not using exsit xml library?
hope i have clearly express my question.
If you're using std::string members to get the data you need, you're almost certainly not parsing the XML in the traditional sense of parsing XML. (That is, you're very probably not making any use of XML's hierarchical structure. Although you are extracting data from XML, "parsing XML" means something much more specific to most people.)
That said, the C equivalents of the std::string members you seem to be OK with, such as memcmp and the GNU extension memmem, just take pointers and lengths. Read their documentation and use them in place of their std:;string-member equivalents.
I am programming a face detection algorithm. In my code I'm parsing an XML file (in a recursion way, very inefficient takes my about 4 minutes to parse the whole XML file). I'd like to save the XML content using Iosteam binary to a file. I'm using a struct in C++ in order to use the raw data.
My goal is to parse the XML only if the raw data file is not exist.
The method work like this:
If the raw data file is not exist, parse the XML file and save the data to a file.
If the raw data file exist, read the raw data from the file
My problem is: whenever I open the raw data file and read from it. I get to read only small amount of byte from the file, I don't know how much, but in a certain point I receive only 0x00 data on my buffer.
My guess: I believe this has to do with the OS buffer, Which has a certain amount of buffer for read and write operations. I might be wrong about this. Though I'm not sure which one from the operations doesn't work well, it's either the write or read.
I was thinking to write / read the raw data char by char or line by line. In the other hand the file doesn't contain a text, which means that I can't read line by line or char by char.
The raw data size is
size_t datasize = DataSize(); == 196876 (Byte)
Which is retrieve in this function
/* Get the upper bound for predefined cascade size */
size_t CCacadeInterpreter::DataSize()
{
// this is an upper boundary for the whole hidden cascade size
size_t datasize = sizeof(HaarClassifierCascade) * TOTAL_CASCADE+
sizeof(HaarStageClassifier)*TOTAL_STAGES +
sizeof(HaarClassifier) * TOTAL_CLASSIFIERS +
sizeof(void*)*(TOTAL_CASCADE+TOTAL_STAGES+TOTAL_CLASSIFIERS);
return datasize;
}
The method work like this
BYTE * CCacadeInterpreter::Interpreter()
{
printf("|Phase - Load cascade from memory | CCacadeInterpreter::Interpreter | \n");
size_t datasize = DataSize();
// Create a memory structure
nextFreeSpace = pStartMemoryLocation = new BYTE [datasize];
memset(nextFreeSpace,0x00,datasize);
// Try to open a predefined cascade file on the current folder (instead of parsing the file again)
fstream stream;
stream.open(cascadeSavePath); // ...try existing file
if (stream.is_open())
{
stream.seekg(0,ios::beg);
stream.read((char*)pStartMemoryLocation , datasize); // **ream from file**
stream.close();
printf("|Load cascade from saved memory location | CCacadeInterpreter::Interpreter | \n");
printf("Completed\n\n");
stream.close();
return pStartMemoryLocation;
}
// Open the cascade file and parse the cascade xml file
std::fstream cascadeFile;
cascadeFile.open(cascadeDestanationPath, std::fstream::in); // open the file with read only attributes
if (!cascadeFile.is_open())
{
printf("Error: couldn't open cascade XML file\n");
delete pStartMemoryLocation;
return NULL;
}
// Read the file XML file , line by line
string buffer, str;
getline(cascadeFile,str);
while(cascadeFile)
{
buffer+=str;
getline(cascadeFile,str);
}
cascadeFile.close();
split(buffer, '<',m_tokens);
// Parsing begins
pHaarClassifierCascade = (HaarClassifierCascade*)nextFreeSpace;
nextFreeSpace += sizeof(HaarClassifierCascade);
pHaarClassifierCascade->count=0;
pHaarClassifierCascade->orig_window_size_height=20;
pHaarClassifierCascade->orig_window_size_width=20;
m_deptInTree=0;
m_numOfStage = 0;
m_numOfTotalClassifiers=0;
while (m_tokens.size())
{
Parsing();
}
// Save the current cascade into a file
SaveBlockToMemory(pStartMemoryLocation,datasize);
printf("\nCompleted\n\n");
return pStartMemoryLocation;
}
bool CCacadeInterpreter::SaveBlockToMemory(BYTE * pStartMemoryLocation,size_t dataSize)
{
fstream stream;
if (stream.is_open() )
stream.close();
stream.open(cascadeSavePath); // ...try existing file
if (!stream.is_open()) // ...else, create new file...
stream.open(cascadeSavePath, ios_base::in | ios_base::out | ios_base::trunc);
stream.seekg(0,ios::beg);
stream.write((char*)pStartMemoryLocation,dataSize);
stream.close();
return true;
}
Try using the Boost IOstreams library.
It has an easy to use wrrapers for file handling
I'm trying to find a way to replace all instances of a string token in a file with another string.
How can I do this in C++ with the win32 API?
In other languages this is an easy thing to do, but in C++ I am just lost.
EDIT: For some context, this is for a WiX custom action. So portability is not a main priority, just the most simplest solution.
If the file fits in memory – it's simpler. Call OpenFile() to open file, GetFileSize() to determine file size, allocate enough memory, call ReadFile() to read file, then CloseFile. Do replacement in memory (use strstr() or similar function), then again OpenFile(), WriteFile(), CloseFile().
If the file is large - create a temporary file and read the source file in chunks and write filtered text to the temporary file, then call DeleteFile() to delete the original file and MoveFile() to move the filtered file.
You could use the Boost.Regex Library which should resemble most of the functionality you find on other platforms.
It would work like this:
In this example you’ll find how you can replace a string matching a pattern.
#include <boost/regex.hpp>
#include <string>
int main()
{
boost::regex pattern ("b.lug",boost::regex_constants::icase|boost::regex_constants::perl);
std::string stringa ("Searching for bolug");
std::string replace ("BgLug");
std::string newString;
newString = boost::regex_replace (stringa, pattern, replace);
printf("The new string is: |%s|\n",newString.c_str());
return 0;
}
but you would have of course to add the file reading/writing.
As per sharptooth's solution, I knocked up some C code to do a find and replace on a file. I used stdio calls (strlen, strstr, strcpy and strcat) to do the string manipulation (rather than win32 calls), so your only dependancy is the C run time.
This is certainly not code I would use in a production system. I would use stuff from toolkit string manipulation libraries to make this much cleaner (and not so much with the fixed length buffers). I probably wouldn't use boost, I don't like the overhead. But I figured you might like an example with just the basics (N.B. This writes the altered buffers out to .temp).
#include <stdio.h>
#define BUF_LEN 2048
int findAndReplace (const char * file, const char * find, const char * replace)
{
int replaceCount = 0;
FILE * f = fopen (file, "rt");
if (strstr(replace, find))
return 0; // replacing blah with stuff_blah_stuff
unsigned int findLen = strlen (find);
char tempFile [BUF_LEN];
strcpy (tempFile, file);
strcat (tempFile, ".temp");
FILE * writeF = fopen (tempFile, "wt");
if (!f || !writeF)
return 0;
printf ("Processing %s - %s to %s\n", file, find, replace);
char lineBuf [BUF_LEN];
memset (lineBuf, 0, BUF_LEN);
char tempLineBuf [BUF_LEN];
memset (tempLineBuf, 0, BUF_LEN);
// read each line of the file
while (fgets (lineBuf, BUF_LEN, f))
{
// get the position of find in the line buffer
char * pos = strstr (lineBuf, find);
while (pos)
{
strncpy (tempLineBuf, lineBuf, pos - lineBuf);
strcat (tempLineBuf, replace);
strcat (tempLineBuf, pos + findLen);
replaceCount++;
// replace the current buf with the replaced buffer
strncpy (lineBuf, tempLineBuf, BUF_LEN);
memset (tempLineBuf, 0, BUF_LEN);
pos = strstr (lineBuf, find);
}
printf ("writing new line %s\n", lineBuf);
fputs (lineBuf, writeF);
}
fclose (f);
fclose (writeF);
return replaceCount;
}
int main ()
{
printf ("Made %d replacements\n", findAndReplace ("blah.txt", "marker", "testing_blah"));
}
Why do you have to use the Win32 API? It's easy enough using straight C++, I wouldn't confuse the issue by adding artificial constraints. Just open your input file, open an output file, and read a line from your input. While you haven't hit EOF in your input file, use a regex to look for your token. If you find it, then replace it with your text. Write the line to the output file. Read another line from the input. When you get EOF on the input, close it. Be sure any pending output gets flushed from the output buffer. Close the output file. Done.
I'm attempting to use TinyXML to read and save from memory, instead of only reading and saving files to disk.
It seems that the documnent's parse function can load a char *. But then I need to save the document to a char * when I'm done with it. Does anyone know about this?
Edit: The printing & streaming functions aren't what I'm looking for. They output in a viewable format, I need the actual xml content.
Edit: Printing is cool.
Here's some sample code I am using, adapted from the TiXMLPrinter documentation:
TiXmlDocument doc;
// populate document here ...
TiXmlPrinter printer;
printer.SetIndent( " " );
doc.Accept( &printer );
std::string xmltext = printer.CStr();
A simple and elegant solution in TinyXml for printing a TiXmlDocument to a std::string.
I have made this little example
// Create a TiXmlDocument
TiXmlDocument *pDoc =new TiXmlDocument("my_doc_name");
// Add some content to the document, you might fill in something else ;-)
TiXmlComment* comment = new TiXmlComment("hello world" );
pDoc->LinkEndChild( comment );
// Declare a printer
TiXmlPrinter printer;
// attach it to the document you want to convert in to a std::string
pDoc->Accept(&printer);
// Create a std::string and copy your document data in to the string
std::string str = printer.CStr();
I'm not familiar with TinyXML, but from the documentation it seems that by using operator << to a C++ stream (so you can use C++ string streams) or a TiXMLPrinter class you can get an STL string without using a file. See TinyXML documentation (look for the "Printing" section)
Don't quite get what you are saying; your question is not clear. I'm guessing you are wanting to load a file into memory so that you can pass it to the document parse function. In that case, the following code should work.
#include <stdio.h>
The following code reads a file into memory and stores it in a buffer
FILE* fd = fopen("filename.xml", "rb"); // Read-only mode
int fsize = fseek(fd, 0, SEEK_END); // Get file size
rewind(fd);
char* buffer = (char*)calloc(fsize + 1, sizeof(char));
fread(buffer, fsize, 1, fd);
fclose(fd);
The file is now in the variable "buffer" and can be passed to whatever function required you to provide a char* buffer of the file to it.