I've been staring at this to long...
I have made a program that logs weather data from different sensors. It handles the data in a double linkedlist and saves it to a binary file. To store different "compressions" of the data different files are used. e.g no compression, hours, days, etc.
The main program first loads the content of the correct file (defined by the WeatherSeries constructor) and adds all content from the file into the linkedlist. Then adds the new element and then saves it all. It adds to the list from oldest to newest and in the file it is also saved so the newest record is the last to be added to the file.
The error that occurs is that it seems that I lose a couple of hours of recorded data. I have observed that the data have existed. I.e. seen that there were data recorded between e.g. 9PM and 10PM and then in the morning after this data is gone.
The weird thing is the following:
Error only occurs intermittent and only for the barometric sensor that delivers values with 6 digits compared to the humidity and temperature sensors which delivers values with four digits.
It has only happened to the "no compression" and never for any of the other compressions. This means that the program that retrieves the data from the sensors works. Also it means that the function that adds data to the double linkedlist works.
Left is the functions that opens and saves the data to the files.
Can you please see if you can find any errors in my code?
void weatherSeries::saveSeries()
{
ostringstream s;
s << "WLData/" << mSensorNbr << "_" << mSensorType << "_" << mTimeBase << ".dat";
ofstream file(s.str().c_str(), ios::out | ios::trunc | ios::binary);
if (!file)
{
file.clear();
file.open(s.str().c_str(), ios::out | ios::trunc | ios::binary);
}
if(file.is_open())
{
for (current = tail; current != NULL; current = current->prev)
{
file.write((char*)¤t->time_stamp, sizeof(time_t));
file.write((char*)¤t->val_avg, sizeof(double));
file.write((char*)¤t->min, sizeof(double));
file.write((char*)¤t->max, sizeof(double));
file.write((char*)¤t->nbrOfValues, sizeof(unsigned long int));
}
}
else
{
cerr << "Unable to open for saving to " << mSensorNbr << "_" << mSensorType << "_" << mTimeBase << ".dat";
}
file.close();
}
void weatherSeries::openSeries()
{
deleteAll();
ostringstream s;
s << "WLData/" << mSensorNbr << "_" << mSensorType << "_" << mTimeBase << ".dat";
ifstream file(s.str().c_str(), ios::in | ios::binary);
if (!file)
{
file.clear();
file.open(s.str().c_str(), ios::in | ios::binary);
}
if(file.is_open())
{
time_t tmp_TS = 0;
double tmp_val_avg = 0;
double tmp_min = 0;
double tmp_max = 0;
unsigned long int tmp_nbrOfValues = 0;
while (file.read((char*)&tmp_TS, sizeof(time_t)))
{
file.read((char*)&tmp_val_avg, sizeof(double));
file.read((char*)&tmp_min, sizeof(double));
file.read((char*)&tmp_max, sizeof(double));
file.read((char*)&tmp_nbrOfValues, sizeof(unsigned long int));
addToSeries(tmp_TS, tmp_val_avg, tmp_min, tmp_max, tmp_nbrOfValues, true);
}
}
else
{
cerr << "Unable to open for opening from " << mSensorNbr << "_" << mSensorType << "_" << mTimeBase << ".dat";
}
file.close();
}
Note: deleteAll() clears the double linkedlist.
You were correct. The error was found in another part of the program. When I started logging different things in the code.
More or less different mechanism instantiate the program and it happened to happen at the same time causing the file to be manipulated by to instances at the same time.
Related
I'm trying to save binary data from a file metadata.mfs into the postgres database and then reading from the database, grab the saved binary data and load it into a file. For the purpose of illustrating my goal lets also call it metadata.mfs with the same name but in a different directory. When I run md5sum metadata.mfs on both files I am expecting to see the same hash. (Essentially I want whatever is saved into the database from file1 be exactly the same as what I extract from the database in file2)
Currently I am not able to achieve that.
Below is what I have so far:
string readFile2(const string &fileName)
{
ifstream ifs(fileName.c_str(), ios::in | ios::binary | ios::ate);
ifs.seekg(0, ios::end);
ifstream::pos_type fileSize = ifs.tellg();
ifs.seekg(0, ios::beg);
vector<char> bytes(fileSize);
ifs.read(bytes.data(), fileSize);
cout.write(bytes.data(),bytes.size());
cout << "\n";
cout << fileSize;
cout << "\n";
// return bytes.data();
return string(bytes.data(), fileSize);
}
int main() {
string content;
string test = "h";
char test1 = 'C';
try {
cout << "A1 \n";;
content = readFile2("/var/opt/lizardfs/lib/lizardfs/metadata.mfs");
pqxx::connection c("postgresql://mark#localhost:26257/metadata");
pqxx::nontransaction w(c);
w.exec("CREATE TABLE IF NOT EXISTS binary (id INT PRIMARY KEY, meta bytea)");
w.exec("INSERT INTO binary (id,meta) VALUES (18, '"+w.esc_raw(content)+"')");
pqxx::result r = w.exec("SELECT meta FROM binary WHERE id='18'");
std::ofstream outfile("metadata.mfs");
for (auto row: r) {
cout << row[0] << endl;
outfile << row[0] << endl;
}
outfile.close();
w.commit()
}
The problem is that cout.write(bytes.data(),bytes.size()); prints out exactly the same as what I would see in the linux terminal if I run cat metadata.mfs, but from cout << row[0] << endl I see everything in hex, ie, \x4c495a4d20322e39000021....
I suspect this is because I am using w.esc_raw() on the binary content before inserting into postgres. Does that mean I need to unescape, ie using w.unesc_raw() after extracting the binary data from the database? But how would I do that? I've been looking at the docs here: https://libpqxx.readthedocs.io/en/6.1.1/a00225.html
I'll try to be as clear as I can: Whenever I try to stream data to my file before my 'do' loop and my pointers reading and writing, my program goes nuts! It appears to be running an infinite loop.
fstream fileHandler; //Can also be done via constructor fstream fileHanlder("myData.txt", ios::out);
//fileHandler.open("myData.txt", ios::out);//Default is in AND out
fileHandler.open("test.txt", ios::in | ios::binary | ios::out);
if (fileHandler.is_open()) {
//fileHandler << "anything" <---HERE IS THE PROBLEM
cout << "The file has been opened and edited properly.";
fileHandler.seekg(0, ios::end);
streampos sizeOfFile = fileHandler.tellg();//tellg returns type streampos
fileHandler.seekg(0, ios::beg);
do{
string buffer;
fileHandler >> buffer;
cout << buffer << endl;
}while(!fileHandler.eof());
if ((fileHandler.rdstate()^ifstream::eofbit) == 0) {
fileHandler.clear();
cout << fileHandler.tellg() << endl;
}
fileHandler.close();
} else cout << "There was a problem opening the file!";
My file has nothing but a simple phrase.
EDIT: fixed the title according to new information
Thanks for any attention!
Removing the binary flag fixed it for some reason.
I am trying to build a "fileUpdater" which will copy an original file into multiple directories, where a file with the same name and extension was previously found.
bool update_files(const string inputPath, const vector<string> outputPaths)
{
ifstream src(inputPath);
if(!src.is_open())
{
cout << "Unable to open input file\n" << inputPath <<endl;
return false;
}
else
{
ofstream dst;
for(unsigned int i=0; i<= outputPaths.size()-1; i++)
{
dst.open(outputPaths[i]);
try
{
dst << src.rdbuf();
dst.close();
}
catch(int e)
{
cout << "Unable to replace file\n" <<endl;
cout << outputPaths[i] <<"\n"<< endl;
cout << "Error code: " <<e<<endl;
}
}
};
src.close();
return true;
}
Exactly after executing
dst.open(outputPaths[i]);
in the second iteration, the original file opened by
ifstream src(inputPath);
gets wiped and only an empty file is copied into the remaining directories.
I also tried
dst.clear();
dst.close();
and
src.clear();
src.seekg(0,ios::beg);
before entering the next iteration, but it made no difference.
UPDATE
After trying different files, I realised the behavior depends on the input file. Above behavior appeared for .m-files (MatLab).
After testing it with .txt files, all files were wiped.
The way you're copying the file, with dst << src.rdbuf();, will leave the current file position at the end of your input file. On the second iteration, that same read won't read anything (leaving an empty copy of the file) because you're already at the end of the input file.
The solution is to seek back to the beginning of the input file before every read, using seekg. You should call tellg before reading anything (right after opening the file), then seek to that position.
auto startpos = src.tellg();
ofstream dst;
// ...
src.seekg(startpos);
dst << src.rdbuf();
None of the proposed methods work.
Neither resetting the pointer, nor pulling ifstream into the loop, which would result in opening the input file (which is not supposed to change) unnecessarily often.
It is still unclear why dst.open(outputPaths[i]); is wiping the input file. Also the exact moment of the wipe depends on used types of files.
I implemented following workaround, effectively reading the input file into a string and closing it beforehand, in order to protect it from further read/write action.
bool update_files( const string inputPath, const vector<string> outputPaths)
{
const char * in = inputPath.c_str();
ifstream src(in);
if(!src.is_open())
{
cout << "Unable to open input file\n" << inputPath <<endl;
return false;
}
else
{
string buffer;
streamsize s=src.gcount();
src.seekg(0,ios::end);
buffer.reserve(src.tellg());
src.seekg(0,ios::beg);
buffer.assign((istreambuf_iterator<char>(src)), istreambuf_iterator<char>());
src.close();
for(unsigned int i=0; i<= outputPaths.size()-1; i++)
{
const char * out = outputPaths[i].c_str();
ofstream dst(out);
try
{
dst << buffer;
dst.close();
}
catch(int e)
{
cout << "Unable to replace file\n" <<endl;
cout << outputPaths[i] <<"\n"<< endl;
cout << "Error code: " <<e<<endl;
}
}
};
src.close();
return true;
}
I'm trying to write a program that replaces a specific number with an 'x' character. The task requires every number to be in its own line, but it seems like '\n' is causing the read/write pointers to behave out of this world. Here's a picture of the output.
My questions are:
why are the pointers behaving this way?
How far do I need to move the write pointer backwards to overwrite a line to make this work?
is there an easier workaround?
Here's my code:
void input(int n)
{
fstream file;
file.open("numbers.txt", ios::out);
while(n --> 0)
{
file << n;
file << '\n';
}
file.close();
}
void read()
{
fstream file;
string tmp;
file.open("numbers.txt", ios::in);
while(true)
{
getline(file,tmp);
if(file.eof())
break;
cout << tmp << endl;
cout << "tellg: " << file.tellg() << " tellp: " << file.tellp() << endl;
}
file.close();
}
void replace()
{
fstream file;
string tmp;
file.open("numbers.txt", ios::in | ios::out);
while(true)
{
file >> tmp;
if(tmp == "6")
{
//cout << file.tellg() << endl;
file.seekp(file.tellg() - tmp.length()-1);
file << "x";
}
if(file.eof())
break;
}
file.close();
}
int main()
{
input(10);
replace();
read();
return 0;
}
Since you open your file in text mode, you need to account for the potential that the underlying stream may use a line end sequence (\r\n) rather than just a \n. I guess, this is the primary problem. The easiest remedy is probaly to open the file in binary mode:
file.open("numbers.txt", std::ios_base::binary | std::ios_base::in | std::ios_base::out);
That said, since you switch from writing to reading without intervening seek, your code is undefined behavior, i.e., anything can happen. You should seek to the current location between writing and reading.
Personally, I'd refrain from rewriting files in-place. It generally gets unnecessary trick. If I were to rewrite files in place, I'd use seekg() to get the current position before a read, saving the position and restoring it prior to the write (I essentially never use the seek operations, i.e., I may have got the signatures wrong):
for (std::streampos pos = (in >> std::ws).tellg();
in >> tmp; pos = (in >> ws).tellg()) {
if (need_to_overwrite) {
in.seekp(pos);
// ...
in.seekg(0, std::ios_base::cur);
}
}
The use of in >> std::ws is to make sure that whitespace is skipped before storing the position.
Also note that your check for file.eof() is wrong: the last line is processed twice. When reading from a file the result shall be tested before using the read string, e.g.:
while (in >> tmp) {
// ...
}
I'm new to the forum, but not to this website. I've been searching for weeks on how to process a large data file quickly using C++ 11. I'm trying to have a function with a member that will capture the trace file name, open and process the data. The trace file contains 2 million lines of data, and each line is structured with a read/write operation and a hex address:
r abcdef123456
However, with a file having that much data, I need to read in and parse those 2 values quickly. My first attempt to read the file was the following:
void getTraceData(string filename)
{
ifstream inputfile;
string file_str;
vector<string> op, addr;
// Open input file
inputfile.open(filename.c_str());
cout << "Opening file for reading: " << filename << endl;
// Determine if file opened successfully
if(inputfile.fail())
{
cout << "Text file failed to open." << endl;
cout << "Please check file name and path." << endl;
exit(1);
}
// Retrieve and store address values and operations
if(inputfile.is_open())
{
cout << "Text file opened successfully." << endl;
while(inputfile >> file_str)
{
if((file_str == "r") || (file_str == "w"))
{
op.push_back(file_str);
}
else
{
addr.push_back(file_str);
}
}
}
inputfile.close();
cout << "File closed." << endl;
}
It worked, it ran, and read in the file. Unfortunately, it took the program 8 minutes to run and read the file. I modified the first program to the second program, to try and read the file in faster. It did, reading the file into a buffer in a fraction of a second versus 8 mins. using ifstream:
void getTraceData()
{
// Setup variables
char* fbuffer;
ifstream ifs("text.txt");
long int length;
clock_t start, end;
// Start timer + get file length
start = clock();
ifs.seekg(0, ifs.end);
length = ifs.tellg();
ifs.seekg(0, ifs.beg);
// Setup buffer to read & store file data
fbuffer = new char[length];
ifs.read(fbuffer, length);
ifs.close();
end = clock();
float diff((float)end - (float)start);
float seconds = diff / CLOCKS_PER_SEC;
cout << "Run time: " << seconds << " seconds" << endl;
delete[] fbuffer;
}
But when I added the parsing portion of the code, to get each line, and parsing the buffer contents line-by-line to store the two values in two separate variables, the program silently exits at the while-loop containing getline from the buffer:
void getTraceData(string filename)
{
// Setup variables
char* fbuffer;
ifstream ifs("text.txt");
long int length;
string op, addr, line;
clock_t start, end;
// Start timer + get file length
start = clock();
ifs.seekg(0, ifs.end);
length = ifs.tellg();
ifs.seekg(0, ifs.beg);
// Setup buffer to read & store file data
fbuffer = new char[length];
ifs.read(fbuffer, length);
ifs.close();
// Setup stream buffer
const int maxline = 20;
char* lbuffer;
stringstream ss;
// Parse buffer data line-by-line
while(ss.getline(lbuffer, length))
{
while(getline(ss, line))
{
ss >> op >> addr;
}
ss.ignore( strlen(lbuffer));
}
end = clock();
float diff((float)end - (float)start);
float seconds = diff / CLOCKS_PER_SEC;
cout << "Run time: " << seconds << " seconds" << endl;
delete[] fbuffer;
delete[] lbuffer;
}
I was wondering, once my file is read into a buffer, how do I retrieve it and store it into variables? For added value, my benchmark time is under 2 mins. to read and process the data file. But right now, I'm just focused on the input file, and not the rest of my program or the machine it runs on (the code is portable to other machines). The language is C++ 11 and the OS is a Linux computer. Sorry for the long posting.
Your stringstream ss is not associated to fbuffer at all. You are trying to getline from an empty stringstream, thus nothing happens. Try this:
string inputedString(fbuffer);
istringstream ss(fbuffer);
And before ss.getline(lbuffer, length), please allocate memory for lbuffer.
Actually you can directly read your file into a string to avoid the copy construction. Check this Reading directly from an std::istream into an std::string .
Last but not least, since your vector is quite large, you'd better reserve enough space for it before push_back the items one by one. When a vector reaches its capacity, attempt to push_back another item into it will result in reallocation and copy of all previous items in order to ensure continuous storage. Millions of items will make that happen quite a few times.