Storing data from an unknown number of files - c++

Question

I have used the following piece of code to read from multiple .dat files and parse them. This code uses 3D vectors to store data after the reading process. However, I would like that the data corresponding to each single file be independent from the others. The issue is that the number of files varies, and is unknown at compile time; hence, the number of vectors varies too. I would like to know if there is any solution for this.
vector<vector<vector<string>>> masterList;
for (int i = 0; i < files.size(); ++i) {
cout << "file name: " << files[i] << endl;
fin.open(files[i].c_str());
if (!fin.is_open()) {
// error occurs!!
// break or exit according to your needs
cout<<"error"<<endl;
}
std::vector<vector<string>> tokens;
int current_line = 0;
std::string line;
while (std::getline(fin, line))
{
cout<<"line number: "<<current_line<<endl;
// Create an empty vector for this line
tokens.push_back(vector<string>());
//copy line into is
std::istringstream is(line);
std::string token;
int n = 0;
//parsing
while (getline(is, token, DELIMITER))
{
tokens[current_line].push_back(token);
cout<<"token["<<current_line<<"]["<<n<<"] = " << token <<endl;
n++;
}
cout<<"\n";
current_line++;
}
fin.clear();
fin.close();
masterList.push_back(tokens);
}
So, the main issue I'm facing is: how to create a variable number of 2D vectors to store the data corresponding to each single file, when I don't know how many files there are at compile time.

Answer 1

Modify the list of files in the main to adapt the size of your "master data". If the length of file names is variable, then parse it first (or get it one way or another first), and then execute the parsing on the dat files. If the filenames are known at run time only, and asynchronously with that, then add a new element in the list each time you get a new filename (you can use events for that for example, take a look at https://github.com/Sheljohn/siglot).
Note that list elements are independent in memory, and that lists support deletion/insertion in constant time. That way, data corresponding to each file is independent from the other. If you want to retrieve the data specific to a file (knowing the filename), either iterate on the list to find the corresponding file (linear time) or trade the list for an unordered_map (amortized constant time).
#include <string>
#include <list>
#include <vector>
#include <iostream>
#include <sstream>
#include <fstream>
#include <iterator>
#include <algorithm>
using namespace std;
#define AVG_LINES_PER_FILE 100
/**
* [tokenize_string Tokenize input string 'words' and put elements in vector 'tokens'.]
* #param words [Space separated data-string.]
* #param tokens [Vector of strings.]
*/
void tokenize_string( string& words, vector<string>& tokens )
{
unsigned n = count( words.begin(), words.end(), ' ' );
tokens.reserve(n);
istringstream iss(words);
copy(
istream_iterator<string>(iss),
istream_iterator<string>(),
back_inserter<vector<string> >(tokens)
);
}
/**
* Contains data parsed from a single .dat file
*/
class DATFileData
{
public:
typedef vector<string> line_type;
typedef vector<line_type> data_type;
DATFileData( const char* fname = nullptr )
{
m_fdata.reserve(AVG_LINES_PER_FILE);
m_fdata.clear();
if ( fname ) parse_file(fname);
}
// Check if the object contains data
inline operator bool() const { return m_fdata.size(); }
// Parse file
bool parse_file( const char* fname )
{
string line;
m_fdata.clear();
ifstream fin( fname );
if ( fin.is_open() )
{
while ( fin.good() )
{
getline(fin,line);
m_fdata.push_back(line_type());
tokenize_string( line, m_fdata.back() );
}
fin.close();
m_fname = fname;
cout << "Parsed " << m_fdata.size() << " lines in file '" << fname << "'." << endl;
return true;
}
else
{
cerr << "Could not parse file '" << fname << "'!" << endl;
return false;
}
}
// Get data
inline unsigned size() const { return m_fdata.size(); }
inline const char* filename() const { return m_fname.empty() ? nullptr : m_fname.c_str(); }
inline const data_type& data() const { return m_fdata; }
inline const line_type& line( const unsigned& i ) const { return m_fdata.at(i); }
private:
string m_fname;
data_type m_fdata;
};
int main()
{
unsigned fcount = 0;
vector<string> files = {"some/file/path.dat","another/one.dat"};
list<DATFileData> data(files.size());
for ( DATFileData& d: data )
d.parse_file( files[fcount++].c_str() );
cout << endl << files.size() << " files parsed successfully." << endl;
}