How to read CSV file and assign to Eigen Matrix? - c++

Question

I try to read a large cvs file into Eigen Matrix, below the code found having problem where it can not detect each line of \n in cvs file to create multiple rows in the matrix. (It read entire file with single row). Not sure what's wrong with the code. Can anyone suggest here?
Im also looking for a effective way to read csv file with 10k of rows and 1k of cols. Not so sure the code below will be the best effective way? Very appreciated with your comment.
#include <stdio.h>
#include <stdlib.h>
#include <iostream>
#include <fstream>
#include <istream> //DataFile.fail() function
#include <vector>
#include <set>
#include <string>
using namespace std;
#include <Eigen/Core>
#include <Eigen/Dense>
using namespace Eigen;
void readCSV(istream &input, vector< vector<string> > &output)
{
int a = 0;
int b = 0;
string csvLine;
// read every line from the stream
while( std::getline(input, csvLine) )
{
istringstream csvStream(csvLine);
vector<string> csvColumn;
MatrixXd mv;
string csvElement;
// read every element from the line that is seperated by commas
// and put it into the vector or strings
while( getline(csvStream, csvElement, ' ') )
{
csvColumn.push_back(csvElement);
//mv.push_back(csvElement);
b++;
}
output.push_back(csvColumn);
a++;
}
cout << "a : " << a << " b : " << b << endl; //a doen't detect '\n'
}
int main(int argc, char* argv[])
{
cout<< "ELM" << endl;
//Testing to load dataset from file.
fstream file("Sample3.csv", ios::in);
if(!file.is_open())
{
cout << "File not found!\n";
return 1;
}
MatrixXd m(3,1000);
// typedef to save typing for the following object
typedef vector< vector<string> > csvVector;
csvVector csvData;
readCSV(file, csvData);
// print out read data to prove reading worked
for(csvVector::iterator i = csvData.begin(); i != csvData.end(); ++i)
{
for(vector<string>::iterator j = i->begin(); j != i->end(); ++j)
{
m(i,j) = *j;
cout << *j << ", ";
}
cout << "\n";
}
}
I will also attach a sample cvs file. https://onedrive.live.com/redir?resid=F1507EBE7BF1C5B!117&authkey=!AMzCnpBqxUyF1BA&ithint=file%2ccsv

Answer 1

Here's something you can actually copy-paste
Writing your own "parser"
Pros: lightweight and customizable
Cons: customizable
#include <Eigen/Dense>
#include <vector>
#include <fstream>
using namespace Eigen;
template<typename M>
M load_csv (const std::string & path) {
std::ifstream indata;
indata.open(path);
std::string line;
std::vector<double> values;
uint rows = 0;
while (std::getline(indata, line)) {
std::stringstream lineStream(line);
std::string cell;
while (std::getline(lineStream, cell, ',')) {
values.push_back(std::stod(cell));
}
++rows;
}
return Map<const Matrix<typename M::Scalar, M::RowsAtCompileTime, M::ColsAtCompileTime, RowMajor>>(values.data(), rows, values.size()/rows);
}
Usage:
MatrixXd A = load_csv<MatrixXd>("C:/Users/.../A.csv");
Matrix3d B = load_csv<Matrix3d>("C:/Users/.../B.csv");
VectorXd v = load_csv<VectorXd>("C:/Users/.../v.csv");
Using the armadillo library's parser
Pros: supports other formats as well, not just csv
Cons: extra dependency
#include <armadillo>
template <typename M>
M load_csv_arma (const std::string & path) {
arma::mat X;
X.load(path, arma::csv_ascii);
return Eigen::Map<const M>(X.memptr(), X.n_rows, X.n_cols);
}

Answer 2

Read the CSV file into your vector < vector > as you please (e.g. Lucas's answer). Instead of the vector< vector<string> > construct, use a vector< vector<double> > or even better a simple vector< double >. To assign the vector of vectors to an Eigen matrix efficiently using vector< vector< double > >, use the following:
Eigen::MatrixXcd mat(rows, cols);
for(int i = 0; i < rows; i++)
mat.row(i) = Eigen::Map<Eigen::VectorXd> (csvData[i].data(), cols).cast<complex<double> >();
If you opted to use the vector< double > option, it becomes:
Eigen::MatrixXcd mat(rows, cols);
mat = Eigen::Map<Eigen::VectorXd> (csvData.data(), rows, cols).cast<complex<double> >().transpose();

Answer 3

This will read from a csv file correctly:
std::ifstream indata;
indata.open(filename);
std::string line;
while (getline(indata, line))
{
std::stringstream lineStream(line);
std::string cell;
while (std::getline(lineStream, cell, ','))
{
//Process cell
}
}
Edit: Also, since your csv is full of numbers, make sure to use std::stod or the equivalent conversion once you expect to treat them as such.