Sort .csv in multidimensional arrays - c++

Question

I'm trying to read specific values (i.e. values#coordinate XY) from a .csv file and struggle with a proper way to define multidimensional arrays within that .csv.
Here's an example of the form from my .csv file
NaN,NaN,1.23,2.34,9.99
1.23,NaN,2.34,3.45,NaN
NaN,NaN,1.23,2.34,9.99
1.23,NaN,2.34,3.45,NaN
1.23,NaN,2.34,3.45,NaN
NaN,NaN,1.23,2.34,9.99
1.23,NaN,2.34,3.45,NaN
NaN,NaN,1.23,2.34,9.99
1.23,NaN,2.34,3.45,NaN
1.23,NaN,2.34,3.45,NaN
NaN,NaN,1.23,2.34,9.99
1.23,NaN,2.34,3.45,NaN
NaN,NaN,1.23,2.34,9.99
1.23,NaN,2.34,3.45,NaN
1.23,NaN,2.34,3.45,NaN
...
Ok, in reality, this file becomes very large. You can interpret rows=latitudes and columns=longitudes and thus each block is an hourly measured coordinate map. The blocks usually have the size of row[361] column[720] and time periods can range up to 20 years (=24*365*20 blocks), just to give you an idea of the data size.
To structure this, I thought of scanning through the .csv and define each block as a vector t, which I can access by choosing the desired timestep t=0,1,2,3...
Then, within this block I would like to go to a specific line (i.e. latitude) and define it as a vector longitudeArray.
The outcome shall be a specified value from coordinate XY at time Z.
As you might guess, my coding experience is rather limited and this is why my actual question might be very simple: How can I arrange my vectors in order to be able to call any random value?
This is my code so far (sadly it is not much, cause I don't know how to continue...)
#include <fstream>
#include <iostream>
#include <iomanip>
#include <sstream>
#include <string>
#include <vector>
#include <algorithm>
using namespace std;
int main()
{
int longitude, latitude; //Coordinates used to specify desired value
int t; //Each array is associated to a specific time t=0,1,2,3... (corresponds to hourly measured data)
string value;
vector<string> t; //Vector of each block
vector<string> longitudeArray; //Line of array, i.e. latitude
ifstream file("swh.csv"); //Open file
if (!file.is_open()) //Check if file is opened, if not
print "File could..."
{
cout << "File could not open..." << endl;
return 1;
}
while (getline(file, latitude, latitude.empty())) //Scan .csv (vertically) and delimit every time a white line occurs
{
longitudeArray.clear();
stringstream ss(latitude);
while(getline(ss,value,',') //Breaks line into comma delimited fields //Specify line number (i.e. int latitude) here??
{
latitudeArray.push_back(value); //Adds each field to the 1D array //Horizontal vector, i.e. latitude
}
t.push_back(/*BLOCK*/) //Adds each block to a distinct vector t
}
cout << t(longitudeArray[5])[6] << endl; //Output: 5th element of longitudeArray in my 6th block
return 0;
}
If you have any hint, especially if there is a better way handling large .csv files, I'd be very grateful.
Ps: C++ is inevitable for this project...
Tüdelüü,
jtotheakob

Answer 1

As usual you should first think in terms of data and data usage. Here you have floating point values (that can be NaN) that should be accessible as a 3D thing along latitude, longitude and time.
If you can accept simple (integer) indexes, the standard ways in C++ would be raw arrays, std::array and std::vector. The rule of thumb then says: if the sizes are known at compile time arrays (or std::array if you want operation on global arrays) are fine, else go with vectors. And if unsure std:vector is your workhorse.
So you will probably end with a std::vector<std::vector<std::vector<double>>> data, that you would use as data[timeindex][latindex][longindex]. If everything is static you could use a double data[NTIMES][NLATS][NLONGS] that you would access more or less the same way. Beware if the array is large, most compilers will choke if you declare it inside a function (including main), but it could be a global inside one compilation unit (C-ish but still valid in C++).
So read the file line by line, feeding values in your container. If you use statically defined arrays just assign each new value in its position, if you use vectors, you can dynamically add new elements with push_back.
This is too far from your current code for me to show you more than trivial code.
The static (C-ish) version could contain:
#define NTIMES 24*365*20
#define NLATS 361
#define NLONGS 720
double data[NTIMES][NLATS][NLONGS];
...
int time, lat, long;
for(time=0; time<NTIMES; time++) {
for (lat=0; lat<NLATS; lat++) {
for (long=0; long<NLONGS; long++) {
std::cin >> data[time][lat][long];
for (;;) {
if (! std::cin) break;
char c = std::cin.peek();
if (std::isspace(c) || (c == ',')) std::cin.get();
else break;
}
if (! std::cin) break;
}
if (! std::cin) break;
}
if (! std::cin) break;
}
if (time != NTIMES) {
//Not enough values or read error
...
}
A more dynamic version using vectors could be:
int ntimes = 0;
const int nlats=361; // may be a non compile time values
const int nlongs=720; // dito
vector<vector<vector<double>>> data;
int lat, long;
for(;;) {
data.push_back(vector<vector<double>>);
for(lat=0; lat<nlats; lat++) {
data[ntimes].push_back(vector<double>(nlongs));
for(long=0; long<nlongs; long++) {
std::cin >> data[time][lat][long];
for (;;) {
if (! std::cin) break;
char c = std::cin.peek();
if (std::isspace(c) || (c == ',')) std::cin.get();
else break;
}
if (! std::cin) break;
}
if (! std::cin) break;
}
if (! std::cin) break;
if (lat!=nlats || long!=nlongs) {
//Not enough values or read error
...
}
ntimes += 1;
}
This code will successfully process NaN converting it the special not a number value, but it does not check the number of fields per line. To do that, read a line with std::getline and use a strstream to parse it.

Answer 2

Thanks, I tried to transfer both versions to my code, but I couldn't make it run.
Guess my poor coding skills aren't able to see what's obvious to everyone else. Can you name the additional libs I might require?
For std::isspace I do need #include <cctype>, anything else missing which is not mentioned in my code from above?
Can you also explain how if (std::isspace(c) || (c == ',')) std::cin.get(); works? From what I understand, it will check whether c (which is the input field?) is a whitespace, and if so, the right term becomes automatically "true" because of ||? What consequence results from that?
At last, if (! std::cin) break is used to stop the loop after we reached the specified array[time][lat][long]?
Anyhow, thanks for your response. I really appreciate it and I have now an idea how to define my loops.

Answer 3

Again thank you all for your ideas.
Unfortunately, I was not able to run the script... but my task changed slightly, thus the need to read very large arrays is not required anymore.
However, I've got an idea of how to structure such operations and most probably will transfer it to my new task.
You may close this topic now ;)
Cheers
jtothekaob