Reading in from CSV to template vector - c++

Question

I've been having difficulty all week trying to get one of my projects up and running. I'm required to read in from a 10,000 line CSV file from a meteorological database and output certain fields with a few demonstrations (Max blah blah).
I'm meant to design this using a self made template vector and aren't allowed access to the STL libraries.
As i'm just learning and this has been a few weeks in the making I think i've over complicated it for myself and now i'm stuck not knowing how to progress.
The main issue here is my confusion of how i'm going to not only read into a struct and parse the information to only read in what i need but then transform that data into the template vector.
Anyway, without further ado, here is my source code:
#include <iostream>
#include <fstream>
#include "Date.h"
#include "Time.h"
#include "Vector.h"
typedef struct {
Date d;
Time t;
float speed;
} WindLogType;
int main()
{
Vector<WindLogType> windlog;
std::string temp;
std::ifstream inputFile("MetData-31-3.csv");
int timeIndex, windSpeedIndex;
//18 Elements per line
//Need the elements at index 0 & 10
while(!inputFile.eof())
{
getline(inputFile, WindLogType.d,' ');
getline(inputFile, WindLogType.t,',');
for(int i = 0; i < 9; i++)
{
getline(inputFile, temp, ',');
}
getline(inputFile, WindLogType.speed);
windlog.push_back(WindLogType);
}
return 0;
}
Vector.h
#ifndef VECTOR_H
#define VECTOR_H
template <class elemType>
class Vector
{
public:
bool isEmpty() const;
bool isFull() const;
int getLength() const;
int getMaxSize() const;
void sort();
// T* WindLogType;
Vector(int nMaxSize = 64); //Default constructor, array size of 64.
Vector(const Vector&); //Copy constructor
~Vector(); //Destructor
void push_back(int);
int operator[](int);
int at(int i);
private:
int maxSize, length;
elemType* anArray;
void alloc_new();
};
template <class elemType>
bool Vector<elemType>::isEmpty() const
{
return (length == 0);
}
template <class elemType>
bool Vector<elemType>::isFull() const
{
return (length == maxSize);
}
template <class elemType>
int Vector<elemType>::getLength() const
{
return length;
}
template <class elemType>
int Vector<elemType>::getMaxSize() const
{
return maxSize;
}
//Constructor that takes the max size of vector
template <class elemType>
Vector<elemType>::Vector(int nMaxSize)
{
maxSize = nMaxSize;
length = 0;
anArray = new elemType[maxSize];
}
//Destructor
template <class elemType>
Vector<elemType>::~Vector()
{
delete[] anArray;
}
//Sort function
template <class elemType>
void Vector<elemType>::sort()
{
int i, j;
int min;
elemType temp;
for(i = 0; i < length; i++)
{
min = i;
for(j = i+1; j<length; ++j)
{
if(anArray[j] < anArray[min])
min = j;
}
temp = anArray[i];
anArray[i] = anArray[min];
anArray[min] = temp;
}
}
//Check if vector is full, if not add the item to the vector
template <class elemType>
void Vector<elemType>::push_back(int i)
{
if(length+1 > maxSize)
alloc_new();
anArray[length]=i;
length++;
}
template <class elemType>
int Vector<elemType>::operator[](int i)
{
return anArray[i];
}
//Return the vector at position 'i'
template <class elemType>
int Vector<elemType>::at(int i)
{
if(i < length)
return anArray[i];
throw 10;
}
//If the vector is about to get full, create a new temporary
//vector of double size and copy the contents across.
template <class elemType>
void Vector<elemType>::alloc_new()
{
maxSize = length*2;
int* tmp=new int[maxSize];
for(int i = 0; i < length; i++)
tmp[i]= anArray[i];
delete[] anArray;
anArray = tmp;
}
/**
//Copy Constructor, takes a reference to a vector and copies
//the values across to a new vector.
Vector::Vector(const Vector& v)
{
maxSize= v.maxSize;
length = v.length;
anArray = new int[maxSize];
for(int i=0; i<v.length; i++)
{
anArray[i] = v.anArray[i];
}
}**/
#endif
There are some things in the vector class that are completely unnecessary, they were just from a bit of practice.
Here is a sample of the CSV file:
WAST,DP,Dta,Dts,EV,QFE,QFF,QNH,RF,RH,S,SR,ST1,ST2,ST3,ST4,Sx,T
31/03/2016 9:00,14.6,175,17,0,1013.4,1016.9,1017,0,68.2,6,512,22.7,24.1,25.5,26.1,8,20.74
31/03/2016 9:10,14.6,194,22,0.1,1013.4,1016.9,1017,0,67.2,5,565,22.7,24.1,25.5,26.1,8,20.97
31/03/2016 9:20,14.8,198,30,0.1,1013.4,1016.9,1017,0,68.2,5,574,22.7,24,25.5,26.1,8,20.92
31/03/2016 9:30,15.1,215,27,0,1013.4,1016.8,1017,0,66.6,5,623,22.6,24,25.5,26.1,8,21.63
I require the elements in the WAST column and the S column, as WAST contains the date and S contains windspeed.
By no means do i want people to give me just the solution, I need to understand how i would read in and parse this data utilizing the struct & template vector.
There's no real "error" per se, I just lack the fundamental understanding of where to go next.
Any help would be greatly appreciated!
Thankyou

Answer 1

One easy and efficient way would be to have a vector per column, aka column-oriented storage. Column-oriented storage minimizes space requirements and allows you to easily apply linear algebra algorithms (including SIMD optimized), whithout having to pick individual struct members (as would be the case with row-oriented storage).
You can then parse each line using fscanf, each value into a separate variable. And then push_back the variables into the corresponding columns.
As fscanf does not parse dates, you would need to extract the date string into a char[64] and then parse that into struct tm which then can be converted to time_t.
The above assumes that you know the layout of the CSV and the types of the columns.
Pseudo-code:
vector<time_t> timestamps;
vector<double> wind_speeds;
for(;;) {
// Parse the CSV line into variables.
char date_str[64 + 1];
double wind_speed;
fscanf(file, "%64[^,], ..., %lf,...", date_str, ..., &wind_speed, ...);
time_t timestamp = parse_date(date_str);
// Store the parsed variables into the vectors.
timestamps.push_back(timestamp);
wind_speed.push_back(wind_speed);
}
double average_wind_speed = std::accumulate(wind_speeds.begin(), wind_speeds.end(), 0.) / wind_speeds.size();

Answer 2

.csv files are a representation of a table, delimited by "," (coma) to change cell and ";" (semi-column) for the end of the line.
EDIT: In the case of the ; does not work, the usual "\n" works. The below algorithm can easily be applied with the "\n"
In fact, there are no need to create a complicate program.. just if and while are enough. Here is an idea on how to proceed, I hope it can help you to understand a method, as it is what you are requesting.
1- Read every character (store it in a char) and add it to a string (the string += the char).
1.1- If the character is a ",", increase a counter and then you compare the string to the value desired (Here WAST).
1.1.2- If the string equales the desired value, save the counter in an integer (It allows knowing the position of the column you want.)
1.1.2- If not, continue until the end of the line ";" (which means in your case the desired column does not exist) or until you have a match (your string == "WAST")
NB: You can do it with different counters so that you know WAST position, S position etc.
Then:
Initialise a new counter
2- Compare the new counter to the saved value in 1.1.2.
2.1.1- If the values match, store the char contents in a string until you have a new coma.
2.1.2- If not, read every char until you find a new coma. Then increase your counter and restart from 2.
3- Continue to read the characters until you find a semi-column ";", and restart at step 2, until you finish to read the file.
To summarise, in this case the first step it to read every columns names, until finding the one you want or arriving at the end of the line. Store its position (noticed by the "," (comas)) thanks to a counter1.
Read every other line and storing the string in the desired column position (noticed by the "," (comas)) with counter1 compared to a new counter.
It may not be the most powerful algorithm by far, but it works and is easy to understand.
I tried to avoid writting it in C so that you can understand the steps without seeing the programmed solution. I hope it suits you.