Related
I have a program that scans a very large txt file (.pts file actually) that looks like this :
437288479
-6.9465 -20.49 -1.3345 70
-6.6835 -20.82 -1.3335 83
-7.3105 -20.179 -1.3325 77
-7.1005 -20.846 -1.3295 96
-7.3645 -20.759 -1.2585 79
...
The first line is the number of points contained in the file, and every other line corresponds to a {x,y,z,intensity} point in a 3D space. This file above is ~11 GB but I have more files to process that can be up to ~50 GB.
Here's the code I use to read this file :
#include <iostream>
#include <chrono>
#include <vector>
#include <algorithm>
#include <tuple>
#include <cmath>
// boost library
#include <boost/iostreams/device/mapped_file.hpp>
#include <boost/iostreams/stream.hpp>
struct point
{
double x;
double y;
double z;
};
void readMappedFile()
{
boost::iostreams::mapped_file_source mmap("my_big_file.pts");
boost::iostreams::stream<boost::iostreams::mapped_file_source> is(mmap, std::ios::binary);
std::string line;
// get rid of the first line
std::getline(is, line);
while (std::getline(is, line))
{
point p;
sscanf(line.c_str(),"%lf %lf %lf %*d", &(p.x), &(p.y), &(p.z));
if (p.z > minThreshold && p.z < maxThreshold)
{
// do something with p and store it in the vector of tuples
// O(n) complexity
}
}
}
int main ()
{
readMappedFile();
return 0;
}
For my 11 GB file, scanning all the lines and storing data in point p takes ~13 minutes to execute.
Is there a way to make it way faster ? Because each time I scan a point, I also have to do some stuff with it. Which will make my program to take several hours to execute in the end.
I started looking into using several cores but it seems it could be problematic if some points are linked together for some reason. If you have any advice on how you would proceed, I'll gladly hear about it.
Edit1 : I'm running the program on a laptop with a CPU containing 8 cores - 2.9GHz, ram is 16GB and I'm using an ssd. The program has to run on similar hardware for this purpose.
Edit2 : Here's the complete program so you can tell me what I've been doing wrong.
I localize each point in a sort of 2D grid called slab. Each cell will contain a certain amount of points and a z mean value.
#include <iostream>
#include <chrono>
#include <vector>
#include <algorithm>
#include <tuple>
#include <cmath>
// boost library
#include <boost/iostreams/device/mapped_file.hpp>
#include <boost/iostreams/stream.hpp>
struct point
{
double x;
double y;
double z;
};
/*
Compute Slab
*/
float slabBox[6] = {-25.,25.,-25.,25.,-1.,0.};
float dx = 0.1;
float dy = 0.1;
int slabSizeX = (slabBox[1] - slabBox[0]) / dx;
int slabSizeY = (slabBox[3] - slabBox[2]) / dy;
std::vector<std::tuple<double, double, double, int>> initSlab()
{
// initialize the slab vector according to the grid size
std::vector<std::tuple<double, double, double, int>> slabVector(slabSizeX * slabSizeY, {0., 0., 0., 0});
// fill the vector with {x,y} cells coordinates
for (int y = 0; y < slabSizeY; y++)
{
for (int x = 0; x < slabSizeX; x++)
{
slabVector[x + y * slabSizeX] = {x * dx + slabBox[0], y * dy + slabBox[2], 0., 0};
}
}
return slabVector;
}
std::vector<std::tuple<double, double, double, int>> addPoint2Slab(point p, std::vector<std::tuple<double, double, double, int>> slabVector)
{
// find the region {x,y} in the slab in which coord {p.x,p.y} is
int x = (int) floor((p.x - slabBox[0])/dx);
int y = (int) floor((p.y - slabBox[2])/dy);
// calculate the new z value
double z = (std::get<2>(slabVector[x + y * slabSizeX]) * std::get<3>(slabVector[x + y * slabSizeX]) + p.z) / (std::get<3>(slabVector[x + y * slabSizeX]) + 1);
// replace the older z
std::get<2>(slabVector[x + y * slabSizeX]) = z;
// add + 1 point in the cell
std::get<3>(slabVector[x + y * slabSizeX])++;
return slabVector;
}
/*
Parse the file
*/
void readMappedFile()
{
boost::iostreams::mapped_file_source mmap("my_big_file.pts");
boost::iostreams::stream<boost::iostreams::mapped_file_source> is(mmap, std::ios::binary);
std::string line;
std::getline(is, line);
auto slabVector = initSlab();
while (std::getline(is, line))
{
point p;
sscanf(line.c_str(),"%lf %lf %lf %*d", &(p.x), &(p.y), &(p.z));
if (p.z > slabBox[4] && p.z < slabBox[5])
{
slabVector = addPoint2Slab(p, slabVector);
}
}
}
int main ()
{
readMappedFile();
return 0;
}
If you use HDD to store your file just reading with 100Mb/s will spend ~2min and it is a good case. Try to read a block of the file and process it in another thread while the next block will be reading.
Also, you have something like:
std::vector<...> addPoint2Slab(point, std::vector<...> result)
{
...
return result;
}
slabVector = addPoint2Slab(point, slabVector);
I suppose it will bring unnecessary copying of the slabVector on every call (actually, a compiler might optimize it).
Try to check speed if you pass vector as follow:
std::vector<...> addPoint2Slab(point, std::vector<...> & result);
And call:
addPoint2Slab(point, slabVector);
And if it will get a speed bonus you can check how to forward results without the overhead.
Using memory maps is good. Using IOStreams isn't. Here's a complete take using Boost Spirit to do the parsing:
An Easy Starter
I'd suggest some cleanup around the typenames
using Record = std::tuple<double, double, double, int>;
std::vector<Record> initSlab()
{
// initialize the slab vector according to the grid size
std::vector<Record> slabVector(slabSizeX * slabSizeY, {0., 0., 0., 0});
// fill the vector with {x,y} cells coordinates
for (int y = 0; y < slabSizeY; y++) {
for (int x = 0; x < slabSizeX; x++) {
slabVector[x + y * slabSizeX] = {
x * dx + slabBox[0],
y * dy + slabBox[2],
0.,
0,
};
}
}
return slabVector;
}
You could just use a struct instead of the tuple, but that's an exercise for the reader
Don't Copy The SlabVector All The Time
You had addPoint2Slab taking the slabVector by value (copying) and returning the modified vector. Even if that's optimized to a couple of moves, it's still at least allocating a temporary copy each time addPoint2Slab is called. Instead, make it a mutating function as intended:
void addPoint2Slab(point const p, std::vector<Record>& slabVector)
{
// find the region {x,y} in the slab in which coord {p.x,p.y} is
int x = (int) floor((p.x - slabBox[0])/dx);
int y = (int) floor((p.y - slabBox[2])/dy);
auto& [ix, iy, iz, icount] = slabVector[x + y * slabSizeX];
iz = (iz * icount + p.z) / (icount + 1);
icount += 1;
}
Note also that the tuple handling has been greatly simplified with structured bindings. You can even see what the code is doing, which was nearly impossible before - let alone verify.
ReadMappedFile
auto readMappedFile(std::string fname)
{
auto slabVector = initSlab();
boost::iostreams::mapped_file_source mmap(fname);
auto handle = [&](auto& ctx) {
using boost::fusion::at_c;
point p{at_c<0>(_attr(ctx)), at_c<1>(_attr(ctx)), at_c<2>(_attr(ctx))};
//auto intensity = at_c<3>(_attr(ctx));
if (p.z > slabBox[4] && p.z < slabBox[5])
addPoint2Slab(p, slabVector);
};
namespace x3 = boost::spirit::x3;
static auto const line_ =
x3::float_ >> x3::float_ >> x3::float_ >> x3::int_;
auto first = mmap.data(), last = first + mmap.size();
try {
bool ok = x3::phrase_parse( //
first, last,
x3::expect[x3::uint_ >> x3::eol] //
>> line_[handle] % x3::eol //
// expect EOF here
>> *x3::eol >> x3::expect[x3::eoi], //
x3::blank);
// ok is true due to the expectation points
assert(ok);
} catch (x3::expectation_failure<char const*> const& ef) {
auto where = ef.where();
auto till = std::min(last, where + 32);
throw std::runtime_error("Expected " + ef.which() + " at #" +
std::to_string(where - mmap.data()) + " '" +
std::string(where, till) + "'...");
}
return slabVector;
}
Here we use Boost Spirit X3 to generate a parser that reads the lines and calls handle on each, much like you had before. A modicum of error handling has been added.
Let's Test It
Here's the test driver I used
#include <fmt/ranges.h>
#include <fstream>
#include <random>
#include <ranges>
using std::ranges::views::filter;
int main()
{
std::string const fname = "T032_OSE.pts";
#if 0 || defined(GENERATE)
using namespace std;
// generates a ~12Gib file
ofstream ofs(fname);
mt19937 prng{random_device{}()};
uniform_real_distribution<float> x(-25, 25), y(-25, +25), z(-1, 0);
uniform_int_distribution<> n(0, 100);
auto N = 437288479;
ofs << N << "\n";
while (N--)
ofs << x(prng) << " " << y(prng) << " " << z(prng) << " " << n(prng) << "\n";
#else
auto sv = readMappedFile(fname);
auto has_count = [](Record const& tup) { return get<3>(tup) > 0; };
fmt::print("slabVector:\n{}\n", fmt::join(sv | filter(has_count), "\n"));
#endif
}
Notice how you can use the conditionally compiled code to generate an input file (because I don't have your large file).
On this ~13GiB file (compressed copy online) it runs in 1m14s on my machine:
slabVector:
(-25, -25, -0.49556059843940164, 1807)
(-24.899999618530273, -25, -0.48971092838941654, 1682)
(-24.799999237060547, -25, -0.49731256076256386, 1731)
(-24.700000762939453, -25, -0.5006042266973916, 1725)
(-24.600000381469727, -25, -0.5000671732885645, 1784)
(-24.5, -25, -0.4940826157717386, 1748)
(-24.399999618530273, -25, -0.5045350563593015, 1720)
(-24.299999237060547, -25, -0.5088279537549671, 1812)
(-24.200000762939453, -25, -0.5065565364794715, 1749)
(-24.100000381469727, -25, -0.4933392542558793, 1743)
(-24, -25, -0.4947248105973453, 1808)
(-23.899999618530273, -25, -0.48640208470636714, 1696)
(-23.799999237060547, -25, -0.4994672590531847, 1711)
(-23.700000762939453, -25, -0.5033631130808075, 1782)
(-23.600000381469727, -25, -0.4995593140170436, 1760)
(-23.5, -25, -0.5009948279948179, 1737)
(-23.399999618530273, -25, -0.4995986820225158, 1732)
(-23.299999237060547, -25, -0.49833906199795897, 1764)
(-23.200000762939453, -25, -0.5013796942594327, 1728)
(-23.100000381469727, -25, -0.5072275248223541, 1700)
(-23, -25, -0.4949060352670081, 1749)
(-22.899999618530273, -25, -0.5026246990689665, 1740)
(-22.799999237060547, -25, -0.493411989775698, 1746)
// ... ~25k lines skipped...
(24.200000762939453, 24.900001525878906, -0.508382879738258, 1746)
(24.299999237060547, 24.900001525878906, -0.5064457874896565, 1740)
(24.400001525878906, 24.900001525878906, -0.4990733400392924, 1756)
(24.5, 24.900001525878906, -0.5063144518978036, 1732)
(24.60000228881836, 24.900001525878906, -0.49988387744959534, 1855)
(24.700000762939453, 24.900001525878906, -0.49970549673984693, 1719)
(24.799999237060547, 24.900001525878906, -0.48656442707683384, 1744)
(24.900001525878906, 24.900001525878906, -0.49267272688797675, 1705)
Remaining Notes
Beware of numerical error. You used float in some places, but with data sets this large it's very likely you will get noticeably large numeric errors in the running average calculation. Consider switching to [long] double or use a "professional" accumulator (many existing correlation frameworks or Boost Accumulator will do better).
Full Code
Live On Compiler Explorer
#include <algorithm>
#include <chrono>
#include <cmath>
#include <iostream>
#include <tuple>
#include <vector>
#include <fmt/ranges.h>
// boost library
#include <boost/iostreams/device/mapped_file.hpp>
#include <boost/iostreams/stream.hpp>
struct point { double x, y, z; };
/*
Compute Slab
*/
using Float = float; //
Float slabBox[6] = {-25.,25.,-25.,25.,-1.,0.};
Float dx = 0.1;
Float dy = 0.1;
int slabSizeX = (slabBox[1] - slabBox[0]) / dx;
int slabSizeY = (slabBox[3] - slabBox[2]) / dy;
using Record = std::tuple<double, double, double, int>;
std::vector<Record> initSlab()
{
// initialize the slab vector according to the grid size
std::vector<Record> slabVector(slabSizeX * slabSizeY, {0., 0., 0., 0});
// fill the vector with {x,y} cells coordinates
for (int y = 0; y < slabSizeY; y++) {
for (int x = 0; x < slabSizeX; x++) {
slabVector[x + y * slabSizeX] = {
x * dx + slabBox[0],
y * dy + slabBox[2],
0.,
0,
};
}
}
return slabVector;
}
void addPoint2Slab(point const p, std::vector<Record>& slabVector)
{
// find the region {x,y} in the slab in which coord {p.x,p.y} is
int x = (int) floor((p.x - slabBox[0])/dx);
int y = (int) floor((p.y - slabBox[2])/dy);
auto& [ix, iy, iz, icount] = slabVector[x + y * slabSizeX];
iz = (iz * icount + p.z) / (icount + 1);
icount += 1;
}
/* Parse the file */
#include <boost/spirit/home/x3.hpp>
auto readMappedFile(std::string fname)
{
auto slabVector = initSlab();
boost::iostreams::mapped_file_source mmap(fname);
auto handle = [&](auto& ctx) {
using boost::fusion::at_c;
point p{at_c<0>(_attr(ctx)), at_c<1>(_attr(ctx)), at_c<2>(_attr(ctx))};
//auto intensity = at_c<3>(_attr(ctx));
if (p.z > slabBox[4] && p.z < slabBox[5])
addPoint2Slab(p, slabVector);
};
namespace x3 = boost::spirit::x3;
static auto const line_ =
x3::double_ >> x3::double_ >> x3::double_ >> x3::int_;
auto first = mmap.data(), last = first + mmap.size();
try {
bool ok = x3::phrase_parse( //
first, last,
x3::expect[x3::uint_ >> x3::eol] //
>> line_[handle] % x3::eol //
// expect EOF here
>> *x3::eol >> x3::expect[x3::eoi], //
x3::blank);
// ok is true due to the expectation points
assert(ok);
} catch (x3::expectation_failure<char const*> const& ef) {
auto where = ef.where();
auto till = std::min(last, where + 32);
throw std::runtime_error("Expected " + ef.which() + " at #" +
std::to_string(where - mmap.data()) + " '" +
std::string(where, till) + "'...");
}
return slabVector;
}
#include <fmt/ranges.h>
#include <fstream>
#include <random>
#include <ranges>
using std::ranges::views::filter;
int main()
{
std::string const fname = "T032_OSE.pts";
#if 0 || defined(GENERATE)
using namespace std;
// generates a ~12Gib file
ofstream ofs(fname);
mt19937 prng{random_device{}()};
uniform_real_distribution<Float> x(-25, 25), y(-25, +25), z(-1, 0);
uniform_int_distribution<> n(0, 100);
auto N = 437288479;
ofs << N << "\n";
while (N--)
ofs << x(prng) << " " << y(prng) << " " << z(prng) << " " << n(prng) << "\n";
#else
auto sv = readMappedFile(fname);
auto has_count = [](Record const& tup) { return get<3>(tup) > 0; };
fmt::print("slabVector:\n{}\n", fmt::join(sv | filter(has_count), "\n"));
#endif
}
Get rid of std::getline. iostreams are pretty slow compared to direct "inmemory" processing of strings. Also do not use sscanf.
Allocate a large chunk of memory, i.e. 128MB or more. Read all of it from file in one call. Then parse this chunk until you reach the end.
Sort of like this:
std::vector<char> huge_chunk(128*1024*1024);
ifstream in("my_file");
do {
in.read(huge_chunk.data(), huge_chunk.size());
parse(huge_chunk.data, in.gcount());
} while (in.good());
you get the idea.
Parse the chunk with strtof, find and the like.
Parsing the chunk will leave a few characters at the end of the chunk which do not form a complete line. You need to store them temporarily and resume parsing the next chunk from there.
Generally speaking: The fewer calls to ifstream, the better. And using "lower API" functions such as strtof, strtoul etc... is usually faster than sscanf, format etc...
This usually does not matter for small files <1MB, but can make a huge difference with very large files.
Also: Use a profiler to find out exactly where your program is waiting. Intels VTune profiler is free, afaik. It is part of the OneAPI Toolkit and is one of the best tools I know.
I am trying to write an Octave C++ .oct function that uses the linasm-1.13 library but I cannot seem to get even basic loading of tzdata from /usr/share/zoneinfo/ to work. My simple test function so far is
#include <octave/oct.h>
#include <Time.h> // the linasm-1.13 library
DEFUN_DLD ( tz, args, nargout,
"-*- texinfo -*-\n\
#deftypefn {Function File} {} tz (#var{YYYYMMDDHHMMSS})\n\
\n\
#end deftypefn" )
{
octave_value_list retval_list ;
unsigned int tz ;
const char *ny_time = "/usr/share/zoneinfo/America/New_York" ;
tz = Time::LoadTimeZone( ny_time ) ;
return retval_list ;
which, on compiling with mkoctfile, gives this error
>> mkoctfile tz.cc
tz.cc: In function ‘octave_value_list Ftz(const octave_value_list&, int)’:
tz.cc:24:34: error: cannot call member function ‘unsigned int Time::LoadTimeZone(const char*)’ without object
tz = Time::LoadTimeZone( ny_time ) ;
^
warning: mkoctfile: building exited with failure status
My understanding of this is that ny_time is not an object that is recognised, but I have tried casting ny_time as a string literal as detailed in this accepted SO answer.
I am doing things this way because the input for LoadTimeZone according to the linasm page should be a "path to tzfile, which describes required time zone." Where am I going wrong?
I think you have to #include "source.cc" files also, not just the #include "header.h" files. In your case, I guess you should add: #include "Time.cc" or something like that. I don't know why but this worked for me when working with Rafat's Hussain wavemin library, but I had only 4 files, it must be incredibly tedious with lots of files.
This is what I did (it's a modified version of the test code provided by Rafat with his library).
#include "wavemin.h"
#include "waveaux.h"
#include "wavemin.cc"
#include "waveaux.cc"
#include <octave/oct.h>
double ensayo();
double absmax(double *array, int N);
DEFUN_DLD(helloctave2, argv, , "Usage: hello()"){
wave_object obj;
wt_object wt;
double *inp, *out, *diff;
int N, i, J;
char *name = "db4";
obj = wave_init(name);// Initialize the wavelet
N = 14; //Length of Signal
inp = (double*)malloc(sizeof(double)* N); //Input signal
out = (double*)malloc(sizeof(double)* N);
diff = (double*)malloc(sizeof(double)* N);
//wmean = mean(temp, N);
for (i = 0; i < N; ++i) {
inp[i] = i;
}
J = 1; //Decomposition Levels
wt = wt_init(obj, "dwt", N, J);// Initialize the wavelet transform object
setDWTExtension(wt, "sym");// Options are "per" and "sym". Symmetric is the default option
setWTConv(wt, "direct");
dwt(wt, inp);// Perform DWT
//DWT output can be accessed using wt->output vector. Use wt_summary to find out how to extract appx and detail coefficients
for (i = 0; i < wt->outlength; ++i) {
octave_stdout << wt->output[i];
octave_stdout << "\n";
}
idwt(wt, out);// Perform IDWT (if needed)
// Test Reconstruction
for (i = 0; i < wt->siglength; ++i) {
diff[i] = out[i] - inp[i];
}
octave_stdout << absmax(diff, wt->siglength);
octave_stdout << "\n";
octave_value_list retval;
return retval;
}
double
absmax(double *array, int N) {
double max;
int i;
max = 0.0;
for (i = 0; i < N; ++i) {
if (fabs(array[i]) >= max) {
max = fabs(array[i]);
}
}
return max;
}
I have following cpp code
#include <stdio.h> /*utiliser printf*/
#include <fcntl.h>
#include <math.h> /*utiliser pour les formules de math*/
#include <malloc.h>
#include <iostream.h>
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
/* Le type nombre complexe */
typedef struct {
double Preel;
double Pimaginaire;
} COMPLEXE;
#define ALLOC_ERROR 1
void indienne(double *MatE, int tE, int nE, double *SortieExp, double *Tempbis)
{
double *TempE=NULL, *SortieE=NULL;
int *dec=NULL;
int i, tampon, kE;
kE=(int)(log(nE)/log(2));
if(nE==8)
kE=3;
/* ALLOCATION DES MATRICES*/
if (!(TempE = (double *)calloc(tE * tE, sizeof(double))))
exit(ALLOC_ERROR);
printf("check1 te=%d, nE=%d",tE,nE);
if (!(dec = (int *)realloc(kE , sizeof(int))))
exit(ALLOC_ERROR);
if (!(SortieE = (double *)calloc(tE * tE, sizeof(double))))
exit(ALLOC_ERROR);
printf("check2 te=%d",tE);
memcpy(TempE,MatE,tE * tE * sizeof(double));
for (i=0; i<tE; i++)
*(Tempbis+(tE * i) + i) = 1.0;
if (nE==1)
{
memcpy(SortieExp, MatE, tE*tE*sizeof(double));
}
else
{
printf("kE=%d, nE=%d\n", kE, nE);
if (nE%2==0)
decompose(kE, nE,dec);
else
decompose(kE, nE-1, dec);
for (i=0; i<kE; i++)
{
carre(TempE, tE, SortieE);
memcpy(TempE, SortieE, tE*tE*sizeof(double));
tampon=*(dec+i);
if (tampon==1)
{
mult(Tempbis, tE, tE, SortieE, tE, tE, SortieExp);
memcpy(Tempbis, SortieExp, tE*tE*sizeof(double));
}
}
if (nE%2 !=0)
{
memcpy(Tempbis, SortieExp, tE*tE*sizeof(double));
mult(Tempbis, tE, tE, MatE, tE, tE, SortieExp);
}
}
free(TempE);
free(SortieE);
free(dec);
}
When I compile this code following error occurres
invalid conversion from 'int' to 'void*' [-fpermissive]|
that is about following line of code
if (!(dec = (int *)realloc(kE , sizeof(int))))
How can I remove this error?
You are passing int kE as the first parameter here:
realloc(kE , sizeof(int))
However, realloc is declared like this:
void *realloc(void *ptr, size_t size);
In other words it expects a pointer! Please read the manual page I link to above to get more details. In short, you probably want something like this for the error line:
if (!(dec = (int *)realloc(dec , sizeof(int))))
Note that this is slightly bad, because if realloc fails, you lose the original value of dec, causing a memory leak. It doesn't really matter if you are going to exit on error, but otherwise you should keep the original value of dec, so you can handle error more gracefully than just exiting.
Another note, you really should use C++ container classes like vector instead of fooling around with C memory allocation functions.
There are probably other issues in your code, but this answer doesn't try to be code review, but just explain why you get the error you get.
Try without do it like this :
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
int *pa = malloc(10 * sizeof *pa); // allocate an array of 10 int
if(pa) {
printf("%zu bytes allocated. Storing ints: ", 10*sizeof(int));
for(int n = 0; n < 10; ++n)
printf("%d ", pa[n] = n);
}
int *pb = realloc(pa, 1000000 * sizeof *pb); // reallocate array to a larger size
if(pb) {
printf("\n%zu bytes allocated, first 10 ints are: ", 1000000*sizeof(int));
for(int n = 0; n < 10; ++n)
printf("%d ", pb[n]); // show the array
free(pb);
} else { // if realloc failed, the original pointer needs to be freed
free(pa);
}
}
This question already has answers here:
What is an undefined reference/unresolved external symbol error and how do I fix it?
(39 answers)
Closed 8 years ago.
I've been trying to compile my code and finally got it down to just one error. If someone could explain whats going it would be great!
Here is my header file
rates.h
#ifndef RATES_H
#define RATES_H
#include <vector>
namespace power {
//! Energy usage over a period of time.
struct EnergyInterval
{
//! Constant or average power used during the interval [W].
const double power;
//! Length of the interval [s].
const double time;
};
//! The cost of using energy, within a specified bound.
struct Rate
{
/*!
* How much power this rate applies to, e.g., "the
* first 650 kWh", "the next 400 kWh" [kWh]
*/
const double amount;
//! The price of energy in this block [cents/kWh]
const double price;
};
/*!
* Calculates the cost of using energy over a billing period,
* broken down into time slices. Costs are calculated with the
* commonly-employed "so much at this price, so much at that price"
* model of rate schedules.
*
* #param[in] usage energy consumed in the billing period
* #param[in] rateSchedule the schedule of energy costs
*
* #returns total cost of energy used in the billing period [cents]
*/
double EnergyCost(const std::vector<EnergyInterval>& usage,
const std::vector<Rate>& rateSchedule);
} // namespace power
#endif
and rates.cpp
#include "rates.h"
#include <vector>
using namespace power;
double EnergyCost(const std::vector<EnergyInterval>& usage,
const std::vector<Rate>& rateSchedule){
double energyUsage = 0;
double cost = 0;
const double uSize = usage.size();
const double rsSize = rateSchedule.size();
double totalAmount = 0;
double secondTotal = 0;
for(int i=0; i < uSize; i++){
energyUsage += (usage[i].power*(usage[i].time/3600));
}
for(int j=0; j < rsSize; j++){
totalAmount += rateSchedule[j].amount;
if (energyUsage > totalAmount){
cost =+ rateSchedule[j].amount*rateSchedule[j].price;
}else if(energyUsage < totalAmount){
cost =+ (energyUsage - secondTotal)*rateSchedule[j].price;
break;
}
secondTotal += rateSchedule[j].amount;
}
return cost;
}
and test.cpp
#include "rates.h"
#include <vector>
#include <iostream>
using namespace std;
using namespace power;
int main()
{
vector<EnergyInterval> usage =
{
{.power = 60, .time = 60*60},
{.power = 2460, .time = 35*60},
{.power = 60, .time = 60*60}
};
vector<Rate> rateSchedule =
{
{.amount = 2000, .price = 10},
{.amount = 5000, .price = 5}
};
double totalCost = EnergyCost(usage, rateSchedule);
cout<<"The total cost is: " << totalCost;
return 0;
}
and when compiling in the command line I use:
g++ -std=c++11 -g test.cpp rates.cpp -o test
This is the error
C:>g++ -std=c++11 -g test.cpp rates.cpp -o test
C:\cchqT8Ro.o: In function `main':
C:\test.cpp:22: undefined reference to `power::EnergyCost(std::vector<power::EnergyInterval,
std::allocator<power::EnergyInterval> > const&, std::vector<power::Rate,
std::allocator<power::Rate> > const&)'
collect2.exe: error: ld returned 1 exit status
Any help or a push in the right direction would be great! Thanks!
In rates.cpp, you are creating a global function EnergyCost which is not the same as power::EnergyCost. This means that power::EnergyCost is still unimplemented.
You should implement EnergyCost inside the power namespace as follows:
// rates.cpp
namespace power {
double EnergyCost(const std::vector<EnergyInterval>& usage,
const std::vector<Rate>& rateSchedule){
// ..
// your code
// ..
}
}
I think EngergyCost function implementation should contained in the power namespace.
namespace Power {
double EnergyCost(const std::vector<EnergyInterval>& usage,
const std::vector<Rate>& rateSchedule){
double energyUsage = 0;
double cost = 0;
const double uSize = usage.size();
const double rsSize = rateSchedule.size();
double totalAmount = 0;
//:
}
}
I have been searching all morning how to do this and I can't quite find what I am looking for. I should mention that I am fairly new to C.
I am trying to modulise my code (that is working until I try and do this)
Presently in the main code I have :
#include<stdio.h>
#include<rfftw.h>
#include<stdlib.h>
#include<math.h>
#include<string.h>
#include <fstream>
#include <iomanip>
#include<complex>
#include<omp.h>
struct basic_gal {
double ra,dec,z,dist,fkp,nbar;
double cp[3];
double RSD[3];
};
struct basic_gal *gal;
int NGAL_MAX =200000;
main() {
if(!(gal = (struct basic_gal*)malloc(NGAL_MAX*sizeof(struct basic_gal))-1))
printf("memory allocation problem for galaxies\n");
etc etc..
}
I then go on to read in a file and allocate the various attributes to the structure.
What I would like to do is pass the empty structure (or a pointer to it) to a function, fill in the elements of the structure in the function and then return it to use in my main program again.
This requires 3 files, the main.c, the header file, header.h and the file that contains the function actions bin_gals.c
In the header file I have
#include <stdlib.h>
#include <fstream>
#include <iomanip>
#include <stdio.h>
#include <math.h>
#include <string>
void bin_NGP(int,int*,struct basic_gal*,int);
In the bin_gals.c file I have
#include "header.h"
#include <all_the_others>
void bin_NGP(int NGAL_MAX, int *NGAL, basic_gal *gal, int flag) {
/*read in files and add data etc*/
}
And in the main.c file I have
#include "header.h"
#include <all_the_others>
struct basic_gal {
double ra,dec,z,dist,fkp,nbar;
double cp[3];
double RSD[3];
};
struct basic_gal *gal;
int NGAL_MAX = 200000;
main() {
if(!(gal = (struct basic_gal*)malloc(NGAL_MAX*sizeof(struct basic_gal))-1))
printf("memory allocation problem for galaxies\n");
int NGAL =0;
int *ipNGAL =&NGAL;
bin_NGP(NRAN_MAX,ipNGAL,gal,1);
}
The problem is that I think I am not passing the structure properly but cannot figure out how to do this.
My error messages look like
bin_gals.c: In function ‘void bin_NGP(int, int*, basic_gal*, int)’:
bin_gals.c:150: error: ISO C++ forbids comparison between pointer and integer
bin_gals.c:151: error: invalid types ‘basic_gal*[int*]’ for array subscript
and I am not sure how to fix this.
Any help would be much appreciated!
EDIT: The actual content of the bin_NGP part of bin_gals.c is:
void bin_NGP(const char *data, int NGAL_MAX, int *NGAL, double *min_x, double *min_y ,double *min_z, double *max_x, double *max_y ,double *max_z ,struct basic_gal *gal, int flag) {
FILE *fp_rand;
if((fp_rand=fopen(data,"r"))==NULL) printf("data file %d not opened\n", flag);
const int bsz=100; char buf[bsz];
fgets(buf, bsz, fp_rand); //header line
while((fgets(buf, bsz, fp_rand))!=NULL) {
double ra, dec, cz;
sscanf(buf,"%lf %lf %lf\n",&ra,&dec,&cz);
if(++NGAL>NGAL_MAX) { NGAL--; break; }
gal[NGAL].ra = ra*pi/180.;
gal[NGAL].dec = dec*pi/180.;
gal[NGAL].z = cz;
gal[NGAL].dist = calc_dp(gal[NGAL].z);
gal[NGAL].cp[0] = (gal[NGAL].dist*cos(gal[NGAL].dec)*cos(gal[NGAL].ra));
gal[NGAL].cp[1] = (gal[NGAL].dist*cos(gal[NGAL].dec)*sin(gal[NGAL].ra));
gal[NGAL].cp[2] = (gal[NGAL].dist*sin(gal[NGAL].dec));
if (flag ==1) {
if (gal[NGAL].cp[0] > max_x) max_x = gal[NGAL].cp[0];
if (gal[NGAL].cp[1] > max_y) max_y = gal[NGAL].cp[1];
if (gal[NGAL].cp[2] > max_z) max_z = gal[NGAL].cp[2];
if (gal[NGAL].cp[0] < min_x) min_x = gal[NGAL].cp[0];
if (gal[NGAL].cp[1] < min_y) min_y = gal[NGAL].cp[1];
if (gal[NGAL].cp[2] < min_z) min_z = gal[NGAL].cp[2];
}
}
fclose(fp_rand);
}
I tried to simplify the question by removing some of the input parameters.. maybe I was disguising a mistake.
EDIT: fixed. For clarity I have included the fixed code incase anyone has the same problem.
void bin_NGP(const char *data, int NGAL_MAX, int *NGAL, double *min_x, double *min_y ,double *min_z, double *max_x, double *max_y ,double *max_z ,struct basic_gal *gal, int flag) {
FILE *fp_rand;
if((fp_rand=fopen(data,"r"))==NULL) printf("data file %d not opened\n", flag);
const int bsz=100; char buf[bsz];
fgets(buf, bsz, fp_rand); //header line
while((fgets(buf, bsz, fp_rand))!=NULL) {
double ra, dec, cz;
sscanf(buf,"%lf %lf %lf\n",&ra,&dec,&cz);
if(++(*NGAL) > NGAL_MAX) { *NGAL--; break; }
gal[*NGAL].ra = ra*pi/180.;
gal[*NGAL].dec = dec*pi/180.;
gal[*NGAL].z = cz;
gal[*NGAL].dist = calc_dp(gal[*NGAL].z);
gal[*NGAL].cp[0] = (gal[*NGAL].dist*cos(gal[*NGAL].dec)*cos(gal[*NGAL].ra));
gal[*NGAL].cp[1] = (gal[*NGAL].dist*cos(gal[*NGAL].dec)*sin(gal[*NGAL].ra));
gal[*NGAL].cp[2] = (gal[*NGAL].dist*sin(gal[*NGAL].dec));
if (flag ==1) {
if (gal[*NGAL].cp[0] > *max_x) *max_x = gal[*NGAL].cp[0];
if (gal[*NGAL].cp[1] > *max_y) *max_y = gal[*NGAL].cp[1];
if (gal[*NGAL].cp[2] > *max_z) *max_z = gal[*NGAL].cp[2];
if (gal[*NGAL].cp[0] < *min_x) *min_x = gal[*NGAL].cp[0];
if (gal[*NGAL].cp[1] < *min_y) *min_y = gal[*NGAL].cp[1];
if (gal[*NGAL].cp[2] < *min_z) *min_z = gal[*NGAL].cp[2];
}
}
fclose(fp_rand);
}
You need to forward declare struct basic_gal* for the declaration of bin_NGP() as struct basic_gal is defined in main.c. However, the definition of struct basic_gal will need to be available to the definition of bin_NGP() so you should move its definition out of main.c into a separate module:
/* basic_gal.h */
#ifndef BASIC_GAL_DEFINITION
#define BASIC_GAL_DEFINITION
struct basic_gal {
double ra,dec,z,dist,fkp,nbar;
double cp[3];
double RSD[3];
};
#endif
Note, if this is C you need to use struct basic_gal.
After edit, this is the offending line:
if(++NGAL>NGAL_MAX) { NGAL--; break; }
as NGAL is an int* and NGAL_MAX is an int. Dereference NGAL:
if(++*NGAL > NGAL_MAX) { (*NGAL)--; break; }
Dereference NGAL when using as array index:
gal[*NGAL].ra = ra*pi/180.;
You need to make your structure visible to both main.c and bin_gals.c. Modify your header bin_gals.h to something like:
#include <stdlib.h>
#include <fstream>
#include <iomanip>
#include <stdio.h>
#include <math.h>
#include <string>
struct basic_gal {
double ra,dec,z,dist,fkp,nbar;
double cp[3];
double RSD[3];
};
void bin_NGP(int,int*,struct basic_gal*,int);
bin_gals.c then would look like
#include "bin_gals.h"
#include <all_the_others>
void bin_NGP(int NGAL_MAX, int *NGAL, basic_gal *gal, int flag) {
/*read in files and add data etc*/
}
And your main file would be like the following:
#include "bin_gals.h"
#include <all_the_others>
struct basic_gal *gal;
int NGAL_MAX = 200000;
main() {
if(!(gal = (struct basic_gal*)malloc(NGAL_MAX*sizeof(struct basic_gal))-1))
printf("memory allocation problem for galaxies\n");
int NGAL =0;
int *ipNGAL =&NGAL;
bin_NGP(NRAN_MAX,ipNGAL,gal,1);
}