c++ vector not filling correctly - c++

I'm making an engine that is supposed to read formatted text files and output them as a text based adventure. The world is being written into a vector matrix. However, my program only seems to fill the matrix in one Dimension and only with the information from the very first cell of the matrix.
The WorldReader reads the World file and returns a specified line:
std::string WorldReader(std::string file,int line)
{
std::string out[n];
int i = 0;
World.open(file + "World.txt");
if(!World.good())
return "Bad File";
else while(i<n && getline(World, out[i]))
{
i++;
}
World.close();
return out[line];
}
Here is the write loop:
for(j=0; j<(width*height); j++)
{
int x;
int y;
stringstream Coordx(WorldReader(loc, 4+j*10));
Coordx >> x;
stringstream Coordy(WorldReader(loc, 5+j*10));
Coordy >> y;
std::string Desc = WorldReader(loc, 6+j*10);
W1.writeCell(x,y,0,Desc);
}
and here is the writeCell function:
std::vector<std::string> Value;
std::vector<std::vector<std::string> > wH;
std::vector< std::vector<std::vector<std::string> > > grid;
void World::writeCell(int writelocW, int writelocH, int ValLoc, std::string input)
{
if (wH.size() > writelocH)
{
Value.insert(Value.begin()+ValLoc,1,input);
wH.insert(wH.begin() + writelocH,1,Value);
grid.insert(grid.begin() + writelocW,1,wH);
}
else
{
wH.insert(wH.begin(),1,Value);
grid.insert(grid.begin(),1,wH);
}
}
also the matrix is getting immensely bloated even though i resized it to 3x3.
tips and help appreciated.

Ok. I think I know where your problem is. Please note this is extremely difficult to analyze without genuinely-runnable code. The high-point is this: You're inserting a new 2D matrix for every value you process into your grid, and I hope it is clear why this is the case. It explains the mass-bloat (and inaccurate data) you're experiencing.
Your original code
void World::writeCell(int writelocW, int writelocH, int ValLoc, std::string input)
{
if (wH.size() > writelocH)
{
// inserts "input" into the Value member.
Value.insert(Value.begin()+ValLoc,1,input);
// inserts a **copy** of Value into wH
wH.insert(wH.begin() + writelocH,1,Value);
// inserts a **copy** of wH into the grid.
grid.insert(grid.begin() + writelocW,1,wH);
}
else
{ // inserts a **copy** of Value into wH
wH.insert(wH.begin(),1,Value);
// inserts a **copy** of wH into the grid.
grid.insert(grid.begin(),1,wH);
}
}
As you can plainly see. there is a whole lot of unintended copying going on here. You have three variables, each of which is independent.
std::vector<std::string> Value;
std::vector<std::vector<std::string> > wH;
std::vector< std::vector<std::vector<std::string> > > grid;
During the course of writeCell you are trying to insert your string into a 3D location, but only "dereferencing" to at-most one of those dimensions. And copies o-festival ensues
From your variable names I'm assuming your grid dimensionality is based on:
writeocW * writelocH * ValLoc
You need to unwind the dimensions in most-to-least significant order, starting with grid. ultimately that is how it is accessed anyway. I personally would use a sparse std::map<> series for this, as the space utilization would be much more efficient, but we're working with what you have. I'm writing this off-the-cuff with no nearby compiler to check for mistakes, so grant me a little latitude.
Proposed Solution
This is a stripped down version of the World class you no-doubt have. I've changed the names of the params to traditional 3D coords (x,y,z) in an effort to make it clear how to do what I think you want:
class World
{
public:
typedef std::vector<std::string> ValueRow;
typedef std::vector<ValueRow> ValueTable;
typedef std::vector<ValueTable> ValueGrid;
ValueGrid grid;
// code omitted to get to your writeCell()
void writeCell(size_t x, size_t y, size_t z, const std::string& val)
{
// resize grid to hold enough tables if we would
// otherwise be out of range.
if (grid.size() < (x+1))
grid.resize(x+1);
// get referenced table, then do the same as above,
// this time making appropriate space for rows.
ValueTable& table = grid[x];
if (table.size() < (y+1))
table.resize(y+1);
// get referenced row, then once again, just as above
// make space if needed to reach the requested value
ValueRow& row = table[y];
if (row.size() < (z+1))
row.resize(z+1);
// and finally. store the value.
row[z] = val;
}
};
I think that will get you where you want. Note that using large coords can quickly grow this cube.
Alternate Solution
Were it up to me I would use something like this:
typedef std::map<size_t, std::string> ValueMap;
typedef std::map<size_t, ValueMap> ValueRowMap;
typedef std::map<size_t, ValueRowMap> ValueGridMap;
ValueGridMap grid;
Since you'd be enumerating these when doing whatever it is you're doing with this grid, order of the keys (the 0-based indexes) is important, thus usage of std::map rather than std::unordered_map. An std::map has a very nice feature with its operator[]() accessor: It adds the referenced key slot if it doesn't already exist. Thus your writeCell function would collapse to this:
void writeCell(size_t x, size_t y, size_t z, const std::string& val)
{
grid[x][y][z] = val;
}
Obviously this would radically alter the way you use the container, as you would have to be conscious of the "skipped" indexes you're not using, and you would detect this while enumerating with the appropriate iterator for the dimension(s) being used. Regardless, your storage would be much more efficient.
Anyway, I hope this helps at least a little.

Related

Rotate elements in a vector and how to return a vector

c++ newbie here. So for an assignment I have to rotate all the elements in a vector to the left one. So, for instance, the elements {1,2,3} should rotate to {2,3,1}.
I'm researching how to do it, and I saw the rotate() function, but I don't think that will work given my code. And then I saw a for loop that could do it, but I'm not sure how to translate that into a return statement. (i tried to adjust it and failed)
This is what I have so far, but it is very wrong (i haven't gotten a single result that hasn't ended in an error yet)
Edit: The vector size I have to deal with is just three, so it doesn't need to account for any sized vector
#include <vector>
using namespace std;
vector<int> rotate(const vector<int>& v)
{
// PUT CODE BELOW THIS LINE. DON'T CHANGE ANYTHING ABOVE.
vector<int> result;
int size = 3;
for (auto i = 0; i < size - 1; ++i)
{
v.at(i) = v.at(i + 1);
result.at(i) = v.at(i);
}
return result;
// PUT CODE ABOVE THIS LINE. DON'T CHANGE ANYTHING BELOW.
}
All my teacher does it upload textbook pages that explain what certain parts of code are supposed to do but the textbook pages offer NO help in trying to figure out how to actually apply this stuff.
So could someone please give me a few pointers?
Since you know exactly how many elements you have, and it's the smallest number that makes sense to rotate, you don't need to do anything fancy - just place the items in the order that you need, and return the result:
vector<int> rotate3(const vector<int>& x) {
return vector<int> { x[1], x[2], x[0] };
}
Note that if your collection always has three elements, you could use std::array instead:
std::array<int,3>
First, just pay attention that you have passed v as const reference (const vector<int>&) so you are forbbiden to modify the state of v in v.at(i) = v.at(i + 1);
Although Sergey has already answered a straight forward solution, you could correct your code like this:
#include <vector>
using namespace std;
vector<int> left_rotate(const vector<int>& v)
{
vector<int> result;
int size = v.size(); // this way you are able to rotate vectors of any size
for (auto i = 1; i < size; ++i)
result.push_back(v.at(i));
// adding first element of v at end of result
result.push_back(v.front());
return result;
}
Use Sergey's answer. This answer deals with why what the asker attempted did not work. They're damn close, so it's worth going though it, explaining the problems, and showing how to fix it.
In
v.at(i) = v.at(i + 1);
v is constant. You can't write to it. The naïve solution (which won't work) is to cut out the middle-man and write directly to the result vector because it is NOT const
result.at(i) = v.at(i + 1);
This doesn't work because
vector<int> result;
defines an empty vector. There is no at(i) to write to, so at throws an exception that terminates the program.
As an aside, the [] operator does not check bounds like at does and will not throw an exception. This can lead you to thinking the program worked when instead it was writing to memory the vector did not own. This would probably crash the program, but it doesn't have to1.
The quick fix here is to ensure usable storage with
vector<int> result(v.size());
The resulting code
vector<int> rotate(const vector<int>& v)
{
// PUT CODE BELOW THIS LINE. DON'T CHANGE ANYTHING ABOVE.
vector<int> result(v.size()); // change here to size the vector
int size = 3;
for (auto i = 0; i < size - 1; ++i)
{
result.at(i) = v.at(i + 1); // change here to directly assign to result
}
return result;
// PUT CODE ABOVE THIS LINE. DON'T CHANGE ANYTHING BELOW.
}
almost works. But when we run it on {1, 2, 3} result holds {2, 3, 0} at the end. We lost the 1. That's because v.at(i + 1) never touches the first element of v. We could increase the number of for loop iterations and use the modulo operator
vector<int> rotate(const vector<int>& v)
{
// PUT CODE BELOW THIS LINE. DON'T CHANGE ANYTHING ABOVE.
vector<int> result(v.size());
int size = 3;
for (auto i = 0; i < size; ++i) // change here to iterate size times
{
result.at(i) = v.at((i + 1) % size); // change here to make i + 1 wrap
}
return result;
// PUT CODE ABOVE THIS LINE. DON'T CHANGE ANYTHING BELOW.
}
and now the output is {2, 3, 1}. But it's just as easy, and probably a bit faster, to just do what we were doing and tack on the missing element after the loop.
vector<int> rotate(const vector<int>& v)
{
// PUT CODE BELOW THIS LINE. DON'T CHANGE ANYTHING ABOVE.
vector<int> result(v.size());
int size = 3;
for (auto i = 0; i < size - 1; ++i)
{
result.at(i) = v.at(i + 1);
}
result.at(size - 1) = v.at(0); // change here to store first element
return result;
// PUT CODE ABOVE THIS LINE. DON'T CHANGE ANYTHING BELOW.
}
Taking this a step further, the size of three is an unnecessary limitation for this function that I would get rid of and since we're guaranteeing that we never go out of bounds in our for loop, we don't need the extra testing in at
vector<int> rotate(const vector<int>& v)
{
// PUT CODE BELOW THIS LINE. DON'T CHANGE ANYTHING ABOVE.
if (v.empty()) // nothing to rotate.
{
return vector<int>{}; // return empty result
}
vector<int> result(v.size());
for (size_t i = 0; i < v.size() - 1; ++i) // Explicitly using size_t because
// 0 is an int, and v.size() is an
// unsigned integer of implementation-
// defined size but cannot be larger
// than size_t
// note v.size() - 1 is only safe because
// we made sure v is not empty above
// otherwise 0 - 1 in unsigned math
// Becomes a very, very large positive
// number
{
result[i] = v[i + 1];
}
result.back() = v.front(); // using direct calls to front and back because it's
// a little easier on my mind than math and []
return result;
// PUT CODE ABOVE THIS LINE. DON'T CHANGE ANYTHING BELOW.
}
We can go further still and use iterators and range-based for loops, but I think this is enough for now. Besides at the end of the day, you throw the function out completely and use std::rotate from the <algorithm> library.
1This is called Undefined Behaviour (UB), and one of the most fearsome things about UB is anything could happen including giving you the expected result. We put up with UB because it makes for very fast, versatile programs. Validity checks are not made where you don't need them (along with where you did) unless the compiler and library writers decide to make those checks and give guaranteed behaviour like an error message and crash. Microsoft, for example, does exactly this in the vector implementation in the implementation used when you make a debug build. The release version of Microsoft's vector make no checks and assumes you wrote the code correctly and would prefer the executable to be as fast as possible.
I saw the rotate() function, but I don't think that will work given my code.
Yes it will work.
When learning there is gain in "reinventing the wheel" (e.g. implementing rotate yourself) and there is also gain in learning how to use the existing pieces (e.g. use standard library algorithm functions).
Here is how you would use std::rotate from the standard library:
std::vector<int> rotate_1(const std::vector<int>& v)
{
std::vector<int> result = v;
std::rotate(result.begin(), result.begin() + 1, result.end());
return result;
}

Why does the unordered_map perform about 100 times slower than a 2d vector for look-ups?

I tried implementing the knapsack algorithm for large data sets.
The 2d vector solution works perfectly for medium data sets around 100 items.
Since the 2d vector won't be feasible for large datasets involving around 1000 items, I decided to use a hashtable and cache the values as required.
I have used hash_value() from boost to hash std::pair into the unordered_map.
But I don't understand why this solution works incredibly slower than
the 2d vector solution. Aren't hashtables meant for super fast look ups?
Both the implementations fail to process the large data set in finite time.
I've attached the code and both the "medium" and "large" data sets.
The code has both unordered_map and 2d vector implementations with the latter commented out.
It would be really helpful if someone could point out the reason for the slow performance and suggest some optimization so that it is able to process the large dataset.
The input file is of the form.
(eg):
6 4 //weight, no of items
3 4
2 3
4 2
4 3
The optimal solution is 8.
Download Link for Large dataset (1000 items)
Download Link for Medium dataset (100 items)
Download Link for Source Code
//code follows:
//Headers, Macros and Global variables:
#include<iostream>
#include<vector>
#include<algorithm>
#include<fstream>
#include<string>
#include<sstream>
#include<unordered_map>
#include<boost/functional/hash.hpp>
using namespace std;
typedef vector< vector<long long> > vii;
typedef vector< pair<int,int> > vp;
typedef pair<int,int> myPair;
typedef unordered_map< myPair, long long, boost::hash<myPair> > myMap;
vp elmnts;
//vii arr2d;
myMap arr;
//Knapsack function:
long long knapsack(int n, int w)
{
//arr2d.resize(n+1, vector<long long>(w+1));
int vi,wi;
for(int j=0; j<=w; ++j)
// arr2d[0][j] = 0;
arr.emplace(make_pair(0,j), 0);
for(int i=1; i<=n; ++i)
{
vi = elmnts[i-1].first;
wi = elmnts[i-1].second;
for(int j=0; j<=w; ++j)
// arr2d[i][j] = (wi > j) ? arr2d[i-1][j] : max(arr2d[i-1][j], arr2d[i-1][j-wi] + vi);
arr.emplace(make_pair(i,j), (wi > j) ? arr[make_pair(i-1,j)] : max(arr[make_pair(i-1,j)], arr[make_pair(i-1,j-wi)]+ vi));
}
//return arr2d[n][w];
return arr[make_pair(n,w)];
}
//Main fucntion
int main()
{
ifstream file("/home/tauseef/Desktop/DAA2/knapsack1.txt");
int n,w;
string line;
pair<int,int> elmnt;
getline(file, line);
stringstream ss(line);
ss >> w;
ss >> n;
while(getline(file, line))
{
stringstream ss1(line);
ss1 >> elmnt.first;
ss1 >> elmnt.second;
elmnts.push_back(elmnt);
}
cout<<"\nThe optimal solution is: "<<knapsack(n,w)<<endl;
file.close();
}
Didn't expect the difference to be so huge: On my machine the array version is 100 times faster than the hash_map version. But after thinking about it...
You should expect the map being slower - there is a lot of overhead: invoking make_pair, creating a pair-object, calculating hash, searching it in the map, constructing return value, copying objects back and forth opposed to just looking-up the value!
On the other hand you don't profit from your switch to the map at all because in the end, as it is coded right now, you have the same elements in your map as in the array. Your change would make sense if you would leave some elements out from your map, but you don't do it.
But the bigger problem in your code is that you use the pseudo-polinomial algorithm from wikipedia which needs O(n*W) memory. That means you would need 32GB memory for the bigger test cases, which could mean swapping memory with the hard disc, depending on how big your system is and getting really sloooow.
The solution is to take the version of the algorithm which needs O(W) memory:
long long knapsack(const vector<myPair> &objects, int W)
{
vector<long long> bests(W+1, -1L);//-1 = this value is not reachable
//the only possible configuration at start: an empty set, value is also 0
bests[0]=0;
for(const myPair &object:objects){//try out all objects
int v = object.first;
int w = object.second;
//update all possible configurations:
for(int cur=W;cur>=0;--cur){//backwards->object at most once in every configuration!
int next=cur+w;
if(bests[cur]!=-1 && next<=W)//consider only reachable configurations and discard too big weights
bests[next]=std::max(bests[next], bests[cur]+v);
}
}
return *max_element(bests.begin(), bests.end());
}
The most important part is that we go backwards through the possible configurations and thus can update the configurations in-place (the updated configurations are the ones already proceeded in the current sweep).
I guess this version should need less than 1 minute for the bigger case (which is pretty reasonable considering how big the input is). I don't guaranty that this is bug-free, but hope you can get the gist of it.

Vector performance suffering

I've been working on state space exploration and was originally using a map to store the assignment of the world states like map<Variable *, int>, where variables are objects in the world with a domain from 0 to n where n is finite. The implementation was extremely quick for performance, but I noticed that it does not scale well with the size of the state space. I changed the states to use vector<int> instead, where I use the id of a variable to find its index in the vector. Memory usage improved greatly, but the efficiency of the solver has tanked (gone from <30 seconds to 400+). The only code that I modified was generating the states and validating if the state is the goal. I can't figure out why using a vector has degraded performance, especially since the vector operations should only take linear time at worst.
Originally this is was how I generated nodes:
State * SuccessorGen::generate_successor(const Operator &op, map<Variable *, int> &var_assignment){
map<Variable *, int> values;
values.insert(var_assignment.begin(), var_assignment.end());
vector<Operator::Effect> effect = op.get_effect();
vector<Operator::Effect>::const_iterator eff_it = effect.begin();
for (; eff_it != effect.end(); eff_it++){
values[eff_it->var] = eff_it->after;
}
return new State(values);
}
And in my new implementation:
State* SuccessorGen::generate_successor(const Operator &op, const vector<int> &assignment){
vector<int> child;
child = assignment;
vector<Operator::Effect> effect = op.get_effect();
vector<Operator::Effect>::const_iterator eff_it = effect.begin();
for (; eff_it != effect.end(); eff_it++){
Variable *v = eff_it->var;
int id = v->get_id();
child[id] = eff_it->after;
}
return new State(child);
}
(The goal checking is similar, just looping over the goal assignment instead of operator effects.)
Are these vector operations really that much slower than using a map? Is there an equally efficient STL container I can use that has a lower overhead? The number of variables is relatively small (<50) and the vector never needs to be resized or modified after the for loop.
Edit:
I tried timing one loop through all the operators to see timing comparisons, with the effect list and assignment the vector version runs one loop in 0.3 seconds, while the map version is a little over 0.4 seconds. When I comment that section out the map was about the same, yet the vector jumped up to closer to 0.5 seconds. I added child.reserve(assignment.size()) but that did not make any change.
Edit 2:
From user63710's answer, I've also been digging through the rest of the code and noticed something really strange going on in the heuristic calculation. The vector version works fine, but for the map I use this line Node *n = new Node(i, transition.value, label_cost); open_list.push(n);, but once the loop finishes filling the queue the node gets totally screwed up. Nodes are a simple struct as:
struct Node{
// Source Value, Destination Value
int from;
int to;
int distance;
Node(int &f, int &t, int &d) : from(f), to(t), distance(d){}
};
Instead of having from, to, distance, it replaces from and to with id with some random number, and that search does not do what it should and is returning much faster then it should. When I tweak the map version to convert the map to a vector and run this:
Node n(i, transition.value, label_cost); open_list.push(n);
the performance is about equal to that of the vector. So that fixes my main issue, but this leaves me wondering why using Node *n gets this behaviour opposed to Node n()?
If as you say, the sizes of these structures are fairly small (~50 elements), I have to think that the issue is somewhere else. At least, I don't think it involves the memory accesses or allocation of the vector/map.
Some example code I made to test: Map version:
unique_ptr<map<int, int>> make_successor_map(const vector<int> &ids,
const map<int, int> &input)
{
auto new_map = make_unique<map<int, int>>(input.begin(), input.end());
for (size_t i = 0; i < ids.size(); ++i)
swap((*new_map)[ids[i]], (*new_map)[i]);
return new_map;
}
int main()
{
auto a_map = make_unique<map<int, int>>();
// ids to access
vector<int> ids;
const int n = 100;
for (int i = 0; i < n; ++i)
{
a_map->insert({i, rand()});
ids.push_back(i);
}
random_shuffle(ids.begin(), ids.end());
for (int i = 0; i < 1e6; ++i)
{
auto temp_map = make_successor_map(ids, *a_map);
swap(temp_map, a_map);
}
cout << a_map->begin()->second << endl;
}
Vector version:
unique_ptr<vector<int>> make_successor_vec(const vector<int> &ids,
const vector<int> &input)
{
auto new_vec = make_unique<vector<int>>(input);
for (size_t i = 0; i < ids.size(); ++i)
swap((*new_vec)[ids[i]], (*new_vec)[i]);
return new_vec;
}
int main()
{
auto a_vec = make_unique<vector<int>>();
// ids to access
vector<int> ids;
const int n = 100;
for (int i = 0; i < n; ++i)
{
a_vec->push_back(rand());
ids.push_back(i);
}
random_shuffle(ids.begin(), ids.end());
for (int i = 0; i < 1e6; ++i)
{
auto temp_vec = make_successor_vec(ids, *a_vec);
swap(temp_vec, a_vec);
}
cout << *a_vec->begin() << endl;
}
The map version takes around 15 seconds to run on my old Core 2 Duo T9600, and the vector version takes 0.406 seconds. Both we're compiled on G++ 4.9.2 with g++ -O3 --std=c++1y. So if your code takes 0.4s per iteration (note that it took my example code 0.4s for 1 million calls), then I'm really thinking your problem is somewhere else.
That's not to say you aren't having performance decreases due to switching from map->vector, but that the code you posted doesn't show much reason for that to happen.
The problem is that you create vectors without reserving space. Vectors store elements contiguously. That ensures constant access to elements.
So everytime you add an item to the vector (for example via your inserter), the vector has to reallocate more space and eventuelly move all the existing elements to a reallocated memory location. This causes slowdown and considerable heap fragmentation.
The solution to this is to reserve() elements if you know in advance how many elements you'll have. Or if you don't reserve() larger chunks and compare size() and capacity() to check if it's time to reserve more.

Access Violation reading elements of an array

I've written my own code to parse an .obj model file - essentially just ASCII text. The file gets parsed and stored in the class correctly according to my tests. I can read back the values (from data members) just fine in the loading function.
The problem occurs when I try to read back the values in my main rendering loop. There is an access violation error on the line beginning "int v":
for(int i = 0; i<data.numFaces; i++){
for(int j = 0; j<3; j++){ //Assuming triangles for now.
int v = data.faceList[i].vertex[j]; // Access violation here.
double vX = data.vertexList[v].x;
double vY = data.vertexList[v].y;
double vZ = data.vertexList[v].z;
glVertex3d(vX, vY, vZ);
}
}
I'm not exactly sure why this happens, and I've checked everything I could possibly think of. I'm not very experienced in C++. Most of my programming experience is in Java, Python and PHP although I have previously written a medium sized project in C++.
I'm sure the problem is something basic related to memory allocation or pointers used for the dynamic arrays.
Here are the relevant parts of code in the obj loading class:
ObjData ObjLoader::LoadObj(std::string filename){
//... Initalization ...
// 1st pass: find number of elements so array sizes can be defined.
while(!file.eof()){
//...
}
//...close file...
_data.faceList = new ObjFace[_data.numFaces];
_data.vertexList = new ObjVert[_data.numVertices];
_data.uvList = new ObjUV[_data.numUVcoords];
_data.normalList = new ObjNormal[_data.numNormals];
//TODO: Make size dynamic according to each face. Just use the first 3 points for now.
for (int i = 0; i < _data.numFaces; i++){
_data.faceList[i].vertex = new int[3];
_data.faceList[i].normal = new int[3];
_data.faceList[i].uv = new int[3];
}
//... file stuff ...
// 2nd pass: read values into arrays.
while(!file.eof()){
//...
if(type=="v"){
_data.vertexList[currentVertex].x = atof(param1.c_str());
_data.vertexList[currentVertex].y = atof(param2.c_str());
_data.vertexList[currentVertex].z = atof(param3.c_str());
currentVertex++;
}else if(type=="vt"){
_data.uvList[currentUV].u = atof(param1.c_str());
_data.uvList[currentUV].v = atof(param2.c_str());
currentUV++;
}else if(type=="vn"){
_data.normalList[currentNormal].x = atof(param1.c_str());
_data.normalList[currentNormal].y = atof(param2.c_str());
_data.normalList[currentNormal].z = atof(param3.c_str());
currentNormal++;
}else if(type=="f"){
//...Within loop over each vertex in a single face ...
if(endPos != string::npos){
// Value before 1st "/" (Vertex index).
// ...find value in string...
_data.faceList[currentFace].vertex[i] = atoi(token.c_str()) -1; // File format begins indices from 1.
// Value between slashes (UV index).
// ...find value in string...
_data.faceList[currentFace].uv[i] = atoi(token.c_str()) -1;
// Value after 2nd "/" (Normal index).
// ...find value in string...
_data.faceList[currentFace].normal[i] = atoi(token.c_str()) -1;
}
//...End of loop over every vertex in a single face...
currentFace++;
}
}
return _data;
}
And the structs ObjFace, ObjVert, ObjUV and ObjNormal are defined as:
struct ObjVert{
float x, y, z;
};
struct ObjUV{
float u, v;
};
struct ObjNormal{
float x, y, z;
};
// Contains indexes.
struct ObjFace{
int* vertex;
int* uv;
int* normal;
};
Thanks for any help. Also, any good sources on avoiding these types of errors in future would be appreciated.
I had typed up some silly response which I realized wasn't right...but I had to keep thinking on it, and I came up with another idea.
Where is this object being assigned?
Your code isn't clear if data and _data are the same object or not, but I noticed that your method there seems to return _data as an object. I'm led to believe that perhaps you are using an assignment like ObjData data = LoadObj("myfilename"); somewhere?
If this is the case, I believe your problem may come from a lack of a copy constructor or overloaded assignment operator for your ObjData class. (I'm not a C++ guru, so I don't recall exactly which one this would fall under. Hopefully someone else can confirm if we're on the right track).
If your pointers are not being copied correctly during the copy and assignment (returning from LoadObj calls a copy constructor iirc, and then the obvious assignment to data), then even though you intended to already have an array of int at that location, you may in fact be accessing uninitialized memory, thus causing your Access Violation.
I'm not an expert with either copy constructors or overloaded assignment operators, but a quick way around this would be to return a pointer to an ObjData rather than to return an object itself.
At first glance I didn't see anything blatantly wrong. But if as you say it is exploding at int v = data.faceList[i].vertex[j]; then it seems very likely that the problem is either i or j or both are too big or too small.
Aside from the obvious approach of getting cozy with your debugger and flattening this bug out, the best approach to solving problems like these is probably to avoid them altogether. Certain things programmers do are more prone to errors than others. The list is long, but you do at least two of them, in spades, in your code.
1) You use dynamically-allocated arrays
2) You use hand-crafted loops
Try to avoid doing these things by using the tools that C++ gives you. Start with #1, and get rid of the dynamically-allocated arrays.
You have a struct:
struct ObjFace{
int* vertex;
int* uv;
int* normal;
};
...with 3 pointers-to-arrays-of-int. Instead of doing that, use a vector:
struct ObjFace{
vector<int> vertex;
vector<int> uv;
vector<int> normal;
};
...and then a whole lot of code you had to write before becomes much simpler now, and much less prone to mistakes:
// all this goes away
//_data.faceList = new ObjFace[_data.numFaces];
//_data.vertexList = new ObjVert[_data.numVertices];
//_data.uvList = new ObjUV[_data.numUVcoords];
//_data.normalList = new ObjNormal[_data.numNormals];
...and:
// now you ask the vector how many elements it really has
for(int i = 0; i<data.faceList.size(); i++){
for(int j = 0; j<data.faceList.size(); j++){ //Ask the vector instead of assuming triangles
int v = data.faceList[i].vertex[j]; // Access violation here.
double vX = data.vertexList[v].x;
double vY = data.vertexList[v].y;
double vZ = data.vertexList[v].z;
glVertex3d(vX, vY, vZ);
}
}
Now, look at that loop. Loops are a very common source of errors. The best loop is the loop you never have to write. So use the STL's algorithms instead. Add a function to ObjFace to execute glVertex3d on each of it's elements:
struct ObjFace{
//...
void do_vertex() const
{
typedef vector<int> ints;
for( ints::iterator it = vertex.begin(); it != vertex.end(); ++it )
glVertex3d(it->x, it->y, it->z);
}
};
...then go back and whittle down that original loop:
(psudocode, actual syntax is more complex)
typedef vector<ObjFace> ObjFaces;
for( ObjFaces::iterator it = data.faceList.begin(); it != data.faceList.end(); ++it )
it->do_vertex();
...or, with a little more effort:
for_each( data.faceList.begin(), data.faceList.end(), &ObjFace::do_vertex );

What is the best way to create a sparse array in C++?

I am working on a project that requires the manipulation of enormous matrices, specifically pyramidal summation for a copula calculation.
In short, I need to keep track of a relatively small number of values (usually a value of 1, and in rare cases more than 1) in a sea of zeros in the matrix (multidimensional array).
A sparse array allows the user to store a small number of values, and assume all undefined records to be a preset value. Since it is not physically possibly to store all values in memory, I need to store only the few non-zero elements. This could be several million entries.
Speed is a huge priority, and I would also like to dynamically choose the number of variables in the class at runtime.
I currently work on a system that uses a binary search tree (b-tree) to store entries. Does anyone know of a better system?
For C++, a map works well. Several million objects won't be a problem. 10 million items took about 4.4 seconds and about 57 meg on my computer.
My test application is as follows:
#include <stdio.h>
#include <stdlib.h>
#include <map>
class triple {
public:
int x;
int y;
int z;
bool operator<(const triple &other) const {
if (x < other.x) return true;
if (other.x < x) return false;
if (y < other.y) return true;
if (other.y < y) return false;
return z < other.z;
}
};
int main(int, char**)
{
std::map<triple,int> data;
triple point;
int i;
for (i = 0; i < 10000000; ++i) {
point.x = rand();
point.y = rand();
point.z = rand();
//printf("%d %d %d %d\n", i, point.x, point.y, point.z);
data[point] = i;
}
return 0;
}
Now to dynamically choose the number of variables, the easiest solution is to represent index as a string, and then use string as a key for the map. For instance, an item located at [23][55] can be represented via "23,55" string. We can also extend this solution for higher dimensions; such as for three dimensions an arbitrary index will look like "34,45,56". A simple implementation of this technique is as follows:
std::map data<string,int> data;
char ix[100];
sprintf(ix, "%d,%d", x, y); // 2 vars
data[ix] = i;
sprintf(ix, "%d,%d,%d", x, y, z); // 3 vars
data[ix] = i;
The accepted answer recommends using strings to represent multi-dimensional indices.
However, constructing strings is needlessly wasteful for this. If the size isn’t known at compile time (and thus std::tuple doesn’t work), std::vector works well as an index, both with hash maps and ordered trees. For std::map, this is almost trivial:
#include <vector>
#include <map>
using index_type = std::vector<int>;
template <typename T>
using sparse_array = std::map<index_type, T>;
For std::unordered_map (or similar hash table-based dictionaries) it’s slightly more work, since std::vector does not specialise std::hash:
#include <vector>
#include <unordered_map>
#include <numeric>
using index_type = std::vector<int>;
struct index_hash {
std::size_t operator()(index_type const& i) const noexcept {
// Like boost::hash_combine; there might be some caveats, see
// <https://stackoverflow.com/a/50978188/1968>
auto const hash_combine = [](auto seed, auto x) {
return std::hash<int>()(x) + 0x9e3779b9 + (seed << 6) + (seed >> 2);
};
return std::accumulate(i.begin() + 1, i.end(), i[0], hash_combine);
}
};
template <typename T>
using sparse_array = std::unordered_map<index_type, T, index_hash>;
Either way, the usage is the same:
int main() {
using i = index_type;
auto x = sparse_array<int>();
x[i{1, 2, 3}] = 42;
x[i{4, 3, 2}] = 23;
std::cout << x[i{1, 2, 3}] + x[i{4, 3, 2}] << '\n'; // 65
}
Boost has a templated implementation of BLAS called uBLAS that contains a sparse matrix.
https://www.boost.org/doc/libs/release/libs/numeric/ublas/doc/index.htm
Eigen is a C++ linear algebra library that has an implementation of a sparse matrix. It even supports matrix operations and solvers (LU factorization etc) that are optimized for sparse matrices.
Complete list of solutions can be found in the wikipedia. For convenience, I have quoted relevant sections as follows.
https://en.wikipedia.org/wiki/Sparse_matrix#Dictionary_of_keys_.28DOK.29
Dictionary of keys (DOK)
DOK consists of a dictionary that maps (row, column)-pairs to the
value of the elements. Elements that are missing from the dictionary
are taken to be zero. The format is good for incrementally
constructing a sparse matrix in random order, but poor for iterating
over non-zero values in lexicographical order. One typically
constructs a matrix in this format and then converts to another more
efficient format for processing.[1]
List of lists (LIL)
LIL stores one list per row, with each entry containing the column
index and the value. Typically, these entries are kept sorted by
column index for faster lookup. This is another format good for
incremental matrix construction.[2]
Coordinate list (COO)
COO stores a list of (row, column, value) tuples. Ideally, the entries
are sorted (by row index, then column index) to improve random access
times. This is another format which is good for incremental matrix
construction.[3]
Compressed sparse row (CSR, CRS or Yale format)
The compressed sparse row (CSR) or compressed row storage (CRS) format
represents a matrix M by three (one-dimensional) arrays, that
respectively contain nonzero values, the extents of rows, and column
indices. It is similar to COO, but compresses the row indices, hence
the name. This format allows fast row access and matrix-vector
multiplications (Mx).
Small detail in the index comparison. You need to do a lexicographical compare, otherwise:
a= (1, 2, 1); b= (2, 1, 2);
(a<b) == (b<a) is true, but b!=a
Edit: So the comparison should probably be:
return lhs.x<rhs.x
? true
: lhs.x==rhs.x
? lhs.y<rhs.y
? true
: lhs.y==rhs.y
? lhs.z<rhs.z
: false
: false
Hash tables have a fast insertion and look up. You could write a simple hash function since you know you'd be dealing with only integer pairs as the keys.
The best way to implement sparse matrices is to not to implement them - atleast not on your own. I would suggest to BLAS (which I think is a part of LAPACK) which can handle really huge matrices.
Since only values with [a][b][c]...[w][x][y][z] are of consequence, we only store the indice themselves, not the value 1 which is just about everywhere - always the same + no way to hash it. Noting that the curse of dimensionality is present, suggest go with some established tool NIST or Boost, at least read the sources for that to circumvent needless blunder.
If the work needs to capture the temporal dependence distributions and parametric tendencies of unknown data sets, then a Map or B-Tree with uni-valued root is probably not practical. We can store only the indice themselves, hashed if ordering ( sensibility for presentation ) can subordinate to reduction of time domain at run-time, for all 1 values. Since non-zero values other than one are few, an obvious candidate for those is whatever data-structure you can find readily and understand. If the data set is truly vast-universe sized I suggest some sort of sliding window that manages file / disk / persistent-io yourself, moving portions of the data into scope as need be. ( writing code that you can understand ) If you are under commitment to provide actual solution to a working group, failure to do so leaves you at the mercy of consumer grade operating systems that have the sole goal of taking your lunch away from you.
Here is a relatively simple implementation that should provide a reasonable fast lookup (using a hash table) as well as fast iteration over non-zero elements in a row/column.
// Copyright 2014 Leo Osvald
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef UTIL_IMMUTABLE_SPARSE_MATRIX_HPP_
#define UTIL_IMMUTABLE_SPARSE_MATRIX_HPP_
#include <algorithm>
#include <limits>
#include <map>
#include <type_traits>
#include <unordered_map>
#include <utility>
#include <vector>
// A simple time-efficient implementation of an immutable sparse matrix
// Provides efficient iteration of non-zero elements by rows/cols,
// e.g. to iterate over a range [row_from, row_to) x [col_from, col_to):
// for (int row = row_from; row < row_to; ++row) {
// for (auto col_range = sm.nonzero_col_range(row, col_from, col_to);
// col_range.first != col_range.second; ++col_range.first) {
// int col = *col_range.first;
// // use sm(row, col)
// ...
// }
template<typename T = double, class Coord = int>
class SparseMatrix {
struct PointHasher;
typedef std::map< Coord, std::vector<Coord> > NonZeroList;
typedef std::pair<Coord, Coord> Point;
public:
typedef T ValueType;
typedef Coord CoordType;
typedef typename NonZeroList::mapped_type::const_iterator CoordIter;
typedef std::pair<CoordIter, CoordIter> CoordIterRange;
SparseMatrix() = default;
// Reads a matrix stored in MatrixMarket-like format, i.e.:
// <num_rows> <num_cols> <num_entries>
// <row_1> <col_1> <val_1>
// ...
// Note: the header (lines starting with '%' are ignored).
template<class InputStream, size_t max_line_length = 1024>
void Init(InputStream& is) {
rows_.clear(), cols_.clear();
values_.clear();
// skip the header (lines beginning with '%', if any)
decltype(is.tellg()) offset = 0;
for (char buf[max_line_length + 1];
is.getline(buf, sizeof(buf)) && buf[0] == '%'; )
offset = is.tellg();
is.seekg(offset);
size_t n;
is >> row_count_ >> col_count_ >> n;
values_.reserve(n);
while (n--) {
Coord row, col;
typename std::remove_cv<T>::type val;
is >> row >> col >> val;
values_[Point(--row, --col)] = val;
rows_[col].push_back(row);
cols_[row].push_back(col);
}
SortAndShrink(rows_);
SortAndShrink(cols_);
}
const T& operator()(const Coord& row, const Coord& col) const {
static const T kZero = T();
auto it = values_.find(Point(row, col));
if (it != values_.end())
return it->second;
return kZero;
}
CoordIterRange
nonzero_col_range(Coord row, Coord col_from, Coord col_to) const {
CoordIterRange r;
GetRange(cols_, row, col_from, col_to, &r);
return r;
}
CoordIterRange
nonzero_row_range(Coord col, Coord row_from, Coord row_to) const {
CoordIterRange r;
GetRange(rows_, col, row_from, row_to, &r);
return r;
}
Coord row_count() const { return row_count_; }
Coord col_count() const { return col_count_; }
size_t nonzero_count() const { return values_.size(); }
size_t element_count() const { return size_t(row_count_) * col_count_; }
private:
typedef std::unordered_map<Point,
typename std::remove_cv<T>::type,
PointHasher> ValueMap;
struct PointHasher {
size_t operator()(const Point& p) const {
return p.first << (std::numeric_limits<Coord>::digits >> 1) ^ p.second;
}
};
static void SortAndShrink(NonZeroList& list) {
for (auto& it : list) {
auto& indices = it.second;
indices.shrink_to_fit();
std::sort(indices.begin(), indices.end());
}
// insert a sentinel vector to handle the case of all zeroes
if (list.empty())
list.emplace(Coord(), std::vector<Coord>(Coord()));
}
static void GetRange(const NonZeroList& list, Coord i, Coord from, Coord to,
CoordIterRange* r) {
auto lr = list.equal_range(i);
if (lr.first == lr.second) {
r->first = r->second = list.begin()->second.end();
return;
}
auto begin = lr.first->second.begin(), end = lr.first->second.end();
r->first = lower_bound(begin, end, from);
r->second = lower_bound(r->first, end, to);
}
ValueMap values_;
NonZeroList rows_, cols_;
Coord row_count_, col_count_;
};
#endif /* UTIL_IMMUTABLE_SPARSE_MATRIX_HPP_ */
For simplicity, it's immutable, but you can can make it mutable; be sure to change std::vector to std::set if you want a reasonable efficient "insertions" (changing a zero to a non-zero).
I would suggest doing something like:
typedef std::tuple<int, int, int> coord_t;
typedef boost::hash<coord_t> coord_hash_t;
typedef std::unordered_map<coord_hash_t, int, c_hash_t> sparse_array_t;
sparse_array_t the_data;
the_data[ { x, y, z } ] = 1; /* list-initialization is cool */
for( const auto& element : the_data ) {
int xx, yy, zz, val;
std::tie( std::tie( xx, yy, zz ), val ) = element;
/* ... */
}
To help keep your data sparse, you might want to write a subclass of unorderd_map, whose iterators automatically skip over (and erase) any items with a value of 0.