I want to use Eigen3 on data coming from another library. An earlier answer by ggael indicates a way for Eigen::Matrix to adopt preexsisting data with the new keyword. However, this is not sufficient for me, because the resulting Matrix still seems to acquire ownership of the data, meaning that it will free the data when going out of scope. To wit, this is a crasher if data eventually gets deleted by the library it comes from:
void crasher(double* data, size_t dim)
{
MatrixXd m;
new (&m) Map<MatrixXd>(data,dim,dim); // matrix adopts the passed data
m.setRandom(); cout<<m<<endl; // manipulation of the passed data via the Matrix interface
} // data deleted by m => potential problem in scope of function call
I have come up with two workarounds:
void nonCrasher1(double* data, size_t dim)
{
MatrixXd m; // semantically, a non-owning matrix
const Map<const MatrixXd> cache(m.data(),0,0); // cache the original „data” (in a const-correct way)
new (&m) Map<MatrixXd>(data,dim,dim); // matrix adopts the passed data
m.setRandom(); cout<<m<<endl; // manipulation of the passed data via the Matrix interface
new (&m) Map<const MatrixXd>(cache); // re-adopting the original data
} // original data deleted by m
This is rather inconvenient because of the presence of cache. The other is free from this problem:
void nonCrasher2(double* data, size_t dim) // no need for caching
{
MatrixXd m; // semantically, a non-owning matrix
new (&m) Map<MatrixXd>(data,dim,dim); // matrix adopts the passed data
m.setRandom(); cout<<m<<endl; // manipulation of the passed data via the Matrix interface
new (&m) Map<MatrixXd>(nullptr,0,0); // adopting nullptr for subsequent deletion
} // nullptr „deleted” by m (what happens with original data [if any]?)
However, here it’s unclear what happens with the original data of m (if any – all this is not completely clear from the documentation of Eigen3).
My question is whether there is a canonical way for Eigen::Matrix to release ownership of it’s data (self-allocated or adopted).
In nonCrasher2 nothing happen to data, except that it has been filled with random values on purpose.
However, this still look like hackish, and the clean approach be to use a Map object instead of MatrixXd:
Map<MatrixXd> m(data, dim, dim);
m.setRandom();
If you need m to be a MatrixXd because you need to call functions taking MatrixXd objects, and such functions cannot be templated, then you might consider generalizing these functions to take Ref<MatrixXd> objects. By default, a Ref<MatrixXd> can accept any expression which storage reassemble to a MatrixXd with arbitrary leading dimension. It has been introduced in Eigen 3.2, check the doc for more details.
Related
I have a quite complex Problem and couldn't find out, where the Problem is. Part of the Problem might be, that i didn't fully understand the data structures. I have the following custom template class, with this member variables:
template <class T>
class Matrix
{
private:
T * *M; /* array of row pointers */
size_t n_rows, /* number of rows */
n_cols; /* number of columns */
size_t row_start, /* first row of submatrix */
col_start, /* first column of submatrix */
origcols; /* refers to original matrix */
bool hasownvalues, /* has matrix own value? */
initialized; /* are matrix values initialed? */
I also made a custom constructor, which makes a 2D array out of a file with tab-separated data:
Matrix(const std::string &filename)
{ do.stuff }
I'm not sure, if it is really important how this exactly works, it quite a bunch of code. if so, i could edit it later, but the constructor works well as long as i declare a single variable for such an object.
My problem occurs when i try to make a bunch of the objects and try to wrap them up in a vector. Lets say i have x different files, each containing data for a matrix. I also have a vector in which the filenames are stored:
std::vector<std::string> filenames{};
for (int idx = 3; idx < argc; idx++)
{
filenames.push_back(argv[idx]);
}
So far this code works and it is mandatory that i keep it unchanged to keep the rest of the program working.
The following is flexible and thats where i have problem:
std::vector<Matrix<T>> coredata; /* initialize the vector for the wrap */
...
for (auto& it : filenames)
{
Matrix<T> M(it); /* call the special constructor*/
coredata.push_back(M); /* add new object to the vector */
}
The first assignment works well, but from the second on it seems that it appends the new Matrix, but also overwrites the old one or at least a part of it. And in the third one i get a segmentation fault when running the program.
Some information of the program around:
the constructor works with space allocated by the "new" command, maybe that could be part of the problem.
the real size of the Matrix object isn't known before run time. (the arrays can be of different size), but i could arrange, that the read data in fact always is of the same size
T is constant within the vector. it can't happen, that M is sometimes Matrix int and sometimes Matrix float
the vector would be embedded in another superclass and the snippet above is part of a constructor of this class, coredata is a member variable.
I'm quite puzzled what the problem exactly is. Maybe some variables don't live long enough. I also thought of assigning just a reference to the address in the vector, but as far as i understand, a vector already stores just a reference, so it seems not that plausible.
I also thought of initializing the vector with a bunch of "0"-matrices and then overwrite them with the constructed objects. But i don't know how to do that and on top of that,
i don't know at compile time how many rows and columns the read Matrices will have.
Maybe a vector isn't a useful solution at all, i also thought of making a plain C-array.
But there i have the problem again, that i have to initialize this array first and can't just append another Matrix-Object to it.
I have a function called DataView that performs the "slicing" (creation of the array_view) as follows:
template <class T>
typename Boost<T>::Array2D::template array_view<2>::type MyArray<T>::DataView(int x_start, int x_finish, int y_start, int y_finish)
{
using range = boost::multi_array_types::index_range;
return _Data[boost::indices[range().start(x_start).finish(x_finish).stride(1)][range().start(y_start).finish(y_finish).stride(1)]];
}
...where _Data is a private member of type Array2D (boost::multi_array) containing the data portion of MyArray. The slicing (view creation) is happening without a problem as follows:
Boost<double>::Array2D::array_view<2>::type array_view = my_array->DataView(x1, x2, y1, y2);
The problem occurs when I want to assign the newly acquired view (array_view) back to the original multi_array (my_array). I have tried to do this by creating this function on the class MyArray:
template <class T>
void MyArray<T>::Data(const typename Boost<T>::Array2D &inData)
{
// find the dimensions of inData
int xdim = static_cast<int>(inData.shape()[0]);
int ydim = static_cast<int>(inData.shape()[1]);
// resize _Data to match the size of inData
_Data.resize(boost::extents[xdim][ydim]);
// assign _Data to the new set of data
_Data = inData;
}
...and calling the function with this line of code...
my_array->Data(array_view);
While the build is successful, my application is producing the following error:
base.hpp:178: Reference boost::detail::multi_array::value_accessor_one<T>::access(boost::type<Reference>, boost::detail::multi_array::value_accessor_one<T>::index, TPtr, const size_type*, const index*, const index*) const [with Reference = double&; TPtr = double*; T = double; boost::detail::multi_array::value_accessor_one<T>::index = long int; boost::detail::multi_array::multi_array_base::size_type = long unsigned int]: Assertion `size_type(idx - index_bases[0]) < extents[0]' failed.
What I really need is a simple example of how I can do one of the following:
Use my current strategy of creating an array_view from my original multi_array and assigning the new view back to the original array, OR
Slicing the original multi_array (in place). I've looked for an example of this solution and cannot seem to find it.
Thank you.
This turned out to be something unrelated within my program. I had other variables that were dependent on the dimensions of my array (which had changed due to the reassignment to the view) that I failed to set.
The answer to your question "assign a view back to the same array?" is simple: "don't do that".
There are two reasons for that, one trivial and the other one is more interesting.
The trivial one is that most subviews of an array are of a different size as the original array.
So when a subview is assigned to an array, Boost.MultiArray will reject the assignment at runtime, simply because the library doesn't support assignment from-to arrays of different sizes (shape).
(see runtime error message here: https://godbolt.org/z/ffPo3e3hj)
So, to start, you need to use an array library that accepts assignment of arrays of arbitrary size.
And even if you could and more interestingly, array data structures and algorithms are subject to something called "aliasing" for efficiency, which means that in general you cannot use two things that share memory on the LHS and the RHS of an equation or assignment.
What you need to do is to make a copy first and then (resize and) assign back to the original array.
This can be done with Boost.MultiArray but it is messy because of the trivial limitation I mention above.
Incidentally, I have a library similar to Boost.MultiArray, in which I use unary plus to break aliasing by making a copy.
multi::array<double, 2> A({3, 3});
A = + A({0, 2}, {0, 2}) ; // make an independent copy of the view first, the assign to the original array
https://godbolt.org/z/ahExP787T
I know these conversion topics have been raised multiple times but I could not find one considering move semantics.
I am wondering if the following is safe:
Eigen::MatrixXd A(n,n);
//fill matrix A
std::vector<double> A_vec;
A_vec.insert(A_vec.end(), std::make_move_iterator(A.data()), std::make_move_iterator(A.data() + A.size()));
//destruct A
Assuming, you actually want to do this for a non-trivial scalar (instead of double) which has move-semantics, this should be safe -- of course, as long as you don't use A afterwards. For double it is safe as well, but it makes no difference.
In your case it may actually be better to map the data of a std::vector to an Eigen::Map, something like this:
std::vector<double> A_vec(n*n); // allocate data into A_vec
Eigen::Map<Eigen::MatrixXd> A(A_vec.data(), n, n);
// fill matrix A.
// data is immediately stored into A_vec
I want to know whether I need to free up the memory occupied by locations and values objects after sparse matrix has been created. Here is the code:
void load_data(umat &locations, vec& values){
// fill up locations and values
}
int main(int argc, char**argv){
umat loc;
vec val;
load_data(loc,val);
sp_mat X(loc,val);
return 0;
}
In the above code, load_data() fills up the locations and values objects and then sparse matrix is created in main(). My Q: Do I need to free up memory used by locations and values after construction of X? The reason is that X could be large and I am on low RAM. I know when main returns, OS will free up locations and values as well as X. But, the real question is whether the memory occupied by X is the same as by locationsand values OR X is allocated memory separately and I need to free locations and values in the case.
The constructor (SpMat_meat.hpp:231) you are using
template<typename T1, typename T2>
inline SpMat(const Base<uword,T1>& locations, const Base<eT,T2>& values, const bool sort_locations = true);
fills the sparse matrix with copies of the values in values.
I understand you are worried that you will run out of memory and if you keep loc, val and X separately you basically have two copies of the same data, taking up twice as much memory as actually needed (this is indeed what happens in your code snippet), so I will try to focus on addressing this problem and give you a few options:
1) If you are fine with keeping two copies of the data for a short while, the easiest solution is to dynamically allocate loc and val and delete them right after initialization of X
int main(int argc, char**argv){
umat* ploc;
vec* pval;
load_data(*ploc,*pval);
// at this point we have one copy of the data
sp_mat X(*ploc,*pval);
// at this point we have two copies of the data
delete ploc;
delete pval;
// at this point we have one copy of the data
return 0;
}
Of course you can use safe pointers instead of C style ones, but you get the idea.
2) If you absolutely don't want to have two copies of the data at any time, I would suggest that you modify your load_data routine to sequentially load values one by one and directly insert them into X
void load_data(umat &locations, vec& values, sp_mat& X){
// fill up locations and values one by one into X
}
Other options would be to i) use move semantics to directly move the values in val into X or ii) directly use the memory allocated for val as the memory for X, similar to the advanced constructor for matrices
Mat(eT* aux_mem, const uword aux_n_rows, const uword aux_n_cols, const bool copy_aux_mem = true, const bool strict = false)
Both options would however require modifications on the level of the armadillo library, as such a functionality is not supplied yet for sparse matrices (There is only a plain move constructor so far). It would be a good idea to request these features from the developers though!
I'm trying to interface with software that just uses raw pointers for holding data.
I feel pretty confident that I can avoid the issues with using shallow_array_adaptor
if I make the wrapping vector instances const even though the documentation for shallow_array_adaptor claims that it is "very dangerous!".
I'd like to do things like:
using namespace boost::numeric::ublas
void f(vector<double>& acc, double* arr){
// assume acc is already correctly sized, and arr is correctly allocated
// this syntax is not correct, I'm pretty sure I'll have to construct
// the instance of shallow_array_adaptor, but don't know how to
// get it to the vector's constructor
const vector< double , shallow_array_adaptor< double > > view( arr );
acc+=view; // by wrapping the raw pointer, I get to use nice syntax here.
}
without having to copy the values stored in arr.
I'm not sure what I have to do to construct the vector such that it uses
the given pointer as it's storage.