I am using pybind11 to convert scipy's csr_matrix into a C++ object through pybind11. To this end, I've defined a Vector class, which is converted from the numpy data, indices, and indptr fields.
template<typename T>
class Vector {
public:
T *data;
const std::array<ssize_t, 1> shape;
Vector() = delete;
Vector(T *data, const std::array<ssize_t, 1> shape) : data(data), shape(shape) {};
};
I later register several instantiations of the class using a template function which does this.
py::class_<Vector<T>>(m, "...Vector", py::buffer_protocol())
.def("__init__", [](Vector<T> &v, py::array_t<T, py::array::c_style | py::array::forcecast> data) {
py::buffer_info info = data.request();
if (info.ndim != 1) throw std::invalid_argument("must be a 1d array!");
std::array<ssize_t, 1> shape_ = {info.shape[0]};
new(&v) Vector<T>(static_cast<T *>(info.ptr), shape_);
})
.def_buffer([](Vector<T> &v) -> py::buffer_info {
return py::buffer_info(
v.data, sizeof(T), py::format_descriptor<T>::format(), 1, v.shape, {sizeof(T)}
);
});
So the vector class works fine, however, I then define a csr_matrix class like so
template<typename T>
class csr_matrix {
public:
Vector<T> data;
Vector<ssize_t> indices;
Vector<ssize_t> indptr;
const std::array<ssize_t, 2> shape;
csr_matrix() = delete;
csr_matrix(Vector<T>& data, Vector<ssize_t>& indices, Vector<ssize_t>& indptr, const std::array<ssize_t, 2>& shape)
: data(data), indices(indices), indptr(indptr), shape(shape) {}
};
which is then registered in the same way as Vector using templates, so I can register csr_matrices for floats, doubles, and ints.
py::class_<csr_matrix<T>>(m, "...csr_matrix"))
.def("__init__", [](
csr_matrix<T> &matrix,
py::array_t<T, py::array::c_style> data,
py::array_t<ssize_t, py::array::c_style> indices,
py::array_t<ssize_t, py::array::c_style> indptr,
py::array_t<ssize_t, py::array::c_style | py::array::forcecast> shape
) {
py::buffer_info data_info = data.request();
py::buffer_info indices_info = indices.request();
py::buffer_info indptr_info = indptr.request();
// ... some validity checks
auto vec_data = new Vector<T>(static_cast<T *>(data_info.ptr), {data_info.shape[0]});
auto vec_indices = new Vector<ssize_t>(static_cast<ssize_t *>(indices_info.ptr), {indices_info.shape[0]});
auto vec_indptr = new Vector<ssize_t>(static_cast<ssize_t *>(indptr_info.ptr), {indptr_info.shape[0]});
std::array<ssize_t, 2> shape_ = {*shape.data(0), *shape.data(1)};
new(&matrix) csr_matrix<T>(*vec_data, *vec_indices, *vec_indptr, shape_);
})
.def_readonly("data", &csr_matrix<T>::data)
.def_readonly("indices", &csr_matrix<T>::indices)
.def_readonly("indptr", &csr_matrix<T>::indptr);
Now, I write a simple unit test in Python to make sure everything is working properly and I get the most baffling error
x = sp.csr_matrix([
[0, 0, 1, 2],
[0, 0, 0, 0],
[3, 0, 4, 0],
], dtype=np.float32)
cx = matrix.Float32CSRMatrix(x.data, x.indices, x.indptr, x.shape)
np.testing.assert_equal(x.data, np.asarray(cx.data, dtype=np.float32))
np.testing.assert_equal(x.indices, np.asarray(cx.indices, dtype=np.uint64))
np.testing.assert_equal(x.indptr, np.asarray(cx.indptr, dtype=np.uint64)) # fails here!
I throw a couple of prints after each line and this is the output
print(1, np.asarray(cx.data, dtype=np.float32), np.asarray(cx.indices, dtype=np.uint64), np.asarray(cx.indptr, dtype=np.uint64))
# data indices indptr
- [1. 2. 3. 4.] [2 3 0 2] [0 2 2 4] # original, python object
0 [1. 2. 3. 4.] [2 3 0 2] [0 2 2 4] # after matrix.Float32CSRMatrix(...)
1 [1. 2. 3. 4.] [2 3 0 2] [0 2 2 4] # after assert on data
2 [1. 2. 3. 4.] [2 3 0 2] [2 3 0 2] # after assert on indices
# Fails at assert on indptr!
So somewhere, something changed the values of indptr into indices, and I have no clue what and where. What's even more baffling is that if I change the order of the asserts so that indptr is checked before indices, like this
np.testing.assert_equal(x.data, np.asarray(cx.data, dtype=np.float32))
np.testing.assert_equal(x.indptr, np.asarray(cx.indptr, dtype=np.uint64))
np.testing.assert_equal(x.indices, np.asarray(cx.indices, dtype=np.uint64)) # fails here!
then this is the output
- [1. 2. 3. 4.] [2 3 0 2] [0 2 2 4]
0 [1. 2. 3. 4.] [2 3 0 2] [0 2 2 4]
1 [1. 2. 3. 4.] [2 3 0 2] [0 2 2 4]
2 [1. 2. 3. 4.] [0 2 2 4] [0 2 2 4]
# Now fails at assert indices; we do the assert on indptr before, and it passes
So now, it's indices that are being overridden with indptr, and not the other way around. I've been banging my head against this wall for well over a day now and I have no clue what's going on. Object lifetime is not an issue, vectors are constructed at the start and destructed when the csr_matrix goes away. And I have pasted here all the relevant code, and I am not doing anything that might cause this.
Any and all help would be greatly appreciated.
Related
Please explain the following behavior
import sys
input=sys.stdin.readline
for _ in range(int(input())):
n=int(input())
m=int(input())
vertices=[[0, 0]]*n
edges=[[0, 0, 0]]*m
for i in range(m):
edges[i][0], edges[i][1], edges[i][2]=list(map(int, input().split()))
for i in range(m):
print(edges[i][0], edges[i][1], edges[i][2])
When the following input is fed into the program:
1
5
7
1 2 1
2 3 2
2 4 6
5 2 1
5 1 3
4 5 2
3 4 3
This output is being produced:
3 4 3
3 4 3
3 4 3
3 4 3
3 4 3
3 4 3
3 4 3
Why is only the last inputs are being stored in all of the elements?
Let's look at this interpreter session:
>>> a = [[0,0,0]] * 5
>>> a
[[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> a[0][0] = 1
>>> a
[[1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0]]
This is actually a common pitfall in Python. The code translates to something like this
inner = [0, 0, 0]
a = [inner, inner, inner, inner, inner]
This way of writing makes it clearer that you actually create a list where all entries point to the same object (inner).
One way to fix this is to use a list comprehension, this way a new inner list will be created for each list entry:
vertices=[[0, 0] for _ in range(n)]
edges=[[0, 0, 0] for _ in range(m)]
Another fix would be using numpy and declaring a 2D matrix using numpy.zeros(n, 2). Numpy is generally useful for math/numerics-intensive applications.
I have a 2D binary array where the value can take 0 and 1 only.
I have a set of indices to check whether the entries of the binary matrix for those indices are duplicate or not. I want to get the matrix with duplicate rows removed and the set of duplicate indices.
For example,
>>>> a
array([[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
])
I am given set of indices (0,2,3,4,6,7). From the set, the rows corresponding to (0,2) and (4,6,7) are duplicates. I want the resulting matrix with the duplicates removed (as shown below)
>>>> b
array([[1, 0, 1, 0],
[0, 0, 1, 1],
[0, 0, 1, 1],
[1, 1, 1, 0],
[1, 1, 1, 0],
])
and a method for reconstruction of the matrix 'a' from 'b'
If the order in the output array is not relevant, then you can probably just use Eelco Hoogendoorn's answer. However, if you want to keep the same relative order as in the original array, here is another possible approach.
import numpy as np
a = np.array([
[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
])
idx = np.array([0, 2, 3, 4, 6, 7])
# Make an array of row numbers
r = np.arange(len(a))
# Replace row numbers in idx with -1
# (use assume_unique only if indices in idx are unique)
r[np.isin(r, idx, assume_unique=True)] = -1
# Add the column to the array
a2 = np.concatenate([a, r[:, np.newaxis]], axis=-1)
# Find unique indices and inverse indices
_, uniq_idx, inv_idx = np.unique(a2, return_index=True, return_inverse=True, axis=0)
# Sort indices to make output array and inverse indices
s = np.argsort(uniq_idx)
a_uniq = a[uniq_idx[s]]
inv_idx = s[inv_idx]
print(a_uniq)
# [[1 0 1 0]
# [0 0 1 1]
# [0 0 1 1]
# [1 1 1 0]
# [1 1 1 0]]
print(np.all(a_uniq[inv_idx] == a))
# True
EDIT: Some further explanation.
The idea in the solution above is to apply np.unique, but in a way that the rows that are not included in idx are not affected by it. In order to do that, you can just add a new number to each row. For the rows included in idx, this number will always be -1, and for the rest of rows it will be a different number for each. That way, it is impossible that rows that are not in idx get removed by np.unique. In order to do that, I build r, first with np.arange(len(a)), which gives you a number per row:
[0 1 2 3 4 5 6 7]
Then I check which of those are in idx with np.isin(r, idx, assume_unique=True) (assume_unique can only be used if elements in idx are guaranteed to be unique), so r[np.isin(r, idx, assume_unique=True)] = -1 will turn all indices idx into -1:
[-1 1 -1 -1 -1 5 -1 -1]
That is added as new column to a into a2:
[[ 1 0 1 0 -1]
[ 0 0 1 1 1]
[ 1 0 1 0 -1]
[ 0 0 1 1 -1]
[ 1 1 1 0 -1]
[ 1 1 1 0 5]
[ 1 1 1 0 -1]
[ 1 1 1 0 -1]]
Now it's just a matter of applying np.unique to a2. As expected, only rows in idx may be eliminated. However, since we want to keep the original relative order, we cannot use the output of np.unique, because it is sorted. We use return_index and return_inverse to get the indices that make the array of unique rows and the indices that get you back to the original array, and actually discard the new array.
To form the final array, you need to sort uniq_idx to keep the relative order, and then inv_idx accordingly. np.argsort gives you the indices that sort uniq_idx into s. uniq_idx[s] is just the array of unique row indices sorted, and s[inv_idx] will map every inverse index in inv_idx to the corresponding one in the resorted array. So, finally, a[uniq_idx[s]] gives you the output array, and the new inv_idx takes you back to the original one.
It feels like you could phrase your question at a higher level to get a more elegant solution; but this seems to solve the literal problem as stated.
idx = [0,2,3,4,6,7]
b = np.concatenate([np.unique(a[idx], axis=0), np.delete(a, idx, axis=0)], axis=0)
Say I have
A = [1 2 3]
[4 5 6]
[7 8 9]
I want to pad it with the first row and first column or last row and last column as many times as needed to create A nxn. For example, A 4x4 would be
A = [1 1 2 3]
[1 1 2 3]
[4 4 5 6]
[7 7 8 9]
and A 5x5 would be
A = [1 1 2 3 3]
[1 1 2 3 3]
[4 4 5 6 6]
[7 7 8 9 9]
[7 7 8 9 9]
I'm aware that I could do A.conservativeResize(4,4) which gets me
A = [1 2 3 0]
[4 5 6 0]
[7 8 9 0]
[0 0 0 0]
then I could copy things around one by one, but is there a more efficient way to do this using Eigen?
You can workaround using a nullary-expression:
#include <iostream>
#include <Eigen/Dense>
using namespace Eigen;
using namespace std;
int main()
{
Matrix3i A;
A.reshaped() = VectorXi::LinSpaced(9,1,9);
cout << A << "\n\n";
int N = 5;
MatrixXi B(N,N);
B = MatrixXi::NullaryExpr(N, N, [&A,N] (Index i,Index j) {
return A( std::max<Index>(0,i-(N-A.rows())),
std::max<Index>(0,j-(N-A.cols())) ); } );
cout << B << "\n\n";
}
Another approach would be to create a clamped sequence of indices like [0 0 0 1 2]:
struct pad {
Index size() const { return m_out_size; }
Index operator[] (Index i) const { return std::max<Index>(0,i-(m_out_size-m_in_size)); }
Index m_in_size, m_out_size;
};
B = A(pad{3,N}, pad{3,N});
This version requires the head of Eigen.
You can easily build on those examples to make them even more general and/or wrap them within functions.
Just as a note, it's not true that A.conservativeResize(4,4) will get you a matrix with the added rows filled with zeros. The Eigen documentation says,
In case values need to be appended to the matrix they will be uninitialized.
The new rows and columns will be filled with garbage, and seeing zeros is only a coincidence (unless you are compiling with a special preprocessor directive to Eigen). But this means that no unnecessary time is wasted writing zeros that you will overwrite anyway.
Note: this code demonstrates how to get a matrix with your original matrix in the top left corner:
The best way to fill multiple values at once is to use Eigen's block operations and setConstant. For example, if A is a matrix of size old_sizexold_size:
A.conservativeResize(n, n);
for (int i = 0; i < n; ++i) {
// Fill the end of each row and column
A.row(i).tail(n - old_size).setConstant(A(i, old_size - 1));
A.col(i).tail(n - old_size).setConstant(A(old_size - 1, i));
}
// Fill the bottom right block
A.bottomRightCorner(n - old_size, n - old_size).setConstant(A(old_size - 1, old_size - 1));
More importantly than being "efficient", these functions express your intent as a programmer.
Edit: To get a padded matrix with your original matrix in the middle:
I just noticed your example pads around the original matrix in the middle, not in the top left. In this case, there is little point to using conservativeResize(), because the original values will only be copied to the top left corner. An outline of the solution is:
Construct a new nxn matrix B of the desired size
Copy your original matrix to the middle using
int start = (n - old_size + 1)/2;
B.block(start, start, old_size, old_size) = A;
Fill in the outside values using block operations similar to my example above.
I am working on a problem that requires iterating over all combinations of elements of K vectors taken one at a time. So for example for K=2 vectors v1 = [0 1] and v2 = [3 4], I would iterate over (0,3), (0,4), (1,3), (1,4).
Since K is determined at run-time, I cannot use explicit for loops. My current approach is based on this solution that implements an "odometer" incrementing an index for each vector.
#include <vector>
#include <iostream>
int main(int argc, char * argv[])
{
std::vector<int> v1( {1, 2, 3} );
std::vector<int> v2( {-2, 5} );
std::vector<int> v3( {0, 1, 2} );
std::vector<std::vector<int> > vv( {v1, v2 ,v3} );
// Iterate combinations of elems in v1, v2, v3, one at a time
std::vector<std::vector<int>::iterator> vit;
for (auto& v : vv)
vit.push_back(v.begin());
int K = vv.size();
while (vit[0] != vv[0].end())
{
std::cout << "Processing combination: [";
for (auto& i : vit)
std::cout << *i << " ";
std::cout << "]\n";
// increment "odometer" by 1
++vit[K-1];
for (int i = K-1; (i > 0) && (vit[i] == vv[i].end()); --i)
{
vit[i] = vv[i].begin();
++vit[i-1];
}
}
return 0;
}
Output:
Processing combination: [1 -2 0 ]
Processing combination: [1 -2 1 ]
Processing combination: [1 -2 2 ]
Processing combination: [1 5 0 ]
Processing combination: [1 5 1 ]
Processing combination: [1 5 2 ]
Processing combination: [2 -2 0 ]
Processing combination: [2 -2 1 ]
Processing combination: [2 -2 2 ]
Processing combination: [2 5 0 ]
Processing combination: [2 5 1 ]
Processing combination: [2 5 2 ]
Processing combination: [3 -2 0 ]
Processing combination: [3 -2 1 ]
Processing combination: [3 -2 2 ]
Processing combination: [3 5 0 ]
Processing combination: [3 5 1 ]
Processing combination: [3 5 2 ]
However, this is somewhat messy and requires a lot of boilerplate code that I'd rather move elsewhere for clarity. Ideally I would like to have a custom iterator class, say my_combination_iterator, that would allow me to do things much cleaner, e.g.:
for (my_combination_iterator it = vv.begin(); it != vv.end(); ++it)
// process combination
So far, I have looked at Boost iterator_facade. But my case seems more complicated than the one in the tutorial since I would need an iterator over a vector of Values as opposed to a single value type to define the required operators for the custom iterator.
How could such an iterator be implemented?
Why would you like to use custom iterators?
One could instead implement a very simple class that will iterate through all combinations:
class Combinator
{
public:
Combinator(std::vector<std::vector<int> >& vectors)
: m_vectors(vectors)
{
m_combination.reserve(m_vectors.size());
for(auto& v : m_vectors)
m_combination.push_back(v.begin());
}
bool next()
{
// iterate through vectors in reverse order
for(long long i = m_vectors.size() - 1; i >= 0; --i)
{
std::vector<int>& v = m_vectors[i];
std::vector<int>::iterator& it = m_combination[i];
if(++it != v.end())
return true;
it = v.begin();
}
return false;
}
std::vector<std::vector<int>::iterator> combination() const
{
return m_combination;
}
private:
std::vector<std::vector<int> >& m_vectors; // reference to data
std::vector<std::vector<int>::iterator> m_combination;
};
Live Demo
UPDATE:
If you would still like to use iterators, I suggest iterating over combinations. One can put all the combinations from Combinator into a container and then work with container's own iterators. In my opinion it's a cleaner solution. The only drawback is the extra-memory needed to store all combinations explicitly.
I wrote a simple bubble sorting algorithm, however it is returning strange results when ran...
I am attempting to sort parallel vectors (not sure if that is correct terminology, I'm taking the concept of parallel arrays and applying it to vectors so I can modify the size at runtime) that make up a contact list.
The code pertaining to the vectors and bubble sort:
#include <cstdlib>
#include <iostream>
#include <iomanip>
#include <string>
#include <vector>
using namespace std;
int main(int argc, char** argv) {
vector <long long int> numbers;
vector <string> names;
names.push_back("jack");
numbers.push_back(6515551234);
names.push_back("jill");
numbers.push_back(6515554321);
names.push_back("bob");
numbers.push_back(6515557777);
names.push_back("aaron");
numbers.push_back(6665559999);
names.push_back("fred");
numbers.push_back(1115552222);
int index = -1;
long int temp = -1;
int pass = -1;
string tempName;
int sortChoice;
for (pass = 0; pass < names.size(); pass++) {
for (index = 0; index < names.size() - pass; index++) {
if ( names[index] > names[index + 1] ) {
temp = numbers[index];
numbers[index] = numbers[index + 1];
numbers[index + 1] = temp;
tempName = names[index];
names[index] = names[index + 1];
names[index + 1] = tempName;
}
}
}
for(int i = 0; i < numbers.size(); i++) {
cout << names[i] << ": " << numbers[i] << endl;
}
return 0;
}
And the output:
: 465675315720
aaron: 6665559999
bob: 6515557777
fred: 1115552222
jack: 6515551234
And it seems that no matter how I change the vectors themselves, the last one is always cut off, and put at the beginning with no name and a (seemingly) random number. Has anyone seen this before?
The only thing I can think of is that while the sort is processing, something goes awry in memory, causing pointing to incorrect memory locations?... And the fact that it is always the last element makes me think the issue pertains to the last iteration of the loop.
Also, I made a version of this to sort by numbers rather than names, and it works perfectly every time. It's the same code but using numbers[index] in place of names[index] in the algorithm loop.
P.S. - I'm still rather new to StackOverflow and coding in general, so if I made any mistakes in formatting this post let me know and I will change them. And I hope I provided enough information.
Thank you in advance!
for each loop, you should compare each neighborhood, from the first to the end.
your error:
for (index = 0; index < names.size() - pass; index++) {
should be
for (index = 0; index < names.size() - 1; index++) {
here is a sample
start: 5 4 3 2 1
loop: 0
index: 0 : [5 4] 3 2 1 => [4 5] 3 2 1
index: 1 :4 [5 3] 2 1 => 4 [3 5] 2 1
index: 2 :4 3 [5 2] 1 => 4 3 [2 5] 1
index: 3 :4 3 2 [5 1] => 4 3 2 [1 5]
loop: 1
index: 0 : [4 3] 2 1 5 => [3 4] 2 1 5
index: 1 :3 [4 2] 1 5 => 3 [2 4] 1 5
index: 2 :3 2 [4 1] 5 => 3 2 [1 4] 5
index: 3 :3 2 1 [4 5] => 3 2 1 [4 5]
loop: 2
index: 0 : [3 2] 1 4 5 => [2 3] 1 4 5
index: 1 :2 [3 1] 4 5 => 2 [1 3] 4 5
index: 2 :2 1 [3 4] 5 => 2 1 [3 4] 5
index: 3 :2 1 3 [4 5] => 2 1 3 [4 5]
loop: 3
index: 0 : [2 1] 3 4 5 => [1 2] 3 4 5
index: 1 :1 [2 3] 4 5 => 1 [2 3] 4 5
index: 2 :1 2 [3 4] 5 => 1 2 [3 4] 5
index: 3 :1 2 3 [4 5] => 1 2 3 [4 5]
loop: 4
index: 0 : [1 2] 3 4 5 => [1 2] 3 4 5
index: 1 :1 [2 3] 4 5 => 1 [2 3] 4 5
index: 2 :1 2 [3 4] 5 => 1 2 [3 4] 5
index: 3 :1 2 3 [4 5] => 1 2 3 [4 5]
final result: 1 2 3 4 5
the code that you have written above is not bubble sort !
however , try this instead , this is bubble sort:
for (pass = 0; pass < names.size(); pass++) {
for (index = pass+1; index < names.size(); index++) {
if ( names[pass] > names[index] ) {
temp = numbers[index];
numbers[index] = numbers[pass];
numbers[pass] = temp;
tempName = names[index];
names[index] = names[pass];
names[pass] = tempName;
}
}
}
i offer you use a pair instead of two vectors