Eigen find set difference between two matrices - c++

Question

I'd like to obtain the set difference between two Eigen matrices. The code:
void diffMatrix(
MatrixXi &M1, // First Matrix
MatrixXi &M2, // Second Matrix
MatrixXi &M3, // Matrix set difference
VectorXi &I3 // Matrix set difference indices
)
{
// find rows in first matrix that aren't in second matrix
// cols of M1 = M2
assert(M1.cols() == M2.cols());
M3.resize(M1.rows(), M1.cols());
I3.resize(M1.rows());
bool m2r_nonex;
size_t k = 0;
// get M1 rows
for (size_t i = 0; i < M1.rows(); i++)
{
m2r_nonex = true;
auto m1r = M1.row(i);
// NOTE: this is slow
// check M1 row is in M2
for (size_t j = 0; j < M2.rows(); j++)
{
auto m2r = M2.row(j);
if (m1r == m2r)
m2r_nonex = false;
}
// if it's not in m2, add it to M3
if (m2r_nonex)
{
M3.row(k) = m1r;
I3(k) = i;
k++;
}
}
M3.conservativeResize(k, NoChange);
I3.conservativeResize(k, NoChange);
}
MatrixXi M1, M2, M3;
VectorXi I3;
M1.resize(3, 3);
M2.resize(2, 3);
M1 << 0, 0, 0, 1, 1, 1, 2, 2, 2;
M2 << 1, 1, 1, 2, 2, 2;
diffMatrix(M1, M2, M3, I3);
===========================================
M3 (Rows: 1 Cols: 3)
===========================================
[[0, 0, 0]]
The provided code works of course, but is slow. Ideally one would substitute the inner for loop for some more compact expression, maybe something allowing to count all occurrence of a M1 row in M2 rows in a single statement...
Is that possible?
===== EDIT =====
(following Homer512 answer)
Yes, it works. To give an idea of the performance increase... After loading 10000 records:
MatrixXi M1, M2, M3;
VectorXi I3;
size_t rows = 10000;
M1.resize(rows, 3);
M2.resize(rows, 3);
for (size_t i = 0; i < rows; i++)
{
M1(i,0) = i;
M1(i,1) = i;
M1(i,2) = i;
M2(i,0) = i + 1;
M2(i,1) = i + 1;
M2(i,2) = i + 1;
}
with first method proposed elapsed is: 43.826125 secs; with second method proposed elapsed is: 0.017632 secs
It's about some orders of magnitude faster... Thank you for this.

Answer 1

Here is a version that ought to be relatively efficient.
We start by transposing the input matrices. You want to compare rows. But Eigen organizes its matrices column-major which means that consecutive elements within a row are not stored consecutively in memory. This makes everything you do rowwise slow and everything you do column-wise fast. Ideally, you want to skip this step and simply start with the proper orientation. Or switch to row-major matrices as described in the linked Eigen documentation.
const Eigen::MatrixXi left_transp = M1.transpose();
const Eigen::MatrixXi right_transp = M2.transpose();
Now for the actual star of the show: We want to use a hash set to check whether an element is in the set. We have a hash set we can use, the std::unordered_set but we need to define a suitable key to represent a vector, ideally without copying data.
Thankfully C++17 introduced std::string_view and related type definitions including std::u32string_view; perfect for binary int32 data. These types come with std::hash specializations and comparisons.
At this point we can build a hash table for all entries in M2.
std::unordered_set<std::u32string_view> right_set;
right_set.reserve(static_cast<std::size_t>(right_transp.cols()));
const std::size_t rows = static_cast<std::size_t>(right_transp.rows());
for(auto col: right_transp.colwise()) {
static_assert(sizeof(char32_t) == sizeof(int));
assert(col.innerStride() == 1);
const char32_t* ptr = reinterpret_cast<const char32_t*>(col.data());
right_set.emplace(ptr, rows);
}
In the next step we can do the same with M1 and search the hash table for duplicates.
const Eigen::Index left_n = left_transp.cols();
I3.resize(left_n);
Eigen::Index count = 0;
for(Eigen::Index i = 0; i < left_n; ++i) {
const char32_t* ptr = reinterpret_cast<const char32_t*>(
left_transp.col(i).data());
const std::u32string_view view{ ptr, rows };
if(! right_set.count(view))
I3[count++] = i;
}
I3.conservativeResize(count);
All that's left is using the indices to build M3. Again, preferably without transposing.
M3.resize(static_cast<Eigen::Index>(rows), count);
for(Eigen::Index i = 0; i < count; ++i)
M3.col(i) = left_transp.col(I3[i]);
M3.transposeInPlace();