Scatter/Gather like Numpy in ArrayFire - c++

I want to scatter and gather elements from an array X at specific indices along one axis.
So given an array of indices idx, I want to select the idx(0)th element along the 0th column, the idx(1)th element along the 1st column, etc..
In Numpy, the following statement:
X = np.array([[1, 2, 3], [4, 5, 6]])
print(X[[0, 1, 1], range(3)])
prints [1, 5, 6].
Furthermore, I can do this process in reverse:
Y = np.zeros((2, 3))
Y[[0, 1, 1], range(3)] = [1, 5, 6]
print(Y)
This will print
[[1. 0. 0.]
[0. 5. 6.]]
However, when I try to replicate this behavior in ArrayFire:
float elements[] = {1, 2, 3, 4, 5, 6};
af::array X = af::array(3, 2, elements);
int idx_elements[] = {0, 1, 1};
af::array idx = af::array(3, idx_elements);
af::print("", X(af::span, idx));
I get an array of shape [3, 3, 1, 1] with the elements
1.0000 4.0000 4.0000
2.0000 5.0000 5.0000
3.0000 6.0000 6.0000
So how can I achieve the desired numpy-like behavior for scattering and gathering elements in ArrayFire?
To perform the gather operation on a matrix, I can extract the diagonal of the resulting matrix but that may not work in the multidimensional case and it doesn't work in the other (scatter) direction.

X
[3 2 1 1]
1.0000 4.0000
2.0000 5.0000
3.0000 6.0000
idx
[3 1 1 1]
0
1
1
ArrayFire does Cartesian product when af::array are involved. Hence, the output.
Please see the below indices because of that.
Col\Row 0 1 1 from array
0 (0, 0) (0,1) (0, 1)
1 (1, 0) (1,1) (1, 1)
2 (2, 0) (2,1) (2, 1)
^
^ from sequence
Thus, the output of X(af::span, idx)) is a 3x3 matrix.
To gather elements based on coordinates, you would need different function
approx2. Note that this function takes it's indices as floating point arrays only.
float idx_elements[] = {0, 1, 1}; // changed the idx to floats
af::array colIdx = af::array(3, idx_elements);
af::array rowIdx = af::iota(3); // same effect as span
af::array out = approx2(X, rowIdx, colIdx);
af_print(out);
// out
// [3 1 1 1]
// 1.0000
// 5.0000
// 6.0000
To set the values for given indices, you would have to flatten the array because of very reason
that array::operator() considers cartesian product when af::array is involved.
af::array A = af::constant(0, 3, 2); // same size as X
af::array B = af::flat(A); // flatten the array, this involves meta data modification only
B(rowIdx + 3 * colIdx) = out; // use row & col indices to fetch linear indices
// rowIdx + 3 * colIdx
// [3 1 1 1]
// 0.0000
// 4.0000
// 5.0000
B = moddims(B, A.dims()); // reset the dimensions to original A dims
af_print(B);
// B
// [3 2 1 1]
// 1.0000 0.0000
// 0.0000 5.0000
// 0.0000 6.0000
You can look more details in our indexing tutorial.

Related

replacing matrix of indices with corresponding vector with armadillo

I have an arma::umat matrix containing indices corresponding to an arma::vec vector containing either 1 or -1:
arma::umat A = { {8,9,7,10,6}, {5,3,1,2,4}};
arma::vec v = {-1, 1, 1, 1, -1, -1, 1, -1, -1 ,1};
I would like to replace each element in the matrix with the corresponding value in the vector, so the output look like this:
A = {{-1,-1,1,1,-1},{-1,1,-1,1,1,1}}
Any suggestions?
Thanks
Saving the result into A is not an option, since A contains unsigned integers, and your v vector has doubles. Just create an arma::mat to contain the result and loop for each row to index v accordingly. One way to do this is using .each_row member.
#include <armadillo>
int main(int argc, char *argv[]) {
arma::umat A = {{7, 8, 6, 9, 5}, {4, 2, 0, 1, 3}};
arma::vec v = {-1, 1, 1, 1, -1, -1, 1, -1, -1, 1};
arma::mat result(A.n_rows, A.n_cols);
auto lineIdx = 0u;
// We capture everything by reference and increase the line index after usage.
// The `.st()` is necessary because the result of indexing `v` is
// a "column vector" and we need a "row vector".
A.each_row([&](auto row) { result.row(lineIdx++) = v(row).st(); });
result.print("result");
return 0;
}
This code prints
result
-1.0000 -1.0000 1.0000 1.0000 -1.0000
-1.0000 1.0000 -1.0000 1.0000 1.0000

check if the rows from a set of indices are duplicates or not and reconstruction

I have a 2D binary array where the value can take 0 and 1 only.
I have a set of indices to check whether the entries of the binary matrix for those indices are duplicate or not. I want to get the matrix with duplicate rows removed and the set of duplicate indices.
For example,
>>>> a
array([[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
])
I am given set of indices (0,2,3,4,6,7). From the set, the rows corresponding to (0,2) and (4,6,7) are duplicates. I want the resulting matrix with the duplicates removed (as shown below)
>>>> b
array([[1, 0, 1, 0],
[0, 0, 1, 1],
[0, 0, 1, 1],
[1, 1, 1, 0],
[1, 1, 1, 0],
])
and a method for reconstruction of the matrix 'a' from 'b'
If the order in the output array is not relevant, then you can probably just use Eelco Hoogendoorn's answer. However, if you want to keep the same relative order as in the original array, here is another possible approach.
import numpy as np
a = np.array([
[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 0, 1, 0],
[0, 0, 1, 1],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
[1, 1, 1, 0],
])
idx = np.array([0, 2, 3, 4, 6, 7])
# Make an array of row numbers
r = np.arange(len(a))
# Replace row numbers in idx with -1
# (use assume_unique only if indices in idx are unique)
r[np.isin(r, idx, assume_unique=True)] = -1
# Add the column to the array
a2 = np.concatenate([a, r[:, np.newaxis]], axis=-1)
# Find unique indices and inverse indices
_, uniq_idx, inv_idx = np.unique(a2, return_index=True, return_inverse=True, axis=0)
# Sort indices to make output array and inverse indices
s = np.argsort(uniq_idx)
a_uniq = a[uniq_idx[s]]
inv_idx = s[inv_idx]
print(a_uniq)
# [[1 0 1 0]
# [0 0 1 1]
# [0 0 1 1]
# [1 1 1 0]
# [1 1 1 0]]
print(np.all(a_uniq[inv_idx] == a))
# True
EDIT: Some further explanation.
The idea in the solution above is to apply np.unique, but in a way that the rows that are not included in idx are not affected by it. In order to do that, you can just add a new number to each row. For the rows included in idx, this number will always be -1, and for the rest of rows it will be a different number for each. That way, it is impossible that rows that are not in idx get removed by np.unique. In order to do that, I build r, first with np.arange(len(a)), which gives you a number per row:
[0 1 2 3 4 5 6 7]
Then I check which of those are in idx with np.isin(r, idx, assume_unique=True) (assume_unique can only be used if elements in idx are guaranteed to be unique), so r[np.isin(r, idx, assume_unique=True)] = -1 will turn all indices idx into -1:
[-1 1 -1 -1 -1 5 -1 -1]
That is added as new column to a into a2:
[[ 1 0 1 0 -1]
[ 0 0 1 1 1]
[ 1 0 1 0 -1]
[ 0 0 1 1 -1]
[ 1 1 1 0 -1]
[ 1 1 1 0 5]
[ 1 1 1 0 -1]
[ 1 1 1 0 -1]]
Now it's just a matter of applying np.unique to a2. As expected, only rows in idx may be eliminated. However, since we want to keep the original relative order, we cannot use the output of np.unique, because it is sorted. We use return_index and return_inverse to get the indices that make the array of unique rows and the indices that get you back to the original array, and actually discard the new array.
To form the final array, you need to sort uniq_idx to keep the relative order, and then inv_idx accordingly. np.argsort gives you the indices that sort uniq_idx into s. uniq_idx[s] is just the array of unique row indices sorted, and s[inv_idx] will map every inverse index in inv_idx to the corresponding one in the resorted array. So, finally, a[uniq_idx[s]] gives you the output array, and the new inv_idx takes you back to the original one.
It feels like you could phrase your question at a higher level to get a more elegant solution; but this seems to solve the literal problem as stated.
idx = [0,2,3,4,6,7]
b = np.concatenate([np.unique(a[idx], axis=0), np.delete(a, idx, axis=0)], axis=0)

Arrayfire sparse matrix issues

Getting confused with something that should be simple. Spent a bit of time trying to debug this and am not getting too far. Would appreciate if someone could help me out.
I am trying to define a sparse matrix in arrayfire by specifying the value/column/row triples as specified in this function. I want to store the following matrix as sparse:
3 3 4
3 10 0
4 0 3
I code it up as follows:
int row[] = {0,0,0,1,1,2,2};
int col[] = {0,1,2,0,1,0,2};
double values[] = { 3,3, 4,3,10,4,3};
array rr = sparse(3,3,array(7,values),array(7,row),array(7,col));
af_print(rr);
af_print(dense(rr));
I get the following output:
rr
Storage Format : AF_STORAGE_CSR
[3 3 1 1]
rr: Values
[7 1 1 1]
1.0000
2.0000
4.0000
3.0000
10.0000
4.0000
3.0000
rr: RowIdx
[7 1 1 1]
0
0
0
1
1
2
2
rr: ColIdx
[7 1 1 1]
0
1
2
0
1
0
2
dense(rr)
[3 3 1 1]
0.0000 0.0000 0.0000
0.0000 0.0000 3.0000
3.0000 0.0000 0.0000
When printing out stored matrix in dense format, I get something completely different than intended.
How do I make the output of printing the dense version of rr give:
3 3 4
3 10 0
4 0 3
Arrayfire uses (a modified) CSR format, so the rowarray has to be of length number_of_rows + 1. Normally it would be filled with the number of non-zero entries per row, i.e. {0, 3 ,2, 2}. But for Arrayfire, you need to take the cumulative sum, i.e. {0, 3, 5, 7}. So this works for me:
int row[] = {0,3,5,7};
int col[] = {0,1,2,0,1,0,2};
float values[] = {3,3,4,3,10,4,3};
array rr = sparse(3,3,array(7,values),array(4,row),array(7,col));
af_print(rr);
af_print(dense(rr));
However, this is not really convenient, since it is quite different from your input format. As an alternative, you could specify the COO format:
int row[] = {0,0,0,1,1,2,2};
int col[] = {0,1,2,0,1,0,2};
float values[] = { 3,3, 4,3,10,4,3};
array rr = sparse(3,3,array(7,values),array(7,row),array(7,col), AF_STORAGE_COO);
af_print(rr);
af_print(dense(rr));
which produces:
rr
Storage Format : AF_STORAGE_COO
[3 3 1 1]
rr: Values
[7 1 1 1]
3.0000
3.0000
4.0000
3.0000
10.0000
4.0000
3.0000
rr: RowIdx
[7 1 1 1]
0
0
0
1
1
2
2
rr: ColIdx
[7 1 1 1]
0
1
2
0
1
0
2
dense(rr)
[3 3 1 1]
3.0000 3.0000 4.0000
3.0000 10.0000 0.0000
4.0000 0.0000 3.0000
See also https://github.com/arrayfire/arrayfire/issues/2134.

Geting the k-smallest values of each column in sorted order using Numpy.argpartition

Using np.argpartition, it does not sort the entire array. It only guarantees that the kth element is in sorted position and all smaller elements will be moved before it. Thus, the first k elements will be the k-smallest elements
>>> num = 3
>>> myBigArray=np.array([[1,3,2,5,7,0],[14,15,6,5,7,0],[17,8,9,5,7,0]])
>>> top = np.argpartition(myBigArray, num, axis=1)[:, :num]
>>> print top
[[5 0 2]
[3 5 2]
[5 3 4]]
>>> myBigArray[np.arange(myBigArray.shape[0])[:, None], top]
[[0 1 2]
[5 0 6]
[0 5 7]]
This returns the k-smallest values of each column. Note that these may not be in sorted order.I use this method because To get the top-k elements in sorted order in this way takes O(n + k log k) time
I want to get the k-smallest values of each column in sorted order, without increasing the time complexity.
Any suggestions??
To use np.argpartition and maintain the sorted order, we need to use those range of elements as range(k) instead of feeding in just the scalar kth param -
idx = np.argpartition(myBigArray, range(num), axis=1)[:, :num]
out = myBigArray[np.arange(idx.shape[0])[:,None], idx]
You can use the exact same trick that you used in the case of rows; combining with #Divakar's trick for sorting, this becomes
In [42]: num = 2
In [43]: myBigArray[np.argpartition(myBigArray, range(num), axis=0)[:num, :], np.arange(myBigArray.shape[1])[None, :]]
Out[43]:
array([[ 1, 3, 2, 5, 7, 0],
[14, 8, 6, 5, 7, 0]])
A bit of indirect indexing does the trick. Pleaese note that I worked on rows since you started off on rows.
fdim = np.arange(3)[:, None]
so = np.argsort(myBigArray[fdim, top], axis=-1)
tops = top[fdim, so]
myBigArray[fdim, tops]
# array([[0, 1, 2],
[0, 5, 6],
[0, 5, 7]])
A note on argpartition with range argument: I strongly suspect that it is not O(n + k log k); in any case it is typically several-fold slower than a manual argpartition + argsort see here

The results of OpenCV idft() and MATLAB ifft2 does not match

So i'm testing my algorithm in MATLAB and it's done.
Then now doing cording for porting on C++ with OpenCV 2.4.5.
The problem is inverse fourier transform methods of two platforms, OpenCV and MATLAB.
So i have tested with simple matrix.
Here's test results.
The subject matrix is... 3 by 3 2-D.
1 2 3
4 5 6
7 8 9
-MATLAB-
test = [ 1, 2, 3;
4, 5, 6;
7, 8, 9];
ifft2(test);
result
5.0000 + 0.0000i -0.5000 - 0.2887i -0.5000 + 0.2887i
-1.5000 - 0.8660i 0.0000 + 0.0000i 0.0000 + 0.0000i
-1.5000 + 0.8660i 0.0000 + 0.0000i 0.0000 + 0.0000i
-OPENCV-
Note:Elements are same values.
Mat a = Mat::zeros(3, 3, CV_64FC1);
Mat b = Mat::zeros(3, 3, CV_64FC1);
a.at<double>(0,0) = 1;
a.at<double>(0,1) = 2;
a.at<double>(0,2) = 3;
a.at<double>(1,0) = 4;
a.at<double>(1,1) = 5;
a.at<double>(1,2) = 6;
a.at<double>(2,0) = 7;
a.at<double>(2,1) = 8;
a.at<double>(2,2) = 9;
idft(a, b, DFT_SCALE, 0);
result
4.33333 -4.13077 2.79743
-2.10313 -0.103134 -2.83518
-0.563533 2.16852 1.43647
I still didnt have found the solution. Even this couldn't gave me a solution.
EDIT: The problem has been solved. I put the CV_64FC1 to idft() as an input and CV_64FC2 as an output. A two matrices must be have same depth, both input and output are have to be 64_FC2. And flags DFT+COMPLEX_OUTPUT+DFT_SCALE is same as MATLAB's ifft2.
-SOLVED-
Mat input = Mat::zeros(3, 3, CV_64FC2);
Mat output = Mat::zeros(3, 3, CV_64FC2);
idft(input, output, DFT_COMPLEX_OUTPUT+DFT_SCALE, 0);
I believe you need cv::DFT_COMPLEX_OUTPUT+cv::DFT_SCALE since the input to idft clearly results in a complex-valued matrix.
Also, I think you'll need a 2-channel array for the output (type CV_64FC2), similarly for the input. As with any multi-channel image in OpenCV, you then access elements with the appropriate vector type (e.g. for doubles, .at<cv::Vec2d>(i,j), where the Vec2d stores the real and imaginary components at location i,j).
I think if you use 2 channel input matrices (CV_64FC2) you should use
a.at<Vec2d>(0,0)[0] = 1; // Re - part
a.at<Vec2d>(0,0)[1] = 0; // Im - part
instead of:
a.at<double>(0,0) = 1;