Retrieving data from af::array via host() results in wrong data - c++

When trying to retrieve data from an af::array (arrayfire) from the device via host(), my output data on the host is wrong (i.e. wrong values). For testing that, I wrote a small code sample (based on https://stackoverflow.com/a/29212923/2546099):
int main(void) {
size_t vector_size = 16;
af::array in_test_array = af::constant(1., vector_size), out_test_array = af::constant(0., vector_size);
af_print(in_test_array);
double *local_data_ptr = new double[vector_size]();
for(int i = 0; i < vector_size; ++i)
std::cout << local_data_ptr[i] << '\t';
std::cout << '\n';
in_test_array.host(local_data_ptr);
for(int i = 0; i < vector_size; ++i)
std::cout << local_data_ptr[i] << '\t';
std::cout << '\n';
delete[] local_data_ptr;
out_test_array = in_test_array;
af_print(out_test_array);
return 0;
}
My output is
in_test_array
[16 1 1 1]
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0.007813 0.007813 0.007813 0.007813 0.007813 0.007813 0.007813 0.007813 0 0 0 0 0 0 0 0
out_test_array
[16 1 1 1]
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
Why are half the values in the pointer set to 0.007813, and not all values to 1? When changing the default value for in_test_array to 2, half the values are set to 2, and for 3 those values are set to 32. Why does that happen?

The datatypes between arrayfire and C are in conflict.
For float use:
af::array in_test_array = af::constant(1., vector_size),
out_test_array = af::constant(0., vector_size);
float *local_data_ptr = new float[vector_size]();
For double use:
af::array in_test_array = af::constant(1., vector_size, f64),
out_test_array = af::constant(0., vector_size, f64)
double *local_data_ptr = new double[vector_size]();
IN both cases above, you will see that arrayfire will return you 1.0 in the local_data_ptr buffer, although with different data types.

Related

Arrayfire sparse matrix issues

Getting confused with something that should be simple. Spent a bit of time trying to debug this and am not getting too far. Would appreciate if someone could help me out.
I am trying to define a sparse matrix in arrayfire by specifying the value/column/row triples as specified in this function. I want to store the following matrix as sparse:
3 3 4
3 10 0
4 0 3
I code it up as follows:
int row[] = {0,0,0,1,1,2,2};
int col[] = {0,1,2,0,1,0,2};
double values[] = { 3,3, 4,3,10,4,3};
array rr = sparse(3,3,array(7,values),array(7,row),array(7,col));
af_print(rr);
af_print(dense(rr));
I get the following output:
rr
Storage Format : AF_STORAGE_CSR
[3 3 1 1]
rr: Values
[7 1 1 1]
1.0000
2.0000
4.0000
3.0000
10.0000
4.0000
3.0000
rr: RowIdx
[7 1 1 1]
0
0
0
1
1
2
2
rr: ColIdx
[7 1 1 1]
0
1
2
0
1
0
2
dense(rr)
[3 3 1 1]
0.0000 0.0000 0.0000
0.0000 0.0000 3.0000
3.0000 0.0000 0.0000
When printing out stored matrix in dense format, I get something completely different than intended.
How do I make the output of printing the dense version of rr give:
3 3 4
3 10 0
4 0 3
Arrayfire uses (a modified) CSR format, so the rowarray has to be of length number_of_rows + 1. Normally it would be filled with the number of non-zero entries per row, i.e. {0, 3 ,2, 2}. But for Arrayfire, you need to take the cumulative sum, i.e. {0, 3, 5, 7}. So this works for me:
int row[] = {0,3,5,7};
int col[] = {0,1,2,0,1,0,2};
float values[] = {3,3,4,3,10,4,3};
array rr = sparse(3,3,array(7,values),array(4,row),array(7,col));
af_print(rr);
af_print(dense(rr));
However, this is not really convenient, since it is quite different from your input format. As an alternative, you could specify the COO format:
int row[] = {0,0,0,1,1,2,2};
int col[] = {0,1,2,0,1,0,2};
float values[] = { 3,3, 4,3,10,4,3};
array rr = sparse(3,3,array(7,values),array(7,row),array(7,col), AF_STORAGE_COO);
af_print(rr);
af_print(dense(rr));
which produces:
rr
Storage Format : AF_STORAGE_COO
[3 3 1 1]
rr: Values
[7 1 1 1]
3.0000
3.0000
4.0000
3.0000
10.0000
4.0000
3.0000
rr: RowIdx
[7 1 1 1]
0
0
0
1
1
2
2
rr: ColIdx
[7 1 1 1]
0
1
2
0
1
0
2
dense(rr)
[3 3 1 1]
3.0000 3.0000 4.0000
3.0000 10.0000 0.0000
4.0000 0.0000 3.0000
See also https://github.com/arrayfire/arrayfire/issues/2134.

Sending Vector Winsock Problems

first time asking a question!
I have a small server that sends a vector of objects to a client:
void sendFoo(vector<Foo> &foo){
int num = foo.size();
sSend = accept(sListen, (SOCKADDR*)&addr, &addrLen);
for(int i = 0; i < num; i++){
if(sSend == NULL){
cout << "Not connected" << endl;
} else {
//send vector size
send(sSend, (char*)&num, sizeof(num), NULL);
//send foo(s)
send(sSend, (char*)&foo[i], sizeof(foo[i]), NULL);
}
}
closesocket(sSend);
}
The foo class contains an int id and a double transformation[3][4].
My client app prints out the ids and the transformations it receives:
sRecv = socket(AF_INET, SOCK_STREAM, NULL);
if (connect(sRecv, (SOCKADDR*)&addr, addrLen) != 0) {
cout << "Not connected to server" << endl;
} else {
recv(sRecv, (char*)&num, sizeof(num), NULL);
vector<Foo> foo(num);
for (int i = 0; i < num; i++){
recv(sRecv, (char*)&foo[i], sizeof(foo[i]), NULL);
//prints them out
}
}
If I create 1 Foo and send it across the client correctly prints this:
Foo ID: 1
Foo Transformation:
1.0000 2.0000 3.0000 4.0000
1.0000 2.0000 3.0000 4.0000
1.0000 2.0000 3.0000 4.0000
But when I try to send across more than one I get what look like memory references every other block:
Foo ID: 1
Foo 1 Transformation:
1.0000 2.0000 3.0000 4.0000
1.0000 2.0000 3.0000 4.0000
1.0000 2.0000 3.0000 4.0000
Foo ID: 4
Foo 4 Transformation:
-6.27744e+66 -6.27744e+66 -6.27744e+66 -6.27744e+66
-6.27744e+66 -6.27744e+66 -6.27744e+66 -6.27744e+66
-6.27744e+66 -6.27744e+66 -6.27744e+66 -6.27744e+66
Foo ID: 2
Foo 2 Transformation:
1.0000 2.0000 3.0000 4.0000
1.0000 2.0000 3.0000 4.0000
1.0000 2.0000 3.0000 4.0000
Foo ID: 4
Foo 4 Transformation:
1.70592e-314 5.29981e-315 5.30499e-315 5.30758e-315
5.31017e-315 5.29981e-315 5.30499e-315 5.30758e-315
5.31017e-315 5.29981e-315 5.30499e-315 5.30758e-315
//etc
[SOLVED] I was trying to send the size of the vector multiple times and the client was only set up to receive it once. Thanks for the answers!
Well, I'd say a debugger will be your best friend. I'm also a little surprised the received end gave the printed output you show, perhaps more issues that aren't shown.
Look at the location of the following line:
send(sSend, (char*)&num, sizeof(num), NULL);
And compare to where you have the receiving of that:
recv(sRecv, (char*)&num, sizeof(num), NULL);
Hint, you call send more than recv.
To send your class over a socket you need to serialize it in a raw chunk of memory. you can't send the C++ object directly.
Make a function uint_32_t Foo::exportRaw( void **buffer) ; that will allocate raw memory for a buffer, put you class' members in it, then return the size allocated to send it over the socket.
Don't forget to free your buffer memory when it is sent and useless.
Here is an example :
void *buffer;
uint32_t sizeAlloc;
sizeAlloc = foo[i].exportRaw(&buffer); // serialize object
//send foo(s)
send(sSend, (char*)buffer, sizeAlloc, NULL);
free(buffer);

Complex Number Matrix multiplication Eigen vs Matlab

Can someone explain to me why the results are different.
Code in C++:
MatrixXcd testTest;
testTest.resize(3,3);
testTest.real()(0,0) = 1;
testTest.real()(0,1) = 2;
testTest.real()(0,2) = 3;
testTest.real()(1,0) = 1;
testTest.real()(1,1) = 2;
testTest.real()(1,2) = 3;
testTest.real()(2,0) = 1;
testTest.real()(2,1) = 2;
testTest.real()(2,2) = 3;
testTest.imag()(0,0) = 1;
testTest.imag()(0,1) = 2;
testTest.imag()(0,2) = 3;
testTest.imag()(1,0) = 1;
testTest.imag()(1,1) = 2;
testTest.imag()(1,2) = 3;
testTest.imag()(2,0) = 1;
testTest.imag()(2,1) = 2;
testTest.imag()(2,2) = 3;
cout<< endl << testTest << endl;
cout<< endl << testTest.transpose() << endl;
cout<< endl << testTest*testTest.transpose() << endl;
cout<< endl << testTest << endl;
Results from C++:
(1,1) (2,2) (3,3)
(1,1) (2,2) (3,3)
(1,1) (2,2) (3,3)
(1,1) (1,1) (1,1)
(2,2) (2,2) (2,2)
(3,3) (3,3) (3,3)
(0,28) (0,28) (0,28)
(0,28) (0,28) (0,28)
(0,28) (0,28) (0,28)
(1,1) (2,2) (3,3)
(1,1) (2,2) (3,3)
(1,1) (2,2) (3,3)
And the same thing written in Matlab:
testTest = [ complex(1,1) complex(2,2) complex(3,3);
complex(1,1) complex(2,2) complex(3,3);
complex(1,1) complex(2,2) complex(3,3)];
testTest
testTest'
testTest*testTest'
testTest
Matlab results:
testTest =
1.0000 + 1.0000i 2.0000 + 2.0000i 3.0000 + 3.0000i
1.0000 + 1.0000i 2.0000 + 2.0000i 3.0000 + 3.0000i
1.0000 + 1.0000i 2.0000 + 2.0000i 3.0000 + 3.0000i
ans =
1.0000 - 1.0000i 1.0000 - 1.0000i 1.0000 - 1.0000i
2.0000 - 2.0000i 2.0000 - 2.0000i 2.0000 - 2.0000i
3.0000 - 3.0000i 3.0000 - 3.0000i 3.0000 - 3.0000i
ans =
28 28 28
28 28 28
28 28 28
testTest =
1.0000 + 1.0000i 2.0000 + 2.0000i 3.0000 + 3.0000i
1.0000 + 1.0000i 2.0000 + 2.0000i 3.0000 + 3.0000i
1.0000 + 1.0000i 2.0000 + 2.0000i 3.0000 + 3.0000i
Multiplication of testTest * testTest' in C returns returns complex numbers with real part 0 and imag part 28. Matlab returns just dobule with value 28.
' in Matlab does the transpose and takes the complex conjugate (http://uk.mathworks.com/help/matlab/ref/ctranspose.html). If you want to just do the transpose use .' (with a dot infront).
Thus, if you change your MATLAB test to
testTest*testTest.'
the results should be the same.
If you want the complex transpose in eigen then you can go matrix.adjoint() (or matrix.conjugate().transpose())

Convert my list output to a dataframe in pandas

How do i convert my list output to a data frame? below is a sample of the code and data
import pandas as pd
import numpy as np
from datetime import datetime
dat=pd.read_csv()
dat.Date = dat.Date.apply(lambda d: datetime.strptime(d, "%d-%m-%Y"))
dat.index = dat.Date
dat = dat.drop(['Date'], axis=1)
################################################################
#Provide Input parameters
Decay=0.4
Decay_Dur=15 #(in days)
Return_Avg_Dur=15 #(in days)
################################################################
Weights=[]
Weights=[pow(i,((2*Decay)-1)) for i in range(1,Decay_Dur+1)] # Calculate Weights
Weights=Weights[::-1] #Reverse the order
fin_dat=[0]
for j in range(1,(dat.shape[0]-Decay_Dur)):
Sum_Weighted_Index=0
for i in range(j,Decay_Dur+j):
temp=Weights[i-j]*dat.iat[i-1,2] #
Sum_Weighted_Index+=temp
fin_dat.append(Sum_Weighted_Index)
Date SPX Index Surprise Index S&P 500 Daily Return
19-07-2007 1553.08 -0.0563 0.0045
20-07-2007 1534.1 0 -0.0122
23-07-2007 1541.57 0 0.0049
24-07-2007 1511.04 0 -0.0198
25-07-2007 1518.09 0 0.0047
26-07-2007 1482.66 0 -0.0233
27-07-2007 1458.95 0 -0.016
30-07-2007 1473.91 0 0.0103
31-07-2007 1455.27 -0.0867 -0.0126
01-08-2007 1465.81 -0.1529 0.0072
02-08-2007 1472.2 0 0.0044
03-08-2007 1433.06 -0.0848 -0.0266
06-08-2007 1467.67 0 0.0242
07-08-2007 1476.71 0 0.0062
08-08-2007 1497.49 0 0.0141
09-08-2007 1453.09 0 -0.0296
10-08-2007 1453.64 0 0.0004
13-08-2007 1452.92 0.0138 -0.0005
14-08-2007 1426.54 0 -0.0182
15-08-2007 1406.7 0 -0.0139
16-08-2007 1411.27 -0.1289 0.0032
17-08-2007 1445.94 0 0.0246
20-08-2007 1445.55 0 -0.0003
21-08-2007 1447.12 0 0.0011
22-08-2007 1464.07 0 0.0117
23-08-2007 1462.5 0 -0.0011
24-08-2007 1479.37 0 0.0115
27-08-2007 1466.79 0 -0.0085
I tried to use your code and then create new version using pandas functions.
It's all my "notes" - and result at the end.
Check whether the results are correct.
import pandas as pd
#--- generate some data ---
#dates = pd.date_range( '01-01-2010', periods=30, freq='D' )
#values = range(0,30)
#dat = pd.DataFrame( {'Date':dates, 'val1':values, 'val2':values} )
#dat.index = dat.Date
#print dat
data = '''Date SPX Surprise S&P-500
19-07-2007 1553.08 -0.0563 0.0045
20-07-2007 1534.1 0 -0.0122
23-07-2007 1541.57 0 0.0049
24-07-2007 1511.04 0 -0.0198
25-07-2007 1518.09 0 0.0047
26-07-2007 1482.66 0 -0.0233
27-07-2007 1458.95 0 -0.016
30-07-2007 1473.91 0 0.0103
31-07-2007 1455.27 -0.0867 -0.0126
01-08-2007 1465.81 -0.1529 0.0072
02-08-2007 1472.2 0 0.0044
03-08-2007 1433.06 -0.0848 -0.0266
06-08-2007 1467.67 0 0.0242
07-08-2007 1476.71 0 0.0062
08-08-2007 1497.49 0 0.0141
09-08-2007 1453.09 0 -0.0296
10-08-2007 1453.64 0 0.0004
13-08-2007 1452.92 0.0138 -0.0005
14-08-2007 1426.54 0 -0.0182
15-08-2007 1406.7 0 -0.0139
16-08-2007 1411.27 -0.1289 0.0032
17-08-2007 1445.94 0 0.0246
20-08-2007 1445.55 0 -0.0003
21-08-2007 1447.12 0 0.0011
22-08-2007 1464.07 0 0.0117
23-08-2007 1462.5 0 -0.0011
24-08-2007 1479.37 0 0.0115
27-08-2007 1466.79 0 -0.0085'''
from StringIO import StringIO
dat = pd.DataFrame.from_csv( StringIO(data), sep='\s+')
#------------------------------------------
decay = 0.4
decay_dur = 15 # (in days)
return_avg_dur = 15 # (in days)
#--- old version ---
weights = [ pow(i,(2*decay)-1) for i in range(1,decay_dur+1) ] # Calculate Weights
weights = weights[::-1] #Reverse the order
#weights = [ pow(i,(2*decay)-1) for i in range(1,decay_dur+1) ][::-1]
#fin_dat=[0]
dat['old'] = 0.0
for j in range(1,(dat.shape[0]-decay_dur)):
sum_weighted_index = 0
for i in range(j,decay_dur+j):
#sum_weighted_index += weights[i-j] * dat.iat[i-1,2] #
sum_weighted_index += weights[i-j] * dat['S&P-500'].iat[i-1] #
#fin_dat.append(sum_weighted_index)
dat['old'].iat[j] = sum_weighted_index
#print sum_weighted_index
#--- new version ---
#def sum_weighted_index(data):
# result = 0
# for w, d in zip(weights, data):
# result += w * d
# return result
def sum_weighted_index(data):
return sum( w * d for w, d in zip(weights, data) )
dat['new'] = pd.rolling_apply(dat['S&P-500'], decay_dur, sum_weighted_index).shift(-decay_dur+2).fillna(0)
print dat
result
SPX Surprise S&P-500 old new
Date
2007-07-19 1553.08 -0.0563 0.0045 0.000000 0.000000
2007-07-20 1534.10 0.0000 -0.0122 -0.010550 -0.010550
2007-07-23 1541.57 0.0000 0.0049 -0.044731 -0.044731
2007-07-24 1511.04 0.0000 -0.0198 -0.034384 -0.034384
2007-07-25 1518.09 0.0000 0.0047 -0.036309 -0.036309
2007-07-26 1482.66 0.0000 -0.0233 -0.042091 -0.042091
2007-07-27 1458.95 0.0000 -0.0160 -0.055676 -0.055676
2007-07-30 1473.91 0.0000 0.0103 -0.035502 -0.035502
2007-07-31 1455.27 -0.0867 -0.0126 -0.000058 -0.000058
2007-01-08 1465.81 -0.1529 0.0072 -0.008301 -0.008301
2007-02-08 1472.20 0.0000 0.0044 -0.000615 -0.000615
2007-03-08 1433.06 -0.0848 -0.0266 0.006442 0.006442
2007-06-08 1467.67 0.0000 0.0242 0.001076 0.001076
2007-07-08 1476.71 0.0000 0.0062 0.000000 0.027115
2007-08-08 1497.49 0.0000 0.0141 0.000000 0.002560
2007-09-08 1453.09 0.0000 -0.0296 0.000000 0.000000
2007-10-08 1453.64 0.0000 0.0004 0.000000 0.000000
2007-08-13 1452.92 0.0138 -0.0005 0.000000 0.000000
2007-08-14 1426.54 0.0000 -0.0182 0.000000 0.000000
2007-08-15 1406.70 0.0000 -0.0139 0.000000 0.000000
2007-08-16 1411.27 -0.1289 0.0032 0.000000 0.000000
2007-08-17 1445.94 0.0000 0.0246 0.000000 0.000000
2007-08-20 1445.55 0.0000 -0.0003 0.000000 0.000000
2007-08-21 1447.12 0.0000 0.0011 0.000000 0.000000
2007-08-22 1464.07 0.0000 0.0117 0.000000 0.000000
2007-08-23 1462.50 0.0000 -0.0011 0.000000 0.000000
2007-08-24 1479.37 0.0000 0.0115 0.000000 0.000000
2007-08-27 1466.79 0.0000 -0.0085 0.000000 0.000000

opencv accessing color element of CV_32FC3 BGR cv::Mat

i have a kind of problem. i've seen some more similar question but i can't find a solution.
my problem is that i have a CV_32FC3 cv::Mat, where values are stored in range between 0 and 255, let's call it S.
I have to create a matrix, called P, where store BGR values of all pixel.
So P should have as rows number the total element of S pixel, and as cols number the number of channels (3).
this is what i've tried to do:
int n_pixels = S.cols * S.rows;
p = Mat::zeros(n_pixels, 3, CV_32FC1);
for(int i=0; i<n_pixels; i++) {
Scalar pixel = S.at<Scalar>(i); // i've tried also Vec3f, Point3_, etc..
p.at<float>(i,0) = pixel[0];
p.at<float>(i,1) = pixel[1];
p.at<float>(i,2) = pixel[2];
}
i've also tried low level c-api data access like this:
for(int i=0; i<selection.rows; i++) {
p.at<float>(i,0) = S.ptr<float>(i)[0];
p.at<float>(i,1) = S.ptr<float>(i)[1];
p.at<float>(i,2) = S.ptr<float>(i)[2];
}
and also splitting channels (but splitted channels have type CV_8U so i think it is wrong):
vector<Mat> bgr;
cv::split(S, bgr);
for(int i=0; i<n_pixels; i++) {
p.at<float>(i,0) = bgr[0].data[i];
p.at<float>(i,1) = bgr[1].data[i];
p.at<float>(i,2) = bgr[2].data[i];
}
but every time i get really weird result if coutting the pixels, like:
172.2042 0.0000 0.0000
2771.1414 0.0000 0.0000
2939505920.0000 3.3468 0.0000
3079446986752.0000 0.0129 0.0000
192669347217408.0000 0.8367 0.0000
51956177323392237568.0000 16301891256320.0000 0.0509
58314208864123224064.0000 3.3945 0.0029
180449.1406 0.0000 0.0000
0.6531 0.0000 0.0000
0.0100 0.0000 0.0000
2.7373 0.0000 0.0000
10957.3184 0.0000 0.0000
739729604608.0000 3.3778 0.0000
0.0000 0.0000 0.0000
0.0000 0.0000 0.0000
0.0000 0.0000 0.0000
0.0000 0.0000 0.0000
and, in some cases also segfault..
any delucidations? how to resolve it? why my trials are wrong?
EDIT: i think the problems are data types conversion
for(int row = 0; row < resiz.rows; ++row) {
float* p = resiz.ptr<float>(row);
for(int col = 0; col < resiz.cols; ++col,p+=3) {
p[0]=; p[1]=; p[2]=;
}
}
I think you are accessing your matrix elements the wrong way.
S.at<Scalar>(i);
which is the same as
S.at<Scalar>(Point(i,0));
But you need to supply the x and y coordinates.
For me this code gives the expected results:
for(int i=0; i<n_pixels; i++)
{
Vec3f pixel = S.at<Vec3f>(i/S.cols, i%S.cols);
p.at<float>(i,0) = pixel[0];
p.at<float>(i,1) = pixel[1];
p.at<float>(i,2) = pixel[2];
}