I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and specifically numpy arrays.
The wrapping I am using does two different types of return for array data (which is of particular interest to me):
ctypes Array: When I do type(x) (where x is the ctypes array, I get a <class 'module_name.wrapper_class_name.c_double_Array_12000'> in return. I know that this data is a copy of the internal data from the documentation and I am able to get it into a numpy array easily:
>>> np.ctypeslib.as_array(x)
This returns a 1D numpy array of the data.
ctype pointer to data: In this case from the library's documentation, I understand that I am getting a pointer to the data stored and used directly to the library. Whey I do type(y) (where y is the pointer) I get <class 'module_name.wrapper_class_name.LP_c_double'>. With this case I am still able to index through the data like y[0][2], but I was only able to get it into numpy via a super awkward:
>>> np.frombuffer(np.core.multiarray.int_asbuffer(
ctypes.addressof(y.contents), array_length*np.dtype(float).itemsize))
I found this in an old numpy mailing list thread from Travis Oliphant, but not in the numpy documentation. If instead of this approach I try as above I get the following:
>>> np.ctypeslib.as_array(y)
...
... BUNCH OF STACK INFORMATION
...
AttributeError: 'LP_c_double' object has no attribute '__array_interface__'
Is this np.frombuffer approach the best or only way to do this? I am open to other suggestions but must would still like to use numpy as I have a lot of other post-processing code that relies on numpy functionality that I want to use with this data.
Creating NumPy arrays from a ctypes pointer object is a problematic operation. It is unclear who actually owns the memory the pointer is pointing to. When will it be freed again? How long is it valid? Whenever possible I would try to avoid this kind of construct. It is so much easier and safer to create arrays in the Python code and pass them to the C function than to use memory allocated by a Python-unaware C function. By doing the latter, you negate to some extent the advantages of having a high-level language taking care of the memory management.
If you are really sure that someone takes care of the memory, you can create an object exposing the Python "buffer protocol" and then create a NumPy array using this buffer object. You gave one way of creating the buffer object in your post, via the undocumented int_asbuffer() function:
buffer = numpy.core.multiarray.int_asbuffer(
ctypes.addressof(y.contents), 8*array_length)
(Note that I substituted 8 for np.dtype(float).itemsize. It's always 8, on any platform.) A different way to create the buffer object would be to call the PyBuffer_FromMemory() function from the Python C API via ctypes:
buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory
buffer_from_memory.restype = ctypes.py_object
buffer = buffer_from_memory(y, 8*array_length)
For both these ways, you can create a NumPy array from buffer by
a = numpy.frombuffer(buffer, float)
(I actually do not understand why you use .astype() instead of a second parameter to frombuffer; furthermore, I wonder why you use np.int, while you said earlier that the array contains doubles.)
I'm afraid it won't get much easier than this, but it isn't that bad, don't you think? You could bury all the ugly details in a wrapper function and don't worry about it any more.
Another possibility (which may require more recent versions of libraries than is available when the first answer was written -- I tested something similar with ctypes 1.1.0 and numpy 1.5.0b2) is to convert from the pointer to the array.
np.ctypeslib.as_array(
(ctypes.c_double * array_length).from_address(ctypes.addressof(y.contents)))
This seems to still have the shared ownership semantics, so you probably need to make sure that you free the underlying buffer eventually.
Neither of these worked for me in Python 3. As a general solution for converting a ctypes pointer into a numpy ndarray in python 2 and 3 I found this worked (via getting a read-only buffer):
def make_nd_array(c_pointer, shape, dtype=np.float64, order='C', own_data=True):
arr_size = np.prod(shape[:]) * np.dtype(dtype).itemsize
if sys.version_info.major >= 3:
buf_from_mem = ctypes.pythonapi.PyMemoryView_FromMemory
buf_from_mem.restype = ctypes.py_object
buf_from_mem.argtypes = (ctypes.c_void_p, ctypes.c_int, ctypes.c_int)
buffer = buf_from_mem(c_pointer, arr_size, 0x100)
else:
buf_from_mem = ctypes.pythonapi.PyBuffer_FromMemory
buf_from_mem.restype = ctypes.py_object
buffer = buf_from_mem(c_pointer, arr_size)
arr = np.ndarray(tuple(shape[:]), dtype, buffer, order=order)
if own_data and not arr.flags.owndata:
return arr.copy()
else:
return arr
np.ctypeslib.as_array is all you need here.
From an array:
c_arr = (c_float * 8)()
np.ctypeslib.as_array(c_arr)
From a pointer
c_arr = (c_float * 8)()
ptr = ctypes.pointer(c_arr[0])
np.ctypeslib.as_array(ptr, shape=(8,))
Using np.ndarrays as ctypes arguments
The preferable approach is using ndpointer, as mentioned in the numpy-docs.
This approach is more flexible than using, for example,
POINTER(c_double), since several restrictions can be specified, which
are verified upon calling the ctypes function. These include data
type, number of dimensions, shape and flags. If a given array does not
satisfy the specified restrictions, a TypeError is raised.
Minimal, Reproducible Example
Calling memcpy from python. Eventually the filename of the standard C-library libc.so.6 needs to be adjusted.
import ctypes
import numpy as np
n_bytes_f64 = 8
nrows = 2
ncols = 5
clib = ctypes.cdll.LoadLibrary("libc.so.6")
clib.memcpy.argtypes = [
np.ctypeslib.ndpointer(dtype=np.float64, ndim=2, flags='C_CONTIGUOUS'),
np.ctypeslib.ndpointer(dtype=np.float64, ndim=1, flags='C_CONTIGUOUS'),
ctypes.c_size_t]
clib.memcpy.restype = ctypes.c_void_p
arr_from = np.arange(nrows * ncols).astype(np.float64)
arr_to = np.empty(shape=(nrows, ncols), dtype=np.float64)
print('arr_from:', arr_from)
print('arr_to:', arr_to)
print('\ncalling clib.memcpy ...\n')
clib.memcpy(arr_to, arr_from, nrows * ncols * n_bytes_f64)
print('arr_from:', arr_from)
print('arr_to:', arr_to)
Output
arr_from: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
arr_to: [[0.0e+000 4.9e-324 9.9e-324 1.5e-323 2.0e-323]
[2.5e-323 3.0e-323 3.5e-323 4.0e-323 4.4e-323]]
calling clib.memcpy ...
arr_from: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
arr_to: [[0. 1. 2. 3. 4.]
[5. 6. 7. 8. 9.]]
If you modify the ndim=1/2 arguments of ndpointer to be inconsistent with the dimensions of arr_from/arr_to, the code fails with an ArgumentError.
As the title of this question is quite general, ...
Constructing a np.ndarray from a ctypes.c_void_p result
Minimal, Reproducible Example
In the following example, some memory is allocated by malloc and filled with 0s by memset. Then a numpy array is constructed, to access this memory. Of course the occur some ownership issues, as python will not free memory, which was allocated in c. To avoid memory leaks, one has to free the allocated memory again by ctypes. The copy method can be used for the np.ndarray to acquire ownership.
import ctypes
import numpy as np
n_bytes_int = 4
size = 7
clib = ctypes.cdll.LoadLibrary("libc.so.6")
clib.malloc.argtypes = [ctypes.c_size_t]
clib.malloc.restype = ctypes.c_void_p
clib.memset.argtypes = [
ctypes.c_void_p,
ctypes.c_int,
ctypes.c_size_t]
clib.memset.restype = np.ctypeslib.ndpointer(
dtype=np.int32, ndim=1, flags='C_CONTIGUOUS')
clib.free.argtypes = [ctypes.c_void_p]
clib.free.restype = ctypes.c_void_p
pntr = clib.malloc(size * n_bytes_int)
ndpntr = clib.memset(pntr, 0, size * n_bytes_int)
print(type(ndpntr))
ctypes_pntr = ctypes.cast(ndpntr, ctypes.POINTER(ctypes.c_int))
print(type(ctypes_pntr))
print()
arr_noowner = np.ctypeslib.as_array(ctypes_pntr, shape=(size,))
arr_owner = np.ctypeslib.as_array(ctypes_pntr, shape=(size,)).copy()
# arr_owner = arr_noowner.copy()
print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))
print('\nfree allocated memory again ...\n')
_ = clib.free(pntr)
print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))
print('\njust for fun: free some python-memory ...\n')
_ = clib.free(arr_owner.ctypes.data_as(ctypes.c_void_p))
print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))
Output
<class 'numpy.ctypeslib.ndpointer_<i4_1d_C_CONTIGUOUS'>
<class '__main__.LP_c_int'>
arr_noowner (at 104719884831376): [0 0 0 0 0 0 0]
arr_owner (at 104719884827744): [0 0 0 0 0 0 0]
free allocated memory again ...
arr_noowner (at 104719884831376): [ -7687536 24381 -28516336 24381 0 0 0]
arr_owner (at 104719884827744): [0 0 0 0 0 0 0]
just for fun: free some python-memory ...
arr_noowner (at 104719884831376): [ -7687536 24381 -28516336 24381 0 0 0]
arr_owner (at 104719884827744): [ -7779696 24381 -28516336 24381 0 0 0]
If you are ok with creating arrays in python, the following example with 2d array works in python3:
import numpy as np
import ctypes
OutType = (ctypes.c_float * 4) * 6
out = OutType()
YourCfunction = ctypes.CDLL('./yourlib.so').voidreturningfunctionwithweirdname
YourCfunction.argtypes = [ctypes.POINTER(ctypes.c_float)]*3, ctypes.POINTER(ctypes.c_float)]*5, OutType]
YourCfunction(input1, input2, out)
out = np.array(out) # convert it to numpy
print(out)
numpy and ctypes versions are 1.11.1 and 1.1.0
Related
I could use your support on this. Here is my issue:
I've got a 2D buffer of floats (in a data object) in a C++ code, that I write in a binary file using:
ptrToFile.write(reinterpret_cast<char *>(&data->array[0][0]), nbOfEltsInArray * sizeof(float));
The data contains 8192 floats, and I (correctly ?) get a 32 kbytes (8192 * 4 bytes) file out of this line of code.
Now I want to read that binary file using MATLAB. The code is:
hdr_binaryfile = fopen(str_binaryfile_path,'r');
res2_raw = fread(hdr_binaryfile, 'float');
res2 = reshape(res2_raw, int_sizel, int_sizec);
But it's not happening as I expect it to happen. If I print the array of data in the C++ code using std::cout, I get:
pCarte_bin->m_size = 8192
pCarte_bin->m_sizel = 64
pCarte_bin->m_sizec = 128
pCarte_bin->m_p[0][0] = 1014.97
pCarte_bin->m_p[0][1] = 566946
pCarte_bin->m_p[0][2] = 423177
pCarte_bin->m_p[0][3] = 497375
pCarte_bin->m_p[0][4] = 624860
pCarte_bin->m_p[0][5] = 478834
pCarte_bin->m_p[1][0] = 2652.25
pCarte_bin->m_p[2][0] = 642077
pCarte_bin->m_p[3][0] = 5.33649e+006
pCarte_bin->m_p[4][0] = 3.80922e+006
pCarte_bin->m_p[5][0] = 568725
And on the MATLAB side, after I read the file using the little block of code above:
size(res2) = 64 128
res2(1,1) = 1014.9659
res2(1,2) = 323288.4063
res2(1,3) = 2652.2515
res2(1,4) = 457593.375
res2(1,5) = 642076.6875
res2(1,6) = 581674.625
res2(2,1) = 566946.1875
res2(3,1) = 423177.1563
res2(4,1) = 497374.6563
res2(5,1) = 624860.0625
res2(6,1) = 478833.7188
The size (lines, columns) is OK, as well as the very first item ([0][0] in C++ == [1][1] in MATLAB). But:
I'm reading the C++ line elements along the columns: [0][1] in C++ == [1][2] in MATLAB (remember that indexing starts at 1 in MATLAB), etc.
I'm reading one correct element out of two along the other dimension: [1][0] in C++ == [1][3] in MATLAB, [2][0] == [1][5], etc.
Any idea about this ?
Thanks!
bye
Leaving aside the fact there seems to be some precision difference (likely the display settings in MATLAB) the issue here is likely the difference between row major and column major ordering of data. Without more details it will be hard to be certain. In particular MATLAB is column major meaning that contiguous memory on disk is interpreted as detailing sequential elements in a column rather than a row.
The likely solution is to reverse the two sizes in your reshape, and access the elements with indices reversed. That is, swap the int_size1 and int_size2, and then read elements expecting
pCarte_bin->m_p[0][0] = res2(1,1)
pCarte_bin->m_p[0][1] = res2(2,1)
pCarte_bin->m_p[0][2] = res2(3,1)
pCarte_bin->m_p[0][3] = res2(4,1)
pCarte_bin->m_p[1][0] = res2(1,2)
etc.
You could also transpose the array in MATLAB after read, but for a large array that could be costly in itself
If I run the following code:
use BlockDist;
config const dimension: int = 5;
const space = {0..#dimension, 0..#dimension};
const matrixBlock: domain(2) dmapped Block(boundingBox=space) = space;
var A : [matrixBlock] int;
[a in A] a = a.locale.id;
writeln(A);
on 4 Locales, I get:
0 0 0 1 1
0 0 0 1 1
0 0 0 1 1
2 2 2 3 3
2 2 2 3 3
Is there a A.<function> which returns the matrix (below)?
0 1
2 3
Or, is this something I should implement?
The expression A.targetLocales() gives you almost what you asked for, and maybe something you'll find even more useful: Rather than the array of ints you requested, it gives you an array of the target locales themselves. Thus, writeln(A.targetLocales()) prints the 2x2 locale array:
LOCALE0 LOCALE1
LOCALE2 LOCALE3
This routine, and others related to locality queries on arrays, can be found in the domain and array operations section of the online documentation, under the array type.
The expression A.targetLocales().id ought to give you what you want, but due to a longstanding unimplemented feature does not (at least, as of version 1.15 of Chapel). In short, this asks each locale for its ID and should result in an array of ints with the same size and shape as the target locales array; yet because promotion doesn't preserve shape as intended, the shape is lost if you don't preserve it. For example, writeln(A.targetLocales.id) results in:
0 1 2 3
rather than:
0 1
2 3
However, you can assign such promoted expressions into an array of the desired shape. So one way to get your desired array of ints today would be to write:
// declare an array whose domain is the same as that of A's target locales
// and initialize the array using the IDs of A's targetLocales array:
var IDs: [A.targetLocales().domain] int = A.targetLocales().id;
Finally, note that you can pass your own array of locales into the Block() distribution's constructor if you wish to specify a specific target locale set, rather than using the default set of target locales that it sets up for you. For example, adding the two following lines:
const locGridSpace = {0..#numLocales, 0..0};
const locGrid: [locGridSpace] locale = [(i,j) in locGridSpace] Locales[i];
will create a numLocales x 1 array of locales which can then be passed into your call to Block() as follows:
const matrixBlock: domain(2) dmapped Block(boundingBox=space,
targetLocales=locGrid) = space;
Alternatively, you could arrange some or all of the locales into some other shape or ordering. The main restriction is that the rank of the targetLocales array matches that of the domains to which the distribution is applied. (So targetLocales must be 2D when distributing 2D domains and arrays and 3D for 3D domains and arrays).
Arrays, domains, and distributions all have a targetLocales() method that returns the array of locales over which the array/domain/distribution is distributed. See: Domain and Array Operations documentation.
The following calls:
writeln(A.targetLocales());
writeln(A.domain.targetLocales());
writeln(A.domain.dist.targetLocales());
Will all print:
LOCALE0 LOCALE1
LOCALE2 LOCALE3
To extract the integral ids, you can then use the .id accessor:
var targetLocs = A.targetLocales();
var targetLocIDs: [targetLocs.domain] int = targetLocs.id;
writeln(targetLocIDs);
Prints:
0 1
2 3
I have a function which is supposed to unpack an H5PY dataset, temp.hdf5, which only contains one example so it can be evaluated:
def getprob():
test_set = H5PYDataset('temp.hdf5', which_sets=('test',))
handle = test_set.open()
test_data = test_set.get_data(handle, slice(0,1))
xx = test_data[0]
YY = test_data[1]
l, prob, rho_orig, rho_larger, rho_largest = f(xx)
return prob[9][0]
Where test_data[0] is 28x28 array of integers and test_data[1] is an integer between 0 and 9.
The problem is that, from within the function, test_data[0] is always a 28x28 array of zeros, even though it is not within 'temp.hdf5'. test_data[1] always loads properly, though.
When these lines of code are run outside of the function, everything works just fine.
What is going on here?
def bruteForce( dictionary = {}):
key = 0
for i in range(len(dictionary)):
keyRank = 0
for k in range(68719476736):
attempt = decrypt(dictionary[i], k)
if(i != attempt):
keyRank = 0
break
else:
keyRank += 1
key = k
print 'key attempt: {0:b}'.format(key)
if(keyRank == len(dictionary)):
print 'found key: {0:b}'.format(key)
break
The key is 36 bits
I get a memory error on the for k in range() line of code
Why is this a memory issue? Does python build an actual list of ints before running this line? Is there a better way to write this loop?
I'm brand new to Python and this wouldn't be a problem in C or Java.
This is a known-plaintext/ciphertext attack. dictionary is a mapping of P:C pairs.
Its on a VM, I can up the memory if needed, but want to know both why its failing and a code-based workaround or better idiomatic approach.
In python 2, range() will build the entire list in memory.
xrange() is a sequence object that evaluates lazily.
In python 3, range() does what xrange() did.
i am trying to compare feature vectors present in test and train data set.These feature vectors are stored in sparse format using scikitlearn library load_svmlight_file.The dimension of feature vectors of both the dataset is same.However,I am getting this error :"The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()."
Why am I getting this error?
How can I resolve it?
Thanks in advance!
from sklearn.datasets import load_svmlight_file
pathToTrainData="../train.txt"
pathToTestData="../test.txt"
X_train,Y_train= load_svmlight_file(pathToTrainData);
X_test,Y_test= load_svmlight_file(pathToTestData);
for ele1 in X_train:
for ele2 in X_test:
if(ele1==ele2):
print "same vector"
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-3-c1f145f984a6> in <module>()
7 for ele1 in X_train:
8 for ele2 in X_test:
----> 9 if(ele1==ele2):
10 print "same vector"
/Users/rkasat/anaconda/lib/python2.7/site-packages/scipy/sparse/base.pyc in __bool__(self)
181 return True if self.nnz == 1 else False
182 else:
--> 183 raise ValueError("The truth value of an array with more than one "
184 "element is ambiguous. Use a.any() or a.all().")
185 __nonzero__ = __bool__
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
You can use this condition to check whether the two sparse arrays are exactly equal without needing to densify them:
if (ele1 - ele2).nnz == 0:
# Matched, do something ...
The nnz attribute gives the number of nonzero elements in the sparse array.
Some simple test runs to show the difference:
import numpy as np
from scipy import sparse
A = sparse.rand(10, 1000000).tocsr()
def benchmark1(A):
for s1 in A:
for s2 in A:
if (s1 - s2).nnz == 0:
pass
def benchmark2(A):
for s1 in A:
for s2 in A:
if (s1.toarray() == s2).all() == 0:
pass
%timeit benchmark1(A)
%timeit benchmark2(A)
Some results:
# Computer 1
10 loops, best of 3: 36.9 ms per loop # with nnz
1 loops, best of 3: 734 ms per loop # with toarray
# Computer 2
10 loops, best of 3: 28 ms per loop
1 loops, best of 3: 312 ms per loop
If your arrays are dense you can run into the same problem, and there the solution is straightforward. Replace
if(ele1==ele2):
with
if (ele1 == ele2).all():
However, since you are working with sparse matrices, this problem is actually not that easy in general. Notably, the functions all and any aren't implemented for sparse matrices (which, at least for all is understandable, because all can only return True if the matrix tested is densely filled with values that evaluate to True).
In your case, since you are only comparing lines of your sparse matrices, you may find it acceptable to densify them and then do the comparison. Try replacing the mentioned line by
if (ele1.toarray() == ele2).all(): # Densifying one of them casts the other to dense too
On a more general note, you seem to want to compare the lines of 2 matrices. Depending on the number of entries, this can be done a lot more efficiently by defining a vectorized comparison function, like this:
def compare(A, B):
return zip(*np.where((np.array(A.multiply(A).sum(1)) +
np.array(B.multiply(B).sum(1)).T) - 2 * A.dot(B.T).toarray() == 0))
This function will return a list of couples of indices, telling you which rows correspond to each other and is a lot more efficient than the double for loop used in your code.
Explanation: The function compare calculates pairwise euclidean distances using the binomial formula (a - b) ** 2 == a ** 2 + b ** 2 - 2 * a * b. This formula also works for l2 norm and scalar products. If the matrices weren't sparse, the formula would become much simpler: squared_distances = (A ** 2).sum(axis=1) + (B ** 2).sum(axis=1) - 2 * A.dot(B.T). Then we check which of these entries are equal to 0 using np.where and return them as tuples.
Benchmarking this, we obtain:
import numpy as np
from scipy import sparse
rng = np.random.RandomState(42)
A = sparse.rand(10, 1000000, random_state=rng).tocsr()
In [12]: %timeit compare(A, A)
100 loops, best of 3: 10.2 ms per loop