How to Load custom data sets in neon nervana - python-2.7

Question

If any one is familiar with nervana's neon please can you give me example of how to load custom dataset in this example of neon.

Answer 1

Here is an example dataset. You can also check their docs. You'll see a refernce later to a "DataSet" in __iter__ but that is just contains some function generate an item. The key is to make sure you create contiguous X, y pairs, set them on a backend tensor and yield. Hope that helps.
import numpy as np
from data import DataSet
from operator import mul
from neon.data import NervanaDataIterator
class CustomLoader(NervanaDataIterator):
def __init__(self, in_data, img_shape, n_classes):
# Load the numpy data into some variables. We divide the image by 255 to normalize the values
# between 0 and 1.
self.shape = img_shape # shape of the input data (e.g. for images, (C, H, W))
# 1. assign some required and useful attributes
self.start = 0 # start at zero
self.ndata = in_data.shape[0] # number of images in X (hint: use X.shape)
self.nfeatures = reduce(mul, img_shape, 1) # number of features in X (hint: use X.shape)
# number of minibatches per epoch
# to calculate this, use the batchsize, which is stored in self.be.bsz
self.nbatches = self.ndata / self.be.bsz
# 2. allocate memory on the GPU for a minibatch's worth of data.
# (e.g. use `self.be` to access the backend.). See the backend documentation.
# to get the minibatch size, use self.be.bsz
# hint: X should have shape (# features, mini-batch size)
# hint: use some of the attributes previously defined above
self.dev_X = self.be.zeros((self.nfeatures, self.be.bsz))
self.dev_Y = self.be.zeros((n_classes, self.be.bsz))
self.data_loader = DataSet(in_data, self.be.bsz)
self.data_loader.start()
def reset(self):
self.data_loader.stop()
self.start = 0
self.data_loader.start()
def __iter__(self):
# 3. loop through minibatches in the dataset
for index in xrange(self.nbatches):
# 3a. grab the right slice from the numpy arrays
inputs, targets, _ = self.data_loader.batch()
inputs = inputs.ravel()
# The arrays X and Y data are in shape (batch_size, num_features),
# but the iterator needs to return data with shape (num_features, batch_size).
# here we transpose the data, and then store it as a contiguous array.
# numpy arrays need to be contiguous before being loaded onto the GPU.
inputs = np.ascontiguousarray(inputs.T / 255.0)
targets = np.ascontiguousarray(targets.T)
# here we test your implementation
# your slice has to have the same shape as the GPU tensors you allocated
assert inputs.shape == self.dev_X.shape, \
"inputs has shape {}, but dev_X is {}".format(inputs.shape, self.dev_X.shape)
assert targets.shape == self.dev_Y.shape, \
"targets has shape {}, but dev_Y is {}".format(targets.shape, self.dev_Y.shape)
# 3b. transfer from numpy arrays to device
# - use the GPU memory buffers allocated previously,
# and call the myTensorBuffer.set() function.
self.dev_X.set(inputs)
self.dev_Y.set(targets)
# 3c. yield a tuple of the device tensors.
# X should be of shape (num_features, batch_size)
# Y should be of shape (4, batch_size)
yield (self.dev_X, self.dev_Y)