TensorFlow CNN with 3D input is too slow to train - python-2.7

Question

Problem
I am building a TensorFlow CNN in Python 2.7 to classify 40x40x40 3D images into one of the two categories. The training data is stored in a HDF5 file as a (5000,40,40,40,1) array (5000 training images, 1 color channel). However, when I am training the network, each iteration of a batch of 32 images takes about a minute to complete. Through Activity Monitor I see that there are about 6GB data written to the disk in each iteration. The HDF5 itself is only about 500MB. What is happening?
This is the code I used to load data:
f = h5py.File(file_name, 'r')
images = f.get('key_name')
images = np.array(images)
images = images.astype(np.float32)
images = np.multiply(images, 1.0 / 100.0)
I also tried to directly use a HDF5 object for each iteration, instead of loading all the data into memory at once. But the problem remains:
f = h5py.File(file_name, 'r')
images = np.array(f.get('key_name')[:batch_size])
CNN Structure
The CNN I built has two 3D convolution layer, one flatten layer, one fully connected layer and one output layer. Here is the complete code for the structure:
num_channels = 1
filter_size_conv1 = 5
filter_size_conv2 = 5
num_filters_conv1 = 32
num_filters_conv2 = 64
fc_layer_size = 64
x = tf.placeholder(tf.float32, shape=[None, img_size, img_size, img_size, num_channels], name='x')
y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')
y_true_cls = tf.argmax(y_true, dimension=1)
def create_weights(shape):
return tf.Variable(tf.truncated_normal(shape, stddev=0.05))
def create_biases(size):
return tf.Variable(tf.constant(0.05,shape=[size]))
def create_convolutional_layer(input,
num_input_channels,
conv_filter_size,
num_filters):
weights = create_weights(shape=[
conv_filter_size, conv_filter_size, conv_filter_size, num_input_channels, num_filters])
biases = create_biases(num_filters)
layer = tf.nn.conv3d(input=input,
filter=weights,
strides=[1, 1, 1, 1, 1],
padding='SAME')
layer += biases
layer = tf.nn.max_pool3d(input=layer,
ksize=[1, 4, 4, 4, 1],
strides=[1, 4, 4, 4, 1],
padding='SAME')
layer = tf.nn.relu(layer)
return layer
def create_flatten_layer(layer):
layer_shape = layer.get_shape()
num_features = layer_shape[1:5].num_elements()
layer = tf.reshape(layer, [-1, num_features])
return layer
def create_fc_layer(input,
num_inputs,
num_outputs,
use_relu=True):
weights = create_weights(shape=[num_inputs, num_outputs])
biases = create_biases(num_outputs)
layer = tf.matmul(input, weights) + biases
if use_relu:
layer = tf.nn.relu(layer)
return layer
layer_conv1 = create_convolutional_layer(input=x,
num_input_channels=num_channels,
conv_filter_size=filter_size_conv1,
num_filters=num_filters_conv1)
layer_conv2 = create_convolutional_layer(input=layer_conv1,
num_input_channels=num_filters_conv1,
conv_filter_size=filter_size_conv2,
num_filters=num_filters_conv2)
layer_flat = create_flatten_layer(layer_conv2)
layer_fc1 = create_fc_layer(input=layer_flat,
num_inputs=layer_flat.get_shape()[1:5].num_elements(),
num_outputs=fc_layer_size,
use_relu=True)
layer_fc2 = create_fc_layer(input=layer_fc1,
num_inputs=fc_layer_size,
num_outputs=num_classes,
use_relu=False)
Any help would be appreciated. Thank you so much!

Answer 1

I actually found the same problem here. I'm using conv3d and it takes like forever to finish. Probably related to the optimization of conv3d