GPU utilization is zero when running batch transform in Amazon SageMaker - amazon-web-services

Question

I want to run a batch transform job on AWS SageMaker. I have an image classification model which I have trained on a local GPU. Now I want to deploy it on AWS SageMaker and make predictions using Batch Transform. While the batch transform job runs successfully, the GPU utilization during the job is always zero (GPU Memory utilization, however, is at 97%). That's what CloudWatch is telling me. Also, the job takes approx. 7 minutes to process 500 images, I would expect it to run much faster than this, at least when comparing it to the time it takes to process the images on a local GPU.
My question: Why doesn't the GPU get used during batch transform, even though I am using a GPU instance (I am using an ml.p3.2xlarge instance)? I was able to deploy the very same model to an endpoint and send requests. When deploying to an endpoint instead of using batch transform, the GPU actually gets used.
Model preparation
I am using a Keras Model with TensorFlow backend. I converted this model to a sagemaker model using this guide https://aws.amazon.com/blogs/machine-learning/deploy-trained-keras-or-tensorflow-models-using-amazon-sagemaker/ :
import tensorflow as tf
from tensorflow.python.saved_model import builder
from tensorflow.python.saved_model.signature_def_utils import predict_signature_def
from tensorflow.python.saved_model import tag_constants
import tarfile
import sagemaker
# deactivate eager mode
if tf.executing_eagerly():
tf.compat.v1.disable_eager_execution()
builder = builder.SavedModelBuilder(export_dir)
# Create prediction signature to be used by TensorFlow Serving Predict API
signature = predict_signature_def(
inputs={"image_bytes": model.input}, outputs={"score_bytes": model.output})
with tf.compat.v1.keras.backend.get_session() as sess:
# Save the meta graph and variables
builder.add_meta_graph_and_variables(
sess=sess, tags=[tag_constants.SERVING], signature_def_map={"serving_default": signature})
builder.save()
with tarfile.open(tar_model_file, mode='w:gz') as archive:
archive.add('export', recursive=True)
sagemaker_session = sagemaker.Session()
s3_uri = sagemaker_session.upload_data(path=tar_model_file, bucket=bucket, key_prefix=sagemaker_model_dir)
Batch Transform
Container image used for batch transform: 763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-inference:2.0.0-gpu
framework = 'tensorflow'
instance_type='ml.p3.2xlarge'
image_scope = 'inference'
tf_version = '2.0.0'
py_version = '3.6'
sagemaker_model = TensorFlowModel(model_data=MODEL_TAR_ON_S3, role=role, image_uri=tensorflow_image)
transformer = sagemaker_model.transformer(
instance_count = 1,
instance_type = instance_type,
strategy='MultiRecord',
max_concurrent_transforms=8,
max_payload=10, # in MB
output_path = output_data_path,
)
transformer.transform(data = input_data_path,
job_name = job_name,
content_type = 'application/json',
logs=False,
wait=True
)
Log file excerpts
Loading the model takes quite long (several minutes). During this time, the following error message is getting logged:
2020-11-08T15:14:12.433+01:00 2020/11/08 14:14:12 [error]
14#14: *3066 no live upstreams while connecting to upstream, client:
169.254.255.130, server: , request: "GET /ping HTTP/1.1", subrequest: "/v1/models/my_model:predict", upstream:
"http://tfs_upstream/v1/models/my_model:predict", host:
"169.254.255.131:8080" 2020-11-08T15:14:12.433+01:00
169.254.255.130 - - [08/Nov/2020:14:14:12 +0000] "GET /ping HTTP/1.1" 502 157 "-" "Go-http-client/1.1" 2020-11-08T15:14:12.433+01:00
2020/11/08 14:14:12 [error] 14#14: *3066 js: failed ping#015
2020-11-08T15:14:12.433+01:00 502 Bad
Gateway#015 2020-11-08T15:14:12.433+01:00
#015 2020-11-08T15:14:12.433+01:00 502
Bad Gateway#015 2020-11-08T15:14:12.433+01:00
nginx/1.16.1#015 2020-11-08T15:14:12.433+01:00
#015 2020-11-08T15:14:12.433+01:00 #015
There was a log entry about NUMA node read:
successful NUMA node read from SysFS had negative value (-1), but
there must be at least one NUMA node, so returning NUMA node zero
And about a serving warmup request:
No warmup data file found at
/opt/ml/model/export/my_model/1/assets.extra/tf_serving_warmup_requests
And this warning:
[warn] getaddrinfo: address family for nodename not supported