aws sagemker does not save model artifacts after training RLEstimator - amazon-web-services

Question

I am training a RLEstimator with COACH toolkit and TENSORFLOW framework.
My code is based on sagemaker docs:
import sagemaker
bucket = sagemaker.Session().default_bucket()
role = sagemaker.get_execution_role()
from sagemaker.rl import RLEstimator, RLToolkit, RLFramework
instance_type = "ml.c5.2xlarge"
estimator = RLEstimator(source_dir='src',
entry_point="train-coach.py",
dependencies=["common/sagemaker_rl"],
toolkit=RLToolkit.COACH,
toolkit_version='0.11.1',
framework=RLFramework.TENSORFLOW,
role=role,
instance_count=1,
instance_type=instance_type,
output_path='s3://{}/'.format(bucket),
base_job_name='my-job-name')
estimator.fit()
Training is performed normally, the last lines in the output :
2021-06-15 06:10:02,088 sagemaker-containers INFO Reporting training SUCCESS
2021-06-15 06:10:20 Uploading - Uploading generated training model
2021-06-15 06:10:20 Completed - Training job completed
Training seconds: 136
Billable seconds: 136
But an attempt of deploying the model causes error:
predictor = estimator.deploy(instance_type='ml.m4.xlarge',
initial_instance_count=1)
update_endpoint is a no-op in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
---------------------------------------------------------------------------
ClientError Traceback (most recent call last)
<ipython-input-8-3b92cb4c3461> in <module>
1 # Deploy my estimator to a SageMaker Endpoint and get a MXNetPredictor
2 predictor = estimator.deploy(instance_type='ml.m4.xlarge',
----> 3 initial_instance_count=1)
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/sagemaker/estimator.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, use_compiled_model, wait, model_name, kms_key, data_capture_config, tags, **kwargs)
949 wait=wait,
950 kms_key=kms_key,
--> 951 data_capture_config=data_capture_config,
952 )
953
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/sagemaker/tensorflow/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, update_endpoint)
285 wait=wait,
286 data_capture_config=data_capture_config,
--> 287 update_endpoint=update_endpoint,
288 )
289
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
761 self._base_name = "-".join((self._base_name, compiled_model_suffix))
762
--> 763 self._create_sagemaker_model(instance_type, accelerator_type, tags)
764 production_variant = sagemaker.production_variant(
765 self.name, instance_type, initial_instance_count, accelerator_type=accelerator_type
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/sagemaker/model.py in _create_sagemaker_model(self, instance_type, accelerator_type, tags)
329 vpc_config=self.vpc_config,
330 enable_network_isolation=enable_network_isolation,
--> 331 tags=tags,
332 )
333
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/sagemaker/session.py in create_model(self, name, role, container_defs, vpc_config, enable_network_isolation, primary_container, tags)
2554
2555 try:
-> 2556 self.sagemaker_client.create_model(**create_model_request)
2557 except ClientError as e:
2558 error_code = e.response["Error"]["Code"]
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
384 "%s() only accepts keyword arguments." % py_operation_name)
385 # The "self" in this scope is referring to the BaseClient.
--> 386 return self._make_api_call(operation_name, kwargs)
387
388 _api_call.__name__ = str(py_operation_name)
~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
703 error_code = parsed_response.get("Error", {}).get("Code")
704 error_class = self.exceptions.from_code(error_code)
--> 705 raise error_class(parsed_response, operation_name)
706 else:
707 return parsed_response
ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Could not find model data at s3://<my-bucket>/sagemaker-rl-tensorflow-2021-06-15-06-05-55-347/output/model.tar.gz.
/sagemaker-rl-tensorflow-2021-06-15-06-05-55-347/output/ folder contains intermediate/ folder and output.tar.gz, but no model.tar.gz exists.
Why model artifacts are not saved during training?
How can I deploy my model to sagemeker endpoint?

Answer 1

you need your code to save a model in /opt/ml/model.
At the end of training, SageMaker checks this location and uploads its content to S3.

Answer 2

You should first check how many episodes that you're actually running. If your training printout doesn't contain any checkpoint logs that send to /opt/ml/model/data...etc... then your _save_tf_model function may not have any files to save. I had a similar issue where I only ran a single episode, and thus there were no checkpoints sending *.onnx files to my checkpoint directory.
Also, check line number 202 in your sagemaker_rl.coach_launcher.py file. In an older version I was using, it was missing the single line:
import tensorflow as tf