I'm new to AWS SageMaker and I'm trying to deploy a simple pre-trained model to SageMaker to create endpoint and then make predictions.
The model is a sklearn linear regression model, the input is a vectorized sparse matrix, derived from a string of text (customer's review), and output the star-rating value (1 to 5).
I have trained the model locally and export its artifact to a model.joblib file.
Then I configure the inference.py file to zip it together with the model.joblib file into a model.tar.gz file, which is then uploaded to S3 for model registration and endpoint creation.
However, when I invoke the endpoint on a sample text, the following error is returned in the CloudWatch log:
File "/miniconda3/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 498, in _check_vocabulary
raise NotFittedError("Vocabulary not fitted or provided")
I understand that this means SageMaker is complaining about the trained model artifact being not fitted, and there is no problem with other parts (such as the inference.py file). However the pre-trained model was fitted before exporting.
I'm not sure which part was wrong, so I didn't upload any more codes not to cluster.
Thank you.
I've trained a custom model on sagemaker based on PyTorch estimator.
Training has been completed, and I verified that the model artifacts have been saved into s3 location.
I want to load my trained model into my sagemaker notebooks so I can perform analysis/inference so on ...
I did as below but I am not sure if this is the right method to do this as it asks for instance type, and to my knowledge, If I were to load the already trained estimator, I would need to declare which type of computing instance I use once I start deploying the model for inference.
estimator = PyTorch(
model_data = ModelArtifact_S3_LOCATION,
entry_point ='train.py',
source_dir = 'code',
role = role,
framework_version = '1.5.0',
py_version = 'py3',)
If training has been completed and you want to setup for inference then you want to point to your tar.gz model artifact file to create an endpoint or take your training estimator directly. The following code block is the general flow that you want to follow for training, inference, and predictions.
# Train my estimator
pytorch_estimator = PyTorch(entry_point='train_and_deploy.py',
instance_type='ml.p3.2xlarge',
instance_count=1,
framework_version='1.8.0',
py_version='py3')
pytorch_estimator.fit('s3://my_bucket/my_training_data/')
# Deploy my estimator to a SageMaker Endpoint and get a Predictor
predictor = pytorch_estimator.deploy(instance_type='ml.m4.xlarge',
initial_instance_count=1)
# `data` is a NumPy array or a Python list.
# `response` is a NumPy array.
response = predictor.predict(data)
For more information check out the following link for deploying PyTorch models on SageMaker.
https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#deploy-pytorch-models
I work for AWS & my opinions are my own
I am trying to deploy a model trained with sklearn to an endpoint and serve it as an API for predictions. All I want to use sagemaker for, is to deploy and server model I had serialised using joblib, nothing more. every blog I have read and sagemaker python documentation showed that sklearn model had to be trained on sagemaker in order to be deployed in sagemaker.
When I was going through the SageMaker documentation I learned that sagemaker does let users load a serialised model stored in S3 as shown below:
def model_fn(model_dir):
clf = joblib.load(os.path.join(model_dir, "model.joblib"))
return clf
And this is what documentation says about the argument model_dir:
SageMaker will inject the directory where your model files and
sub-directories, saved by save, have been mounted. Your model function
should return a model object that can be used for model serving.
This again means that training has to be done on sagemaker.
So, is there a way I can just specify the S3 location of my serialised model and have sagemaker de-serialise(or load) the model from S3 and use it for inference?
EDIT 1:
I used code in the answer to my application and I got below error when trying to deploy from notebook of SageMaker studio. I believe SageMaker is screaming that training wasn't done on SageMaker.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-4-6662bbae6010> in <module>
1 predictor = model.deploy(
2 initial_instance_count=1,
----> 3 instance_type='ml.m4.xlarge'
4 )
/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, use_compiled_model, wait, model_name, kms_key, data_capture_config, tags, **kwargs)
770 """
771 removed_kwargs("update_endpoint", kwargs)
--> 772 self._ensure_latest_training_job()
773 self._ensure_base_job_name()
774 default_name = name_from_base(self.base_job_name)
/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in _ensure_latest_training_job(self, error_message)
1128 """
1129 if self.latest_training_job is None:
-> 1130 raise ValueError(error_message)
1131
1132 delete_endpoint = removed_function("delete_endpoint")
ValueError: Estimator is not associated with a training job
My code:
import sagemaker
from sagemaker import get_execution_role
# from sagemaker.pytorch import PyTorchModel
from sagemaker.sklearn import SKLearn
from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
sm_role = sagemaker.get_execution_role() # IAM role to run SageMaker, access S3 and ECR
model_file = "s3://sagemaker-manual-bucket/sm_model_artifacts/model.tar.gz" # Must be ".tar.gz" suffix
class AnalysisClass(RealTimePredictor):
def __init__(self, endpoint_name, sagemaker_session):
super().__init__(
endpoint_name,
sagemaker_session=sagemaker_session,
serializer=json_serializer,
deserializer=json_deserializer, # To be able to use JSON serialization
content_type='application/json' # To be able to send JSON as HTTP body
)
model = SKLearn(model_data=model_file,
entry_point='inference.py',
name='rf_try_1',
role=sm_role,
source_dir='code',
framework_version='0.20.0',
instance_count=1,
instance_type='ml.m4.xlarge',
predictor_cls=AnalysisClass)
predictor = model.deploy(initial_instance_count=1,
instance_type='ml.m4.xlarge')
Yes you can. AWS documentation focuses on end-to-end from training to deployment in SageMaker which makes the impression that training has to be done on sagemaker. AWS documentation and examples should have clear separation among Training in Estimator, Saving and loading model, and Deployment model to SageMaker Endpoint.
SageMaker Model
You need to create the AWS::SageMaker::Model resource which refers to the "model" you have trained and more. AWS::SageMaker::Model is in CloudFormation document but it is only to explain what AWS resource you need.
CreateModel API creates a SageMaker model resource. The parameters specifie the docker image to use, model location in S3, IAM role to use, etc. See How SageMaker Loads Your Model Artifacts.
Docker image
Obviously you need the framework e.g. ScikitLearn, TensorFlow, PyTorch, etc that you used to train your model to get inferences. You need a docker image that has the framework, and HTTP front end to respond to the prediction calls. See SageMaker Inference Toolkit and Using the SageMaker Training and Inference Toolkits.
To build the image is not easy. Hence AWS provides pre-built images called AWS Deep Learning Containers and available images are in Github.
If your framework and the version is listed there, you can use it as the image. Otherwise you need to build by yourself. See Building a docker container for training/deploying our classifier.
SageMaker Python SDK for Frameworks
Create SageMaker Model by yourself using API is hard. Hence AWS SageMaker Python SDK has provided utilities to create the SageMaker models for several frameworks. See Frameworks for available frameworks. If it is not there, you may still be able to use sagemaker.model.FrameworkModel and Model to load your trained model. For your case, see Using Scikit-learn with the SageMaker Python SDK.
model.tar.gz
For instance if you used PyTorch and save the model as model.pth. To load the model and the inference code to get the prediction from the model, you need to create a model.tar.gz file. The structure inside the model.tar.gz is explained in Model Directory Structure. If you use Windows, beware of the CRLF to LF. AWS SageMaker runs in *NIX environment. See Create the directory structure for your model files.
|- model.pth # model file is inside / directory.
|- code/ # Code artefacts must be inside /code
|- inference.py # Your inference code for the framework
|- requirements.txt # only for versions 1.3.1 and higher. Name must be "requirements.txt"
Save the tar.gz file in S3. Make sure of the IAM role to access the S3 bucket and objects.
Loading model and get inference
See Create a PyTorchModel object. When instantiating the PyTorchModel class, SageMaker automatically selects the AWS Deep Learning Container image for PyTorch for the version specified in framework_version. If the image for the version does not exist, then it fails. This has not been documented in AWS but need to be aware of. SageMaker then internally calls the CreateModel API with the S3 model file location and the AWS Deep Learning Container image URL.
import sagemaker
from sagemaker import get_execution_role
from sagemaker.pytorch import PyTorchModel
from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
role = sagemaker.get_execution_role() # IAM role to run SageMaker, access S3 and ECR
model_file = "s3://YOUR_BUCKET/YOUR_FOLDER/model.tar.gz" # Must be ".tar.gz" suffix
class AnalysisClass(RealTimePredictor):
def __init__(self, endpoint_name, sagemaker_session):
super().__init__(
endpoint_name,
sagemaker_session=sagemaker_session,
serializer=json_serializer,
deserializer=json_deserializer, # To be able to use JSON serialization
content_type='application/json' # To be able to send JSON as HTTP body
)
model = PyTorchModel(
model_data=model_file,
name='YOUR_MODEL_NAME_WHATEVER',
role=role,
entry_point='inference.py',
source_dir='code', # Location of the inference code
framework_version='1.5.0', # Availble AWS Deep Learning PyTorch container version must be specified
predictor_cls=AnalysisClass # To specify the HTTP request body format (application/json)
)
predictor = model.deploy(
initial_instance_count=1,
instance_type='ml.m5.xlarge'
)
test_data = {"body": "YOUR PREDICTION REQUEST"}
prediction = predictor.predict(test_data)
By default, SageMaker uses NumPy as the serialization format. To be able to use JSON, need to specify the serializer and content_type. Instead of using RealTimePredictor class, you can specify them to predictor.
predictor.serializer=json_serializer
predictor.predict(test_data)
Or
predictor.serializer=None # As the serializer is None, predictor won't serialize the data
serialized_test_data=json.dumps(test_data)
predictor.predict(serialized_test_data)
Inference code sample
See Process Model Input, Get Predictions from a PyTorch Model and Process Model Output. The prediction request is sent as JSON in HTTP request body in this example.
import os
import sys
import datetime
import json
import torch
import numpy as np
CONTENT_TYPE_JSON = 'application/json'
def model_fn(model_dir):
# SageMaker automatically load the model.tar.gz from the S3 and
# mount the folders inside the docker container. The 'model_dir'
# points to the root of the extracted tar.gz file.
model_path = f'{model_dir}/'
# Load the model
# You can load whatever from the Internet, S3, wherever <--- Answer to your Question
# NO Need to use the model in tar.gz. You can place a dummy model file.
...
return model
def predict_fn(input_data, model):
# Do your inference
...
def input_fn(serialized_input_data, content_type=CONTENT_TYPE_JSON):
input_data = json.loads(serialized_input_data)
return input_data
def output_fn(prediction_output, accept=CONTENT_TYPE_JSON):
if accept == CONTENT_TYPE_JSON:
return json.dumps(prediction_output), accept
raise Exception('Unsupported content type')
Related
Using Models Trained Outside of Amazon SageMaker
Note
SageMaker team keeps changing the implementations and the documentations are frequently obsolete. When you are sure you did follow the documents and it does not work, obsolete documentation is quite likely. In such case, need to clarify with AWS support, or open an issue in the Github.
I have my own trained TF Object Detection model. If I try to deploy/implement the same model in AWS Sagemaker. It was not working.
I have tried TensorFlowModel() in Sagemaker. But there is an argument called entrypoint- how to create that .py file for prediction?
entrypoint is a argument which contains the file name inference.py,which means,once you create a endpoint and try to predict the image using the invoke endpoint api. the instance will be created based on you mentioned and it will go to the inference.py script and execute the process.
Link : Documentation for tensor-flow model deployment in amazon sage-maker
.
The inference script must contain a methods input_handler and output_handler or handler which will cover both the function in inference.py script, this for pre and post processing of your image.
Example for Deploying the tensor flow model
In the above link, i have mentioned a medium post, this will be helpful for your doubts.
Tensorflow's Estimator provides a method to get desired variable values after training/testing using get_variable_value.
Does there exist similar functionality in Sagemaker's Estimator, so that I am able to obtain weights after my model is trained.
For the Estimator object in the SageMaker Python SDK, after you call fit() you can call get the S3 URL of your model artifacts with
model_artifacts_url = estimator.create_model().model_data
The model itself is saved into your S3 bucket as a tarball at this location. So from here you can get the model parameters out of S3.