I'm new to AWS SageMaker and I'm trying to deploy a simple pre-trained model to SageMaker to create endpoint and then make predictions.
The model is a sklearn linear regression model, the input is a vectorized sparse matrix, derived from a string of text (customer's review), and output the star-rating value (1 to 5).
I have trained the model locally and export its artifact to a model.joblib file.
Then I configure the inference.py file to zip it together with the model.joblib file into a model.tar.gz file, which is then uploaded to S3 for model registration and endpoint creation.
However, when I invoke the endpoint on a sample text, the following error is returned in the CloudWatch log:
File "/miniconda3/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 498, in _check_vocabulary
raise NotFittedError("Vocabulary not fitted or provided")
I understand that this means SageMaker is complaining about the trained model artifact being not fitted, and there is no problem with other parts (such as the inference.py file). However the pre-trained model was fitted before exporting.
I'm not sure which part was wrong, so I didn't upload any more codes not to cluster.
Thank you.
I am trying to deploy a model trained with sklearn to an endpoint and serve it as an API for predictions. All I want to use sagemaker for, is to deploy and server model I had serialised using joblib, nothing more. every blog I have read and sagemaker python documentation showed that sklearn model had to be trained on sagemaker in order to be deployed in sagemaker.
When I was going through the SageMaker documentation I learned that sagemaker does let users load a serialised model stored in S3 as shown below:
def model_fn(model_dir):
clf = joblib.load(os.path.join(model_dir, "model.joblib"))
return clf
And this is what documentation says about the argument model_dir:
SageMaker will inject the directory where your model files and
sub-directories, saved by save, have been mounted. Your model function
should return a model object that can be used for model serving.
This again means that training has to be done on sagemaker.
So, is there a way I can just specify the S3 location of my serialised model and have sagemaker de-serialise(or load) the model from S3 and use it for inference?
EDIT 1:
I used code in the answer to my application and I got below error when trying to deploy from notebook of SageMaker studio. I believe SageMaker is screaming that training wasn't done on SageMaker.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-4-6662bbae6010> in <module>
1 predictor = model.deploy(
2 initial_instance_count=1,
----> 3 instance_type='ml.m4.xlarge'
4 )
/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, use_compiled_model, wait, model_name, kms_key, data_capture_config, tags, **kwargs)
770 """
771 removed_kwargs("update_endpoint", kwargs)
--> 772 self._ensure_latest_training_job()
773 self._ensure_base_job_name()
774 default_name = name_from_base(self.base_job_name)
/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in _ensure_latest_training_job(self, error_message)
1128 """
1129 if self.latest_training_job is None:
-> 1130 raise ValueError(error_message)
1131
1132 delete_endpoint = removed_function("delete_endpoint")
ValueError: Estimator is not associated with a training job
My code:
import sagemaker
from sagemaker import get_execution_role
# from sagemaker.pytorch import PyTorchModel
from sagemaker.sklearn import SKLearn
from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
sm_role = sagemaker.get_execution_role() # IAM role to run SageMaker, access S3 and ECR
model_file = "s3://sagemaker-manual-bucket/sm_model_artifacts/model.tar.gz" # Must be ".tar.gz" suffix
class AnalysisClass(RealTimePredictor):
def __init__(self, endpoint_name, sagemaker_session):
super().__init__(
endpoint_name,
sagemaker_session=sagemaker_session,
serializer=json_serializer,
deserializer=json_deserializer, # To be able to use JSON serialization
content_type='application/json' # To be able to send JSON as HTTP body
)
model = SKLearn(model_data=model_file,
entry_point='inference.py',
name='rf_try_1',
role=sm_role,
source_dir='code',
framework_version='0.20.0',
instance_count=1,
instance_type='ml.m4.xlarge',
predictor_cls=AnalysisClass)
predictor = model.deploy(initial_instance_count=1,
instance_type='ml.m4.xlarge')
Yes you can. AWS documentation focuses on end-to-end from training to deployment in SageMaker which makes the impression that training has to be done on sagemaker. AWS documentation and examples should have clear separation among Training in Estimator, Saving and loading model, and Deployment model to SageMaker Endpoint.
SageMaker Model
You need to create the AWS::SageMaker::Model resource which refers to the "model" you have trained and more. AWS::SageMaker::Model is in CloudFormation document but it is only to explain what AWS resource you need.
CreateModel API creates a SageMaker model resource. The parameters specifie the docker image to use, model location in S3, IAM role to use, etc. See How SageMaker Loads Your Model Artifacts.
Docker image
Obviously you need the framework e.g. ScikitLearn, TensorFlow, PyTorch, etc that you used to train your model to get inferences. You need a docker image that has the framework, and HTTP front end to respond to the prediction calls. See SageMaker Inference Toolkit and Using the SageMaker Training and Inference Toolkits.
To build the image is not easy. Hence AWS provides pre-built images called AWS Deep Learning Containers and available images are in Github.
If your framework and the version is listed there, you can use it as the image. Otherwise you need to build by yourself. See Building a docker container for training/deploying our classifier.
SageMaker Python SDK for Frameworks
Create SageMaker Model by yourself using API is hard. Hence AWS SageMaker Python SDK has provided utilities to create the SageMaker models for several frameworks. See Frameworks for available frameworks. If it is not there, you may still be able to use sagemaker.model.FrameworkModel and Model to load your trained model. For your case, see Using Scikit-learn with the SageMaker Python SDK.
model.tar.gz
For instance if you used PyTorch and save the model as model.pth. To load the model and the inference code to get the prediction from the model, you need to create a model.tar.gz file. The structure inside the model.tar.gz is explained in Model Directory Structure. If you use Windows, beware of the CRLF to LF. AWS SageMaker runs in *NIX environment. See Create the directory structure for your model files.
|- model.pth # model file is inside / directory.
|- code/ # Code artefacts must be inside /code
|- inference.py # Your inference code for the framework
|- requirements.txt # only for versions 1.3.1 and higher. Name must be "requirements.txt"
Save the tar.gz file in S3. Make sure of the IAM role to access the S3 bucket and objects.
Loading model and get inference
See Create a PyTorchModel object. When instantiating the PyTorchModel class, SageMaker automatically selects the AWS Deep Learning Container image for PyTorch for the version specified in framework_version. If the image for the version does not exist, then it fails. This has not been documented in AWS but need to be aware of. SageMaker then internally calls the CreateModel API with the S3 model file location and the AWS Deep Learning Container image URL.
import sagemaker
from sagemaker import get_execution_role
from sagemaker.pytorch import PyTorchModel
from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
role = sagemaker.get_execution_role() # IAM role to run SageMaker, access S3 and ECR
model_file = "s3://YOUR_BUCKET/YOUR_FOLDER/model.tar.gz" # Must be ".tar.gz" suffix
class AnalysisClass(RealTimePredictor):
def __init__(self, endpoint_name, sagemaker_session):
super().__init__(
endpoint_name,
sagemaker_session=sagemaker_session,
serializer=json_serializer,
deserializer=json_deserializer, # To be able to use JSON serialization
content_type='application/json' # To be able to send JSON as HTTP body
)
model = PyTorchModel(
model_data=model_file,
name='YOUR_MODEL_NAME_WHATEVER',
role=role,
entry_point='inference.py',
source_dir='code', # Location of the inference code
framework_version='1.5.0', # Availble AWS Deep Learning PyTorch container version must be specified
predictor_cls=AnalysisClass # To specify the HTTP request body format (application/json)
)
predictor = model.deploy(
initial_instance_count=1,
instance_type='ml.m5.xlarge'
)
test_data = {"body": "YOUR PREDICTION REQUEST"}
prediction = predictor.predict(test_data)
By default, SageMaker uses NumPy as the serialization format. To be able to use JSON, need to specify the serializer and content_type. Instead of using RealTimePredictor class, you can specify them to predictor.
predictor.serializer=json_serializer
predictor.predict(test_data)
Or
predictor.serializer=None # As the serializer is None, predictor won't serialize the data
serialized_test_data=json.dumps(test_data)
predictor.predict(serialized_test_data)
Inference code sample
See Process Model Input, Get Predictions from a PyTorch Model and Process Model Output. The prediction request is sent as JSON in HTTP request body in this example.
import os
import sys
import datetime
import json
import torch
import numpy as np
CONTENT_TYPE_JSON = 'application/json'
def model_fn(model_dir):
# SageMaker automatically load the model.tar.gz from the S3 and
# mount the folders inside the docker container. The 'model_dir'
# points to the root of the extracted tar.gz file.
model_path = f'{model_dir}/'
# Load the model
# You can load whatever from the Internet, S3, wherever <--- Answer to your Question
# NO Need to use the model in tar.gz. You can place a dummy model file.
...
return model
def predict_fn(input_data, model):
# Do your inference
...
def input_fn(serialized_input_data, content_type=CONTENT_TYPE_JSON):
input_data = json.loads(serialized_input_data)
return input_data
def output_fn(prediction_output, accept=CONTENT_TYPE_JSON):
if accept == CONTENT_TYPE_JSON:
return json.dumps(prediction_output), accept
raise Exception('Unsupported content type')
Related
Using Models Trained Outside of Amazon SageMaker
Note
SageMaker team keeps changing the implementations and the documentations are frequently obsolete. When you are sure you did follow the documents and it does not work, obsolete documentation is quite likely. In such case, need to clarify with AWS support, or open an issue in the Github.
I have a model running on my jupyter notebook instance with very basic SVM classifier
# Text lassifier - Algorithm - SVM
# fit the training dataset on the classifier
SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto',probability=True)
SVM.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("SVM Accuracy Score -> ",accuracy_score(predictions_SVM, Test_Y)*100)
Use Case : Host the model on Sagemaker and create an endpoint.Use the end point via Lambda for text classification
I saw AWS has few posts on creating an endpoint E.g. https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-train-model.html but majority of the content is not applicable to scikit-learn : SVM
Is there an another approach I should be looking at ?
If your model is small enough, you can create a lambda function that loads the model and does predictions based on input passed in from user.
I have followed an Amazon tutorial for using SageMaker and have used it to create the model in the tutorial (https://aws.amazon.com/getting-started/tutorials/build-train-deploy-machine-learning-model-sagemaker/).
This is my first time using SageMaker, so my question may be stupid.
How do you actually view the model that it has created? I want to be able to see a) the final formula created with the parameters etc. b) graphs of plotted factors etc. as if I was reviewing a GLM for example.
Thanks in advance.
If you followed the SageMaker tutorial you must have trained an XGBoost model. SageMaker places the model artifacts in a bucket that you own, check the output S3 location in the AWS SageMaker console.
For more information about XGBoost you can check the AWS SageMaker documentation https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html#xgboost-sample-notebooks and the example notebooks, e.g. https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/xgboost_abalone/xgboost_abalone.ipynb
To consume the XGBoost artifact generated by SageMaker, check out the official documentation, which contains the following code:
# SageMaker XGBoost uses the Python pickle module to serialize/deserialize
# the model, which can be used for saving/loading the model.
# To use a model trained with SageMaker XGBoost in open source XGBoost
# Use the following Python code:
import pickle as pkl
model = pkl.load(open(model_file_path, 'rb'))
# prediction with test data
pred = model.predict(dtest)
Tensorflow's Estimator provides a method to get desired variable values after training/testing using get_variable_value.
Does there exist similar functionality in Sagemaker's Estimator, so that I am able to obtain weights after my model is trained.
For the Estimator object in the SageMaker Python SDK, after you call fit() you can call get the S3 URL of your model artifacts with
model_artifacts_url = estimator.create_model().model_data
The model itself is saved into your S3 bucket as a tarball at this location. So from here you can get the model parameters out of S3.