Invoke sagemaker endpoint with custom inference script - amazon-web-services

Question

I've deployed a sagemaker endpoint using the following code:
from sagemaker.pytorch import PyTorchModel
from sagemaker import get_execution_role, Session
sess = Session()
role = get_execution_role()
model = PyTorchModel(model_data=my_trained_model_location,
role=role,
sagemaker_session=sess,
framework_version='1.5.0',
entry_point='inference.py',
source_dir='.')
predictor = model.deploy(initial_instance_count=1,
instance_type='ml.m4.xlarge',
endpoint_name='my_endpoint')
If I run:
import numpy as np
pseudo_data = [np.random.randn(1, 300), np.random.randn(6, 300), np.random.randn(3, 300), np.random.randn(7, 300), np.random.randn(5, 300)] # input data is a list of 2D numpy arrays with variable first dimension and fixed second dimension
result = predictor.predict(pseudo_data)
I can generate the result with no errors. However, if I want to invoke the endpoint and make prediction by running:
from sagemaker.predictor import RealTimePredictor
predictor = RealTimePredictor(endpoint='my_endpoint')
result = predictor.predict(pseudo_data)
I'd get an error:
Traceback (most recent call last):
File "default_local.py", line 77, in <module>
score = predictor.predict(input_data)
File "/home/biggytruck/.local/lib/python3.6/site-packages/sagemaker/predictor.py", line 113, in predict
response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
File "/home/biggytruck/.local/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/biggytruck/.local/lib/python3.6/site-packages/botocore/client.py", line 608, in _make_api_call
api_params, operation_model, context=request_context)
File "/home/biggytruck/.local/lib/python3.6/site-packages/botocore/client.py", line 656, in _convert_to_request_dict
api_params, operation_model)
File "/home/biggytruck/.local/lib/python3.6/site-packages/botocore/validate.py", line 297, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Body
From my understanding, the error occurs because I didn't pass in inference.py as the entry point file, which is required to handle the input since it's not in a standard format supported by Sagemaker. However, sagemaker.predictor.RealTimePredictor doesn't allow me to define the entry point file. How can I solve this?

Answer 1

The error you're seeing is raised from the clientside SageMaker Python SDK library, not the remote endpoint that you have published.
Here is the documentation for the data argument (in your case, this is pseudo_data)
data (object) – Input data for which you want the model to provide inference. If a serializer was specified when creating the RealTimePredictor, the result of the serializer is sent as input data. Otherwise the data must be sequence of bytes, and the predict method then sends the bytes in the request body as is.
Source: https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html#sagemaker.predictor.RealTimePredictor.predict
My guess is that pseudo_data is not the type that the SageMaker Python SDK is expecting, which is a sequence of bytes.