"errorMessage": "Parameter validation failed in Lambda calling SageMaker endpoint - amazon-web-services

I am trying to invoke a SageMaker enpoint from AWS Lambda using a lambda function.
This is a sample API call to the endpoint from SageMaker Studio, working as expected:
here's my Lambda function (inspired from documentation):
import os
import io
import boto3
import json
ENDPOINT_NAME = 'iris-autoscale-6'
runtime= boto3.client('runtime.sagemaker')
def lambda_handler(event, context):
# print("Received event: " + json.dumps(event, indent=2))
payload = json.loads(json.dumps(event))
print(payload)
response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME, ContentType='application/json', Body=payload)
print(response)
result = json.loads(response['Body'].read().decode())
print(result)
return result
My error message:
Test Event Name
ProperTest
Response
{
"errorMessage": "Parameter validation failed:\nInvalid type for parameter Body, value: {'sepal_length': [5.1, 4.9, 4.7, 4.6, 5], 'sepal_width': [3.5, 3, 3.2, 3.1, 3.6], 'petal_length': [1.4, 1.4, 1.3, 1.5, 1.4], 'petal_width': [0.2, 0.2, 0.2, 0.2, 0.2]}, type: <class 'dict'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object",
"errorType": "ParamValidationError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 17, in lambda_handler\n response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME, ContentType='application/json', Body=payload)\n",
" File \"/var/runtime/botocore/client.py\", line 386, in _api_call\n return self._make_api_call(operation_name, kwargs)\n",
" File \"/var/runtime/botocore/client.py\", line 678, in _make_api_call\n api_params, operation_model, context=request_context)\n",
" File \"/var/runtime/botocore/client.py\", line 726, in _convert_to_request_dict\n api_params, operation_model)\n",
" File \"/var/runtime/botocore/validate.py\", line 319, in serialize_to_request\n raise ParamValidationError(report=report.generate_report())\n"
]
}
Function Logs
START RequestId: 70278b9f-f75e-4ac9-a827-7ad35d162512 Version: $LATEST
{'sepal_length': [5.1, 4.9, 4.7, 4.6, 5], 'sepal_width': [3.5, 3, 3.2, 3.1, 3.6], 'petal_length': [1.4, 1.4, 1.3, 1.5, 1.4], 'petal_width': [0.2, 0.2, 0.2, 0.2, 0.2]}
[ERROR] ParamValidationError: Parameter validation failed:
Invalid type for parameter Body, value: {'sepal_length': [5.1, 4.9, 4.7, 4.6, 5], 'sepal_width': [3.5, 3, 3.2, 3.1, 3.6], 'petal_length': [1.4, 1.4, 1.3, 1.5, 1.4], 'petal_width': [0.2, 0.2, 0.2, 0.2, 0.2]}, type: <class 'dict'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 17, in lambda_handler
    response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME, ContentType='application/json', Body=payload)
  File "/var/runtime/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 678, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/var/runtime/botocore/client.py", line 726, in _convert_to_request_dict
    api_params, operation_model)
  File "/var/runtime/botocore/validate.py", line 319, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
END RequestId: 70278b9f-f75e-4ac9-a827-7ad35d162512
REPORT RequestId: 70278b9f-f75e-4ac9-a827-7ad35d162512 Duration: 26.70 ms Billed Duration: 27 ms Memory Size: 128 MB Max Memory Used: 76 MB Init Duration: 343.10 ms
Here's the policy attached to the lambda function:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sagemaker:InvokeEndpoint",
"Resource": "arn:aws:sagemaker:ap-south-1:<my-account-id>:endpoint/iris-autoscale-6"
}
]
}

The issue is that your payload has invalid format. It should be one of:
<class 'bytes'>, <class 'bytearray'>, file-like object
The following should address the error (note: you may have many other issues in your code):
payload = json.dumps(event)
print(payload)
response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME, ContentType='application/json', Body=payload.encode())

Related

Google Vertex AI hyperparameter tuning: 500 Internal Error encountered

I was trying to run a hyperparameter tuning job on Vertex AI using the Python SDK described here. Around 2 hours ago it was successful sending the job to run. I noticed there are some errors in my code so the run failed, and I went back and fix it, and I re-run the code and what I got was as below.
Traceback (most recent call last):
File "/workspace/.pip-modules/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
return callable_(*args, **kwargs)
File "/workspace/.pip-modules/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/workspace/.pip-modules/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered."
debug_error_string = "{"created":"#1623393121.374988331","description":"Error received from peer ipv4:142.251.33.74:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Internal error encountered.","grpc_status":13}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/workspace/ariel_ml_2021/hparam_tuning.py", line 140, in <module>
create_hyperparameter_tuning_job_python_package()
File "/workspace/ariel_ml_2021/hparam_tuning.py", line 133, in create_hyperparameter_tuning_job_python_package
response = client.create_hyperparameter_tuning_job(
File "/workspace/.pip-modules/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/job_service/client.py", line 1363, in create_hyperparameter_tuning_job
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/workspace/.pip-modules/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/workspace/.pip-modules/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.InternalServerError: 500 Internal error encountered.
I thought this might be due to my changes in the python code that causes the error, so I try with the original copy (without making any changes), and the error persists. The code for hyperparameter tuning is as below if required.
from google.cloud import aiplatform
def create_hyperparameter_tuning_job_python_package(
project: str = "<my_project_id>",
display_name: str = "<some_description>",
executor_image_uri: str = "us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-4:latest",
package_uri: str = "gs://<bucket_name>/",
python_module: str = "train_second", # located at gs://<bucket_name>/train_second.py
location: str = "us-central1",
api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
client_options = {"api_endpoint": api_endpoint}
client = aiplatform.gapic.JobServiceClient(client_options=client_options)
metric = {
"metric_id": "ariel_score",
"goal": aiplatform.gapic.StudySpec.MetricSpec.GoalType.MAXIMIZE,
}
conditional_param_H1 = {
"parameter_spec": {
"parameter_id": "H1",
"discrete_value_spec": {"values": [4, 8, 16, 32, 64, 128, 256, 512, 1024]},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
conditional_param_H2 = {
"parameter_spec": {
"parameter_id": "H2",
"discrete_value_spec": {"values": [64, 128, 256, 512, 1024]},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
conditional_param_H3 = {
"parameter_spec": {
"parameter_id": "H3",
"discrete_value_spec": {"values": [4, 8, 16, 32, 64, 128, 256, 512, 1024]},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
conditional_param_D1 = {
"parameter_spec": {
"parameter_id": "D1",
"double_value_spec": {"min_value": 0.01, "max_value": 0.5},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
conditional_param_mean = {
"parameter_spec": {
"parameter_id": "mean",
"discrete_value_spec": {"values": [0., 1.]},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
conditional_param_std = {
"parameter_spec": {
"parameter_id": "std",
"double_value_spec": {"min_value": 0.005, "max_value": 0.5},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
conditional_param_lr = {
"parameter_spec": {
"parameter_id": "lr",
"discrete_value_spec": {"values": [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3]},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
},
"parent_discrete_values": {"values": [10, 25, 50, 100]}
}
parameter = {
"parameter_id": "batch_size",
"discrete_value_spec": {"values": [10, 25, 50, 100]},
"scale_type": aiplatform.gapic.StudySpec.ParameterSpec.ScaleType.UNIT_LINEAR_SCALE,
"conditional_parameter_specs": [
conditional_param_H1,
conditional_param_H2,
conditional_param_H3,
conditional_param_D1,
conditional_param_mean,
conditional_param_std,
conditional_param_lr,
],
}
# Trial job spec
machine_spec = {
"machine_type": "e2-standard-4",
}
worker_pool_spec = {
"machine_spec": machine_spec,
"replica_count": 1,
"python_package_spec": {
"executor_image_uri": executor_image_uri,
"package_uris": [package_uri],
"python_module": python_module,
"args": [],
}
}
# hparam tuning job
hyperparameter_tuning_job = {
"display_name": display_name,
"max_trial_count": 2,
"parallel_trial_count": 2,
"study_spec": {
"metrics": [metric],
"parameters": [parameter],
},
"trial_job_spec": {"worker_pool_specs": [worker_pool_spec]},
}
parent = f"projects/{project}/locations/{location}"
response = client.create_hyperparameter_tuning_job(
parent=parent, hyperparameter_tuning_job=hyperparameter_tuning_job
)
print(f"response:", response)
if __name__ == "__main__":
create_hyperparameter_tuning_job_python_package()
Thanks in advance.
It seems that the endpoint at us-central1 is experiencing the problem. The workaround is to use another endpoint such as with us-east1 and the problem is solved.
Your package_uri is incorrect. It should point to a file containing the Python package (i.e. a tar.bz file with all the code), not a directory or a bucket.

Django Rest Framework Elastic Search: RequestError 400 parsing_exception

When trying to query for event documents that match this condition, I'm getting a parsing exception and I'm not sure what's causing it. This is occurring in my custom get_queryset method. In my get_query in my document view set I'm getting an error.
def get_queryset(self):
qs = super().get_queryset()
user = self.request.user
if hasattr(user, 'userprofile'):
user_universities = user.userprofile.universities.all().values_list("id")
user_universities_campus = user.userprofile.universities.all().values_list("main_campus__id")
query = query | Q('bool', must=[
Q('match', visibility_scope=Event.UNIVERSITY),
Q('bool', must=[
Q('terms', university__id=list(user_universities)),
Q('bool', should=[
Q('terms', university__main_campus__id=list(user_universities)),
Q('terms', university__main_campus__id=list(user_universities_campus))
])
])
])
qs = qs.query(query)
return qs
I'm getting this error:
if self.count == 0 and not self.allow_empty_first_page:
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\django\utils\functional.py", line 80, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\django\core\paginator.py", line 91, in count
return c()
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\elasticsearch_dsl\search.py", line 679, in count
**self._params
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\elasticsearch\client\utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\elasticsearch\client\__init__.py", line 529, in count
"POST", _make_path(index, doc_type, "_count"), params=params, body=body
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\elasticsearch\transport.py", line 358, in perform_request
timeout=timeout,
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\elasticsearch\connection\http_urllib3.py", line 261, in perform_request
self._raise_error(response.status, raw_data)
File "C:\Users\fendy\.virtualenvs\cul\lib\site-packages\elasticsearch\connection\base.py", line 182, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', '[terms] unknown token [END_ARRAY] after [university.id]')
Query String printout:
Bool(should=[Bool(must=[Match(visibility_scope=2), Bool(must=[Terms(university__id=[(42809,)]), Bool(should=[Terms(university__main_campus__id=[(42809,)]), Terms(university__main_campus__id=[(None,)])])])]), Match(visibility_scope=0)])
Your query in string should be like this:
{
"bool": {
"should":[
"bool" : {
"must":[
{"match":{
"visibility_scope" : 2
},
"bool":{
"must":[
{
"terms": {
}
....
.....
.....
}
]
}
}
]
}
]
}
}
Can you update your question with a querystring? I may be able to complete it.
It is because of queryset. Please set flat=True.
user_universities = user.userprofile.universities.all().values_list("id", flat=True)

grep including new line

I have a log file, and I want to get the content using pattern
Log file will look like this
2019-05-15 16:40 +07:00: data { data:
[ { audio_incremental_num: 1,
session_id: 'openrJEe7A_1557912549',
stream_time: 88,
duration: 291,
audio_id: '749f7c75-9fe1-4dbc-b5d8-770aadfe94bc'
version: '1.2' },
{ audio_incremental_num: 1,
session_id: 'openrJEe7A_1557912549',
stream_time: 88,
duration: 291,
audio_id: '749f7c75-9fe1-4dbc-b5d8-770aadfe94bc'
version: '1.2' }] }
2019-05-15 16:50 +07:00: data { data:
[ { audio_incremental_num: 1,
session_id: 'openrJEe7A_1557912549',
stream_time: 88,
duration: 291,
audio_id: '749f7c75-9fe1-4dbc-b5d8-770aadfe94bc'
version: '1.2' },
{ audio_incremental_num: 1,
session_id: 'openrJEe7A_1557912549',
stream_time: 88,
duration: 291,
audio_id: '749f7c75-9fe1-4dbc-b5d8-770aadfe94bc'
version: '1.2' }] }
I have tried using these but no luck
grep -zo '2019-05-[0-9][1-9] [0-9][0-9]:[0-9][0-9] +07:00: data { data:[[:space:]]'
grep -P '2019-05-[0-9]{2} [0-9]{2}:[0-9]{2} \+07:00: data { data:(\s.*)*.*'
Note: My log file actually is mixed with other log string content, so its not 100% JSON log
Your log file look like a json format you can use jq in bash to parse that is very useful check this link Working with JSON in bash using jq

Cookie for Django

I use Django 1.6 and the following error occurred:
Exception happened during processing of request from ('server ip ', 54687)
Traceback (most recent call last):
File "/usr/local/Python2.7.5/lib/python2.7/SocketServer.py", line 593, in process_request_thread
self.finish_request(request, client_address)
File "/usr/local/Python2.7.5/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/local/Python2.7.5/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 126, in __init__
super(WSGIRequestHandler, self).__init__(*args, **kwargs)
File "/usr/local/Python2.7.5/lib/python2.7/SocketServer.py", line 649, in __init__
self.handle()
File "/usr/local/Python2.7.5/lib/python2.7/wsgiref/simple_server.py", line 124, in handle
handler.run(self.server.get_app())
File "/usr/local/Python2.7.5/lib/python2.7/wsgiref/handlers.py", line 92, in run
self.close()
File "/usr/local/Python2.7.5/lib/python2.7/wsgiref/simple_server.py", line 33, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
I set cookie from client program. Why it happens?
It is client C# of Unity code to create cookie.
Dictionary<string,string> headers = www.responseHeaders;
foreach (KeyValuePair<string,string> kvPair in headers)
{
if (kvPair.Key.ToLower ().Equals ("set-cookie"))
{
string stHeader = "";
string stJoint = "";
string[] astCookie = kvPair.Value.Split (
new string[] {"; "}, System.StringSplitOptions.None);
foreach (string stCookie in astCookie)
{
if (!stCookie.Substring (0, 5).Equals ("path="))
{
stHeader += stJoint + stCookie;
stJoint = "; ";
}
}
if (stHeader.Length > 0)
{
this.hsHeader ["Cookie"] = stHeader;
}
else
{
this.hsHeader.Clear ();
}
}
}
And I set like this
WWW www = new WWW (openURL, null, this.hsHeader);
I think server side is okey.
Because, from browser it runs fine.
stHeader is like this
csrftoken=2YiHLunt9QzqIavUbxfhp2sPVU22g3Vv; expires=Sat, 09-Apr-2016 00:11:19 GMT; Max-Age=31449600

log errors in celery to django-sentry

How would I log errors to django-sentry from celery tasks?
I've tried try: except: blocks and using raven_client.create_from_exception but that blows up every time with:
Traceback (most recent call last):
File "/home/controlpanel/current/env/lib/python2.6/site-packages/celery/execute/trace.py", line 47, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/home/controlpanel/current/env/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
return self.run(*args, **kwargs)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
return fun(*args, **kwargs)
File "/home/controlpanel/deployments/1323265958/src/deploy/tasks.py", line 60, in async_deploy_bundle
raven_client.create_from_exception(exc_info=exc_info)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/contrib/django/__init__.py", line 120, in create_from_exception
return super(DjangoClient, self).create_from_exception(exc_info, **kwargs)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/base.py", line 285, in create_from_exception
frames = varmap(shorten, get_stack_info(iter_traceback_frames(exc_traceback)))
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/stacks.py", line 128, in get_stack_info
'vars': transform(frame.f_locals.items()),
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 78, in transform
ret = type(value)(transform_rec(o) for o in value)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 78, in <genexpr>
ret = type(value)(transform_rec(o) for o in value)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 72, in <lambda>
transform_rec = lambda o: transform(o, stack + [value], context)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 78, in transform
ret = type(value)(transform_rec(o) for o in value)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 78, in <genexpr>
ret = type(value)(transform_rec(o) for o in value)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 72, in <lambda>
transform_rec = lambda o: transform(o, stack + [value], context)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/raven/utils/encoding.py", line 78, in transform
ret = type(value)(transform_rec(o) for o in value)
File "/home/controlpanel/current/env/lib/python2.6/site-packages/git/util.py", line 588, in __init__
raise ValueError("First parameter must be a string identifying the name-property. Extend the list after initialization")
ValueError: First parameter must be a string identifying the name-property. Extend the list after initialization
My task:
#task
def async_deploy_bundle(bundle_id):
try:
do_stuff(bundle_id)
except:
exc_info = sys.exc_info()
traceback.print_exception(*exc_info)
try:
raven_client = get_client()
raven_client.create_from_exception(exc_info=exc_info)
except:
pass
raise
What about this:
myLogger = logging.getLogger('mylogger.info')
myLogger.setLevel(logging.INFO)
myLogger.info(exc_info)
?
In my settings.py:
'loggers': {
'mylogger.info': {
'level': 'INFO',
'handlers': ['sentry'],
'propagate': False,
},
}