i need a little bit of help understanding a problem that i have with user defined exceptions in python 2.7.11.
I have two files main.py and myErrors.py .in main i post data and receive a response and and in myErrors i handle the errors.
What i'm trying to do is execute the version error in the try:except statement, but it doesn't get executed even thought it should be. what i'm doing is that i pass the response to myErrors and update that data to a dictionary in the errors file.-
my question was badly phrased. What I want to do is, is pass the response to the error handler, but i don't want to execute it, until we get to the Try:except clause in on_response method. So when we get the response and if it's not successful, then check the error code and raise the exception. Now what i'm doing is checking first for errors and then executing the check for success (error code)
Here is the main
def send_messages(self):
response = cm.postData(url=simulateSasServer, jsondata=json_data)
self.on_response(response)
def on_response(self, response):
myERRORS.myERRORS(response)
# if registration is succesful change state to REGISTERED.
if 'registrationResponse' in response:
try:
responseObjects = response['registrationResponse']
for responseObject in responseObjects:
if responseObject['error']['errorCode'] == 0:
do_action
except myErrors.Version():
raise ("version_message")
Here is the myErrors
class myERRORS(Exception):
error_code = {'SUCCESS': 0,
'VERSION': 100,
}
response_data = {}
def __init__(self, response):
self.response_data.update(response)
class Version(myERRORS):
def __init__(self):
self.name = "VERSION"
self.err_code = self.error_code['VERSION']
self.msg = "SAS protocol version used by CBSD is not supported by SAS"
self.version_error()
if self.version_error() is True:
print (self.name, self.err_code, self.msg)
raise Exception(self.name, self.err_code, self.msg)
def version_error(self):
response_objects = self.response_data.values()[0]
if 'registrationResponse' in self.response_data:
for r_object in response_objects:
if r_object['error']['errorCode'] == self.error_code['VERSION']:
return True
Any help is much appreciated.
There isn't really anything special about exceptions. They are classes. What you did is create an instance of a class. You did not raise it. Change:
myERRORS.myERRORS(response)
to:
raise myERRORS.myERRORS(response)
Related
I have a custom prediction routine in google-ml-engine. Works very well.
I now am doing input checking on the instance data, and want to return error responses from my predict routine.
The example: https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines
Raises exceptions on input errors, etc. However, when this happens the response body always has {'error': Prediction failed: unknown error}. I can see the correct errors are being logged in google cloud console, but the https response is always the same unknown error.
My question is:
How to make the Custom prediction routine return a proper error code and error message string?
Instead of returning a prediction, I can return an error string/code in prediction -but it ends up in the prediction part of the response which seems hacky and doesn't get any of the google errors eg based on instance size.
root:test_deployment.py:35 {'predictions': {'error': "('Instance does not include required sensors', 'occurred at index 0')"}}
What's the best way to do this?
Thanks!
David
Please take a look at the following code, I created a _validate function inside predict and use a custom Exception class.
Basically, I validate instances, before I call the model predict method and handle the exception.
There may be some overhead to the response time when doing this validation, which you need to test for your use case.
requests = [
"god this episode sucks",
"meh, I kinda like it",
"what were the writer thinking, omg!",
"omg! what a twist, who would'v though :o!",
99999
]
api = discovery.build('ml', 'v1')
parent = 'projects/{}/models/{}/versions/{}'.format(PROJECT, MODEL_NAME, VERSION_NAME)
parent = 'projects/{}/models/{}'.format(PROJECT, MODEL_NAME)
response = api.projects().predict(body=request_data, name=parent).execute()
{'predictions': [{'Error code': 1, 'Message': 'Invalid instance type'}]}
Custom Prediction class:
import os
import pickle
import numpy as np
import logging
from datetime import date
import tensorflow.keras as keras
class CustomModelPredictionError(Exception):
def __init__(self, code, message='Error found'):
self.code = code
self.message = message # you could add more args
def __str__(self):
return str(self.message)
def isstr(s):
return isinstance(s, str) or isinstance(s, bytes)
def _validate(instances):
for instance in instances:
if not isstr(instance):
raise CustomModelPredictionError(1, 'Invalid instance type')
return instances
class CustomModelPrediction(object):
def __init__(self, model, processor):
self._model = model
self._processor = processor
def _postprocess(self, predictions):
labels = ['negative', 'positive']
return [
{
"label":labels[int(np.round(prediction))],
"score":float(np.round(prediction, 4))
} for prediction in predictions]
def predict(self, instances, **kwargs):
try:
instances = _validate(instances)
except CustomModelPredictionError as c:
return [{"Error code": c.code, "Message": c.message}]
else:
preprocessed_data = self._processor.transform(instances)
predictions = self._model.predict(preprocessed_data)
labels = self._postprocess(predictions)
return labels
#classmethod
def from_path(cls, model_dir):
model = keras.models.load_model(
os.path.join(model_dir,'keras_saved_model.h5'))
with open(os.path.join(model_dir, 'processor_state.pkl'), 'rb') as f:
processor = pickle.load(f)
return cls(model, processor)
Complete code in this notebook.
If it is still relevant to you, I found a way by using google internal libraries (not sure if it would be endorsed by Google though).
AI platform custom prediction wrapping code only returns custom error message if the Exception thrown is a specific one from their internal library.
It might also not be super reliable as you would have very little control in case Google wants to change it.
class Predictor(object):
def predict(self, instances, **kwargs):
# Your prediction code here
# This is an internal google library, it should be available at prediction time.
from google.cloud.ml.prediction import prediction_utils
raise prediction_utils.PredictionError(0, "Custom error message goes here")
#classmethod
def from_path(cls, model_dir):
# Your logic to load the model here
You would get the following message in your HTTP response
Prediction failed: Custom error message goes here
I have been adapting this article How to Add Custom Action Buttons to Django Admin to my particular circumstance and things are working pretty well except for this one strange error I am getting. In my forms.py I have defined class DenyForm: (Note the request parameter in form_action is not the usual API request. I have a model class Request.) What should form_action be returning?
class DenyForm(ApproveOrDenyForm):
def form_action(self, request, admin_approver ):
print "forms DenyForm Called."
justification = self.cleaned_data['justification']
request.admin_deny(admin_approver=admin_approver,justification=justification)
return [request]
#return request.admin_deny(admin_approver=admin_approver,justification=justification)
This results in a message of
“‘Please correct the errors below.’ need more than 1 value to unpack”.
I have tried different endings to my form_action method. If my last line is:
return request
I get “‘Please correct the errors below.’ ‘Request’ object is not iterable”.
If the last line is:
return request.admin_deny(admin_approver=admin_approver,justification=justification)
I get this … “‘Please correct the errors below.’ ‘NoneType’ object is not iterable”. That is because admin_deny() does not return anything. What should form_action be returning?
Update: Here is the code that call form_action():
class ApproveOrDenyForm(forms.Form):
justification = forms.CharField(
required=True,
widget=forms.Textarea,
)
def save(self, req, login ):
try:
user = User.objects.filter(login=login).get()
req, action = self.form_action(req, user )
except Exception as e:
error_message = str(e)
self.add_error(None, error_message)
raise
return req, action
When you call form_action you are expecting it to return an iterable of two items - req and action. So you need to ensure that form_action() returns a list or tuple of exactly those two things.
It is not clear to me from the post you linked to, or your code, what the action returned is supposed to be - maybe just a result of the action performed. You need to check that to decide whether your action method needs to return something else.
Something like this should work:
def form_action(self, request, admin_approver ):
justification = self.cleaned_data['justification']
action = request.admin_deny(admin_approver=admin_approver, justification=justification)
# action will be None if admin_deny does not return anything,
# but the code calling this function expects it, so return it anyway.
return (request, action)
How could one handle exceptions globally with Flask? I have found ways to use the following to handle custom db interactions:
try:
sess.add(cat2)
sess.commit()
except sqlalchemy.exc.IntegrityError, exc:
reason = exc.message
if reason.endswith('is not unique'):
print "%s already exists" % exc.params[0]
sess.rollback()
The problem with try-except is I would have to run that on every aspect of my code. I can find better ways to do that for custom code. My question is directed more towards global catching and handling for:
apimanager.create_api(
Model,
collection_name="models",
**base_writable_api_settings
)
I have found that this apimanager accepts validation_exceptions: [ValidationError] but I have found no examples of this being used.
I still would like a higher tier of handling that effects all db interactions with a simple concept of "If this error: show this, If another error: show something else" that just runs on all interactions/exceptions automatically without me including it on every apimanager (putting it in my base_writable_api_settings is fine I guess). (IntegrityError, NameError, DataError, DatabaseError, etc)
I tend to set up an error handler on the app that formats the exception into a json response. Then you can create custom exceptions like UnauthorizedException...
class Unauthorized(Exception):
status_code = 401
#app.errorhandler(Exception)
def _(error):
trace = traceback.format_exc()
status_code = getattr(error, 'status_code', 400)
response_dict = dict(getattr(error, 'payload', None) or ())
response_dict['message'] = getattr(error, 'message', None)
response_dict['traceback'] = trace
response = jsonify(response_dict)
response.status_code = status_code
traceback.print_exc(file=sys.stdout)
return response
You can also handle specific exceptions using this pattern...
#app.errorhandler(ValidationError)
def handle_validation_error(error):
# Do something...
Error handlers get attached to the app, not the apimanager. You probably have something like
app = Flask()
apimanager = ApiManager(app)
...
Put this somewhere using that app object.
My preferred approach uses decorated view-functions.
You could define a decorator like the following:
def handle_exceptions(func):
#wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except ValidationError:
# do something
except HTTPException:
# do something else ...
except MyCustomException:
# do a third thing
Then you can simply decorate your view-functions, e.g.
#app.route('/')
#handle_exceptions
def index():
# ...
I unfortunately do not know about the hooks Flask-Restless offers for passing view-functions.
I'm new to celery and an overall python noob. I must have stumbled upon the right solution during my research but I just don't seem to understand what I need to do for what seems to be a simple case scenario.
I followed the following guide to learn about flask+celery.
What I understand:
There seems there is something obvious I'm missing about how to trigger a task after the first one is finished. I tried using callbacks, using loops, even tried using Celery Flower and Celery beat to realise this has nothing with what I'm doing...
Goal:
After filling the form, I want to send an email with attachements (result of the task) or a failure email otherwise. Without having to wonder what my user is doing on the app (no HTTP requests)
My code:
class ClassWithTheTask:
def __init__(self, filename, proxies):
# do stuff until a variable results is created
self.results = 'this contains my result'
#app.route('/', methods=['GET', 'POST'])
#app.route('/index', methods=['GET', 'POST'])
def index():
form = MyForm()
if form.validate_on_submit():
# ...
# the task
my_task = task1.delay(file_path, proxies)
return redirect(url_for('taskstatus', task_id=my_task.id, filename=filename, email=form.email.data))
return render_template('index.html',
form=form)
#celery.task(bind=True)
def task1(self, filepath, proxies):
task = ClassWithTheTask(filepath, proxies)
return results
#celery.task
def send_async_email(msg):
"""Background task to send an email with Flask-Mail."""
with app.app_context():
mail.send(msg)
#app.route('/status/<task_id>/<filename>/<email>')
def taskstatus(task_id, filename, email):
task = task1.AsyncResult(task_id)
if task.state == 'PENDING':
# job did not start yet
response = {
'state': task.state,
'status': 'Pending...'
}
elif task.state != 'FAILURE':
response = {
'state': task.state,
'status': task.info.get('status', '')
}
if 'results' in task.info:
response['results'] = task.info['results']
response['untranslated'] = task.info['untranslated']
msg = Message('Task Complete for %s !' % filename,
recipients=[email])
msg.body = 'blabla'
with app.open_resource(response['results']) as fp:
msg.attach(response['results'], "text/csv", fp.read())
with app.open_resource(response['untranslated']) as fp:
msg.attach(response['untranslated'], "text/csv", fp.read())
# the big problem here is that it will send the email only if the user refreshes the page and get the 'SUCCESS' status.
send_async_email.delay(msg)
flash('task finished. sent an email.')
return redirect(url_for('index'))
else:
# something went wrong in the background job
response = {
'state': task.state,
'status': str(task.info), # this is the exception raised
}
return jsonify(response)
I don't get the goal of your method for status check. Anyway what you are describing can be accomplished this way.
if form.validate_on_submit():
# ...
# the task
my_task = (
task1.s(file_path, proxies).set(link_error=send_error_email.s(filename, error))
| send_async_email.s()
).delay()
return redirect(url_for('taskstatus', task_id=my_task.id, filename=filename, email=form.email.data))
Then your error task will look like this. The normal task can stay the way it is.
#celery.task
def send_error_email(task_id, filename, email):
task = AsyncResult(task_id)
.....
What happens here is that you are using a chain. You are telling Celery to run your task1, if that completes successfully then run send_async_email, if it fails run send_error_email. This should work, but you might need to adapt the parameters, consider it as pseudocode.
This does not seem right at all:
def task1(self, filepath, proxies):
task = ClassWithTheTask(filepath, proxies)
return results
The line my_task = task1.delay(file_path, proxies) earlier in your code suggests you want to return task but you return results which is not defined anywhere. (ClassWithTheTask is also undefined). This code would crash, and your task would never execute.
I am attempting to asynchronously consume a web service because it takes up to 45 seconds to return. Unfortunately, this web service is also somewhat unreliable and can throw errors. I have set up django-celery and have my tasks executing, which works fine until the task fails beyond max_retries.
Here is what I have so far:
#task(default_retry_delay=5, max_retries=10)
def request(xml):
try:
server = Client('https://www.whatever.net/RealTimeService.asmx?wsdl')
xml = server.service.RunRealTimeXML(
username=settings.WS_USERNAME,
password=settings.WS_PASSWORD,
xml=xml
)
except Exception, e:
result = Result(celery_id=request.request.id, details=e.reason, status="i")
result.save()
try:
return request.retry(exc=e)
except MaxRetriesExceededError, e:
result = Result(celery_id=request.request.id, details="Max Retries Exceeded", status="f")
result.save()
raise
result = Result(celery_id=request.request.id, details=xml, status="s")
result.save()
return result
Unfortunately, MaxRetriesExceededError is not being thrown by retry(), so I'm not sure how to handle the failure of this task. Django has already returned HTML to the client, and I am checking the contents of Result via AJAX, which is never getting to a full fail f status.
So the question is: How can I update my database when the Celery task has exceeded max_retries?
The issue is that celery is trying to re-raise the exception you passed in when it hits the retry limit. The code for doing this re-raising is here: https://github.com/celery/celery/blob/v3.1.20/celery/app/task.py#L673-L681
The simplest way around this is to just not have celery manage your exceptions at all:
#task(max_retries=10)
def mytask():
try:
do_the_thing()
except Exception as e:
try:
mytask.retry()
except MaxRetriesExceededError:
do_something_to_handle_the_error()
logger.exception(e)
You can override the after_return method of the celery task class, this method is called after the execution of the task whatever is the ret status (SUCCESS,FAILED,RETRY)
class MyTask(celery.task.Task)
def run(self, xml, **kwargs)
#Your stuffs here
def after_return(self, status, retval, task_id, args, kwargs, einfo=None):
if self.max_retries == int(kwargs['task_retries']):
#If max retries are equals to task retries do something
if status == "FAILURE":
#You can do also something if the tasks fail instead of check the retries
http://readthedocs.org/docs/celery/en/latest/reference/celery.task.base.html#celery.task.base.BaseTask.after_return
http://celery.readthedocs.org/en/latest/reference/celery.app.task.html?highlight=after_return#celery.app.task.Task.after_return
With Celery version 2.3.2 this approach has worked well for me:
class MyTask(celery.task.Task):
abstract = True
def after_return(self, status, retval, task_id, args, kwargs, einfo):
if self.max_retries == self.request.retries:
#If max retries is equal to task retries do something
#task(base=MyTask, default_retry_delay=5, max_retries=10)
def request(xml):
#Your stuff here
I'm just going with this for now, spares me the work of subclassing Task and is easily understood.
# auto-retry with delay as defined below. After that, hook is disabled.
#celery.shared_task(bind=True, max_retries=5, default_retry_delay=300)
def post_data(self, hook_object_id, url, event, payload):
headers = {'Content-type': 'application/json'}
try:
r = requests.post(url, data=payload, headers=headers)
r.raise_for_status()
except requests.exceptions.RequestException as e:
if self.request.retries >= self.max_retries:
log.warning("Auto-deactivating webhook %s for event %s", hook_object_id, event)
Webhook.objects.filter(object_id=hook_object_id).update(active=False)
return False
raise self.retry(exc=e)
return True