Flask API: what's the best way to handle exceptions? - flask

I'm building a very simple "to-do" API which only has 3 files: a controller with all the endpoints, a service with some logic to manipulate data and a DAO for DB access. So whenever an endpoint gets a request, it calls a method from the service and this method calls a DAO method to interact with the DB.
For example, the service has an add_item method which converts a field ("status") into an enum value, and a KeyError occurs when the value is something else. So my method goes something like this (self.dao is an instance of the DAO class and ValidStatus is the enum):
def add_item(self, name: str, status: str) -> int:
valid_status = ValidStatus[status.upper()]
inserted_id = self.dao.insert(item_name=name, item_status=valid_status)
return inserted_id
Since I'm now trying to handle exceptions, I modified the method to add a try-except block:
def add_item(self, name: str, status: str) -> Union[int, str]:
try:
valid_status = ValidStatus[status.upper()]
inserted_id = self.dao.insert_task(item_name=name, item_status=valid_status)
return inserted_id
except KeyError as e:
return (f"{e} status is not valid. Please enter {ValidStatus.TODO} or {ValidStatus.COMPLETED}.")
But I'm not sure this is a good approach, since now the method returns 2 different value types: the id of the new item if everthing is ok, or a string with the error message if the exception occurs. And I'm not sure that's the best choice, consistency-wise (and because all my type hinting would get messed up with Unions).
I thought another option would be to always return a tuple with a dict to create a json formatted response and an http code. So when everything goes well it would be something like return {"new item id": inserted_id}, 200 and when there's an exception it would be something like
return {"error": "Something went wrong"}, 400. But I'm not sure this is a good approach either, since then the service would be dealing with http status codes and json structured dicts, and that sounds more like a controller's responsibility.
So is there an advised best practice for this?

Well, generally speaking, there are two types of errors to handle, one is user input-related errors, such as the status key error in your case, which usually return 4xx, and the other is server-side processing errors, such as database errors, which usually return 5xx.
For the first type of error, we often use some kind of validator, such as marshmallow, and handle those errors in the controller. For example,
#bp.route("", methods=["POST"])
#login_required
def new_paper():
req = request.json
try:
cleaned_data = CreateUpdatePaperSchema(unknown=EXCLUDE).load(req)
except ValidationError as err:
return err_response(err_msg=fmt_validate_err(err), status_code=400)
For the second type of error, we often use a global exception catching mechanism to handle that uniformly. For example,
#app.errorhandler(Exception)
def handle_exception(exc):
db.session.rollback()
app.logger.error(exc, exc_info=True)
err_msg = "Something went wrong"
return err_response(err_msg=err_msg, status_code=500)

Related

Refactor try and Except codeblocks in Python

Currently my app is working with several external APIs, and those functions can raise some kind of error, and I would like to be aware of all kind of situations and don't crash my app. In the following piece of code you can see my function, using slack api (it's just an example, not real example), and you can see how I'm creating a space giving the name and the emails. That function can returns several errors, like APIerror, TypeError... etc etc
Also you can see, all the functions in that file, can raise some errors so it is using try and except and try to don't break our app. In the except part, I don't want to do anything special, just return the error or I don't know, raise an custom error. In the future would be cool if I can enqueue that task if something goes wrong.
slack_client.py
def create_space():
try:
slack_api.create.space(spaceName='space name', email=emails)
except (...):
...
def delete_space():
try:
slack_api.delete.space(spaceId=space_id)
except (...):
...
I would like to refactor the try and except part, because I don't want to copy and paste try and except during all my slack_client class, but in case the connection is down, or some parameter is wron, the "Slack" API will raise me an error.
I thought about create a class, to run each function inside try and catch, but it's a bit weird and I will loose the arguments part like "spaceName='space name'", which I think it's really cool and clear.
def execute_func(func, *args):
try:
func(args)
except (..):
...
So would like some help for this kind of situation, because handle external API is not always really easy... (at the moment working with 2 external APIs)
In a previous life I used VAX/VMS and still remember great pleasure at discoverting LIB$SIG_TO_RET which converted a signal to a return value. You can do exactly the same in not many lines of Python, along the lines you indicated
def execute_func(func, *args, **kwargs):
try:
result = func( *args, **kwargs)
return (True, result)
except Exception as exc:
return ( False, exc)
Usage
result = execute_func( func, a1, a2, ... k1=v1, k2=v2, ...)
if result[0]:
func_result = result[1] # if it's useful
...
else:
# it went wrong, deal with the exception
caught_exception = result[1]
print( caught_exception)
You can catch any exception that is thrown and capture it in a variable to log or raise. You could even raise your own custom exception. Something like this should work:
First create your exception:
class CustomSlackException(Exception):
pass
Then raise it when the API you called throws some exception:
try:
slack_api.create.space(spaceName='space name', email=emails)
except Exception as e:
print(e)
raise CustomSlackException("Custom error message!")
Now you can catch your exception and handle it as needed.

why is Django MultipleObjectsReturned Error hogging memory?

I am using Django 1.9.10:
I have a model called Details has a unique_id column which is indexed:
try:
detail = Detail.objects.get(unique_id=UUID)
except MultipleObjectsReturned as m:
logger.error("uuid {} returned multiple objects - {}".format(UUID, str(m)))
Due to some error in code UUID=None this resulted in MultipleObjectsReturned error getting raised. but we noticed that almost 2-GB of memory is getting used up and it is slowing the system down a lot.
On printing str(m) in the error logs we found following error
MultipleObjectsReturned: get() returned more than one Details -- it returned 451424!
My Question is why is Django fetching so much data in memory just to raise an error? Django can just fetch the count?
I know I can use filter() to over come this issue but I am just surprised by this and want to understand why django is doing this?
Because that's how it's done internally:
def get(self, *args, **kwargs):
"""
Perform the query and return a single object matching the given
keyword arguments.
"""
clone = self.filter(*args, **kwargs) # calling `filter()` inside
if self.query.can_filter() and not self.query.distinct_fields:
clone = clone.order_by()
num = len(clone)
if num == 1:
return clone._result_cache[0]
if not num:
raise self.model.DoesNotExist(
"%s matching query does not exist." %
self.model._meta.object_name
)
raise self.model.MultipleObjectsReturned(
"get() returned more than one %s -- it returned %s!" %
(self.model._meta.object_name, num)
)
See the full source here.
I guess you wonder why it can't just fetch the number? Because it will make two requests to the database instead of one. One request to fetch the count and one request to fetch the data.
How can you fix it? You may change your application logic to avoid this situation. Assert that UUID is not None. Or fetch count before making an actual query.
Docs about get: https://docs.djangoproject.com/en/dev/topics/db/queries/#retrieving-a-single-object-with-get
Because you use .get when you know you shouldn't get multiple objects.
Choosing to use it in a case where 451424 items are returned instead isn't something it was designed for.
Because you have more then one with same ID in database.
Rewrite your query to this: Detail.objects.filter(unique_id=UUID)

Should unrecoverable exceptions from HTTP parameter problems be caught?

Is it necessary to catch errors stemming from HTTP inputs? Is it ever a good idea to let it fail naturally (allow the exception to bubble up)?
I have a Django view for a server side interface for an AJAX call that looks something like this:
def some_view(request):
try:
some_int = int(request.POST.get('some_int')) # May raise ValueError or TypeError
except (ValueError, TypeError):
return HttpResponseBadRequest('some_int must be an int')
# ... Code that assumes some_int is an int
return HttpResponse('The normal response')
Is it ever acceptable in production code to have something like this?
def some_view(request):
some_int = int(request.POST.get('some_int')) # Ignore ValueError or TypeError raised
# ... Code that assumes some_int is an int
return HttpResponse('normal_response')
As I accept more parameters, I find that it is frustrating to maintain so many try/except blocks which are mostly the same and I end up with a ton of boiler plate code.
Of course I tried to refactor this into a separate function but since Django requires an HttpResponse to be returned, not raised as an exception, I can't plug it into a view without a try/except block. Also, conversions to int aren't the only thing I check... there are a lot of business logic sanity checks performed depending on the input as well. For example, I would validate that the JSON passed is of a specific format (i.e. array of objects of int array, etc.).
My views end up being 70+ lines of code just for sanity checks and a few lines of code that actually generate the response. Somehow I feel like there should be a more elegant way but I haven't found one so I'm considering forgoing all checks and just letting Django take care of it. Is this a bad idea?
I'm aware of the following potential problems if I don't catch the exceptions:
The same HTTP 500 is returned for all errors
If logging is enabled in production, it would probably log an error every time an invalid input occurs
Are there other problems I should be aware of? It just feels wrong not to catch exceptions from user inputs even though there's not much I can do about it in terms of recovery logic.
I think the best way to handle this is by writing your own middleware that catches the exceptions and turns them into your desired response.
That might look something like this:
# views.py
def some_view(request):
some_int = int(request.POST.get('some_int'))
# ... Code that assumes some_int is an int
return HttpResponse('normal_response')
# middleware.py
class MyValidationMiddleware(object):
def process_exception(self, request, e):
if isinstance(e, ValueError):
return HttpResponseBadRequest('Input did not validate')
else:
return None # let it bubble up
(Since middleware is site-wide, you might want to explicitly define your own input-validation exceptions to distinguish them from other sorts of exceptions that might occur.)
Alternatively, you could do the same sort of thing with per-view decorators.

How to use exception handling in Django view

Suppose i have this code
if form.is_valid():
form.save()
Now suppose my form is valid i have exception that foregin key value is linked to more than one column so it will raise exception
Now i want to know is there any way to grab that exception value and pass to jquery via AJAX
Because form is valid so it comes inside the loop but its not able to go after form.save
So how can i program that if exception occurs it get passed to jquery like
if exception
return HttpResponse(exception)
I get this exception
MultipleObjectsReturned at
/manage/Source/create/ get() returned
more than one Account -- it returned
3! Lookup parameters were
{'account_number':
u'121121'}
What type of exception it is
MultipleObjectsReturned is the exception.
try:
#do something
except MultipleObjectsReturned:
return HttpResponse('MultipleObjectsReturned')
I wouldn't recommend using a bare try/except to catch all exceptions, as you won't know exactly what's wrong.

how show personalized error with get_object_or_404

I would like to know how to show personalized errors with the get_object_or_404 method. I don't want the normal Http404 pages, but I want to display a custom message with the message: the result is none.
Thanks :)
The get_object_or_404() is essentially a simple 5-line function. Unless you have some specific reason for using it, just do:
try:
instance = YourModel.objects.get(pk=something)
except YourModel.DoesNotExist:
return render_to_response('a_template_with_your_error_message.html')
If for whatever reason you have to use get_object_or_404(), you can try putting it in a try: ... except Http404: ... block, but I honestly can't think of a plausible reason for that.
As stated by michael, when using get_object_or_404 you cannot customize the message given on http 404. The message provided in DEBUG does offer information about the exception however: "No MyModel matches the given query."
Check out the doc on this. There are three arguments: Model, *args, and **kwargs. The last two are used to build an argument for either get() or filter() on the Model.
The reason I wrote, however, is to address the question of why we would want to use a helper function such as get_object_or_404() instead of, for example, catching it with an exception like Model.DoesNotExist.
The later solution couples the view layer to the model layer. In an effort to relax this coupling we can take advantage of the controlled coupling offered in the django.shortcuts module[1].
And why exactly aren't you using your server's capeability to do just that?
get_object_or_404() is redirecting to the default 404 page right?
If you are on debug mode you won't see it but when deployed django will just refer to the server's 404 html page.
That can't be done with that shortcut. It will only raise a Http404 exception. Your best bet is a try catch if you want full control. Eg.
try:
obj = Model.objects.get(pk = foo)
except:
return HttpResponseRedirect('/no/foo/for/you')
#or
return render_to_response ...