Is it necessary to catch errors stemming from HTTP inputs? Is it ever a good idea to let it fail naturally (allow the exception to bubble up)?
I have a Django view for a server side interface for an AJAX call that looks something like this:
def some_view(request):
try:
some_int = int(request.POST.get('some_int')) # May raise ValueError or TypeError
except (ValueError, TypeError):
return HttpResponseBadRequest('some_int must be an int')
# ... Code that assumes some_int is an int
return HttpResponse('The normal response')
Is it ever acceptable in production code to have something like this?
def some_view(request):
some_int = int(request.POST.get('some_int')) # Ignore ValueError or TypeError raised
# ... Code that assumes some_int is an int
return HttpResponse('normal_response')
As I accept more parameters, I find that it is frustrating to maintain so many try/except blocks which are mostly the same and I end up with a ton of boiler plate code.
Of course I tried to refactor this into a separate function but since Django requires an HttpResponse to be returned, not raised as an exception, I can't plug it into a view without a try/except block. Also, conversions to int aren't the only thing I check... there are a lot of business logic sanity checks performed depending on the input as well. For example, I would validate that the JSON passed is of a specific format (i.e. array of objects of int array, etc.).
My views end up being 70+ lines of code just for sanity checks and a few lines of code that actually generate the response. Somehow I feel like there should be a more elegant way but I haven't found one so I'm considering forgoing all checks and just letting Django take care of it. Is this a bad idea?
I'm aware of the following potential problems if I don't catch the exceptions:
The same HTTP 500 is returned for all errors
If logging is enabled in production, it would probably log an error every time an invalid input occurs
Are there other problems I should be aware of? It just feels wrong not to catch exceptions from user inputs even though there's not much I can do about it in terms of recovery logic.
I think the best way to handle this is by writing your own middleware that catches the exceptions and turns them into your desired response.
That might look something like this:
# views.py
def some_view(request):
some_int = int(request.POST.get('some_int'))
# ... Code that assumes some_int is an int
return HttpResponse('normal_response')
# middleware.py
class MyValidationMiddleware(object):
def process_exception(self, request, e):
if isinstance(e, ValueError):
return HttpResponseBadRequest('Input did not validate')
else:
return None # let it bubble up
(Since middleware is site-wide, you might want to explicitly define your own input-validation exceptions to distinguish them from other sorts of exceptions that might occur.)
Alternatively, you could do the same sort of thing with per-view decorators.
Related
I'm building a very simple "to-do" API which only has 3 files: a controller with all the endpoints, a service with some logic to manipulate data and a DAO for DB access. So whenever an endpoint gets a request, it calls a method from the service and this method calls a DAO method to interact with the DB.
For example, the service has an add_item method which converts a field ("status") into an enum value, and a KeyError occurs when the value is something else. So my method goes something like this (self.dao is an instance of the DAO class and ValidStatus is the enum):
def add_item(self, name: str, status: str) -> int:
valid_status = ValidStatus[status.upper()]
inserted_id = self.dao.insert(item_name=name, item_status=valid_status)
return inserted_id
Since I'm now trying to handle exceptions, I modified the method to add a try-except block:
def add_item(self, name: str, status: str) -> Union[int, str]:
try:
valid_status = ValidStatus[status.upper()]
inserted_id = self.dao.insert_task(item_name=name, item_status=valid_status)
return inserted_id
except KeyError as e:
return (f"{e} status is not valid. Please enter {ValidStatus.TODO} or {ValidStatus.COMPLETED}.")
But I'm not sure this is a good approach, since now the method returns 2 different value types: the id of the new item if everthing is ok, or a string with the error message if the exception occurs. And I'm not sure that's the best choice, consistency-wise (and because all my type hinting would get messed up with Unions).
I thought another option would be to always return a tuple with a dict to create a json formatted response and an http code. So when everything goes well it would be something like return {"new item id": inserted_id}, 200 and when there's an exception it would be something like
return {"error": "Something went wrong"}, 400. But I'm not sure this is a good approach either, since then the service would be dealing with http status codes and json structured dicts, and that sounds more like a controller's responsibility.
So is there an advised best practice for this?
Well, generally speaking, there are two types of errors to handle, one is user input-related errors, such as the status key error in your case, which usually return 4xx, and the other is server-side processing errors, such as database errors, which usually return 5xx.
For the first type of error, we often use some kind of validator, such as marshmallow, and handle those errors in the controller. For example,
#bp.route("", methods=["POST"])
#login_required
def new_paper():
req = request.json
try:
cleaned_data = CreateUpdatePaperSchema(unknown=EXCLUDE).load(req)
except ValidationError as err:
return err_response(err_msg=fmt_validate_err(err), status_code=400)
For the second type of error, we often use a global exception catching mechanism to handle that uniformly. For example,
#app.errorhandler(Exception)
def handle_exception(exc):
db.session.rollback()
app.logger.error(exc, exc_info=True)
err_msg = "Something went wrong"
return err_response(err_msg=err_msg, status_code=500)
Currently my app is working with several external APIs, and those functions can raise some kind of error, and I would like to be aware of all kind of situations and don't crash my app. In the following piece of code you can see my function, using slack api (it's just an example, not real example), and you can see how I'm creating a space giving the name and the emails. That function can returns several errors, like APIerror, TypeError... etc etc
Also you can see, all the functions in that file, can raise some errors so it is using try and except and try to don't break our app. In the except part, I don't want to do anything special, just return the error or I don't know, raise an custom error. In the future would be cool if I can enqueue that task if something goes wrong.
slack_client.py
def create_space():
try:
slack_api.create.space(spaceName='space name', email=emails)
except (...):
...
def delete_space():
try:
slack_api.delete.space(spaceId=space_id)
except (...):
...
I would like to refactor the try and except part, because I don't want to copy and paste try and except during all my slack_client class, but in case the connection is down, or some parameter is wron, the "Slack" API will raise me an error.
I thought about create a class, to run each function inside try and catch, but it's a bit weird and I will loose the arguments part like "spaceName='space name'", which I think it's really cool and clear.
def execute_func(func, *args):
try:
func(args)
except (..):
...
So would like some help for this kind of situation, because handle external API is not always really easy... (at the moment working with 2 external APIs)
In a previous life I used VAX/VMS and still remember great pleasure at discoverting LIB$SIG_TO_RET which converted a signal to a return value. You can do exactly the same in not many lines of Python, along the lines you indicated
def execute_func(func, *args, **kwargs):
try:
result = func( *args, **kwargs)
return (True, result)
except Exception as exc:
return ( False, exc)
Usage
result = execute_func( func, a1, a2, ... k1=v1, k2=v2, ...)
if result[0]:
func_result = result[1] # if it's useful
...
else:
# it went wrong, deal with the exception
caught_exception = result[1]
print( caught_exception)
You can catch any exception that is thrown and capture it in a variable to log or raise. You could even raise your own custom exception. Something like this should work:
First create your exception:
class CustomSlackException(Exception):
pass
Then raise it when the API you called throws some exception:
try:
slack_api.create.space(spaceName='space name', email=emails)
except Exception as e:
print(e)
raise CustomSlackException("Custom error message!")
Now you can catch your exception and handle it as needed.
I'm trying to utilize django's row-level-locking by using the select_for_update utility. As per the documentation, this can only be used when inside of a transaction.atomic block. The side-effect of using a transaction.atomic block is that if my code throws an exception, all the database changes get rolled-back. My use case is such that I'd actually like to keep the database changes, and allow the exception to propagate. This leaves me with code looking like this:
with transaction.atomic():
user = User.objects.select_for_update.get(id=1234)
try:
user.do_something()
except Exception as e:
exception = e
else:
exception = None
if exception is not None:
raise exception
This feels like a total anti-pattern and I'm sure I must be missing something. I'm aware I could probably roll-my-own solution by manually using transaction.set_autocommit to manage the transaction, but I'd have thought that there would be a simpler way to get this functionality. Is there a built in way to achieve what I want?
I ended up going with something that looks like this:
from django.db import transaction
class ErrorTolerantTransaction(transaction.Atomic):
def __exit__(self, exc_type, exc_value, traceback):
return super().__exit__(None, None, None)
def error_tolerant_transaction(using=None, savepoint=True):
"""
Wraps a code block in an 'error tolerant' transaction block to allow the use of
select_for_update but without the effect of automatic rollback on exception.
Can be invoked as either a decorator or context manager.
"""
if callable(using):
return ErrorTolerantTransaction('default', savepoint)(using)
return ErrorTolerantTransaction(using, savepoint)
I can now put an error_tolerant_transaction in place of transaction.atomic and exceptions can be raised without a forced rollback. Of course database-related exceptions (i.e. IntegrityError) will still cause a rollback, but that's expected behavior given that we're using a transaction. As a bonus, this solution is compatible with transaction.atomic, meaning it can be nested inside an atomic block and vice-versa.
When I want to select objects with a get() function like
personalProfile = World.objects.get(ID=personID)
If get function doesn't return find a value, a "matching query does not exist." error occurs.
If I don't need this error, I'll use try and except function
try:
personalProfile = World.objects.get(ID=personID)
except:
pass
But I think this is not the best way since I use
except:
pass
Please recommend some idea or code sample to fight with this issue
That depends on what you want to do if it doesn't exist..
Theres get_object_or_404:
Calls get() on a given model manager, but it raises Http404 instead of the model’s DoesNotExist exception.
get_object_or_404(World, ID=personID)
Which is very close to the try except code you currently do.
Otherwise theres get_or_create:
personalProfile, created = World.objects.get_or_create(ID=personID)
Although, If you choose to continue with your current approach, at least make sure the except is localised to the correct error and then do something with that as necessary
try:
personalProfile = World.objects.get(ID=personID)
except MyModel.DoesNotExist:
raise Http404("No MyModel matches the given query.")
The above try/except handle is similar to what is found in the docs for get_object_or_404...
A get_or_none() function has been proposed, multiple times now. The rejection notice is feature creep, which you might or might not agree with. The functionality is present --with slightly different semantics-- in the first() queryset method.
But first things first:
The manager throws World.DoesNotExist, a specialized subclass of ObjectDoesNotExist when a World object was not found:
try:
personalProfile = World.objects.get(ID=personID)
except World.DoesNotExist:
pass
There's also get_object_or_404() which raises a Http404 exception when the object was not found.
You can also roll your own get_or_none(). A possible implementation could be:
def get_or_none(queryset, *args, **kwargs):
try:
return queryset.get(*args, **kwargs)
except ObjectDoesNotExist:
return None
Note that this still raises MultipleObjectsReturned when more than one matching object is found. If you always want the first object regardless of any others, you can simplify using first(), which returns None when the queryset is empty:
def get_or_none(queryset, *args, **kwargs):
return queryset.filter(*args, **kwargs).first()
Note however, for this to work reliably, you need a proper order for objects, because in the presence of multiple objects first() might be non-deterministic (it probably returns the first object from the database index used to filter the query and neither indexes not the underlying tables need be sorted or even have a repeatable order).
Use both, however, only when the use of the object to retrieve is strictly optional for the further program flow. When failure to retrieve an object is an error, use get_object_or_404(). When an object should be created when it does not exist, use get_or_create(). In those cases, both are better suited to simplify program flow.
As alasdair mentioned you could use the built in first() method.
It returns the object if it exists or None if it's not
personalProfile = World.objects.filter(ID=personID).first()
I would like to know how to show personalized errors with the get_object_or_404 method. I don't want the normal Http404 pages, but I want to display a custom message with the message: the result is none.
Thanks :)
The get_object_or_404() is essentially a simple 5-line function. Unless you have some specific reason for using it, just do:
try:
instance = YourModel.objects.get(pk=something)
except YourModel.DoesNotExist:
return render_to_response('a_template_with_your_error_message.html')
If for whatever reason you have to use get_object_or_404(), you can try putting it in a try: ... except Http404: ... block, but I honestly can't think of a plausible reason for that.
As stated by michael, when using get_object_or_404 you cannot customize the message given on http 404. The message provided in DEBUG does offer information about the exception however: "No MyModel matches the given query."
Check out the doc on this. There are three arguments: Model, *args, and **kwargs. The last two are used to build an argument for either get() or filter() on the Model.
The reason I wrote, however, is to address the question of why we would want to use a helper function such as get_object_or_404() instead of, for example, catching it with an exception like Model.DoesNotExist.
The later solution couples the view layer to the model layer. In an effort to relax this coupling we can take advantage of the controlled coupling offered in the django.shortcuts module[1].
And why exactly aren't you using your server's capeability to do just that?
get_object_or_404() is redirecting to the default 404 page right?
If you are on debug mode you won't see it but when deployed django will just refer to the server's 404 html page.
That can't be done with that shortcut. It will only raise a Http404 exception. Your best bet is a try catch if you want full control. Eg.
try:
obj = Model.objects.get(pk = foo)
except:
return HttpResponseRedirect('/no/foo/for/you')
#or
return render_to_response ...