Manually fire Django 1.3's traceback/exception log - django

I'm using djutils's async decorator, which has the nasty side effect of not sending traceback emails when an exception is raised, since it runs on a separate thread.
It does, however, have the following place to put a logger.
def worker_thread():
while 1:
func, args, kwargs = queue.get()
try:
func(*args, **kwargs)
except:
pass # <-- log error here
finally:
queue.task_done()
I've confirmed this will work, but even with the try/except removed, it won't trip Django's traceback logger.
While it'd be pretty easy to tell it to write to a db/file on exception, I'd really like it to send a regular traceback as defined in settings. How can I do that?
Edit: answer seems to involve django.utils.log.AdminEmailHandler - but I'm having a hard time finding an example.
Edit 2: Here's my current (99% likely to be wrong) attempt.
from django.utils.log import AdminEmailHandler
def worker_thread():
while 1:
func, args, kwargs = queue.get()
try:
func(*args, **kwargs)
except:
import logging
from django.conf import settings
print settings.EMAIL_HOST
logger = logging.getLogger("async.logger")
logger.exception("Async exploded")
AdminEmailHandler
pass # <-- log error here
finally:
queue.task_done()

first, configure your logging settings i settings.py:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler'
}
},
'loggers': {
'myproject': {
'handlers': ['mail_admins'],
'level': 'INFO',
'propagate': True,
},
'django.request': {
'handlers': ['mail_admins'],
'level': 'ERROR',
'propagate': True,
},
}
}
from now, all loggers which starts with 'myproject' should use AdminEmailHandler
your code should look like this:
import logging
logger = logging.getLogger('myproject.optional.path')
# example
# logger = logging.getLogger('myprojects.myapp.views')
def worker_thread():
while 1:
func, args, kwargs = queue.get()
try:
func(*args, **kwargs)
except:
logger.exception("Async exploded")
finally:
queue.task_done()

Related

Why my Django exception middleware does not log stack trace

I am trying to set up a custom exception handler.
# settings.py
LOGGING = {
// ...
'handlers': {
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
},
},
'loggers': {
'django': {
'handlers': ['applogfile', 'mail_admins'],
'level': 'ERROR',
'propagate': True,
}
}
}
When I use the Django default exception handler and there is a Django error, I get an email containing the debug page, including the stack trace. However, I want to have my own exception middleware, so I added the following:
# middleware.py
from django.utils.deprecation import MiddlewareMixin
class MyExceptionMiddleware(MiddlewareMixin):
def process_exception(self, request, exception):
return render(request, template_name='error_message.html', context={
'title':'Error',
'message':'Sorry, we encountered an unexpected error'
})
# settings.py
MIDDLEWARE = [
# other middlewares
...
'middleware.MyExceptionMiddleware',
]
After I added this, the debug email that I get is missing the stack trace. It tells me that there is an "Internal server error" at the url, but it doesn't give any details of the errors.
I know I can override error handlers (https://docs.djangoproject.com/en/3.0/topics/http/views/#customizing-error-views) which would give me the full stack trace, but I am wondering why when I use a middleware to catch exceptions I don't get the stack trace.

Log SQL queries even if DEBUG=False

This is my logging config in settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
'file': {
'class': 'logging.FileHandler',
'filename': os.path.join(BASE_DIR, 'logs', 'django.log'),
},
},
'loggers': {
'django': {
'handlers': ['file', 'console'],
'level': 'DEBUG',
},
'django.template': {
'handlers': ['file', 'console'],
'level': 'INFO',
},
'App': {
'handlers': ['file', 'console'],
'level': 'DEBUG',
},
},
}
There's a behavior that I can't explain: if I run with debug=True I can see all SQL queries being logged to the console, but when debug=False that doesn't happen, even if I don't change the configuration above.
Why is this? How can I determine in my logging config whether or not SQL queries are logged to the console?
Query logging goes way deeper as Django wraps the database cursor based on the truthiness of a computed property.
From Line 226, django.db.backends.base.base v2.2.6 sources
def _prepare_cursor(self, cursor):
"""
Validate the connection is usable and perform database cursor wrapping.
"""
self.validate_thread_sharing()
if self.queries_logged:
wrapped_cursor = self.make_debug_cursor(cursor)
else:
wrapped_cursor = self.make_cursor(cursor)
return wrapped_cursor
This computed property gets to be decided on where debugging is forced or debugging is enabled in project settings
From Line 149, django.db.backends.base.base v2.2.6 sources
#property
def queries_logged(self):
return self.force_debug_cursor or settings.DEBUG
You should be able to get a proxy to the default connection and force the debug cursor to be used.
from django.db import connection
connection.force_debug_cursor = True
I however advise against this approach and favor query audits in the database.
That's because existing loggers already filtered your logs. In this case is Django's logger.
DEFAULT_LOGGING = {
'filters': {
'require_debug_true': {
'()': 'django.utils.log.RequireDebugTrue',
},
},
'handlers': {
'console': {
'level': 'INFO',
'filters': ['require_debug_true'],
'class': 'logging.StreamHandler',
},
...
}
and django.utils.log.RequireDebugTrue
class RequireDebugTrue(logging.Filter):
def filter(self, record):
return settings.DEBUG
To solve this issue, you could disable existing loggers by set disable_existing_loggers: True or you can override the filter like so:
LOGGING = {
...
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'filters': [],
},
...
}
Step 1: Write patch_django function in patchers.py
# Created by BaiJiFeiLong#gmail.com at 2022/6/6
import importlib
import importlib.util
import sys
# noinspection DuplicatedCode
def patch_module(module_name, modification_func):
spec = importlib.util.find_spec(module_name)
source = spec.loader.get_source(module_name)
source = modification_func(source)
module = importlib.util.module_from_spec(spec)
code = compile(source, module.__spec__.origin, 'exec')
exec(code, module.__dict__)
sys.modules[module_name] = module
return module
def patch_django():
patch_module("django.db.backends.base.base", lambda x: x.replace(
"self.force_debug_cursor = False",
"self.force_debug_cursor = True"))
Step2: Call patch_django before django imported.
manage.py
__import__("importlib").import_module("patchers").patch_django() # noqa
import os
import sys
def main():
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
wsgi.py
__import__("importlib").import_module("patchers").patch_django() # noqa
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
application = get_wsgi_application()
asgi.py
__import__("importlib").import_module("patchers").patch_django() # noqa
import os
from django.core.asgi import get_asgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
application = get_asgi_application()
Step 3: Make sure django.db.backends is in debug level
'django.db.backends': {
'level': 'DEBUG',
},

django: How can I create a custom Logging Filter for SuspiciousOperation exception?

After migrate to 1.11 ( from 1.8 ) I'm receiving some SuspiciousOperation errors from logging.
It seems it comes from JS request who keeps session alive if user move their mouse. But this is not important.
How can I filter just this exception?
What I tried:
I just created a filter somewhere:
import logging
from django.core.exceptions import SuspiciousOperation
class StopSuspiciousOperation(logging.Filter):
def filter(self, record):
if record.exc_info:
exc_value = record.exc_info[1]
return isinstance(exc_value, SuspiciousOperation)
return True
Then I added this filter to my configuration:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse',
},
'stop_suspicious_operation': {
'()': 'aula.utils.loggingFilters.StopSuspiciousOperation',
}
},
'handlers': {
'mail_admins': {
'level': 'ERROR',
'filters': ['require_debug_false',
'stop_suspicious_operation',], #<-- here
'class': 'django.utils.log.AdminEmailHandler'
}
},
'loggers': {
'django.request': {
'handlers': ['mail_admins'],
'level': 'ERROR',
'propagate': True,
},
}
}
But I'm still receiving the error:
Internal Server Error: /keepalive
SuspiciousOperation at /keepalive
The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example.
Request Method: GET
Request URL: https://XXXXXX/YYYYYY
Django Version: 1.11.9
Python Executable: /usr/bin/python
Python Version: 2.7.3
I am not sure about the correct answer, but I think django is catching the SuspiciousOperation at WSGI level and is logging an ERROR. See:
https://docs.djangoproject.com/en/dev/ref/exceptions/#suspiciousoperation
If a SuspiciousOperation exception reaches the WSGI handler level it
is logged at the Error level and results in a HttpResponseBadRequest.
You maybe just want to filter out the bad requests like this:
from logging import LogRecord
def filter_400(record: LogRecord) -> bool:
'''Filter out HTTP Error Code 400 Bad Request'''
return record.status_code != 400

Send error mail to admin in Django

I am trying to send a mail to ADMIN when any exception is occurred in django. So i tried but not able to find the solution. I don't have a clear idea about how to send a error mail to admin
Any help would be appreciated.
Views.py
def emit(self, record):
try:
request = record.request
subject = '%s (%s IP): %s' % (
record.levelname,
('internal' if request.META.get('REMOTE_ADDR') in settings.INTERNAL_IPS
else 'EXTERNAL'),
record.getMessage()
)
filter = get_exception_reporter_filter(request)
request_repr = '\n{}'.format(force_text(filter.get_request_repr(request)))
except Exception:
subject = '%s: %s' % (
record.levelname,
record.getMessage()
)
request = None
request_repr = "unavailable"
subject = self.format_subject(subject)
if record.exc_info:
exc_info = record.exc_info
else:
exc_info = (None, record.getMessage(), None)
message = "%s\n\nRequest repr(): %s" % (self.format(record), request_repr)
reporter = ExceptionReporter(request, is_email=True, *exc_info)
html_message = reporter.get_traceback_html() if self.include_html else None
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
def send_mail(self, subject, message, *args, **kwargs):
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
def connection(self):
return get_connection(backend=self.email_backend, fail_silently=True)
def format_subject(self, subject):
"""
Escape CR and LF characters, and limit length.
RFC 2822's hard limit is 998 characters per line. So, minus "Subject: "
the actual subject must be no longer than 989 characters.
"""
formatted_subject = subject.replace('\n', '\\n').replace('\r', '\\r')
return formatted_subject[:989]
def test_view(request):
try:
raise Exception
except Exception as e:
send_mail(self, subject, message, *args, **kwargs)
logger_setting.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse',
},
'require_debug_true': {
'()': 'django.utils.log.RequireDebugTrue',
},
},
'handlers': {
'console': {
'level': 'INFO',
'filters': ['require_debug_true'],
'class': 'logging.StreamHandler',
},
'null': {
'class': 'logging.NullHandler',
},
'mail_admins': {
'level': 'ERROR',
'filters': ['require_debug_false'],
'class': 'django.utils.log.AdminEmailHandler'
}
},
'loggers': {
'django': {
'handlers': ['console'],
},
'django.request': {
'handlers': ['mail_admins'],
'level': 'ERROR',
'propagate': False,
},
'django.security': {
'handlers': ['mail_admins'],
'level': 'ERROR',
'propagate': False,
},
'py.warnings': {
'handlers': ['console'],
},
}
}
I also tried by changing my view.py code as following but didn't any error mail
import sys
import traceback
from django.core import mail
from django.views.debug import ExceptionReporter
def send_manually_exception_email(request, e):
exc_info = sys.exc_info()
reporter = ExceptionReporter(request, is_email=True, *exc_info)
subject = e.message.replace('\n', '\\n').replace('\r', '\\r')[:989]
message = "%s\n\n%s" % (
'\n'.join(traceback.format_exception(*exc_info)),
reporter.filter.get_request_repr(request)
)
mail.mail_admins(
subject, message, fail_silently=True,
html_message=reporter.get_traceback_html()
)
Generating a test exception-
def test_view(request):
try:
raise Exception
except Exception as e:
send_manually_exception_email(request, e)
First follow these instructions: https://docs.djangoproject.com/en/dev/howto/error-reporting/#email-reports
and set the settings for EMAIL_HOST, EMAIL_HOST_USER, EMAIL_HOST_PASSWORD, ADMINS, and SERVER_EMAIL.
You'll also need to set an email server. For starters, any of these will do:
an email program (like sendmail)
an email server (like Sendgrid)
the python debugging email server python -m smtpd -n -c DebuggingServer localhost:1025
The values for your settings must match the email server that you choose.
Don't worry about the using email with logging until you get the simple case configured from settings first.

django.request logger not working for get_object_or_404

I have this code for UserDetailsView in Django Rest Framework. I am using django 1.9 and DRF 3.
class UserDetailsView(RetrieveUpdateAPIView):
"""
API endpoint that allows a user to be viewed or edited.
"""
serializer_class = UserDetailsSerializer
permission_classes = (IsAuthenticated,)
def get_object(self):
pk = self.kwargs.get('pk', None)
if not pk or (pk == str(self.request.user.pk)):
return self.request.user
else:
try:
return get_object_or_404(User, id=pk)
except ValueError:
return get_object_or_404(User, username=pk)
I have have my django logger configured as per these settings.
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'simple': {
'format': '[%(levelname)s] %(message)s'
},
},
'handlers': {
'default': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': 'logs/backend_common.log',
'maxBytes': 1024*1024*10,
'backupCount': 10,
'formatter': 'simple',
},
'request_handler': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': 'logs/backend_requests.log',
'maxBytes': 1024*1024*10,
'backupCount': 10,
'formatter': 'simple',
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
}
},
'loggers': {
'': {
'handlers': ['default'],
'propagate': True,
},
'django.request': {
'handlers': ['request_handler', 'mail_admins'],
'level': 'DEBUG',
'propagate': False,
}
}
}
Now, all django 4xx, 5xx error status code should ideally be logged into backend_requests.log file and they are except 404 status resulting from get_object_or_404. I think 404 errors resulting from this view should also get logged. Any help is highly appreciated.
There are actually two issues here.
The first is a bug in Django, which was fixed just a few days ago. The bit of code that was logging 404s was running a little too early. The next release of Django will work and your 404s will be logged as you would expect.
For other exceptions however, the problem is as follows. Django will log errors if the exceptions that caused them are bubbled up to the core request handler. However if the view or some middleware catches the exception and handles it, then this portion of Django's code never gets called. This is what is happening with DRF:
REST framework's views handle various exceptions, and deal with returning appropriate error responses.
The handled exceptions are:
Subclasses of APIException raised inside REST framework.
Django's Http404 exception.
Django's PermissionDenied exception.
In each case, REST framework will return a response with an appropriate status code and content-type. The body of the response will include any additional details regarding the nature of the error.
Because DRF catches the exception and returns a response object to Django, Django will just render the response and not log the error. This makes sense - it is possible for some middleware to take what was originally a 404 but return a different response instead (the flatpage middleware is a good example).
If you want to log exceptions handled by DRF you can specify your own EXCEPTION_HANDLER in the DRF config, or write your own middleware that logs the errors.
Note that 5xx exceptions are not handled by DRF and these should still propagate up to Django. These should be getting logged for you even now.