I have a middleware where I a want to log every request/response. How can I access to POST data? - django

Question

I have this middleware
import logging
request_logger = logging.getLogger('api.request.logger')
class LoggingMiddleware(object):
def process_response(self, request, response):
request_logger.log(logging.DEBUG,
"GET: {}. POST: {} response code: {}. response "
"content: {}".format(request.GET, request.DATA,
response.status_code,
response.content))
return response
The problem is that request in process_response method has no .POST nor .DATA nor .body. I am using django-rest-framework and my requests has Content-Type: application/json
Note, that if I put logging to process_request method - it has .body and everything I need. However, I need both request and response in a single log entry.

Answer 1

Here is complete solution I made
"""
Api middleware module
"""
import logging
request_logger = logging.getLogger('api.request.logger')
class LoggingMiddleware(object):
"""
Provides full logging of requests and responses
"""
_initial_http_body = None
def process_request(self, request):
self._initial_http_body = request.body # this requires because for some reasons there is no way to access request.body in the 'process_response' method.
def process_response(self, request, response):
"""
Adding request and response logging
"""
if request.path.startswith('/api/') and \
(request.method == "POST" and
request.META.get('CONTENT_TYPE') == 'application/json'
or request.method == "GET"):
request_logger.log(logging.DEBUG,
"GET: {}. body: {} response code: {}. "
"response "
"content: {}"
.format(request.GET, self._initial_http_body,
response.status_code,
response.content), extra={
'tags': {
'url': request.build_absolute_uri()
}
})
return response
Note, this
'tags': {
'url': request.build_absolute_uri()
}
will allow you to filter by url in sentry.

Answer 2

Andrey's solution will break on concurrent requests. You'd need to store the body somewhere in the request scope and fetch it in the process_response().
class RequestLoggerMiddleware(object):
def process_request(self, request):
request._body_to_log = request.body
def process_response(self, request, response):
if not hasattr(request, '_body_to_log'):
return response
msg = "method=%s path=%s status=%s request.body=%s response.body=%s"
args = (request.method,
request.path,
response.status_code,
request._body_to_log,
response.content)
request_logger.info(msg, *args)
return response

Answer 3

All answers above have one potential problem -- big request.body passed to the server. In Django request.body is a property. (from framework)
#property
def body(self):
if not hasattr(self, '_body'):
if self._read_started:
raise RawPostDataException("You cannot access body after reading from request's data stream")
try:
self._body = self.read()
except IOError as e:
six.reraise(UnreadablePostError, UnreadablePostError(*e.args), sys.exc_info()[2])
self._stream = BytesIO(self._body)
return self._body
Django framework access body directly only in one case. (from framework)
elif self.META.get('CONTENT_TYPE', '').startswith('application/x-www-form-urlencoded'):
As you can see, property body read the entire request into memory. As a result, your server can simply crash. Moreover, it becomes vulnerable to DoS attack.
In this case I would suggest using another method of HttpRequest class. (from framework)
def readlines(self):
return list(iter(self))
So, you no longer need to do this
def process_request(self, request):
request._body_to_log = request.body
you can simply do:
def process_response(self, request, response):
msg = "method=%s path=%s status=%s request.body=%s response.body=%s"
args = (request.method,
request.path,
response.status_code,
request.readlines(),
response.content)
request_logger.info(msg, *args)
return response
EDIT: this approach with request.readlines() has problems. Sometimes it does not log anything.

Answer 4

It's frustrating and surprising that there is no easy-to-use request logging package in Django.
So I created one myself. Check it out: https://github.com/rhumbixsf/django-request-logging.git
Uses the logging system so it is easy to configure. This is what you get with DEBUG level:
GET/POST request url
POST BODY if any
GET/POST request url - response code
Response body

Answer 5

It is like accessing the form data to create a new form.
You must use request.POST for this (perhaps request.FILES is something you'd log as well).
class LoggingMiddleware(object):
def process_response(self, request, response):
request_logger.log(logging.DEBUG,
"GET: {}. POST: {} response code: {}. response "
"content: {}".format(request.GET, request.POST,
response.status_code,
response.content))
return response
See Here for request properties.

Answer 6

You can use like below:
"""
Middleware to log requests and responses.
"""
import socket
import time
import json
import logging
request_logger = logging.getLogger(__name__)
class RequestLogMiddleware:
"""Request Logging Middleware."""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
log_data = {}
# add request payload to log_data
req_body = json.loads(request.body.decode("utf-8")) if request.body else {}
log_data["request_body"] = req_body
# request passes on to controller
response = self.get_response(request)
# add response payload to our log_data
if response and response["content-type"] == "application/json":
response_body = json.loads(response.content.decode("utf-8"))
log_data["response_body"] = response_body
request_logger.info(msg=log_data)
return response
# Log unhandled exceptions as well
def process_exception(self, request, exception):
try:
raise exception
except Exception as e:
request_logger.exception("Unhandled Exception: " + str(e))
return exception
You can also check this out - log requests via middleware explains this

Answer 7

Also note, that response.content returns bytestring and not unicode string so if you need to print unicode, you need to call response.content.decode("utf-8").

Answer 8

You cannot access request.POST (or equivalently request.body) in the process_response part of the middleware. Here is a ticket raising the issue. Though you can have it in the process_request part. The previous answers give a class-based middleware. Django 2.0+ and 3.0+ allow function based middlewares.
from .models import RequestData # Model that stores all the request data
def requestMiddleware(get_response):
# One-time configuration and initialization.
def middleware(request):
# Code to be executed for each request before
# the view (and later middleware) are called.
try : metadata = request.META ;
except : metadata = 'no data'
try : data = request.body ;
except : data = 'no data'
try : u = str(request.user)
except : u = 'nouser'
response = get_response(request)
w = RequestData.objects.create(userdata=u, metadata=metadata,data=data )
w.save()
return response
return middleware
Model RequestData looks as follows -
class RequestData(models.Model):
time = models.DateTimeField(auto_now_add=True)
userdata = models.CharField(max_length=10000, default=' ')
data = models.CharField(max_length=20000, default=' ')
metadata = models.CharField(max_length=20000, default=' ')