How can I log the requests and responses that the Django server handles?
I have a Django server application has tables and relations. Many tables are exposed through several API endpoints. There are also several custom functions that take GET/POST requests with params and run a function in python and return a result.
As the server admin, I would like to log all the incoming requests and the outgoing responses for each of these requests. It is essential to me that I capture the GET/POST params made with each requests and the data that is sent out with each response. What would be the best way for me to implement this?
I would suggest you DON'T do this through Django, all you're going to do is add to the request work load. You're going to make your application slower, all so you can have it in "one place", remember, "the right tool for the job".
I highly suggest you log to a standard logger file/syslog, and then use something like GrayLog/Loggly/Splunk/Logstash/etc. -- you're going to thank me in the long run, you're not going to grind your Django server and/or DB to the ground.
If you think I'm kidding, think of this: you're more likely going to put this in a middleware right? BEFORE auth, right? If I wanted to make your life a living hell all I have to do is while true; do curl https://yourdomain.com/doesnt-need-to-exist &; done i'm going to fill your DB up in a couple of days, not to mention degrade your service as a whole. Do everyone a favor and don't use the db, specially a prod db to store logs
django-wiretap is a package that stores the requests and responses in the database via models.
Related
I am not able to comprehend what would be pros and cons of the following approaches in making a single page backbone application using RESTful APIs from Django Rest Framework.
Render the whole app from within Django's template.
Serve the backbone app from another server ie node server. With nginx in the front for both servers.
Serve the HTML/Templates and JS from a separate CDN.
What are the things to take care ie points of caution in each strategy. Is there any other way to tie them up which I am missing?
This is a very broad question, and really it has nothing to do with Django or Backbone. What you're really asking about is a "thick-client" architecture vs. a "thin-client" architecture. In other words, having your user interface rendered on the client vs. having it rendered on the server.
First, allow me to recap a few things to make sure we're on the same page. The "thin-client" approach is the traditional/old school model, and the model Django itself is based on. The server renders HTML, sends it to the client, and whenever the client wants to do something it sends data back to the server and asks for fresh HTML.
In contrast the more modern "thick-client" approach lets the client render all of the UI. Whenever the client needs to do something it makes an AJAX request to a (presumably REST-ful) API, powered by a library like Django REST Framework. That API just returns the relevant data, and leaves it up to the client to render it appropriately.
There are advantages and disadvantages to both approaches, but the thick-client approach is becoming more and more popular because:
network transactions are faster: because your server is only sending the exact JSON you need instead of a mess of HTML, the "payload" of the response is much smaller
you can fetch all data "behind the scenes"; this makes things appear faster to the user, and lets you implement UI paradigms (eg. infinite scroll) that a thin client can't
the client/server relationship is simpler, because the people writing server code never have to even think about HTML or any other presentation logic; they get to just focus on the data (which, being server engineers, is probably the part they're most interested in anyway)
This is why a lot of companies (including the one I work for) have all but abandoned Django proper in favor of API endpoints served by Django REST Framework.
So, if you want to go with a thick client architecture, Django should never serve anything except the very first HTML page (and even that could be served by ngnix if you wanted, since it's just static HTML). After that you'd use a Backbone.Router and Backbone.Views to render your site. Whenever you need new information from the server you'd fetch a Backbone.Model or Backbone.Collection (with its url property pointing to your Django REST Framework endpoint).
I can attest that this whole approach works great; the site I work on is very complex, with many endpoints, and Backbone + Django REST Framework handles it beautifully. The only (slightly) tricky part is caching: in the thin client approach the browser automatically caches pages for you, but since there are no "pages" in a thick client (just AJAX responses with data) there is no automatic caching. This means that if you want to cache data you'll need to do it yourself, for instance with a Backbone.Collection devoted to that purpose.
Hope that helps.
P.S. Back in the day Django REST Framework didn't handle Django authentication stuff (ie. logging in/out) quite the way we wanted, so we wound up serving one other page, our login page, from Django. However I'm pretty sure the current Django REST Framework handles authentication stuff much better now, so this likely won't be an issue for you.
I am writing a Django management command that visits a few pages, logged in as a superuser, and saves the results to a set of .html files.
Right now I'm just using the requests library and running the command with the development server running. Is there an easy way to generate the HTML from a view response so I do without actual HTTP requests?
I could create a request object from scratch but that seems like more overhead than the current solution. I was hoping for something simple.
Django has a RequestFactory which seems to suit your needs.
While it's not exactly meant for this purpose, an option would be to use the testing framework's Client to fake a request to the url - be sure to use client.login() before making your requests, to ensure you have superuser capabilities.
In django app I need to periodically check for if new object of particular model are created.
I want to this by ajax.
I was thinking about something like this:
render current timestamp into template, load current objects.
Then, every x seconds do ajax request and ask for objects which are created later then this timestamp.
What do you think? Is there maybe a better way?
What you want is a way for the client to know whether something has changed in the server. Generally there are three ways to stimulate this subscriber/broadcaster, or pull/push, relationship. The first is Ajax long-polling, which is roughly what you described. The second is implemented via WebSocket, which unfortunately not all browser supports. The third is HTTP streaming, or a long polling at the HTTP level. All three are available in https://github.com/ziyan/django-comet
A newer technology is Webhooks, which allows you to subscribe to server changes via URL (http://en.wikipedia.org/wiki/Webhook). Check it out here for an early Django adaptation: https://github.com/johnboxall/django_webhooks
I have an django app, a model which stores data entered via a web interface by a user.
I need to consume an third party REST api when viewing / saving a model instance. I know how to do this but, what I am unsure about is where this code should live with the django app.
my gut is to put this code with in the model class, but then you could also use a view... I am just not sure.
How has this been done before, there are lots of posts asking how to do this, but none stating best place to put the code.
any guidance would be gratefully received.
Cheers
This is a subjective question, so here is a subjective answer.
First of all, ensure that any code that interacts with this external REST API resides in a separate module. E.g, if you're grabbing word definitions from a dictionary API, all the code that talks to this API should ideally be in a separate dictionary module, which you can then import into your view.
Secondly, your models.py should merely declare your application's data model and define the operations on this model, and little else. They should not be concerned with request/response cycles, reading files, rendering templates, making HTTP calls, or anything else. By this logic, you should be making these REST API calls from your views and, if required, passing the returned data into your models.
And finally, think twice about making REST calls from your Django app. Python does synchronous (blocking) I/O by default, which means as long the app is waiting for the REST call to finish, it can't service any incoming HTTP requests. This is not an issue if you don't have too many users, but it's something to keep in mind for apps that need to scale. You might want to look into async I/O libraries for Python.
I have a middleware in my Django application that redirects mobile clients to a user-configurable mobile domain. It's not a simple m.[current domain], since users define the domain themselves. To save queries, I can store a mapping similar to {'www.example.com': 'mobile-version.example.com'}. However, I'd like to save the wsgi server and full Django stack from being reached on mobile requests, because this simple logic is the only thing that is happening. My thought was, if I could place this logic in Nginx somehow, I'd be able to bypass Django altogether, saving some resources. Is this possible? I've read where people have served entire sites via memcached (seems like a cheaper replacement for simple Varnish usage), but the methodology seems to be a bit different.
The logic would be something like:
$mobile_domain = memcached.get_by_key("mobile_domain_for:" + $current_domain)
IF $mobile_domain:
redirect $mobile_domain + $path_info + $query_strings
It looks like the third-party memc-nginx-module has the ability to look up specific memcached keys.