I am going to outline my workflow and I would like some suggestions on how to improve the efficiency of this. It seems right now a bit cumbersome and repetitive (something I hate), so I am looking for some improvements. Keep in mind I'm still new to django and how it works but I'm a pretty fluent coder (IMHO). So here goes...
Tools (I use these everyday so I'm not inclined to shift):
Mac OSX Leopard
TextMate
Terminal w/tabs
Perforce
Assumptions
Django Basics (Did the tutorials/bought the books)
Python Fluent (running 2.6 with IDLE Support)
Starting my first Application working on models.py
Starting out
Create a TextMate Project with the entire django Tree inside of it.
TextMate Project http://img.skitch.com/20090821-g48cpt38pyfwk4u95mf4gk1m7d.jpg
In the first tab of the terminal start the server
python ./manage.py runserver
In the second tab of the terminal window start the shell
python ./manage.py shell
This spawns up iPython and let's me start the development workflow
Workflow
Create and build a basic Model called models.py
Build a basic Model
class P4Change(models.Model):
"""This simply expands out 'p4 describe' """
change = models.IntegerField(primary_key=True)
client = models.ForeignKey(P4Client)
user = models.ForeignKey(P4User)
files = models.ManyToManyField(P4Document)
desc = models.TextField()
status = models.CharField(max_length=128)
time = models.DateField(auto_now_add=True)
def __unicode__(self):
return str(self.change)
admin.site.register(P4Change)
In the first terminal (Running server) stop it ^C and syncdb start server
> python ./manage.py syncdb
Creating table perforce_p4change
Installing index for perforce.P4Change model
In the shell terminal window load it..
> python ./manage.py shell
Python 2.6.2 (r262:71600, Apr 23 2009, 14:22:01)
Type "copyright", "credits" or "license" for more information.
IPython 0.10 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]: from perforce.models import *
In [2]: c = P4Client.objects.get_or_create("nellie")
Did it break yes/no if it did not work then do this:
Stop the shell
Clear the database
Rebuild the database
Fix the code
Reload the shell
Reload the modules
PRAY...
Issues / Comments / Thoughts
Is it me or does this seem terribly inefficient?
It seems like I should be able to do a reload(module) but I can't figure out how to do this.. Anyone?
It would seem as though I should be able to test this from within TextMate?? Anyone??
Even to just get out of the shell I have to verify I want to leave..
The point of this is for all of you geniuses out there to show me the light on a more productive way to work. I am completely open to reasonable suggestions. I'm not inclined to shift tools but I am open to criticisms.
First of all, no need to do a ./manage.py runserver until your models are in place.
Second, clear the database/rebuild the database should be done after fixing the code, and can be done in one fell swoop with ./manage.py reset perforce
Third, the things that you are typing out in the shell each time (import models, try creating an object) should be written in a test suite instead. Then you can do ./manage.py test perforce instead of firing up the shell and typing it again. Actually, if you're using the test suite, you won't need to, because it will create a clean dummy db each time, and break it down for you when it's done.
Fourth, Instead of "PRAY...", try "Watch tests pass."
I find it smoother to write unit tests more often and only use the shell when something is failing and it's not obvious why and you want to poke around to figure it out. It is a little more inefficient at the very beginning, but quickly becomes a wonderful way to work.
I also tend to concentrate on getting the model more or less stable and complete (at least as far as what will affect table structure) before I work on the views and need to run the server. That tends to front-load as many resets as possible so you're doing them when it's cheap.
Thanks to everyone who read this and is looking for a better way. I think unit tests are definately the simpler approach.
So according to the docs you simply need to create a file tests.py parallel to models.py and put tests in there.
from django.test import TestCase
from perforce.models import P4User, P4Client
class ModelTests(TestCase):
def setUp(self):
self.p4 = P4.P4()
self.p4.connect()
def test_BasicP4(self):
"""
Make sure we are running 2009.1 == 65
"""
self.failUnlessEqual(self.p4.api_level, 65)
def test_P4User_get_or_retrieve(self):
"""
This will simply verify we can get a user and push it into the model
"""
user = self.p4.run(("users"))[0]
dbuser = P4User.objects.get_or_retrieve(user.get('User'))
# Did it get loaded into the db?
self.assertEqual(dbuser[1], True)
# Do it again but hey it already exists..
dbuser = P4User.objects.get_or_retrieve(user.get('User'))
# Did it get loaded into the db?
self.assertEqual(dbuser[1], False)
# Verify one field of the data matches
dbuser = dbuser[0]
self.assertEqual(dbuser.email, user.get("Email"))
Now you can simply fire up the terminal and do python manage.py test and that will run the tests but again that's a pretty limited view and still requires you to swap in/out of programs.. So here is how you do this directly from Textmate using ⌘R.
Add an import line at the top and a few line at the bottom.
from django.test.simple import run_tests
#
# Unit tests from above
#
if __name__ == '__main__':
run_tests(None, verbosity=1, interactive=False)
And now ⌘R will work directly from TextMate.
OK, I'll bite :-) Here's what I use:
MAMP. You get a fully functional Apache + MySQL + PHP + phpMyAdmin stack to manage the web and DB layers. It's great for apps that go beyond basic SQLite. Basic version is free but I went ahead and popped for Pro because I use it so much and wanted to support the devs. A good way to test and make sure everything works is start with the Django test server, then deploy and test under MAMP on your own machine, and finally push it out to your deployment site. (You could try to automate the process with something like Fabric).
Eclipse + PyDev + PyDev extensions. Once configured properly you get Python code completion, a nice development environment, and full debugging. You can configure it so it runs the Django test server for you and you can set breakpoints on any line in Django source or your own code. The thing I like about Eclipse is that once you get used to the environment, you can also it for C/C++, Java, JavaScript, Python, and Flex coding.
Aptana for Eclipse. It helps when developing AJAX front-ends and editing Django templates to have a decent Javascript + HTML editor/debugger.
TextMate. I've created a TextMate project that includes all of Django sources and saved it in the Django source directory. This way, I can quickly do project searches through Django source and single-click open the source file. You can also set it up so you can go back and forth between Eclipse and TextMate editors and have them auto-reload.
A decent MySQL or SQLite editor. phpMySQLAdmin is OK but sometimes it's good to have a standalone tool. SequelPro (formerly CocoaMySQL) and Navicat are all pretty good for MySQL. One advantage is that once your app is deployed, you can use these tools to remotely access the deployment DB server and tweak it from your desktop. On the SQLite side SQLiteManager and Base are good commercial tools, as is the freebie FireFox SQLite Manager. At the very least you can watch what Django's doing under the hood.
I use Subversion for version control mostly because it runs on a standalone Mac Mini which saves to a Drobo RAID array plus auto-backups everything to a couple other external drives. This on top of Time Machine (yes, I'm paranoid :-) I used to use Eclipse's SVN support but now I'm a big fan of Versions. At some point when I can figure out a good mirroring scheme I'll switch to Mercurial, Git, or Bazaar, but for now this works pretty well.
Terminal plus a bunch of shell scripts. Everyone has their own version of this. I'm pretty lazy when it comes to these things so I set up a bunch of bash shortcuts to help speed up repetitive Django admin tasks. I posted these up a while back.
Most of these can be had for free or a moderate fee (< $100). But if I had to pick the 'must have' items for Django development on the Mac, it would be Eclipse and PyDev.
I'm sure there are some I've missed. Be great to hear what tools everyone else is using.
Related
I've used Django to make the site and now I'd like to be able to use the bot to grab info from Discord to use. For example how many members are in a particular server or only letting a person through to a certain page if they're in the server. Is there a way I can run the bot alongside everything else, or an alternative solution? Thanks in advance for any help!
Edit: Thank you for the suggestions below, however I read over everything several times and I just couldn't make much sense of how it would work, or how I could apply it. I just went for the easiest solution which I explain below
Edit 2: Forgot to edit this post again, but a while ago when I was trying to fix this, I actually scrapped the solution I put below and instead used a get request to get the number of members, a much better solution.
The solution I used (Read edit 2 before this):
This is probably one of the "harder to do" solutions, but since I found that I'm capable of doing it I went with it. Inside the main Django app I made a Python file for running the bot and a json file in which the bot could save data to. Then from that json file, the views.py file could grab information. The only "problem" here is that may be hard for others to keep a bot running consistently with an operating system that they dedicate to specifically that. I am using a Raspberry Pi that I keep on all day to run the bot straight from the project directory (I was already using the Raspberry Pi to run various Python bots so this just adds another bot). I don't know if there're any ways of running it from something such as Heroku and being able to have access to the project files.
This is a bit late, but for the next ones who have the same problem
You can use a django management command ( https://docs.djangoproject.com/en/4.0/howto/custom-management-commands/ )
just create a .py file under your_project/your_app/management/commands/<your_command_name.py>
and wrap the bot like this
from django.core.management.base import BaseCommand
from django.conf import settings
from .bot import DiscordBot
class Command(BaseCommand):
help = "Run a discord bot"
def handle(self, *args, **options):
bot = DiscordBot()
bot.run(settings.DISCORD_BOT_TOKEN)
When it's done you can run your bot using:
python manage.py <your_command_name>
I am very new to python/django programming, as in I have no programming background. I am working on a class online and I just would like to know exactly what the manage.py file does. I've tried googling it, but I have not found any answers other than it puts a thin shell around django-admin.py. I still don't understand what the file does. I just know I need to type it whenever I do certain things.
(I assume you've read the documentation. But if not, take note that one of Django's great strong points is its documentation -- I recommend going there first before hitting Google.)
You can think of the arguments you pass to manage.py as subcommands. It is your tool for executing many Django-specific tasks -- starting a new app within a project, running the development server, running your tests...
It is also an extension point where you can access custom commands you write yourself that are specific to your apps.
I've just started using Django and one thing I find that I'm doing is starting a lot of new projects. I'm finding this process to be pretty tedious every time, even using manage.py startproject * I'm constantly changing settings in settings.py like media_root and template paths. Just a little background, I come from PHP and CodeIgniter. I never used a stock CI directory. I modified it to meet my needs for a new project. When I needed a new project, I would just copy that directory. manage.py seems to generate the files on the fly so this approach doesn't seem that possible. Does anyone else have any advice on this?
Lincoln loop has some best practices, they suggest importing settings from a different file. http://lincolnloop.com/django-best-practices/projects/modules/settings.html
Also checkout pip requirements, you might be able to use this to install the settings module from an external source like a git repo.
I'm using Paver to automate my Django project setup.
I have a Bitbucket repository with my own bootstrap setup. Eventually I'll make this generic, but for now it might give you some example code
Sounds like you're starting new projects very often. I assume that's because you're learning. Sure, if there's a custom settings.py that will save you some typing as you generate your learning projects, create it and use it. You could make your template the whole project directory, but since you're unlikely to have a lot of project-level boilerplate outside of settings.py, just focus on that one file. The settings file is the essence of the project.
Django development is all about apps. As you learn more, apps will start to become your focus. My advice would be not to pour too much energy into making an efficient assembly line for project creation.
Also, please learn and use use version control. For bonus points, also learn and use virtualenv :)
My django application has become painfully slow on the production. Probably it is due to some complex or unindexed queries.
Is there any django-ish way to profile my application?
Try the Django Debug Toolbar. It will show you what queries are executed on each page and how much time they take. It's a really useful, powerful and easy to use tool.
Also, read recommendations about Django performance in Database access optimization from the documentation.
And Django performance tips by
Jacob Kaplan-Moss.
Just type "django-profiling" on google, you'll get these links (and more):
http://code.djangoproject.com/wiki/ProfilingDjango
http://code.google.com/p/django-profiling/
http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/
Personally I'm using the middleware approach - i.e. each user can toggle a "profiling" flag stored in a session, and if my profiling middleware notices that a flag has been set, it uses Python's hotshot module like this:
def process_view(self, request, view_func, view_args, view_kwargs):
# setup things here, along with: settings.DEBUG=True
# to get a SQL dump in connection.queries
profiler = hotshot.Profile(fname)
response = profiler.runcall(view_func, request, *view_args, **view_kwargs)
profiler.close()
# process results
return response
EDIT: For profiling SQL queries http://github.com/robhudson/django-debug-toolbar mentioned by Konstantin is a nice thing - but if your queries are really slow (probably because there are hundreds or thousands of them), then you'll be waiting insane amount of time until it gets loaded into a browser - and then it'll be hard to browse due to slowness. Also, django-debug-toolbar is by design unable to give useful insight into the internals of AJAX requests.
EDIT2: django-extensions has a great profiling command built in:
https://github.com/django-extensions/django-extensions/blob/master/docs/runprofileserver.rst
Just do this and voila:
$ mkdir /tmp/my-profile-data
$ ./manage.py runprofileserver --kcachegrind --prof-path=/tmp/my-profile-data
For profiling data access (which is where the bottleneck is most of the time) check out django-live-profiler. Unlike Django Debug Toolbar it collects data across all requests simultaneously and you can run it in production without too much performance overhead or exposing your app internals.
Shameless plug here, but I recently made https://github.com/django-silk/silk for this purpose. It's somewhat similar to django toolbar but with history, code profiling and more fine grained control over everything.
For all you KCacheGrind fans, I find it's very easy to use the shell in tandem with Django's fantastic test Client for generating profile logs on-the-fly, especially in production. I've used this technique now on several occasions because it has a light touch — no pesky middleware or third-party Django applications are required!
For example, to profile a particular view that seems to be running slow, you could crack open the shell and type this code:
from django.test import Client
import hotshot
c = Client()
profiler = hotshot.Profile("yourprofile.prof") # saves a logfile to your pwd
profiler.runcall(c.get, "/pattern/matching/your/view/")
profiler.close()
To visualize the resulting log, I've used hotshot2cachegrind:
http://kcachegrind.sourceforge.net/html/ContribPython.html
But there are other options as well:
http://www.vrplumber.com/programming/runsnakerun/
https://code.djangoproject.com/wiki/ProfilingDjango
I needed to profile a Django app recently and tried many of these suggestions. I ended up using pyinstrument instead, which can be added to a Django app using a single update to the middleware list and provides a stack-based view of the timings.
Quick summary of my experience with some other tools:
Django Debug Toolbar is great if you the issue is due to SQL queries and works well in combination with pyinstrument
django-silk works well, but requires adding a context manager or decorator to each part of the stack where you want sub-request timings. It also provides an easy way to access cProfile timings and automatically displays ajax timings, both of which can be really helpful.
djdt-flamegraph looked promising, but the page never actually rendered on my system.
Compared to the other tools I tried, pyinstrument was dramatically easier to install and to use.
When the views are not HTML, for example JSON, use simple middleware methods for profiling.
Here are a couple examples:
https://gist.github.com/1229685 - capture all sql calls went into the view
https://gist.github.com/1229681 - profile all method calls used to create the view
You can use line_profiler.
It allows to display a line-by-line analysis of your code with the time alongside of each line (When a line is hit several times, the time is summed up also).
It's used with not-Django python code but there's a little trick to use it on Django in fact: https://stackoverflow.com/a/68163807/1937033
I am using silk for live profiling and inspection of Django application. This is a great tool. You can have a look on it.
https://github.com/jazzband/django-silk
For school, I have made a CMS in django for my major assessment task for Software. My teacher has asked to get the source code and, if applicable, the program compiled.
Now, because i dont want my teacher to install django (Something might go wrong, he may get a different version, missing dependences), how can i package up my django app, plus the django source and make the whole thing runnable (on the development server) by running a single script?
He has python, so that does not need to be included and the target OS would be OS X, but Windows can do as well.
Pip and VirtualENV will make this task much easier. (not sure the support for windows though)
PIP will help with the requirements installation.
http://pypi.python.org/pypi/pip
VirtualENV provides an isolated python environment.
URL: http://pypi.python.org/pypi/virtualenv
Reading through this blog post on installing Pinax will give you a good understanding on how the two work together: http://uswaretech.com/blog/2009/03/create-a-new-social-networking-site-in-few-hours-using-pinax-platform-django/
Perhaps Instant Django will set you in the right direction. It's for windows, but it might be of help.
Without having actually tested it, you should be able to copy the main django directory (/usr/lib/python2.6/site-packages/django for me) over into your project directory, and archive the whole thing. This will continue to keep everything importable (from django import ...), and make it so there's just one archive to extract.
Now, I wouldn't say this is a good way, but it's simple, and I think it'll work. I think.
I belive this is what your looking for (it's not pretty but it gets the job done).
It describes how to package django, a web server, and everything else needed to make a stand alone django application. To make it work for osx you should just be able to substitue py2app (http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html) instead of py2exe and it should (cross your fingers) work, however I have not tested it. Hope this helps!
Ps: sorry for not hyperlinking py2app im a new user and cant post 2 links yet :(