I have an application written in Django that has now become quite sizeable, the application is in constant use throughout the day and making changes 'on-the-fly' risks disruption.
I am fairly new to software development and am not sure what the best way is to develop a django application where I can issue updates grouped together and release a 'version'. So instead of me updating the main application as and when in a live environment, I'd like if I can have a development server where I can develop & test any updates and then I can roll these out once a month.
I can just copy the view/model files over and overwrite the old ones but what is the best way to handle database changes? I assume I will have to write SQL to add/drop changed columns and overwrite the django_content_type table completely?
Any advice appreciated!
I'd go further than Ashish: you must use version control. You should not be "copying files over" and overwriting old ones. In 2014 there is no excuse for not using something like git (or Mercurial, or even SVN).
For the database changes, you should of course be using migrations. In version 1.7, due to be released very soon now, these are included in core Django. In previous versions, you will need to install the third-party library South.
As for the source files, you can use git. You test your changes on a dev (or test) server, and once you're satisfied, push the changes to git. On the prod server, you then pull the changes and restart server.
The database changes can be accomplished using south and migrations. Again, you need to test your migrations on dev or test before pushing them to git. Once you're satisfied, you can then move the changes to prod.
The flow is ...
develop --> test --> commit changes to remote git --> pull changes on production --> migrate --> restart server
Look here for important links
git --> http://git-scm.com
githut --> http://github.com
south --> http://south.aeracode.org/
Related
I have few questions about this plugin.
1- what does it do?
Is it for exchanging databases between teams or changing their schema or creating tables based on models or something else?
2- if it is not meant to create tables based on models where can I find a script that does this?
3-can it work under windows?
thanks
The Migrations plugin allows versioning of your db changes. Much like is available in other PHP frameworks and Rails.
You essentially start with your original schema and create the initial migration. Each time you make a change you generate a 'diff' that gets stored in the filesystem.
When you run a migration, the database is updated with the changes you made. Think deployment to a staging or production server where you want the structure to be the same as your code is moved from environment to environment.
We are starting to look at this plugin so we can automate our deployments, as the DB changes are done manually right now.
I have setup a git repository and cloned open source code which I am planning on modifying from github to start development. I committed the codebase in our repository.
I now have added few users and posts and other "stuff" into database.
I want to commit this change as well so that my teammate can check out and we have same settings and database throughout.
Is this possible by using south migrations? i.e will database bot contents and schema be in sync as well?
I have the project where I am writing the code as well the actual app. Should I commit both of them.
What should the github repository look like after doing the "right" thing
Data and database structure
This is possible using south migrations, data migrations and fixtures.
The easiest way for development is to just use a SQLite database, which is a binary file that you can commit. The test_project of django-autocomplete-light demonstrates such a possibility: http://django-autocomplete-light.readthedocs.org/en/latest/demo.html
you must use south anyway !
Apps in the project repo
I think you should keep apps as small and loosely coupled as possible.
If sound, make another repository and python package for the app:
In some cases it makes sense at the beginning, ie. a "blog" app that you know you will reuse,
In some cases it makes sense later, ie. you tought your app was really project specific but then you want to reuse it in another project,
In some cases it never makes sense (ie. the app is only useful to that particular project).
Best practice
As for best practices, there is http://12factor.net, http://lincolnloop.com/django-best-practices/ and pinax projects which are really interresting.
If you're going to reuse and extend external apps, then maybe this article on best practice reusing apps can help.
If there is data that every developer needs, these can be provided by initial_fixtures. I have just begun to use south, but I think it is mainly for migrations, (changing your models, without having to delete your data, and resync) not for sharing data. Schema should always be in sync using south, because a developer can pull the south migration, and apply it.
During development, I like the idea of frameworks like Entity Framework 4.3 Migrations (although I need it to work with sql scripts instead of Migration classes) keeping all developer databases up-to-date. Someone does an update to get the latest source, they try to run the application and get an error that they need to update their database to the latest migration (or have the migration happen automatically). Migration files are timestamped so devs don't have to worry about naming two files the same or the sequence the files need to be executed in.
When I get ready to build a WebDeploy deployment package, I want the package to contain the scripts required to move the production database to the latest db version. So somehow, MSBuild or WebDeploy need to make a decision about which scripts must be packaged. For deployment I don't want the application to try to update itself like what EF offers. I want to either hand the package to the IT department or do an auto-deploy through a deployment server.
Some of my questions are:
Can EF 4.3 work with sql scripts instead of DBMigration classes for my development needs (I'm already using it for my ORM so it would be nice if it could)?
Does MSBuild or WebDeploy understand the concept of database migrations (e.g. does it recognize the EF. 4.3 migrationHistory table) or do I have to make sure to give it just the scripts it needs to run that will bring my prod db to the latest migration? Manually deciding which scripts should be pakaged is not something I want to do so is there a MS WebDeploy extension that understands migrations?
Are my concerns and ideas valid? I'm just researching this stuff so I don't really know.
I think that your concerns are valid. During development anything that keeps the developer machines in sync will do. When deploying, more control is needed.
In my project we've decided to only use code-based migrations, to ensure that all databases are migrated in the same order with the same steps. Automatic migration and db creation is disabled by a custom initialization strategy that only checks that the db exists and is valid.
I've written more about the details in Prevent EF Migrations from Creating or Changing the Database. I've also looked a bit into merge conflicts which will eventually happen with multiple developers.
You can run SQL in the classes by calling the Sql method, or generate SQL from a migration by using the -script parameter with the update-database command.
No. They were looking at adding support for webdeploy but apparently decided against it before rtm. They did however release a command line app (and powershell script obviously) which you could call from either.
What I do in my projects is create an startup module in my applications and run any migrations that haven't been deployed automatically - Triggering EF migration at application startup by code. It's not perfect, developers have to run the app after getting latest before changing the db, but it works for both get latest and deployment.
I am building a Django powered web app that has a large database component. I am wondering, how would I go about continuing to develop the web app while users are using the live, production version? There are two parts to the problem, as I see it, as follows:
Making changes to templates, scripts, and other files
Making database schema changes
Now, the first problem is easy to manage with a SVN system. Heck, I could just have a "dev" directory which have all my in-development files and, once ready, just copy them into the "production" directory.
However, the second problem is more confusing to me. How do I test/develop new database changes without affecting the main/live database? I have been using South to do schema migrations during the initial creation stages of the web app, but surely I wouldn't want to make changes to the database while it is being used. Especially if I make changes that I don't want to keep.
Any thoughts/ideas?
You need another server on which to do your development. Typically, this is a personal machine, like your laptop. Often, you also have a copy of your production environment on a server, known as the staging server.
Your workflow would be like this:
Work on your code on your development machine, make all the changes you want, it's just you using it.
When the code is ready for production, you push it to the staging server to see that it really works properly in a server environment.
When you're sure it's ready for production, push it to the production server.
Overview
I'm building a website in django. I need to allow people to begin to add flatpages, and set some settings in the admin. These changes should be definitive, since that information comes from the client. However, I'm also developing the backend, and as such will am creating and migrating tables. I push these changes to the hub.
Tools
django
git
south
postgres
Problem
How can I ensure that I get the database changes from the online site down to me on my lappy, and also how can I push my database changes up to the live site, so that we have a minimum of co-ordination needed? I am familiar with git hooks, so that option is in play.
Addendum:
I guess I know which tables can be modified via the admin. There should not be much overlap really. As I consider further, the danger really is me pushing data that would overwrite something they have done.
Thanks.
For getting your schema changes up to the server, just use South carefully. If you modify any table they might have data in, make sure you write both a schema migration and as necessary a data migration to preserve the sense of their data.
For getting their updated data back down to you (which doesn't seem critical, but might be nice to work with up-to-date test data as you're developing), I generally just use Django fixtures and the dumpdata and loaddata commands. It's easy enough to dump a fixture and commit it to your repo, then a loaddata on your end.
You could try using git hooks to automate some of this, but if you want automation I do recommend trying something like Fabric instead. Much of this stuff doesn't need to be run every single time you push/pull (in particular, I usually wouldn't want to dump a new data fixture that frequently).
You should probably take a look at South:
http://south.aeracode.org/
It seems to me that you could probably create a git hook that triggers off South if you are doing some sort of continuous integration system.
Otherwise, every time you do a push you will have to manually execute the migration steps yourself. Don't forget to put up the "site is under maintenance" message. ;)
I recommend that you use mk-table-sync to pull changes from live server to your laptop.
mk-table-sync takes a lot of parameters so you can automate this process by using fabric. You would basically create a fabric function that executes mk-table-sync on each tablet that you want to pull from the server.
This means that you can not make dabatase changes yourself, because they will be overwritten by the pull.
The only changes that you would be making to the live database are using South. You would push the code to the server and then run migrate to update the database schema.