I have a Django 1.9-driven website run on Ubuntu, and I very often face a strange issue that some error vanishes when I run the clone of the project locally from my PC using 127.0.0.1:8000 url. Locating the error in such cases is EXTREMELY time consuming and I wonder what are the best practices for debugging a large-scale project, especially when the website is already partially in use.
Just to be as specific as possible, I provide a step-by-step description of what goes wrong.
Step 1. I type some url, say, 10.8.0.1:8000/show_students/
Step 2. Do some action on the page, say, save a student profile. The operation does not end successfully, yielding an error.
Step 3. I copy-paste the project directory located on the remote server onto a local directory on my PC and try to run the CLONE. I see that the error does not take place.
Real-life example,
task_email_recipients = TaskEmailRecipients.objects.get(task_type =
task_instance.type, legal_entity_own = legal_entity_own_instance)
This line throws exception saying that LegalEntityOwn has no field named (yes, I did not omit anything. It is empty string after "field named")
If I run the same view from 127.0.0.1, the error goes away.
What should be my actions ?
BTW, I use Eclipse if this makes any difference. And I have MS Windows 10 on my local PC.
Summing up, my goal is to debug the project run from 10.8.0.1
UPDATE for – Paul Becotte's comment
I've always ignored this warning, but when running the project, it gives a warning
You have unapplied migrations; your app may not work properly until
they are applied. Run 'python manage.py migrate' to apply them.
So, let me explain a few concepts.
A. Source Control (Git) lets you keep track of all the changes to your source code. This is pretty important so that you can feel confident that you are running the same version of your code on your development machine as your deployed server without trying to do something like copy the files back and forth. A command like git status can show you if you changed something and maybe give you tips on what is different between the two environments. If you are not using git, you should immediately start!
B. Migrations are like source control for the schema of your database. A SQL database like Mysql or Postgres has a fixed schema- you have THIS many tables, with THESE names, and table A has three columns with one of them called Name and one called ID and so forth. Migrations are designed to give you visibility into what these schema are- instead of logging into the database and running CREATE TABLE A ... you build a migration file that contains the necessary commands, and then stamps the database with a version number. Then you run those command files so that if the databases are on the same version, you know they have the same structure (which allows you to get your local database to match your deployed one). Django has a helpful migration system all built in... manage.py migrate is a command to apply all of the migration files to the current database. If you are getting the error message you listed, there is basically no chance that your app IS going to work properly, because your database schema, somewhere, is out of sync with your model files. Based on your very limited description, you added a field to a model that now exists in your local database but does not exist in your production database.
C. I mentioned a deploy script- this is a single command you can run to get your code running on your remote server so that you are sure it happens the same way every time. In this case it might be something like...
ssh production
git pull
python manage.py migrate
uwsgi
Set up a script like that so that you know what is going on, and you can rule out accidentally skipping steps as an error vector.
Related
I have successfully set up and used Django with MySQL on my local machine, but now if I put my project on say GitHub what should the requirements be for any other person to be able to run it?
I am asking this because my project uses a database that I have stored in my local machine and while uploading the project on GitHub I have not uploaded the database. in sqlite3 there is a database file inside the project itself but this does not happen for MySQL whose database is stored in a different location.
I mean Django accesses the database from a different location(var/lib/MySQL) and when I try to copy the database from there to the project folder and specify its location in settings.py, I get an access denied error.
So how can I solve this?
You would typically have a seed file for others to use. Others will create a database on their own systems and use your seed file to get started with the project.
It should not be necessary to copy the database files. Also, you should not just copy the MySQL directory like that. If you copy the whole directory then you might replace what somebody already has on their system, but if you copy only some of the files then you might be missing things like the MySQL user accounts. Besides, that is a backup procedure, not a deployment or distribution procedure.
For somebody else to get started with your project the normal process is:
manually create the appropriate MySQL user and database (or provide a script to automate it)
Run migrations: python manage.py migrate
Import initial data:
This can be with fixtures: python manage.py loaddata my_data.json
Or with a custom management command: python manage.py load_my_data
However, if you really need to provide somebody with an almost ready database, you can use mysqldmp which will produce a SQL text file, but the other person still needs to create the user account manually.
I want to add some with himank that if you need to provide some additional data for database you can up your fixture-datalink in fixture folder. Then other person will able to load those manually with command or even able to run a script link to populate initially data to database.
As the title says... I'm not sure if Django migrations should live in source control.
For:
If they get accidentally deleted from my local machine, it's going to cause me issues next time I want to run a migration... right? So it would be useful for me to have them.
Against:
Devs setting up the project for the first time shouldn't need to run them, they can just work straight from the models file.
They seem like machine-specific cruft.
Could they potentially reveal things I don't want about the database?
Yes, absolutely!!
From the docs:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
One big point is that migrations should always be tested before you deploy them in production. You should never create migrations on production, only apply them.
You also want to synchronise the state of the models in source control with the state of the database. If someone pulls your branch, has to find a bug, and goes back in the source control's history, he'd need the migration files to change the state of the database to match that point in time. If he has to create his own migration files, they won't include the intermediate state, and he runs into a problem where his models are out-of-sync with the database.
I'm trying to develop a Django website with Heroku. Having no previous experience with databases (except the sqlite3 one from the tutorial), it seems to me a good idea to have the following file structure:
Projects
'-MySite
|-MySite
'-MyDB
I'm finding it hard to figure out how to do it, with psql commands preferring to put the databases in some obscure directory instead. Perhaps it's not such a good idea?
Eventually I want to be able to test and develop my site (it'll be just a blog for a while, I'm still learning) locally (ie. add a post, play with the CSS) and sync with the Heroku repository, but I also want to be able to add posts via the website itself occasionally.
The underlying data files (MyDb) has nothing to do with your project files and should not be under your project.
EDIT
added two ways to sync your local database with the database ON the Heroku server
1) export-import
This is the most simple way, do the following steps every now and then:
make an export on the Heroku server by using the pg_dump utility
download the dump file
import the dump into your local database by using the psql utility
2) replication
A more sophisticated way for keeping your local db in sync all the time is Replication. It is used in professional environments and it is probably an overkill for you at the moment. You can read more about it here: http://www.postgresql.org/docs/9.1/static/high-availability.html
During development, I like the idea of frameworks like Entity Framework 4.3 Migrations (although I need it to work with sql scripts instead of Migration classes) keeping all developer databases up-to-date. Someone does an update to get the latest source, they try to run the application and get an error that they need to update their database to the latest migration (or have the migration happen automatically). Migration files are timestamped so devs don't have to worry about naming two files the same or the sequence the files need to be executed in.
When I get ready to build a WebDeploy deployment package, I want the package to contain the scripts required to move the production database to the latest db version. So somehow, MSBuild or WebDeploy need to make a decision about which scripts must be packaged. For deployment I don't want the application to try to update itself like what EF offers. I want to either hand the package to the IT department or do an auto-deploy through a deployment server.
Some of my questions are:
Can EF 4.3 work with sql scripts instead of DBMigration classes for my development needs (I'm already using it for my ORM so it would be nice if it could)?
Does MSBuild or WebDeploy understand the concept of database migrations (e.g. does it recognize the EF. 4.3 migrationHistory table) or do I have to make sure to give it just the scripts it needs to run that will bring my prod db to the latest migration? Manually deciding which scripts should be pakaged is not something I want to do so is there a MS WebDeploy extension that understands migrations?
Are my concerns and ideas valid? I'm just researching this stuff so I don't really know.
I think that your concerns are valid. During development anything that keeps the developer machines in sync will do. When deploying, more control is needed.
In my project we've decided to only use code-based migrations, to ensure that all databases are migrated in the same order with the same steps. Automatic migration and db creation is disabled by a custom initialization strategy that only checks that the db exists and is valid.
I've written more about the details in Prevent EF Migrations from Creating or Changing the Database. I've also looked a bit into merge conflicts which will eventually happen with multiple developers.
You can run SQL in the classes by calling the Sql method, or generate SQL from a migration by using the -script parameter with the update-database command.
No. They were looking at adding support for webdeploy but apparently decided against it before rtm. They did however release a command line app (and powershell script obviously) which you could call from either.
What I do in my projects is create an startup module in my applications and run any migrations that haven't been deployed automatically - Triggering EF migration at application startup by code. It's not perfect, developers have to run the app after getting latest before changing the db, but it works for both get latest and deployment.
I have a small python script that pushes data to my django postgres db.
It imports the relevant model from a django project and uses the .save function to save the data to the db without issue.
Yesterday the system was running fine. I started and stopped both my django project and the python script many times over the course of the day, but never rebooted or powered off my computer, until the end of the day.
Today I have discovered that the data is no longer in the db!
This seems silly, as I probably forgotten to do something obvious, but I thought that when the save function is called from a model, the data is committed to the db.
So this answer is "where to start troubleshooting problems like this" since the question is quite vague and we don't have enough info to troubleshoot effectively.
If this ever happens again, the first thing to do is to turn on statement logging for PostgreSQL and look at the statements as they come in. This should show you begin and commit statements as well as the queries. It's virtually impossible to troubleshoot this sort of problem without access to the queries. Things to look for include missing COMMITs, and missing statements.
After that, the next thing to do is to look at the circumstances under which your computer rebooted. Is it possible it did so before an expected commit? Or did it lose power and not have the transaction log flushed to disk in time?
Those two should rule out just about all possible causes on the db side in a development environment. In a production environment for old versions of PostgreSQL you do want to verify that the system has autovacuum running properly and that you aren't getting warnings about xid wraparound. In newer versions this is not a problem because PostgreSQL will refuse to accept queries when approaching xid wraparound.