I have a Django 1.7 app, developed against sqlite3, and moving to Postgresql. One model table called Place has a parent relation to itself. I used dumpdata to create a .json extract.
When I attempt to use loaddata to read the extract into the Postgres db, Django (not postgres) gives me:
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/ugliest/project/thub/dbdump25jan_nat.json': Place matching query does not exist;
This tells me that django is validating the data before loading it into the database. Is there a way to allow this data in before validation? I know that Postgres has the ability to defer constraint checking, but even if I could trigger that from django, it wouldn't prevent django from validating. How to get the data in?
I feel like I'm missing something obvious. This seems like it would happen often, and have a known simple fix. Do I really have to write some custom mess to reorder the extract file?
Edit: FWIW, never found an answer, and wrote a custom mess to reorder things.
Related
i'm looking for a "best-practice" guide/solution to the following situation.
I have a Django project with a MySql DB which i created and manage. I have to import data, every 5 minutes, from a second (external, not managed by me) db in order to do some actions. I have read rights for the external db and all the necessary information.
I have read the django docs regarding the usage of multiple database: register the db in settings.py, migrate using the --database flag, query/access data by routing to the db (short version) and multiple question on this matter on stackoverflow.
So my plan is:
Register the second database in settings.py, use inspectdb to add to the model, migrate, define a method which reads data from the external db and add it to the internal (own) db.
However I do have some questions:
Do i have to register the external db if i don't manage it?
(Most probably yes in order to use ORM or the cursors to access the data)
How can i migrate the model if I don't manage the DB and don't have write permissions? I also don't need all the tables (around 250, but only 5 needed).
(is fake migration an option worth considering? I would use inspectdb and migrate only the necessary tables.)
Because I only need to retrieve data from the external db and not to write back, would it suffice to have a method that constantly gets the latest data like the second solution suggested in this answer
Any thoughts/ideas/suggestions are welcomed!
I would not use Django's ORM for it, but rather just access the DB with psycopg2 and SQL, get the columns you care about into dicts, and work with those. Otherwise any minor change to that external DB's tables may break your Django app, because the models don't match anymore. That could create more headaches than an ORM is worth.
For the app I'm building I need to be able to create a new data model in models.py as fast as possible automatically.
I created a way to do this by making a seperate python program that opens models.py, edits it, closes it, and does server migrations automatically but there must be a better way.
edit: my method works on my local server but not on pythonanywhere
In the Django documentation, I found SchemaEditor, which is exactly what you want. Using the SchemaEditor, you can create Models, delete Models, add fields, delete fields etc..
Here's an excerpt:
Django’s migration system is split into two parts; the logic for
calculating and storing what operations should be run
(django.db.migrations), and the database abstraction layer that turns
things like “create a model” or “delete a field” into SQL - which is
the job of the SchemaEditor.
Don't rewrite your models.py file automatically, that is not how it's meant to work. When you need more flexibility in the way you store data, you should do the following:
think hard about what kind of data you want to store and make your data model more abstract to fit more cases, if needed.
Use JSON fields to store arbitrary JSON data with your model (e.g. for the Postgres database)
if it's not a fit, don't use Django's ORM and use a different store (e.g. Redis for key-value or MongoDB for JSON documents)
All I did was add a field to a model, and now I get an error that says this column does not exist.
In an attempt to rebuild the database I used -flush (i dont care about losing the data), thinking this would rebuild the database, but I still get the same error.
I was told by someone else to use South because I'm running Django 1.6.
I followed the tutorial and literally the first time I ran syncdb, I got the following (probably unrelated) error:
dist-packages/easy-thumbnails/
raise Improperly_Configured(SOUTH_ERROR_MESSAGE)
django.core.exceptions.ImproperlyConfigured:
For South Support, customize the SOUTH_MIGRATION_MODULES setting like so:
South_Migration_Module = {
'easy_thumbnails': 'easy_thumbnails.south_migrations',
}
Ultimately all I want to do is have my db reflect my models. Back when I was working on this project in my dev environment I would literally just drag my sqlite file to the trash and then run syncdb, but I cannot do that now because I'm using postgres.
So my question is how can I accomplish this seemingly simple task? Whether that means addressing the South error, or just not using South altogether (which I would prefer), I would appreciated any help.
You can manually open up a db shell (>>> python manage.py dbshell) and drop the table with DROP TABLE <table_name>;. In order for syncdb to recreate a table with the new field, it is not enough to empty the table.
To solve the error, just do exactly what it says: add the SOUTH_MIGRATION_MODULES setting to your settings.
With that said, I'd definitely advice you to use south. It eases making changes to your models, and it allows you to preserve test data in your development environment that you'd otherwise have to recreate. That's all nice, but when your project goes live, it is absolutely mandatory that your data is preserved when making a change to your models. South is the best tool for that.
South has been such an integral part of pretty much any Django project, that Django has worked together with the developers of South to include it as a core feature from Django 1.7 onwards.
I'm not sure how to handle the following case (thus my question, obviously).
I have a django setup with postgresql to contains all the django model data, but I also have mongoengine managing (let's call them) extended data.
I also have a circular reference between the two (mongo_id points from django model to mongoengine document PK, and db_id points from mongoengine to django model PK).
Obviously, if I run dumpdata, I only get django model data. How can I make it to also dump data from mongoengine? Is there a way for me to achieve this?
This is to get a backup of the data. Backup of referenced files can be easily done by just grabbing the file on disk.
I did not define another DATABASES in the settings.py file (mainly because I was not required to). Is that what I need to do?
Thanks for any pointers.
As a bonus, I would appreciate if I could those mongoengine in the admin interface, but also the base django models.
First of all you can dump your data using mongodump
In one project we had to move data from one database to another with a significantly different schema so we created a management command that would do that. If you would want to use it in a similar manner it would have the advantage of moving only valid data for your current Document definitions and leaving out any possible leftovers from the older ones.
The dumping management command should contain something like
from bson import json_utils
json_util.dumps(map(lambda x: x.to_mongo(), SomeDocument.objects.all()))
I just tested it myself. I had Django models, and there have already been instances of the models in the database.
Then I added a dummy integer field to a model and ran manage.py syncdb. Checked the database, and nothing happened to the table. I don't see the extra field added in.
Is this the expected behavior? What's the proper way of modifying the model, and how will that alter the data that's already in the database?
Django will not alter already existing tables, they even say so in the documentation. The reason for this is that django can not guarantee that there will be no information lost.
You have two options if you want to change existing tables. Either drop them and run syncdb again, but you will need to store your data somehow if you want to keep it. The other options is to use a migrations tool to do this for you. Django can show you the SQL for the new database schema and you can diff that to the current version of the database to create the update script.
You could even update your database mannually if it is a small change and you don't want to bother with migrations tools, though I would recommend to use one.
Please use south for any kind of changes to get reflected to your database tables,
here goes the link for using south
Link for South documentation