Am trying to load some generated data into Django without disrupting the existing data in the site. What I have:
Saved the data as a valid JSON (validated here).
The JSON format matches the Django documentation. In previous attempts I also aligned it to the Django documentation here (slightly different field order, the result was the same).
Output errors I'm receiving are very generic and not helpful, even with verbosity=3.
The Error Prompt
Operations to perform:
Apply all migrations: workoutprogrammes
Running migrations:
Applying workoutprogrammes.0005_auto_20220415_2021...Loading '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures/datafile_2' fixtures...
Checking '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures' for fixtures...
Installing json fixture 'datafile_2' from '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures'.
Traceback (most recent call last):
File "/Users/Robert/Desktop/Projects/Powerlifts/venv/lib/python3.8/site-packages/django/core/serializers/json.py", line 70, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/Users/Robert/Desktop/Projects/Powerlifts/venv/lib/python3.8/site-packages/django/core/serializers/python.py", line 93, in Deserializer
Model = _get_model(d["model"])
KeyError: 'model'
The above exception was the direct cause of the following exception:... text continues on...
for obj in objects:
File "/Users/Robert/Desktop/Projects/Powerlifts/venv/lib/python3.8/site-packages/django/core/serializers/json.py", line 74, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/Users/Robert/Desktop/Projects/Powerlifts/src/workoutprogrammes/fixtures/datafile_2.json':
auto_2022_migration.py file:
from django.db import migrations
from django.core.management import call_command
def db_migration(apps, schema_editor):
call_command('loaddata', '/filename.json', verbosity=3)
class Migration(migrations.Migration):
dependencies = [('workoutprogrammes', '0004_extable_delete_ex_table'),]
operations = [migrations.RunPython(db_migration),]
JSON file extract (start... end)
NB: all my PKs are UUIDs generated from postgresql
[{"pk":"af82d5f4-2814-4d52-b2b1-6add6cf18d3c","model":"workoutprogrammes.ex_table","fields":{"exercise_name":"Cable Alternating Front Raise","utility":"Auxiliary","mechanics":"Isolated","force":"Push","levator_scapulae":"Stabilisers",..."obliques":"Stabilisers","psoas_major":""}}]
I'm not sure what the original error was. I suspect it had to do with either editing the table/schema using psql separately from Django, OR using the somewhat outdated uuid-ossp package instead of pgcrypto package (native to Djagno via CryptoExtension()). I was never able to confirm. I did however manage to get it working by:
Deleting the table (schema, data) using psql and the app using Django. Creating a new app in Django.
Load the prev model data into this new_app/models.py. Make new migration file, first updating the CryptoExtension() operation before committing the migration.
Create Fixtures directory and relevant file (per Django specs, validated JSON online.
Validate my UUIDs online
Load the fixture data into the app. python3 manage.py loaddata /path/to/data/file_name.json
Related
I'm doing a Django project and try to improve computing speed in backend.
The task is something like a CPU-bound conversion process
Here's my environment
Python 3.6.1
Django 1.10
PostgreSQL 9.6
And I stuck with following errors when I try to parallel a computing API by python multi-processing library.
File "D:\\project\apps\converter\models\convert_manager.py", line 1, in <module>
from apps.conversion.models import Conversion
File "D:\\project\apps\conversion\models.py", line 5, in <module>
class Conversion(models.Model):
File "C:\\virtenv\lib\site-packages\django\db\models\base.py", line 105, in __new__
app_config = apps.get_containing_app_config(module)
File "C:\\virtenv\ib\site-packages\django\apps\registry.py", line 237, in get_containing_app_config
self.check_apps_ready()
File "C:\\lib\site-packages\django\apps\registry.py", line 124, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
look like each process import Conversion model
and Conversion model is like
from django.db import models
Conversion(model.Model):
conversion_name = models.CharField(max_length=63)
conversion_user = models.CharField(max_length=31)
conversion_description = models.TextField(blank=True)
...
Below is my sample function which I want to parallel, each iteration is independent but will access or insert data into SQL.
Class ConversionJob():
...
def run(self, p_list):
list_merge_result = []
for p in p_list:
list_result = self.Couputing_api(p)
list_merge_result.extend(list_result)
and I'm try to do is
from multiprocessing import Pool
Class ConversionJob():
...
def run(self, p_list):
list_merge_result = []
p = Pool(process=4)
list_result = p.map(self.couputing_api, p_list)
list_merge_result.extend(list_result)
In computing_api(), it'll try to get current conversion's info which has completed and save into SQL before this api call, but this caused the error.
My question is
Why import Conversion model will caused Apps aren't loaded yet errors, I had google lots of article but not actually solve my problems.
I can see each Process SpawnPoolWorker-x generated and try to boot django server again(why?), each worker will stop at same errors.
computing API will try to access sql , I haven't think about how to deal with this work. (share db connections or create new connection in each process)
For others that might stumble upon this in future:
If you encounter this issue while running Python 3.8 and trying to use multiprocessing package, chances are that it is due to the sub processed are 'spawned' instead of 'forked'. This is a change with Python 3.8 on Mac OS where the default process start method is changed from 'fork' to 'spawn'.
This is a known issue with Django.
To get around it:
import multiprocessing as mp
mp.set_start_method('fork')
This post can solve the problem.
Django upgrading to 1.9 error “AppRegistryNotReady: Apps aren't loaded yet.”
I had found this answer before, but not actually solve my problems at that time.
After I repeated test, I have to add these codes before import another model,
otherwise, child-process will booting failed and give the error.
import django
django.setup()
from another.app import models
Since upgrading to django 1.8 I've been having some issues with datetime fields in my models not migrating correctly.
I was seeing this message repeatedly:
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
I run makemigrations and I get this:
operations = [
migrations.AlterField(
model_name='profile',
name='date_of_hire',
field=models.DateField(default=datetime.date(2016, 6, 5)),
),
]
So I run manage.py migrate and then i get:
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
So I run make migrations again and I get an new migration identical to the one above.
here's my problem field:
date_of_hire = models.DateField(default=datetime.date.today())
Looking at the migration I can see that the date is getting explicitly set to a fixed date. So now if I change my field to this:
date_of_hire = models.DateField(auto_now_add=True)
or this:
date_of_hire = models.DateTimeField(auto_now_add=True)
I get the error below when trying to run makemigrations or start my server:
File "/urls.py", line 13, in <module>
import profiles.views as profile_views
File "/profiles/views.py", line 9, in <module>
from profiles.forms import CompanyProfileForm
File "/profiles/forms.py", line 19, in <module>
class ProfileForm(ModelForm):
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 295, in __new__
raise FieldError(message)
django.core.exceptions.FieldError: Unknown field(s) (date_of_hire) specified for Profile
If I comment out that field in the forms.py fields list everything except the for the form works. I can make migrations and apply them, run the server, etcetera, but as soon as I uncomment that field the app take a crap. So I am at a loss...
In your default, you should pass the callable datetime.date.today, instead of calling it:
date_of_hire = models.DateField(default=datetime.date.today)
When you use default=datetime.date.today(), Django calls today() every time you load your models.py. This changes the default, so Django thinks a new migration is required.
You'll have to create one more migration to change the default to datetime.date.today (or edit the existing migrations but that will be trickier).
I have a very simple test as follows:
import models
from django.test import TestCase
MyViewTest(TestCase):
def setUp(self):
self.trip = models.Trip.objects.order_by('?')[0]
def test_something(self):
# Blah Blah
whenever i run test it throws the error mentioned below:
Traceback (most recent call last):
File "/home/amyth/Projects/test/trips/tests.py", line 8, in setUp
self.trip = models.Trip.objects.order_by('?')[0]
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 207, in __getitem__
return list(qs)[0]
IndexError: list index out of range
I also tried changing the query to models.Trip.objects.all()[0] and it still throws the same error. What's strange is if I use any of the above queries within the shell it works. Then howcome it is not working within a test ?
See the documentation on testing in django. A new 'test' database is created, and your 'production' database is not used. Unless you create Trip entries in the TestCase setUp method, it is empty. Also, after each TestCase is run, the database is truncated, so if you need to use the Trips in multiple TestCases, you will need to create the database entry for it in each TestCase setUp.
I've just started using south with an existing app, and after adding new models to the db, I can view the models in the admin, but when clicking on them to view the model details an error
I try to do the south equivalent of syncdb:
python manage.py schemamigration directory --initial
python manage.py migrate directory
where directory is the app name.
So when i try and view the model in admin I get the following:
Exception Type: DatabaseError
Exception Value: (1146, "Table 'omada.directory_drift' doesn't exist")
where Drift is the model I added to models.py, then registered in admin.py - omada is the site name.
Further Information:
Traceback from the django site ends with:
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py" in execute
114. return self.cursor.execute(query, args) File "/usr/local/lib/python2.7/dist-packages/MySQL_python-1.2.4b5-py2.7-linux-i686.egg/MySQLdb/cursors.py" in execute
201. self.errorhandler(self, exc, value) File "/usr/local/lib/python2.7/dist-packages/MySQL_python-1.2.4b5-py2.7-linux-i686.egg/MySQLdb/connections.py" in defaulterrorhandler
36. raise errorclass, errorvalue
When executing
python manage.py migrate directory
I get an error that starts with:
FATAL ERROR - The following SQL query failed: CREATE TABLE `directory_building`
! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.
which doesn't sound very promising :S, and ends with:
File "/usr/local/lib/python2.7/dist-packages/MySQL_python-1.2.4b5-py2.7-linux-i686.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
django.db.utils.DatabaseError: (1050, "Table 'directory_building' already exists")
Thanks in advance to everyone who takes the time to read this and offer help!
You may find this guide useful.
Since you are only starting with South I would advice to start all over:
Drop your south table (south_migrationhistory) from your database
Delete the migrations from the app.migrations folder
Follow the guide
It is quite easy to get South in a bad state so you just need to get some experience with it. You however can't live without it as your projects grow in size.
So I was developing an app in Django and needed a function from the 1.4 version so I decided to update.
But then a weird error appeared when I wanted to do syncdb
I am using the new manage.py and as You can see it makes some of the tables but then fails :
./manage.py syncdb
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Traceback (most recent call last):
File "./manage.py", line 9, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/commands/syncdb.py", line 91, in handle_noargs
sql, references = connection.creation.sql_create_model(model, self.style, seen_models)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/creation.py", line 44, in sql_create_model
col_type = f.db_type(connection=self.connection)
TypeError: db_type() got an unexpected keyword argument 'connection'
I had the same issue, the definition for my custom field was missing the connection parameter.
from django.db import models
class BigIntegerField(models.IntegerField):
def db_type(self, connection):
return "bigint"
Although already old, answered and accepted question but I am adding my understanding I have added it because I am not using customized type and it is a Django Evolution error (but not syncdb)evolve --hint --execute. I think it may be helpful for someone in future. .
I am average in Python and new to Django. I also encounter same issue when I added some new features to my existing project. To add new feature I had to add some new fields of models.CharField() type,as follows.
included_domains = models.CharField(
"set of comma(,) seprated list of domains in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
excluded_domains = models.CharField(
"set of comma(,) seprated list of domains NOT in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
The Django version I am using is 1.3.1:
$ python -c "import django; print django.get_version()"
1.3.1 <--------# version
$python manage.py syncdb
Project signature has changed - an evolution is required
Django Evolution: Django Evolution is an extension to Django that allows you to track changes in your models over time, and to update the database to reflect those changes.
$ python manage.py evolve --hint
#----- Evolution for messagingframework
from django_evolution.mutations import AddField
from django.db import models
MUTATIONS = [
AddField('MessageConfiguration', 'excluded_domains', models.CharField, initial=u'', max_length=300),
AddField('MessageConfiguration', 'included_domains', models.CharField, initial=u'', max_length=300)
]
#----------------------
Trial evolution successful.
Run './manage.py evolve --hint --execute' to apply evolution.
The trial was susses and when I tried to apply changes in DB
$ python manage.py evolve --hint --execute
Traceback (most recent call last):
File "manage.py", line 25, in <module>
execute_manager(settings)
File "/var/www/sites/www.taxspanner.com/django/core/management/__init__.py", line 362, in execute_manager
utility.execute()
File "/var/www/sites/www.taxspanner.com/django/core/management/__init__.py", line 303, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/www/sites/www.taxspanner.com/django/core/management/base.py", line 195, in run_from_argv
self.execute(*args, **options.__dict__)
File "/var/www/sites/www.taxspanner.com/django/core/management/base.py", line 222, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/management/commands/evolve.py", line 60, in handle
self.evolve(*app_labels, **options)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/management/commands/evolve.py", line 140, in evolve
database))
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/mutations.py", line 426, in mutate
return self.add_column(app_label, proj_sig, database)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/mutations.py", line 438, in add_column
sql_statements = evolver.add_column(model, field, self.initial)
File "/usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9.dev_r225-py2.7.egg/django_evolution/db/common.py", line 142, in add_column
f.db_type(connection=self.connection), # <=== here f is field class object
TypeError: db_type() got an unexpected keyword argument 'connection'
To understand this exception I check that this exception is something similar to:
>>> def f(a):
... print a
...
>>> f('b', b='a')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f() got an unexpected keyword argument 'b'
>>>
So the function signature has been changed.
Because I have not added any new customized or enum fields but only two similar fields that was already in model and char type field is supported by most of database (I am ussing PostgreSQL) even I was getting this error!
Then I read from #: Russell Keith-Magee-4 Reply.
What you've hit here is the end of the deprecation cycle for code that
doesn't support multiple databases.
In Django 1.2, we introduced multiple database support; in order to
support this, the prototype for get_db_preb_lookup() and
get_db_prep_value() was changed.
For backwards compatibility, we added a shim that would transparently
'fix' these methods if they hadn't already been fixed by the
developer.
In Django 1.2, the usage of these shims raised a
PendingDeprecationWarning. In Django 1.3, they raised a
DeprecationWarning.
Under Django 1.4, the shim code was been removed -- so any code that
wasn't updated will now raise errors like the one you describe.
But I am not getting any DeprecationWarning warning assuming because of newer version of Django Evolution.
But from above quote I could understand that to support multiple databases function signature is added and an extra argument connection is needed. I also check the db_type() signature in my installation of Django as follows:
/django$ grep --exclude-dir=".svn" -n 'def db_type(' * -R
contrib/localflavor/us/models.py:8: def db_type(self):
contrib/localflavor/us/models.py:24: def db_type(self):
:
:
Ialso refer of Django documentation
Field.db_type(self, connection):
Returns the database column data type for the Field, taking into account the connection
object, and the settings associated with it.
And Then I could understand that to resolve this issue I have to inherited models.filed class and overwrite def db_type() function. And because I am using PostgreSQL in which to create 300 chars type field I need to return 'char(300)'. In my models.py I added:
class CharMaxlengthN(models.Field):
def db_type(self, connection):
return 'char(%d)' % self.max_length # because I am using postgresql
If you encounter similar problem please check your underline DB's manual that which type of column you need to create and return a string.
And changed the definition of new fields (that I need to add) read comments:
included_domains = CharMaxlengthN( # <--Notice change
"set of comma(,) seprated list of domains in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
excluded_domains = CharMaxlengthN( # <-- Notice change
"set of comma(,) seprated list of domains NOT in target emails",
default="",
max_length=it_len.EMAIL_LEN*5)
Then I executed same command that was failing previously:
t$ python manage.py evolve --hint --execute
You have requested a database evolution. This will alter tables
and data currently in the None database, and may result in
IRREVERSABLE DATA LOSS. Evolutions should be *thoroughly* reviewed
prior to execution.
Are you sure you want to execute the evolutions?
Type 'yes' to continue, or 'no' to cancel: yes
Evolution successful.
I also check my DB and tested my new added features It is now working perfectly, and no DB problem.
If you wants to create ENUM field read Specifying a mySQL ENUM in a Django model.
Edit: I realized instead of sub classing models.Field I should have inherit more specific subclass that is models.CharField.
Similarly I need to create Decimal DB fields so I added following class in model:
class DecimalField(models.DecimalField):
def db_type(self, connection):
d = {
'max_digits': self.max_digits,
'decimal_places': self.decimal_places,
}
return 'numeric(%(max_digits)s, %(decimal_places)s)' % d