So, in a bit more detail I have a model with a field like: permalink = models.IntegerField(default=0)
I've not actually been using this field - but would now like to.
However, it seems all models on this table, permalink is now 57295730 - on all 2000 models!
In an attempt to debug, I tried completly wiping the DB, running migrate (~100 migrations) - but then creating a instance of the model, I am told permalink violates the not-null constraint though I am definitely passing it a value! I also get a list of the values I am passing it, but am not sure how to know which value/column relates to which field?
I've even tried removing DB, removing migrations, running a new makemigrations - and still get the null violation...
even stranger, it looks like this field has not been touched since the initial migration!
migrations$ egrep permalink *
0001_initial.py: ('permalink', models.IntegerField(default=0)),
migrations$
I'm running (k)ubuntu 14.04, postgres 9.3, python 3.4, django 1.9.4
Though I'd love to know how to fix this - my question is really "What can I do to debug this kind of situation?"
Well Not the answer I want - but a working answer:
do automated testing
use CI! Prevent this problem in the first place
And if you are not doing the above...
use git bisect (or if you cant, manually use git reset --hard HEAD~1 to find the problem!)
in mycase, I was over-riding the save function of the model in a... stupid way!
edit:
in a little more detail, I was setting permalink to be 1 greater than the current biggest value - but in an earlier commit, had removed the + 1
However, I did not notice this error quickly, as it did not happen with data in the DB.
So! the error was actually quite informative - had I been running my tests more often (or using CI) I would have been informed of the error instantly, and saved myself quite a headache!
so, in short: **write tests, run them - automatically **
Related
I'm working on a Django Project with Wagtail which uses two databases. The first one is the standard sql lite database for all django models (called db_tool.sqlite3), the other one is also sql lite but for a wagtail integration (called db.sqlite3).
I wanted to migrate to the db_tool.sqlite3 with the following command
python manage.py make migrations
python manage.py migrate --database db_tool
but now I get the following error message regarding wagtail, which I never got before.
django.db.utils.IntegrityError: NOT NULL constraint failed: wagtailcore_page.draft_title
First of all: I don't understand this, because I named the db_tool in particular and I wonder, why the wagtail integration raises an error when I try to migrate to db_tool.
Second: I see no particular field at my wagtail-pages called draft_title and I don't have any draft page at the moment.
Third: the error message also relates to a migration file of wagtail that can be found in the side-packages (see below). So maybe this is the root of the error, but I don't understand the correlation to the other error message, because since now it worked fine and I changed nothing exept of some content of my wagtail pages.
File "C:\Users\pubr\.conda\envs\iqps_web\lib\site-packages\wagtail\core\migrations\0001_squashed_0016_change_page_url_path_to_text_field.py", line 23, in initial_data
root = Page.objects.create(
The wagtail version I use here is wagtail 2.15.2 and I haven't updated it since I started the project...
Due to the fact, that my wagtail-database has the name of the default django-database, could it be possible, that I accidentally tried a migration which was ment for the tool_db.sqlite3 without naming it in the migrate-command and caused this error by doing that?
So I would be very grateful if anyone knows, where the error comes from, or at least, what I could try out to fix it...
Kind regards and thank you!
It isn't clear to me if your database is currently broken or not. Hopefully not, but if it is, please take a back up of each before doing anything else.
This does sound like you might have been trying to operate in the wrong database. Do you have DATABASE_ROUTERS configured? I think that might help you prevent code from one app from getting introduced into the wrong database. The example in the Django docs is mostly focused on read replicas but should be adaptable to your situation: https://docs.djangoproject.com/en/4.1/topics/db/multi-db/#an-example
If your databases are in an incorrect state, start by looking at the django_migrations file in each and then carefully pruning the messed up one until you get back to the separation you have been enforcing.
This question already has an answer here:
Migrating problems when porting Django project to Python 3 and Django 2
(1 answer)
Closed 3 years ago.
I'm working on upgrading my legacy application from Django 1.11.13 to 2.2.8. I've dutifully addressed every compatibility issue, but I've hit one I can't figure out how to resolve. When I try to start the webserver in my local environment, I get this error (only showing the end of the full error trace that appears):
File "/Users/me/my_app/my_model/migrations/0001_initial.py", line 37, in Migration
('entry', models.ForeignKey(to='my_model.Entry')),
TypeError: __init__() missing 1 required positional argument: 'on_delete'
I understand why on_delete is now required -- I just spent a while updating my models everywhere to accommodate this change -- but I have no idea how to fix this particular issue without going through dozens of old migrations files to make them conform?!
I tried squashmigrations to at least collapse the number of places I have to clean up, but I got the same exact TypeError.
I tried to use the old version of Django for squashmigrations. I was successful in avoiding the TypeError, but ended up with a huge mess of circular import errors.
Since I don't actually need the migration history to roll back, I tried to follow these instructions (scenario 2) to clear the migration history while keeping the existing database, but I couldn't run makemigrations to catch up on the changes I made to make my models Django 2.2 compliant, and when I decided I'd skip ahead and deal with that later, showmigrations failed with the same TypeError. (Is there some other way to get a fresh set of initial migrations based on the current database? It can't be based off the models since the models have upgrade-related changes not yet reflected in the database.)
I moved the migrations to a non-standard location, which got the server to start, but that makes it impossible to actually do anything migration related ever again, and of course once I move back, everything breaks again...
I've considered just deleting my entire database and all migration history, building the tables from scratch with a fresh set of initial migrations, and then resetting the data from a backup, but there are a few huge tables which would make this take quite a while... and this rather seems like the nuclear approach. Am I stuck with editing a large number of very old migrations to be compliant with Django 2.2 for no actual reason since I'm never going to roll my project that far back? How can that be right?
As Iain Shelvington mentions in a comment under the question,
First delete all of your migration files and folder, then run makemigrations with the "on_delete" - this should create some "initial" migration files. Then you'll have to log in to your DB and delete all entries for your apps and then you need to run manage.py migrate --fake - this will enter into the DB entries for the newly created migrations but will not apply them
I have a Video and VideoLog model. Until now I've been happy enough annotating on a count to see how many times a video has been watched:
Video.objects.annotate(
views=Count('logs'), # logs is the related name from the VideoLog model
)
This works fine on SQLite but we've just upgraded to Postgres 9.6 and trying to run that gives me
ProgrammingError: column "video_video.name" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: SELECT "video_video"."id", "video_video"."name", "video_vide...
Here's the formatted query it's running:
SELECT "video_video"."id",
"video_video"."name",
"video_video"."shorttext",
"video_video"."text",
"video_video"."thumbnail",
"video_video"."product_id",
"video_video"."slug",
"video_video"."available_sizes",
"video_video"."preview_image",
"video_video"."certificate",
"video_video"."download",
Count("video_videolog"."id") AS "views"
FROM "video_video"
LEFT OUTER JOIN "video_videolog"
ON ( "video_video"."id" = "video_videolog"."video_id" )
GROUP BY "video_video"."id"
The confounding thing is that in another project on the same database, with the similar dependencies, both on fresh virtualenvs, this works fine. I guess I'm going to have to pick through this line-by-line until I find something.
I used pgloader to upgrade the database. Turns out it also broke a few things. Not sure where, but it must have been pretty subtle because everything else was working quite well.
I know people rag on the ./manage.py dumpdata → ./manage.py loaddata workflow (somebody in IRC called it a toy yesterday when I asked for a better option) but if your data is sane, it's bloody accurate. It just saved my bacon.
We've changed our database, using django migrations (django v1.7+).
The data that exists in the database is no longer valid.
Basically I want to test a migration by, inside a unittest, constructing the pre-migration database, adding some data, applying the migration, then confirming everything went smoothly.
How does one:
hold back the new migration when loading the unittest
I found some stuff about overriding settings.MIGRATION_MODULES but couldn't work out how to use it. When I inspect executor.loader.applied_migrations it still lists everything. The only way I could prevent the new migration was to actually remove the file; not a solution I can use.
create a record in the unittest database (using the old model)
If we can prevent the migration then this should be pretty straightforward. myModel.object.create(...)
apply the migration
I think I can probably work this out now that I've found the test_executor: set a plan pointing to the migration file and execute it? Um, right? Got any code for that :-D
confirm the old data in the database now matches the new model
Again, I expect this should be pretty easy: just fetch the instance created before the migration and confirm it has changed in all the right ways.
So the challenge is really just working out how to prevent the unittest from applying the latest migration script and then applying it when we're ready?
Perhaps I have the wrong approach? Should I create fixtures, and just confirm that they're all good at the end? Do fixtures get loaded before the migrations are applied, or after they're all done?
By using the MigrationExecutor and picking out specific migrations with .migrate I've been able to, maybe?, roll it back to a specific state, then roll forward one-by-one. But that is popping up doubts; currently chasing down sqlite fudging around due to the lack of an actual ALTER TABLE instruction. Jury still out.
I wasn't able to prevent the unittest from starting with the current database schema, but I did find it is quite easy to revert to earlier points in the migration history:
Where "0014_nulls_permitted" is a file in the migrations directory...
from django.db.migrations.executor import MigrationExecutor
executor.migrate([("workflow_engine", "0014_nulls_permitted")])
executor.loader.build_graph()
NB: running the executor.loader.build_graph between invocations of executor.migrate seems to be a very important part of completing the migration and making things behave as one might expect
The migrations which are currently applicable to the database can be checked with something like:
print [x[1] for x in sorted(executor.loader.applied_migrations)]
[u'0001_initial', u'0002_fix_foreignkeys', ... u'0014_nulls_permitted']
I created a model instance via the ORM then ensured the database was in the old state by running some SQL directly:
job = Job.objects.create(....)
from django.db import connection
cursor = connection.cursor()
cursor.execute('UPDATE workflow_engine_job SET next_job_state=NULL')
Great. Now I know I have a database in the old state, and can test the forwards migration. So where 0016_nulls_banished is a migration file:
executor.migrate([("workflow_engine", "0016_nulls_banished")])
executor.loader.build_graph()
Migration 0015 goes through the database converting all the NULL fields to a default value. Migration 0016 alters the schema. You can scatter some print statements around to confirm things are happening as you think they should be.
And now the test can confirm that the migration has worked. In this case by ensuring there are no nulls left in the database.
jobs = Job.objects.all()
self.assertTrue(all([j.next_job_state is not None for j in jobs]))
We have used the following code in settings_test.py to ignore the migration for the tests:
MIGRATION_MODULES = dict(
(app.split('.')[-1], '.'.join([app, 'nonexistent_django_migrations_module']))
for app in INSTALLED_APPS
)
The idea here being that none of the apps have a nonexistent_django_migrations_module folder, and thus django will simply find no migrations.
I have a Django web site running a mini CMS we've built internally years ago, it's using postgresql. When saving a simple title and a paragraph of text I get the following error:
value too long for type character varying(100)
The weird thing is, not a single column is varying(100) they are all 200 or 250, even the default Django ones have been changed from the 100 to 200 due to a re-opened ticket mentioned here
Does anyone know of a solution to this problem?
I can bet money you have a models.SlugField without length set. The default length is 50 characters, most likely it's not enough for your use case.
Change it to models.SlugField(max_length=255) and migrate your database schema.
I also had this problem when using a filefield and was scratching my head for a while. Of course the default FileField instances are created with a 100 character limit.
https://docs.djangoproject.com/en/dev/ref/models/fields/#filefield
This is an error message from Postgres and not django.
You seem to have changed the length of the field in the models.py, but that doesn't change the database length which was created when you did a manage.py syncdb.
You have to alter the length of the field in the database, directly.
Django 2.1
I encountered this problem while switching from sqlite3 to postgresql.
Remove the migration files in each app's migrations folder except __init__.py
Then re-run migration
(venv)myapp$python manage.py makemigrations
(venv)myapp$python manage.py migrate
(venv)myapp$python manage.py runserver
I had a similar problem with django-autoslugfield
I was using a similar package and then switched over to django-autoslugfield
I was getting this error:
value too long for type character varying(50)
despite the fact that my models.py had:
slug = AutoSlugField(max_length=255, populate_from='name', unique=True)
and in my db the it the type was
character varying 255
once i remove max_length=255 from the field i.e.
slug = AutoSlugField(populate_from='name', unique=True)
then it worked fine
i went through this same error. and when i made changes in the modele, i kept having the same error.
Here is how i fixed it.
It might be necessary to skip few migrations for the program to only use the migration where changes have been made for the CharField max_lenght.
for that you have to run
python manage.py showmigrations
to see which migrations have not been made.
then you skip them until you get to the last with the command
python manage.py migrate <app> 000_migration_number --fake
I realize the question is already answered but for others that come here when looking for the error message:
In my case the problem was that my table name exceeded 50 characters. Apparently this is not allowed. Changing the table name solved the problem.
Read more here: https://code.djangoproject.com/ticket/18959
Michael Samoylov's answer pointed me in the right direction. I had the same error come up, except it was with a FileField.
Fields have a max_length, even if you have not explicitly set a max_length. Increase the value so that your data fits to avoid the error.
In my case, the data was too large to reasonably store in the database. I resorted to saving my file to disk, then saving the file path in the database.
I had this problem when I wanted to set max_length to a lower value on a FileField.
Interestingly, the error message states value too long but in my case the value was too short.
The solution was to set back the max_length to its old value. It's quite weird that the migration couldn't be done.
I also had to delete the wrongly generated migrations files and rerun python manage.py makemigrations and python manage.py migrate.
If it's not SlugField, FileField, or any other field mentioned here--scroll back to where the migration got stuck in the terminal. For me it was AddField
Good talk.
First, try setting max_length to something reasonable on all applicable field types in your model.
For example: MyText = models.CharField(max_length=2000)
If you don't set a max_length, your database might be applying a default max_length, shorter than the length of your input data, causing your value too long for type character error.
If that doesn't work and you started in SQLite and changed databases to PostgreSQL, the previous migrations from SQLite might be interfering with the new PostgreSQL migrations.
Go into the migrations folder for your project and delete the old migration files to get a fresh start. Then try makemigrations and migrate again :)
predefined fields in model.py creates the problem. Extend it to desired length, i think problem will be resolved.
For the FileField and the ImageField, from the Django docs:
FileField instances are created in your database as varchar columns with a default max length of 100 characters. As with other fields, you can change the maximum length using the max_length argument.
The value can be 100 or 25 or 17 whatever , the reason behind that is models , search where you have added that particular length(17,25) and you will get the bug ! ,
You are trying to impose a field length more than mentioned one.
This solved my issue , I hope it will help you too