I'm trying to migrate django apps, and i have problem with one. When i run
python manage.py migrate --list |grep -v "*"
i get output
dbtemplates
( ) 0001_initial
( ) 0002_auto__del_unique_template_name
( ) 0003_initial
I jut downloaded dbtemplates package, and in downloaded folder migrations are 3 files:
__init__.py
0001_initial.py
0002_auto__del_unique_template_name.py
So in my django project 0003 shouldn't be here i think. Should i remove 0003, mark as fake? If i need to remove 0003 migration, where can i find this package?
python manage.py shell
[1]: import dbtemplates
[2]: dbtemplates
This show the path package
Related
I have deployed my project to Heroku and currently trying to load the data dump from local sqlite database to the Heroku database. The remote database is clean and untouched other than the initial migrate command.
I have tried the following combinations of dump but all of them returned an error
python manage.py dumpdata --exclude contenttypes --> data.json
python manage.py dumpdata --exclude auth.permission --exclude contenttypes --indent 2 > data.json
python manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude auth.user --indent 2 > data.json
and the error is:
django.db.utils.IntegrityError: Problem installing fixture
'/app/data.json': Could not load wellsurfer.Profile(pk=6): duplicate
key value violates unique constraint "wellsurfer_profile_user_id_key"
DETAIL: Key (user_id)=(1) already exists.
i would like to post the json file here but it is about 120,000 lines. But i can provide specific portions if needed. The error clearly says the key exists but the database is clean in the beginning. Obviously, i am doing something very basic thing wrong and i hope you can point me in the right direction. I have tried recommendations that i found on Stackoverflow with no success. How to manage.py loaddata in Django
I had the same problem, and this is what worked for me
source (local sqlite)
python manage.py dumpdata --natural-foreign --indent 4 > datadump.json
(this will include everything, even the auth app / users)
destination (heroku postgres)
python manage.py migrate
python manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
>>> quit()
Finally, run following command to load the json data:
python manage.py loaddata datadump.json
Fairly new to Docker, I am trying to add the execution of a custom sql script (triggers and functions) to Django's migration process and I am starting to feel a bit lost. Overall, What I am trying to achieve follows this pretty clear tutorial. In this tutorial, migrations are achieved by the execution of an entry point script. In the Dockerfile:
# run entrypoint.sh
ENTRYPOINT ["/usr/src/my_app/entrypoint.sh"]
Here is the entrypoint.sh:
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
# tried several with and without combinations
python manage.py flush --no-input
python manage.py makemigrations my_app
python manage.py migrate
exec "$#"
So far so good. Turning to the question of integrating the execution of custom sql scripts in the migration process, most articles I read (this one for instance) recommend to create an empty migration to add the execution of sql statements. Here is what I have in
my_app/migrations/0001_initial_data.py
import os
from django.db import migrations, connection
def load_data_from_sql(filename):
file_path = os.path.join(os.path.dirname(__file__), '../sql/', filename)
sql_statement = open(file_path).read()
with connection.cursor() as cursor:
cursor.execute(sql_statement)
class Migration(migrations.Migration):
dependencies = [
('my_app', '0001_initial'),
]
operations = [
migrations.RunPython(load_data_from_sql('my_app_base.sql'))
]
As stated by dependencies, this step depends on the initial one (0001_initial.py):
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='Unpayed',
fields=[
etc etc
[The Issue] However, even when I try to manually migrate (docker-compose exec web python manage.py makemigrations my_app), I get the following error because the db in the postgresql container is empty:
File "/usr/src/my_app/my_app/migrations/0001_initial_data.py", line 21, in Migration
migrations.RunPython(load_data_from_sql('my_app_base.sql'))
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute
....
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "auth_user" does not exist
[What I do not understand] However, when I log in the container, remove 0001_initial_data.py and run ./entrypoint.sh, everything works like a charm and tables are created. I can add 0001_initial_data.py manually later on, run entrypoint.sh angain and have my functions. Same when I remove this file before running docker-compose up -d --build: tables are created.
I feel like I am missing some obvious way and easier around trying integrate sql script migrations in this canonical way. All I need is this script to be run after 0001_initial migration is over. How would you do it?
[edit] docker-compose.yml:
version: '3.7'
services:
web:
build:
context: ./my_app
dockerfile: Dockerfile
command: python /usr/src/my_app/manage.py runserver 0.0.0.0:8000
volumes:
- ./my_app/:/usr/src/my_app/
ports:
- 8000:8000
environment:
- SECRET_KEY='o##xO=jrd=p0^17svmYpw!22-bnm3zz*%y(7=j+p*t%ei-4pi!'
- SQL_ENGINE=django.db.backends.postgresql
- SQL_DATABASE=postgres
- SQL_USER=postgres
- SQL_PASSWORD=N0tTh3D3favlTpAssw0rd
- SQL_HOST=db
- SQL_PORT=5432
depends_on:
- db
db:
image: postgres:10.5-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
volumes:
postgres_data:
django:2.2
python:3.7
I believe the issue has to do with you naming the migration file and manually making your dependencies with the same prefix "0001" The reason I say this is because when you do reverse migrations, you simply can just reference the prefix. IE if you wanted to go from your 7th migration to your 6th migration. The command looks like this python manage.py migrate my_app 0006 Either way, I would try deleting and creating a new migration file via python manage.py makemigrations my_app --empty and moving your code into that file. This should also write the dependencies for you.
The error message alongside the test you ran with adding he migration file after is indicative of the issue though. Some how initial migrations aren't running before the other one. I would also try dropping your DB as it may have persisted some bad state ./manage.py sqlflush
[The easiest way I could find] I simply disentangled django migrations from the creation of custom functions in the DB. Migration are run first so that the tables exists when creating the functions. Here is the entrypoint.sh
#!/bin/sh
if [ "$DATABASE" = "postgres" ]
then
echo "Waiting for postgres..."
while ! nc -z $SQL_HOST $SQL_PORT; do
sleep 0.1
done
echo "PostgreSQL started"
fi
python manage.py flush --no-input
python manage.py migrate
# add custom sql functions to db
cat my_app/sql/my_app_base.sql | python manage.py dbshell
python manage.py collectstatic --no-input
exec "$#"
Keep in mind that manage.py dbshell requires a postgresql-client to run. I just needed to add it in the Dockerfile:
# pull official base image
FROM python:3.7-alpine
...........
# install psycopg2
RUN apk update \
&& apk add --virtual build-deps gcc python3-dev musl-dev \
&& apk add postgresql-dev postgresql-client\
&& pip install psycopg2 \
&& apk del build-deps
(I am aware that a number of Django users have had the same issue.
I have looked at a number of solutions online but none has worked for me so far.)
I have set up my apps.py, settings.py and models.py files as explained in Django official tutorial (please see the 3 files below).
When I enter in the terminal:
$ python3 manage.py makemigrations munichliving_app
It returns:
No changes detected in app 'munichliving_app'
(file settings.py) in INSTALLED_APP --> I added and tested both one at a time:
'munichliving_app' and
'munichliving_app.apps.MunichLivingConfig'
apps.py file: https://pastebin.com/raw/qaYy1x44
setting.py file: https://pastebin.com/raw/cSsbfPsx
models.py: https://pastebin.com/raw/U0QeM16k
Django official tutorial states that I should see something along the lines of:
Migrations for 'polls':
polls/migrations/0001_initial.py:
- Create model Choice
- Create model Question
- Add field question to choice
Thank you.
Your app is munichliving (the module that contains models.py), but you have munichliving_app in your INSTALLED_APPS setting. The munichlivin_app is the project folder (the one that contains settings.py). It doesn't normally contain models so you shouldn't usually have to add it to INSTALLED_APPS or make migrations for it.
Replace 'munichliving_app' with 'munichliving' in your INSTALLED_APPS.
Next, I would remove your apps.py because it doesn't appear to be used. If you do keep it, then change it to name='munichliving', then use'munichliving.apps.MunichLivingConfig'inINSTALLED_APPS`.
Finally, create migrations with
./manage.py makemigrations munichliving
Try this:
python manage.py migrate --fake appname
Or delete the migration folder in your app, go to the database and delete the file in django_migrations table, then migrate again:
python manage.py makemigrations
python manage.py migrate
In Django-South:
I changed I've run the initial migration successfully for myapp but for some reason, after I've made a change to my model and go to
./manage.py schemamigration myapp --auto
./manage.py migrate myapp
And I get a lot of traceback which ends in:
(1050, "Table 'my_table' already exists")
After much googling, I found and tried this:
./manage.py migrate myapp --fake
And then I proceed to migrate it, to no avail; same error.
Any suggestions?
I just got this same error, and found this question by search.
My problem was that my second migration I'd created using the --initial flag, i.e.
$ ./manage.py startapp foo
$ ./manage.py schemamigration --initial foo
$ ./manage.py migrate foo
... make some changes to foo ...
$ ./manage.py schemamigration --initial foo
(oops!)
$ ./manage.py migrate foo
... and I get the error, and the migration fails because in the second migration, South is trying to create a table its already created.
Solution
In my migrations folder:
$ ls foo/migrations
0001_initial.py 0002_initial.py
remove that second migration and re-export the second migration with the correct --auto flag:
$ rm foo/migrations/0002_initial.py
$ ./manage.py schemamigration --auto foo
$ ./manage.py migrate foo
Success!
There may be other things that cause this error, but that was my bad!
Is it an existing app?
In that case you will need to convert it in addition to the fake bit.
There are good docs here on converting an existing app.
Although they are quite tricky to find if you don't know where they are already ( ;
For converting, after adding south to your installed apps:
./manage.py syncdb
./manage.py convert_to_south myapp
./manage.py migrate myapp 0001 --fake
this problem actually happens if one of the cases:
1) You made "schemamigration app_name --initial" after one is "--auto"
2) You interrupted the last migration you have made.
To resolve such problem you apply the following:
1) mark your last schema migration as fake.
python manage.py schemamigration app_name --fake
Note: Make sure that the schema of models is same as schema of tables in database.
2) apply the migration again by doing
python manage.py schemamigration app_Name --auto
python manage.py migrate app-Name
Note: sometimes you might add manually a specific field you already added using the following syntax.
python manage.py schemamigration app_name --add-field My_model.added_field
For more info. regarding south, you could check its documentation here.
Is there anyway to load one fixture and have it load multiple fixtures?
I'd ideally like to type:
python manage.py loaddata all_fixtures
And have that load all of the data instead of having to type everything. Is this possible?
Using $ python manage.py loaddata myfixtures/*.json would work as Bash will substitute the wildcard to a list of matching filenames.
I have multiple apps on the project directory and have each app with its 'fixtures' directory. So using some bash I can do:
python3 manage.py loaddata */fixtures/*.json
And that expands all of the json files inside of the fixtures directory on each app in my project. You can test it by simply doing:
ls */fixtures/*.json
Why not create a Makefile that pulls in all your fixtures? eg something like:
load_all_fixtures:
./manage.py loaddata path/to/fixtures/foo.json
./manage.py loaddata path/to/fixtures/bar.json
./manage.py loaddata path/to/fixtures/baz.json
And then at the shell prompt, run
make load_all_fixtures
(This kind of approach is also good for executing unit tests for certain apps only and ignoring others, if need be)
This thread shows up among the first results with a Google search "load data from all fixtures" and doesn't mention what IMO is the correct solution for this, ie the solution that allows you to load any fixtures you want without any wildcard tricks nor a single modification of the settings.py file (I also used to do it this way)
Just make your apps' fixtures directories flat (and not the usual Django scheme that e.g. goes app_name/templates/app_name/mytemplate.html), ie app_name/fixtures/myfixture.[json, yaml, xml]
Here's what the django doc says :
For example:
django-admin loaddata foo/bar/mydata.json
would search /fixtures/foo/bar/mydata.json for each installed application, /foo/bar/mydata.json for each directory in FIXTURE_DIRS, and the literal path foo/bar/mydata.json.
What that means is that if you have a fixtures/myfixture.json in all your app directories, you just have to run
./manage.py loaddata myfixture
to load all the fixtures that are located there within your project ... And that's it ! You can even restrict what apps you load fixtures from by using --app or --exclude arguments.
I'll mention that I use my fixtures only to populate my database while doing some development so I don't mind having a flat structure in my 'fixtures' directories ... But even if you use your fixtures for tests it seems like having a flat structure is the Django-esque way to go, and as
that answer suggests, you would reference the fixture from a specific app by just writing something like :
class MyTestCase(TestCase):
fixtures = ['app_name/fixtures/myfixture.json']
My command is this, simple. (django 1.6)
python manage.py loaddata a.json b.json c.json
If you want to have this work on linux and windows you simply could use this for loading all your json-Fixtures:
import os
files = os.listdir('path/to/my/fixtures')
def loaddata(file):
if os.path.splitext(file)[1] == '.json' and file != 'initial_data.json':
print file
os.system("python manage.py loaddata %s" % file)
map(loaddata, files)
Works for me with Django-admin version 3.1.4
python manage.py loaddata json_file_1 json_file_2
My folder structure is like this -
app_name_1
├──fixtures
├────json_file_1.json
├────json_file_2.json
app_name_2
├──fixtures
├────json_file_3.json
Manage.py loaddata will look automatically in certain places, so if you name your fixtures the same in each app, or put all your fixtures in the same folder it can be easy to load them all. If you have many different fixtures, and need a more complex naming schema, you can easily load all your fixtures using find with -exec
find . -name "*.json" -exec manage.py loaddata {} \;
As I state in this [question][2], I also have this in a fabfile. EDIT: use python manage.py if manage.py is not in your VE path.
If your fixtures are located into the same folder, you can simply ls and xargs: ls myfolder | xargs django-admin loaddata.
Example with this structure:
$ tree fixtures/
root_dir/fixtures/
├── 1_users.json
├── 2_articles.json
└── 3_addresses.json
$ ls -d fixtures/* | xargs django-admin loaddata
would do the same as:
$ django-admin loaddata 1_users.json
$ django-admin loaddata 2_articles.json
$ django-admin loaddata 3_addresses.json
After doing a bit of searching, I ended up writing this script. It searches through all directories named "fixtures" for .json files and runs a "python manage.py loaddata {fixture_name}.json". Sometimes ordering matters for foreign key constraints, so it leaves a fixture in the queue if the constraint cannot be resolved.
(Note: It requires the pip package simple_terminal that I wrote. And I set it up to be run by 'python manage.py runscript ', which requires django-extensions.)
# load_fixture.py
#
# A script that searches for all .json files in fixtures directories
# and loads them into the working database. This is meant to be run after
# dropping and recreating a database then running migrations.
#
# Usage: python manage.py runscript load_fixtures
from simple_terminal import Terminal
from django.core.management import call_command
from django.db.utils import IntegrityError
def load_fixture(fixture_location):
# runs command: python manage.py loaddata <fixture_location>
call_command('loaddata', fixture_location)
def run():
with Terminal() as t:
# get all .json files in a fixtures directory
fixture_locations = t.command(
'find . -name *.json | grep fixtures | grep -v env')
while fixture_locations:
# check that every iteration imports are occuring
errors = []
length_before = len(fixture_locations)
for fl in fixture_locations:
try:
# try to load fixture and if loaded remove it from the array
load_fixture(fl)
print("LOADED: {}".format(fl))
fixture_locations.remove(fl)
except IntegrityError as e:
errors.append(str(e))
# if import did not occur this iteration raise exception due to
# missing foreign key reference
length_after = len(fixture_locations)
if length_before == length_after:
raise IntegrityError(' '.join(errors))
This works well for me; it finds all files located in fixtures directories within the the src directory:
python manage.py loaddata \
$(ls -1 src/**/fixtures/* | tr '\n' '\0' | xargs -0 -n 1 basename | tr '\n' ' ')
python manage.py loaddata ./*/fixtures/*.json
This command will look for the folder fixture in all the directory and then it will pick up all the files with json extension and will install the fixtures.
This way you won't have to have just one folder for fixtures instead you can have fixtures on app level and in multiple apps.
It will work with the ideal folder structure like this -
app_name_1
├──fixtures
├────json_file_1.json
├────json_file_2.json
app_name_2
├──fixtures
├────json_file_3.json