I have a Django app with a SQLite database. This app is deployed on Heroku.
When someone uses the app and add data into the database, the sqlite database is modified. But when I make a code change on Github and then deploy this new version of my app, I also deploy the sqlite database (which doesn't have the new changes made by the users) and so I remove the changes.
What is the best process to prevent this ?
The solution is simple: you must not use sqlite on Heroku. As you have discovered, the file system on Heroku is ephemeral, and changes aren't persisted between deploys. Plus, if you scaled to more than one dyno, they wouldn't share the same database.
Use the Postgres add-on instead.
I don't know much about Heroku but if there's a seperate copy of your databse files on Heroku then you should add your sqlite files to the .gitignore file. This way you sqlite files won't be sent to Heroku along with the rest of the code. Also you'll need to delete the sqlite files from your repo which have already been added. Search more about gitignore. A general .gitignore for Django projects contains the following -
*.pyc
__pycache__
myvenv
db.sqlite3
.DS_Store
settings.py
Just copy this to your existing gitignore or make a new if not already present. A gigignore file is usually present in the root directory of the project.
Related
I have a django REST project and a PostgreSQL database deployed to DigitalOcean. When I develop locally, I have a separate dockerized REST server and a separate PostgreSQL database to test backend features without touching production data.
My question arises when I'm adding/modifying model fields that require me to make migrations using python [manage.py](https://manage.py) makemigrations and python [manage.py](https://manage.py) migrate command. Here is my current situation so far:
What I was supposed to do
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
What I did so far
IN LOCAL ENV, created the migration files,
python manage.py makemigrations
python manage.py migrate
I committed & pushed the changes to production WITHOUT the created migration file
IN PRODUCTION ENV, ran BOTH commands.
python manage.py makemigrations
python manage.py migrate
The production server successfully added the isActive field to the database and is working fine, but I still have a 0011_user_isActive.py migration file in my local changes that hasn't been staged/committed/pushed to github repo.
And because I ran makemigrations command in production env, it probably created the same migration file that I haven't pushed from local env.
My questions are:
What happens if I push the local migration file to production? Wouldn't it create a conflict when I run migration command on digitalocean console in the future?
How should I fix this situation?
I am just scared I'm going to corrupt/conflict my production database as I'm very inexperienced in databases and have too much to risk at the moment. Would appreciate any tips on best practices when dealing with such situations!
As docs says:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
So it's best practice is to push your migration files and make your local and production migration files sync.
And if you got conflict when pushing migraions files and pulling them, the makemigrations --merge command is for solving that.
Also docs says:
Because migrations are stored in version control, you’ll occasionally come across situations where you and another developer have both committed a migration to the same app at the same time, resulting in two migrations with the same number.
Don’t worry - the numbers are just there for developers’ reference, Django just cares that each migration has a different name. Migrations specify which other migrations they depend on - including earlier migrations in the same app - in the file, so it’s possible to detect when there’s two new migrations for the same app that aren’t ordered.
When this happens, Django will prompt you and give you some options. If it thinks it’s safe enough, it will offer to automatically linearize the two migrations for you. If not, you’ll have to go in and modify the migrations yourself - don’t worry, this isn’t difficult, and is explained more in Migration files below.
Also be aware that in case of updating existed data in production, you can use RunPython in migration file. Read about it here.
I took over a Django project and discovered that migrations for various apps were not being tracked by Git. That seemed a bit problematic since my understanding was that one should always track the migrations. Is that pretty much the consensus or are there reasons not to do it?
The second part of the question has to do with the fact that I have done some work involving tweaking some migrations locally. I would now like to push the changes to production. However, I am unsure as to what the best way to combine the possible conflicts would be.
For instance, in production, I have the following:
Untracked files:
(use "git add <file>..." to include in what will be committed)
mysite/aldryn_forms/migrations/0019_auto_20200730_1455.py
mysite/apps/common/migrations/0016_auto_20200624_2028.py
mysite/apps/common/migrations/0016_auto_20200625_1125.py
mysite/apps/common/migrations/0017_merge_20200625_1129.py
mysite/apps/common/migrations/0018_auto_20200720_1743.py
mysite/apps/payment/migrations/0005_auto_20200624_2028.py
mysite/apps/payment/migrations/0005_auto_20200625_1125.py
mysite/apps/payment/migrations/0006_merge_20200625_1129.py
mysite/apps/payment/migrations/0007_auto_20200720_1743.py
mysite/apps/payment/migrations/0008_paymentmodel_course.py
mysite/apps/payment/migrations/0009_paymentmodel_user.py
mysite/apps/plugins/migrations/0016_auto_20200624_2028.py
mysite/apps/plugins/migrations/0016_auto_20200625_1125.py
mysite/apps/plugins/migrations/0017_merge_20200625_1129.py
mysite/apps/xyz/migrations/0005_auto_20200730_1455.py
Locally, I have the following:
Untracked files:
(use "git add <file>..." to include in what will be committed)
mysite/aldryn_forms/migrations/0019_auto_20201108_1623.py
mysite/apps/common/migrations/0016_auto_20201108_1623.py
mysite/apps/common/migrations/0017_auto_20201108_1806.py
mysite/apps/payment/migrations/0005_auto_20201108_1623.py
mysite/apps/plugins/migrations/0016_auto_20201108_1623.py
mysite/apps/xyz/migrations/0005_auto_20201108_1623.py
These are the files with custom work:
mysite/apps/common/migrations/0016_auto_20201108_1623.py
mysite/apps/common/migrations/0017_auto_20201108_1806.py
It appears that all the migrations existing on the production server have been applied to the production database. Hence, I have concluded that they correctly describe the state of the production DB.
What should I do to sync my local work with that currently in-production?
First I would review code for conflicts in your new migrations and migrations aplied in production.
Then backup evrything (your site and production).
if you don't finde logic errors, try copy migrations files from production to yours site and go with :
python manage.py makemigrations –merge
python manage.py migrate
if it works then it will propably work in production.
There is article about it :
https://www.algotech.solutions/blog/python/django-migrations-and-how-to-manage-conflicts/#:~:text=So%2C%20in%20order%20to%20allow,manage.py%20makemigrations%20%E2%80%93merge)
once i had it too.. annoing is that you will have to do this -merge evry new migration.
At some point I reset migrations.
When you will have database in order. You can do:
backup db and remove insert from db table : migrations
delete all migration files from porject dir
qoute all paterns in urls.py
go with makemigrations and migrate
populete new db with old data (exept migration table)
oh there is even simpler method with --fake:
https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html
that will put you with initial migration.
I wanna make some changes to my Django site running on Heroku, how can i upload my new changes with git push without touching my database, so i don't loose any data?
You would typically have your database on a seperate dyno on heroku. For e.g. you might have configured heroku-postgresql to keep your DB. When you git push to heroku it only moves your application to the heroku server, and doesn't overwrite the database, i.e. unless you're using some DB like sqlite3 which might be saving the data in a local file within the application directory.
I've deployed my Django application for a while now, but all this time I've kept migrations out of souce control. What I did was run makemigrations and migrate twice-- once in development, and once in production. So now how can I seamlessly put my migrations into source control? Theoretically the migrations should be the same (given my models.py files haven't changed), but how would I go about matching my migrations up without accidentally overriding something?
I'm not sure if I understand the problem...
Django keeps track of which migrations have been applied using the database. But it uses the names, not the contents. So as long as you're not changing the contents of migrations that have already been applied, you should be good.
My team has had the Django migration files in .gitignore , and so hasn't been committing these files. Instead we've been making migrations on our production server. We discovered that this is not the recommended practice (upon encountering Should I be adding the Django migration files in the .gitignore file? ). Would it cause problems to - at this point in time - remove migrations from .gitignore, makemigrations on the development machine, commit, push, and apply the migrations on the production server? If so, how can we get around these problems?
This question is motivated largely because we need to apply a custom migration that we partially wrote ourselves. (As such there's actually an additional step between making migrations on the development machine and committing them, namely adding in our custom code to the migration file).
I think the only way is to copy over all the migrations from production server to your repo then commit them. This is most likely a manual process because your production server is the only place that tracks all migrations. You don't need to worry about how to migrate your production server because it keeps the original copy. However, all these should be done before any new migrations are added and applied.
After all is fixed, you should create new migrations in your local dev environment, add to git and push your migration to production then apply the migration. Remember to have your CI or something else check for duplicate migration files.