I have a media folder that stores all my uploaded images during development and I'm pushing my django(1.5.1) project to a dev server on heroku. Inside the media folder I have
media/
# cache and images were commited before .gitignored was added
cache/ # store thumbnails
images/ # store images
.gitignore
the .gitignore has
*
!.gitignore
The problem is whenever I git push to heroku, all my testing uploads are wiped out by git. Is there a way to deal with this?
This isn't an issue with Git, but rather with Heroku. Heroku's file system is ephemeral, and is reset between deploys. Use a service like Amazon S3 to store uploaded files.
Related
I am running a django application on Heroku, and currently using AWS S3 to serve my static files. We store our static files both in static folders per app, and also in a static/ folder at the root of the directory. The static/ folder at the root is about 40Mb large.
Whenever we deploy our app to Heroku, the static files are included in the Heroku slug, so that
heroku run python manage.py collectstatic --no-input can be run from the Dyno itself, which then copies any changed/new static files to our S3 bucket so that they can be served.
The issue is after we go through this process, we now have a static/ folder on the Dyno which takes up about 40Mb of space, and is seemingly useless since our files are being served from our S3 bucket!
Is there a better way to go about deploying our application, and collecting our static files to our S3 bucket but not copying the static files to Heroku?
One way I was thinking was to add all static files to Heroku's .slugignore file, and then configure a way to upload static files to our S3 bucket without using Heroku at all. I'm not sure if this is the correct way to go about it, however, and would appreciate advice on this.
The reason we have been looking into this is our Heroku slug size is starting to grow far too large (~450Mb), and we need to start reducing it.
After some more digging, I found examples of people doing exactly what I was describing above, which is uploading static files directly to S3 without using any intermediary storage. This article shows how to configure Django and S3 so that running python manage.py collectstatic on your local machine will copy the static files directly to S3.
This configuration, in combination with disabling collectstatic on Heroku (https://devcenter.heroku.com/articles/django-assets#disabling-collectstatic) and adding our static files to .slugignore, would be exactly what I was looking for, which was to upload static files directly to S3 without uploading them first to Heroku.
More reading from Django' docs
I have a Django application which is present in my local system.
As I want to ignore db.sqlite3 files while transferring the repository, I have put the following in .gitignore
db.sqlite3
I push it to Github using:
git push origin master
When I do this, the updated db.sqlite3 from local system is NOT transferred to git.
As the next step, I need to transfer the files from local system to Heroku using:
git push heroku master
However, it seems that the file from Github is copied to heroku, which is weird.
Perhaps my understanding of the git push heroku master is incorrect.
Deployment method is using heroku cli
To check this weird way of working :
I added couple of entries in db.sqlite3 in my local system
I made a small change to the code in my local system
I made new entries in the Django application which is deployed to heroku
I pushed the application to Github using git push origin master and checked the timestamp on db.sqlite3 in git and it wasn't changed - I downloaded the db.sqlite3 from git and checked, the new entries that I made to the local system weren't there. This is good.
I pushed the application to Heroku using git push heroku master and found that the entries which I made in step 3 are gone and the entries in step 1 are also not reflected.
I checked my Github db.sqlite3 file and heroku db.sqlite3 file and they matched.
My requirements are as follows :
The changes to the data in db that I make in my local system should not reflect in the application deployed to heroku (I believe therefore .gitignore -> db.sqlite3)
The structural and the application changes should only go to production.
Any pointers in the right direction ?
I figured this out, like my last two queries.
I was misled by this command :
git update-index --assume-unchanged db.sqlite3
Though this link clearly tells not do to so.
For the solution, git and .gitignore works perfectly fine (stating the obvious) . It requires only one entry called db.sqlite3 and you need to ensure that you do not send db.sqlite3 to heroku. You need to have your .gitignore file updated with db.sqlite3 and use PostgreSQL in heroku.
When I did this, I received an error called django-session not setup. Basically it meant that PostgreSQL is not ready for use. You need to ensure that you are ready to follow the steps below.
Few things to remember :
When experimenting with Django, locally you use db.sqlite3 and eager to send the database file db.sqlite3 to heroku and do not make entries in .gitignore. Don't do that.
In local, use db.sqlite3 and while deploying to heroku , use PostgreSQL
Create a virtual environment using pipenv
Use pipenv install psycopg2
Use heroku run bash -a <appname>
Go to manage.py folder and run python manage.py migrate
Create your superuser python manage.py createsuperuser
This worked for me. I shall come back and update this a bit more. Three days of brain-wreck.
Finally keep searching in Github , we have a goldmine of problems and solutions already provided. Sometimes we just need to connect the dots.
I have make a first push to Gitlab with git, but the.gitignore wasn't completely configure.
I configure the .gitignore like this now :
# Environments
env/
# Byte-compiled / optimized / DLL files
poc_project/poc_project/__pycache__
poc_project/pollution/__pycache__
__pycache__/
*.py[cod]
*$py.class
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
The env/ and poc_project/poc_project/__pycache__ aren't push but the db.sqlite3 and poc_project/pollution/__pycache__ are already on the distant repository (Gitlab).
I use this https://github.com/github/gitignore/blob/master/Python.gitignore for configure my .gitignore because I use django. My teammate will soon start working on the project.
It's a problem to have pycache file and db.sqlite3 for team work on distant repository with Django?
If yes how can I delete these files correctly from gitlab repository ?
If they do not contain any sensitive information: just delete the files and commit the deletion. If they do contain sensitive information they would still be present in the history, even after you delete the files. That requires some additional work, see: Remove sensitive files and their commits from Git history
I wanna make some changes to my Django site running on Heroku, how can i upload my new changes with git push without touching my database, so i don't loose any data?
You would typically have your database on a seperate dyno on heroku. For e.g. you might have configured heroku-postgresql to keep your DB. When you git push to heroku it only moves your application to the heroku server, and doesn't overwrite the database, i.e. unless you're using some DB like sqlite3 which might be saving the data in a local file within the application directory.
So I was finally able to set up local + prod test project I'm working on.
# wsgi.py
from dj_static import Cling, MediaCling
application = Cling(MediaCling(get_wsgi_application()))
application = DjangoWhiteNoise(application)
I set up static files using whitenoise (without any problems) and media (file uploads) using dj_static and Postgres for local + prod. Everything works fine at first... static files, file uploads.
But after the Heroku dynos restart I lose all the file uploads. My question is, --- Since I'm serving the media files from the Django app instead of something like S3, does the dyno restart wipe all that out too?
PS: I'm aware I can do this with AWS, etc, but I just want to know if thats the reason I'm losing all the uploads.
Since I'm serving the media files from the Django app instead of something like S3, does the dyno restart wipe all that out too?
Yes!. That's right. According to the Heroku docs:
Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code.
See, also this answer and this answer.
Conclusion: For media files (the uploaded ones), you must use some external service (like S3 or something). whitenoise is just for static files. See here why whitenoise is not suitable for serving user-uploaded (media) files.