Sqlite table data not loaded into Postgresql database on Heroku [duplicate] - django

This question already has answers here:
Django: What are the best practices to migrate a project from sqlite to PostgreSQL
(7 answers)
Providing initial data for models - Django
(1 answer)
Closed 4 months ago.
This post was edited and submitted for review 4 months ago and failed to reopen the post:
Original close reason(s) were not resolved
I've been working on a Django project locally and used SQLITE as the database. I've now deployed the project to Heroku-website and used Postgresql as the database. The database schemas have moved over fine but the data from the tables has not migrated.
What is my problem?
Here is my database code from settings file.
DATABASES = {
'default': {
"ENGINE": "django.db.backends.sqlite3",
"NAME": os.path.join(BASE_DIR, "db.sqlite3"),
}
}
import dj_database_url
db_from_env = dj_database_url.config(conn_max_age=600)
DATABASES['default'].update(db_from_env)

You should use engine name only in database while in production.
In settings.py file:
"ENGINE": "django.db.backends.postgresql_psycopg2"
Transfer data from local to production.
Data transfer
Install the Heroku PGBackups add-on:
heroku addons:add pgbackups
Dump your local database:
pg_dump -h localhost -Fc library > db.dump
In order for Heroku to access db dump, you need to upload it to the Internet somewhere.
Import the dump to Heroku:
heroku pgbackups:restore DATABASE http://www.example.com/db.dump

Related

Issue with django DB connectivity on heroku

I am new to django and might be missing something obvious.
I intend to create a simple project and deploy on heroku.
Since sqlite3 is not supported on heroku, I have setup a postgres instance on elephantsql.
I have changed the settings.py file and able to connect to the postgres instance from http://127.0.0.1:8000/.
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "dummy",
"USER": "dummy",
"PASSWORD": "dummy",
"HOST": "dummy",
"PORT": "5432"
}
}
However, I am still not able to connect from heroku.
ProgrammingError at /
relation "home_signup" does not exist
LINE 1: ...e_signup"."email", "home_signup"."timestamp" FROM "home_sign...
^
I have gone through the postgres db which heroku offers, but the free version is limited to 10rows only.So, thought of using elephantsql which has better free offering.
Any pointer will be great help.
Some more details:
I created an instance in elephantsql > ran the migrations locally > checked using pgAdmin4 tool > table is created
The project just takes in text from an input field and adds to a col in the table.It is able to insert records without any issues locally.I am using pgadmin to check if the record is inserted or not.
I am not doing any select or any other db operation.
Use django-heroku.
I used this video to deploy my app: https://www.youtube.com/watch?v=6DI_7Zja8Zc
Finally I found the solution.
Heroku doesn't honor the database details as provided in settings.py
Heroku has a section called settings under its dashboard.
Under the config vars section, we need to add the DB TNS as a key value pair - with key=DATABASE_URL and value=DB TNS.
Screenshot:
Note: If you have a heroku provisioned postgre DB, the DATABASE_URL will point to the heroku postgre and we can't edit the value. We need to remove the postgre DB add-on first.

Data always resets to original state on heroku free account after some time

I have deployed my django app into heroku free account.(first time)
In my working environment I use SQLLite on Heroku I assume POstgres should work.
However I am confused whats going on.
So first of all the app is up and running the data that was in my SQLlite database is currently shows at my deployed app.
Is it because my SQLLite DB was copied to heroku and this is what I actually see?
Another issue that indicates somthing is wron with DB is that if I create superuser via terminal I cant log with it to my app.
Another issue is that it looks data constantly restores to its original state after some time .
I followed the instructions and this is what I have in my settings for DB
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
import dj_database_url
db_from_env = dj_database_url.config()
DATABASES['default'].update(db_from_env)
Because that is how Heroku works. The file system is ephemeral and you must not store anything on it; but sqlite does store is data there, so it will get reset every time a new dyno is started.
Your reference to postgres is confusing. You are not using postgres, but you must.

Django migrations: sqlite3 development db, Amazon Elastic Beanstalk and Amazon RDS postgresql live database

I'm wondering how the community would handle this particular scenario.
I have a Django app that I develop locally using an SQLite3 database as my development database.
The live application is hosted on Amazon Elastic Beanstalk and uses an Amazon RDS PostgreSQL database for production.
To deploy the app, I simply push the Django app to Elastic Beanstalk with eb deploy (which pushes the latest committed version from the local git repository).
settings.py configures the database and checks if the environment is live like so:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
if 'RDS_DB_NAME' in os.environ:
from settings_live import *
and settings_live.py changes the database configuration to the production settings like so:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': CREDENTIALS['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
This all works fine, but issues come up when it comes to migrations. For example: in my development environment I create a new model in an app's models.py. After making the change, I run manage.py makemigrations myapp and manage.py migrate. The migrations are properly applied to my sqlite3 development database. No problems.
Then I commit my changes in preparation for live deployment. My .gitignore file is configured to ignore db.sqlite3 as well as */migrations (since these migrations are only applicable to the development database).
Then I push my latest commit (which doesn't contain my dev database or associated migrations) to Elastic Beanstalk with eb deploy. I have configured an .ebextentions file (.ebextensions/02_commands.config) to run migrations on the production database like so:
03_makemigrations:
command: "django-admin.py makemigrations myapp1 myapp2"
leader_only: true
04_migrate:
command: "django-admin.py migrate"
leader_only: true
Here's the problem: any previous migrations that were generated in the Elastic Beanstalk environment with makemigrations no longer exist in app/migrations since the eb deploy deployment process overwrites the old app with the new one (which only contains a blank migrations directory). This leads to some unexpected behaviour such as tables not being created in the production database.
One solution I've considered (but haven't even begun to implement) is to create a script that copies migration files from an S3 bucket to */migrations and configure 02_commands.config to run this prior to running makemigrations and migrate. Then run another script afterwards that copies the new migrations files back to the S3 bucket. I just wonder if my whole workflow is wrong if it has come to this though.
Your mistake is in saying that the migrations are only applicable to the development database. That's just false. The whole point of migrations is that they are exactly intended to keep your development and production databases in sync. They are part of your code; they should be committed along with all the rest of the code, deployed to production, and run there.

Migrate Django development database (.sql3) to Heroku

How does one migrate their Django .sql3 development database to heroku?
Per here, and here I tried: heroku pg:psql --app sblic < database.sql3 but my Django admin shows no new uploads (even after syncdb/migrate/ or collectstatic
Perhaps there may be a way to directly upload an sql3 file to Heroku, but I went with the path of clearest certainty (convert local sql3 db to postgre db and upload a dump of postgre db to Heroku via pgbackups tool):
Create a dump of your sql3 database as a json file
With PostgreSql installed and with its bin directory in the Path environment variable, create a postgre user and database (or just plan on using your initial super user account created upon installing postgresql)
Update your settings.py with a reference to your newly created postgre database (note, 'HOST' may need to be set as 'localhost', 'user' is your postgre user login)
run python manage.py syncdb to initiate your new postgre db
Optional: if necessary, truncate your postgre db of contenttypes
load your dump from step 1 (if you have fixture issues, see here)
Now you have a working postgre local db. To migrate, first create a postgre dump
Post your postgre dump somewhere accessible by URL (such as drop box or amazon s3 as suggested in previous link).
Perform pgbackups restore command, referencing your dump url
Your heroku app should now reference the contents of your local db.
Heroku command line tool uses the psql binary. You have to install PostgreSQL on your local development machine to have psql available. From the (documentation)[https://devcenter.heroku.com/articles/heroku-postgresql#pg-psql]:
You must have PostgreSQL installed on your system to use heroku pg:psql.
You can in fact keep using your SQLite database with Heroku, but this is not recommended as it will be rewritten with your local copy if you re-deploy it to another dyno. Migrating the data to psql is recommended as described at https://devcenter.heroku.com/articles/sqlite3

force heroku / django to use local database

I have a django based herokuapp site. I set everything up a long time ago and am now unable to get things working with a local instance of postgresql. In my settings file, I updated:
DATABASES['default'] = dj_database_url.config()
to work with a local database:
DATABASES['default'] = dj_database_url.config(default='postgres://localhost/appDB')
When running foreman, I can view the site, but the database is not currently populated (although I did create the empty DB). Running:
heroku run python manage.py dumpdata
Returns the contents of the remote (herokuapp) database, while a syncdb command results in "Installed 0 object(s) from 0 fixture(s)". So it looks like I'm still contacting the remote database. I'm pretty sure the postgresql DB is setup correctly locally; how can I force the app to use it?
I'm sure this is simple, but I haven't seen anything useful yet. I did try
export DATABASE_URL=postgres:///appDB
but that hasn't helped.
Cheers