I succesfully deployed my first Django/Heroku app, and now I only need to transfer my database. It was previously on a MySql db on a Win7 PC. I looked around for ways to import csv into the Heroku db but didn't find anything. They suggest using a ruby gem to do it, or using taps and this command:
heroku db:push mysql://root:mypass#localhost/mydb.
My database is pretty small, only around 1000 columns and 2 tables, so it would be pretty simple to do it import the CSV files, but I cant find how to do it. Anyone knows?
Here's a few ideas that should get you going very quickly:
First, a quick and dirty approach:
Install Postgres on your local machine.
Import the CSV into a local Postgres database.
Push that database to Heroku.
Alternatively, a slightly less quick and still a little dirty approach:
Use the CSV reader Python module.
Create a task in your Django app that loads the CSV, iterates over each CSV row, creates a new corresponding model, and saves it in the server's database if the model is valid.
Run the task via the heroku CLI.
Once you're finished and you've verified the server database, you can remove the task and CSV from the repo.
Otherwise, taps may suit you well too!
As per enter link description here, you can use the copy command to load a CSV into postgres from you local filesystem. You should be able to use this with your Heroku DB with something similar to:
PGPASSWORD=passwordhere psql -h hostname -U username dbname -c "\copy ..."
You can import a local csv file as a table in your heroku postgres by the below command
PGPASSWORD=<your password> psql -h <your heroku host> -U <heroku user> <heroku postgres database name> -c "\copy bank (ifsc, bank_id, branch, address, city, district, state, bank_name) FROM '<local file path location>' CSV HEADER DELIMITER E'\t';"
Please alter the DELIMITER value according to your needs. The 'E' before delimiter value is to denote that the command contains escape characters, otherwise it wil; throw exception
Related
We three working with Django and postgres is the database. Whenever we push the code to GitHub. The database data is not reflecting. The data I stored is visible to me only. The postgres user, password and database name are same on all our laptops. How to make when I push that has to go their databases also.
If you are talking about changes in db schema, then take a look django migrations - https://docs.djangoproject.com/en/3.1/topics/migrations/. The workflow is following:
change model (e.g. add new field, change existing field...)
generate migration file by running python manage.py makemigrations. This generates migration file in <app_folder>/migrations
Run python manage.py migrate to apply changes in models to your database
Add migration file to your version control (github or whaever) and push it
Now when your colleagues gets updated code from version control they need to run python manage.py migrate to apply model changes to their local database.
I found that it is not possible to send postgress data directly. But sqlite can be used for it. This link aids in that - https://manuelvanrijn.nl/blog/2012/01/18/convert-postgresql-to-sqlite/ . But i send data by taking backup like
From Postgres to Postgres: taking backup :
pg_dump dbname > dbname.bak
and then on new Postgres restored with:
psql test < dbname.bak
I have a django app that I put inside of a docker container for deployment. I have some initial data that I want to load into the database via the dumpdata and loaddata commands. The initial data lives on my local hard drive. I choose a very naive approach and simply copied the data_backup.json file to the server via scp.
Now, I want to load the data_backup.json file (the file sits on the server not in the docker container) by executing:
sudo docker-compose exec restapi python manage.py loaddata --settings=rest.settings.production ./data_backup_20191004.json
But Django only searches the internal directories for fixtures.
I am looking for a way to populate the database with the data_backup.json file inside the docker container. Can someone help?
Ultimately, I am looking for a way to dump data directly to S3 and load it from there if needed (for db backups). If you have any tips on how to achieve that, this would also be super helpful - I don't seem to be able to find material on that.
Just in case someone has this question in the future. It is possible to loaddata from stdin. So you can just take the backup file and pipe it into the database (within the container with a command like this:
cat <<fixture_name.json>> | sudo docker exec -i <<container_name_or_id>> python manage.py loaddata --format=json -
The last dash tells django that you want to load the data from stdin.
DOCS
You could copy the file into the docker container before running the command with docker cp:
docker cp ./data_backup_20191004.json <container_id>:django_dir/data_backup_20191004.json
Or, if the file is located on an S3 server, you could execute a curl inside the docker container and install from there:
sudo docker-compose exec restapi curl http://s3.example.com/path/to/data.json > data.json
sudo docker-compose exec restapi python manage.py loaddata --settings=rest.settings.production ./data.json
I am looking for a way to populate the database with the
data_backup.json file inside the docker container. Can someone help?
See the answer of Xen_mar, which I think it's perfect.
Ultimately, I am looking for a way to dump data directly to S3 and
load it from there if needed (for db backups). If you have any tips on
how to achieve that, this would also be super helpful - I don't seem
to be able to find material on that.
This seems to be a complete different question. I would consider using some django package like Django Smuggler, which allows you to load and dump fixtures from the admin, and I assume that it may be possible to configure django smuggler upload directory to be handled by django storages. I'm not sure that is possible, so if you try let me know.
How does one migrate their Django .sql3 development database to heroku?
Per here, and here I tried: heroku pg:psql --app sblic < database.sql3 but my Django admin shows no new uploads (even after syncdb/migrate/ or collectstatic
Perhaps there may be a way to directly upload an sql3 file to Heroku, but I went with the path of clearest certainty (convert local sql3 db to postgre db and upload a dump of postgre db to Heroku via pgbackups tool):
Create a dump of your sql3 database as a json file
With PostgreSql installed and with its bin directory in the Path environment variable, create a postgre user and database (or just plan on using your initial super user account created upon installing postgresql)
Update your settings.py with a reference to your newly created postgre database (note, 'HOST' may need to be set as 'localhost', 'user' is your postgre user login)
run python manage.py syncdb to initiate your new postgre db
Optional: if necessary, truncate your postgre db of contenttypes
load your dump from step 1 (if you have fixture issues, see here)
Now you have a working postgre local db. To migrate, first create a postgre dump
Post your postgre dump somewhere accessible by URL (such as drop box or amazon s3 as suggested in previous link).
Perform pgbackups restore command, referencing your dump url
Your heroku app should now reference the contents of your local db.
Heroku command line tool uses the psql binary. You have to install PostgreSQL on your local development machine to have psql available. From the (documentation)[https://devcenter.heroku.com/articles/heroku-postgresql#pg-psql]:
You must have PostgreSQL installed on your system to use heroku pg:psql.
You can in fact keep using your SQLite database with Heroku, but this is not recommended as it will be rewritten with your local copy if you re-deploy it to another dyno. Migrating the data to psql is recommended as described at https://devcenter.heroku.com/articles/sqlite3
I am creating a Django app on Heroku. While I am testing it locally, finally I am checking by uploading everything to heroku and seeing if it works for real on the remote server as well
To quickstart, I went to the admin panel and created a bunch of data to run the view functions on. When I am pushing the source code to heroku via git (git push heroku master), I also want to push up the database from local to heroku, so that I dont need to enter it again on the server side
How do I achieve this?
Thanks a lot
Grab your the postgres string from your Heroku database using heroku config:get
Look for the Heroku Postgres url (example: HEROKU_POSTGRESQL_RED_URL: postgres://user3123:passkja83kd8#ec2-117-21-174-214.compute-1.amazonaws.com:6212/db982398).
Next, run this on your command line:
pg_dump --host=<host_name> --port=<port> --username=<username> --password --dbname=<dbname> > output.sql
The terminal will ask for your password then run it and dump it into output.sql.
Then import it:
psql -d <heroku postgres string> -f output.sql
Note: in each place where you see <.....> in the above code, be sure to substitute your specific information. Thus you would put your username where it says <username>, and your heroku database url where it says <heroku postgres string>.
Similar answer here.
I know there are a lot of questions floating around there relating to similar issues, but I think I have a specific flavor which hasn't been addressed yet. I'm attempting to create my local postgresql database so that I can do local development in addition to pushing to Heroku.
I have found basic answers on how to do this, for example (which I think is a wee bit outdated):
'#DATABASES = {'default': dj_database_url.config(default='postgres://fooname:barpass#localhost/dbname')}'
This solves the "ENGINE" is not configured error. However, when I run 'python manage.py syncdb' I get the following error:
'OperationalError: FATAL: password authentication failed for user "foo"
FATAL: password authentication failed for user "foo"'
This happens for all conceivable combinations of username/pass. So my ubuntu username/pass, my heroku username/pass, etc. Also this happens if I just try to take out the Heroku component and build it locally as if I was using postgresql while following the tutorial. Since I don't have a database yet, what the heck do those username/pass values refer to? Is the problem exactly that, that I need to create a database first? If so how?
As a side note I know I could get the db from heroku using the process outlined here: Should I have my Postgres directory right next to my project folder? If so, how?
But assuming I were to do so, where would the new db live, how would django know how to access it, and would I have the same user/pass problems?
Thanks a bunch.
Assuming you have postgres installed, connect via pgadmin or psql and create a new user. Then create a new database and with your new user as the owner. Make sure you can connect via psql with the new user into to the database. you will then need to set up an env variable in your postactivate file in your virtualenv's bin folder and save it. Here is what I have for the database:
export DATABASE_URL='postgres://{{username}}:{{password}}#localhost:5432/{{database}}'
Just a note: adding this value to your postactivate doesn't do anything. The file is not run upon saving. You will either need to run this at the $ prompt, or simply deactivate and active your virtualenv.
Your settings.py should read from this env var:
DATABASES = {'default': dj_database_url.config()}
You will then configure Heroku with their CLI tool to use your production database when deployed. Something like:
heroku config:set DATABASE_URL={{production value here}}
(if you don't have Heroku's CLI tool installed, you need to do it)
If you need to figure how exactly what that value you need for your production database, you can get it by logging into heroku's postgresql subdomain (at the time this is being written, it's https://postgres.heroku.com/) and selecting the db from the list and looking at the "Connection Settings : URL" value.
This way your same settings.py value will work for both local and production and you keep your usernames/passwords out of version control. They are just env config values.