I'm currently developing a screener webapp using heroku. The data is saved in csv (because its is a screener, it takes a lot of time to run if using api). Right now, I updated manually by updating and saving the data in my local then push it to heroku.
Is there a way to cron so that it updates the csv data once a day?
Heroku dynos have an ephemeral filesystem. Any changes you make will be lost when the dynos restart. This happens frequently (at least once per day).
For this reason, file-based data stores are not recommended on Heroku. If you move your data into a client-server database (Heroku's own Postgres service would be a good fit or you can pick something else), you could easily updated it whenever you like.
The Heroku Scheduler can be used to run tasks daily. Write a script to update your database table and schedule it to run whenever you like.
Related
So, I often am working with projects where there will be multiple dockerized django application servers behind a loadbalancer, and frequently I will need to deploy to them. I frequently use Watchtower to do pull-based deploys. I build a new image, push it to dockerhub, and watchtower is responsible for pulling those images down to the servers and replacing the running containers. That all works pretty well.
I would like to start automating the running of django migrations. One way I could accomplish this would be to simply add a run of the manage.py migrate to the entrypoint, and have every container automatically attempt a migration when the container comes online. This would work, and it would avoid the hassle of needing to come up with a way to do a lockout or leader election; but without some sort of way to prevent multiple runs, there is a risk that multiple instances of the migration could run at the same time. If I went this route, is there any chance that multiple migrations running at the same time could cause problems? Should I be looking for some other way to kick off these migrations once and only once?
Running migrations at the same time in parallel is not safe. I've tested it by running the migrate command in parallel on a data migration, and Django will run the migration twice. It will add 2 rows to the django_migrations table.
Check out this Google Groups post discussing the issue
And this article about running migrations on container statup.
I have deployed a Django site on GoDaddy Django "droplet" for a while and users have been using the site to keep their records.
Now that GoDaddy is discontinuing the service, I would like to migrate the entire site with all records intact to DigitalOcean.
How does one go about doing this?
Here's what I'd do:
dump my data into a file see dumpdata
stop the server
remove all .pyc files
copy paste the whole folder of the website to the destination server
restore data using loaddata
run the server
I did this 4 times without any problems (dev to test environment, test to pre-prod and pre-prod to prod).
The reply by Oliver is good for small sites, but big companies (or any) won't be happy if you stop the server, also some records/sales might be done between the time you dump and the time you stop the server
I think this would be better but I'm unsure:
make a new database and make it sync with the old one, whenever old db changes changes should be synced to the new db (I know for example postgres has a feature that triggers some kind of alerts on creations/updates, this is indeed the hardest step and needs quite a lot of research to pull it off, both databases must be on sync)
upload the new(copy) page next to the new database connected to it
modify the DNS records to point to the new server, traffic will organically move to the new server, at some point you might have people making database updates on both servers so it is important that the sync goes both ways
take the first web-page's server down, cut the database syncing and remove old
page and old database
remove the pipe/method that was letting you sync the
new db with the old one
I create a new deployment for every new customer on Heroku platform.
However as my app gets deployed to Heroku some default data should be uploaded. Each time I update - existing data should remain.
What is my best option? Thanks!
My Django application (a PoC, not a final product) with a backend library uses a SQLite database - read only. The SQLite database is part of the repo and deployed to Heroku. This is working fine.
I have the requirement to allow updates to this database via the Django admin interface. This is not a Django managed database, so from Django's point of view just a binary file.
I could allow for a FileField to handle this, overwriting the database; I guess this would work in a self-managed server, but I am on Heroku and have the constraints imposed by Disk Backed Storage. My SQLite is not my webapp database, but limitations apply the same: I can not write to the webapp's filesystem and get any guarantee the new data will be visible by the running webapp.
I can think of alternatives, all with drawbacks:
Put the SQLite database in another server (a "media" server), and access it remotely: this will severely impact performance. Besides, accessing SQLite databases over the network does not seem easy.
Create a deploy script for the customer to upload the database via the usual deploy mechanisms. Since the customer is not technically fit, and I can not provide direct support, this is unfeasible.
Move out of Heroku to a self-managed server, so I can implement this quick-and-dirty upload without complications.
Do you have another suggestion?
PythonAnywhere.com
deploy your app and you can easily access all of your files and update them and your Sqlite3 database is going to be updated in real time without losing data.
herokuapp.com erase your Sqlite3 database every 24 hours that's why it's not preferred for Sqlite3 having web apps
I am building a small financial web app with django. The app requires that the database has a complete history of prices, regardless of whether someone is currently using the app. These prices are freely available online.
The way I am currently handling this is by running simultaneously a separate python script (outside of django) which downloads the price data and records it in the django database using the sqlite3 module.
My plan for deployment is to run the app on an AWS EC2 instance, change the permissions of the folder where the db file resides, and separately run the download script.
Is this a good way to deploy this sort of app? What are the downsides?
Is there a better way to handle the asynchronous downloads and the deployment? (PythonAnywhere?)
You can write the daemon code and follow this approach to push data to DB as soon as you get it from Internet. Since your daemon would be running independently from the Django, you'd need to take care of data synchronisation related issues as well. One possible solution could be to use DateTimeField in your Django model with auto_now_add = True, which will give you idea of time when data was entered in DB. Hope this helps you or someone else looking for similar answer.