I recently deployed a couple of web applications built using django (on webfaction).
These would be some of the first projects of this scale that i am working on, so I wanted to know what an effective backup strategy was for maintaining backups both on webfaction and an alternate location.
EDIT:
What i want to backup?
Database and user uploaded media. (my code is managed via git)
I'm not sure there is a one size fits all answer especially since you haven't said what you intend to backup. My usual MO:
Source code: use source control such as svn or git. This means that you will usually have: dev, deploy and repository backups for code (specially in a drsc).
Database: this also depends on usage, but usually:
Have a dump_database.py management command that will introspect settings and for each db will output the correct db dump command (taking into consideration the db type and also the database name).
Have a cron job on another server that connects through ssh to the application server, executes the dump db management command, tars the sql file with the db name + timestamp as the file name and uploads it to another server (amazon's s3 in my case).
Media file: e.g. user uploads. Keep a cron job on another server that can ssh into the application server and calls rsync to another server.
The thing to keep in mind though, it what is the intended purpose of the backup.
If it's accidental (be it disk failure, bug or sql injection) data loss or simply restoring, you can keep those cron jobs on the same server.
If you also want to be safe in case the server is compromised, you cannot keep the remote backup credentials (sshkeys, amazon secret etc) on the application server! Or else an attacker will gain access to the backup server.
Related
Since there is a nightly backup of SQL we are wondering of a good way to restore this backup to a different database in the same MySQL server instance. We have prod_xxxx for all our production databases AND we have staging_xxxx for all our staging databases (yes not that good in that they are all on the same mysql instance right now).
Anyways, we would love to restore all tables/constraints/etc and data from prod_incomingdb to staging_incomingdb. Is this possible in cloud SQL?
Since this is over a productive instance I recommend you to perform a backup before start, in order to avoid any data corruption.
To clone a database within the same instance, there is not a direct way to perform the task (this is a missing feature on MySQL).
I followed this path in order to successfully clone a database within same MySQL Cloud SQL instance.
1.- Create a dump of the desired database using the Google Cloud Console (Web UI) by follow these steps
*it is very important to only dump the desired database in format SQL, please not select multiple databases on the dump.
After finish the process, the dump will be available in a Google Cloud Storage Bucket.
2.- Download the dump file to a Compute Engine VM or to any local machine with linux.
3.- please replace the database name (the old one) in the USE clauses.
I used this sed command over my downloaded dump to change the names of the databases
sed -i 's/USE `employees`;/USE `emp2`;/g' employees.sql
*this can take some seconds depending the size of your file.
4.- Upload the updated file to the Cloud storage bucket.
5.- Create a new empty database on your Cloud SQL instance, in this case my target instance is called emp2.
6.- Import the modified dump by following these steps
I could not figure out the nightly backups as it seems to restore an entire instance. I think the answer to the above is no. I did find out that I can export and then import (not exactly what I wanted though as I didn't want to be exporting our DB during the day but for now, we may go with that and automate a nightly export later).
I have this small project that specifies sqlite as the database choice.
For this particular project, the framework is Django, and the server is hosted by Heroku. In order for the database to work, it must be set up with migration commands and credentials whenever the project is deployed to continuous integration tools or development site.
The question is, that many of these environments do not actually use the my_project.sqlite3 file that comes with the source repository, which we version control with git. How do I incorporate changes to the deployed database? Is a script that set up the database suitable for this scenario? Meanwhile, it is worth notice that there are security credentials that should not appear in a script in unencrypted ways, which makes the situation tricky.
that many of these environments do not actually use the my_project.sqlite3 file that comes with the source repository
If your deployment platform does not support your chosen database, then your development environment should probably be moved to using one of the databases they do support. It is possible to run different databases in development and production, but just seems like the source of headaches.
I have found a number of articles that state that Heroku just doesn't support SQLite in production and instead recommends Postgres.
How do I incorporate changes to the deployed database? Is a script that set up the database suitable for this scenario?
I assume that you are just extracting data from one database to give to another, so yes,as long as that script is a one time batch operation each time the code is updated, then it should be fine. You will want something else if you are adding/manipulating data in production and then exporting it to your git.
Meanwhile, it is worth notice that there are security credentials that should not appear in a script in unencrypted ways
An environment variable should solve that. You set your host machine to have environment variables with your credentials and then just retrieve them within the script. You are looking to have something like this:
# Set environment vars
os.environ['USER'] = 'username'
os.environ['PASSWORD'] = 'password'
# Get environment vars
USER = os.getenv('USER')
PASSWORD = os.environ.get('PASSWORD')
I have deployed a Django site on GoDaddy Django "droplet" for a while and users have been using the site to keep their records.
Now that GoDaddy is discontinuing the service, I would like to migrate the entire site with all records intact to DigitalOcean.
How does one go about doing this?
Here's what I'd do:
dump my data into a file see dumpdata
stop the server
remove all .pyc files
copy paste the whole folder of the website to the destination server
restore data using loaddata
run the server
I did this 4 times without any problems (dev to test environment, test to pre-prod and pre-prod to prod).
The reply by Oliver is good for small sites, but big companies (or any) won't be happy if you stop the server, also some records/sales might be done between the time you dump and the time you stop the server
I think this would be better but I'm unsure:
make a new database and make it sync with the old one, whenever old db changes changes should be synced to the new db (I know for example postgres has a feature that triggers some kind of alerts on creations/updates, this is indeed the hardest step and needs quite a lot of research to pull it off, both databases must be on sync)
upload the new(copy) page next to the new database connected to it
modify the DNS records to point to the new server, traffic will organically move to the new server, at some point you might have people making database updates on both servers so it is important that the sync goes both ways
take the first web-page's server down, cut the database syncing and remove old
page and old database
remove the pipe/method that was letting you sync the
new db with the old one
I have deployed my app (PHP Buildpack) to production with cf push app-name. After that I worked on further features and bugfixes. Now I would to push my local changes to production. But when I do that all the images (e.g. profile image) which are being saved on the production server get lost with every push.
How do I take over only the changes in the code without losing any stored files on the production server?
It should be like a "git pull"
Your application container should be stateless. To persist data, you should use the offered services. The Swisscom Application Cloud offers an S3 compatible Dynamic Storage (e.g. for pictures or user avatars) or different database services (MongoDB, MariaDB and others). If you need to save user data, you should save it in one of these services instead of the local filesystem of the app's container. If you keep your app stateless, you can migrate and scale it more easily. You can find more information about how your app should be structured to run in a modern cloud environment here. To get more information about how to use your app with a service, please check this link.
Quote from Avoid Writing to the Local File System
Applications running on Cloud Foundry should not write files to the
local file system for the following reasons:
Local file system storage is short-lived. When an application instance
crashes or stops, the resources assigned to that instance are
reclaimed by the platform including any local disk changes made since
the app started. When the instance is restarted, the application will
start with a new disk image. Although your application can write local
files while it is running, the files will disappear after the
application restarts.
Instances of the same application do not share a
local file system. Each application instance runs in its own isolated
container. Thus a file written by one instance is not visible to other
instances of the same application. If the files are temporary, this
should not be a problem. However, if your application needs the data
in the files to persist across application restarts, or the data needs
to be shared across all running instances of the application, the
local file system should not be used. We recommend using a shared data
service like a database or blobstore for this purpose.
In future your problem will be "solved" with Volume Services (Experimental). You will have a persistent disk for your app.
Cloud Foundry application developers may want their applications to
mount one or more volumes in order to write to a reliable,
non-ephemeral file system. By integrating with service brokers and the
Cloud Foundry runtime, providers can offer these services to
developers through an automated, self-service, and on-demand user
experience.
Please subscribe to our newsletter for feature announcements. Please also monitor the CF community for upstream development.
My Django application (a PoC, not a final product) with a backend library uses a SQLite database - read only. The SQLite database is part of the repo and deployed to Heroku. This is working fine.
I have the requirement to allow updates to this database via the Django admin interface. This is not a Django managed database, so from Django's point of view just a binary file.
I could allow for a FileField to handle this, overwriting the database; I guess this would work in a self-managed server, but I am on Heroku and have the constraints imposed by Disk Backed Storage. My SQLite is not my webapp database, but limitations apply the same: I can not write to the webapp's filesystem and get any guarantee the new data will be visible by the running webapp.
I can think of alternatives, all with drawbacks:
Put the SQLite database in another server (a "media" server), and access it remotely: this will severely impact performance. Besides, accessing SQLite databases over the network does not seem easy.
Create a deploy script for the customer to upload the database via the usual deploy mechanisms. Since the customer is not technically fit, and I can not provide direct support, this is unfeasible.
Move out of Heroku to a self-managed server, so I can implement this quick-and-dirty upload without complications.
Do you have another suggestion?
PythonAnywhere.com
deploy your app and you can easily access all of your files and update them and your Sqlite3 database is going to be updated in real time without losing data.
herokuapp.com erase your Sqlite3 database every 24 hours that's why it's not preferred for Sqlite3 having web apps