I am developing a Django application which will use multiple database backends. I would like to put an sqlite database on the machine running the django application, and sync to a remote mysql database. The tricky part is that this machine running the application will not always have an internet connection, so the mysql database is not always availalble. There will be multiple machines running the application, each with it's own local sqlite DB, but all using the same remote mysql DB.
I haven't written the code yet, but here is what I have in mind. Every time I run an insert or update I would like to write it to both databases, unless the remote DB is unavailable, in which case I will save the sql statement in a table on the local database to run when the remote DB is available.
Can this be done with database routers, or do I need to manually implement this with each db statement?
Note on PK: Not directly related, but sure to be asked. The primary key will be generated locally on each machine. In the mysql DB there will be a field for this key, and a field with a unique identifier for each instance of the application, which together will provide a unique key.
Suppose you have a model called Blog then you can use the following to store it locally and remotely (assuming that you have configured access to the remote db).
blog = Blog('test')
blog.save() #Assuming that sqlite is the default db
try:
blog.save(using='mysql')
except NoInternetConnection:
pass
Make sure you have defined and configured 'mysql' in settings.py and that you handle the cases when there is no Internet connection.
Side note: I am not quite sure why you actually would want to do this. If this is for backup purposes then I would use standard backup procedures. For more information about using multiple databases see: http://docs.djangoproject.com/en/dev/topics/db/multi-db/#using-raw-cursors-with-multiple-databases
I took DrDee's code and attached it to the post_save signal (+1 for the help).
#receiver(models.signals.post_save) #new signal decorator in Django 1.3
def save_to_remote(sender,instance,using,**kwargs):
if using == 'default' and instance.__module__ == 'mymodel.myapp.models':
try:
instance.save(using='remote')
except:
pending_instance=Pending_Remote(
pk_default=instance.pk,
model_default=instance.__class__.__name__
)
pending_instance.save()
This also saves a record of what was not saved to the remote database. Note that the model Pending_Remote must not be in 'myapp'.
Related
I have created a Django-based webpage where different vendor company employees can logins and can change their Shift Timing. (Now we are controlling this job with Linux script but due to large user size ~8k doing it for all requests is a difficult task).
To resolve this I have created a Django webpage( 6 separate models/DB) and used default SQLite DB.
Requirement:
The user is working on some application which needs to be controlled by updated shift timing on the portal.
Question:
How to consolidate OR store DB data in a centralized place? so that if tomorrow I have to reset the Timing for all the users in the portal to default consider General Shift.
I have the below Idea to do this but not sure if this is the best way to complete this work.
by using the REST API I will get the JSON data.OR
manage.py dumpdata apple.CompanyName(Model) --indent 5
any help/Suggestion on this would be appreciated.
For the database u could use an Hosted db like heroku postgres database, If ur new
to database else u can run ur own postgres database in the server.AS u mention there 8k its not good to use SQLite DB as it is file system based.To update the shift timing u can use the default Django admin. I am not sure about ur model structure but as long as you have necessary validation in logic it can be updated from admin anytime
i'm looking for a "best-practice" guide/solution to the following situation.
I have a Django project with a MySql DB which i created and manage. I have to import data, every 5 minutes, from a second (external, not managed by me) db in order to do some actions. I have read rights for the external db and all the necessary information.
I have read the django docs regarding the usage of multiple database: register the db in settings.py, migrate using the --database flag, query/access data by routing to the db (short version) and multiple question on this matter on stackoverflow.
So my plan is:
Register the second database in settings.py, use inspectdb to add to the model, migrate, define a method which reads data from the external db and add it to the internal (own) db.
However I do have some questions:
Do i have to register the external db if i don't manage it?
(Most probably yes in order to use ORM or the cursors to access the data)
How can i migrate the model if I don't manage the DB and don't have write permissions? I also don't need all the tables (around 250, but only 5 needed).
(is fake migration an option worth considering? I would use inspectdb and migrate only the necessary tables.)
Because I only need to retrieve data from the external db and not to write back, would it suffice to have a method that constantly gets the latest data like the second solution suggested in this answer
Any thoughts/ideas/suggestions are welcomed!
I would not use Django's ORM for it, but rather just access the DB with psycopg2 and SQL, get the columns you care about into dicts, and work with those. Otherwise any minor change to that external DB's tables may break your Django app, because the models don't match anymore. That could create more headaches than an ORM is worth.
I am connecting to a legacy mysql database in cloud using my django project.
i need to fetch data and insert data if required in DB table.
do i need to write model for the tables which are already present in the db?
if so there are 90 plus tables. so what happens if i create model for single table?
how do i talk to database other than creating models and migrating? is there any better way in django? or model is the better way?
when i create model what happens in the backend? does it create those tables again in the same database?
There are several ways to connect to a legacy database; the two I use are either by creating a model for the data you need from the legacy database, or using raw SQL.
For example, if I'm going to be connecting to the legacy database for the foreseeable future, and performing both reads and writes, I'll create a model containing only the fields from the foreign table I need as a secondary database. That method is well documented and a bit more time consuming.
However, if I'm only reading data from a legacy database which will be retired, I'll create a read-only user on the legacy database, and use raw SQL to grab what I need like so:
from django.db import connections
cursor = connections["my_secondary_db"].cursor()
cursor.execute("SELECT * FROM my_table")
for row in cursor.fetchall():
insert_data_into_my_new_system_model(row)
I'm doing this right now with a legacy SQL Server database from our old website as we migrate our user and product data to Django/PostgreSQL. This has served me well over the years and saved a lot of time. I've used this to create sync routines all within a single app as Django management commands, and then when the legacy database is done being migrated to Django, I've completely deleted the single app containing all of the sync routines, for a clean break. Good luck!
I was wondering this could produce any problem if I directly add rows or remove some from a model table. I thought maybe Django records the number of rows in all tables? or this could mess up the auto-generated id's?
I don't think it matters but I'm using MySql.
No, it's not a problem because Django does the same that you do "directly" to the database, it execute SQL statements, and the auto generated id is handled by the database server (MySql server in this case), no matter where that SQL queries comes from, whatever it is Mysql Client or Django.
Since you can have Django work on a pre-existing database (one that wasn't created by Django), I don't think you will have problems if you access/write the tables of your own app (you might want to avoid modifying Django's internal tables like auth, permission, content_type etc until you are familiar with them)
When you create a model through Django, Django doesn't store the count or anything (unless your app does), so it's okay if you create the model with Django on the SQL database, and then have another app write/read from that same SQL table
If you use Django signals, those will not be triggered by modifying the SQL table directly through the DB, so you might want to pay attention to side effects like that.
Your RDBMS handles it's own auto generated IDs and referential integrity, counts etc, so you don't have to worry about messing it up.
I've recently changed the database server on my project from sqlite3 to Postgresql and I have a few questions that I hope will give an answer to my issues.
I understand that switching from sqlite to Postgres implies that I create the new database and the tables inside it, right? I've done that but I haven't seen any new files created in my project to show me that the database I've made is visible. (Btw, I've changed the database name in settings.py)
I probably should mention that I'm working in a virtual environment and I would like to know if that affects my references in any way. I've tried to import the tables in Django to try and count the number of records in a table but I get the error: "No module named psdemo". (psdemo is my database name and i'm trying to import the table with:
from ps.psdemo import Product
where ps is my application, psdemo is my database and Product the table in the database.
In conclusion I'm trying to get access to my database and tables but I can't manage to find them. I repeat, there is no new database file in my project or in my virtual environment (I've searched thoroughly) but if I use a terminal connection I can connect to my virtual environment and change directories to get to the application folder then if I connect to the Postgresql server I can create the database, the tables and can Insert into them, make queries etc, but I cannot access them from the Django code.
I understand that switching from sqlite to Postgres implies that I create the new database and the tables inside it, right? I've done that but I haven't seen any new files created in my project to show me that the database I've made is visible. (Btw, I've changed the database name in settings.py)
All you have to do with postgres is create the database. Not the tables. Django will create the tables, and anything else it thinks are useful, once you call syncdb.
You won't have any new files in your project like you did in sqlite. If you want to view your database, you should download and install pgadminIII (which I would recommend in any event)
I probably should mention that I'm working in a virtual environment and I would like to know if that affects my references in any way. I've tried to import the tables in Django to try and count the number of records in a table but I get the error: "No module named psdemo". (psdemo is my database name and i'm trying to import the table with:
Here, you import models via normal python syntax and it then references your tables. Each model should represent a single table. You define your models first, and then call
python manage.py syncdb
In conclusion I'm trying to get access to my database and tables but I can't manage to find them.
See above, but you should definitely read about postgres installation from the postgres docs, and read the psycopg2 docs as well as the Django docs for setting up a postgres database.
I understand that switching from sqlite to Postgres implies that I
create the new database and the tables inside it, right? I've done
that but I haven't seen any new files created in my project to show me
that the database I've made is visible. (Btw, I've changed the
database name in settings.py)
Database files are not created in the project directory with postgresql. They are created in the database server data directory (like /var/lib/postgres it depends on the distribution). You should generally query it through a PostgreSQL client that connects to the PostgreSQL server rather than messing with the files directly.
You can for example run command:
manage.py dbshell
As to your first issue, see #jpic's answer.
On your second issue, your database is not a package, and you do not import models from your database. If you were able to import your models correctly before you made any changes, change your import statements back to how they were.