I have a multi-database Django Project (Python 2.7, Django 1.4 ) and I need that the database used in every single transaction triggered in a request be selected through the url from that request.
Ex:
url='db1.myproject.com'
Would trigger the use of 'db1' in every transaction, like this:
MyModel.objects.using('db1').all()
That is where my problem resides.
Its virtualy impossible include in every call of 'Model.objects' the '.using(dbname)' function (this project already have 2 dozen of models, and its build with scaling in mind, so it could have hundreds of models, and almost everymodel has (or might have) a implementation a of post_save/post_delete that acts as database triggers).
Is there a way of telling django that he should change the default database for this or that request ?
I already have a Custom QuerySet and a Custom Manager, so, if i could tell THEM the database should be this or that could do the trick to (i dont find anywhere how to get request data inside them).
Any help would be apreciated.
I think my answer to a similar question here will do what you need:
https://stackoverflow.com/a/21382032/202168
...although I'm a bit worried you may have chosen an unconventional or 'wrong' way of doing database load balancing, maybe this article can help with some better ideas:
http://engineering.hackerearth.com/2013/10/07/scaling-database-with-django-and-haproxy/
Related
I have lots of models in my project like Advertisements, UserDetails etc. Currently I have to delete the entire database every year so as to not create any conflicts between this year data and previous year data.
I want to implement a feature that can allow me to switch between different years. What can be the best way to implement this?
I think you could switch schemas in PostgreSQL. It's not completely straightforward. There are several ways to do that you can look into. The way I did it was to use a default search path for the Django database user account (e.g. user2018, user2019, etc) that only included the schema I wanted to use. I can't check the exact settings right now because my office network is down. You can also do it in settings.py or in each individual model using db_table according to what I've read, although both those solutions seem more convoluted that using the search path.
You would have to shutdown, change the database username in settings.py (or change the search path in PostgreSQL, change the schema over to a new one, and then run migrate to create the tables again. If you have reference data in any of the tables then schema-to-schema copies are easy to do.
Try searching for change django database schema postgresql to see what options there are for specifying the schema.
I'm a beginner at Django, and as a practice project I would like to create a webpage with a dashboard to track investments in a particular p2p platform. They do not have a nice dashboard (but provide excel file with all data). As I see it, main steps that I need to do in this project are as follow:
Create login so that users would have account where they upload their excel files.
Make it possible to import excel file to a database
Manipulate/calculate data for it to be later used in dashboard
Create dashboard.
Host webpage.
After some struggle I have implemented point no. 2, and will deal with 1 and 5 later. But number 3 is my biggest issue now.
I'm completely unsure what I need to do, and google did not help. I need to calculate data before I can make dashboard from it. Union two of the tables, and then join them together with a third table, creating some additional needed calculated fields. Do I create a view in the database and somehow fetch this data to Django? Or do I need to create some rules so that new table would be created at the time of the import? I think having table instead of a view would have better performance. Or maybe I'm doing it completely wrong, and should take completely different approach for this kind of task? Also, is SQLite a good database for a task (I'm using it, because it was a default in Django)?
I assume for vizualization part I will need to do it with some JavaScript library, such as D3? Which then would use data from step 3.
For part 3 there is 2 way, either do these stuff and save the result in your database or you can do it when you need it using django model features like annotation, aggregation and etc.
Option 1 requires to add a table for you calculation which is Models in django.
Option 2 requires to create a doing the annotations in a view or model managers and then using them in views.
Django docs: Aggregation
Which is the best is depended on how big your data is, how complicated the calculation is and how often you need them.
And for database; SQLite is just a database for development use not the production and surly not with a lot of data and a lot of calculations. The recommended database for django is postgresql which is pretty good at handling millions and even billions of data and doing heavy calculation.
And for vizualization you should handle it on the template side which is basically HTML, CSS and JS.
I'm thinking of creating a log system for my django web application. The web application is quite comprehensive in its use (covers all aspects of a business's processes) so I'd like to track every event that happens. Specifically, I'd like to log every view that runs and not just the "main" ones and, potentially, log what is happening within the view as its executed.
While I'm in the "idea" stage of the logging system, I've quickly hit a few questions that leave me unsure how to proceed. Here are the main questions I have:
I'm thinking of logging all of the events in the same MySQL database that the main web app holds its data. The concern I have is bloating the MySQL database into a massive DB. Also, if the DB crashes or is destroyed somehow (yes I have backups) I'll loose my log too which blows away any ability to track down the problem. Do I use a seperate DB or just go with text files?
How granular do I go? Initially I was thinking of simply logging things like, "Date - In view myView". However, as I'm thinking about it, it would be nice to log all the stuff that happens within the view. Doing this could make the log massive! and would also make my code ugly with so many log entry lines mixed into the code. This kind of detail:
Date - entered view myView
Date - in view myView, retrieved object myObject from the DB
Date - in view myView, setting myObject field myField to myNewValue
Date - leaving myView
Those are my main thoughts at this point. Any advice on this front?
Thanks
I think the best and right way is to create your own custom middleware where you can log actually everything you need.
Here are some links on the subject:
middleware snippets
http://djangosnippets.org/snippets/2624/
http://djangosnippets.org/snippets/290/
http://djangosnippets.org/snippets/264/
django-logging-middleware (pretty old by may give you an idea)
django-request
django.db.backends logging
Is there a Django middleware/plugin that logs all my requests in a organized fashion?
Django verbose request logging
log all sql queries
django orm, how to view (or log) the executed query?
Also, consider using sentry error logging and aggregation platform instead of writing logs into the database. FYI, see using a database for logging.
If you want to log any action run in every view, you can for example, replace entered view A and exited view A by a line in these words: view A - 147ms.
As alecxe stated, you can log requests/SQL, there are plenty of ways to do it with middleware. About database (object) actions, you can tie individual saves, updates and deletes with signals.
For bulk updates and deletes, you could (it's not a clean way but it would work) monkey-patch manager and queryset methods to add logging.
This way you can log actions rather than SQL.
I would see lines like this:
[2013/09/11 15:11:12.0153] view app.module.view 200 148ms
[2013/09/11 15:11:12.0189] orm save:auth.User,id=1 3ms
This is a quick and dirty proposal, but, maybe it's worth it.
If you want user to submit say articles to your website but you want to approve these articles before they are added to a list of articles, how would you go about doing so?
The only method I can think of is to have 2 databases; one for 'awaiting approval' and another for 'approved and ready to be displayed'. However, my issue is whether there is a fast method to go about transferring the information between the two databases? I only know about doing this manually.
Or is there already a Django module that handles this?
Thank You for any help.
As jordi has said, add an extra field, but you might also want to write a custom manager for your table that selects posts according to status.
See the Django docs: https://docs.djangoproject.com/en/dev/topics/db/managers/ where they set up a manager that filters on sex="M" to only return Male people. This makes your code neater, and means as long as you go through the right manager you'll not have to keep remembering to test for state="Approved" all the time.
If you want to get more complex, the thing you are doing is called "Workflow" and there are django and python packages that implement this - but it gets very complex very quickly...
Just add a column to the database table called status
1 = waiting
2 = approved
3 = denied
I have an authentication backend based off a legacy database. When someone logs in using that database and there isn't a corresponding User record, I create one. What I'm wondering is if there is some way to alert the Django system to this fact, so that for example I can redirect the brand-new user to a different page.
The only thing I can think of is adding a flag to the users' profile record called something like is_new which is tested once and then set to False as soon as they're redirected.
Basically, I'm wondering if someone else has figured this out so I don't have to reinvent the wheel.
I found the easiest way to accomplish this is to do exactly as you've said. I had a similar requirement on one of my projects. We needed to show a "Don't forget to update your profile" message to any new member until they had visit their profile for the first time. The client wanted it quickly so we added a 'visited_profile' field to the User's profile and defaulted that to False.
We settled on this because it was super fast to implement, didn't require tinkering with the registration process, worked with existing users, and didn't require extra queries every page load (since the user and user profile is retrieved on every page already). Took us all of 10 minutes to add the field, run the South migration and put an if tag into the template.
There's two methods that I know of to determine if an object has been created:
1) When using get_or_create a tuple is returned of the form (obj, created) where created is a boolean indicating obviously enough whether the object was created or not
2) The post_save signal passes a created paramater, also a boolean, also indicating whether the object was created or not.
At the simplest level, you can use either of these two hooks to set a session var, that you can then check and redirect accordingly.
If you can get by with it, you could also directly redirect either after calling get_or_create or in the post_save signal.
You can use a file-based cache to store the users that aren't yet saved to the database. When the user logs in for the second time, you can look in the cache, find the user object, and save it to the database for good.
Here's some info on django caching: http://docs.djangoproject.com/en/dev/topics/cache/?from=olddocs
PS: don't use Memcached because it will delete all information in the situation of a computer crash or shut down.