I have a few models that look like this:
class System(models.Model):
'''Defines a system'''
system_desc = models.CharField('System Name', max_length=50)
class SystemGroup(models.Model):
'''Groups within a 'System' (ie. Customers, regions, divisions, etc. '''
group_desc = models.CharField('Group Name',max_length=25)
system = models.ForeignKey(System)
class QueryData(models.Model):
'''Queries related to a 'System' (ie. Active users against System1, Orders today in system2, etc. '''
qry_desc = models.CharField('Query Description', max_length=50)
system = models.ForeignKey(System)
class UserDashboard(models.Model):
'''User specific Dashboard '''
user = models.ForeignKey(User)
board_name = models.CharField('Dashboard Name', max_length=50)
class UserDashboardGroup(models.Model):
'''Groups on the dashboard (ie. 'System 1's Key Indicators', 'System 2's Hot Sheet', etc. '''
group_desc = models.CharField('Group Display Title', max_length=50)
user = models.ForeignKey(User)
dashboard = models.ForeignKey(UserDashboard)
system = models.ForeignKey(System)
system_group = models.ForeignKey(SystemGroup)
class UserGroupQuery(models.Model):
'''Queries that run in a specific group on the dashboard (ie. Within 'System 1's Key Indicators, we run queries for active users today, orders in the last hour, etc. '''
user = models.ForeignKey(User)
dashboard = ChainedForeignKey(UserDashboard, chained_field='user', chained_model_field='user', show_all=False, auto_choose=True)
user_dashboard_group = ChainedForeignKey(UserDashboardGroup, chained_field='dashboard', chained_model_field='dashboard')
query = models.ForeignKey(QueryData)
I am having problems with the very last part of this. I want to restrict the 'query' that appears in a admin page based on the selected user_dashboard_group. I'm not sure how I can do this, based on my current models though. query and user_dashboard_group both have a foreign key to System, but not to each other. To get this though, I'd have to get the user_dashboard_group's system and then use that to filter the QueryData. How can I do this?
Edit
I'm adding in a picture to (hopefully) describe a little better what I want to do.
In step 1, the user inputs a name for this group of queries. This group is associated with a system (#2) and a predefined group within the system (#3) (think of #3 as a 'customer' or a 'region', etc and #1 and #3 are NOT the same thing, despite the similar naming). They then select 'Save and Continue editing' on this inline form and the drop down at step 4 becomes populated with information from the above form. Once step #4 has a selection made, I want #5 to populate with data only from the associated system. Since #2 contains this information, I am hoping it is fairly easy to do this, but I can't figure out the chaining.
I also realized that I didn't mention I was using django-smart-selects
I have never worked with django-smart-selects, but following the docs, I would expect
query = models.ChainedForeignKey(QueryData,**kwargs)
instead of
query = models.ForeignKey(QueryData)
since you want the options in query depend on the other selections. Is that understanding correct? If so, it was only about defining the right **kwargs.
For **kwargs, I would propose something like this
Update
chained_field='query', chained_model_field='user_dashboard_group__system__query_set__query'
assuming that the fields name is 'system in both cases.
I am pretty sure that this describes the relationship correctly. I am just not sure, if django-smart-selects supports that syntax.
Same way you would do it across one relation, except across two.
field__relation1__relation2
Related
I have Django project with two database models: Device and DeviceTest.
Every device in system should walk through some test stages from manufacturing to sale. And therefore many DeviceTest objects are connected to Device object through foreign key relationship:
class Device(models.Model):
created_at = models.DateTimeField(auto_now_add=True)
name = models.CharField(max_length=255)
class DeviceTest(models.Model):
device = models.ForeignKey(Device)
created_at = models.DateTimeField(auto_now_add=True)
status = models.CharField(max_length=255)
tester = models.CharField(max_length=255)
action = models.CharField(max_length=255)
In my project there 2 kind of pages:
1) page with all tests for individual device
2) page with all devices with their latest status and action
Now I'm trying to optimize 2) page. To get latest test data I use this code:
status_list = []
last_update_list = []
last_action_list = []
for dev in device_list:
try:
latest_test = DeviceTest.objects.filter(device_id=dev.pk).latest('created_at')
status_list.append(latest_test.status)
last_update_list.append(latest_test.created_at)
last_action_list.append(latest_test.action)
except ObjectDoesNotExist:
status_list.append("Not checked")
last_update_list.append("Not checked")
last_action_list.append("Not checked")
For now in my database ~600 devices and ~4000 tests. And this is the main bottleneck in page loading.
What are the ways to speed up this calculation?
I came up with idea of adding extra field to Device model: foreign key to its last DeviceTest. In this scenario there wouldn't be any complicated requests to database at all.
And now I have a few questions:
Is it a good practice to add redundant field to model?
Is it possible to write migration rule to fill this redundant field to all current Devices?
And the most important, what are other choices to speed up my calculations?
id_list = [dev.id for dev in device_list]
devtests = DeviceTest.objects.filter(
device_id__in=id_list).order_by('-created_at').distinct('device')
That should give you, in one database call, in devtests only the latest entries for each device_id by create_at value.
Then do your loop and take the values from the list, instead of calling the database on each iteration.
However, it could also be a good idea to denormalize the database, like you suggested. Using "redundant fields" can definitely be good practice. You can automate the denormalization in the save() method or by listening to a post_save() signal from the related model.
Edit
First a correction: should be .distinct('device') (not created_at)
A list comprehension to fetch only the id values from the device_list. Equivalent to Device.objects.filter(...).values_list('id', flat=True)
id_list = [dev.id for dev in device_list]
Using the list of ids, we fetch all related DeviceTest objects
devtests = DeviceTest.objects.filter(device_id__in=id_list)
and order them by created_at but with the newest first -created_at. That also means, for every Device, the newest related DeviceTest will be first.
.order_by('-created_at')
Finally, for every device we only select the first related value we find (that would be the newest, because we sorted the values that way).
.distinct('device')
Additionally, you could also combine the device id and DeviceTest lookups
devtests = DeviceTest.objects.filter(device_in=Device.objects.filter(...))
then Django would create the SQL for it to do the JOIN in the database, so you don't need to load and loop the id list in Python.
I have the following models:
class Deal(models.Model):
date = models.DateTimeField(auto_now_add=True)
retailer = models.ForeignKey(Retailer, related_name='deals')
description = models.CharField(max_length=255)
...etc
class CustomerProfile(models.Model):
saved_deals = models.ManyToManyField(Deal, related_name='saved_by_customers', null=True, blank=True)
dismissed_deals = models.ManyToManyField(Deal, related_name='dismissed_by_customers', null=True, blank=True)
What I want to do is retrieve deals for a customer, but I don't want to include deals that they have dismissed.
I'm having trouble wrapping my head around the many-to-many relationship and am having no luck figuring out how to do this query. I'm assuming I should use an exclude on Deal.objects() but all the examples I see for exclude are excluding one item, not what amounts to multiple items.
When I naively tried just:
deals = Deal.objects.exclude(customer.saved_deals).all()
I get the error: "'ManyRelatedManager' object is not iterable"
If I say:
deals = Deal.objects.exclude(customer.saved_deals.all()).all()
I get "Too many values to unpack" (though I feel I should note there are only 5 deals and 2 customers in the database right now)
We (our client) presumes that he/she will have thousands of customers and tens of thousands of deals in the future, so I'd like to stay performance oriented as best I can. If this setup is incorrect, I'd love to know a better way.
Also, I am running django 1.5 as this is deployed on App Engine (using CloudSQL)
Where am I going wrong?
Suggest you use customer.saved_deals to get the list of deal ids to exclude (use values_list to quickly convert to a flat list).
This should save you excluding by a field in a joined table.
deals = Deals.exclude( id__in=customer.saved_deals.values_list('id', flat=True) )
You'd want to change this:
deals = Deal.objects.exclude(customer.saved_deals).all()
To something like this:
deals = Deal.objects.exclude(customer__id__in=[1,2,etc..]).all()
Basically, customer is the many-to-many foreign key, so you can't use it directly with an exclude.
Deals saved and deals dismissed are two fields describing almost same thing. There is also a risk too much columns may be used in database if these two field are allowed to store Null values. It's worth to consider remove dismissed_deals at all, and use saved_deal only with True or False statement.
Another thing to think about is move saved_deals out of CustomerProfile class to Deals class. Saved_deals are about Deals so it can prefer to live in Deals class.
class Deal(models.Model):
saved = models.BooleandField()
...
A real deal would have been made by one customer / buyer rather then few. A real customer can have milions of deals, so relating deals to customer would be good way.
class Deal(models.Model):
saved = models.BooleanField()
customer = models.ForeignKey(CustomerProfile)
....
What I want to do is retrieve deals for a customer, but I don't want to include deals that they have dismissed.
deals_for_customer = Deals.objects.all().filter(customer__name = "John")
There is double underscore between customer and name (customer__name), which let to filter model_name (customer is related to CustomerProfile which is model name) and name of field in that model (assuming CutomerProfile class has name attribute)
deals_saved = deals_for_customer.filter(saved = True)
That's it. I hope I could help. Let me know if not.
I am trying to add an additional custom field to a django model. I have been having quite a hard time figuring out how to do the following, and I will be awarding a 150pt bounty for the first fully correct answer when it becomes available (after it is available -- see as a reference Improving Python/django view code).
I have the following model, with a custom def that returns a video count for each user --
class UserProfile(models.Model):
user = models.ForeignKey(User, unique=True)
positions = models.ManyToManyField('Position', through ='PositionTimestamp', blank=True)
def count(self):
from django.db import connection
cursor = connection.cursor()
cursor.execute(
"""SELECT (
SELECT COUNT(*)
FROM videos_video v
WHERE v.uploaded_by_id = p.id
OR EXISTS (
SELECT NULL
FROM videos_videocredit c
WHERE c.video_id = v.id
AND c.profile_id = p.id
)
) AS Total_credits
FROM userprofile_userprofile p
WHERE p.id = %d"""%(int(self.pk))
)
return int(cursor.fetchone()[0])
I want to be able to order by the count, i.e., UserProfile.objects.order_by('count'). Of course, I can't do that, which is why I'm asking this question.
Previously, I tried adding a custom model Manager, but the problem with that was I also need to be able to filter by various criteria of the UserProfile model: Specifically, I need to be able to do: UserProfile.objects.filter(positions=x).order_by('count'). In addition, I need to stay in the ORM (cannot have a raw sql output) and I do not want to put the filtering logic into the SQL, because there are various filters, and would require several statements.
How exactly would I do this? Thank you.
My reaction is that you're trying to take a bigger bite than you can chew. Break it into bite size pieces by giving yourself more primitives to work with.
You want to create these two pieces separately so you can call on them:
Does this user get credit for this video? return boolean
For how many videos does this user get credit? return int
Then use a combination of #property, model managers, querysets, and methods that make it easiest to express what you need.
For example you might attach the "credit" to the video model taking a user parameter, or the user model taking a video parameter, or a "credit" manager on users which adds a count of videos for which they have credit.
It's not trivial, but shouldn't be too tricky if you work for it.
"couldn't you use something like the "extra" queryset modifier?"
see the docs
I didn't put this in an answer at first because I wasn't sure it would actually work or if it was what you needed - it was more like a nudge in the (hopefully) right direction.
in the docs on that page there is an example
query
Blog.objects.extra(
select={
'entry_count': 'SELECT COUNT(*) FROM blog_entry WHERE blog_entry.blog_id = blog_blog.id'
},
)
resulting sql
SELECT blog_blog.*, (SELECT COUNT(*) FROM blog_entry WHERE blog_entry.blog_id = blog_blog.id) AS entry_count
FROM blog_blog;
Perhaps doing something like that and accessing the user id which you currently have as p.id as appname_userprofile.id
note:
Im just winging it so try to play around a bit.
perhaps use the shell to output the query as sql and see what you are getting.
models:
class Positions(models.Model):
x = models.IntegerField()
class Meta:
db_table = 'xtest_positions'
class UserProfile(models.Model):
user = models.ForeignKey(User, unique=True)
positions = models.ManyToManyField(Positions)
class Meta:
db_table = 'xtest_users'
class Video(models.Model):
usr = models.ForeignKey(UserProfile)
views = models.IntegerField()
class Meta:
db_table = 'xtest_video'
result:
test = UserProfile.objects.annotate(video_views=Sum('video__views')).order_by('video_views')
for t in test:
print t.video_views
doc: https://docs.djangoproject.com/en/dev/topics/db/aggregation/
This is either what you want, or I've completely misunderstood!.. Anywhoo... Hope it helps!
Previously had a go at asking a more specific version of this question, but had trouble articulating what my question was. On reflection that made me doubt if my chosen solution was correct for the problem, so this time I will explain the problem and ask if a) I am on the right track and b) if there is a way around my current brick wall.
I am currently building a web interface to enable an existing database to be interrogated by (a small number of) users. Sticking with the analogy from the docs, I have models that look something like this:
class Musician(models.Model):
first_name = models.CharField(max_length=50)
last_name = models.CharField(max_length=50)
dob = models.DateField()
class Album(models.Model):
artist = models.ForeignKey(Musician)
name = models.CharField(max_length=100)
class Instrument(models.Model):
artist = models.ForeignKey(Musician)
name = models.CharField(max_length=100)
Where I have one central table (Musician) and several tables of associated data that are related by either ForeignKey or OneToOneFields. Users interact with the database by creating filtering criteria to select a subset of Musicians based on data the data on the main or related tables. Likewise, the users can then select what piece of data is used to rank results that are presented to them. The results are then viewed initially as a 2 dimensional table with a single row per Musician with selected data fields (or aggregates) in each column.
To give you some idea of scale, the database has ~5,000 Musicians with around 20 fields of related data.
Up to here is fine and I have a working implementation. However, it is important that I have the ability for a given user to upload there own annotation data sets (more than one) and then filter and order on these in the same way they can with the existing data.
The way I had tried to do this was to add the models:
class UserDataSets(models.Model):
user = models.ForeignKey(User)
name = models.CharField(max_length=100)
description = models.CharField(max_length=64)
results = models.ManyToManyField(Musician, through='UserData')
class UserData(models.Model):
artist = models.ForeignKey(Musician)
dataset = models.ForeignKey(UserDataSets)
score = models.IntegerField()
class Meta:
unique_together = (("artist", "dataset"),)
I have a simple upload mechanism enabling users to upload a data set file that consists of 1 to 1 relationship between a Musician and their "score". Within a given user dataset each artist will be unique, but different datasets are independent from each other and will often contain entries for the same musician.
This worked fine for displaying the data, starting from a given artist I can do something like this:
artist = Musician.objects.get(pk=1)
dataset = UserDataSets.objects.get(pk=5)
print artist.userdata_set.get(dataset=dataset.pk)
However, this approach fell over when I came to implement the filtering and ordering of query set of musicians based on the data contained in a single user data set. For example, I could easily order the query set based on all of the data in the UserData table like this:
artists = Musician.objects.all().order_by(userdata__score)
But that does not help me order by the results of a given single user dataset. Likewise I need to be able to filter the query set based on the "scores" from different user data sets (eg find all musicians with a score > 5 in dataset1 and < 2 in dataset2).
Is there a way of doing this, or am I going about the whole thing wrong?
edit: nevermind, it's wrong. I'll keep it so you can read, but then I'll delete afterward.
Hi,
If I understand correctly, you can try something like this:
artists = Musician.objects.select_related('UserDataSets').filter( Q(userdata__score_gt=5, userdata__id=1) | Q(userdata__sorce_lt=2, userdata__id=2 )
For more info on how to use Q, check this: Complex lookups with Q objects.
this is a model of the view table.
class QryDescChar(models.Model):
iid_id = models.IntegerField()
cid_id = models.IntegerField()
cs = models.CharField(max_length=10)
cid = models.IntegerField()
charname = models.CharField(max_length=50)
class Meta:
db_table = u'qry_desc_char'
this is the SQL i use to create the table
CREATE VIEW qry_desc_char as
SELECT
tbl_desc.iid_id,
tbl_desc.cid_id,
tbl_desc.cs,
tbl_char.cid,
tbl_char.charname
FROM tbl_desC,tbl_char
WHERE tbl_desc.cid_id = tbl_char.cid;
i dont know if i need a function in models or views or both. i want to get a list of objects from that database to display it. This might be easy but im new at Django and python so i having some problems
Django 1.1 brought in a new feature that you might find useful. You should be able to do something like:
class QryDescChar(models.Model):
iid_id = models.IntegerField()
cid_id = models.IntegerField()
cs = models.CharField(max_length=10)
cid = models.IntegerField()
charname = models.CharField(max_length=50)
class Meta:
db_table = u'qry_desc_char'
managed = False
The documentation for the managed Meta class option is here. A relevant quote:
If False, no database table creation
or deletion operations will be
performed for this model. This is
useful if the model represents an
existing table or a database view that
has been created by some other means.
This is the only difference when
managed is False. All other aspects of
model handling are exactly the same as
normal.
Once that is done, you should be able to use your model normally. To get a list of objects you'd do something like:
qry_desc_char_list = QryDescChar.objects.all()
To actually get the list into your template you might want to look at generic views, specifically the object_list view.
If your RDBMS lets you create writable views and the view you create has the exact structure than the table Django would create I guess that should work directly.
(This is an old question, but is an area that still trips people up and is still highly relevant to anyone using Django with a pre-existing, normalized schema.)
In your SELECT statement you will need to add a numeric "id" because Django expects one, even on an unmanaged model. You can use the row_number() window function to accomplish this if there isn't a guaranteed unique integer value on the row somewhere (and with views this is often the case).
In this case I'm using an ORDER BY clause with the window function, but you can do anything that's valid, and while you're at it you may as well use a clause that's useful to you in some way. Just make sure you do not try to use Django ORM dot references to relations because they look for the "id" column by default, and yours are fake.
Additionally I would consider renaming my output columns to something more meaningful if you're going to use it within an object. With those changes in place the query would look more like (of course, substitute your own terms for the "AS" clauses):
CREATE VIEW qry_desc_char as
SELECT
row_number() OVER (ORDER BY tbl_char.cid) AS id,
tbl_desc.iid_id AS iid_id,
tbl_desc.cid_id AS cid_id,
tbl_desc.cs AS a_better_name,
tbl_char.cid AS something_descriptive,
tbl_char.charname AS name
FROM tbl_desc,tbl_char
WHERE tbl_desc.cid_id = tbl_char.cid;
Once that is done, in Django your model could look like this:
class QryDescChar(models.Model):
iid_id = models.ForeignKey('WhateverIidIs', related_name='+',
db_column='iid_id', on_delete=models.DO_NOTHING)
cid_id = models.ForeignKey('WhateverCidIs', related_name='+',
db_column='cid_id', on_delete=models.DO_NOTHING)
a_better_name = models.CharField(max_length=10)
something_descriptive = models.IntegerField()
name = models.CharField(max_length=50)
class Meta:
managed = False
db_table = 'qry_desc_char'
You don't need the "_id" part on the end of the id column names, because you can declare the column name on the Django model with something more descriptive using the "db_column" argument as I did above (but here I only it to prevent Django from adding another "_id" to the end of cid_id and iid_id -- which added zero semantic value to your code). Also, note the "on_delete" argument. Django does its own thing when it comes to cascading deletes, and on an interesting data model you don't want this -- and when it comes to views you'll just get an error and an aborted transaction. Prior to Django 1.5 you have to patch it to make DO_NOTHING actually mean "do nothing" -- otherwise it will still try to (needlessly) query and collect all related objects before going through its delete cycle, and the query will fail, halting the entire operation.
Incidentally, I wrote an in-depth explanation of how to do this just the other day.
You are trying to fetch records from a view. This is not correct as a view does not map to a model, a table maps to a model.
You should use Django ORM to fetch QryDescChar objects. Please note that Django ORM will fetch them directly from the table. You can consult Django docs for extra() and select_related() methods which will allow you to fetch related data (data you want to get from the other table) in different ways.