For the following models I want to retrieve all the devices that have an entry in the History table with transition_date between a specified interval:
class History(models.Model):
device = models.ForeignKey(DeviceModel, to_field='id')
transition_date = models.DateTimeField()
class Meta:
db_table = 'History'
class DeviceModel(models.Model):
id = models.IntegerField()
name = models.CharField()
class Meta:
db_table = 'Devices'
I have this code that filters for the specified interval:
devices = DeviceModel.objects.filter(history__transition_date__range=(startDate, endDate))
That gives me as many rows as History table has with transition_date in the specified range.
The filter function performs an INNER JOIN between DeviceModel and History on device id retrieving only DeviceModel fields. My question is how do I retrieve data from both History and DeviceModel at the same time while joining them as with filter/select_related on device id.
I'd rather not write a custom SQL query.
In your models Device and History models are related with a foreign key from History to DeviceModel, this mean when you have a History object you can retrieve the Device model related to it, and viceversa (if you have a Device you can get its History).
Example:
first_history = History.objects.all()[0]
first_history.device # This return the device object related with first_history
first_history.device.name # This return the name of the device related with first_history
But it works also in the other way, you could do:
first_device = Device.objects.all()[0]
first_device.history # This return the history object related with device
first_device.history.transition_date # Exactly as before, can access history fields
So in your query:
devices = DeviceModel.objects.filter(history__transition_date__range=(startDate, endDate))
This return a device list, but you can access to the history related with each device object
Isn't that enough for you ? You have a Device list, and each device can access to its related History object
Info: When you declare a ForeignKey field the models are related by id for default, I say this because you're doing:
device = models.ForeignKey(DeviceModel, to_field='id')
as you can see you're using to_field='id' but this relation is done by default, if you do:
device = models.ForeignKey(DeviceModel)
You'll get same results
(EDIT) Using .values() to obtain list [device.name, history.date]
To get a list like you said [device.name, history.date] you can use .values() function of Django QuerySet, official documentation here
You can try something like:
devices = DeviceModel.objects.filter(history__transition_date__range=(startDate, endDate)).values('name','history__transition_date')
# Notice that it is 'history _ _ transition_date with 2 underscores
Related
I have users who create (or receive) transactions. The transaction hierarchy I have is a multi-table inheritance, with Transaction as the base model containing the common fields between all transaction types, such as User (FK), amount, etc. I have several transaction types, which extend the Transaction model with type specific data.
For the sake of this example, a simplified structure illustrating my problem can be found below.
from model_utils.managers import InheritanceManager
class User(models.Model):
pass
class Transaction(models.Model):
DEPOSIT = 'deposit'
WITHDRAWAL = 'withdrawal'
TRANSFER = 'transfer'
TYPES = (
(DEPOSIT, DEPOSIT),
(WITHDRAWAL, WITHDRAWAL),
(TRANSFER, TRANSFER),
)
type = models.CharField(max_length=24, choices=TYPES)
user = models.ForeignKey(User)
amount = models.PositiveIntegerField()
objects = InheritanceManager()
class Meta:
indexes = [
models.Index(fields=['user']),
models.Index(fields=['type'])
]
class Withdrawal(Transaction):
TYPE = Transaction.WITHDRAWAL
bank_account = models.ForeignKey(BankAccount)
class Deposit(Transaction):
TYPE = Transaction.DEPOSIT
card = models.ForeignKey(Card)
class Transfer(Transaction):
TYPE = Transaction.Transfer
recipient = models.ForeignKey(User)
class Meta:
indexes = [
models.Index(fields=['recipient'])
]
I then set each transaction's type in the inherited model's .save() method. This is all fine and well.
The problem comes in when I would like to fetch the a user's transactions. Specifically, I require the sub-model instances (deposits, transfers and withdrawals), rather than the base model (transactions). I also require transactions that the user both created themselves AND transfers they have received. For the former I use django-model-utils's fantastic IneritanceManager, which works great. Except that when I include the filtering on the transfer submodel's recipient FK field the DB query increases by an order of magnitude.
As illustrated above I have placed indexes on the Transaction user column and the Transfer recipient column. But it appeared to me that what I may need is an index on the Transaction subtype, if that is at all possible. I have attempted to achieve this effect by putting an index on the Transaction type field and including it in the query, as you will see below, but this appears to have no effect. Furthermore, I use .select_related() for the user objects since they are required in the serializations.
The query is structured as such:
from django.db.models import Q
queryset = Transaction.objects.select_related(
'user',
'transfer__recipient'
).select_subclasses().filter(
Q(user=request.user) |
Q(type=Transaction.TRANSFER, transfer__recipient=request.user)
).order_by('-id')
So my question is, why is there an order of magnitude difference on the DB query when including the Transfer.recipient in the query? Have I missed something? Am I doing something silly? Or is there a way I can optimise this further?
I have Django project with two database models: Device and DeviceTest.
Every device in system should walk through some test stages from manufacturing to sale. And therefore many DeviceTest objects are connected to Device object through foreign key relationship:
class Device(models.Model):
created_at = models.DateTimeField(auto_now_add=True)
name = models.CharField(max_length=255)
class DeviceTest(models.Model):
device = models.ForeignKey(Device)
created_at = models.DateTimeField(auto_now_add=True)
status = models.CharField(max_length=255)
tester = models.CharField(max_length=255)
action = models.CharField(max_length=255)
In my project there 2 kind of pages:
1) page with all tests for individual device
2) page with all devices with their latest status and action
Now I'm trying to optimize 2) page. To get latest test data I use this code:
status_list = []
last_update_list = []
last_action_list = []
for dev in device_list:
try:
latest_test = DeviceTest.objects.filter(device_id=dev.pk).latest('created_at')
status_list.append(latest_test.status)
last_update_list.append(latest_test.created_at)
last_action_list.append(latest_test.action)
except ObjectDoesNotExist:
status_list.append("Not checked")
last_update_list.append("Not checked")
last_action_list.append("Not checked")
For now in my database ~600 devices and ~4000 tests. And this is the main bottleneck in page loading.
What are the ways to speed up this calculation?
I came up with idea of adding extra field to Device model: foreign key to its last DeviceTest. In this scenario there wouldn't be any complicated requests to database at all.
And now I have a few questions:
Is it a good practice to add redundant field to model?
Is it possible to write migration rule to fill this redundant field to all current Devices?
And the most important, what are other choices to speed up my calculations?
id_list = [dev.id for dev in device_list]
devtests = DeviceTest.objects.filter(
device_id__in=id_list).order_by('-created_at').distinct('device')
That should give you, in one database call, in devtests only the latest entries for each device_id by create_at value.
Then do your loop and take the values from the list, instead of calling the database on each iteration.
However, it could also be a good idea to denormalize the database, like you suggested. Using "redundant fields" can definitely be good practice. You can automate the denormalization in the save() method or by listening to a post_save() signal from the related model.
Edit
First a correction: should be .distinct('device') (not created_at)
A list comprehension to fetch only the id values from the device_list. Equivalent to Device.objects.filter(...).values_list('id', flat=True)
id_list = [dev.id for dev in device_list]
Using the list of ids, we fetch all related DeviceTest objects
devtests = DeviceTest.objects.filter(device_id__in=id_list)
and order them by created_at but with the newest first -created_at. That also means, for every Device, the newest related DeviceTest will be first.
.order_by('-created_at')
Finally, for every device we only select the first related value we find (that would be the newest, because we sorted the values that way).
.distinct('device')
Additionally, you could also combine the device id and DeviceTest lookups
devtests = DeviceTest.objects.filter(device_in=Device.objects.filter(...))
then Django would create the SQL for it to do the JOIN in the database, so you don't need to load and loop the id list in Python.
I have a person, which has a foreign key to User:
p = Personne.objects.get(user=self.request.user)
This person has "friends" through the model PersonneRelation where there's a src person and a dst person, so I'm retrieving all the person's friends like this:
friends = [a.dst.pk for a in PersonneRelation.objects.filter(src=p)]
I have a model Travel which embeds a description of a travel. I have also a model "activity" which is an activity the friend's current user have done (Activite). So here's the way I'm retrieving all the activities related to the current user:
context['activites'] = Activite.objects.filter(
Q(travel__personne__pk__in=friends) | Q(relation__src__pk__in=friends),
)
Everything works fine. I have also a model PersonneLiked where you precise if you liked an Activite. Thus this model has a foreign key to Activite.
class PersonneLiked(models.Model):
src = models.ForeignKey('Personne', related_name='liked_src')
dst = models.ForeignKey('Personne', related_name='liked_dst')
activite = models.ForeignKey('Activite', related_name='liked_activite')
# liked = thumb added *OR* removed :
liked = models.BooleanField(default=True)
What is the code I should do to retrieve all the PersonneLiked of context['activites']? It's like making an OUTER JOIN in SQL, but I dont know how I could do this with Django.
Since context['activities'] is a queryset of the appropriate Activite objects then you can do:
PersonneLiked.objects.filter(activite__in=context['activities'])
(Note that if this were a M2M relationship rather than a ForeignKey, this method would create duplicate entries in your final queryset.)
I am implementing a web interface for email lists. When a list administrator logs in, the site will visually display which lists they are an owner of and corresponding information about the lists. For this I have decided to have two tables:
1) An owner table which contains entries for information about list administrators. Each of these entries contains a 'ManyToManyField' which holds the information about which lists the owner in any given entry is an administrator for.
2) A list table which contains entries with information about each email list. Each entry contains the name of the list a 'ManyToManyField' holding information about which owners are administrators the list.
Here is the code in models.py:
from django.db import models
class ListEntry(models.Model):
name = models.CharField(max_length=64)
owners = models.ManyToManyField('OwnerEntry')
date = models.DateTimeField('date created')
class Meta:
ordering = ('name',)
class OwnerEntry(models.Model):
name = models.CharField(max_length=32)
lists = models.ManyToManyField('ListEntry')
class Meta:
ordering = ('name',)
I have already set up a simple local database to create a basic working website with. I have populated it with test entries using this code:
from list_app.models import *
from datetime import *
le1 = ListEntry(
name = "Physics 211 email list",
date = datetime.now(),
)
le1.save()
le2 = ListEntry(
name = "Physics 265 email list",
date = datetime(2014,1,1),
)
le2.save()
oe1 = OwnerEntry(
name = 'wasingej',
)
oe1.save()
oe1.lists.add(le1,le2)
le1.owners.add(oe1)
le2.owners.add(oe1)
oe2 = OwnerEntry(
name = 'doej',
)
oe2.save()
oe2.lists.add(le1)
le1.owners.add(oe2)
Here is where my error occurs: When the user has logged in via CAS, I have them redirected to this page in views.py:
def login_success(request):
u = OwnerEntry(name=request.user)
print(u.name)
print(u.lists)
return HttpResponse("login success!")
At the line 'print(u.lists)', I get the error "" needs to have a value for field "ownerentry" before this many-to-many relationship can be used.
What am I doing wrong here?
Your model structure is broken, for a start. You don't need ManyToManyFields on both sides of the relationship, only one - Django will provide the accessor for the reverse relationship.
Your issue is happening because you are not querying an existing instance from the database, you are instantiating an unsaved one. To query, you use model.objects.get():
u = OwnerEntry.objects.get(name=request.user.username)
You need to provide the actual class to the ManyToManyField constructor, not a string.
https://docs.djangoproject.com/en/dev/topics/db/examples/many_to_many/
Is there an easy way to fetch the ManyToMany objects from a query that returns more than one object? The way I am doing it now doesn't feel as sexy as I would like it to. Here is how I am doing it now in my view:
contacts = Contact.objects.all()
# Use Custom Manager Method to Fetch Each Contacts Phone Numbers
contacts = PhoneNumber.objects.inject(contacts)
My Models:
class PhoneNumber(models.Model):
number = models.CharField()
type = models.CharField()
# My Custom Manager
objects = PhoneNumberManager()
class Contact(models.Model):
name = models.CharField()
numbers = models.ManyToManyField(PhoneNumber, through='ContactPhoneNumbers')
class ContactPhoneNumbers(models.Model):
number = models.ForeignKey(PhoneNumber)
contact = models.ForeignKey(Contact)
ext = models.CharField()
My Custom Manager:
class PhoneNumberManager(models.Manager):
def inject(self, contacts):
contact_ids = ','.join([str(item.id) for item in contacts])
cursor = connection.cursor()
cursor.execute("""
SELECT l.contact_id, l.ext, p.number, p.type
FROM svcontact_contactphonenumbers l, svcontact_phonenumber p
WHERE p.id = l.number_id AND l.contact_id IN(%s)
""" % contact_ids)
result = {}
for row in cursor.fetchall():
id = str(row[0])
if not id in result:
result[id] = []
result[id].append({
'ext': row[1],
'number': row[2],
'type': row[3]
})
for contact in contacts:
id = str(contact.id)
if id in result:
contact.phonenumbers = result[id]
return contacts
There are a couple things you can do to find sexiness here :-)
Django does not have any OOTB way to inject the properties of the through table into your Contact instance. A M2M table with extra data is a SQL concept, so Django wouldn't try to fight the relations, nor guess what should happen in the event of namespace collision, etc... . In fact, I'd go so far as to say that you probably do not want to inject arbitrary model properties onto your Contact object... if you find yourself needing to do that, then it's probably a sign you should revise your model definition.
Instead, Django provides convenient ways to access the relation seamlessly, both in queries and for data retrieval, all the while preserving the integrity of the entities. In this case, you'll find that your Contact object offers a contactphonenumbers_set property that you can use to access the through data:
>>> c = Contact.objects.get(id=1)
>>> c.contactphonenumbers_set.all()
# Would produce a list of ContactPhoneNumbers objects for that contact
This means, in your case, to iterate of all contact phone numbers (for example) you would:
for contact in Contact.objects.all():
for phone in contact.contactphonenumbers_set.all():
print phone.number.number, phone.number.type, phone.ext
If you really, really, really want to do the injection for some reason, you'll see you can do that using the 3-line code sample immediately above: just change the print statements into assignment statements.
On a separate note, just for future reference, you could have written your inject function without SQL statements. In Django, the through table is itself a model, so you can query it directly:
def inject(self, contacts):
contact_phone_numbers = ContactPhoneNumbers.objects.\
filter(contact__in=contacts)
# And then do the result construction...
# - use contact_phone_number.number.phone to get the phone and ext
# - use contact_phone_number.contact to get the contact instance