I'm trying to improve the performance of this code:
orderitems = OrderItem.objects.filter(order__contact=contact)
for orderitem in orderitems:
try:
pv = ProductVariation.objects.get(product=orderitem.product)
if pv.parent_id == parent_product.id:
return True
except:
pass
Essentially I want to get rid of the 'for' loop because its slow. I would like to do it using a single queryset if possible, but I just can't get my head around the syntax. Here is the SQL that I effectively want to reproduce. It creates a list which is fairly short so I can iterate through that looking for a match:
SELECT parent_id
FROM configurable_productvariation
WHERE product_id IN (
SELECT product_id
FROM shop_orderitem
WHERE order_id
IN (
SELECT id
FROM shop_order
WHERE contact_id = 4));
The '4' is the 'contact' referred to in the first line of python.
Many thanks,
Thomas
This should generate sql like yours
product_ids = OrderItem.objects.filter(order__contact=contact).values_list('product_id', flat=True)
ProductVariation.objects.filter(product_id__in=product_ids)
Related
I have a programme where users should be able to filter different types of technologies by their attributes. My question is, how would I filter the technologies when there's potential conflicts and empty values in the parameters I use to filter?
Forms.py:
class FilterDataForm(forms.ModelForm):
ASSESSMENT = (('', ''),('Yes', 'Yes'),('No', 'No'),)
q01_suitability_for_task_x = forms.ChoiceField(label='Is the technology suitable for x?',
choices=ASSESSMENT, help_text='Please select yes or no', required=False,)
q02_suitability_for_environment_y = forms.ChoiceField(label='Is the technology suitable for environment Y?',
choices=ASSESSMENT, help_text='Please select yes or no', required=False)
There are many fields in my model like the ones above.
views.py
class TechListView(ListView):
model = MiningTech
def get_queryset(self):
q1 = self.request.GET.get('q01_suitability_for_task_x', '')
q2 = self.request.GET.get('q02_suitability_for_environment_y', '')
object_list = MiningTech.objects.filter(q01_suitability_for_task_x=q1).filter(
q02_suitability_for_environment_y=q2)
return object_list
The difficulty is that not all technology db entries will have data. So in my current setup there's times where I will filter out objects that have one attribute but not another.
For instance if my db has:
pk1: q01_suitability_for_task_x=Yes; q02_suitability_for_environment_y=Yes;
pk2: q01_suitability_for_task_x=Yes; q02_suitability_for_environment_y='';
In the form, if I don't select any value for q01_suitability_for_task_x, and select Yes for q02_suitability_for_environment_y, I get nothing back in the queryset because there are no q01_suitability_for_task_x empty fields.
Any help would be appreciated. I'm also ok with restructuring everything if need be.
The problem is that your self.request.GET.get(...) code defaults to an empty string if there is no value found, so your model .filter() is looking for matches where the string is ''.
I would restructure the first part of get_queryset() to build a dictionary that can be unpacked into your filter. If the value doesn't exist then it doesn't get added to the filter dictionary:
filters = {}
q1 = self.request.GET.get('q01_suitability_for_task_x', None)
q2 = self.request.GET.get('q02_suitability_for_environment_y', None)
if q1 is not None:
filters['q01_suitability_for_task_x'] = q1
... etc ...
object_list = MiningTech.objects.filter(**filters)
If you have a lot of q1, q2, etc. items then consider putting them in a list, looping through and inserting into the dictionary if .get(...) returns anything.
Edit: Because there are indeed a lot possible filters, the final solution looks as follows:
def get_queryset(self):
filters = {}
for key, value in self.request.GET.items():
if value != '':
filters[key] = value
object_list = Tech.objects.filter(**filters)
I want to get a list of max ids for a filter I have in Django
class Foo(models.Model):
name = models.CharField()
poo = models.CharField()
Foo.objects.filter(name__in=['foo','koo','too']).latest_by_id()
End result a queryset having only the latest objects by id for each name. How can I do that in Django?
Edit: I want multiple objects in the end result. Not just one object.
Edit1: Added __in. Once again I need only latest( as a result distinct) objects for each name.
Something like this.
my_id_list = [Foo.objects.filter(name=name).latest('id').id for name in ['foo','koo','too']]
Foo.objects.filter(id__in=my_id_list)
The above works. But I want a more concise way of doing it. Is it possible to do this in a single query/filter annotate combination?
you can try:
qs = Foo.objects.filter(name__in=['foo','koo','too'])
# Get list of max == last pk for your filter objects
max_pks = qs.annotate(mpk=Max('pk')).order_by().values_list('mpk', flat=True)
# after it filter your queryset by last pk
result = qs.filter(pk__in=max_pks)
If you are using PostgreSQL you can do the following
Foo.objects.order_by('name', '-id').distinct('name')
MySQL is more complicated since is lacks a DISTINCT ON clause. Here is the raw query that is very hard to force Django to generate from ORM function calls:
Foo.objects.raw("""
SELECT
*
FROM
`foo`
GROUP BY `foo`.`name`
ORDER BY `foo`.`name` ASC , `foo`.`id` DESC
""")
Here is the raw query set Django ORM:
ob = Shop.objects.raw('SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1 ORDER BY some_field')
listorder = ["check_in","check_out","location"]
This listorder part is dynamic. I don't know how it ll be. It ll change the ordering from time to time & one more thing can't apply ordering on raw query sets because I want the whole data for other purpose.After that only i can apply ordering.
Here want ordering by the list "listorder".
mObj = ob.order_by[*listorder].
In above facing error like can't apply ordering to raw query sets.
Anyone having any idea?
If you want a raw queryset to be ordered by different fields, you can add them to the ORDER BY clause.
ob = Shop.objects.raw('SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1 ORDER BY check_in, check_out, location')
if you want the order to be reversed for a particular field you can change it as
ob = Shop.objects.raw('SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1 ORDER BY check_in, check_out DESC, location')
If the ordering is going to be dynamic, you can create the querystring dynamically.
qs = ''SELECT * from shops GROUP BY
(duplicate_field_name) having COUNT(*) = 1'
# some other code here to decide what your ordering is example
order_fields = ['id','location','check_in','check_out']
qs = qs + "ORDER BY " + ",".join(order_fields)
Then you can query as before
Shop.objects.raw(qs)
Using Django's ORM, I am trying to find instances of myModel based on two of its datetime variables; specifically, where the months of these two datetimes are not equal. I understand to filter by the value of a modelfield, you can use Django's F( ) expressions, so I thought I'd try something like this:
myModel.objects.filter(fixed_date__month=F('closed_date__month'))
I know this wouldn't find instances where they aren't equal, but I thought it'd be a good first step since I've never used the F expressions before. However, it doesn't work as I thought it should. I expected it to give me a queryset of objects where the value of the fixed_date month was equal to the value of the closed_date month, but instead I get an error:
FieldError: Join on field 'closed_date' not permitted. Did you misspell 'month' for the lookup type?
I'm not sure if what I'm trying to do isn't possible or straightforward with the ORM, or if I'm just making a simple mistake.
It doesn't look like django F objects currently support extracting the month inside a DateTimeField, the error message seems to be stating that the F object is trying to convert the '__' inside the string 'closed_date__month' as a Foreignkey between different objects, which are usually stored as joins inside an sql database.
You could carry out the same query by iterating across the objects:
result = []
for obj in myModel.objects.all():
if obj.fixed_date.month != obj.closed_date.month:
result.append(obj)
or as a list comprehension:
result = [obj for obj in myModel.objects.all() if obj.fixed_date.month != obj.closed_date.month]
Alternatively, if this is not efficient enough, the months for the two dates could be cached as IntegerFields within the model, something like:
class myModel(models.Model):
....other fields....
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
fixed_month = models.IntegerField()
closed_month = models.IntegerField()
store the two integers when the relevant dates are updated:
myModel.fixed_month = myModel.fixed_date.month
myModel.save()
Then use an F object to compare the two integer fields:
myModel.objects.filter(fixed_month__ne=F('closed_month'))
The ne modifier will do the not equal test.
Edit - using raw sql
If you are using an sql based database, then most efficient method is to use the .raw() method to manually specify the sql:
myModel.objects.raw('SELECT * FROM stuff_mymodel WHERE MONTH(fixed_date) != MONTH(close_date)')
Where 'stuff_mymodel' is the correct name of the table in the database. This uses the SQL MONTH() function to extract the values from the month fields, and compare their values. It will return a collection of objects.
There is some nay-saying about the django query system, for example: http://charlesleifer.com/blog/shortcomings-in-the-django-orm-and-a-look-at-peewee-a-lightweight-alternative/. This example could be taken as demonstrating another inconsistency in it's query api.
My thinking is this:
class myModel(models.Model):
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
def has_diff_months(self):
if self.fixed_date.month != self.closed_date.month:
return True
return False
Then:
[x for x in myModel.objects.all() if x.has_diff_months()]
However, for a truly efficient solution you'd have to use another column. It makes sense to me that it'd be a computed boolean field that is created when you save, like so:
class myModel(models.Model):
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
diff_months = models.BooleanField()
#overriding save method
def save(self, *args, **kwargs):
#calculating the value for diff_months
self.diff_months = (self.fixed_date.month != self.closed_date.month)
#aaand... saving:
super(Blog, self).save(*args, **kwargs)
Then filtering would simply be:
myModel.objects.filter(diff_months=True)
I have a third-party function which gives me a filtered queryset (e.g. records with 'valid'=True) but I want to remove a particular condition (e.g. to have all records, both valid and invalid).
Is there a way to remove a filter condition to an already-filtered queryset?
E.g.
only_valid = MyModel.objects.filter(valid=True)
all_records = only_valid.**remove_filter**('valid')
(I know that it would be better to define 'all_records' before 'only_valid', but this is just an example...)
Although there is no official way to do this using filter notation, you may easily do it with Q-notation.
For example, if you ensure that third-part function returns a Q object, not a filtered QuerySet, you may do the following:
q = ThirdParty()
q = q | Q(valid=False)
And the resulting SQL conditions will be joined using OR operator.
From the docs:
Each time you refine a QuerySet, you get a brand-new QuerySet that is in no way bound to the previous QuerySet. Each refinement creates a separate and distinct QuerySet that can be stored, used and reused.
I doubt therefore, that there is a standard way to do it. You could dig into the code, see, what filter() does and try a bit. If that doesn't help, my assumption is, you're out of luck and need to re-build the query yourself.
Use this function
from django.db.models import Q
def remove_filter(lookup, queryset):
"""
Remove filter lookup in queryset
```
>>> queryset = User.objects.filter(email='user#gmail.com')
>>> queryset.count()
1
>>> remove_filter('email', queryset)
>>> queryset.count()
1000
```
"""
query = queryset.query
q = Q(**{lookup: None})
clause, _ = query._add_q(q, self.used_aliases)
def filter_lookups(child):
return child.lhs.target != clause.children[0].lhs.target
query.where.children = list(filter(filter_lookups, query.where.children))
Here's what I did in a similar case.
all_records = MyModel.objects.all()
only_valid = MyModel.objects.filter(valid=True)
only_valid.original = all_records
...
all_records = only_valid.original
Obviously this clears any other filters too so it won't be right for every case.
original_query_set = MyModel.objects.filter(**conditions)
model_class = orginal_query_set.model
new_query_set = model_class.objects.filter(**new_conditions)
You can use the .model attribute on a QuerySet to get the model class, then use the model class to make a brand new QuerySet.
Thanks for making me check the source code Boldewyn. So, it seems there's a new method right below filter() with the same parameters... called exclude() (as of commit mini-hash ef6c680, for when it loses its line number)
Return a new QuerySet instance with NOT (args) ANDed to the existing set.
So, to answer the original question:
only_valid = MyModel.objects.filter(valid=True)
filtered_results = only_valid.exclude(the_condition_to_remove=True)