Is there a way to do a query and exclude a list of things, instead of calling exclude multiple times?
Based on your reply to Ned, it sounds like you just want to exclude a list of tags. So you could just use the in filter:
names_to_exclude = [o.name for o in objects_to_exclude]
Foo.objects.exclude(name__in=names_to_exclude)
Does that do what you want?
What's wrong with calling exclude multiple times? Queries are lazy, nothing happens until you try to pull data from it, so there's no downside to using .exclude() more than once.
You can try this also.
exclude_list = ['A', 'B', 'C']
qs = Foo.objects.exclude(items__in=exclude_list)
You can do it pretty easily with the Q object:
from django.db.models import Q
excludes = None
for tag in ignored_tags:
q = Q(tag=tag)
excludes = (excludes and (excludes | q)) or q # makes sure excludes is set properly
set_minus_excluded = Foo.objects.exclude(excludes)
You should also be able to do it dynamically with exclude():
qs = Foo.objects.all()
for tag in ignored_tags:
qs = qs.exclude(tag=tag)
To improve on Daniel Roseman's answer I think it would be better to get the values you need directly from the queryset instead of the for loop that could be expensive on large data sets i.e.
names_to_exclude = objects_to_exclude.values_list('name')
Foo.objects.exclude(name__in=names_to_exclude)
Related
Is it possible to filter a Django queryset by model property?
i have a method in my model:
#property
def myproperty(self):
[..]
and now i want to filter by this property like:
MyModel.objects.filter(myproperty=[..])
is this somehow possible?
Nope. Django filters operate at the database level, generating SQL. To filter based on Python properties, you have to load the object into Python to evaluate the property--and at that point, you've already done all the work to load it.
I might be misunderstanding your original question, but there is a filter builtin in python.
filtered = filter(myproperty, MyModel.objects)
But it's better to use a list comprehension:
filtered = [x for x in MyModel.objects if x.myproperty()]
or even better, a generator expression:
filtered = (x for x in MyModel.objects if x.myproperty())
Riffing off #TheGrimmScientist's suggested workaround, you can make these "sql properties" by defining them on the Manager or the QuerySet, and reuse/chain/compose them:
With a Manager:
class CompanyManager(models.Manager):
def with_chairs_needed(self):
return self.annotate(chairs_needed=F('num_employees') - F('num_chairs'))
class Company(models.Model):
# ...
objects = CompanyManager()
Company.objects.with_chairs_needed().filter(chairs_needed__lt=4)
With a QuerySet:
class CompanyQuerySet(models.QuerySet):
def many_employees(self, n=50):
return self.filter(num_employees__gte=n)
def needs_fewer_chairs_than(self, n=5):
return self.with_chairs_needed().filter(chairs_needed__lt=n)
def with_chairs_needed(self):
return self.annotate(chairs_needed=F('num_employees') - F('num_chairs'))
class Company(models.Model):
# ...
objects = CompanyQuerySet.as_manager()
Company.objects.needs_fewer_chairs_than(4).many_employees()
See https://docs.djangoproject.com/en/1.9/topics/db/managers/ for more.
Note that I am going off the documentation and have not tested the above.
Looks like using F() with annotations will be my solution to this.
It's not going to filter by #property, since F talks to the databse before objects are brought into python. But still putting it here as an answer since my reason for wanting filter by property was really wanting to filter objects by the result of simple arithmetic on two different fields.
so, something along the lines of:
companies = Company.objects\
.annotate(chairs_needed=F('num_employees') - F('num_chairs'))\
.filter(chairs_needed__lt=4)
rather than defining the property to be:
#property
def chairs_needed(self):
return self.num_employees - self.num_chairs
then doing a list comprehension across all objects.
I had the same problem, and I developed this simple solution:
objects = [
my_object
for my_object in MyModel.objects.all()
if my_object.myProperty == [...]
]
This is not a performatic solution, it shouldn't be done in tables that contains a large amount of data. This is great for a simple solution or for a personal small project.
PLEASE someone correct me, but I guess I have found a solution, at least for my own case.
I want to work on all those elements whose properties are exactly equal to ... whatever.
But I have several models, and this routine should work for all models. And it does:
def selectByProperties(modelType, specify):
clause = "SELECT * from %s" % modelType._meta.db_table
if len(specify) > 0:
clause += " WHERE "
for field, eqvalue in specify.items():
clause += "%s = '%s' AND " % (field, eqvalue)
clause = clause [:-5] # remove last AND
print clause
return modelType.objects.raw(clause)
With this universal subroutine, I can select all those elements which exactly equal my dictionary of 'specify' (propertyname,propertyvalue) combinations.
The first parameter takes a (models.Model),
the second a dictionary like:
{"property1" : "77" , "property2" : "12"}
And it creates an SQL statement like
SELECT * from appname_modelname WHERE property1 = '77' AND property2 = '12'
and returns a QuerySet on those elements.
This is a test function:
from myApp.models import myModel
def testSelectByProperties ():
specify = {"property1" : "77" , "property2" : "12"}
subset = selectByProperties(myModel, specify)
nameField = "property0"
## checking if that is what I expected:
for i in subset:
print i.__dict__[nameField],
for j in specify.keys():
print i.__dict__[j],
print
And? What do you think?
i know it is an old question, but for the sake of those jumping here i think it is useful to read the question below and the relative answer:
How to customize admin filter in Django 1.4
It may also be possible to use queryset annotations that duplicate the property get/set-logic, as suggested e.g. by #rattray and #thegrimmscientist, in conjunction with the property. This could yield something that works both on the Python level and on the database level.
Not sure about the drawbacks, however: see this SO question for an example.
how to remove duplicate after merge two different models like this
import intertools
events_list = list(itertools.chain(events_list, speakers_list))
I am getting duplicate values in Django REst serializer
You can use union(); here is doc about union()
qs1.union(qs2)
# no duplicates
By default, union()only gives you distinct values. If you want to allow duplicates, you use
qs1.union(qs2, all=True)
# allow duplicates
A set enforces uniqueness
events_list = set(itertools.chain(events_list, speakers_list))
a = queryset1
b = queryset2
qs = a | b #removes duplicates
distinct() is doing this job :
queryset_final = queryset_with_duplicate.distinct()
Cf doc
By default, a QuerySet will not eliminate duplicate rows. In practice,
this is rarely a problem, because simple queries such as
Blog.objects.all() don’t introduce the possibility of duplicate result
rows. However, if your query spans multiple tables, it’s possible to
get duplicate results when a QuerySet is evaluated. That’s when you’d
use distinct().
I'm trying to query a database based on user input tags. The number of tags can be from 0-5, so I need to create the query dynamically.
So I have a tag list, tag_list, and I want to query the database:
design_list = Design.objects.filter(Q(tags__tag__contains = "tag1") and Q(tags__tag__contains = "tag2") and etc. etc. )
How can I create this feature?
You'll want to loop through the tag_list and apply a filter for each one.
tag_list = ['tag1', 'tag2', 'tag3']
base_qs = Design.objects.all()
for t in tag_list:
base_qs = base_qs.filter(tags__tag__contains=t)
This will give you results matching all tags, as your example indicated with and. If in fact you needed or instead, you will probably need Q objects.
Edit: I think I have what you're looking for now.
tags = ['tag1', 'tag2', 'tag3']
q_objects = Q() # Create an empty Q object to start with
for t in tags:
q_objects |= Q(tags__tag__contains=t) # 'or' the Q objects together
designs = Design.objects.filter(q_objects)
I tested this and it seems to work really well.
Edit 2: Credit to kezabelle in #django on Freenode for the initial idea.
You can use this way:
my_dict = {'field_1': 1, 'field_2': 2, 'field_3': 3, ...} # Your dict with fields
or_condition = Q()
for key, value in my_dict.items():
or_condition.add(Q(**{key: value}), Q.OR)
query_set = MyModel.objects.filter(or_condition)
By this way you can use dynamically generated field names.
Also you can use Q.AND for AND condition.
Just prepare a tag list first then, query like this:
tags = ['tag1', 'tag2',...]
design_list = Design.objects.filter(tags__tag__contains__in = tags)
You may need to add AND and OR conditions
query = (Q(fild1='ENABLE'))
# Filter by list
query.add(Q(fild2__in=[p.code for p in Objects.field.all()]), Q.AND)
# filter OR
q_objects = Q(field3='9999999')
for x in myList:
q_objects.add(Q(field3=x.code), Q.OR)
query.add(q_objects, Q.AND)
Use reduce :
from functools import reduce
design_list = Design.objects.filter(reduce(lambda q1,q2: q1 & q2,
[Q(tags__tag__contains=t)
for t in tag_list]))
I'd like to update a table with Django - something like this in raw SQL:
update tbl_name set name = 'foo' where name = 'bar'
My first result is something like this - but that's nasty, isn't it?
list = ModelClass.objects.filter(name = 'bar')
for obj in list:
obj.name = 'foo'
obj.save()
Is there a more elegant way?
Update:
Django 2.2 version now has a bulk_update.
Old answer:
Refer to the following django documentation section
Updating multiple objects at once
In short you should be able to use:
ModelClass.objects.filter(name='bar').update(name="foo")
You can also use F objects to do things like incrementing rows:
from django.db.models import F
Entry.objects.all().update(n_pingbacks=F('n_pingbacks') + 1)
See the documentation.
However, note that:
This won't use ModelClass.save method (so if you have some logic inside it won't be triggered).
No django signals will be emitted.
You can't perform an .update() on a sliced QuerySet, it must be on an original QuerySet so you'll need to lean on the .filter() and .exclude() methods.
Consider using django-bulk-update found here on GitHub.
Install: pip install django-bulk-update
Implement: (code taken directly from projects ReadMe file)
from bulk_update.helper import bulk_update
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
r = random.randrange(4)
person.name = random_names[r]
bulk_update(people) # updates all columns using the default db
Update: As Marc points out in the comments this is not suitable for updating thousands of rows at once. Though it is suitable for smaller batches 10's to 100's. The size of the batch that is right for you depends on your CPU and query complexity. This tool is more like a wheel barrow than a dump truck.
Django 2.2 version now has a bulk_update method (release notes).
https://docs.djangoproject.com/en/stable/ref/models/querysets/#bulk-update
Example:
# get a pk: record dictionary of existing records
updates = YourModel.objects.filter(...).in_bulk()
....
# do something with the updates dict
....
if hasattr(YourModel.objects, 'bulk_update') and updates:
# Use the new method
YourModel.objects.bulk_update(updates.values(), [list the fields to update], batch_size=100)
else:
# The old & slow way
with transaction.atomic():
for obj in updates.values():
obj.save(update_fields=[list the fields to update])
If you want to set the same value on a collection of rows, you can use the update() method combined with any query term to update all rows in one query:
some_list = ModelClass.objects.filter(some condition).values('id')
ModelClass.objects.filter(pk__in=some_list).update(foo=bar)
If you want to update a collection of rows with different values depending on some condition, you can in best case batch the updates according to values. Let's say you have 1000 rows where you want to set a column to one of X values, then you could prepare the batches beforehand and then only run X update-queries (each essentially having the form of the first example above) + the initial SELECT-query.
If every row requires a unique value there is no way to avoid one query per update. Perhaps look into other architectures like CQRS/Event sourcing if you need performance in this latter case.
Here is a useful content which i found in internet regarding the above question
https://www.sankalpjonna.com/learn-django/running-a-bulk-update-with-django
The inefficient way
model_qs= ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
obj.name = 'foo'
obj.save()
The efficient way
ModelClass.objects.filter(name = 'bar').update(name="foo") # for single value 'foo' or add loop
Using bulk_update
update_list = []
model_qs= ModelClass.objects.filter(name = 'bar')
for model_obj in model_qs:
model_obj.name = "foo" # Or what ever the value is for simplicty im providing foo only
update_list.append(model_obj)
ModelClass.objects.bulk_update(update_list,['name'])
Using an atomic transaction
from django.db import transaction
with transaction.atomic():
model_qs = ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
ModelClass.objects.filter(name = 'bar').update(name="foo")
Any Up Votes ? Thanks in advance : Thank you for keep an attention ;)
To update with same value we can simply use this
ModelClass.objects.filter(name = 'bar').update(name='foo')
To update with different values
ob_list = ModelClass.objects.filter(name = 'bar')
obj_to_be_update = []
for obj in obj_list:
obj.name = "Dear "+obj.name
obj_to_be_update.append(obj)
ModelClass.objects.bulk_update(obj_to_be_update, ['name'], batch_size=1000)
It won't trigger save signal every time instead we keep all the objects to be updated on the list and trigger update signal at once.
IT returns number of objects are updated in table.
update_counts = ModelClass.objects.filter(name='bar').update(name="foo")
You can refer this link to get more information on bulk update and create.
Bulk update and Create
I have the following:
answers = Answer.objects.filter(id__in=[answer.id for answer in answer_set.answers.all()])
then later:
for i in range(len(answers)):
# iterate through all existing QuestionAnswer objects
for existing_question_answer in existing_question_answers:
# if an answer is already associated, remove it from the
# list of answers to save
if answers[i].id == existing_question_answer.answer.id:
answers.remove(answers[i]) # doesn't work
existing_question_answers.remove(existing_question_answer)
I get an error:
'QuerySet' object has no attribute 'remove'
I've tried all sorts to convert the QuerySet to a standard set or list. Nothing works.
How can I remove an item from the QuerySet so it doesn't delete it from the database, and doesn't return a new QuerySet (since it's in a loop that won't work)?
Why not just call list() on the Queryset?
answers_list = list(answers)
This will also evaluate the QuerySet/run the query. You can then remove/add from that list.
You could do this:
import itertools
ids = set(existing_answer.answer.id for existing_answer in existing_question_answers)
answers = itertools.ifilter(lambda x: x.id not in ids, answers)
Read when QuerySets are evaluated and note that it is not good to load the whole result into memory (e.g. via list()).
Reference: itertools.ifilter
Update with regard to the comment:
There are various ways to do this. One (which is probably not the best one in terms of memory and time) is to do exactly the same :
answer_ids = set(answer.id for answer in answers)
existing_question_answers = filter(lambda x: x.answer.id not in answers_id, existing_question_answers)
It is a little hard to follow what you are really trying to do. Your first statement looks like you may be fetching the same exact QuerySet of Answer objects twice. First via answer_set.answers.all() and then again via .filter(id__in=...). Double check in the shell and see if this will give you the list of answers you are looking for:
answers = answer_set.answers.all()
Once you have that cleaned up so it is a little easier for you (and others working on the code) to read you might want to look into .exclude() and the __in field lookup.
existing_question_answers = QuestionAnswer.objects.filter(...)
new_answers = answers.exclude(question_answer__in=existing_question_answers)
The above lookup might not sync up with your model definitions but it will probably get you close enough to finish the job yourself.
If you still need to get a list of id values then you want to play with .values_list(). In your case you will probably want to add the optional flat=True.
answers.values_list('id', flat=True)
By the use of slice operator with step parameter which would cause evaluation of the queryset and create a list.
list_of_answers = answers[::1]
or initially you could have done:
answers = Answer.objects.filter(id__in=[answer.id for answer in
answer_set.answers.all()])[::1]
You can directly convert using the list keyword.
For example:
obj=emp.objects.all()
list1=list(obj)
Using the above code you can directly convert a query set result into a list.
Here list is keyword and obj is result of query set and list1 is variable in that variable we are storing the converted result which in list.
Try this values_list('column_name', flat=True).
answers = Answer.objects.filter(id__in=[answer.id for answer in answer_set.answers.all()]).values_list('column_name', flat=True)
It will return you a list with specified column values
Use python list() function
list(). Force evaluation of a QuerySet by calling list() on it. For
example:
answers = list(answer_set.answers.all())
Why not just call
.values('reqColumn1','reqColumn2') or .values_list('reqColumn1','reqColumn2') on the queryset?
answers_list = models.objects.values('reqColumn1','reqColumn2')
result = [{'reqColumn1':value1,'reqColumn2':value2}]
OR
answers_list = models.objects.values_list('reqColumn1','reqColumn2')
result = [(value1,value2)]
You can able to do all the operation on this QuerySet, which you do for list .
def querySet_to_list(qs):
"""
this will return python list<dict>
"""
return [dict(q) for q in qs]
def get_answer_by_something(request):
ss = Answer.objects.filter(something).values()
querySet_to_list(ss) # python list return.(json-able)
this code convert django queryset to python list
instead of remove() you can use exclude() function to remove an object from the queryset.
it's syntax is similar to filter()
eg : -
qs = qs.exclude(id= 1)
in above code it removes all objects from qs with id '1'
additional info :-
filter() used to select specific objects but exclude() used to remove