I have a list of objects and a model includes ManyToManyField of this object.
I'd like to get objects who have the same list of objects in this field.
that's means __in won't work because he will do OR between the objects in the list and not AND.
I was trying using the AND Q Lookup, but it didn't work (after checking the .query text, it seems that he doing the AND inside the field object, instead of the object itself, so obviously the field object id don't have two different ids..)
here's the interceptor I played with
results_query_set[0]
<Results: Results object (2)>
results_query_set[0].users.all()
<QuerySet [<User: test_user>, <User: test_user_2>]>
users
[<User: test_user>, <User: test_user_2>]
users_q
<Q: (AND: ('users', <User: test_user>), ('users', <User: test_user_2>))>
results_query_set.filter(users=users[0])
<QuerySet [<Results: Results object (2)>]>
results_query_set.filter(users=users[1])
<QuerySet [<Results: Results object (2)>]>
results_query_set.filter(users_q)
<QuerySet []>
results_query_set.filter(Q(users=users[0]) & Q(users=users[1]))
<QuerySet []>
and the result results_query_set.filter(users_q).query.__str__() reproduce is
'SELECT "results_table"."id", "results_table"."date", "results_table"."lookup", "results_table"."value" FROM "results_table" INNER JOIN "results_table_users" ON ("results_table"."id" = "results_table_users"."widgetresults_id") WHERE ("results_table_users"."user_id" = 1 AND "results_table_users"."user_id" = 2)
I can chain .filter for each user, but of course, I'd like to make one query instead of queries by the numbers of my input.
You need to JOIN the target table (User in your case) multiple times (for every single user) in order to build such a query.
The way to do this in Django is by calling .filter multiple times.
users = [user1, user2] # a list of users your are interested to filter on
initial_qs = Results.objects.all() # or whatever your results_query_set is
result_qs = reduce(lambda qs, user: qs.filter(users=user.id), users, initial_qs)
# At this point you will have results containing user1 and user2
# but this also includes results with more users (e.g. users 1, 2 and 3)
# if you want to exclude those, you need to filter by the total users count too
result_qs = result_qs.annotate(cnt=models.Count('users')).filter(cnt=len(users))
I have not tried this, but I think you might be able to use postgresql's array_agg function. The Django implementation is here.
from django.contrib.postgres.aggregates import ArrayAgg
ideal_user_list = [] # some list.
# Or fetch it directly from the db using the below query and `get`
Results.objects.annotate(
related_user_array=ArrayAgg('users__id', ordering='users__id')
).filter(related_user_array=ideal_user_list)
Related
I have a queryset with the users:
<QuerySet [<User: usr0>, <User: usr1>]>
I know that I can verify if the user belongs to the group like so:
In [18]: usr1 = queryset[1]
In [19]: usr1.groups.filter(name='Partners').exists()
Out[19]: True
How can I filter out all the users from the queryset which do not belong to the custom group(s)?
You can filter the queryset, with:
queryset.filter(groups__name='Partners')
Here we construct a queryset that will only contain Users that belong to a group with the name 'Partners'.
I got a ValueError while trying to add model instances with a many-to-many relationship.
ValueError: "(Idea: hey)" needs to have a value for field "id" before this many-to-many relationship can be used.
A lot of responses were given here, but none was helpful.My (idea) solution was to "manually" input the "id" values.
>>> import django
>>> django.setup()
>>> from myapp1.models import Category, Idea
# Notice that I manually add an "id"
>>> id2=Idea.objects.create(
... title_en='tre',
... subtitle_en='ca',
... description_en='mata',
... id=5,
... is_original=True,
... )
>>> id2.save()
>>> cat22=Category(title_en='yo')
>>> cat22.save()
>>> id2.categories.add(cat22)
>>> Idea.objects.all()
<QuerySet [<Idea: tre>]>
>>> exit()
How do i command django to auto-add the "id" field?
Note: I tried adding autoField but failed, thanks
#python_2_unicode_compatible
class Idea(UrlMixin, CreationModificationDateMixin, MetaTagsMixin):
id = models.IntegerField(primary_key=True,)
title = MultilingualCharField(_("Title"), max_length=200,)
subtitle = MultilingualCharField(_("Subtitle"), max_length=200, blank=True,)
description = MultilingualTextField(_("Description"), blank=True,)
is_original = models.BooleanField(_("Original"), default=False,)
categories = models.ManyToManyField(Category,
You're confusing two things here:
With many-to-many relationships, when connecting two objects, both objects must already be saved to the database (have a primary key), because under the hoods, Django creates a third object that points at the two objects to connect them. It can only do that if both have an id, assuming id is the primary key.
When creating an object, you don't have to explicitly set the id (actually you shouldn't). By default, a django Model will have id set as an auto field and as a primary key (you can override that by specifying your own pk, but in general there's no need to). The id is automatically created when the model is saved the first time.
You saw the error because probably one of the objects (idea or category) wasn't saved to the database before you connected them. In your code sample, you don't have to pass id=5, it will work without it, because you save id2 and category before connecting them.
I have the following case. I have a form with 3 fields which are submitted with POST method. Then the fields are captured and a search using Q is made on the database:
query = Model.objects.filter( Q(field1=field1) & Q(field2=field2) & Q(field3=field3))
The problem is that I would like to dynamically use the fields that are filled not the empty ones. That means that the query will contain one or two or three criteria depending on the user.
I have managed to perform the search I describe with nested if but considering adding extra fields it gets bigger and bigger.
thanks
You can write generic function to generate a query based on fields passed-in.
def generate_query(**kwargs):
query = Q()
for database_field, value in kwargs.items():
query_dict = {database_field:value}
if value:
query &= Q(**query_dict)
return query
And use it:
data = {"name":"john", "age__gte":25} # your post data
query = generate_query(**data)
objects = SomeModel.objects.filter(query)
# generally: SomeModel.objects.filter(request.POST)
Consider you have query field like below:
field = {'field1': 'value1', 'field2': 'value2', 'field3': None}
You can filter to remove null field as below:
non_empty_field = dict(filter(lambda x: x[1], field.items())) # output: {'field1': 'value1', 'field2': 'value2'}
For performing and query you can write
Model.objects.filter(**non_empty_field) # equivalent to Model.objects.filter(field1='value1', field2='value2')
to perform and query you don't need Q() object
For performing or query you can write:
Model.objects.filter(eval('|'.join(['(Q({}="{}"))'.format(i,j) for i,j in non_empty_field.items()])))
according to current field example above query will be equivalent to:
Model.objects.filter(Q(field1=value1) | Q(field2=value2))
in a view I am constructing I need to consult multiple databases. What I want to do is use the results of on query_set to search another db table.
I have functioning mydb1_query_set, what I need now is something like this:
for row in mydb1_query_set:
mydb2_query_set = Mytable.objects.filter(id=row.id)
So that I keep adding to the initially empty mydb2_query_set as I iterate. I realize that there is no QuerySet.append, so how do I achieve what I want? Any help much appreciated...
Use a list instead of a queryset, and then you can append or extend as you wish.
mydb2_query = []
for row in mydb1_query_set:
mydb2_query.extend(list(Mytable.objects.filter(id=row.id)))
qs1= <QuerySet [<User: 1#gmail.com>, <User: 2#gmail.com>]>
qs2= <QuerySet [<User: 3#gmail.com>, <User: 4#gmail.com>]>
qs3 = qs1 | qs2
res:
qs3 = <QuerySet [<User: 1#gmail.com>, <User: 2#gmail.com>, <User:3#gmail.com>, <User: 4#gmail.com>]>
EDIT: Link to documentation:
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#operators-that-return-new-querysets
Using python's built in itertools module worked best for me.
from itertools import chain
qs1 = Mytable.objects.filter(id=2)
qs2 = Mytable.objects.filter(id=3)
all_queries = chain(qs1, qs2)
Here's the docs, in case you'd like some reference materials: https://docs.python.org/3/library/itertools.html#itertools.chain
Django's Managers object provide a default method called .get_queryset() which allow you to concatenate extra queries after it:
queryset = MyModel.objects.get_queryset()
if username:
queryset = queryset.filter(username=username)
if country:
queryset = queryset.filter(country=country)
You could easily using .query attribute getting the SQL, for example:
>>> print(queryset.query)
SELECT `myapp_mymodel`.`id`, `myapp_mymodel`.`username`, `myapp_mymodel`.`country` FROM `myapp_mymodel` WHERE `myapp_mymodel`.`country` = foo
I would like to save a queryset criteria to the DB for reuse.
So, if I have a queryset like:
Client.objects.filter(state='AL')
# I'm simplifying the problem for readability. In reality I could have
# a very complex queryset, with multiple filters, excludes and even Q() objects.
I would like to save to the DB not the results of the queryset (i.e. the individual client records that have a state field matching 'AL'); but the queryset itself (i.e. the criteria used in filtering the Client model).
The ultimate goal is to have a "saved filter" that can be read from the DB and used by multiple django applications.
At first I thought I could serialize the queryset and save that. But serializing a queryset actually executes the query - and then I end up with a static list of clients in Alabama at the time of serialization. I want the list to be dynamic (i.e. each time I read the queryset from the DB it should execute and retrieve the most current list of clients in Alabama).
Edit: Alternatively, is it possible to obtain a list of filters applied to a queryset?
Something like:
qs = Client.objects.filter(state='AL')
filters = qs.getFilters()
print filters
{ 'state': 'AL' }
You can do as jcd says, storing the sql.
You can also store the conditions.
In [44]: q=Q( Q(content_type__model="User") | Q(content_type__model="Group"),content_type__app_label="auth")
In [45]: c={'name__startswith':'Can add'}
In [46]: Permission.objects.filter(q).filter(**c)
Out[46]: [<Permission: auth | group | Can add group>, <Permission: auth | user | Can add user>]
In [48]: q2=Q( Q(content_type__model="User") | Q(content_type__model="Group"),content_type__app_label="auth", name__startswith='Can add')
In [49]: Permission.objects.filter(q2)
Out[49]: [<Permission: auth | group | Can add group>, <Permission: auth | user | Can add user>]
In that example you see that the conditions are the objects c and q (although they can be joined in one object, q2). You can then serialize these objects and store them on the database as strings.
--edit--
If you need to have all the conditions on a single database record, you can store them in a dictionary
{'filter_conditions': (cond_1, cond_2, cond_3), 'exclude_conditions': (cond_4, cond_5)}
and then serialize the dictionary.
You can store the sql generated by the query using the queryset's _as_sql() method. The method takes a database connection as an argument, so you'd do:
from app.models import MyModel
from django.db import connection
qs = MyModel.filter(pk__gt=56, published_date__lt=datetime.now())
store_query(qs._as_sql(connection))
You can use http://github.com/denz/django-stored-queryset for that
You can pickle the Query object (not the QuerySet):
>>> import pickle
>>> query = pickle.loads(s) # Assuming 's' is the pickled string.
>>> qs = MyModel.objects.all()
>>> qs.query = query # Restore the original 'query'.
Docs: https://docs.djangoproject.com/en/dev/ref/models/querysets/#pickling-querysets
But: You can’t share pickles between versions
you can create your own model to store your queries.
First field can contains fk to ContentTypes
Second field can be just text field with your query etc.
And after that you can use Q object to set queryset for your model.
The current answer was unclear to me as I don't have much experience with pickle. In 2022, I've found that turning a dict into JSON worked well. I'll show you what I did below. I believe pickling still works, so at the end I will show some more thoughts there.
models.py - example database structure
class Transaction(models.Model):
id = models.CharField(max_length=24, primary_key=True)
date = models.DateField(null=False)
amount = models.IntegerField(null=False)
info = models.CharField()
account = models.ForiegnKey(Account, on_delete=models.SET_NULL, null=True)
category = models.ForeignKey(Category, on_delete=models.SET_NULL, null=True, blank=False, default=None)
class Account(models.Model):
name = models.CharField()
email = models.EmailField()
class Category(models.Model):
name = models.CharField(unique=True)
class Rule(models.Model):
category = models.ForeignKey(Category, on_delete=models.SET_NULL, blank=False, null=True, default=None)
criteria = models.JSONField(default=dict) # this will hold our query
My models store financial transactions, the category the transaction fits into (e.g., salaried income, 1099 income, office expenses, labor expenses, etc...), and a rule to save a query to automatically categorize future transactions without having to remember the query every year when doing taxes.
I know, for example, that all my transactions with my consulting clients should be marked as 1099 income. So I want to create a rule for clients that will grab each monthly transaction and mark it as 1099 income.
Making the query the old-fashioned way
>>> from transactions.models import Category, Rule, Transaction
>>>
>>> client1_transactions = Transaction.objects.filter(account__name="Client One")
<QuerySet [<Transaction: Transaction object (1111111)>, <Transaction: Transaction object (1111112)>, <Transaction: Transaction object (1111113)...]>
>>> client1_transactions.count()
12
Twelve transactions, one for each month. Beautiful.
But how do we save this to the database?
Save query to database in JSONField
We now have Django 4.0 and a bunch of support for JSONField.
I've been able to grab the filtering values out of a form POST request, then add them in view logic.
urls.py
from transactions import views
app_name = "transactions"
urlpatterns = [
path("categorize", views.categorize, name="categorize"),
path("", views.list, name="list"),
]
transactions/list.html
<form action="{% url 'transactions:categorize' %}" method="POST">
{% csrf_token %}
<label for="info">Info field contains...</label>
<input id="info" type="text" name="info">
<label for="account">Account name contains...</label>
<input id="account" type="text" name="account">
<label for="category">New category should be...</label>
<input id="category" type="text" name="category">
<button type="submit">Make a Rule</button>
</form>
views.py
def categorize(request):
# get POST data from our form
info = request.POST.get("info", "")
account = request.POST.get("account", "")
category = request.POST.get("category", "")
# set up query
query = {}
if info:
query["info__icontains"] = info
if account:
query["account__name__icontains"] = account
# update the database
category_obj, _ = Category.objects.get_or_create(name=category)
transactions = Transaction.objects.filter(**query).order_by("-date")
Rule.objects.get_or_create(category=category_obj, criteria=query)
transactions.update(category=category_obj)
# render the template
return render(
request,
"transactions/list.html",
{
"transactions": transactions.select_related("account"),
},
)
That's pretty much it!
My example here is a little contrived, so please forgive any errors.
How to do it with pickle
I actually lied before. I have a little experience with pickle and I do like it, but I am not sure on how to save it to the database. My guess is that you'd then save the pickled string to a BinaryField.
Perhaps something like this:
>>> # imports
>>> import pickle # standard library
>>> from transactions.models import Category, Rule, Transaction # my own stuff
>>>
>>> # create the query
>>> qs_to_save = Transaction.objects.filter(account__name="Client 1")
>>> qs_to_save.count()
12
>>>
>>> # create the pickle
>>> saved_pickle = pickle.dumps(qs_to_save.query)
>>> type(saved_pickle)
<class 'bytes'>
>>>
>>> # save to database
>>> # make sure `criteria = models.BinaryField()` above in models.py
>>> # I'm unsure about this
>>> test_category, _ = Category.objects.get_or_create(name="Test Category")
>>> test_rule = Rule.objects.create(category=test_category, criteria=saved_pickle)
>>>
>>> # remake queryset at a later date
>>> new_qs = Transaction.objects.all()
>>> new_qs.query = pickle.loads(test_rule.criteria)
>>> new_qs.count()
12
Going even further beyond
I found a way to make this all work with my htmx live search, allowing me to see the results of my query on the front end of my site before saving.
This answer is already too long, so here's a link to a post if you care about that: Saving a Django Query to the Database.