I'm running metrics on user data and want exclude users that have bogus emails like '#example.com' or '#test.com'.
I tried
emails_to_exclude = ['#example.com', '#test.com', '#mailinator.com' ....]
Users.objects.exclude(email__endswith__in=emails_to_exclude)
Unfortunately this doesn't work. Looks like endswith and in don't play nice with each other. Any ideas?
Simply loop over the QuerySet, as QuerySets are lazy.
emails_to_exclude = ['#example.com', '#test.com', '#mailinator.com' ....]
users = Users.objects
for exclude_email in emails_to_exclude:
users = users.exclude(email__endswith=exclude_email)
users = users.all()
You can also do this with regular expressions in single query.
emails_to_exclude = ['#example.com', '#test.com', '#mailinator.com' ....]
User.objects.exclude(email__regex = "|".join(emails_to_exclude))
I don't know the efficiency of this query.
This will not work for SQLite, as it has no built in regular expression support.
You can probably loop over the emails and build up a Q Object. Actually, you can probably do a 1-liner if you're clever.
User.objects.exclude(bitwise_or_function[Q(email__endswith=e) for e in emails_to_exclude])
Something like that. I don't remember the function to bitwise-OR a whole list together, my Python's rusty.
This should works with the latest version of Python and Django. The reduce function is a good friend.
from functools import reduce
from operator import or_
from django.db.models import Q
emails_to_exclude = ['#example.com', '#test.com', '#mailinator.com' ....]
users = ( Users.objects
.exclude( reduce( or_, (
Q(( "email__endswith", k ))
for k in emails_to_exclude
) ) )
)
One more way to achieve this:
from django.contrib.auth.models import User
from django.db.models import Q
emails_to_exclude = ['#example.com', '#test.com', '#mailinator.com']
users = User.objects.all()
filters = Q()
for ending in emails_to_exclude:
filters |= Q(email__endswith=ending)
filtered_users = users.filter(~filters)
I changed the exclusion input to make it a set and to not have the "#". Otherwise, this should do what you want.
>>> emails = ['foo#example.com', 'spam#stackoverflow.com', 'bad#test.com']
>>> excludes = {'example.com', 'test.com', 'mailinator.com'}
>>> [email for email in emails if email.split('#')[-1] not in excludes]
['spam#stackoverflow.com']
Related
I like to perform a Django query like the one below:
query = Dbmodel.objects.all().annotate(result=F('fieldname1') + F('fieldname2') + F('fieldname4'))
But instead of using the fieldnames directly, I would like to use a list with the field names:
fields = ['fieldname1', 'fieldname2', 'fieldname4']
Depending on user interaction the number of fieldnames in the list can variate. Is there a way to do it?
As this is my first question on StackOverflow, please let me know, if my question is unclear or I could improve my question.
You can do this by using operator.add to combine the multiple F expressions into a single expression, and pass that into the annotation.
It would look like this:
import operator
from django.db.models import F
# any amount of fields
fields = ["fieldname1", "fieldname2", "fieldname3"]
# combine into a single sum
combined_expression = reduce(operator.add, (F(x) for x in fields))
# the new query
query = Dbmodel.objects.all().annotate(result=combined_expression)
I'm trying to query a database based on user input tags. The number of tags can be from 0-5, so I need to create the query dynamically.
So I have a tag list, tag_list, and I want to query the database:
design_list = Design.objects.filter(Q(tags__tag__contains = "tag1") and Q(tags__tag__contains = "tag2") and etc. etc. )
How can I create this feature?
You'll want to loop through the tag_list and apply a filter for each one.
tag_list = ['tag1', 'tag2', 'tag3']
base_qs = Design.objects.all()
for t in tag_list:
base_qs = base_qs.filter(tags__tag__contains=t)
This will give you results matching all tags, as your example indicated with and. If in fact you needed or instead, you will probably need Q objects.
Edit: I think I have what you're looking for now.
tags = ['tag1', 'tag2', 'tag3']
q_objects = Q() # Create an empty Q object to start with
for t in tags:
q_objects |= Q(tags__tag__contains=t) # 'or' the Q objects together
designs = Design.objects.filter(q_objects)
I tested this and it seems to work really well.
Edit 2: Credit to kezabelle in #django on Freenode for the initial idea.
You can use this way:
my_dict = {'field_1': 1, 'field_2': 2, 'field_3': 3, ...} # Your dict with fields
or_condition = Q()
for key, value in my_dict.items():
or_condition.add(Q(**{key: value}), Q.OR)
query_set = MyModel.objects.filter(or_condition)
By this way you can use dynamically generated field names.
Also you can use Q.AND for AND condition.
Just prepare a tag list first then, query like this:
tags = ['tag1', 'tag2',...]
design_list = Design.objects.filter(tags__tag__contains__in = tags)
You may need to add AND and OR conditions
query = (Q(fild1='ENABLE'))
# Filter by list
query.add(Q(fild2__in=[p.code for p in Objects.field.all()]), Q.AND)
# filter OR
q_objects = Q(field3='9999999')
for x in myList:
q_objects.add(Q(field3=x.code), Q.OR)
query.add(q_objects, Q.AND)
Use reduce :
from functools import reduce
design_list = Design.objects.filter(reduce(lambda q1,q2: q1 & q2,
[Q(tags__tag__contains=t)
for t in tag_list]))
i have a django app that retrieve all subjects from a single table of users. i've also implemented an input search form,
this is the query performed:
all_soggs = Entity.objects.filter(lastname__istartswith=request.GET['query_term']).order_by('lastname')
if(all_soggs.count()==0):
all_soggs = Entity.objects.filter(firstname__istartswith=request.GET['query_term']).order_by('firstname')
as you can see the query first search for matching items by lastname, and then by firstname. this works until i insert the complete name 'firstaname lastname' or 'lastname firstname', in this case there's no results. how can i modify the query to make a better search?
thanks - luke
Copy/paste from: https://stackoverflow.com/a/17361729/1297812
from django.db.models import Q
def find_user_by_name(query_name):
qs = User.objects.all()
for term in query_name.split():
qs = qs.filter( Q(first_name__icontains = term) | Q(last_name__icontains = term))
return qs
You need Q objects and you also need to split your query into separate terms (since no first name will match the full string "Firstname Lastname").
Here's an idea to match any first or last name starting with either "Firstname" or "Lastname" in the search "Firstname Lastname".
This is a generic search - adjust the query to suit your specific needs!
Edit: oops, I really don't like using reduce since it looks confusing, but these need to be ORed together and we can't do a more verbose version because the number of terms is unknown.
import operator
from django.db.models import Q
search_args = []
for term in request.GET['query_term'].split():
for query in ('first_name__istartswith', 'last_name__istartswith'):
search_args.append(Q(**{query: term}))
all_soggs = Entity.objects.filter(reduce(operator.or_, search_args))
To clarify how to use Q objects, given the search "Firstname Lastname" the previous query is equal to:
Entity.objects.filter(
Q(first_name__istartswith="Firstname") | Q(last_name__istartswith="Firstname") |
Q(first_name__istartswith="Lastname") | Q(last_name__istartswith="Lastname")
)
This is a fairly old question but I just ran into the same problem and I thought I would share a more elegant solution.
from django.db.models import Value as V
from django.db.models.functions import Concat
from ..models import User
def find_user_by_name(search_str) -> QuerySey[User]:
q = User.objects.annotate(full_name=Concat('first_name', V(' '), 'last_name'))
q = q.filter(full_name__icontains=search_str)
return q
This is the only solution that I have found which lets gives the behaviour I wanted, IE seaching with a full name string with a space ("John Doe") and with a partial string ("John Do").
Similar question:Querying full name in Django
query = request.GET.get('query')
entities = []
try:
firstname = query.split(' ')[0]
lastname = query.split(' ')[1]
entities += Entity.objects.filter(firstname__icontains=firstname,lastname__icontains=lastname)
entities += Entity.objects.filter(firstname__icontains=lastname,lastname__icontains=firstname)
entities = set(entities)
I have a list of names that I want to match case insensitive, is there a way to do it without using a loop like below?
a = ['name1', 'name2', 'name3']
result = any([Name.objects.filter(name__iexact=name) for name in a])
Unfortunatley, there are no __iin field lookup. But there is a iregex that might be useful, like so:
result = Name.objects.filter(name__iregex=r'(name1|name2|name3)')
or even:
a = ['name1', 'name2', 'name3']
result = Name.objects.filter(name__iregex=r'(' + '|'.join(a) + ')')
Note that if a can contain characters that are special in a regex, you need to escape them properly.
NEWS: In Django 1.7+ it is possible to create your own lookups, so you can actually use filter(name__iin=['name1', 'name2', 'name3']) after proper initialization. See documentation reference for details.
In Postgresql you could try creating a case insensitive index as described here:
https://stackoverflow.com/a/4124225/110274
Then run a query:
from django.db.models import Q
name_filter = Q()
for name in names:
name_filter |= Q(name__iexact=name)
result = Name.objects.filter(name_filter)
Index search will run faster than the regex matching query.
Another way to this using django query functions and annotation
from django.db.models.functions import Lower
Record.objects.annotate(name_lower=Lower('name')).filter(name_lower__in=['two', 'one']
Adding onto what Rasmuj said, escape any user-input like so
import re
result = Name.objects.filter(name__iregex=r'(' + '|'.join([re.escape(n) for n in a]) + ')')
Keep in mind that at least in MySQL you have to set utf8_bin collation in your tables to actually make them case sensitive. Otherwise they are case preserving but case insensitive. E.g.
>>> models.Person.objects.filter(first__in=['John', 'Ringo'])
[<Person: John Lennon>, <Person: Ringo Starr>]
>>> models.Person.objects.filter(first__in=['joHn', 'RiNgO'])
[<Person: John Lennon>, <Person: Ringo Starr>]
So, if portability is not crucial and you use MySQL you may choose to ignore the issue altogether.
I am expanding Exgeny idea into an two liner.
import functools
Name.objects.filter(functools.reduce(lambda acc,x: acc | Q(name_iexact=x)), names, Q()))
After trying many methods, including annotate, which resulted in duplicate objects, I discovered transformers (https://docs.djangoproject.com/en/4.1/howto/custom-lookups/#a-transformer-example) which allow for a simple solution.
Add the following to models.py before model declarations:
class LowerCase(models.Transform):
lookup_name = "lower"
function = "LOWER"
models.CharField.register_lookup(LowerCase)
models.TextField.register_lookup(LowerCase)
You can now use the __lower transformer alongside any lookup, in this case: field__lower__in. You can also add bilateral = True to the transformer class for it to apply to both the field and the list items, which should be functionally equivalent to __iin.
Here is an example of custom User model classmethod to filter users by email case-insensitive
from django.db.models import Q
#classmethod
def get_users_by_email_query(cls, emails):
q = Q()
for email in [email.strip() for email in emails]:
q = q | Q(email__iexact=email)
return cls.objects.filter(q)
If this is a common use case for anyone, you can implement this by adapting the code from Django's In and IExact transformers.
Make sure the following code is imported before all model declarations:
from django.db.models import Field
from django.db.models.lookups import In
#Field.register_lookup
class IIn(In):
lookup_name = 'iin'
def process_lhs(self, *args, **kwargs):
sql, params = super().process_lhs(*args, **kwargs)
# Convert LHS to lowercase
sql = f'LOWER({sql})'
return sql, params
def process_rhs(self, qn, connection):
rhs, params = super().process_rhs(qn, connection)
# Convert RHS to lowercase
params = tuple(p.lower() for p in params)
return rhs, params
Example usage:
result = Name.objects.filter(name__iin=['name1', 'name2', 'name3'])
Is there a way to do a query and exclude a list of things, instead of calling exclude multiple times?
Based on your reply to Ned, it sounds like you just want to exclude a list of tags. So you could just use the in filter:
names_to_exclude = [o.name for o in objects_to_exclude]
Foo.objects.exclude(name__in=names_to_exclude)
Does that do what you want?
What's wrong with calling exclude multiple times? Queries are lazy, nothing happens until you try to pull data from it, so there's no downside to using .exclude() more than once.
You can try this also.
exclude_list = ['A', 'B', 'C']
qs = Foo.objects.exclude(items__in=exclude_list)
You can do it pretty easily with the Q object:
from django.db.models import Q
excludes = None
for tag in ignored_tags:
q = Q(tag=tag)
excludes = (excludes and (excludes | q)) or q # makes sure excludes is set properly
set_minus_excluded = Foo.objects.exclude(excludes)
You should also be able to do it dynamically with exclude():
qs = Foo.objects.all()
for tag in ignored_tags:
qs = qs.exclude(tag=tag)
To improve on Daniel Roseman's answer I think it would be better to get the values you need directly from the queryset instead of the for loop that could be expensive on large data sets i.e.
names_to_exclude = objects_to_exclude.values_list('name')
Foo.objects.exclude(name__in=names_to_exclude)