I made a custom manager that has to randomize my query:
class RandomManager(models.Manager):
def randomize(self):
count = self.aggregate(count=Count('id'))['count']
random_index = random.randint(0, count - 1)
return self.all()[random_index]
When I use the method defined in my manager in the first place, it's works ok:
>>> PostPages.random_objects.randomize()
>>> <PostPages: post 3>
I need to randomize the already filtered query. When I tried to use the manager and the method in chain I got an error:
PostPages.random_objects.filter(image_gallary__isnull=False).randomize()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/home/i159/workspace/shivaroot/shivablog/<ipython-input-9-98f654c77896> in <module>()
----> 1 PostPages.random_objects.filter(image_gallary__isnull=False).randomize()
AttributeError: 'QuerySet' object has no attribute 'randomize'
Result of filtering is not an instance of model class, but it's django.db.models.query.QuerySet, so that it does not have my manager and method, respectively.
Is there a way to use custom manager in chain query?
This is how you chain custom methods on custom manager ie: Post.objects.by_author(user=request.user).published()
from django.db.models.query import QuerySet
class PostMixin(object):
def by_author(self, user):
return self.filter(user=user)
def published(self):
return self.filter(published__lte=datetime.now())
class PostQuerySet(QuerySet, PostMixin):
pass
class PostManager(models.Manager, PostMixin):
def get_query_set(self):
return PostQuerySet(self.model, using=self._db)
Just a code example using the new as_manager() method (see update information from #zzart.
class MyQuerySet(models.query.QuerySet):
def randomize(self):
count = self.aggregate(count=Count('id'))['count']
random_index = random.randint(0, count - 1)
return self.all()[random_index]
class MyModel(models.Model):
.....
.....
objects = MyQuerySet.as_manager()
.....
.....
And then you will be able to use something like this in your code:
MyModel.objects.filter(age__gt=16).randomize()
As you can see, the new as_manager() is really neat:)
Looks like this snippet provides a solution to your situation: Custom managers with chainable filters.
Given that you have an existing models.Manager and you don't want to expose some of the manager method to a chainable queryset, you can use Manager.from_queryset(QuerySet)().
So, you could still place all your chainable queryset method inside the QuerySet and your manager method independently.
Example given in the official site.
Snippet from Django Docs
class BaseManager(models.Manager):
# Available only on Manager.
def manager_only_method(self):
return
class CustomQuerySet(models.QuerySet):
# Available on both Manager and QuerySet.
def manager_and_queryset_method(self):
return
# Available only on QuerySet.
def _private_method(self):
return
CustomManager = BaseManager.from_queryset(CustomQuerySet)
class MyModel(models.Model):
objects = CustomManager()
How about something like below which creates the custom QuerySet dynamically and allows us to 'transplant' our custom queries onto the returned QuerySet instance:
class OfferManager(models.Manager):
"""
Additional methods / constants to Offer's objects manager
"""
### Model (db table) wide constants - we put these and
### not in model definition to avoid circular imports.
### One can access these constants through like
<foo>.objects.STATUS_DISABLED or ImageManager.STATUS_DISABLED
STATUS_DISABLED = 0
...
STATUS_CHOICES = (
(STATUS_DISABLED, "Disabled"),
(STATUS_ENABLED, "Enabled"),
(STATUS_NEGOTIATED, "Negotiated"),
(STATUS_ARCHIVED, "Archived"),
)
...
# we keep status and filters naming a little different as
# it is not one-to-one mapping in all situations
QUERYSET_PUBLIC_KWARGS = {'status__gte': STATUS_ENABLED}
QUERYSET_ACTIVE_KWARGS = {'status': STATUS_ENABLED}
def get_query_set(self):
""" our customized method which transpalats manager methods
as per get_query_set.<method_name> = <method> definitions """
CustomizedQuerySet = QuerySet
for name, function in self.get_query_set.__dict__.items():
setattr(CustomizedQuerySet, name, function)
return CustomizedQuerySet(self.model, using=self._db)
def public(self):
""" Returns all entries accessible through front end site"""
return self.all().filter(**OfferManager.QUERYSET_PUBLIC_KWARGS)
get_query_set.public = public # will tranplat the function onto the
# returned QuerySet instance which
# means 'self' changes depending on context.
def active(self):
""" returns offers that are open to negotiation """
return self.public().filter(**OfferManager.QUERYSET_ACTIVE_KWARGS)
get_query_set.active = active
...
More polished version of this method and django ticket here: https://code.djangoproject.com/ticket/20625.
Related
I have an endpoint that can follow this format:
www.example.com/ModelA/2/ModelB/5/ModelC?word=hello
Model C has a FK to B, which has a FK to A. I should only ever see C's that correspond to the same A and B at one time. In the above example, we should...
filter C by those with an FK to B id = 5 and A id = 2
also filter C by the field 'word' that contains hello.
I know how to use the filter_queryset() method to accomplish #1:
class BaseModelCViewSet(GenericViewSet):
queryset = ModelC.objects.all()
class ModelCViewSet(BaseModelCViewSet, mixins.RetrieveModelMixin, mixins.ListModelMixin):
def filter_queryset(self, queryset):
return queryset.filter(ModelB=self.kwargs["ModelB"], ModelB__ModelA=self.kwargs["ModelA"])
I also know how to use a Filterset class to filter by fields on ModelC to accomplish #2
class ModelCFilterSet(GeoFilterSet):
word = CharFilter(field_name='word', lookup_expr='icontains')
But I can only get one or the other to work. If I add filterset_class = ModelCFilterSet to ModelCViewSet then it no longer does #1, and without it, it does not do #2.
How do I accomplish both? Ideally I want all of this in the ModelCFilterSet
Note - As hinted by the use of GeoFilterSet I will (later on) be using DRF to add a GIS query, this is just a simplified example. So I think that restricts me to using FilterSet classes in some manner.
I'm not sure this would be of help in your situation, but I often use nested urls in DRF in a way that would be convenient to perform the task. I use a library called drf-nested-routers that does part of the job, namely keeps track of the relations by the provided ids. Let me show an example:
# views.py
from rest_framework import exceptions, viewsets
class ModelBViewSet(viewsets.ModelViewSet):
# This is a viewset for the nested part that depends on ModelA
queryset = ModelB.objects.order_by('id').select_related('model_a_fk_field')
serializer_class = ModelBSerializer
filterset_class = ModelBFilterSet # more about it below
def get_queryset(self, *args, **kwargs):
model_a_entry_id = self.kwargs.get('model_a_pk')
model_a_entry = ModelA.objects.filter(id=model_a_entry_id).first()
if not model_a_entry:
raise exceptions.NotFound("MAYDAY")
return self.queryset.filter(model_c_fk_field=model_a_entry)
class ModelAViewSet(viewsets.ModelViewSet):
queryset = ModelA.objects.order_by('id')
serializer_class = ModelASerializer
# urls.py
from rest_framework_nested import routers
router = routers.SimpleRouter()
router.register('model-a', ModelAViewSet, basename='model_a')
model_a_router = routers.NestedSimpleRouter(router, 'model-a', lookup='model_a')
model_a_router.register('model-b', ModelBViewSet, basename='model_b')
...
In this case I can make a query like www.example.com/ModelA/2/ModelB/ that will only return the entries of ModelB that point to the object of ModelA with id 2. Likewise, www.example.com/ModelA/2/ModelB/5 will return only the corresponding object of ModelB in case it is related to ModelA-id2. A further level for ModelC would act correspondingly.
Sticking to the example, by now we have filtered the entries of ModelB related to a particular object of ModelA, that is we have received the relevant queryset. Next, we have to search for a particular subset within this queryset, and here's where FilterSet comes in play. The easiest way to customise its behaviour is by writing specific methods.
# filters.py
import django_filters
class ModelBFilterSet(django_filters.FilterSet):
word = django_filters.CharFilter(
method="get_word",
)
def get_word(self, queryset, name, value):
return queryset.filter(word__icontains=value)
In fact, you don't even have to use a method here; the way you pasted would work as well (word = CharFilter(field_name='word', lookup_expr='icontains')), I just wanted to point out that there is such an option too.
The filter starts its job with the queryset that has already been processed by the viewset and now it will just narrow down our sample using the given parameter.
I haven't tried this with a three-level nested URL, only checked on the example of two levels, but I think the third level should act in the same way.
Figured it out. So creating a filter_queryset() method overwrites the one that is in the GenericAPIView class (which I already knew).
However - that overwritten class is also responsible for using the FilterSet class that I defined. So by overwriting it, I also "broke" the FilterSet.
Solution was adding a super() to call the original class before the one I wrote:
def filter_queryset(self, queryset):
queryset = super().filter_queryset(queryset)
return queryset.filter(asset=self.kwargs["asset"], asset__program=self.kwargs["program"])
Using Django 3.2 -- I will simplify the problem as much as I can.
I have three model classes:
# abstract base class
MyAbstractModel(models.Model)
# derived model classes
Person(MyAbstractModel)
LogoImage(MyAbstractModel)
Each Person has:
image = ForeignKey(LogoImage, db_index=True, related_name="person", null=True,
on_delete=models.PROTECT)
The MyAbstractModel defines a few model managers:
objects = CustomModelManager()
objects_all_states = models.Manager()
as well as a state field, that can be either active or inactive
CustomModelManager is defined as something that'll bring only records that have state == 'active':
class CustomModelManager(models.Manager):
def get_queryset(self):
return super().get_query().filter(self.model, using=self._db).filter(state='active')
In my database I have two objects in two tables:
Person ID 1 state = 'active'
Image ID 1 state = 'inactive'
Person ID 1 has a foreign key connection to Image ID 1 via the Person.image field.
------ NOW for the issue ----------------
# CORRECT: gives me the person object
person = Person.objects.get(id=1)
# INCORRECT: I get the image, but it should not work...
image = person.image
Why is that incorrect? because I queried for the person object using the objects model manager which is supposed to bring only those items with active status. It brought the Person which is fine, because Person (ID=1) is state==active -- but the object under person.image is state==inactive. Why am I getting it?
WORKAROND ATTEMPT:
added base_manager_name = "objects" to the MyAbstractModel class Meta: section
ATTEMPTING AGAIN:
# CORRECT: gives me the person object
person = Person.objects.get(id=1)
# CORRECT: gives me a "Does not Exist" exception.
image = person.image
However..... Now I try this:
# CORRECT: getting the person
person.objects_all_states.get(id=1)
# INCORRECT: throws a DoesNotExist, as it's trying to use the `objects` model manager I hard coded in the `MyAbstractModel` class meta.
image = person.image
Since I got the Person under the objects_all_states which does not care about state==active -- I expect I would also get the person.image in a similar way. But that doesn't work as expected.
THE ROOT ISSUE
How do I force the same model manager used to fetch the parent object (Person) -- in the fetching of every single ForeignKey object a Person has? I can't find the answer. I've been going in circles for days. There is simply no clear answer anywhere. Either I am missing something very fundamental, or Django has a design flaw (which of course I don't really believe) -- so, what am I missing here?
Why they don't play well together
Foreign key classes use separate instances of managers, so there's no shared state.
There's no information about the manager used on the parent instance either.
As per django.db.models.Model._base_manager, Django simply uses _base_manager:
return self.field.remote_field.model._base_manager.db_manager(hints=hints).all()
...where hints would be {'instance': <Person: Person object (1)>}.
Since we have a reference to the parent, in some scenarios, we could support this inference.
Fair warning
Django specifically mentions not to do this.
From django.db.models.Model._base_manager:
Don’t filter away any results in this type of manager subclass
This manager is used to access objects that are related to from some other model. In those situations, Django has to be able to see all the objects for the model it is fetching, so that anything which is referred to can be retrieved.
Therefore, you should not override get_queryset() to filter out any rows. If you do so, Django will return incomplete results.
1. How you could implement this inference
You could:
override get() to actively store some information on the instance (that will be passed as hint) about whether an instance of CustomModelManager was used to get it, and then
in get_queryset, check that and try to fallback on objects_all_states.
class CustomModelManager(models.Manager):
def get(self, *args, **kwargs):
instance = super().get(*args, **kwargs)
instance.hint_manager = self
return instance
def get_queryset(self):
hint = self._hints.get('instance')
if hint and isinstance(hint.__class__.objects, self.__class__):
hint_manager = getattr(hint, 'hint_manager', None)
if not hint_manager or not isinstance(hint_manager, self.__class__):
manager = getattr(self.model, 'objects_all_states', None)
if manager:
return manager.db_manager(hints=self._hints).get_queryset()
return super().get_queryset().filter(state='active')
Limitations
One of possibly many edge cases where this wouldn't work is if you queried person via Person.objects.filter(id=1).first().
2. Using explicit instance context
Usage:
person = Person.objects_all_states.get(id=1)
# image = person.image
with CustomModelManager.disable_for_instance(person):
image = person.image
Implementation:
class CustomModelManager(models.Manager):
_disabled_for_instances = set()
#classmethod
#contextmanager
def disable_for_instance(cls, instance):
is_already_in = instance in cls._disabled_for_instances
if not is_already_in:
cls._disabled_for_instances.add(instance)
yield
if not is_already_in:
cls._disabled_for_instances.remove(instance)
def get_queryset(self):
if self._hints.get('instance') in self._disabled_for_instances:
return super().get_queryset()
return super().get_queryset().filter(state='active')
3. Using explicit thread-local context
Usage:
# person = Person.objects_all_states.get(id=1)
# image = person.image
with CustomModelManager.disable():
person = Person.objects.get(id=1)
image = person.image
Implementation:
import threading
from contextlib import contextmanager
from django.db import models
from django.utils.functional import classproperty
class CustomModelManager(models.Manager):
_data = threading.local()
#classmethod
#contextmanager
def disable(cls):
is_disabled = cls._is_disabled
cls._data.is_disabled = True
yield
cls._data.is_disabled = is_disabled
#classproperty
def _is_disabled(cls):
return getattr(cls._data, 'is_disabled', None)
def get_queryset(self):
if self._is_disabled:
return super().get_queryset()
return super().get_queryset().filter(state='active')
Well, i must point out a few design flaws in your approach.
First - you should not override get_queryset method for manager. Instead, make a separate method to filter specific cases. Even better if you make a custom QuerySet class with those methods, since then you will able to chain them
class ActiveQuerySet(QuerySet):
def active(self):
return self.filter(state="active")
# in your model
objects = ActiveQueryset.as_manager()
Also, you should not place field state in every model and expect, that Django will handle this for you. It will be much easier to handle for you if you decide from domain perspective, which model is your root model and have state there. For example, if Person can be inactive, then probably all of his images are also inactive, so you may safely assume, that Persons status is shared by all related models.
I would specifically look for a way to avoid such issue from design perspective, instead of trying to brutforce Django to process such filtration cases
I got a Customer model which is related to a Company model. I'd like to give my factory the possibility to use a given company (if I need several customer from the same company). I thought to use the inner class Params to achieve that, but I got an issue using LazyAttribute and SubFactory together. Here is my factory:
class CustomerFactory(UserFactory):
"""
To be valid, customer needs to belong to one of these groups:
- manager
- regular
This can be achieved using the `set_group` parameter.
Example:
manager = CustomerFactory(set_group=manager_group)
"""
#lazy_attribute
def _company(self):
if self.use_company:
return self.use_company
else:
return SubFactory('rdt.tests.factories.CompanyFactory')
class Meta:
model = Customer
class Params:
use_company = None
#post_generation
def set_group(self, create, extracted, **kwargs):
if extracted:
self.groups.add(extracted)
I thought to use the factory as:
c1 = factories.CustomerFactory(use_company=my_company)
c2 = factories.CustomerFactory()
I got ValueError. It seems I can't get the parameter value 'use_company' in the factory.
Anyway my factory throws a ValueError.
I found a solution simply using an other post_generation method like this:
#post_generation
def set_group(self, create, extracted, **kwargs):
if extracted:
self.groups.add(extracted)
#post_generation
def company(self, create, extracted, **kwargs):
if extracted:
self._company = extracted
self.save()
My question is about creating a QuerySet Mixin which provides identical QuerySet methods for both a model and a related model. Here is example code, and the first class ByPositionMixin is what I am focused on:
from django.db import models
from django.db.models.query import QuerySet
from django.core.exceptions import FieldError
class ByPositionMixin(object):
def batters(self):
try:
return self.exclude(positions=1)
except FieldError:
return self.exclude(position=1)
class PlayerQuerySet(QuerySet, ByPositionMixin):
pass
class PlayerPositionQuerySet(QuerySet, ByPositionMixin):
pass
class PlayerManager(models.Manager):
def get_query_set(self):
return PlayerQuerySet(self.model, using=self._db)
class PlayerPositionManager(models.Manager):
def get_query_set(self):
return PlayerPositionQuerySet(self.model, using=self._db)
class Position(models.Model):
# pos_list in order ('P', 'C', '1B', '2B', '3B', 'SS', 'LF', 'CF', 'RF')
# pos id / pk correspond to index value of pos_list(pos)
pos = models.CharField(max_length=2)
class Player(models.Model):
name = models.CharField(max_length=100)
positions = models.ManyToManyField(Position, through='PlayerPosition')
objects = PlayerManager()
class PlayerPosition(models.Model):
player = models.ForeignKey(Player)
position = models.ForeignKey(Position)
primary = models.BooleanField()
objects = PlayerPositionManager()
Inside ByPositionMixin, I try exclude(positions=1) which queries against PlayerQuerySet and if that generates a FieldError, I try exclude(position=1) which queries against PlayerPositionQuerySet. The difference in field names is precise, a Player() has positions, but a PlayerPosition() has only one position. So the difference it the exclude() query is 'positions' / 'position'. Since I will have many custom queries (e.g. batters(), pitchers(), by_position() etc.), do I have to write out try / except code for each one?
Or is there a different approach which would let me write custom queries without having to try against one model and then against the other one?
UPDATE: basically, I have decided to write a kwarg helper function, which provides the correct kwargs for both Player and PlayerPosition. It's a little elaborate (and perhaps completely unnecessary), but should be able to be made to simplify the code for several custom queries.
class ByPositionMixin(object):
def pkw(self, **kwargs):
# returns appropriate kwargs, at the moment, only handles one kwarg
key = kwargs.keys()[0] # e.g. 'positions__in'
value = kwargs[key]
key_args = key.split('__')
if self.model.__name__ == 'Player':
first_arg = 'positions'
elif self.model.__name__ == 'PlayerPosition':
first_arg = 'position'
else:
first_arg = key_args[0]
key = '__'.join([first_arg] + key_args[1:])
return {key: value}
def batters(self): # shows how pkw() is used
return self.exclude(**self.pkw(positions=1))
I have a private boolean flag on my model, and a custom manager that overwrites the get_query_set method, with a filter, removing private=True:
class myManager(models.Manager):
def get_query_set(self):
qs = super(myManager, self).get_query_set()
qs = qs.filter(private=False)
return qs
class myModel(models.Model):
private = models.BooleanField(default=False)
owner = models.ForeignKey('Profile', related_name="owned")
#...etc...
objects = myManager()
I want the default queryset to exclude the private models be default as a security measure, preventing accidental usage of the model showing private models.
Sometimes, however, I will want to show the private models, so I have the following on the manager:
def for_user(self, user):
if user and not user.is_authenticated():
return self.get_query_set()
qs = super(myManager, self).get_query_set()
qs = qs.filter(Q(owner=user, private=True) | Q(private=False))
return qs
This works excellently, with the limitation that I can't chain the filter. This becomes a problem when I have a fk pointing the myModel and use otherModel.mymodel_set. otherModel.mymodel_set.for_user(user) wont work because mymodel_set returns a QuerySet object, rather than the manager.
Now the real problem starts, as I can't see a way to make the for_user() method work on a QuerySet subclass, because I can't access the full, unfiltered queryset (basically overwriting the get_query_set) form the QuerySet subclass, like I can in the manager (using super() to get the base queryset.)
What is the best way to work around this?
I'm not tied to any particular interface, but I would like it to be as djangoy/DRY as it can be. Obviously I could drop the security and just call a method to filter out private tasks on each call, but I really don't want to have to do that.
Update
manji's answer below is very close, however it fails when the queryset I want isn't a subset of the default queryset. I guess the real question here is how can I remove a particular filter from a chained query?
Define a custom QuerySet (containing your custom filter methods):
class MyQuerySet(models.query.QuerySet):
def public(self):
return self.filter(private=False)
def for_user(self, user):
if user and not user.is_authenticated():
return self.public()
return self.filter(Q(owner=user, private=True) | Q(private=False))
Define a custom manager that will use MyQuerySet (MyQuerySet custom filters will be accessible as if they were defined in the manager[by overriding __getattr__]):
# A Custom Manager accepting custom QuerySet
class MyManager(models.Manager):
use_for_related_fields = True
def __init__(self, qs_class=models.query.QuerySet):
self.queryset_class = qs_class
super(QuerySetManager, self).__init__()
def get_query_set(self):
return self.queryset_class(self.model).public()
def __getattr__(self, attr, *args):
try:
return getattr(self.__class__, attr, *args)
except AttributeError:
return getattr(self.get_query_set(), attr, *args)
Then in the model:
class MyModel(models.Model):
private = models.BooleanField(default=False)
owner = models.ForeignKey('Profile', related_name="owned")
#...etc...
objects = myManager(MyQuerySet)
Now you can:
¤ access by default only public models:
MyModel.objects.filter(..
¤ access for_user models:
MyModel.objects.for_user(user1).filter(..
Because of (use_for_related_fields = True), this same manager wil be used for related managers. So you can also:
¤ access by default only public models from related managers:
otherModel.mymodel_set.filter(..
¤ access for_user from related managers:
otherModel.mymodel_set.for_user(user).filter(..
More informations: Subclassing Django QuerySets & Custom managers with chainable filters (django snippet)
To use the chain you should override the get_query_set in your manager and place the for_user in your custom QuerySet.
I don't like this solution, but it works.
class CustomQuerySet(models.query.QuerySet):
def for_user(self):
return super(CustomQuerySet, self).filter(*args, **kwargs).filter(private=False)
class CustomManager(models.Manager):
def get_query_set(self):
return CustomQuerySet(self.model, using=self._db)
If you need to "reset" the QuerySet you can access the model of the queryset and call the original manager again (to fully reset). However that's probably not very useful for you, unless you were keeping track of the previous filter/exclude etc statements and can replay them again on the reset queryset. With a bit of planning that actually wouldn't be too hard to do, but may be a bit brute force.
Overall manji's answer is definitely the right way to go.
So amending manji's answer you need to replace the existing "model"."private" = False with ("model"."owner_id" = 2 AND "model"."private" = True ) OR "model"."private" = False ). To do that you will need to walk through the where object on the query object of the queryset to find the relevant bit to remove. The query object has a WhereNode object that represents the tree of the where clause, with each node having multiple children. You'd have to call the as_sql on the node to figure out if it's the one you are after:
from django.db import connection
qn = connection.ops.quote_name
q = myModel.objects.all()
print q.query.where.children[0].as_sql(qn, connection)
Which should give you something like:
('"model"."private" = ?', [False])
However trying to do that is probably way more effort than it's worth and it's delving into bits of Django that are probably not API-stable.
My recommendation would be to use two managers. One that can access everything (an escape hatch of sort), the other with the default filtering applied. The default manager is the first one, so you need to play around with the ordering depending on what you need to do. Then restructure your code to know which one to use - so you don't have the problem of having the extra private=False clause in there already.