Why is a Django model taking so long to load in admin? - django

I have a fairly simple Django set up for a forum, and one of the most basic models is this, for each thread:
class Post(models.Model):
created = models.DateTimeField(auto_now_add=True)
last_reply = models.DateTimeField(auto_now_add=True, blank=True, null=True)
username = models.ForeignKey(User, related_name="forumuser")
fixed = models.BooleanField(_("Sticky"), default=False)
closed = models.BooleanField(default=False)
markdown_enabled = models.BooleanField(default=False)
reply_count = models.IntegerField(default=0)
title = models.CharField(_("Title Post"), max_length=255)
content = models.TextField(_("Content"), blank=False)
rating = models.IntegerField(default=0)
followers = models.IntegerField(default=0)
ip_address = models.CharField(max_length=255)
def __unicode__(self):
return self.title
def get_absolute_url(self):
return "/post/%s/" % self.id
Then we have some replies:
class PostReply(models.Model):
user = models.ForeignKey(User, related_name='replyuser')
post = models.ForeignKey(Post, related_name='replypost')
created = models.DateTimeField(auto_now_add=True)
content = models.TextField()
ip_address = models.CharField(max_length=255)
quoted_post = models.ForeignKey('self', related_name='quotedreply', blank=True, null=True)
rating = models.IntegerField(default=0)
reply_order = models.IntegerField(default=1)
Now, currently there just over 1600 users, 6000 Posts, and 330,000 PostReply objects in the db for this setup. When I run this SQL query:
SELECT * FROM `forum_post` LIMIT 10000
I see that Query took 0.0241 sec which is fine. When I browse to the Django admin section of my site, pulling up an individual Post is rapid, as is the paginated list of Posts.
However, if I try and pull up an individual PostReply, it takes around 2-3 minutes to load.
Obviously each PostReply admin page will have a dropdown list of all the Posts in it, but can anyone tell me why this or anything else would cause such a dramatically slow query? It's worth noting that the forum itself is pretty fast.
Also, if it is something to do with that dropdown list, has anyone got any suggestions for making that more usable?

Try to add all foreign keys in raw_id_fields in admin
class PostReplyAdmin(ModelAdmin):
raw_id_fields = ['user', 'post', 'quoted_post']
This will decrease page's load time in change view. The problem is that django loads ForeignModel.objects.all() for each foreign key's dropdowns.

Another way is to add foreign keys in autocomplete_fields (docs) in admin
class PostReplyAdmin(ModelAdmin):
autocomplete_fields = ['user', 'post', 'quoted_post']
As pointed by #Andrey Nelubin the problem for me was indeed in the page loading all related models for each foreign key's dropdown. However, with autocomplete_fields selects are turned into autocomplete inputs (see figure below), which load options asynchronously.

Related

django orm foreign key latest data using filter query

class Schedule(models.Model):
user = models.ForeignKey(USER, on_delete=models.SET_NULL, null=True)
area = models.ForeignKey(Site, on_delete=models.SET_NULL, null=True)
created_at = models.DateTimeField(auto_now_add=True, null=True)
updated_at = models.DateTimeField(auto_now=True, null=True)
area = Schedule.objects.values("area").annotate(latest=Max('created_at')).values("area")
latest = Schedule.objects.values("area").annotate(latest=Max('created_at')).values("latest")
Schedule.objects.filter(created_at__in=latest, area__in=area)
I got the value I want.
However, I am uneasy when I filter with ForeignKey and DateTimeField. Can't there be a better way?
Also, can't you make it cleaner other than the code above?
Guess you can create a Manager for your Model and use selected_related feature, it will create quick query for you, and it`s ready "out-of-the-box" Django solution. Plz see Django docs.
Here`s example based on your code
In Model.py something like
class SheduleManager(model.Manager):
def all_prefetched_data(self):
qs = self.get_queryset()
qs = qs.select_related(
'area_set'
)
return qs
in the same Model.py you need to assign your manager in Shedule model
listing = SheduleManager
Now in view you can sort db items by creation date, query with the ForiegnKey fetching through all your db. You also can display result, for example in your_template.html with the dictionary b_list
view.py
def GetList(request):
sh_list = Schedul.listing.all().order_by('-created_at')
return render(request, 'your_template.html', {'b_list': sh_list})

Django ORM Query Optimization Issue

I am making a blog website and I am facing some issues with the Query performance.
I have 3 models
User Model -> Users (To store user email, Password etc)
Post Model -> Actual Posts
people Model -> (To store users extra information)
Post Model ->
class Post(models.Model):
user = models.ForeignKey(User, on_delete=models.PROTECT)
category = models.ForeignKey(Category, on_delete=models.PROTECT)
title = models.CharField(max_length=255,null=True)
description = models.CharField(max_length=1000,null=True)
Likes = models.ManyToManyField(to=User, related_name='Post_likes')
favourites = models.ManyToManyField(to=User,blank=True,related_name="favourite")
People Model ->
class People(models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
photo = models.ImageField(upload_to='profile_pics', blank=True,null=True)
Phone_number = models.CharField(max_length=255,null=True,blank=True)
Birth_Date = models.DateField(null=True,blank=True)
Created_date = models.DateTimeField(auto_now_add=True)
Updated_date = models.DateTimeField(auto_now=True)
Now as both of these models are connected to User model. I want to query the Post model and get the user photo in the template. Now when I use post.user.people.photo then for every post it generates a seperate query to DB resulting in slowness. I would like to use Join here to combines multiple tables and fetch all the records at once.
I am currently using following Query ->
posts = Post.objects.select_related().prefetch_related('images_set').annotate(comments_Count = Count('comments_post',distinct=True)).annotate(Count('Likes',distinct=True)).all().order_by('-id')
You can perform a .select_related(…) [Django-doc] on the user and the people with user__people, so:
posts = Post.objects.select_related(
'user__people', 'category'
).prefetch_related('images_set').annotate(
comments_Count = Count('comments_post',distinct=True),
Count('Likes',distinct=True)
).order_by('-id')
Note: It is normally better to make use of the settings.AUTH_USER_MODEL [Django-doc] to refer to the user model, than to use the User model [Django-doc] directly. For more information you can see the referencing the User model section of the documentation.

Django Admin super slow selecting single record

I have Pick model which when I select a record within Django Admin takes an age (about 20 seconds) to retrieve the record (there are about 70k). However, it is quick when I try and create/save a record. I have added the indexes to try and speed up retrieval but don't really know what I should be doing
class Pick (models.Model):
team = models.ForeignKey(Team, on_delete=models.CASCADE, null=True, blank=True)
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, db_index=True)
contest_entry=models.ForeignKey(ContestEntry, on_delete=models.CASCADE, db_index=True)
game_round=models.ForeignKey(GameRound, on_delete=models.CASCADE, db_index=True)
objects = DataFrameManager()
def __str__(self):
return f'%s %s' % (self.contest_entry, self.game_round)
class Meta:
unique_together = ['user', 'contest_entry', 'game_round']
ordering = ['contest_entry','game_round','user','team']
index_together = [["user", "contest_entry", 'game_round'],]
indexes = [
models.Index(fields=['user', 'contest_entry', 'game_round']),
models.Index(fields=['game_round'], name='game_round_idx'),
models.Index(fields=['contest_entry'], name='contest_entry_idx'),
]
class PickAdmin(admin.ModelAdmin):
model = Pick
list_filter= (
('user', RelatedDropdownFilter),
('contest_entry__contest', RelatedDropdownFilter),
('game_round', RelatedDropdownFilter)
)
def get_queryset(self, request):
return super().get_queryset(request).select_related('user','contest_entry','game_round')
admin.site.register(Pick, PickAdmin)
How can I improve performance here?
The key is specifying the raw_id_fields on the admin class. The default <select> widget for foreign key relationships can cause a lot of overhead retrieving and rendering all the options.
class PickAdmin(admin.ModelAdmin):
model = Pick
list_filter= (
('user', RelatedDropdownFilter),
('contest_entry__contest', RelatedDropdownFilter),
('game_round', RelatedDropdownFilter)
)
raw_id_fields = ('user', 'contest_entry', 'game_round',)
def get_queryset(self, request):
return super().get_queryset(request).select_related('user','contest_entry','game_round')
My guess is that what takes so long is rendering the dropdowns for the ForeignKey fields of your model.
I'd suggest looking at the SQL queries that are performed, either by installing Django toolbar or enabling logging in your database server.
You can improve the queries by using select_related for foreignkeys https://docs.djangoproject.com/en/2.2/ref/models/querysets/#select-related

Django - Show only a specific dynamic fields per models in django-eav2

I'm trying to figure it out on how I can show only a specific set of dynamic fields in eav to a unique registered model in my apps.models. But I don't know how to this, I've also read the documents but I can't seem to find anything about it, or maybe I've come across it and didn't understand.
Now, what is happening is that, when I add an attribute in the django admin. It also adds the dynamic field in all the models registered in the eav.
What I want to do is that;
model 1 - dynamic_field1, dynamic_field2, dynamic_field3
model 2 - dynamic_field4, dynamic_field5, dynamic_field6
Btw, I'm currently using the django-eav2 the documentation is in the link. I've found my solution for my initial use case here link
Below codes are basically on how to register my models to the eav. Here is my sample models
class ClientName(models.Model):
name = models.CharField(max_length=250, null=True, blank=True)
description = models.TextField(null=True, blank=True)
is_active = models.BooleanField(default=True)
def __str__(self):
return str(self.name)
class CallDetails(models.Model):
client_name = models.ForeignKey(ClientName, on_delete=models.PROTECT, null=True, blank=True, db_index=True)
letter_info = models.TextField(null=True, blank=True)
def __str__(self):
return str(self.client_name)
class Meta:
verbose_name = 'Call Detail'
ordering = ['client_name']
eav.register(ClientName)
eav.register(CallDetails)
below is my admin.py
class CallDetailsAdminForm(BaseDynamicEntityForm):
model = CallDetails
class CallDetailsAdmin(BaseEntityAdmin):
form = CallDetailsAdminForm
admin.site.register(CallDetails, CallDetailsAdmin)

Excluding objects from Django queryset based on recency

I have a reddit-like Django app where users can post interesting urls (links) and then publicly comment under them. The two data models to represent this are:
class Link(models.Model):
description = models.TextField(validators=[MaxLengthValidator(500)])
submitter = models.ForeignKey(User)
submitted_on = models.DateTimeField(auto_now_add=True)
class Publicreply(models.Model):
submitted_by = models.ForeignKey(User)
answer_to = models.ForeignKey(Link)
submitted_on = models.DateTimeField(auto_now_add=True)
description = models.TextField(validators=[MaxLengthValidator(250)])
How do I query for all Links which have at least 1 or more publicreply, and secondly where the latest publicreply is not by self.request.user? I sense something like the following:
Link.objects.filter(publicreply__isnull=False).exclude(**something here**)
Please advise! Performance is key too, hence the simpler the better!
For performance and simplicity you could cache both the number of replies and the latest reply:
class Link(models.Model):
...
number_of_replies = models.PositiveIntegerField(default=0)
latest_reply = models.ForeignKey('myapp.Publicreply', related_name='+', blank=True, null=True, on_delete=models.SET_NULL)
When a reply is entered, update the corresponding link.number_of_replies and link.latest_reply.
The query would then be:
Link.objects.filter(number_of_replies__gte=1)\
.exclude(latest_reply__user=request.user)