I've been trying to get a list of model objects, randomize their order and paginate them in the template. I thought I've done it until I realize that in each next page call I had been re-randomizing objects rather than using the previously randomized list.
Is there anyway I can randomize the object list only while opening the page first time and after that use the same list without randomizing it while going to the next page?
Thanks.
Views.py
class index(View):
def get(self, request):
all_items = list(Electronics.objects.all())
random.shuffle(all_items)
paginator = Paginator(items, 24)
page = request.GET.get('page')
items = paginator.get_page(page)
return render(request, 'home.html', {'items':items, 'header':'Homepage'})
home.html
<div class='container'>
<div class='row'>
{% for item in items %}
<div class='col-xs-6 col-sm-8 col-lg-4'>
<img src='{{ item.image.url|cut:"%22"|thumb}}'>
<h4>{{item.name}}</h4>
<p>$ {{item.price}}</p>
<form method='get'>
<input value='Add to cart' type='submit' formaction= "{% url 'addtocart' item.id %}">
<input value='View Details' type='submit' formaction= "{% url 'detail' item.id %}">
</form>
<div><br><br>
{% endfor %}
</div>
</div>
<div class='pagination' align='center'>
<span class='step-links'>
{% if items.has_previous %}
« first
previous
{% endif %}
<span class="current">
Page {{items.number}} of {{items.paginator.num_pages}}
</span>
{% if items.has_next %}
next
last »
{% endif %}
</span>
</div>
You can use this great snippet to implement a randomized seeded queryset mixin. This will basically allow you to create a GET request to the view using the mixin (via a url such as /electronics?page=1), and then it will generate a seed which will be cached in your session, and then reused to pull iterative sets of records.
This shouldn't be too much of a burden on your load, since caching a seed is inexpensive - it is the use of order_by('?') which can be expensive, but 3-4 thousand records isn't considered that big for such a task.
Related
I have been working with HTMX and it's pretty cool compared to the dreaded formsets and Javascript. I have it working....My only issue is when the user updates the form anywhere...you have to manually refresh the page to reset the list of todos. My issue is identical to this one...https://stackoverflow.com/questions/66664407/dynamically-update-table-when-creating-new-enty-using-htmx but there is no resolution listed.....
Here's a quick overview of my code...
My view...
def create_to_do(request):
user = User.objects.get(id=request.user.id)
to_dos = NewToDo.objects.filter(created_by=user)
form = ToDoForm(request.POST or None)
if request.method == "POST":
if form.is_valid():
to_do = form.save(commit=False)
to_do.created_by = user
to_do.creation_date = timezone.now()
to_do.save()
return redirect("MyToDoList:detail-to-do", pk=to_do.id)
else:
return render(request, "partials/to_do_form.html", {
"form":form
})
context = {
"form": form,
"user": user,
"to_dos": to_dos,
}
return render(request, "create_to_do.html", context)
Partial detailview....
<button class="button35" hx-get="{% url 'MyToDoList:update-to-do' to_do.id %}" hx-swap="outerHTML">
Update
</button>
<button class="button34" hx-post="{% url 'MyToDoList:delete-to-do' to_do.id %}" hx-swap="outerHTML">
Delete
</button>
</div>
Partial todo form....
<div hx-target="this" hx-swap="outerHTML" class="">
<form method="POST">
{% csrf_token %}
{% if to_do %}
<button class="button35" hx-post="{% url 'MyToDoList:update-to-do' to_do.id %}">
Save
</button>
<button class="button34" hx-get="{% url 'MyToDoList:detail-to-do' to_do.id %}">
Cancel
</button>
</div>
{% else %}
<button class="button35" hx-post=".">
Save
</button>
</div>
{% endif %}
</form>
</div>
My main create form html..
<button class="button36" hx-get="{% url 'MyToDoList:create-to-do-form' %}" hx-swap="beforeend" hx-target="#bookforms">Add Form</button>
<div id="bookforms" class=""></div>
<div class="">
{% if to_dos %}
{% for to_do in to_dos %}
{% include "partials/to_do_detail.html" %}
{% endfor %}
{% endif %}
After a day of playing and pulling my hair out..it's all working as I want...I just need to figure out how to incorporate a dynamic page load if anything changes so that the entire page gets reloaded....so that the records get sorted according to my number field...
Thanks in advance for any thoughts or suggestions.
So thanks to a kind soul on Facebook....I added this Javascript to my code and it works.
$(document).on("click", ".yourclass", function(){
location.reload();
});
In case you need this, I had the problem when I have to update two/mutliple parts of the page after post.
I manage to solve it by trigger_client_event
Thanks to this issue: How do I manually trigger a hx-get event of an element from JavaScript
HTMX rocks!
my checkbox html :
<form method="GET">
{% for subcategory in subcategories %}
<div class="filter-items filter-items-count">
<div class="filter-item">
<div class="custom-control custom-checkbox">
<input type="checkbox" class="custom-control-input" value="{{subcategory.id}}" name="subcategory" id="{{subcategory.id}}" {% if subcategory in subcategories1 %} checked {% endif %}>
<label class="custom-control-label" for="{{subcategory.id}}">{{subcategory}}</label>
</div><!-- End .custom-checkbox -->
<span class="item-count">{{subcategory.products.all|length}}</span>
</div><!-- End .filter-item -->
</div><!-- End .filter-items -->
{% endfor %}
<button type="submit" class="btn btn-primary w-100 mt-3">Filter!</button>
</form>
it work correctly to make filter.
views :
subcatid = request.GET.getlist('subcategory')
query string:
?subcategory=5&subcategory=6
it can be one or more than one, depends on number of subcategories.
but when I go next page i suppose it become like :
?page=2&subcategory=5&subcategory=6
but it remove earliest subcategory i choose and keep the last one, just one, like :
?page=2&subcategory=5
acutely when i put Manually ?page=2&subcategory=5&subcategory=6 in url field it works but not from pagination buttons.
so while all checkboxes in filter has same names, name="subcategory" i made them unique by changing to name="{{subcategory}}", now each checkbox has unique name, now after tapping next page, Everything is kept, and there is no problem like before,
but in views, I don't know how to get them with deafferents names
subcatid = request.GET.getlist('subcategory')
You can add a QueryDict to the object that does not contain any page parameter with:
def subcategory(request, category_url):
# …
reqget = request.GET.copy()
reqget.pop('page', None)
ctx = {
# …
'reqget': reqget
}
return render(request, 'products/subcategory.html', ctx)
Then in the template, the links to go to the previous/next page will urlencode the reqget:
<a href="?page={{ products.previous_page_number }}&{{ reqget.urlencode }}" aria-label="Previous" tabindex="-1" class="page-link page-link-prev" aria-disabled="true">
I'm rendering a template and it's taking anywhere between 2-3 seconds to do it. There are quite a few elements on the page (10k checkboxes which change every query) and I'm wondering if anyone knows any helpful tricks to cut down on load time.
Link to site: http://18.207.127.123/search/
FYI: there are no Django Forms or Formsets in this application! :-) I'm working with objects called Lemmas which have Forms as their children... sorry if this causes confusion.
Update #1
Whatever is taking so long, it's here:
<li class="form-item" data-lemma="{{lemma.id}}" data-group="{{group}}" style="display:none">
<input type="checkbox" name="{{lemma.name}}#{{lemma.latin}}#{{lemma.homonym_id}}#{{group}}#{{form}}" onchange="countCheckboxes(this)" id="{{lemma.id}}#{{group}}#{{form}}" checked>
<label for="{{lemma.id}}#{{group}}#{{form}}">{{ form }}</label>
</li>
are the dot methods what's slowing this down?
Update #2 - Found the bottleneck
Apparently all the {{ var }} calls for each of the 10k items is what's taking so long. Each one costs ~50-200ms.
Update #3 - Remove unnecessary tags and autoescape off
<input id="x"...
<label for="x">
can be replaced with
<label>
<input ...>
</label>
saving six variable calls.
Adding
{% autoescape off %}
# body text
{% endautoescape %}
removes the need to make each variable safe.
These two changes bring the rendering time down about 50%.
Is it database hits?
After inspecting my django connections and profiling, I'm only making two database calls as far as I can tell.
My datastructure is a dict which looks like:
results_dict = { LemmaObj1 : [form11, form12, form13, ...],
LemmaObj2 : [form21, form22, form33, ...],
...
where the LemmaObj are Queryset objects. Generating this dict only takes 0.5 seconds. I force evaluation of the lazy queryset by putting the formij into a list comprehension.
views.py
lemma_qs = Lemma.objects.prefetch_related('form_set')
# ...
# some logic goes here
# ...
for lemma in lemma_qs:
form_qs = lemma.form_set.all()
form_list = [form.name for form in form_qs if some_condition(form)]
if form_list:
results_dict[lemma] = form_list
context['results_dict'] = results_dict
return render(request, "query.html", context)
All this is to say that I'm pretty sure the slowdown isn't coming from database hits.
The template in question
In my query.html I have two for loops.
query.html
<div class="lemma-box">
<ol>
<li class="lemma-title">
<div class="lemma-col-1">
lemma [{{results_dict|length}}] (group {{group}})
</div>
<div class="lemma-col-2">
latin
</div>
<div class="lemma-col-3">
homonym id.
</div>
<div class="lemma-col-4">
<input type="button" class="pushable" value="all" onclick="checkAllLemmas('{{group}}', true)"></input>
<input type="button" class="pushable" value="none" onclick="checkAllLemmas('{{group}}', false)"></input>
</div>
</li>
{% for lemma, form_list in results_dict.items %}
<li class="lemma-item" data-lemma="{{lemma.id}}" data-group="{{group}}" onclick="activateLemma(this)">
<div class="lemma-col-1">
<input type="checkbox" onchange="countCheckboxes(this)" onclick="lemmaToggleAll(this)" id="{{lemma.id}}#{{group}}" checked></input>
<label for="{{lemma.id}}#{{group}}">
{{ lemma.name }}
</label>
</div>
<div class="lemma-col-2">
{{ lemma.latin }}
</div>
<div class="lemma-col-3">
{{ lemma.homonym_id }}
</div>
<div class="lemma-col-4">
{% with form_list|length as total %}
<span class="counter">(<span class="total">{{ total }}</span>/<span>{{ total }}</span>)</span>
{% endwith %}
</div>
</li>
{% endfor %}
{% for item in not_found_items_set %}
<li class="lemma-item-not-found">
{{ item }} not found
</li>
{% endfor %}
</ol>
</div>
<div class="form-box">
<ol>
<li class="form-title">
<div class="form-col-1">
forms (group {{group}})
</div>
<div class="form-col-2" data-group="{{group}}">
<input type="button" class="pushable" value="all" onclick="checkAllForms('{{group}}', true)"></input>
<input type="button" class="pushable" value="none" onclick="checkAllForms('{{group}}', false)"></input>
</div>
</li>
{% for lemma, form_list in results_dict.items %}
{% for form in form_list %}
<li class="form-item" data-lemma="{{lemma.id}}" data-group="{{group}}" style="display:none">
<input type="checkbox" name="{{lemma.name}}#{{lemma.latin}}#{{lemma.homonym_id}}#{{group}}#{{form}}" onchange="countCheckboxes(this)" id="{{lemma.id}}#{{group}}#{{form}}" checked>
<label for="{{lemma.id}}#{{group}}#{{form}}">{{ form }}</label>
</li>
{% endfor %}
{% endfor %}
</ol>
</div>
Is this about as fast as I'm going to get?
Jinja2 the answer?
I tried jinja2 briefly by converting query.html to query.jinja and it seemed to make almost no difference. Do I need extra steps to leverage jinja2?
Models.py
Finally, for inspection, my models.py.
models.py
class Form(models.Model):
name = models.CharField(max_length=100, db_index = True)
lemma = models.ForeignKey("Lemma", on_delete = models.CASCADE)
def __str__(self):
return self.name
class Lemma(models.Model):
name = models.CharField(max_length=100, db_index = True)
latin = models.CharField(max_length=100, db_index = True, blank = True, null = True)
homonym_id = models.IntegerField(null = True)
def __str__(self):
return self.name
One thing to do is remove as many template tags as possible by thinking clearly about what your code needs to do. That will cut down on the rendering time substantially. Such as
<input id="x"...
<label for="x">
being replaced with
<label>
<input ...>
</label>
and saving a series of template tags.
You can mark the variables as safe, which again shaves off the rendering time, such as in
{% autoescape off %}
# body text
{% endautoescape %}
The absolute best way (if you need speed and have a ton of data) is to send the JSON to the client and have the JS render it. This is a little messy, but it's much faster (2-3x faster maybe) and gives you more flexibility to manipulate the DOM piecewise.
I'm going for solution (3) for now.
I have a page with projects for a specific category that begins with the top-x most earned projects, followed by the top-x most viewed and the third segment contains all (other) projects for this category.
This third segment is long as it contains all projects for this specific category. Therefore, I would like to paginate them. But with standard pagination going to the next page will refresh the page, ending on the top of the page, needing to scroll down again to the third section (every single time you select the next page), which isn't very user friendly.
I saw a site (kickstarter) where they were able to paginate only one segment of the page, leaving the other parts of the site untouched, which is exactly what I want. But I am a novice and I have no idea how to do this.
Thanks for any ideas and help!
My HTML code:
{% for project in projects|dictsortreversed:"created_date"%}
{% if project.num_days > 0 %}
<div class="row no-gutters">
<div class="offset-1 col-4 mb-5 ">
{% if project.image %}
<a href="{% url 'project_detail' project.id %}" target="_blank">
<img class="img-fluid mt-5" src="{{MEDIA_URL}}{{ project.image }}">
</a>
{% else %}
<a href="{% url 'project_detail' project.id %}" target="_blank">
<img class="img-fluid mt-5" src="{% static 'img/No-image-available.png' %}" alt="no image available">
</a>
{% endif %}
</div>
<div class="offset-1 col-6 mt-5">
<a href="{% url 'project_detail' project.id %}" target="_blank">
<h3 class="maintext">{{ project.title}}</h3>
</a>
<p class="subtext">{{ project.category }}</p>
<p>{{ project.description |truncatewords:60}}</p>
<p class="subtext mb-5">goal: ${{project.goal}}      ends in: {{ project.num_days }} days</p>
</div>
</div>
{% endif %}
{% endfor%}
My view code
def get_project_category(request, project_category):
"""
Projects for a specific category
"""
projects = Project.objects.filter(
category=Category.objects.get(category=project_category).id).order_by('-created_date')
projects.category = project_category
calculations(projects)
context = {
'projects': projects,
}
return render(request, 'project_category.html', context)
I used https://datatables.net/manual/installation.
It makes a table with automatic search options and pagination. It paginates without having to start on top of the page.
AFter that I used CSS to eliminate the styling as a table. So made the header invisible and added an additional column in the table to create spacing between my images and text.
I am working on a website, where a user can vote and comment on different projects in various categories. Whenever a user returns to the site, their votes will be highlighted and all the comments will be visible.
The problem is that every time the page is loaded there are many database calls due to the design of my models or due to me not finding an efficient way of retrieving the data.
My models are as follows:
class ProjectCategory:
title=models.CharField(...)
class Project:
category=models.ForeignKey(ProjectCategory)
title=models.CharField(...)
def ownedComments(self):
return Comment.objects.filter(project=self).order_by('-submissionTime')
def ownedCommentsPreview(self):
return Comment.objects.filter(project=self).order_by('-submissionTime')[0:3]
def ownedVotes(self):
return Vote.objects.filter(project=self,upVoted=True).count()
class Vote:
project=models.ForeignKey(Project)
user=models.ForeignKey(User)
vote=models.IntegerField(...)
class Comment:
project=models.ForeignKey(Project)
user=models.ForeignKey(User)
title=models.CharField(...)
Here are relevant parts of my views.py:
def getCurrentUserData(request=None):
if (request==None) or (request.user.is_authenticated()==False):
return {'loggedInUsername':'Anonym','userIsLoggedIn':False}
return {'loggedInUsername':request.user.username,'userIsLoggedIn':True}
def getProjectResponse(request,surveyFile=None,additionalInfo={}):
userVotedUp=[]
if surveyFile!=None:
userVotes=Vote.objects.filter(upVoted=True,surveyFile=surveyFile)
for vote in userVotes:
userVotedUp.append(vote.project.id)
answerDict={'userVotedUp':userVotedUp,'prjCategories':PrjCategory.objects.select_related(),'projects': Project.objects.prefetch_related('category').filter(visible=True),'spammerquestion':spammerquestion.SpammerQuestion().getRandomQuestion(),'rotatingLines':helpers.getRandomRotatingLines(),'backgroundimageurl':helpers.getBackgroundImageUrl()}
answerDict.update(getCurrentUserData(request))
answerDict.update(additionalInfo)
return answerDict
def showProject(request,prjNo):
currentPrj=get_object_or_404(Project,prjNo=prjNo,visible=True)
surveyFile=assignSurveyFile(request,create_surveyFile_if_not_existing=False)
return render_to_response('website/singleproject.html', getProjectResponse(request,surveyFile,{'currentPrj':currentPrj,'immediateProjectID':currentPrj.id}),context_instance=RequestContext(request))
The template contains many iterations over projects and categories, there might be several database calls happening here. Relevant parts of the template are:
{% for project in projects %}
self.vote['{{ project.id }}']=ko.observable(new Vote({'totalVoteNumber':ko.observable({{ project.ownedVotes }}),'userVoteState':ko.observable({% if project.id in userVotedUp %}true{% else %}false{% endif %})}));
self.comments['{{ project.id }}']=ko.observableArray([]);
{% for comment in project.ownedCommentsPreview %}
self.comments['{{ project.id }}'].push(new Comment({'description':'{{ comment.description|striptags|safe }}','submissionTime':'{{ comment.submissionTime|humanDT }}','commentUuid':'{{ comment.commentUuid }}','deletable':false}));
{% endfor %}
{% endfor %}
{% for prjCat in prjCategories %}
.prjCatUnderline{{ prjCat.id }} {border-bottom: 0px solid {{ prjCat.color }};}
.prjCatUnderline{{ prjCat.id }}:hover {border-bottom: 3px solid {{ prjCat.color }};}
{% endfor %}
{% for prjCat in prjCategories %}
{{ prjCat.title }}
{% if not forloop.last %}<span class="catpipe">|</span>{% endif %}
{% endfor %}
<script type="text/javascript">
var projectLocations = [{% for project in projects %}{% if project.location != '' %}['{{ project.title }}', {{ project.location }}, {{ project.id }}, '{{ MEDIA_URL }}{{ project.mapImage }}', {{ project.mapImage.width }}, {{ project.mapImage.height }}, '{{ project.id }}'],
{% endif %}{% endfor %}];
</script>
{% for project in projects %}
<div class="item prjCat{{ project.category.id }}">
<div class="prjnumberbox">
<div class="numberbar" style="background-color:{{ project.category.color }}"></div>
<div class="prjnumber">{{ project.prjNo|stringformat:"02d" }}</div>
<div class="numberbar" style="background-color: {{ project.category.color }}"></div>
</div>
<div class="prjheadline"><span>{{ project.title }}</span></div>
<div class="prjimagecontainer">
<!-- ko with: $root.vote['{{ project.id }}'] -->
<div class="prjimageoverlay"><a data-bind="click: function(data,event) { $root.showSingleProjectData('{{ project.id }}',data) },css : { imgVoted :userVoteState }" class="singleProjectOverlay" href="{% url 'website.views.showProject' prjNo=project.prjNo %}"></a></div>
<div class="prjimage"><img src="{{ MEDIA_URL }}{{ project.teaserImage }}" /></div>
<!-- /ko -->
</div>
<div class="prjtextcontainer" {% if project.ownedComments|length_is:"0" %}style="background-image:none;"{% endif %}>
<div class="prjtext">{{ project.teaserText|safe }} <a class="singleProjectOverlay" data-bind="click: function(data,event) { showSingleProjectData('{{ project.id }}',data) }" href="{% url 'website.views.showProject' prjNo=project.prjNo %}">mehr >></a></div>
<div class="votesline"><span class="voteslinefont" data-bind="with: vote['{{ project.id }}']"> <span class="votecount" data-bind="text: totalVoteNumber">{{ project.ownedVotes }}</span> votes </span></div>
<div class="prjcomments nojs">
{% for comment in project.ownedCommentsPreview|slice:":4" %}
<div class="prjcommentitem">
{{ comment.teaserdescription }}
</div>
{% endfor %}
</div>
<div class="prjcomments" data-bind="foreach: comments['{{ project.id }}'], visible: comments['{{ project.id }}']().length > 0">
<!-- ko if: $index() < 4 -->
<div class="prjcommentitem">
<em data-bind="text: submissionTime"> </em><br/>
<span data-bind="text: description"> </span>
[X]
</div>
<!-- /ko -->
</div>
</div>
</div>
{% endfor %}
For the front page I'd like to retrieve all the projects, most recent three comments, the total number of votes for a project and the comments and votes by the user.
It would be great if that was possible with one db call, or with two, a general one to get the number of votes and the comments, and one for the data specific to the user.
CUrrently I am retrieving all the projects with one call, so I am automatically getting all the project categories. Then I get all the comments at once and assign them to the projects in python. I then get the comments and votes for a certain user with one call each.
Is there a way - without using sql directly - to get
comments and votes for a user with a single database call
comments, projects, accumulated number of votes per project and project categories at once?
One idea would be to download the whole database using select_related() and then sorting everything out in Python, but if in the future I have a large number of votes or comments, this might not be a good idea.