Case: paginated page with an overview of users whom sent the 'receiver' a message. Once clicked, the receiver will go to the page with the actual conversation.
The 'overview page' is paginated (works). However, after implementation we noticed that if user X received 2 messages from Y the front-end displays two rows for Y. So we are looking for a solution to group the paginated objects - the below came to life. This however gives us the next challenge. As the pagination only retrieves the first X objects, if the next 'paginated page' contains the user from page 1 (user Y). user Y will occur on page 2 as well. I tried to create a pagination object after making the group 'unique_group' but this is just a list and cannot be paginated.
Retrieving .all() seems inefficient, especially when the application grows.
def group_paginated_msgs(user, Message):
#[Y, Y, Z] = simple example
''' grab user messages '''
page = request.args.get('page', 1, type=int)
messages = user.messages_received.order_by(
Message.timestamp.desc()
).paginate(
page,
5, #from config in prod.
False
)
unique_authors = []
unique_group = []
# group user messages
try:
p = messages.items
for m in p:
if m.author in unique_authors:
continue
else:
unique_authors.append(m.author)
unique_group.append(m)
except:
...
# print(unique_group) -> [Y, Z]
Could you use the .distinct() method to get the unique users from your query before you paginate?
Something like this
messages = Messages.query.filter(receiver==user).distinct(Message.author)
.order_by(
Message.timestamp.desc()
).paginate(
page,
5, #from config in prod.
False
)
This example works by querying the messages that have been received by your user and then filtering for distinct message authors.
I'm not sure exactly what I have written will work, but I think using the distinct() method is a good place to start. You can find more info about the method here.
Related
What I'm trying to do
I'm trying to enter a list of tags in flask that should become passable as a list but I can't figure out how to do it in flask, nor can I find documentation to add lists (of strings) in flask_wtf. Has anyone have experience with this?
Ideally I would like the tags to be selectively delete-able, after you entered them. So that you could enter.
The problem
Thus far my form is static. You enter stuff, hit submit, it gets processed into a .json. The tags list is the last element I can't figure out. I don't even know if flask can do this.
A little demo of how I envisioned the entry process:
How I envisioned the entry process:
The current tags are displayed and an entry field to add new ones.
[Tag1](x) | [Tag2](x)
Enter new Tag: [______] (add)
Hit (add)
[Tag1](x) | [Tag2](x)
Enter new Tag: [Tag3__] (add)
New Tag is added
[Tag1](x) | [Tag2](x) | [Tag3](x)
Enter new Tag: [______]
How I envisioned the deletion process:
Hitting the (x) on the side of the tag should kill it.
[Tag1](x) | [Tag2](x) | [Tag3](x)
Hit (x) on Tag2. Result:
[Tag1](x) | [Tag3](x)
The deletion is kind of icing on the cake and could probably be done, once I have a list I can edit, but getting there seems quite hard.
I'm at a loss here.
I basically want to know if it's possible to enter lists in general, since there does not seem to be documentation on the topic.
Your description is not really clear (is Tag1 the key in the JSON or is it Tag the key, and 1 the index?)
But I had a similar issue recently, where I wanted to submit a basic list in JSON and let WTForms handle it properly.
For instance, this:
{
"name": "John",
"tags": ["code", "python", "flask", "wtforms"]
}
So, I had to rewrite the way FieldList works because WTForms, for some reason, wants a list as "tags-1=XXX,tags-2=xxx".
from wtforms import FieldList
class JSONFieldList(FieldList):
def process(self, formdata, data=None):
self.entries = []
if data is None or not data:
try:
data = self.default()
except TypeError:
data = self.default
self.object_data = data
if formdata:
for (index, obj_data) in enumerate(formdata.getlist(self.name)):
self._add_entry(formdata, obj_data, index=index)
else:
for obj_data in data:
self._add_entry(formdata, obj_data)
while len(self.entries) < self.min_entries:
self._add_entry(formdata)
def _add_entry(self, formdata=None, data=None, index=None):
assert not self.max_entries or len(self.entries) < self.max_entries, \
'You cannot have more than max_entries entries in this FieldList'
if index is None:
index = self.last_index + 1
self.last_index = index
name = '%s-%d' % (self.short_name, index)
id = '%s-%d' % (self.id, index)
field = self.unbound_field.bind(form=None, name=name, id=id, prefix=self._prefix, _meta=self.meta,
translations=self._translations)
field.process(formdata, data)
self.entries.append(field)
return field
On Flask's end to handle the form:
from flask import request
from werkzeug.datastructures import ImmutableMultiDict
#app.route('/add', methods=['POST'])
def add():
form = MyForm(ImmutableMultiDict(request.get_json())
# process the form, form.tags.data is a list
And the form (notice the use of JSONFieldList):
class MonitorForm(BaseForm):
name = StringField(validators=[validators.DataRequired(), validators.Length(min=3, max=5)], filters=[lambda x: x or None])
tags = JSONFieldList(StringField(validators=[validators.DataRequired(), validators.Length(min=1, max=250)], filters=[lambda x: x or None]), validators=[Optional()])
I found a viable solution in this 2015 book, where a tagging system is being build for flask as part of a blog building exercise.
It's based on Flask_SQLAlchemy.
Entering lists therefore is possible with WTForms / Flask by submitting the items to the database via, e.g. FieldList and in the usecase of a tagging system, reading them from the database back to render them in the UI.
If however you don't want to deal with O'Rielly's paywall (I'm sorry, I can't post copyrighted material here) and all you want is a solution to add tags, check out taggle.js by Sean Coker. It's not flask, but javascript, but it does the job.
I have a ReactJS component inside a Django template, where a user clicks on a checkout button, posts the item_code and gets redirected to checkout:
onCheckout = () => {
fetch("/onCheckout/", {
method: "POST",
body: JSON.stringify({'item': this.props.item_info.code})
}).then(window.location.replace("/checkout"))
}
A Django view receives the request and stores it in a session.
def onCheckout(request):
if request.method == "POST":
items = request.session.get('items', [])
new_item = json.loads(request.body.decode('utf-8'))['item']
items.append(new_item)
request.session['items'] = items
I am having a issue with storing data in the session. After the first item gets stored correctly in the array, and I then checkout on a second item, the items array starts acting up:
(Pdb) items
['15130BC.ZZ.8042BC.01']
(Pdb) new_item
'5213G-001'
(Pdb) items
['15130BC.ZZ.8042BC.01']
(Pdb) items
['5213G-001']
If I try to access request.session['item'] from any other view function, I get a KeyError.
I am fairly new to Django, any help would be appreciated. Also, I would like to know if there are better alternatives to accomplish the above.
Sessions Config
settings.SESSION_ENGINE = 'django.contrib.sessions.backends.db'
settings.SESSION_CACHE_ALIAS = 'default'
settings.CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
Some reading on change detection for Django sessions: https://docs.djangoproject.com/en/2.0/topics/http/sessions/#when-sessions-are-saved
Based on your code, it appears to me that the change detection should happen. However, let's try to brute force this, can you add the following line as the last line of your code: request.session.modified = True - see if this fixes your issue?
Update: some basic checks
Can you verify the following
Check if your db backend is configured priestly
If you want to use a database-backed session, you need to add 'django.contrib.sessions' to your INSTALLED_APPS setting. Once you have configured your installation, run manage.py migrate to install the single database table that stores session data.
Check if your session Middleware is enabled
Sessions are implemented via a piece of middleware. The default settings.py created by django-admin startproject has SessionMiddleware activated. To enable session functionality, edit the MIDDLEWARE_CLASSES setting and make sure it contains 'django.contrib.sessions.middleware.SessionMiddleware'.
Update 2: Test the session
Maybe modify a style existing endpoint as follows and see if you are able to store values and persist them in session :
test_keys = request.session.get('test_keys', [])
test_keys.append(random.randint())
request.session['test_keys'] = test_keys
return Response(request.session.get('test_keys', []))
You should see that each time you hit the api, you get a list with one new integer in it in addition to all past values. Lmk how this goes.
I've followed the guide in the queryset documentation as per (https://docs.djangoproject.com/en/1.10/ref/models/querysets/#update-or-create) but I think im getting something wrong:
my script checks against an inbox for maintenance emails from our ISP, and then sends us a calendar invite if you are subscribed and adds maintenance to the database.
Sometimes we get updates on already planned maintenance, of which i then need to update the database with the new date and time, so im trying to use "update or create" for the queryset, and need to use the ref no from the email to update or create the record
#Maintenance
if sender.lower() == 'maintenance#isp.com':
print 'Found maintenance in mail: {0}'.format(subject)
content = Message.getBody(mail)
postcodes = re.findall(r"[A-Z]{1,2}[0-9R][0-9A-Z]? [0-9][A-Z]{2}", content)
if postcodes:
print 'Found Postcodes'
else:
error_body = """
Email titled: {0}
With content: {1}
Failed processing, could not find any postcodes in the email
""".format(subject,content)
SendMail(authentication,site_admins,'Unprocessed Email',error_body)
Message.markAsRead(mail)
continue
times = re.findall("\d{2}/\d{2}/\d{4} \d{2}:\d{2}", content)
if times:
print 'Found event Times'
e_start_time = datetime.strftime(datetime.strptime(times[0], "%d/%m/%Y %H:%M"),"%Y-%m-%dT%H:%M:%SZ")
e_end_time = datetime.strftime(datetime.strptime(times[1], "%d/%m/%Y %H:%M"),"%Y-%m-%dT%H:%M:%SZ")
subscribers = []
clauses = (Q(site_data__address__icontains=p) for p in postcodes)
query = reduce(operator.or_, clauses)
sites = Circuits.objects.filter(query).filter(circuit_type='MPLS', provider='KCOM')
subject_text = "Maintenance: "
m_ref = re.search('\[(.*?)\]',subject).group(1)
if not len(sites):
#try use first part of postcode
h_pcode = postcodes[0].split(' ')
sites = Circuits.objects.filter(site_data__postcode__startswith=h_pcode[0]).filter(circuit_type='MPLS', provider='KCOM')
if not len(sites):
#still cant find a site, send error
error_body = """
Email titled: {0}
With content: {1}
I have found a postcode, but could not find any matching sites to assign this maintenance too, therefore no meeting has been sent
""".format(subject,content)
SendMail(authentication,site_admins,'Unprocessed Email',error_body)
Message.markAsRead(mail)
continue
else:
#have site(s) send an invite and create record
for s in sites:
create record in circuit maintenance
maint = CircuitMaintenance(
circuit = s,
ref = m_ref,
start_time = e_start_time,
end_time = e_end_time,
notes = content
)
maint, CircuitMaintenance.objects.update_or_create(ref=m_ref)
#create subscribers for maintenance
m_ref, is the unique field that will match the update, but everytime I run this in tests I get
sites_circuitmaintenance.start_time may not be NULL
but I've set it?
If you want to update certain fields provided that a record with certain values exists, you need to explicitly provide the defaults as well as the field names.
Your code should look like this:
CircuitMaintenance.objects.update_or_create(default=
{'circuit' : s,'start_time' : e_start_time,'end_time' : e_end_time,'notes' : content}, ref=m_ref)
The particular error you are seeing is because update_or_create is creating an object because one with rer=m_ref does not exist. But you are not passing in values for all the not null fields. The above code will fi that.
I am developing a web application for managing customers. So I have a Customer entity which is made up by usual fields such as first_name, last_name, age etc.
I have a page where these customers are shown as a table. In the same page I have a search field, and I'd like to filter customers and update the table while the user is typing a something in the search field, using Ajax.
Here is how it should work:
Figure 1: The main page showing all of the customers:
Figure 2: As long as the user types letter "b", the table is updated with the results:
Given that partial text matching is not supported in GAE, I have tricked and implemented it arising from what is shown here: TL;DR: I have created a Customers Index, that contains a Search Document for every customer(doc_id=customer_key). Each Search Document contains Atom Fields for every customer's field I want to be able to search on(eg: first_name, last_name): every field is made up like this: suppose the last_name is Berlusconi, the field is going to be made up by these Atom Fields "b" "be" "ber" "berl" "berlu" "berlus" "berlusc" "berlusco" "berluscon" "berlusconi".
In this way I am able to perform full text matching in a way that resembles partial text matching. If I search for "Be", the Berlusconi customer is returned.
The search is made by Ajax calls: whenever a user types in the search field(the ajax is dalayed a little bit to see if the user keeps typing, to avoid sending a burst of requests), an Ajax call is made with the query string, and a json object is returned.
Now, things were working well in debugging, but I was testing it with a few people in the datastore. As long as I put many people, search looks very slow.
This is how I create search documents. This is called everytime a new customer is put to the datastore.
def put_search_document(cls, key):
"""
Called by _post_put_hook in BaseModel
"""
model = key.get()
_fields = []
if model:
_fields.append(search.AtomField(name="empty", value=""),) # to retrieve customers when no query string
_fields.append(search.TextField(name="sort1", value=model.last_name.lower()))
_fields.append(search.TextField(name="sort2", value=model.first_name.lower()))
_fields.append(search.TextField(name="full_name", value=Customer.tokenize1(
model.first_name.lower()+" "+model.last_name.lower()
)),)
_fields.append(search.TextField(name="full_name_rev", value=Customer.tokenize1(
model.last_name.lower()+" "+model.first_name.lower()
)),)
# _fields.append(search.TextField(name="telephone", value=Customer.tokenize1(
# model.telephone.lower()
# )),)
# _fields.append(search.TextField(name="email", value=Customer.tokenize1(
# model.email.lower()
# )),)
document = search.Document( # create new document with doc_id=key.urlsafe()
doc_id=key.urlsafe(),
fields=_fields)
index = search.Index(name=cls._get_kind()+"Index") # not in try-except: defer will catch and retry.
index.put(document)
#staticmethod
def tokenize1(string):
s = ""
for i in range(len(string)):
if i > 0:
s = s + " " + string[0:i+1]
else:
s = string[0:i+1]
return s
This is the search code:
#staticmethod
def search(ndb_model, query_phrase):
# TODO: search returns a limited number of results(20 by default)
# (See Search Results at https://cloud.google.com/appengine/docs/python/search/#Python_Overview)
sort1 = search.SortExpression(expression='sort1', direction=search.SortExpression.ASCENDING,
default_value="")
sort2 = search.SortExpression(expression='sort2', direction=search.SortExpression.ASCENDING,
default_value="")
sort_opt = search.SortOptions(expressions=[sort1, sort2])
results = search.Index(name=ndb_model._get_kind() + "Index").search(
search.Query(
query_string=query_phrase,
options=search.QueryOptions(
sort_options=sort_opt
)
)
)
print "----------------"
res_list = []
for r in results:
obj = ndb.Key(urlsafe=r.doc_id).get()
print obj.first_name + " "+obj.last_name
res_list.append(obj)
return res_list
Did anyone else had my same experience? If so, how have you solved it?
Thank you guys very much,
Marco Galassi
EDIT: names, email, phone are obviously totally invented.
Edit2: I have now moved to TextField, who look a little bit faster, but the problem still persist
I have a bit of code that is causing my page to load pretty slow (49 queries in 128 ms). This is the landing page for my site -- so it needs to load snappily.
The following is my views.py that creates a feed of latest updates on the site and is causing the slowest load times from what I can see in the Debug toolbar:
def product_feed(request):
""" Return all site activity from friends, etc. """
latestparts = Part.objects.all().prefetch_related('uniparts').order_by('-added')
latestdesigns = Design.objects.all().order_by('-added')
latest = list(latestparts) + list(latestdesigns)
latestupdates = sorted (latest, key = lambda x: x.added, reverse = True)
latestupdates = latestupdates [0:8]
# only get the unique avatars that we need to put on the page so we're not pinging for images for each update
uniqueusers = User.objects.filter(id__in = Part.objects.values_list('adder', flat=True))
return render_to_response("homepage.html", {
"uniqueusers": uniqueusers,
"latestupdates": latestupdates
}, context_instance=RequestContext(request))
The query that causes the most time seem to be:
latest = list(latestparts) + list(latestdesigns) (25ms)
There is another one that's 17ms (sitewide annoucements) and 25ms (adding tagged items on each product feed item) respectively that I am also investigating.
Does anyone see any ways in which I can optimize the loading of my activity feed?
You never need more than 8 items, so limit your queries. And don't forget to make sure that added in both models is indexed.
latestparts = Part.objects.all().prefetch_related('uniparts').order_by('-added')[:8]
latestdesigns = Design.objects.all().order_by('-added')[:8]
For bonus marks, eliminate the magic number.
After making those queries a bit faster, you might want to check out memcache to store the most common query results.
Moreover, I believe adder is ForeignKey to User model.
Part.objects.distinct().values_list('adder', flat=True)
Above line is QuerySet with unique addre values. I believe you ment exactly that.
It saves you performing a subuery.