This is the model that has to be searched:
class BlockQuote(models.Model):
debate = models.ForeignKey(Debate, related_name='quotes')
speaker = models.ForeignKey(Speaker, related_name='quotes')
text = models.TextField()
I have around a thousand instances on the database on my laptop (with around 50000 on the production server)
I am creating a 'manage.py' function that will search through the database and returns all 'BlockQuote' objects whose textfield contains the keyword.
I am doing this with the Django's (1.11) Postgres search options in order to use the 'rank' attribute, which sounds like something that would come in handy. I used the official Django fulltext-search documentation for the code below
Yet when I run this code, it matches with all objects, regardless if BlockQuote.text actually contains the queryfield.
def handle(self, *args, **options):
vector = SearchVector('text')
query = options['query'][0]
Search_Instance = Search_Instance.objects.create(query=query)
set = BlockQuote.objects.annotate(rank=SearchRank(vector, query)).order_by('-rank')
for result in set:
match = QueryMatch.objects.create(quote=result, query=Search_Instance)
match.save()
Does anyone have an idea of what I am doing wrong?
I don't see you actually filtering ever.
BlockQuote.objects.annotate(...).filter(rank__gte=0.5)
Related
It is a very specific question regarding Flask-appbuilder. During my development, I found FAB's ModelView is suitable for admin role, but need more user logic handlers/views for complex designs.
There is a many to many relationship between devices and users, since each device could be shared between many users, and each user could own many device. So there is a secondary table called accesses, describes the access control between devices and users. In this table, I add "isHost" to just if the user owns the device. Therefore, we have two roles: host and (regular) user. However, these roles are not two roles defined as other applications, since one man can be either host or user in same time. In a very simple application, enforce the user to switch two roles are not very convinient. That makes things worse.
Anyway, I need design some custom handlers with traditional Flask/Jinja2 templates. For example:
class PageView(ModelView):
# FAB default URL: "/pageview/list"
datamodel = SQLAInterface(Page)
list_columns = ['name', 'date', 'get_url']
#expose("/p/<string:url>")
def p(self, url):
title = urllib.unquote(url)
r = db.session.query(Page).filter_by(name = title).first()
if r:
md = r.markdown
parser = mistune.Markdown()
body = parser(md)
return self.render_template('page.html', title = title, body = body)
else:
return self.render_template('404.html'), 404
Above markdown page URL is simple, since it is a seperate UI. But if I goes to DeviceView/AccountView/AccessView for list/show/add/edit operations. I realized that I need a unique styles of UI.
So, now how can I reuse the existing templates/widgets of FAB with custom sqlalchemy queries? Here is my code for DeviceView.
class DeviceView(ModelView):
datamodel = SQLAInterface(Device)
related_views = [EventView, AccessView]
show_template = 'appbuilder/general/model/show_cascade.html'
edit_template = 'appbuilder/general/model/edit_cascade.html'
#expose('/host')
#has_access
def host(self):
base_filters = [['name', FilterStartsWith, 'S'],]
#if there is not return, FAB will throw error
return "host view:{}".format(repr(base_filters))
#expose('/my')
#has_access
def my(self):
# A pure testing method
rec = db.session.query(Access).filter_by(id = 1).all()
if rec:
for r in rec:
print "rec, acc:{}, dev:{}, host:{}".format(r.account_id, r.device_id, r.is_host)
return self.render_template('list.html', title = "My Accesses", body = "{}".format(repr(r)))
else:
return repr(None)
Besides sqlalchemy code with render_template(), I guess base_filters can also help to define custom queries, however, I have no idea how to get query result and get them rendered.
Please give me some reference code or example if possible. Actually I have grep keywords of "db.session/render_template/expoaw"in FAB's github sources. But no luck.
I am trying to add a completion suggester to enable search-as-you-type for a search field in my Django app (using Elastic Search 5.2.x and elasticseach-dsl). After trying to figure this out for a long time, I am not able to figure yet how to bulk index the suggester. Here's my code:
class SchoolIndex(DocType):
name = Text()
school_type = Keyword()
name_suggest = Completion()
Bulk indexing as follows:
def bulk_indexing():
SchoolIndex.init(index="school_index")
es = Elasticsearch()
bulk(client=es, actions=(a.indexing() for a in models.School.objects.all().iterator()))
And have defined an indexing method in models.py:
def indexing(self):
obj = SchoolIndex(
meta = {'id': self.pk},
name = self.name,
school_type = self.school_type,
name_suggest = {'input': self.name } <--- # what goes in here?
)
obj.save(index="school_index")
return obj.to_dict(include_meta=True)
As per the ES docs, suggestions are indexed like any other field. So I could just put a few terms in the name_suggest = statement above in my code which will match the corresponding field, when searched. But my question is how to do that with a ton of records? I was guessing there would be a standard way for ES to automatically come up with a few terms that could be used as suggestions. For example: using each word in the phrase as a term. I could come up something like that on my own (by breaking each phrase into words) but it seems counter-intuitive to do that on my own since I'd guess there would already be a default way that the user could further tweak if needed. But couldn't find anything like that on SO/blogs/ES docs/elasticsearch-dsl docs after searching for quite sometime. (This post by Adam Wattis was very helpful in getting me started though). Will appreciate any pointers.
I think I figured it out (..phew)
In the indexing function, I need to use the following to enable to the prefix completion suggester:
name_suggest = self.name
instead of:
name_suggest = {'input': something.here }
which seems to be used for more custom cases.
Thanks to this video that helped!
I have a setup of Django+Haystack+Elasticsearch. I was making a search with q="Çakıcı'nın İlk Kurşunu". And no result was appearing. As I look into the text field for this object in Elasticsearch index I see the text is stored as "Çakıcı'nın İlk Kurşunu". Single quote was converted to html safe notation.
So I assume when I create index my db objects are converted by Elasticsearch.
In order to overcome this situation I have changed the q field by overwriting the clean_q method of Form class. I used the django.utils.html.escape method to escape single quote as elasticsearch stored in the index.
Here is my form code.
class BstBookSearchForm(SearchForm):
def no_query_found(self):
return self.searchqueryset.all()
def search(self):
#print self.cleaned_data.get('q')
# First, store the SearchQuerySet received from other processing. (the main work is run internally by Haystack here).
sqs = super(BstBookSearchForm, self).search()
# if something goes wrong
if not self.is_valid():
return self.no_query_found()
# you can then adjust the search results and ask for instance to order the results by title
#sqs = sqs.order_by(book_name)
return sqs
def clean_q(self):
return escape(self.cleaned_data['q'])
Is there a built in way to do this conversion, by Django, by Haystack or by Elasticsearch.
Would you consider my solution a bad practice?
I'm trying to render a form with a combo that shows related entities. Therefore I'm using a ModelChoiceField.
This approach works well, until I needed to limit which entities to show. If I use a simple query expression it also works well, but things break if I use a raw SQL query.
So my code that works, sets the queryset to a filter expression.
class ReservationForm(forms.Form):
location_time_slot = ModelChoiceField(queryset=LocationTimeSlot.objects.all(), empty_label="Select your prefered time")
def __init__(self,*args,**kwargs):
city_id = kwargs.pop("city_id") # client is the parameter passed from views.py
super(ReservationForm, self).__init__(*args,**kwargs)
# TODO: move this to a manager
self.fields['location_time_slot'].queryset = LocationTimeSlot.objects.filter(city__id = city_id )
BUT, if I change that to a raw query I start having problems. Code that does not work:
class ReservationForm(forms.Form):
location_time_slot = ModelChoiceField(queryset=LocationTimeSlot.objects.all(), empty_label="Select your prefered time")
def __init__(self,*args,**kwargs):
city_id = kwargs.pop("city_id") # client is the parameter passed from views.py
super(ReservationForm, self).__init__(*args,**kwargs)
# TODO: move this to a manager
query = """SELECT ts.id, ts.datetime_to, ts.datetime_from, ts.available_reserves, l.name, l.'order'
FROM reservations_locationtimeslot AS ts
INNER JOIN reservations_location AS l ON l.id = ts.location_id
WHERE l.city_id = %s
AND ts.available_reserves > 0
AND ts.datetime_from > datetime() """
time_slots = LocationTimeSlot.objects.raw(query, [city_id])
self.fields['location_time_slot'].queryset = time_slots
The first error I get when trying to render the widget is: 'RawQuerySet' object has no attribute 'all'
I could solve that one thanks to one of the commets in enter link description here, by doing:
time_slots.all = time_slots.__iter__ # Dummy fix to allow default form rendering with raw SQL
But now I'm getting something similar when posting the form:
'RawQuerySet' object has no attribute 'get'
Is there a proper way to prepare a RawQuerySet to be used by ModelChoiceField?
Thanks!
Are you sure you actually need a raw query there? Just looking at that query, I can't see any reason you can't just do it with filter(location__city=city_id, available_reserves__gte=0, datetime_from__gt=datetime.datetime.now()).
Raw query sets are missing a number of methods that are defined on conventional query sets, so just dropping them in place isn't likely to work without writing your own definitions for all those methods.
I temporarily fixed the problem adding the missing methods.
The way I'm currently using the ModelChoiceField I only needed to add the all() and get() methods, but in different scenarios you might need to add some other methods as well. Also this is not a perfect solution because:
1) Defining the get method this way migth produce incorrect results. I think the get() method is used to validate that the selected option is within the options returned by all(). The way I temporarily implemented it only validates that the id exists in the table.
2) I guess the get method is less performant specified this way.
If anyone can think of a better solution, please let me know.
So my temporary solution:
class LocationTimeSlotManager(models.Manager):
def availableSlots(self, city_id):
query = """SELECT ts.id, ts.datetime_to, ts.datetime_from, ts.available_reserves, l.name, l.'order'
FROM reservations_locationtimeslot AS ts
.....
.....
MORE SQL """
time_slots = LocationTimeSlot.objects.raw(query, [city_id])
# Dummy fix to allow default form rendering with raw SQL
time_slots.all = time_slots.__iter__
time_slots.get = LocationTimeSlot.objects.get
return time_slots
Say I have a model Person for which there is a PersonIndex class in search_indexes.py that makes all fields of it searchable. How can I make a search within only those entries where say the has_title field is True?
I tried the following, but it just searches among all entries, not just the ones where has_title is True:
srch = request.GET.get('search', "")
sqs = SearchQuerySet().filter(has_title=True)
clean_query = sqs.query.clean(srch)
results = sqs.raw_search(clean_query)
I am using Whoosh 2.4.1, Django-haystack 1.2.7 and Django 1.4.
Use filter(content=clean_query) instead of raw_search(clean_query). See here for more details.