Wagtail search backend with Postgres - using unaccent with English search config - django

Question

I'm not sure if this is a fault with Wagtail's search engine, or if I'm missing a setting/configuration step along the line.
I'm working with a couple of European sites where they work in English as a common language but there are plenty of people and place names in the content with extended latin characters.
On the db server, I added the unaccent extension and created a new search config based on the built-in english with unaccent added (using the example from postgres):
CREATE TEXT SEARCH CONFIGURATION english_extended ( COPY = english );
ALTER TEXT SEARCH CONFIGURATION english_extended
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, english_stem;
And backends:
WAGTAILSEARCH_BACKENDS = {
'default': {
'BACKEND': 'wagtail.search.backends.database',
'SEARCH_CONFIG': 'english_extended',
},
'es': {
'BACKEND': 'wagtail.search.backends.database',
'SEARCH_CONFIG': 'spanish',
},
}
I have a page title with the word Bodø. To test, I tried a default search from the psql command line:
# select title from wagtailcore_page where to_tsvector(title) ## to_tsquery('Bodo');
title
-------
(0 rows)
No results, as expected. Then again using the new search config:
# select title from wagtailcore_page where to_tsvector('english_extended', title) ## to_tsquery('Bodo');
title
----------------------------------------------------
The old dock at Kjerringøy, Bodø, Nordland, Norway
(1 row)
Page now found, so the unaccent is being processed in the new config.
So far, so good. Unfortunately, in Wagtail, when I search, the unaccent doesn't happen, any word with an accented character seems to get ignored:
In [1]: from wagtail.search.backends import get_search_backend
In [2]: s=get_search_backend()
In [3]: s.config
Out[3]: 'english_extended'
In [4]: s.search('bodø', Page.objects.all())
Out[4]: <SearchResults []>
In [5]: s.search('bodo', Page.objects.all())
Out[5]: <SearchResults []>
In [6]: s.search('Bodø', Page.objects.all())
Out[6]: <SearchResults []>
In [7]: s.search('dock', Page.objects.all())
Out[7]: <SearchResults [<Page: The old dock at Kjerringøy, Bodø, Nordland, Norway>]>
If I use the Spanish backend for example, I can search unaccent terms without any config:
In [1]: from wagtail.search.backends import get_search_backend
In [2]: s=get_search_backend('en')
In [3]: s.search('jeremy' , Page.objects.all())
Out[3]: <SearchResults []>
In [4]: s.search('Jérémy' , Page.objects.all())
Out[4]: <SearchResults []>
In [5]: s=get_search_backend('es')
In [6]: s.config
Out[6]: 'spanish'
In [7]: s.search('jeremy' , Page.objects.all())
Out[7]: <SearchResults [<Page: Jérémy in the title>]>
Is there a step I'm missing or an additional property in the backend definition missing? Or is this bug/'undocumented feature' of Wagtail's search engine?

Answer 1

So the answer to this, if not the solution, is that Wagtail search doesn't support multiple wagtail_backends using the same backend with differing search configs.
All that happens is the index gets rebuilt for each one it encounters, leaving the index built with the last in the list. In the case above, it ended up indexed with 'spanish' which is why none of the unaccenting was happening.
The good news is, with the english_extended last in line, the unaccenting works perfectly, the page containing Bodø can be found with bodø or bodo.
Jérémy álphabét Bodø Strandå Hôtel cañon is indexed as 'alphabet':2B 'bodo':3B 'canon':6B 'hotel':5B 'jeremi':1B 'stranda':4B
We are going walking in the mountains is indexed as 'go':3B 'mountain':7B 'today':8B 'walk':4B
So unaccenting and stop/stem words are treated correctly.
The bad news is there seems to be no way to have a backend for (say) English and one for Spanish.
Issue raised on Wagtail github.