Adding REGEX entities to SpaCy's Matcher - regex

Question

I am trying to add entities defined by regular expressions to SpaCy's NER pipeline. Ideally, I should be able to use any regular expression loaded from a json file with a defined entity type. As an example, I am trying to execute the code below.
The code below shows what I am trying to do, following an example given on Spacy's discussion about custom attributes using regular expressions. I have tried calling the 'set_extension' method in various ways (to Doc, Span, Token), but to no avail. I'm not even sure what I should be setting them to.
nlp = spacy.load("en_core_web_lg")
matcher = Matcher(nlp.vocab)
pattern = [{"_": {"country": {"REGEX": "^[Uu](\.?|nited) ?[Ss](\.|tates)$"}}}]
matcher.add("US", None, pattern)
doc = nlp(u"I'm from the United States.")
matches = matcher(doc)
for match_id, start, end in matches:
string_id = nlp.vocab.strings[match_id]
span = doc[start:end]
print(match_id, string_id, start, end, span.text)
I expect match_id, string_id 3 4 United States to be printed out.
Instead, I am getting AttributeError: [E046] Can't retrieve unregistered extension attribute 'country'. Did you forget to call the 'set_extension' method?

Answer 1

There's documentation around the extension attributes here: https://spacy.io/usage/processing-pipelines#custom-components-attributes
Basically you'll have to define this country variable as an extension attribute, something like this:
Token.set_extension("country", default="")
However, in the code you cited you're never actually setting the _.country attribute to any token (or span), so they're all still at default value, and the matcher will never be able to get a match on them. The line you cited:
pattern = [{"_": {"country": {"REGEX": "^[Uu](\.?|nited) ?[Ss](\.?|tates)$"}}}]
Tries to match the United States regex on the custom attribute values, instead of on the doc text, as you expect (I think).
One solution is just to run the reg-exps on the texts directly:
nlp = spacy.load("en_core_web_lg")
matcher = Matcher(nlp.vocab)
pattern = [{"TEXT": {"REGEX": "^[Uu](\.?|nited)$"}},
{"TEXT": {"REGEX": "^[Ss](\.?|tates)$"}}]
matcher.add("US", None, pattern)
doc = nlp(u"I'm from the United States.")
matches = matcher(doc)
for match_id, start, end in matches:
string_id = nlp.vocab.strings[match_id]
span = doc[start:end]
print(match_id, string_id, start, end, span.text)
Which outputs
15397641858402276818 US 4 6 United States
Then you can use those matches to e.g. set a custom attribute on the Span's or Token's (in this case Span, because your match is potentially involving multiple tokens)