Create patterns for rule-based named entities

Turn phrases, spaCy-style pattern lists or a data.frame of patterns into a spacyr_entity_patterns object: a checked set of rules for spaCy's entity_ruler, which finds named entities by matching phrases or sequences of tokens. Creating the object needs neither Python nor spaCy; pass it to spacy_initialize() as entity_patterns to add the rules to the pipeline.

Usage

as_entity_patterns(x, ...)

# S3 method for class 'character'
as_entity_patterns(x, label = NULL, id = NULL, case_insensitive = FALSE, ...)

# S3 method for class 'list'
as_entity_patterns(x, case_insensitive = FALSE, ...)

# S3 method for class 'data.frame'
as_entity_patterns(x, case_insensitive = FALSE, concatenator = NULL, ...)

# S3 method for class 'dictionary2'
as_entity_patterns(
  x,
  case_insensitive = TRUE,
  valuetype = c("glob", "fixed", "regex"),
  ...
)

Arguments

x

the patterns to convert:

a character vector of phrases, each matched exactly as a sequence of spaCy's tokens. Labels come from the names of x, as in c(ORG = "Acme Widgets"), or from label.
a list of patterns in spaCy's own structure, each a list with the elements label, pattern and optionally id. pattern is either a single phrase or a list of token patterns, one per token, each a named list of token attributes such as list(LOWER = "dr"). See spaCy's rule-based matching for the attributes and operators available. A single pattern may also be given on its own, rather than in a list of patterns.
a data.frame with the columns label and pattern, and optionally id and case_insensitive. pattern is a character column of phrases, or a list column holding phrases or token patterns.
the entities found by spacy_extract_entity(), a data.frame with the columns text and ent_type, which become phrases and their labels. This turns the entities found in one set of texts into rules for others.
the entities found by entity_extract(), with the columns entity and entity_type, given with the concatenator that joined their tokens.
a quanteda dictionary, made by quanteda::dictionary(). Its top-level keys are the labels, with the values of any nested keys under them. A value without wildcards becomes a phrase. A value with wildcards (* for any characters, ? for one) becomes a token pattern with one token for each part separated by spaces, as quanteda matches dictionary values to tokens, so that "dr * who" matches three tokens.
NULL, as spacy_extract_entity() returns when it finds no entities, which gives an empty set of patterns.
a spacyr_entity_patterns object, which is checked again, since it may have been edited as a data.frame.

...

not used.

label

character; the entity label of each phrase, such as "PERSON" or "ORG", either one for all phrases or one per phrase. If NULL (the default), the names of x are used.

id

character; an optional ID for each phrase, either one for all phrases or one per phrase. spaCy gives the ID of the pattern that matched as each entity's ent_id_.

case_insensitive

logical; if TRUE, phrases match regardless of case. The default, FALSE, matches them exactly, as spaCy does, except for a dictionary, where it is TRUE, as in quanteda, and also applies to the parts of values with wildcards. It does not apply to token patterns, which set case through the attribute they match, such as ORTH (exact) or LOWER (any case). For a list, it applies to the patterns that do not set case_insensitive themselves; for a data.frame, it applies when there is no case_insensitive column.

concatenator

for the output of entity_extract(), the character(s) that joined the tokens of each entity, which are replaced by spaces. It must be given, since a token that itself contains the concatenator, such as an underscore, is changed too; the output of spacy_extract_entity() keeps the text of each entity as it was, and is the better input.

valuetype

for a dictionary, how its values are matched: "glob" (the default) for wildcards, "fixed" for exact phrases, or "regex" for regular expressions, as in quanteda::tokens_lookup(). With "regex", each part of a value separated by spaces is a regular expression for one of spaCy's tokens, in the dialect of Python's re module, not of R: for instance, "^\\p{L}+$" works in quanteda but stops spacy_initialize() with the error "bad escape \p". Since quanteda::dictionary() lowercases values by default, which changes escapes such as "\\D" to "\\d", make a dictionary of regular expressions with tolower = FALSE.

Value

A spacyr_entity_patterns object: a data.frame with one row per pattern and the columns

label the entity label;
pattern a list column, holding for each pattern a single phrase or a list of token patterns;
id the pattern's ID, or NA;
case_insensitive for phrases, whether they match regardless of case; NA for token patterns.

Duplicated patterns are removed.

Details

Token patterns are checked for their structure only: that each is a list with one named list per token. spaCy checks the attribute names, operators and values when the patterns are added to a pipeline, and reports any errors then.

Two conversions make patterns from R reach spaCy in the form it expects. The values of spaCy's set operators (IN, NOT_IN, IS_SUBSET, IS_SUPERSET and INTERSECTS) are always stored as lists, so that list(LOWER = list(IN = "dr")) stays a list of one value, rather than becoming a single string. Whole numbers stored as doubles, such as 3 in list(LENGTH = 3), become integers, since spaCy does not accept decimal numbers in token patterns.

Phrases, including dictionary values without wildcards, are split into tokens by the pipeline's own tokenizer, and some of its rules depend on case: spaCy's English models keep "Inc." as one token, but split "inc." into "inc" and ".". So write phrases in their usual case, even with case_insensitive = TRUE, and make dictionaries with quanteda::dictionary(tolower = FALSE), since by default it lowercases their values.

Patterns match spaCy's tokens, not characters. For instance, spaCy's English models make "Prof." a single token, so a pattern for academic titles needs to list both "prof" and "prof.", to match titles written with and without a full stop.

Examples

# phrases, labelled by their names or by `label`
as_entity_patterns(c(ORG = "Acme Widgets", GPE = "San Francisco"))
#> Entity patterns: 2 patterns (ORG: 1, GPE: 1)
#>   ORG  "Acme Widgets"
#>   GPE  "San Francisco"
as_entity_patterns(c("Acme Widgets", "Globex"), label = "ORG")
#> Entity patterns: 2 patterns (ORG: 2)
#>   ORG  "Acme Widgets"
#>   ORG  "Globex"

# a token pattern for people with academic titles: a title, followed by
# the first token of a person the model recognises (ENT_IOB "B") and the
# rest of that person (ENT_IOB "I"), so that it extends one entity without
# running into a person who follows it
titles <- c("dr", "dr.", "prof", "prof.")
as_entity_patterns(list(
  label = "PERSON",
  pattern = list(list(LOWER = list(IN = titles)),
                 list(ENT_TYPE = "PERSON", ENT_IOB = "B"),
                 list(ENT_TYPE = "PERSON", ENT_IOB = "I", OP = "*"))
))
#> Entity patterns: 1 pattern (PERSON: 1)
#>   PERSON  [LOWER in (dr, dr., prof, prof.)] [ENT_TYPE = PERSON, ENT_IOB = B] [ENT_TYPE = PERSON, ENT_IOB = I]*

# a data.frame of patterns
as_entity_patterns(data.frame(label = c("ORG", "ORG"),
                              pattern = c("Acme Widgets", "Globex")))
#> Entity patterns: 2 patterns (ORG: 2)
#>   ORG  "Acme Widgets"
#>   ORG  "Globex"

# a quanteda dictionary, with labels from its keys
if (requireNamespace("quanteda", quietly = TRUE)) {
  dict <- quanteda::dictionary(list(ORG = c("acme widgets", "globex*"),
                                    PERSON = "dr * who"))
  as_entity_patterns(dict)
}
#> Entity patterns: 3 patterns (ORG: 2, PERSON: 1)
#>   ORG     "acme widgets" (any case)
#>   ORG     [LOWER ~ /^globex.*$/]
#>   PERSON  [LOWER = dr] [LOWER ~ /^.*$/] [LOWER = who]

# entities found by spaCy, to use as rules for other texts
if (FALSE) { # \dontrun{
spacy_initialize()
found <- spacy_extract_entity("Acme Widgets hired Jane Smith.")
as_entity_patterns(found)
spacy_finalize()
} # }

# combine sets of patterns
c(as_entity_patterns(c(ORG = "Globex")),
  as_entity_patterns(c(PERSON = "Jane Smith")))
#> Entity patterns: 2 patterns (ORG: 1, PERSON: 1)
#>   ORG     "Globex"
#>   PERSON  "Jane Smith"

Usage

Arguments

Value

Details

See also

Examples