The spacy_parse() function calls spaCy to both tokenize and tag the
texts, and returns a data.frame of the results, with one row per token. The
function provides options for universal (pos) and detailed (tag)
part-of-speech tags, lemmatization (lemma), named entity recognition
(entity), dependency parsing (dependency), noun phrases (nounphrase),
the position of each token in its text (offsets), and any other token
attributes from spaCy (additional_attributes).
Usage
spacy_parse(
x,
pos = TRUE,
tag = FALSE,
lemma = TRUE,
entity = TRUE,
dependency = FALSE,
nounphrase = FALSE,
multithread,
additional_attributes = NULL,
n_process = 1L,
batch_size = NULL,
offsets = FALSE,
...
)Arguments
- x
a character object, a quanteda corpus, or a TIF-compliant corpus data.frame (see https://github.com/ropenscilabs/tif)
- pos
logical whether to return universal dependency POS tagset (https://universaldependencies.org/u/pos/)
- tag
logical whether to return detailed part-of-speech tags; the English language models use the OntoNotes 5 version of the Penn Treebank tag set (https://spacy.io/models/en). The tag sets of other language models are listed under "Label scheme" on https://spacy.io/models.
- lemma
logical; include lemmatized tokens in the output (the quality of lemmatization depends on the language model)
- entity
logical; if
TRUE, report named entities- dependency
logical; if
TRUE, analyse and tag dependencies- nounphrase
logical; if
TRUE, analyse and tag noun phrases tags- multithread
deprecated; it no longer has any effect, and gives a warning if supplied. Documents are always processed in batches with spaCy's
nlp.pipe(); usen_processandbatch_sizeinstead.- additional_attributes
a character vector; this option is for extracting additional attributes of tokens from spaCy. When the names of attributes are supplied, the output data.frame will contain additional variables corresponding to the names of the attributes. For instance, when
additional_attributes = c("is_punct"), the output will include an additional variable namedis_punct, which is a Boolean (in R, logical) variable indicating whether the token is a punctuation. A full list of available attributes is available from https://spacy.io/api/token#attributes.- n_process
the number of processes for spaCy's
nlp.pipe()to use:1(the default) processes the documents in the current R session's Python process, a larger number starts that many Python processes, and-1uses all CPUs. Starting the processes and loading the model in each takes time, so more than one process helps only for large numbers of documents. A custom pipeline component must be defined in an importable Python module to be used in other processes.- batch_size
the number of documents that spaCy's
nlp.pipe()processes together;NULL(the default) uses the pipeline's default (nlp.batch_size). Larger batches can be faster, but use more memory.- offsets
logical; if
TRUE, return the position of each token in its text as it was given inx, in two columns aftertoken:start, the position of the token's first character, andend, that of its last. Positions are counted from 1 in characters as R counts them, whitespace included, so thatsubstring(text, start, end)gives the tokens of a text. Some of spaCy's tokenizers, such as the one for Korean, make a document whose whitespace differs from that of the text; the positions are still those in the text. If a tokenizer changes anything else, so that a token is not found in the text or the text has more than whitespace outside the tokens, this is an error.- ...
not used directly
Value
a data.frame of tokenized, parsed, and annotated tokens, of class
spacyr_parsed. It keeps the names of all the documents of x, also of
those in which spaCy found no tokens, and the document variables of a
quanteda corpus, for the quanteda methods such
as quanteda::as.tokens().
Details
The components of spaCy's pipeline whose results the options do not ask
for are left out of the call, which makes it faster: the named entity
recognizer with entity = FALSE; the lemmatizer with lemma = FALSE; and
the tagger and the attribute ruler when pos, tag, lemma and
nounphrase are all FALSE. The output is the same as when every
component runs, so a component stays whenever it sets something that the
output has, or that a later component may read:
The parser always runs, since it sets the sentences.
An attribute ruler stays if any of its rules sets something that is returned. That of spaCy's English models also sets the dependency of whitespace tokens, so with
dependency = TRUEit stays, and so does the tagger, whose tags its rules read.With entity rules from
spacy_initialize(entity_patterns = )andentity = TRUE, the components that the rules may read stay.A component stays if the model of a later component reads what it sets, as an entity recognizer that was trained to use the part of speech does. The models of spaCy's own pipelines read only the text of the tokens. What a model reads is told from its settings; a model that is not built by spaCy's own functions is taken to read everything.
Nothing is left out when
additional_attributesare asked for, or when the pipeline has a component that is not one of spaCy's standard ones.Nothing is left out either when the pipeline's tokenizer is not spaCy's own. Such a tokenizer can give the documents a function of its own to find the sentences, which may read any annotation.
With
nounphrase = TRUE, nothing is left out either when the pipeline finds its noun phrases by a function that is not spaCy's own for the language, which may read any annotation.
See the article "Improving Performance" on the package website.
Examples
if (FALSE) { # \dontrun{
spacy_initialize()
# See Chap 5.1 of the NLTK book, http://www.nltk.org/book/ch05.html
txt <- "And now for something completely different."
spacy_parse(txt)
spacy_parse(txt, pos = TRUE, tag = TRUE)
spacy_parse(txt, dependency = TRUE)
txt2 <- c(doc1 = "The fast cat catches mice.\\nThe quick brown dog jumped.",
doc2 = "This is the second document.",
doc3 = "This is a \\\"quoted\\\" text." )
spacy_parse(txt2, entity = TRUE, dependency = TRUE)
txt3 <- "We analyzed the Supreme Court with three natural language processing tools."
spacy_parse(txt3, entity = TRUE, nounphrase = TRUE)
spacy_parse(txt3, additional_attributes = c("like_num", "is_punct"))
# the position of each token in the text
parsed <- spacy_parse(txt3, offsets = TRUE)
substring(txt3, parsed$start, parsed$end)
} # }