Turn the positions that spacy_match(), spacy_extract_entity() or
spacy_extract_nounphrases() return into an index of quanteda, as
quanteda::index() makes, so that quanteda::kwic() can show each of them
in its context. Making the index needs neither quanteda nor spaCy.
Arguments
- x
the output of
spacy_match(),spacy_extract_entity()orspacy_extract_nounphrases(): a data.frame with the columnsdoc_id,start_idandlength. It can also beNULL, whichspacy_extract_entity()andspacy_extract_nounphrases()return when they find nothing, and which gives an index with no rows.
Value
An object of class index, as quanteda::index() returns: a
data.frame with one row for each row of x, in the same order, and the
columns
docnamethe document, fromdoc_id, in Unicode's composed form (NFC), as the details describe;fromandtothe positions of the first and last token, fromstart_idandlength;patterna factor: thelabelof a match, theent_typeof an entity, or"nounphrase"for a noun phrase.
Details
The positions count spaCy's tokens from 1 in each document, with its
tokens of whitespace. They are the positions in
quanteda::as.tokens(spacy_parse(x)) for the same texts and the same
pipeline, and not those in quanteda::tokens(x), which are
quanteda's tokens, or in the output of spacy_tokenize(), which by
default leaves out whitespace. So give quanteda::kwic() the tokens of
spacy_parse(), as in the examples. quanteda::kwic() does not check
that an index was made from the tokens it is given: with other tokens, it
shows whatever is at those positions.
spacy_parse() needs the sentences of each document, which come from the
parser of the language model or from a sentence segmenter. In a pipeline
that sets no sentence boundaries, spacy_match() still returns the
positions of its matches, but spacy_parse() fails, and so cannot give
the tokens for a concordance.
quanteda keeps the names of documents in Unicode's composed form
(NFC), so that an "e" followed by a combining accent becomes the single
accented character. as_index() does the same to doc_id when the
stringi package is installed, which quanteda needs: the
matches of a document whose name is not in that form then keep their
place in the concordance, which would otherwise drop them without a
message. Giving the same quanteda::corpus() to spacy_match() and to
spacy_parse() has the same effect, since the names in a corpus are
already in that form.
A match of a dependency pattern has one row for each of its nodes, and so
one row in the index for each node, also when two nodes are the same
token. quanteda::kwic() keeps the rows of the index in their order, and
drops those of documents that the tokens do not have. When the tokens
have every document, row i of the concordance is row i of x, so the
other columns of x, such as match_id and node, can be added to it.
The concordance shows the matches that it is given. Which matches
spacy_match() returns where they overlap is set by its overlap, whose
rule is not that of quanteda::kwic(x, pattern), as the details of
spacy_match() describe. A concordance of matches keeps a "New York"
inside another label's "New York Times", for instance, which
quanteda's own search for the two would drop.
Examples
# an index needs no spaCy: here, of matches written by hand
m <- data.frame(doc_id = c("d1", "d1"), label = c("city", "paper"),
start_id = c(2, 2), length = c(2, 3))
as_index(m)
#> docname from to pattern
#> 1 d1 2 3 city
#> 2 d1 2 4 paper
as_index(NULL)
#> [1] docname from to pattern
#> <0 rows> (or 0-length row.names)
if (FALSE) { # \dontrun{
spacy_initialize()
txt <- c(d1 = "The New York Times is a paper in New York.",
d2 = "Jane Smith wrote the book in Boston.")
toks <- quanteda::as.tokens(spacy_parse(txt))
# matches in context
m <- spacy_match(txt, quanteda::phrase(c("new york", "the book")))
quanteda::kwic(toks, index = as_index(m), window = 3)
# entities and noun phrases
quanteda::kwic(toks, index = as_index(spacy_extract_entity(txt)), window = 3)
quanteda::kwic(toks, index = as_index(spacy_extract_nounphrases(txt)),
window = 3)
spacy_finalize()
} # }