Changelog
Source:NEWS.md
spacyr (development version)
Upgrading from 1.4.0
Most code written for 1.4.0 runs unchanged. These changes can need attention; each is described in full below.
- spacyr now requires R 3.6.0 or later, and spaCy 3.0.0 or later with Python 3.
- The quanteda methods for the output of
spacy_parse(), such asas.tokens()andntoken(), are now part of spacyr and no longer of quanteda. They keep documents that have no tokens, restore the document variables of a corpus, and return counts in the order of the documents. To use them on a parse that was saved to a file, load spacyr as well as quanteda. - The output of
spacy_parse()has a new attribute,documents, so it is no longeridentical()to a data.frame with the same columns. - The
multithreadargument is deprecated and has no effect: usen_processandbatch_size. -
process_document()and theget_*()functions are deprecated: usespacy_parse(). -
spacy_installed()is deprecated: usespacy_is_installed(), the same function under a new name. -
spacyr no longer creates the Python variables
nlp,spobj,textsand others in Python’s__main__namespace.
New features
- New function
entity_visualize()shows parsed documents with their named entities highlighted and labelled, as HTML made by displaCy, spaCy’s visualizer (#214). It takes the output ofspacy_parse(), asentity_extract()andentity_consolidate()do, or that ofentity_consolidate(), and shows the entities of the parse without processing the documents again; these include entities from rules given tospacy_initialize()asentity_patterns. Withspacy_parse(x, offsets = TRUE)the tokens are spaced as they were in the text; otherwise there is a space between every two tokens, and a message, given once in a session, says so.entitieschooses the entity labels to highlight andcolorssets the colours of labels. The result is one HTML fragment for all the documents, in their order, which can be put into a page, an R Markdown document or a Shiny app;view = TRUEalso opens it in the browser. - New function
as_entity_patterns()creates patterns for rule-based named entities, as used by spaCy’sentity_ruler, from phrases, lists of patterns in spaCy’s own structure, or a data.frame (#234). Patterns are checked for their structure, and converted so that they reach spaCy in the form it expects: the values of spaCy’s set operators (IN,NOT_IN,IS_SUBSET,IS_SUPERSETandINTERSECTS) stay lists even when they hold a single value or none, other empty lists become empty dictionaries, and whole numbers become integers. Phrases can match regardless of case withcase_insensitive = TRUE. The new functionsentity_patterns_read()andentity_patterns_write()read patterns from, and write them to, spaCy’s JSONL format, JSON arrays, or (for reading) a text file of phrases; reading and writing JSON needs the jsonlite package, now suggested. Pass the patterns tospacy_initialize()with its new argumententity_patterns. -
as_entity_patterns()also converts the entities found byspacy_extract_entity(), so that entities found in one set of texts can be rules for others, and those found byentity_extract(), given theconcatenatorthat joined their tokens. It converts quanteda dictionaries too, labelling patterns by their top-level keys and matching regardless of case by default. Dictionary values with wildcards (valuetype = "glob", the default) become token patterns with a regular expression for each part that has a wildcard, andvaluetype = "regex"and"fixed"are also supported. - A new article on the package website, “From Text to Auditable Annotations”, shows how the new functions work together, with references to the legislature in four inaugural addresses as the example: finding them with a quanteda dictionary and
spacy_match(), checking each match in a concordance withas_index()andquanteda::kwic(), revising the rules, finding the matches in the original text withoffsets, adding the rules as named entities, and highlighting them withentity_visualize(). - A new article on the package website, “Adding Rule-Based Named Entities”, shows how to add entities by rules: phrases, token patterns such as academic titles, rules and the model’s own entities, quanteda dictionaries, reusing the entities found, and saving rules to files.
- A new article on the package website, “Rule-Based Matching”, shows how to find sequences of tokens, phrases and dictionary values, and tokens connected in the dependency parse, with
spacy_match(), with passive verbs as the example, and what to keep in mind about language models and their parses (#204). - A new article on the package website, “Improving Performance”, shows what makes parsing faster or slower: choosing between
spacy_tokenize()andspacy_parse(), leaving out the named entity recognizer, the language model,batch_sizeandn_process, and long documents (#226). - New function
spacy_match()finds the tokens that match patterns, using spaCy’s rule-based matchers (#204). A pattern describes either a sequence of adjacent tokens, matched by spaCy’sMatcher, or tokens that are connected in the dependency parse, matched by itsDependencyMatcher: a passive verb with its subject and agent, for instance, wherever they are in the sentence.patternis a named list of patterns in spaCy’s own structure, which can hold both kinds: token patterns, with alternatives for a label given as a list of patterns, and dependency patterns, one for each label, whose nodes are named byRIGHT_ID. It can also be anas_entity_patterns()object, whose phrases are matched too, exactly or in any case, or a quanteda dictionary, with its wildcards. Patterns can use any token attribute, including those the model assigns, such asPOS,TAG,DEPandLEMMA. All patterns are matched in one pass over the documents, and the result is one data.frame with the same columns for every pattern:doc_id,match_id,label,node,sentence_id,token_id,text,start_idandlength. A sequence is one row, and a match of a dependency pattern is one row of length 1 for each node, with the node’s name innodeand the samematch_id; positions are as inspacy_parse()andspacy_extract_entity(), and nothing matching gives a data.frame with no rows.overlapsays which sequences to keep where matches of a label overlap:"longest", the default, which chooses among them asquanteda::tokens_lookup()does withnested_scope = "key","first", or"all"; it does not apply to dependency patterns, all of whose matches are returned. An argument thatspacy_match()does not use is an error, not ignored. The pipeline needs a dependency parse for a dependency pattern; a document without one is an error, not a document without matches. Unlikeentity_patterns, the patterns do not change the pipeline or a document’s entities.spacy_match()replacesspacy_extract_matches()andspacy_extract_dependency_matches(), which were in earlier development versions only and are removed, as is their argumentpatterns. -
spacy_match()takes words and phrases as quanteda’s functions do, with the new argumentsvaluetype = c("glob", "regex", "fixed")andcase_insensitive = TRUE.patterncan now be a character vector, in which each element describes one token (spacy_match(x, "tax*")); aquanteda::phrase()object or a list of character vectors, in which each element is a sequence with one string for each token (phrase("new york*")), used as given and not tokenized again by spaCy; or the collocations ofquanteda.textstats::textstat_collocations(), whose words are matched as they are, with onlycase_insensitiveapplying, as quanteda matches them. A match is labelled by the name of its pattern, or else by the pattern’s text. As in quanteda, a character element that contains a space is still one token, so sequences needphrase().valuetypeandcase_insensitivealso apply to a quanteda dictionary. Regular expressions are those of Python’sremodule. Sequences without a wildcard or a regular expression are matched by spaCy’sPhraseMatcher, so many thousands stay fast. Givingvaluetypeorcase_insensitiveis an error whenpatternholds nothing that they apply to: only spaCy’s token or dependency patterns, or anas_entity_patterns()object, whose phrases have their owncase_insensitive. A named list of spaCy patterns can also hold character vectors. -
as_entity_patterns()has the new argumentlevelsfor a quanteda dictionary, as inquanteda::tokens_lookup(): the default,levels = 1, labels values by their top-level key as before, andlevels = 1:2, for instance, by their keys at the first two levels, joined by"."("place.us"). Itsvaluetypenow lists its choices in quanteda’s order,c("glob", "regex", "fixed"), with the same default. A character vector of phrases now takesvaluetypetoo, so thatas_entity_patterns(c(ORG = "acme*"), valuetype = "glob")has a wildcard without a dictionary; its default stays"fixed", andcase_insensitiveapplies to it. - New function
as_index()turns the output ofspacy_match(),spacy_extract_entity()orspacy_extract_nounphrases()into an index of quanteda, asquanteda::index()makes, so thatquanteda::kwic(quanteda::as.tokens(spacy_parse(x)), index = as_index(m))shows each match, entity or noun phrase in its context. The index has one row for each row of its input, in the same order, labelled by the match’slabel, the entity’sent_type, or"nounphrase";NULL, which the two extractors return when they find nothing, gives an index with no rows. The positions are those of spaCy’s tokens, as inspacy_parse(). Document names are put in Unicode’s composed form (NFC), as quanteda keeps them, so thatkwic()does not drop the matches of a document whose name is in another form. The “Rule-Based Matching” article has new sections on matches in context and on counting matches asquanteda::tokens_lookup()does. - New method of
quanteda::as.dictionary()for the entity patterns thatas_entity_patterns()makes, when all of them are phrases: the labels become the keys of a quanteda dictionary and the phrases its values, each kept once and in its own case. Token patterns are an error that names their labels, and a phrase with*or?gives a warning, since quanteda reads those as wildcards unlessvaluetype = "fixed". Theidandcase_insensitiveof a pattern are not kept;as_entity_patterns(d, valuetype = "fixed", case_insensitive = FALSE)makes exact phrases of the dictionary again. The method is registered when quanteda is loaded, so quanteda stays a suggested package; this needs R 3.6.0 or later, which spacyr now requires. The articles show how to make a dictionary of the texts of matches and of entities. - The quanteda methods for the output of
spacy_parse()are now part of spacyr:as.tokens(),docnames(),ndoc(),ntoken(),ntype()andnsentence(). They were part of quanteda, and are documented in?"quanteda-methods". They are registered when quanteda is loaded, so quanteda stays a suggested package; this needs R 3.6.0 or later, which spacyr now requires. To use them on a parse that was saved to a file, load spacyr as well as quanteda. The methods now agree on which documents there are, and differ from those of quanteda 4.5.1 in these ways:- A document in which spaCy found no tokens, such as an empty text, is kept as a document with no tokens:
spacy_parse()now keeps the names of all the documents it was given with its output. Before,as.tokens()and the other methods left such documents out. -
as.tokens()restores the document variables of a quanteda corpus that was parsed, which were lost before. - When rows are taken out of a parse, the documents follow the rows: removing some tokens keeps every document, and removing all the rows of a document that had tokens removes it, and the documents without tokens with it. A parse without the record of its documents, such as one saved by an earlier version, has the documents of its rows, as before.
-
rbind()of parses keeps the documents of all of them, and their document variables when all have the same ones, when no two parses have a document of the same name. When two do, the result has the documents of its rows, without document variables. -
ntoken(),ntype()andnsentence()return integer vectors in the order of the documents, asdocnames()gives them; before, they were in alphabetical order of the document names. -
as.tokens()gives an error foruse_lemma = TRUEorinclude_poswhen the parse has no such column.
- A document in which spaCy found no tokens, such as an empty text, is kept as a document with no tokens:
-
spacy_initialize()has new argumentsentity_patternsandoverwrite_entities, to add rule-based named entities to the pipeline (#234).entity_patternstakes anythingas_entity_patterns()accepts, and adds spaCy’s entity ruler asspacyr_entity_ruler, after every component that sets entities, so that anentity_rulersaved with the model stays in place (if the model already has aspacyr_entity_ruler, the new one isspacyr_entity_ruler_2). Where a rule’s entity overlaps one the model found, the rule’s is kept, unlessoverwrite_entities = FALSE. Phrases markedcase_insensitiveare tokenized with the pipeline’s own tokenizer. The rules’ entities are returned byspacy_parse()andspacy_extract_entity()also whenentity = FALSE, and patterns that need entities found by an earlier component (withENT_TYPEorENT_IOB) give a warning when no component before the ruler sets entities. Phrase patterns give a warning when an earlier component, such asmerge_entities, changes the tokens. spaCy’s message is shown for a pattern it does not accept. -
spacy_initialize()has a new argumentsplit_hyphens(#250). Withsplit_hyphens = FALSE, spaCy’s tokenizer no longer splits words at a single hyphen between letters, or between a letter and a digit, so that “eye-opening” stays one token for the whole pipeline, including the tagger and lemmatizer, instead of becoming “eye”, “-” and “opening”. The tokenizer still splits wherever else it did, as between numbers (“3-4”, which quanteda’stokens(split_hyphens = FALSE)keeps whole), at two or more hyphens and at the start or end of a word. Only the language’s default infix rules are changed; a model saved with rules of its own, or with a tokenizer that is not spaCy’s rule-based one, gives a warning if it still splits some hyphenated words, and so does a ruler saved with the model whose phrase patterns were tokenized before the change. The default,TRUE, keeps the model’s tokenization, unlike the default ofquanteda::tokens(). -
spacy_parse(),spacy_tokenize(),spacy_extract_entity()andspacy_extract_nounphrases()have new argumentsn_processandbatch_size, passed to spaCy’snlp.pipe()(#185).n_processsets the number of processes (-1for all CPUs) andbatch_sizethe number of documents processed together; the defaults,n_process = 1andbatch_size = NULL(the pipeline’s default batch size), process documents as before. The output does not depend on either. With more than one process, pipelines with only spaCy’s built-in components and default error handlers send plain text to the workers. This avoids tokenizing and serializing the input documents in the main process. Pipelines with custom components or error handlers carry the document names through the pipeline, so the names stay with the documents if any are dropped or reordered (#289).spacy_tokenize()usesn_processonly when components of the pipeline run, which is forwhat = "sentence"and forremove_symbols = TRUE: splitting texts into words runs none, so it is always done in one process, since more would only add the time to start them (#289). -
spacy_parse()has a new argumentoffsets(#311). Withoffsets = TRUE, the output has two more columns aftertoken:startandend, the positions of each token’s first and last characters in its document. They are counted from 1 in characters as R counts them, whitespace included, so thatsubstring(text, start, end)gives the tokens of a text. The positions are those in the text as it was given, also when spaCy’s tokenizer makes a document with other whitespace, as the one for Korean can; if a tokenizer changes the text in another way, so that a token cannot be found in it or it has more than whitespace outside the tokens,offsets = TRUEis an error.entity_consolidate()andnounphrase_consolidate()keep the two columns, with a consolidated token running from the start of its first token to the end of its last. -
spacy_extract_entity(),spacy_extract_nounphrases()andspacy_match()have the new argumentoffsetstoo (#311). Withoffsets = TRUE, their data.frame has two more columns at its end:startandend, the positions in the text of the first and last characters of each entity, noun phrase or match, counted as inspacy_parse(), so thatsubstring(text, start, end)gives them as they are in a text. For a dependency pattern they are those of each node’s token. The extractors needoutput = "data.frame"for it. In these three functionsoffsetscomes after..., so it has to be given by its full name, and the arguments before it are matched as they were.
Deprecated
-
spacy_installed()is renamedspacy_is_installed(), a name that says it answers a question, asspacy_install()andspacy_uninstall()say they act.spacy_installed()is deprecated: it still works, gives a warning on every call, and will be removed in a future version. - The
multithreadargument ofspacy_parse(),spacy_tokenize(),spacy_extract_entity()andspacy_extract_nounphrases()is deprecated, and gives a warning when supplied. It has no effect (see below); usen_processandbatch_sizeinstead. -
process_document()and the functions that read its result,get_tokens(),get_tags(),get_attrs(),get_named_entities(),get_dependency(),get_noun_phrases(),get_ntokens()andget_ntokens_by_sent(), are deprecated, and give a warning on every call. They will be removed in a future version.spacy_parse()returns all of their results; each warning names the replacement. The help forget_dependency()shows how to convertspacy_parse()’shead_token_id, which numbers tokens within each sentence, to itshead_id, which numbers them within each document. - The result of
process_document()now holds its own parsed documents. Previously, theget_*()functions read the documents of the most recent call to any spacyr function, so they failed with a PythonKeyErroronce another function, such asspacy_parse(), had been called.process_document()now also checksn_processandbatch_size, and warns aboutmultithread, as the other functions do. It does not keep the language model, whichspacy_finalize()frees as before.
Changes
spacy_parse()now leaves out of a call the components of spaCy’s pipeline whose results its options do not ask for (#308): the named entity recognizer withentity = FALSE, the lemmatizer withlemma = FALSE, and the tagger and the attribute ruler whenpos,tag,lemmaandnounphraseare allFALSE. Before, every component ran whatever the options were, so that turning them off saved almost no time (#188); nowspacy_parse(x, pos = FALSE, lemma = FALSE, entity = FALSE)takes less than half the time of the defaults with spaCy’s small English model. The output is the same as when every component runs, also withn_processgreater than 1. A component stays whenever it sets something that the output has or that a later component may read. So the parser always runs, since it sets the sentences; and an attribute ruler stays if any of its rules sets something that is returned: that of spaCy’s English models also sets the dependency of whitespace tokens, so withdependency = TRUEit stays, and with it the tagger, whose tags its rules read. A component also stays if the model of a later component reads what it sets, as a recognizer trained to use the part of speech does; this is told from the model’s settings, and a model that is not built by spaCy’s own functions is taken to read everything. Nothing is left out whenadditional_attributesare asked for, or when the pipeline has a component that is not one of spaCy’s standard ones, since such a component may read or set anything; with entity rules fromspacy_initialize(entity_patterns = )andentity = TRUE, the components that the rules may read stay. The “Improving Performance” article is updated.spacy_extract_entity()andspacy_extract_nounphrases()leave out components in the same way (#308): with spaCy’s English models, the first no longer runs the tagger, the attribute ruler and the lemmatizer, and the second the lemmatizer and the named entity recognizer, which took about 40 and 25 per cent off their time for the inaugural addresses of quanteda. Their output is the same as when every component runs, by the rules thatspacy_parse()follows. For noun phrases, also those ofspacy_parse(nounphrase = TRUE), nothing is left out when the pipeline finds them by a function that is not spaCy’s own for the language (Vocab.get_noun_chunks), which may read any annotation.spacy_tokenize()withwhat = "sentence"orremove_symbols = TRUE, which ran every component of the pipeline, now runs only those that the sentences or the parts of speech need (#308). With spaCy’s English models, sentences need the parser, and no longer run the tagger, the attribute ruler, the lemmatizer and the named entity recognizer; removing symbols no longer runs the lemmatizer and the recognizer. For the inaugural addresses of quanteda, this took about 60 and 30 per cent off their time. The output is the same, by the rules thatspacy_parse()follows. Splitting texts into words runs no component, as before.spacy_match()now runs only the components that its patterns ask for (#308). Phrases, words and the values of a dictionary are matched on the text of the tokens, and need only the parser, for the sentence of each match: for the inaugural addresses of quanteda they took about 65 per cent less time. A pattern onPOSorTAGalso runs the components that set the tags, one onLEMMAthe lemmatizer, and one onENT_TYPEthe named entity recognizer, which took 20 to 45 per cent less. The output is the same, by the rules thatspacy_parse()follows. Every component runs when a pattern asks for an attribute that is not known, such as an extension attribute. In all the functions, an attribute ruler that stays and has such an attribute in the pattern of a rule keeps every component before it; and one with a rule that sets such an attribute, whose setter may read and set any annotation, always stays, with every component before it.None of these functions leaves out a component when the pipeline’s tokenizer or language class is not spaCy’s own. Such a tokenizer can give each document a function of its own to find the sentences (
Doc.user_hooks["sents"]), which may read any annotation, such as the part of speech.spacy_initialize()now sets the environment variableTOKENIZERS_PARALLELISMto"false", unless it is already set (#233). With versions of Hugging Face’s tokenizers library before 0.22.2, pipelines that use it, such as some built with spacy-transformers, printed “The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks…” again and again, until R was restarted, once the R process had been forked after the library was used, for instance byparallel::mcparallel(); saving a file in RStudio was also reported to trigger it. The setting controls the library’s tokenizer threads; its effect on total runtime depends on the pipeline and the workload. It applies to the whole R process and stays afterspacy_finalize(); if initialization fails, the previous value is restored. A value set beforehand, even an empty one, is kept. Setting it to"true"turns off the library’s safeguard against deadlocks, so a forked worker, fromparallel::mclapply()or fromn_processgreater than 1 when the workers are started by fork, that uses tokenizers after the parent has used it can hang; leave it at"false"whenever that can happen.spacy_download_langmodel(), also used byspacy_install(), now tries a failed model download twice more, after 10 and 30 seconds, and warns if all three attempts fail. Models come from GitHub, which sometimes answers with a temporary server error that pip does not retry. Previously a failed download gave no warning from spacyr.spacyr now requires spaCy 3.0.0 or later, and Python 3.
spacy_initialize()stops with an error if an earlier version of spaCy is installed, andspacy_install()refuses aversionthat allows only earlier versions, whether a bare version ("2.3.9"), a pip-style pin ("==2.3.9","[cuda102]==2.3.9") or a range ("<3"). A range without a lower bound of 3 or above gets>=3.0.0added. Aversionthat is not a valid pip requirement, such as"1.10.1a", is passed to pip unchanged, so that pip’s error shows what was given. The code for Python 2 and spaCy 2 has been removed. Thedeal_utf8argument ofget_attrs()is no longer used.spacy_parse()now builds all its columns in one pass over the documents as spaCy parses them, instead of storing the parsed documents in Python and fetching each column with a separate call. The output is unchanged; the parsed documents are no longer kept in memory after the call, and parsing with several options is somewhat faster.spacy_parse(),spacy_tokenize(),spacy_extract_entity()andspacy_extract_nounphrases()now always process documents in batches with spaCy’snlp.pipe(), and themultithreadargument no longer has any effect. Previously,multithread = FALSE, and a single document in all butspacy_parse(), processed each document withnlp(). With the standard spaCy models the results are the same, but a custom pipeline component whosepipe()method behaves differently from its__call__()method now always has itspipe()method used.When a pipeline component fails on a document and the pipeline’s error handler suppresses the error, spaCy’s
nlp.pipe()leaves that document out. spacyr now keeps the names of the other documents, leaves the failed document out of the output and warns about it. Previously, the results of later documents could be assigned to the wrong document names, andspacy_parse()failed.spacy_tokenize()withremove_punct,remove_numbersorremove_url, and withoutremove_symbols, no longer runs the components of the pipeline, which these options do not need, so they are now about as fast as tokenizing without them (#226). With the standard spaCy models, the tokens are the same as before. With a pipeline whose components change the tokens or their attributes, such as spaCy’smerge_entities, or that fail on some documents and leave them out, these options now give the tokens that plain tokenization gives, from all the documents, since those components no longer run.remove_symbolsstill runs all the components, since it needs the part-of-speech tags, whatever the other options.spacy_tokenize()now stops with an error whenremove_punct,remove_url,remove_numbers,remove_separators,remove_symbolsorpaddingis notTRUEorFALSE. Previously a missing value (NA) either gave an unclear error or was taken asTRUEorFALSEwithout warning.spacyr now loads its Python code as a module with
reticulate::import_from_path()and calls its functions with arguments, instead of running Python code in the__main__namespace and passing data through global variables there. As a result, spacyr no longer creates the Python variablesnlp,spobj,textsand others in__main__, so code that readreticulate::py$nlpafterspacy_initialize()no longer works.-
spacy_parse(),spacy_tokenize(),spacy_extract_entity()andspacy_extract_nounphrases()now share one function for checking and preparing their input, which fixes three edge cases:- A TIF data.frame whose
textcolumn is a factor, asdata.frame()made it by default before R 4.0, is now accepted. Previously it failed with “no applicable method”. (A factordoc_idcolumn already worked.) - A document name that is
NAnow gives the error “Some docnames are missing.” Previously it failed with “missing value where TRUE/FALSE needed”. - An empty character vector no longer gives a warning that two documents (
text1andtext0) were left out.
- A TIF data.frame whose
Bug fixes
-
entity_extract()andentity_consolidate()now keep entity labels that contain underscores, such as a customPUBLIC_AGENCYlabel. Previously they cut the label at its first underscore, givingPUBLIC. The labels of spaCy’s standard models have no underscores, so their results are unchanged.
Documentation
- The vignette and help pages are updated for spaCy 3: language models are named in full (for example
de_core_news_smrather thande), links to spaCy’s documentation point to its current pages, and thespacy_parse()description no longer mentions options that do not exist. The help forentity_extract()andspacy_extract_entity()now lists which entity types count as extended. - The vignette is now precomputed: its source is
vignettes/using_spacyr.Rmd.orig, whichmake vignetteknits, with spaCy, intovignettes/using_spacyr.Rmd, so that R and pkgdown build it without running its code. It replaces the static HTML vignette built with R.rsp, which had not been updated with the vignette’s source, and R.rsp is no longer a suggested package. The pkgdown site is rebuilt with pkgdown 2.2.1, and now uses Bootstrap 5 with the same theme and styling as the quanteda site. The site’s search box, which pkgdown provides without an outside service, is restored. - The help for
spacy_extract_entity()now names the column of entity types as it is returned,ent_type. It and the help forspacy_extract_nounphrases()now say thatstart_idcounts every one of spaCy’s tokens in the document, whitespace tokens included, as the rows ofspacy_parse()do; it is not the position among the tokens ofspacy_tokenize()with its default options, which leaves whitespace tokens out.
Tests
- The
saved-outputCI job now tries installing the pinned spaCy and model versions up to three times, for the same reason. - Added saved-output tests that compare the results of
spacy_parse(),spacy_tokenize(),spacy_extract_entity(),spacy_extract_nounphrases()and theentity_*()andnounphrase_*()functions with those of version 1.4.0, on a fixed set of texts that includes edge cases. Theentity_*()andnounphrase_*()tests do not need Python, so they also run on CRAN. - The saved-output tests compare output only with the spaCy and language model versions it was recorded with. A new
saved-outputCI job installs exactly those versions and fails if any saved-output test is skipped. The 1.4.0 output is kept as a fixed baseline, and the accepted output is updated from a named, accepted revision (seetests/testthat/fixtures/README.md). - Added tests that record which functions process documents with spaCy’s
nlp.pipe()and which callnlp()on each document, using a custom pipeline component that behaves differently in the two. Output with the standard models is the same either way, so the saved-output tests alone would not catch a change of path.
spacyr v1.4.0
CRAN release: 2026-11-04
Changes that can affect results
-
spacy_parse(),spacy_tokenize(),spacy_extract_entity()andspacy_extract_nounphrases()no longer alter the input text. Previously they deleted all backslashes and converted the literal sequences\nand\tinto newlines and tabs, so that for instanceC:\Userswas tokenized asC,:,Users. Code that relied on literal\nbeing converted should now convert it in R before calling spacyr. -
NAdocuments are now treated as empty documents, rather than as the text"NA". -
spacy_tokenize(what = "sentence", remove_separators = FALSE)now keeps trailing whitespace regardless of the value ofmultithread, including a single space after a sentence, so that the sentences join back into the original text. -
spacy_parse(nounphrase = TRUE)now returns rows in the order of the input documents, as it does without noun phrases, instead of sorted by document name. This fixes a bug where thewhitespacecolumn and anyadditional_attributeswere assigned to the wrong tokens when the document names were not in alphabetical order.
New features
- Added
spacy_installed(), which checks without starting Python whether spaCy and, optionally, given language models are installed (#197). - Added a
max_lengthargument tospacy_initialize()to allow documents longer than spaCy’s default limit of 1,000,000 characters (#218).
Installation
-
spacy_install()now creates its environment with a version of Python supported by the spaCy release being installed, as declared on PyPI, rather than whichever Python is found first; if none is found, it offers to install a compatible one. For an existing environment whose Python is not supported, it stops with an error explaining how to recreate the environment, instead of failing during installation. -
spacy_initialize()now explains how to install a language model that cannot be found. - Fixed
spacy_download_langmodel()andspacy_install()not detecting already-installed language models with recent versions of pip (#258). - Fixed a bug in
spacy_download_langmodel()that prevented installing multiple models in one function call (#249). -
spacy_install()andspacy_upgrade()no longer fail on Windows when installing into a different environment after Python has been initialised in the session (#258). - Updated the installation instructions: language models are installed with
spacy_download_langmodel()rather than withpip(#253), and new sections cover changing the Python version, using an existing Python or conda environment (#246), using local models (#180) and the need for git on Windows (#245). The vignette no longer describes arguments that were removed fromspacy_initialize(). - Requires reticulate >= 1.31.
Bug fixes
- Fixed a bug where an error during
spacy_tokenize(), such as a text exceeding spaCy’s maximum length, left the spaCy pipeline disabled, so that later calls in the same session failed or silently returned no entities. - Fixed a
SyntaxWarningabout an invalid escape sequence on Python 3.12+ when initialising spacyr (#257, thanks to @thjbdvlt). -
spacy_parse()no longer warns that lemmatization “may not work properly” for models whose name does not start with “en”. -
spacy_parse()now returns a zero-row result with the usual columns and types when no document contains any tokens (for instance when all documents areNAor empty), instead of failing withdependency = TRUEor returning malformed columns.
spacyr v1.3
- Overhauled the
spacy_install()andspacy_uninstall()functions to provide a more modern and robust installation procedure.
spacyr v1.2
CRAN release: 2019-07-04
- Added an option for using conda package manager instead of
pipinspacy_install()andspacy_upgrade(). - (1.21) All character inputs to
spacy_parse()andspacy_tokenize()are now coerced tocharacter, which allows them to work directly with quanteda v2 corpus objects (which are based on thecharacterclass but with special attributed), without special methods.
spacyr v1.1
- Fixed a bug in
spacy_parse(x, nounphrase = TRUE)that occurred when no noun phrases were found in the text. (#153) - Fixed a bug in
spacy_initialize()(#157).spacy_initialize()ignored Python options supplied by users when a conda environment created throughspacy_install()exists. - General stability enhancements to ensure compatibility with spaCy >= 2.1
spacyr v1.0
CRAN release: 2018-12-15
- Added new commands
spacy_tokenize(),spacy_extract_entity(),spacy_extract_nounphrases(),nounphrase_extract(), andnounphrase_consolidate()for direct extraction of entities, noun phrases, and tokens, and extraction of noun phrases from spacyr parsed tests. - Added a new argument
additional_attributestospacy_parse()allowing the return of any tokens-level attribute available from https://spacy.io/api/token#attributes. - Added a vignette and significantly improved the documentation site https://spacyr.quanteda.io.
spacyr v0.9.9
CRAN release: 2018-04-17
- Added
spacy_install(),spacy_install_virtualenv(), andspacy_upgrade()to make installing or upgrading spaCy (and Python itself) easy and automatic. - Added support for multithreading in
spacy_parse()via themultithreadingargument. This uses the “pipes” functionality in spaCy for improved performance.
spacyr v0.9.6
CRAN release: 2018-02-18
- Create an option to permanently set the default Python through
.Rprofile - Performance enhancement through
spacy_initialize(entity = FALSE)(#91) - Now looks for Python settings from
.bash_profile.
spacyr v0.9.3
CRAN release: 2017-11-10
- Updated for the newer spaCy 2.0 release and new language models.
- Add
ask = FALSEtospacy_initialize(), to find spaCy installations automatically.
spacyr v0.9.2
- Fixed a bug caused by zero-token “sentences” in
spacy_parse(), by changing1:length()toseq_along().