Title: huggingface/transformers: Transformers v4.0.0: Fast tokenizers, model outputs, file reorganization

Type Software Thomas Wolf, Lysandre Debut, Julien Chaumond, Patrick von Platen, Sylvain Gugger, Sam Shleifer, Victor SANH, Stas Bekman, Manuel Romero, Funtowicz Morgan, Aymeric Augustin, Rémi Louf, Stefan Schweter, Denis, Suraj Patil, Julien Plu, erenup, Joe Davison, Matt, Piero Molino, Grégory Châtel, Bram Vanroy, Anthony MOI, Teven, Clement, Kevin Canwen Xu, Gunnlaugur Thor Briem, Tim Rault, Malte Pietsch, Bilal Khan (2020): huggingface/transformers: Transformers v4.0.0: Fast tokenizers, model outputs, file reorganization. Zenodo. Software. https://zenodo.org/record/4298849

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Julien Chaumond (Hugging Face) ; Patrick von Platen ; Sylvain Gugger ; Sam Shleifer (@facebookresearch) ; Victor SANH (@huggingface) ; Stas Bekman (Stasosphere Online Inc.) ; Manuel Romero ; Funtowicz Morgan (HuggingFace) ; Aymeric Augustin (@qonto) ; Rémi Louf (Freelance) ; Stefan Schweter ; Denis ; Suraj Patil (Huggingface) ; Julien Plu (Hugging Face) ; erenup ; Joe Davison (Hugging Face) ; Matt ; Piero Molino ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Bram Vanroy (@UGent) ; Anthony MOI (Hugging Face) ; Teven (HuggingFace) ; Clement (@huggingface) ; Kevin Canwen Xu ; Gunnlaugur Thor Briem (GRID) ; Tim Rault (@huggingface) ; Malte Pietsch (deepset) ; Bilal Khan ;

Links

• Item record in Zenodo
• Digital object URL

Summary

Transformers v4.0.0-rc-1: Fast tokenizers, model outputs, file reorganization Breaking changes since v3.x

Version v4.0.0 introduces several breaking changes that were necessary.

The python and rust tokenizers have roughly the same API, but the rust tokenizers have a more complete feature set. The main breaking change is the handling of overflowing tokens between the python and rust tokenizers.

In version v3.x:

to obtain the same in version v4.x:

The requirement on the SentencePiece dependency has been lifted from the setup.py. This is done so that we may have a channel on anaconda cloud without relying on conda-forge. This means that the tokenizers that depend on the SentencePiece library will not be available with a standard transformers installation.

This includes the slow versions of:

In order to obtain the same behavior as version v3.x, you should install sentencepiece additionally:

In version v3.x:

to obtain the same in version v4.x:

or

The past and foreseeable addition of new models means that the number of files in the directory src/transformers keeps growing and becomes harder to navigate and understand. We made the choice to put each model and the files accompanying it in their own sub-directories.

This is a breaking change as importing intermediary layers using a model's module directly needs to be done via a different path.

In order to obtain the same behavior as version v3.x, you should update the path used to access the layers.

In version v3.x:

to obtain the same in version v4.x:

The return_dict argument enables the return of named-tuples-like python objects containing the model outputs, instead of the standard tuples. This object is self-documented as keys can be used to retrieve values, while also behaving as a tuple as users may retrieve objects by index or by slice.

This is a breaking change as the limitation of that tuple is that it cannot be unpacked: value0, value1 = outputs will not work.

In order to obtain the same behavior as version v3.x, you should specify the return_dict argument to False, either in the model configuration or during the forward pass.

In version v3.x:

to obtain the same in version v4.x:

Attributes that were deprecated have been removed if they had been deprecated for at least a month. The full list of deprecated attributes can be found in #8604.

Here is a list of these attributes/methods/arguments and what their replacements should be:

In several models, the labels become consistent with the other models:

In several models, the caching mechanism becomes consistent with the other models:

Regarding the tokenizer classes:

Regarding the Trainer class:

Regarding the TFTrainer class:

Regarding the TrainerArgument and TFTrainerArgument classes:

Regarding the Transfo-XL model:

Regarding pipelines:

Version 4.0.0 will be the first to include the experimental feature of model templates. These model templates aim to facilitate the addition of new models to the library by doing most of the work: generating the model/configuration/tokenization/test files that fit the API, with respect to the choice the user has made in terms of naming and functionality.

This release includes a model template for the encoder model (similar to the BERT architecture). Generating a model using the template will generate the files, put them at the appropriate location, reference them throughout the code-base, and generate a working test suite. The user should then only modify the files to their liking, rather than creating the model from scratch.

Feedback welcome, get started from the README here.

The T5v1.1 is an improved version of the original T5 model, see here: https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md

The multilingual T5 model (mT5) was presented in https://arxiv.org/abs/2010.11934 and is based on the T5v1.1 architecture.

Multiple pre-trained checkpoints have been added to the library:

Relevant pull requests:

The DPR model has been added in TensorFlow to match its PyTorch counterpart by @ratthachat

Additional heads have been added to the TensorFlow Longformer implementation: SequenceClassification, MultipleChoice and TokenClassification

More information

• DOI: 10.5281/zenodo.4298849

Dates

• Publication date: 2020
• Issued: November 30, 2020

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems