Title: huggingface/transformers: v4.2.0: LED from AllenAI, Generation Scores, TensorFlow 2x speedup, faster import

Type Software Thomas Wolf, Lysandre Debut, Patrick von Platen, Julien Chaumond, Sylvain Gugger, Sam Shleifer, Stas Bekman, Victor SANH, Manuel Romero, Funtowicz Morgan, Aymeric Augustin, Rémi Louf, Julien Plu, Stefan Schweter, Suraj Patil, Denis, erenup, Joe Davison, Matt, Piero Molino, Grégory Châtel, Bram Vanroy, Teven, Anthony MOI, Clement, Kevin Canwen Xu, Gunnlaugur Thor Briem, Tim Rault, Nicolas Patry, Malte Pietsch (2021): huggingface/transformers: v4.2.0: LED from AllenAI, Generation Scores, TensorFlow 2x speedup, faster import. Zenodo. Software. https://zenodo.org/record/4436743

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Patrick von Platen ; Julien Chaumond (Hugging Face) ; Sylvain Gugger ; Sam Shleifer (@facebookresearch) ; Stas Bekman (Stasosphere Online Inc.) ; Victor SANH (@huggingface) ; Manuel Romero ; Funtowicz Morgan (HuggingFace) ; Aymeric Augustin (@qonto) ; Rémi Louf (Freelance) ; Julien Plu (Hugging Face) ; Stefan Schweter ; Suraj Patil (Huggingface) ; Denis ; erenup ; Joe Davison (Hugging Face) ; Matt ; Piero Molino ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Bram Vanroy (@UGent) ; Teven (HuggingFace) ; Anthony MOI (Hugging Face) ; Clement (@huggingface) ; Kevin Canwen Xu ; Gunnlaugur Thor Briem (GRID) ; Tim Rault (@huggingface) ; Nicolas Patry (@huggingface) ; Malte Pietsch (deepset) ;

Links

• Item record in Zenodo
• Digital object URL

Summary

v4.2.0: LED from AllenAI, encoder-decoder templates, fast imports LED from AllenAI (@patrickvonplaten)

Four new models are released as part of the LED implementation: LEDModel, LEDForConditionalGeneration, LEDForSequenceClassification, LEDForQuestionAnswering, in PyTorch. The first two models have a TensorFlow version.

LED is the encoder-decoder variant of the Longformer model by allenai.

The LED model was proposed in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=led

Available notebooks:

Contributions:

The PyTorch generation function now allows to return:

by simply adding return_dict_in_generate to the config or as an input to .generate()

Tweet:

Notebooks for a better explanation:

PR:

The TensorFlow version of the BERT-like models have been updated and are now twice as fast as the previous versions.

This version introduces a new API for TensorFlow saved models, which can now be exported with model.save_pretrained("path", saved_model=True) and easily loaded into a TensorFlow Serving environment.

Initial support for DeepSpeed to accelerate distributed training on several GPUs. This is an experimental feature that hasn't been fully tested yet, but early results are very encouraging (see this comment). Stay tuned for more details in the coming weeks!

The encoder-decoder version of the templates is now part of Transformers! Adding an encoder-decoder model is made very easy with this addition. More information can be found in the README.

The initialization process has been changed to only import what is required. Therefore, when using only PyTorch models, TensorFlow will not be imported and vice-versa. In the best situations the import of a transformers model now takes only a few hundreds of milliseconds (~200ms) compared to more than a few seconds (~3s) in previous versions.

Some models now have improved documentation. The LayoutLM model has seen a general overhaul in its documentation thanks to @NielsRogge.

The tokenizer-only models Bertweet, Herbert and Phobert now have their own documentation pages thanks to @Qbiwan.

There are no breaking changes between the previous version and this one. This will be the first version to require TensorFlow >= 2.3.

More information

• DOI: 10.5281/zenodo.4436743

Dates

• Publication date: 2021
• Issued: January 13, 2021

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems