Title: huggingface/transformers: v4.9.0: TensorFlow examples, CANINE, tokenizer training, ONNX rework

Type Software Thomas Wolf, Lysandre Debut, Sylvain Gugger, Patrick von Platen, Julien Chaumond, Stas Bekman, Sam Shleifer, Victor SANH, Manuel Romero, Funtowicz Morgan, Suraj Patil, Julien Plu, Aymeric Augustin, Rémi Louf, Stefan Schweter, Denis, Matt, erenup, Nicolas Patry, Joe Davison, Philipp Schmid, Teven, Anthony MOI, Piero Molino, Grégory Châtel, Kevin Canwen Xu, Bram Vanroy, Clement, Daniel Stancl, Philip May (2021): huggingface/transformers: v4.9.0: TensorFlow examples, CANINE, tokenizer training, ONNX rework. Zenodo. Software. https://zenodo.org/record/5121485

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Sylvain Gugger ; Patrick von Platen ; Julien Chaumond (Hugging Face) ; Stas Bekman (Stasosphere Online Inc.) ; Sam Shleifer (@facebookresearch) ; Victor SANH (@huggingface) ; Manuel Romero ; Funtowicz Morgan (HuggingFace) ; Suraj Patil (Huggingface) ; Julien Plu (Hugging Face) ; Aymeric Augustin (@qonto) ; Rémi Louf (Freelance) ; Stefan Schweter ; Denis ; Matt (@huggingface) ; erenup ; Nicolas Patry (@huggingface) ; Joe Davison (Hugging Face) ; Philipp Schmid (@huggingface) ; Teven (HuggingFace) ; Anthony MOI (@huggingface) ; Piero Molino ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Kevin Canwen Xu ; Bram Vanroy (@UGent) ; Clement (@huggingface) ; Daniel Stancl (Seznam.cz) ; Philip May (@t-systems-on-site-services-gmbh) ;

Links

• Item record in Zenodo
• Digital object URL

Summary

v4.9.0: TensorFlow examples, CANINE, tokenizer training, ONNX rework ONNX rework

This version introduces a new package, transformers.onnx, which can be used to export models to ONNX. Contrary to the previous implementation, this approach is meant as an easily extendable package where users may define their own ONNX configurations and export the models they wish to export.

Four new models are released as part of the CANINE implementation: CanineForSequenceClassification, CanineForMultipleChoice, CanineForTokenClassification and CanineForQuestionAnswering, in PyTorch.

The CANINE model was proposed in CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation by Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting. It's among the first papers that train a Transformer without using an explicit tokenization step (such as Byte Pair Encoding (BPE), WordPiece, or SentencePiece). Instead, the model is trained directly at a Unicode character level. Training at a character level inevitably comes with a longer sequence length, which CANINE solves with an efficient downsampling strategy, before applying a deep Transformer encoder.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=canine

This version introduces a new method to train a tokenizer from scratch based off of an existing tokenizer configuration.

The TFTrainer is now entering deprecation - and it is replaced by Keras. With version v4.9.0 comes the end of a long rework of the TensorFlow examples, for them to be more Keras-idiomatic, clearer, and more robust.

HuBERT is now implemented in TensorFlow:

When load_best_model_at_end was set to True in the TrainingArguments, having a different save_strategy and eval_strategy was accepted but the save_strategy was overwritten by the eval_strategy (the option to keep track of the best model needs to make sure there is an evaluation each time there is a save). This led to a lot of confusion with users not understanding why the script was not doing what it was told, so this situation will now raise an error indicating to set save_strategy and eval_strategy to the same values, and in the case that value is "steps", save_steps must be a round multiple of eval_steps.

More information

• DOI: 10.5281/zenodo.5121485

Dates

• Publication date: 2021
• Issued: July 22, 2021

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems