Title: huggingface/transformers: v4.4.0: S2T, M2M100, I-BERT, mBART-50, DeBERTa-v2, XLSR-Wav2Vec2

Type Software Thomas Wolf, Lysandre Debut, Sylvain Gugger, Patrick von Platen, Julien Chaumond, Sam Shleifer, Stas Bekman, Victor SANH, Manuel Romero, Funtowicz Morgan, Julien Plu, Aymeric Augustin, Rémi Louf, Suraj Patil, Stefan Schweter, Denis, erenup, Joe Davison, Matt, Nicolas Patry, Anthony MOI, Piero Molino, Grégory Châtel, Bram Vanroy, Teven, Clement, Kevin Canwen Xu, Gunnlaugur Thor Briem, Tim Rault, Malte Pietsch (2021): huggingface/transformers: v4.4.0: S2T, M2M100, I-BERT, mBART-50, DeBERTa-v2, XLSR-Wav2Vec2. Zenodo. Software. https://zenodo.org/record/4608351

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Sylvain Gugger ; Patrick von Platen ; Julien Chaumond (Hugging Face) ; Sam Shleifer (@facebookresearch) ; Stas Bekman (Stasosphere Online Inc.) ; Victor SANH (@huggingface) ; Manuel Romero ; Funtowicz Morgan (HuggingFace) ; Julien Plu (Hugging Face) ; Aymeric Augustin (@qonto) ; Rémi Louf (Freelance) ; Suraj Patil (Huggingface) ; Stefan Schweter ; Denis ; erenup ; Joe Davison (Hugging Face) ; Matt ; Nicolas Patry (@huggingface) ; Anthony MOI (Hugging Face) ; Piero Molino ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Bram Vanroy (@UGent) ; Teven (HuggingFace) ; Clement (@huggingface) ; Kevin Canwen Xu ; Gunnlaugur Thor Briem (GRID) ; Tim Rault (@huggingface) ; Malte Pietsch (deepset) ;

Links

• Item record in Zenodo
• Digital object URL

Summary

v4.4.0: S2T, M2M100, I-BERT, mBART-50, DeBERTa-v2, XLSR-Wav2Vec2 SpeechToText

Two new models are released as part of the S2T implementation: Speech2TextModel and Speech2TextForConditionalGeneration, in PyTorch.

Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech signal. It's a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively.

The Speech2Text model was proposed in fairseq S2T: Fast Speech-to-Text Modeling with fairseq by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=speech_to_text

Two new models are released as part of the M2M100 implementation: M2M100Model and M2M100ForConditionalGeneration, in PyTorch.

M2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks.

The M2M100 model was proposed in Beyond English-Centric Multilingual Machine Translation by Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=m2m_100

Six new models are released as part of the I-BERT implementation: IBertModel, IBertForMaskedLM, IBertForSequenceClassification, IBertForMultipleChoice, IBertForTokenClassification and IBertForQuestionAnswering, in PyTorch.

I-BERT is a quantized version of RoBERTa running inference up to four times faster.

The I-BERT framework in PyTorch allows to identify the best parameters for quantization. Once the model is exported in a framework that supports int8 execution (such as TensorRT), a speedup of up to 4x is visible, with no loss in performance thanks to the parameter search.

The I-BERT model was proposed in I-BERT: Integer-only BERT Quantization by Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney and Kurt Keutzer.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=ibert

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=speech_to_text

MBart-50 is created using the original mbart-large-cc25 checkpoint by extending its embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50 languages.

The MBart model was presented in Multilingual Translation with Extensible Multilingual Pretraining and Finetuning by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=mbart-50

Fixe new models are released as part of the DeBERTa-v2 implementation: DebertaV2Model, DebertaV2ForMaskedLM, DebertaV2ForSequenceClassification, DeberaV2ForTokenClassification and DebertaV2ForQuestionAnswering, in PyTorch.

The DeBERTa model was proposed in DeBERTa: Decoding-enhanced BERT with Disentangled Attention by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It is based on Google's BERT model released in 2018 and Facebook's RoBERTa model released in 2019.

It builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in RoBERTa.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=deberta-v2

The XLSR-Wav2Vec2 model was proposed in Unsupervised Cross-Lingual Representation Learning For Speech Recognition by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael Auli.

The checkpoint corresponding to that model is added to the model hub: facebook/ wav2vec2-large-xlsr-53

A fine-tuning script showcasing how the Wav2Vec2 model can be trained has been added.

The Wav2Vec2 architecture becomes more stable as several changes are done to its architecture. This introduces feature extractors and feature processors as the pre-processing aspect of multi-modal speech models.

Most of the TensorFlow models are now compatible with automatic mixed precision and have XLA support.

We are rolling out experimental support for model parallelism on SageMaker with a new SageMakerTrainer that can be used in place of the regular Trainer. This is a temporary class that will be removed in a future version, the end goal is to have Trainer support this feature out of the box.

ConvBERT fix torch <> tf weights conversion #10314 (@abhishekkrthakur)

fix deprecated reference tokenizer.max_len in glue.py #10220 (@poedator)

[ci, flax] non-existing models are unlikely to pass tests #10409 (@julien-c)

[LED] Correct Docs #10419 (@patrickvonplaten)

More information

• DOI: 10.5281/zenodo.4608351

Dates

• Publication date: 2021
• Issued: March 16, 2021

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems