Title: huggingface/transformers: v4.7.0: DETR, RoFormer, ByT5, HuBERT, support for torch 1.9.0

Type Software Thomas Wolf, Lysandre Debut, Sylvain Gugger, Patrick von Platen, Julien Chaumond, Stas Bekman, Sam Shleifer, Victor SANH, Manuel Romero, Funtowicz Morgan, Julien Plu, Suraj Patil, Aymeric Augustin, Rémi Louf, Stefan Schweter, Denis, erenup, Matt, Nicolas Patry, Joe Davison, Anthony MOI, Philipp Schmid, Teven, Piero Molino, Grégory Châtel, Bram Vanroy, Clement, Kevin Canwen Xu, Daniel Stancl, Philip May (2021): huggingface/transformers: v4.7.0: DETR, RoFormer, ByT5, HuBERT, support for torch 1.9.0. Zenodo. Software. https://zenodo.org/record/4975030

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Sylvain Gugger ; Patrick von Platen ; Julien Chaumond (Hugging Face) ; Stas Bekman (Stasosphere Online Inc.) ; Sam Shleifer (@facebookresearch) ; Victor SANH (@huggingface) ; Manuel Romero ; Funtowicz Morgan (HuggingFace) ; Julien Plu (Hugging Face) ; Suraj Patil (Huggingface) ; Aymeric Augustin (@qonto) ; Rémi Louf (Freelance) ; Stefan Schweter ; Denis ; erenup ; Matt (@huggingface) ; Nicolas Patry (@huggingface) ; Joe Davison (Hugging Face) ; Anthony MOI (@huggingface) ; Philipp Schmid (@huggingface) ; Teven (HuggingFace) ; Piero Molino ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Bram Vanroy (@UGent) ; Clement (@huggingface) ; Kevin Canwen Xu ; Daniel Stancl (Seznam.cz) ; Philip May (@t-systems-on-site-services-gmbh) ;

Links

• Item record in Zenodo
• Digital object URL

Summary

v4.7.0: DETR, RoFormer, ByT5, Hubert, support for torch 1.9.0 DETR (@NielsRogge)

Three new models are released as part of the DETR implementation: DetrModel, DetrForObjectDetection and DetrForSegmentation, in PyTorch.

DETR consists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for object detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use things like region proposals, non-maximum suppression procedure, and anchor generation. Moreover, DETR can also be naturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.

DETR can support any timm backbone.

The DETR model was proposed in End-to-End Object Detection with Transformers by Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=detr

A new tokenizer is released as part of the ByT5 implementation: ByT5Tokenizer. It can be used with the T5 family of models.

The ByT5 model was presented in ByT5: Towards a token-free future with pre-trained byte-to-byte models by Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?search=byt5

14 new models are released as part of the RoFormer implementation: RoFormerModel, RoFormerForCausalLM, RoFormerForMaskedLM, RoFormerForSequenceClassification, RoFormerForTokenClassification, RoFormerForQuestionAnswering and RoFormerForMultipleChoice, TFRoFormerModel, TFRoFormerForCausalLM, TFRoFormerForMaskedLM, TFRoFormerForSequenceClassification, TFRoFormerForTokenClassification, TFRoFormerForQuestionAnswering and TFRoFormerForMultipleChoice, in PyTorch and TensorFlow.

RoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown improved performance on classification tasks with long texts. The RoFormer model was proposed in RoFormer: Enhanced Transformer with Rotary Position Embedding by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=roformer

HuBERT is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.

HuBERT was proposed in HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.

Two new models are released as part of the HuBERT implementation: HubertModel and HubertForCTC, in PyTorch.

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=hubert

On Monday, June 14th, 2021, we released the first part of the Hugging Face Course. The course is focused on the Hugging Face ecosystem, including transformers. Most of the material in the course is now linked from the transformers documentation which now includes videos to explain singular concepts.

The Wav2Vec2 model can now be used in TensorFlow:

More information

• DOI: 10.5281/zenodo.4975030

Dates

• Publication date: 2021
• Issued: June 17, 2021

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems