Title: huggingface/transformers: Bert Seq2Seq models, FSMT, LayoutLM, Funnel Transformer, LXMERT

Type Software Thomas Wolf, Lysandre Debut, Julien Chaumond, Patrick von Platen, Sam Shleifer, Victor SANH, Sylvain Gugger, Manuel Romero, Funtowicz Morgan, Aymeric Augustin, Stas Bekman, Rémi Louf, Stefan Schweter, Denis, Suraj Patil, erenup, Matt, Grégory Châtel, Piero Molino, Julien Plu, Anthony MOI, Bram Vanroy, Clement, Joe Davison, Teven, Gunnlaugur Thor Briem, Kevin Canwen Xu, Tim Rault, Catalin Voss, Lorenzo Ampil (2020): huggingface/transformers: Bert Seq2Seq models, FSMT, LayoutLM, Funnel Transformer, LXMERT. Zenodo. Software. https://zenodo.org/record/4044067

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Julien Chaumond (Hugging Face) ; Patrick von Platen ; Sam Shleifer (Huggingface) ; Victor SANH (@huggingface) ; Sylvain Gugger ; Manuel Romero ; Funtowicz Morgan (HuggingFace) ; Aymeric Augustin (@qonto) ; Stas Bekman (Stasosphere Online Inc.) ; Rémi Louf ; Stefan Schweter ; Denis ; Suraj Patil (Wynum) ; erenup ; Matt ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Piero Molino ; Julien Plu (Hugging Face) ; Anthony MOI (Hugging Face) ; Bram Vanroy (@UGent) ; Clement (@huggingface) ; Joe Davison (Hugging Face) ; Teven (HuggingFace) ; Gunnlaugur Thor Briem (Qlik) ; Kevin Canwen Xu ; Tim Rault (@huggingface) ; Catalin Voss (Stanford University) ; Lorenzo Ampil (@thinkingmachines) ;

Links

• Item record in Zenodo
• Digital object URL

Summary

Bert Seq2Seq models, FSMT, Funnel Transformer, LXMERT BERT Seq2seq models

The BertGeneration model is a BERT model that can be leveraged for sequence-to-sequence tasks using EncoderDecoderModel as proposed in Leveraging Pre-trained Checkpoints for Sequence Generation Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.

It was added to the library in PyTorch with the following checkpoints:

FSMT (FairSeq MachineTranslation) models were introduced in Facebook FAIR's WMT19 News Translation Task Submission by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.

It was added to the library in PyTorch, with the following checkpoints:

The LayoutLM model was proposed in LayoutLM: Pre-training of Text and Layout for Document Image Understandin by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. It's a simple but effective pre-training method of text and layout for document image understanding and information extraction tasks, such as form understanding and receipt understanding.

It was added to the library in PyTorch with the following checkpoints:

The Funnel Transformer model was proposed in the paper Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing. It is a bidirectional transformer model, like BERT, but with a pooling operation after each block of layers, a bit like in traditional convolutional neural networks (CNN) in computer vision.

It was added to the library in both PyTorch and TensorFlow, with the following checkpoints:

The LXMERT model was proposed in LXMERT: Learning Cross-Modality Encoder Representations from Transformers by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders (one for the vision modality, one for the language modality, and then one to fuse both modalities) pre-trained using a combination of masked language modeling, visual-language text alignment, ROI-feature regression, masked visual-attribute modeling, masked visual-object modeling, and visual-question answering objectives. The pretraining consists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering, VQA 2.0, and GQA.

It was added to the library in TensorFlow with the following checkpoints:

The following pipeline was added to the library:

The following community notebooks were contributed to the library:

An additional encoder-decoder architecture was added:

More information

• DOI: 10.5281/zenodo.4044067

Dates

• Publication date: 2020
• Issued: September 22, 2020

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems