Title: huggingface/transformers: FlauBERT, MMBT, Dutch model, improved documentation, training from scratch, clean Python code

Type Software Thomas Wolf, Lysandre Debut, Victor SANH, Julien Chaumond, Aymeric Augustin, Rémi Louf, Funtowicz Morgan, Stefan Schweter, Denis, erenup, Matt, Piero Molino, Grégory Châtel, Patrick von Platen, Tim Rault, MOI Anthony, Catalin Voss, Bilal Khan, Bram Vanroy, Fei Wang, Julien Plu, Malte Pietsch, Louis Martin, Davide Fiocco, dhanajitb, Jinoo, Ananya Harsh Jha, Juha Kiili, Guillem García Subies, Clement (2020): huggingface/transformers: FlauBERT, MMBT, Dutch model, improved documentation, training from scratch, clean Python code. Zenodo. Software. https://zenodo.org/record/3633003

Authors: Thomas Wolf (@huggingface) ; Lysandre Debut (Hugging Face) ; Victor SANH (@huggingface) ; Julien Chaumond (Hugging Face) ; Aymeric Augustin (@canalplus) ; Rémi Louf ; Funtowicz Morgan (HuggingFace) ; Stefan Schweter ; Denis ; erenup ; Matt ; Piero Molino ; Grégory Châtel (DisAItek & Intel AI Innovators) ; Patrick von Platen ; Tim Rault (@huggingface) ; MOI Anthony (Hugging Face) ; Catalin Voss (Stanford University) ; Bilal Khan ; Bram Vanroy (@UGent) ; Fei Wang (@ShannonAI) ; Julien Plu (Leboncoin Lab) ; Malte Pietsch (deepset) ; Louis Martin ; Davide Fiocco ; dhanajitb ; Jinoo ; Ananya Harsh Jha ; Juha Kiili (Juhakiili Oy) ; Guillem García Subies ; Clement (@huggingface) ;

Links

• Item record in Zenodo
• Digital object URL

Summary

FlauBERT, MMBT MMBT was added to the list of available models, as the first multi-modal model to make it in the library. It can accept a transformer model as well as a computer vision model, in order to classify image and text. The MMBT Model is from Supervised Multimodal Bitransformers for Classifying Images and Text by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine (https://github.com/facebookresearch/mmbt/) Added by @suvrat96. A new Dutch BERT model was added under the wietsedv/bert-base-dutch-cased identifier. Added by @wietsedv. Model page A new French model was added, FlauBERT, based on XLM. The FlauBERT model is from FlauBERT: Unsupervised Language Model Pre-training for French (https://github.com/getalp/Flaubert). Four checkpoints are added: small size, base uncased, base cased and large. Model page New TF architectures (@jplu) TensorFlow XLM-RoBERTa was added (@jplu ) TensorFlow CamemBERT was added (@jplu ) Python best practices (@aaugustin) Greatly improved the quality of the source code by leveraging black, isort and flake8. A test was added, check_code_quality, which checks that the contributions respect the contribution guidelines related to those tools. Similarly, optional imports are better handled and raise more precise errors. Cleaned up several requirements files, updated the contribution guidelines and rely on setup.py for the necessary dev dependencies. you can clean up your code for a PR with (more details in CONTRIBUTING.md):make style make quality Documentation (@LysandreJik)

The documentation was uniformized and some better guidelines have been defined. This work is part of an ongoing effort of making transformers accessible to a larger audience. A glossary has been added, adding definitions for most frequently used inputs.

Furthermore, some tips are given concerning each model in their documentation pages.

The code samples are now tested on a weekly basis alongside other slow tests.

The source code was moved from ./transformers to ./src/transformers. Since it changes the location of the source code, contributors must update their local development environment by uninstalling and re-installing the library.

Version 2.3.0 was the last version to support Python 2. As we begin the year 2020, official Python 2 support has been dropped.

Tests can now be run in parallel

An abstract method was added to PreTrainedModel, which is implemented in all models trained with CLM. This abstract method is generate, which offers an API for text generation:

Previously, when stopping a training the only saved values would be the model weights/configuration. Now the different scripts save several other values: the global step, current epoch, and the steps trained in the current epoch. When resuming a training, all those values will be leveraged to correctly resume the training.

This applies to the following scripts: run_glue, run_squad, run_ner, run_xnli.

The run_lm_finetuning.py script now handles training from scratch.

The configuration files now contain the architecture they're referring to. There is no need to have the architecture in the file name as it was necessary before. This should ease the naming of community models.

A new type of AutoModel was added: AutoModelForPreTraining. This model returns the base model that was used during the pre-training. For most models it is the base model alongside a language modeling head, whereas for others it is a different model, e.g. BertForPreTraining for BERT.

The HANS dataset was added to the examples. It allows for testing a model with adversarial evaluation of natural language.

When using PyTorch, certain values can be ignored when computing the loss. In order for the loss function to understand which indices must be ignored, those have to be set to a certain value. Most of our models required those indices to be set to -1. We decided to set this value to -100 instead as it is PyTorch's default value. This removes the discrepancy between user-implemented losses and the losses integrated in the models.

Further help from @r0mainK.

More information

• DOI: 10.5281/zenodo.3633003

Dates

• Publication date: 2020
• Issued: January 31, 2020

Rights

• info:eu-repo/semantics/openAccess Open Access

Format

electronic resource

Relateditems