|
|
--- |
|
|
language: en |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- wikipedia |
|
|
--- |
|
|
|
|
|
# ALBERT Large (dropout) |
|
|
|
|
|
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced |
|
|
in [this paper](https://arxiv.org/abs/1810.04805) and first released |
|
|
in [this repository](https://github.com/google-research-datasets/Zari). The model is initialized from the relevant |
|
|
publicly-available checkpoint and pre-training continued over Wikipedia, with increased dropout rate. |
|
|
|
|
|
Disclaimer: The team releasing ALBERT did not write a model card for this model so this model card has been written by |
|
|
the FairNLP team. |
|
|
|
|
|
|
|
|
### BibTeX entry and citation info |
|
|
|
|
|
``` |
|
|
@misc{zari, |
|
|
title={Measuring and Reducing Gendered Correlations in Pre-trained Models}, |
|
|
author={Kellie Webster and Xuezhi Wang and Ian Tenney and Alex Beutel and Emily Pitler and Ellie Pavlick and Jilin Chen and Slav Petrov}, |
|
|
year={2020}, |
|
|
eprint={2010.06032}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL} |
|
|
} |
|
|
``` |
