
# BigBird base model

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this [paper](https://arxiv.org/abs/2007.14062) and first released in this [repository](https://github.com/google-research/bigbird).

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.

## Model description

BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.

## How to use `TODO: Update`

Here is how to use this model to get the features of a given text in Flax:

```python
from transformers import BigBirdTokenizer, FlaxBigBirdModel

model_id = "flax-community/bigband"

# by default its in `block_sparse` mode with num_random_blocks=3, block_size=64
model = FlaxBigBirdModel.from_pretrained(model_id)

# you can change `attention_type` to full attention like this:
model = FlaxBigBirdModel.from_pretrained(model_id, attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = FlaxBigBirdModel.from_pretrained(model_id, block_size=16, num_random_blocks=2)

tokenizer = BigBirdTokenizer.from_pretrained(model_id)

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors="jax")
output = model(**inputs)
```

## Training Data `TODO: Update`

This model is pre-trained on four publicly available datasets: **Books**, **CC-News**, **Stories** and **Wikipedia**. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2).

## Training Procedure `TODO: Update`

Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.

Model is warm started from RoBERTa’s checkpoint.

{"language": "en", "license": "apache-2.0", "datasets": ["bookcorpus", "wikipedia", "cc_news"]}
transformers big_bird pretraining en dataset:bookcorpus dataset:wikipedia dataset:cc_news arxiv:2007.14062 license:apache-2.0 endpoints_compatible region:us
TAGS
#transformers #big_bird #pretraining #en #dataset-bookcorpus #dataset-wikipedia #dataset-cc_news #arxiv-2007.14062 #license-apache-2.0 #endpoints_compatible #region-us 


# BigBird base model

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository.

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.

## Model description

BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.

## How to use 'TODO: Update'

Here is how to use this model to get the features of a given text in Flax:



## Training Data 'TODO: Update'

This model is pre-trained on four publicly available datasets: Books, CC-News, Stories and Wikipedia. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2).

## Training Procedure 'TODO: Update'

Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.

Model is warm started from RoBERTa’s checkpoint.

# BigBird base model

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository.

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team. ## Model description

BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts. ## How to use 'TODO: Update'

Here is how to use this model to get the features of a given text in Flax: ## Training Data 'TODO: Update'

This model is pre-trained on four publicly available datasets: Books, CC-News, Stories and Wikipedia. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2). ## Training Procedure 'TODO: Update'

Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.

Model is warm started from RoBERTa’s checkpoint.
TAGS
#transformers #big_bird #pretraining #en #dataset-bookcorpus #dataset-wikipedia #dataset-cc_news #arxiv-2007.14062 #license-apache-2.0 #endpoints_compatible #region-us 
 # BigBird base model

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository.

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team. ## Model description

BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts. ## How to use 'TODO: Update'

Here is how to use this model to get the features of a given text in Flax: ## Training Data 'TODO: Update'

This model is pre-trained on four publicly available datasets: Books, CC-News, Stories and Wikipedia. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2). ## Training Procedure 'TODO: Update'

Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.

Model is warm started from RoBERTa’s checkpoint.
passage: TAGS
#transformers #big_bird #pretraining #en #dataset-bookcorpus #dataset-wikipedia #dataset-cc_news #arxiv-2007.14062 #license-apache-2.0 #endpoints_compatible #region-us 
# BigBird base model

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository.

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.## Model description

BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.## How to use 'TODO: Update'

Here is how to use this model to get the features of a given text in Flax:## Training Data 'TODO: Update'

This model is pre-trained on four publicly available datasets: Books, CC-News, Stories and Wikipedia. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2).## Training Procedure 'TODO: Update'

Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.

Model is warm started from RoBERTa’s checkpoint.