# PolitBERT

## Background

This model was created to specialize on political speeches, interviews and press briefings of English-speaking politicians.

## Training
The model was initialized using the pre-trained weights of BERT<sub>BASE</sub> and trained for 20 epochs on the standard MLM task with default parameters.
The used learning rate was 5e-5 with a linearly decreasing schedule and AdamW.
The used batch size is 8 per GPU while beeing trained on two Nvidia GTX TITAN X.
The rest of the used configuration is the same as in ```AutoConfig.from_pretrained('bert-base-uncased')```.
As a tokenizer the default tokenizer of BERT was used (```BertTokenizer.from_pretrained('bert-base-uncased')```)

## Dataset
PolitBERT was trained on the following dataset, which has been split up into single sentences:
<https://www.kaggle.com/mauricerupp/englishspeaking-politicians>

## Usage
To predict a missing word of a sentence, the following pipeline can be applied:

```
from transformers import pipeline, BertTokenizer, AutoModel

fill_mask = pipeline("fill-mask", 
                     model=AutoModel.from_pretrained('maurice/PolitBERT'), 
                     tokenizer=BertTokenizer.from_pretrained('bert-base-uncased'))

print(fill_mask('Donald Trump is a [MASK].'))
```

## Training Results
Evaluation Loss:
![evalloss](evalloss_BERT.png)
Training Loss:
![evalloss](loss_BERT.png)
Learning Rate Schedule:
![evalloss](LR_BERT.png)


{}
transformers pytorch jax bert fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #jax #bert #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

# PolitBERT

## Background

This model was created to specialize on political speeches, interviews and press briefings of English-speaking politicians.

## Training
The model was initialized using the pre-trained weights of BERT<sub>BASE</sub> and trained for 20 epochs on the standard MLM task with default parameters.
The used learning rate was 5e-5 with a linearly decreasing schedule and AdamW.
The used batch size is 8 per GPU while beeing trained on two Nvidia GTX TITAN X.
The rest of the used configuration is the same as in .
As a tokenizer the default tokenizer of BERT was used ()

## Dataset
PolitBERT was trained on the following dataset, which has been split up into single sentences:
<URL

## Usage
To predict a missing word of a sentence, the following pipeline can be applied:



## Training Results
Evaluation Loss:
!evalloss
Training Loss:
!evalloss
Learning Rate Schedule:
!evalloss


# PolitBERT ## Background

This model was created to specialize on political speeches, interviews and press briefings of English-speaking politicians. ## Training
The model was initialized using the pre-trained weights of BERT<sub>BASE</sub> and trained for 20 epochs on the standard MLM task with default parameters.
The used learning rate was 5e-5 with a linearly decreasing schedule and AdamW.
The used batch size is 8 per GPU while beeing trained on two Nvidia GTX TITAN X.
The rest of the used configuration is the same as in .
As a tokenizer the default tokenizer of BERT was used () ## Dataset
PolitBERT was trained on the following dataset, which has been split up into single sentences:
<URL ## Usage
To predict a missing word of a sentence, the following pipeline can be applied: ## Training Results
Evaluation Loss:
!evalloss
Training Loss:
!evalloss
Learning Rate Schedule:
!evalloss
TAGS
#transformers #pytorch #jax #bert #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
 # PolitBERT ## Background

This model was created to specialize on political speeches, interviews and press briefings of English-speaking politicians. ## Training
The model was initialized using the pre-trained weights of BERT<sub>BASE</sub> and trained for 20 epochs on the standard MLM task with default parameters.
The used learning rate was 5e-5 with a linearly decreasing schedule and AdamW.
The used batch size is 8 per GPU while beeing trained on two Nvidia GTX TITAN X.
The rest of the used configuration is the same as in .
As a tokenizer the default tokenizer of BERT was used () ## Dataset
PolitBERT was trained on the following dataset, which has been split up into single sentences:
<URL ## Usage
To predict a missing word of a sentence, the following pipeline can be applied: ## Training Results
Evaluation Loss:
!evalloss
Training Loss:
!evalloss
Learning Rate Schedule:
!evalloss
passage: TAGS
#transformers #pytorch #jax #bert #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
# PolitBERT## Background

This model was created to specialize on political speeches, interviews and press briefings of English-speaking politicians.## Training
The model was initialized using the pre-trained weights of BERT<sub>BASE</sub> and trained for 20 epochs on the standard MLM task with default parameters.
The used learning rate was 5e-5 with a linearly decreasing schedule and AdamW.
The used batch size is 8 per GPU while beeing trained on two Nvidia GTX TITAN X.
The rest of the used configuration is the same as in .
As a tokenizer the default tokenizer of BERT was used ()## Dataset
PolitBERT was trained on the following dataset, which has been split up into single sentences:
<URL## Usage
To predict a missing word of a sentence, the following pipeline can be applied:## Training Results
Evaluation Loss:
!evalloss
Training Loss:
!evalloss
Learning Rate Schedule:
!evalloss