pytorch transformers
## Taglish-Electra

Our Taglish-Electra model was pretrained with two Filipino training datasets and one English dataset to increase improvement against Filipino text with English where speakers may code-switch between the two languages. 

1) Openwebtext (English) 
2) WikiText-TL-39 (Filipino)
3) [TLUnified Large Scale Corpus](https://www.blaisecruz.com/resources/)

This is the discriminator model, which is the main Transformer used for finetuning to downstream tasks. For generation, mask-filling, and retraining, refer to the Generator models.