Create README.md

README.md

@@ -0,0 +1,63 @@
+---
+datasets:
+- xnli
+language:
+- sw
+library_name: transformers
+examples: null
+widget:
+  - text: Uhuru Kenyatta ni rais wa [MASK].
+    example_title: Sentence_1
+  - text: Tumefanya mabadiliko muhimu [MASK] sera zetu za faragha na vidakuzi
+    example_title: Sentence_2
+---
+
+# SW
+
+* Pre-trained model on Swahili language using a masked language modeling (MLM) objective.
+
+## Model description
+
+This is a transformers model pre-trained on a large corpus of Swahili data in a self-supervised fashion. This means it
+was pre-trained on the raw texts only, with no humans labeling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
+was pre-trained with one objective:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
+  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the terms one after the other, or from autoregressive models like
+  GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the
+  sentence.
+
+This way, the model learns an inner representation of the Swahili language that can then be used to extract features
+useful for downstream tasks e.g.
+ * Named Entity Recognition (Token Classification)
+ * Text Classification
+
+The model is based on the Orginal BERT UNCASED which can be found on [google-research/bert readme](https://github.com/google-research/bert/blob/master/README.md)
+
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling, but it's primarily intended to be fine-tuned on a downstream task. 
+Check out this variant TUS_NER-sw, a finetuned version of TUS meant for Named Entity Recognition
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("eolang/SW-v1")
+model = AutoModelForMaskedLM.from_pretrained("eolang/SW-v1")
+
+text = "Hii ni tovuti ya idhaa ya Kiswahili ya BBC ambayo hukuletea habari na makala kutoka Afrika na kote duniani kwa lugha ya Kiswahili."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+### Limitations and Bias
+
+Even if the training data used for this model could be reasonably neutral, this model can have biased
+predictions. This is something we are still working on improving.