
# distilbert-base-nepali

This model is pre-trained on [nepalitext](https://huggingface.co/datasets/Sakonii/nepalitext-language-model-dataset) dataset consisting of over 13 million Nepali text sequences using a masked language modeling (MLM) objective. Our approach trains a Sentence Piece Model (SPM) for text tokenization similar to [XLM-ROBERTa](https://arxiv.org/abs/1911.02116) and trains [distilbert model](https://arxiv.org/abs/1910.01108) for language modeling. Find more details in [this paper](https://aclanthology.org/2022.sigul-1.14/).

It achieves the following results on the evaluation set:

mlm probability|evaluation loss|evaluation perplexity
--:|----:|-----:|
15%|2.349|10.479|
20%|2.605|13.351|

## Model description

Refer to original [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased)

## Intended uses & limitations

This backbone model intends to be fine-tuned on Nepali language focused downstream task such as sequence classification, token classification or question answering. 
The language model being trained on a data with texts grouped to a block size of 512, it handles text sequence up to 512 tokens and may not perform satisfactorily on shorter sequences.

## Usage

This model can be used directly with a pipeline for masked language modeling:

```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='Sakonii/distilbert-base-nepali')
>>> unmasker("मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, <mask>, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।")

[{'score': 0.04128897562623024,
  'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, मौसम, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
  'token': 2605,
  'token_str': 'मौसम'},
 {'score': 0.04100276157259941,
  'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, प्रकृति, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
  'token': 2792,
  'token_str': 'प्रकृति'},
 {'score': 0.026525357738137245,
  'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, पानी, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
  'token': 387,
  'token_str': 'पानी'},
 {'score': 0.02340106852352619,
  'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, जल, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
  'token': 1313,
  'token_str': 'जल'},
 {'score': 0.02055591531097889,
  'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, वातावरण, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
  'token': 790,
  'token_str': 'वातावरण'}]
```

Here is how we can use the model to get the features of a given text in PyTorch:

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained('Sakonii/distilbert-base-nepali')
model = AutoModelForMaskedLM.from_pretrained('Sakonii/distilbert-base-nepali')

# prepare input
text = "चाहिएको text यता राख्नु होला।"
encoded_input = tokenizer(text, return_tensors='pt')

# forward pass
output = model(**encoded_input)
```

## Training data

This model is trained on [nepalitext](https://huggingface.co/datasets/Sakonii/nepalitext-language-model-dataset) language modeling dataset which combines the datasets: [OSCAR](https://huggingface.co/datasets/oscar) , [cc100](https://huggingface.co/datasets/cc100) and a set of scraped Nepali articles on Wikipedia.
As for training the language model, the texts in the training set are grouped to a block of 512 tokens.

## Tokenization

A Sentence Piece Model (SPM) is trained on a subset of [nepalitext](https://huggingface.co/datasets/Sakonii/nepalitext-language-model-dataset) dataset for text tokenization. The tokenizer trained with vocab-size=24576, min-frequency=4, limit-alphabet=1000 and model-max-length=512.

## Training procedure

The model is trained with the same configuration as the original [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased); 512 tokens per instance, 28 instances per batch, and around 35.7K training steps.

### Training hyperparameters

The following hyperparameters were used for training of the final epoch: [ Refer to the *Training results* table below for varying hyperparameters every epoch ]
- learning_rate: 5e-05
- train_batch_size: 28
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1
- mixed_precision_training: Native AMP

### Training results

The model is trained for 4 epochs with varying hyperparameters:

| Training Loss | Epoch | MLM Probability | Train Batch Size | Step  | Validation Loss | Perplexity |
|:-------------:|:-----:|:---------------:|:----------------:|:-----:|:---------------:|:----------:|
| 3.4477        | 1.0   | 15              | 26               | 38864 | 3.3067          | 27.2949    |
| 2.9451        | 2.0   | 15              | 28               | 35715 | 2.8238          | 16.8407    |
| 2.866         | 3.0   | 20              | 28               | 35715 | 2.7431          | 15.5351    |
| 2.7287        | 4.0   | 20              | 28               | 35715 | 2.6053          | 13.5353    |
| 2.6412        | 5.0   | 20              | 28               | 35715 | 2.5161          | 12.3802    |

Final model evaluated with MLM Probability of 15%:

| Training Loss | Epoch | MLM Probability | Train Batch Size | Step  | Validation Loss | Perplexity |
|:-------------:|:-----:|:---------------:|:----------------:|:-----:|:---------------:|:----------:|
| -             | -     | 15              | -                | -     | 2.3494          | 10.4791    |


### Framework versions

- Transformers 4.16.2
- Pytorch 1.9.1
- Datasets 1.18.3
- Tokenizers 0.10.3

{"license": "apache-2.0", "tags": ["generated_from_trainer"], "datasets": "Sakonii/nepalitext-language-model-dataset", "mask_token": "<mask>", "widget": [{"text": "\u092e\u093e\u0928\u0935\u093f\u092f \u0917\u0924\u093f\u0935\u093f\u0927\u093f\u0932\u0947 \u092a\u094d\u0930\u093e\u0924\u0943\u0924\u093f\u0915 \u092a\u0930\u094d\u092f\u093e\u0935\u0930\u0928 \u092a\u094d\u0930\u0928\u093e\u0932\u0940\u0932\u093e\u0908 \u0905\u092a\u0930\u093f\u092e\u0947\u092f \u0915\u094d\u0937\u0924\u093f \u092a\u0941\u094d\u0930\u094d\u092f\u093e\u090f\u0915\u094b \u091b\u0964 \u092a\u0930\u093f\u0935\u0930\u094d\u0924\u0928\u0936\u093f\u0932 \u091c\u0932\u0935\u093e\u092f\u0941\u0932\u0947 \u0916\u093e\u0927, \u0938\u0941\u0930\u0915\u094d\u0937\u093e, <mask>, \u091c\u092e\u093f\u0928, \u092e\u094c\u0938\u092e\u0932\u0917\u093e\u092f\u0924\u0932\u093e\u0908 \u0905\u0938\u0902\u0916\u094d\u092f \u0924\u0930\u093f\u0915\u093e\u0932\u0947 \u092a\u094d\u0930\u092d\u093e\u0935\u093f\u0924 \u091b\u0964", "example_title": "Example 1"}, {"text": "\u0905\u091a\u0947\u0932 \u0935\u093f\u0926\u094d\u092f\u093e\u0932\u092f \u0930 \u0915\u0932\u0947\u091c\u0939\u0930\u0942\u0932\u0947 \u0938\u094d\u092e\u093e\u0930\u093f\u0915\u093e \u0915\u0924\u094d\u0924\u093f\u0915\u094b \u092a\u094d\u0930\u0915\u093e\u0936\u0928 \u0917\u0930\u094d\u091b\u0928\u094d, \u092f\u0915\u093f\u0928 \u091b\u0948\u0928\u202f\u0964 \u0915\u0947\u0939\u0940 \u0935\u0930\u094d\u0937\u092a\u0939\u093f\u0932\u0947\u0938\u092e\u094d\u092e \u0917\u093e\u0909\u0901\u0938\u0939\u0930\u0915\u093e \u0938\u093e\u0928\u093e\u0920\u0942\u0932\u093e <mask> \u0938\u0902\u0938\u094d\u0925\u093e\u0939\u0930\u0942\u092e\u093e \u092a\u0941\u0917\u094d\u0926\u093e \u0936\u093f\u0915\u094d\u0937\u0915 \u0935\u093e \u0915\u0930\u094d\u092e\u091a\u093e\u0930\u0940\u0932\u0947 \u0938\u0902\u0938\u094d\u0925\u093e\u092c\u093e\u091f \u092a\u094d\u0930\u0915\u093e\u0936\u093f\u0924 \u092a\u0924\u094d\u0930\u093f\u0915\u093e, \u0938\u094d\u092e\u093e\u0930\u093f\u0915\u093e \u0930 \u092a\u0941\u0938\u094d\u0924\u0915 \u0915\u094b\u0938\u0947\u0932\u0940\u0915\u093e \u0930\u0942\u092a\u092e\u093e \u0925\u092e\u093e\u0909\u0901\u0925\u0947\u202f\u0964", "example_title": "Example 2"}, {"text": "\u091c\u0932\u0935\u093f\u0926\u094d\u092f\u0941\u0924\u094d \u0935\u093f\u0915\u093e\u0938\u0915\u094b \u0967\u0967\u0966 \u0935\u0930\u094d\u0937\u0915\u094b \u0907\u0924\u093f\u0939\u093e\u0938 \u092c\u0928\u093e\u090f\u0915\u094b \u0928\u0947\u092a\u093e\u0932\u092e\u093e \u0939\u093e\u0932 \u0938\u0930\u0915\u093e\u0930\u0940 \u0930 \u0928\u093f\u091c\u0940 \u0915\u094d\u0937\u0947\u0924\u094d\u0930\u092c\u093e\u091f \u0917\u0930\u0940 \u0915\u0930\u093f\u092c \u0968 \u0939\u091c\u093e\u0930 \u092e\u0947\u0917\u093e\u0935\u093e\u091f <mask> \u0909\u0924\u094d\u092a\u093e\u0926\u0928 \u092d\u0907\u0930\u0939\u0947\u0915\u094b \u091b\u202f\u0964", "example_title": "Example 3"}], "model-index": [{"name": "distilbert-base-nepali", "results": []}]}
transformers pytorch safetensors distilbert fill-mask generated_from_trainer dataset:Sakonii/nepalitext-language-model-dataset arxiv:1911.02116 arxiv:1910.01108 license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #generated_from_trainer #dataset-Sakonii/nepalitext-language-model-dataset #arxiv-1911.02116 #arxiv-1910.01108 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

distilbert-base-nepali
======================


This model is pre-trained on nepalitext dataset consisting of over 13 million Nepali text sequences using a masked language modeling (MLM) objective. Our approach trains a Sentence Piece Model (SPM) for text tokenization similar to XLM-ROBERTa and trains distilbert model for language modeling. Find more details in this paper.


It achieves the following results on the evaluation set:



Model description
-----------------


Refer to original distilbert-base-uncased


Intended uses & limitations
---------------------------


This backbone model intends to be fine-tuned on Nepali language focused downstream task such as sequence classification, token classification or question answering.
The language model being trained on a data with texts grouped to a block size of 512, it handles text sequence up to 512 tokens and may not perform satisfactorily on shorter sequences.


Usage
-----


This model can be used directly with a pipeline for masked language modeling:


Here is how we can use the model to get the features of a given text in PyTorch:


Training data
-------------


This model is trained on nepalitext language modeling dataset which combines the datasets: OSCAR , cc100 and a set of scraped Nepali articles on Wikipedia.
As for training the language model, the texts in the training set are grouped to a block of 512 tokens.


Tokenization
------------


A Sentence Piece Model (SPM) is trained on a subset of nepalitext dataset for text tokenization. The tokenizer trained with vocab-size=24576, min-frequency=4, limit-alphabet=1000 and model-max-length=512.


Training procedure
------------------


The model is trained with the same configuration as the original distilbert-base-uncased; 512 tokens per instance, 28 instances per batch, and around 35.7K training steps.


### Training hyperparameters


The following hyperparameters were used for training of the final epoch: [ Refer to the *Training results* table below for varying hyperparameters every epoch ]


* learning\_rate: 5e-05
* train\_batch\_size: 28
* eval\_batch\_size: 8
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 1
* mixed\_precision\_training: Native AMP


### Training results


The model is trained for 4 epochs with varying hyperparameters:



Final model evaluated with MLM Probability of 15%:



### Framework versions


* Transformers 4.16.2
* Pytorch 1.9.1
* Datasets 1.18.3
* Tokenizers 0.10.3



### Training hyperparameters


The following hyperparameters were used for training of the final epoch: [ Refer to the *Training results* table below for varying hyperparameters every epoch ]


* learning\_rate: 5e-05
* train\_batch\_size: 28
* eval\_batch\_size: 8
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 1
* mixed\_precision\_training: Native AMP ### Training results


The model is trained for 4 epochs with varying hyperparameters:



Final model evaluated with MLM Probability of 15%: ### Framework versions


* Transformers 4.16.2
* Pytorch 1.9.1
* Datasets 1.18.3
* Tokenizers 0.10.3
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #generated_from_trainer #dataset-Sakonii/nepalitext-language-model-dataset #arxiv-1911.02116 #arxiv-1910.01108 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used for training of the final epoch: [ Refer to the *Training results* table below for varying hyperparameters every epoch ]


* learning\_rate: 5e-05
* train\_batch\_size: 28
* eval\_batch\_size: 8
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 1
* mixed\_precision\_training: Native AMP ### Training results


The model is trained for 4 epochs with varying hyperparameters:



Final model evaluated with MLM Probability of 15%: ### Framework versions


* Transformers 4.16.2
* Pytorch 1.9.1
* Datasets 1.18.3
* Tokenizers 0.10.3
passage: TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #generated_from_trainer #dataset-Sakonii/nepalitext-language-model-dataset #arxiv-1911.02116 #arxiv-1910.01108 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used for training of the final epoch: [ Refer to the *Training results* table below for varying hyperparameters every epoch ]


* learning\_rate: 5e-05
* train\_batch\_size: 28
* eval\_batch\_size: 8
* seed: 42
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* num\_epochs: 1
* mixed\_precision\_training: Native AMP### Training results


The model is trained for 4 epochs with varying hyperparameters:



Final model evaluated with MLM Probability of 15%:### Framework versions


* Transformers 4.16.2
* Pytorch 1.9.1
* Datasets 1.18.3
* Tokenizers 0.10.3