
# RuGPT3MediumSumGazeta

## Model description

This is the model for abstractive summarization for Russian based on [rugpt3medium_based_on_gpt2](https://huggingface.co/sberbank-ai/rugpt3medium_based_on_gpt2).


## Intended uses & limitations

#### How to use

Colab: [link](https://colab.research.google.com/drive/1eR-ev0Y5ISWIwGnzYYoHyGMaSIUz8GTN)

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "IlyaGusev/rugpt3medium_sum_gazeta"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

article_text = "..."

text_tokens = tokenizer(
    article_text,
    max_length=600,
    add_special_tokens=False, 
    padding=False,
    truncation=True
)["input_ids"]
input_ids = text_tokens + [tokenizer.sep_token_id]
input_ids = torch.LongTensor([input_ids])

output_ids = model.generate(
    input_ids=input_ids,
    no_repeat_ngram_size=4
)

summary = tokenizer.decode(output_ids[0], skip_special_tokens=False)
summary = summary.split(tokenizer.sep_token)[1]
summary = summary.split(tokenizer.eos_token)[0]
print(summary)
```

## Training data

- Dataset: [Gazeta](https://huggingface.co/datasets/IlyaGusev/gazeta)

## Training procedure

- Training script: [train.py](https://github.com/IlyaGusev/summarus/blob/master/external/hf_scripts/train.py)
- Config: [gpt_training_config.json](https://github.com/IlyaGusev/summarus/blob/master/external/hf_scripts/configs/gpt_training_config.json)

## Eval results

* Train dataset: **Gazeta v1 train**
* Test dataset: **Gazeta v1 test**
* Source max_length: **600**
* Target max_length: **200**
* no_repeat_ngram_size: **4**
* num_beams: **5**

| Model                     | R-1-f | R-2-f | R-L-f | chrF | METEOR | BLEU | Avg char length |
|:--------------------------|:------|:------|:------|:-------|:-------|:-----|:-----|
| [mbart_ru_sum_gazeta](https://huggingface.co/IlyaGusev/mbart_ru_sum_gazeta)       | **32.4**  | 14.3  | 28.0  | 39.7 | **26.4** | 12.1 | 371 |
| [rut5_base_sum_gazeta](https://huggingface.co/IlyaGusev/rut5_base_sum_gazeta)      | 32.2  | **14.4**  | **28.1** | **39.8** | 25.7 | **12.3** | 330 |
| [rugpt3medium_sum_gazeta](https://huggingface.co/IlyaGusev/rugpt3medium_sum_gazeta) | 26.2 | 7.7 | 21.7 | 33.8 | 18.2 | 4.3 | 244 |

* Train dataset: **Gazeta v1 train**
* Test dataset: **Gazeta v2 test**
* Source max_length: **600**
* Target max_length: **200**
* no_repeat_ngram_size: **4**
* num_beams: **5**

| Model                     | R-1-f | R-2-f | R-L-f | chrF | METEOR | BLEU | Avg char length |
|:--------------------------|:------|:------|:------|:-------|:-------|:-----|:-----|
| [mbart_ru_sum_gazeta](https://huggingface.co/IlyaGusev/mbart_ru_sum_gazeta)        | **28.7**  | **11.1**  | 24.4  | **37.3** | **22.7**  | **9.4** | 373 |
| [rut5_base_sum_gazeta](https://huggingface.co/IlyaGusev/rut5_base_sum_gazeta)      | 28.6 | **11.1** | **24.5** | 37.2 | 22.0 | **9.4** | 331 |
| [rugpt3medium_sum_gazeta](https://huggingface.co/IlyaGusev/rugpt3medium_sum_gazeta) | 24.1 | 6.5 | 19.8 | 32.1 | 16.3 | 3.6 | 242 |

Evaluation script: [evaluate.py](https://github.com/IlyaGusev/summarus/blob/master/evaluate.py)

Flags: --language ru --tokenize-after --lower

{"language": ["ru"], "license": ["apache-2.0"], "tags": ["causal-lm", "summarization"], "datasets": ["IlyaGusev/gazeta"], "inference": false, "widget": [{"text": "\u0412\u044b\u0441\u043e\u0442\u0430 \u0431\u0430\u0448\u043d\u0438 \u0441\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442 324 \u043c\u0435\u0442\u0440\u0430 (1063 \u0444\u0443\u0442\u0430), \u043f\u0440\u0438\u043c\u0435\u0440\u043d\u043e \u0442\u0430\u043a\u0430\u044f \u0436\u0435 \u0432\u044b\u0441\u043e\u0442\u0430, \u043a\u0430\u043a \u0443 81-\u044d\u0442\u0430\u0436\u043d\u043e\u0433\u043e \u0437\u0434\u0430\u043d\u0438\u044f, \u0438 \u0441\u0430\u043c\u043e\u0435 \u0432\u044b\u0441\u043e\u043a\u043e\u0435 \u0441\u043e\u043e\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 \u041f\u0430\u0440\u0438\u0436\u0435. \u0415\u0433\u043e \u043e\u0441\u043d\u043e\u0432\u0430\u043d\u0438\u0435 \u043a\u0432\u0430\u0434\u0440\u0430\u0442\u043d\u043e, \u0440\u0430\u0437\u043c\u0435\u0440\u043e\u043c 125 \u043c\u0435\u0442\u0440\u043e\u0432 (410 \u0444\u0443\u0442\u043e\u0432) \u0441 \u043b\u044e\u0431\u043e\u0439 \u0441\u0442\u043e\u0440\u043e\u043d\u044b. \u0412\u043e \u0432\u0440\u0435\u043c\u044f \u0441\u0442\u0440\u043e\u0438\u0442\u0435\u043b\u044c\u0441\u0442\u0432\u0430 \u042d\u0439\u0444\u0435\u043b\u0435\u0432\u0430 \u0431\u0430\u0448\u043d\u044f \u043f\u0440\u0435\u0432\u0437\u043e\u0448\u043b\u0430 \u043c\u043e\u043d\u0443\u043c\u0435\u043d\u0442 \u0412\u0430\u0448\u0438\u043d\u0433\u0442\u043e\u043d\u0430, \u0441\u0442\u0430\u0432 \u0441\u0430\u043c\u044b\u043c \u0432\u044b\u0441\u043e\u043a\u0438\u043c \u0438\u0441\u043a\u0443\u0441\u0441\u0442\u0432\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u043e\u0440\u0443\u0436\u0435\u043d\u0438\u0435\u043c \u0432 \u043c\u0438\u0440\u0435, \u0438 \u044d\u0442\u043e\u0442 \u0442\u0438\u0442\u0443\u043b \u043e\u043d\u0430 \u0443\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u043b\u0430 \u0432 \u0442\u0435\u0447\u0435\u043d\u0438\u0435 41 \u0433\u043e\u0434\u0430 \u0434\u043e \u0437\u0430\u0432\u0435\u0440\u0448\u0435\u043d\u0438\u044f \u0441\u0442\u0440\u043e\u0438\u0442\u0435\u043b\u044c\u0441\u0442\u0432\u043e \u0437\u0434\u0430\u043d\u0438\u044f \u041a\u0440\u0430\u0439\u0441\u043b\u0435\u0440 \u0432 \u041d\u044c\u044e-\u0419\u043e\u0440\u043a\u0435 \u0432 1930 \u0433\u043e\u0434\u0443. \u042d\u0442\u043e \u043f\u0435\u0440\u0432\u043e\u0435 \u0441\u043e\u043e\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0434\u043e\u0441\u0442\u0438\u0433\u043b\u043e \u0432\u044b\u0441\u043e\u0442\u044b 300 \u043c\u0435\u0442\u0440\u043e\u0432. \u0418\u0437-\u0437\u0430 \u0434\u043e\u0431\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u0432\u0435\u0449\u0430\u0442\u0435\u043b\u044c\u043d\u043e\u0439 \u0430\u043d\u0442\u0435\u043d\u043d\u044b \u043d\u0430 \u0432\u0435\u0440\u0448\u0438\u043d\u0435 \u0431\u0430\u0448\u043d\u0438 \u0432 1957 \u0433\u043e\u0434\u0443 \u043e\u043d\u0430 \u0441\u0435\u0439\u0447\u0430\u0441 \u0432\u044b\u0448\u0435 \u0437\u0434\u0430\u043d\u0438\u044f \u041a\u0440\u0430\u0439\u0441\u043b\u0435\u0440 \u043d\u0430 5,2 \u043c\u0435\u0442\u0440\u0430 (17 \u0444\u0443\u0442\u043e\u0432). \u0417\u0430 \u0438\u0441\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435\u043c \u043f\u0435\u0440\u0435\u0434\u0430\u0442\u0447\u0438\u043a\u043e\u0432, \u042d\u0439\u0444\u0435\u043b\u0435\u0432\u0430 \u0431\u0430\u0448\u043d\u044f \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0432\u0442\u043e\u0440\u043e\u0439 \u0441\u0430\u043c\u043e\u0439 \u0432\u044b\u0441\u043e\u043a\u043e\u0439 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u043e \u0441\u0442\u043e\u044f\u0449\u0435\u0439 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439 \u0432\u043e \u0424\u0440\u0430\u043d\u0446\u0438\u0438 \u043f\u043e\u0441\u043b\u0435 \u0432\u0438\u0430\u0434\u0443\u043a\u0430 \u041c\u0438\u0439\u043e.<s>", "example_title": "\u0412\u0438\u043a\u0438\u043f\u0435\u0434\u0438\u044f"}]}
transformers pytorch gpt2 text-generation causal-lm summarization ru dataset:IlyaGusev/gazeta license:apache-2.0 autotrain_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #gpt2 #text-generation #causal-lm #summarization #ru #dataset-IlyaGusev/gazeta #license-apache-2.0 #autotrain_compatible #text-generation-inference #region-us 

RuGPT3MediumSumGazeta
=====================


Model description
-----------------


This is the model for abstractive summarization for Russian based on rugpt3medium\_based\_on\_gpt2.


Intended uses & limitations
---------------------------


#### How to use


Colab: link


Training data
-------------


* Dataset: Gazeta


Training procedure
------------------


* Training script: URL
* Config: gpt\_training\_config.json


Eval results
------------


* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v1 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v2 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



Evaluation script: URL


Flags: --language ru --tokenize-after --lower



#### How to use


Colab: link


Training data
-------------


* Dataset: Gazeta


Training procedure
------------------


* Training script: URL
* Config: gpt\_training\_config.json


Eval results
------------


* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v1 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v2 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



Evaluation script: URL


Flags: --language ru --tokenize-after --lower
TAGS
#transformers #pytorch #gpt2 #text-generation #causal-lm #summarization #ru #dataset-IlyaGusev/gazeta #license-apache-2.0 #autotrain_compatible #text-generation-inference #region-us 
 #### How to use


Colab: link


Training data
-------------


* Dataset: Gazeta


Training procedure
------------------


* Training script: URL
* Config: gpt\_training\_config.json


Eval results
------------


* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v1 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v2 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



Evaluation script: URL


Flags: --language ru --tokenize-after --lower
passage: TAGS
#transformers #pytorch #gpt2 #text-generation #causal-lm #summarization #ru #dataset-IlyaGusev/gazeta #license-apache-2.0 #autotrain_compatible #text-generation-inference #region-us 
#### How to use


Colab: link


Training data
-------------


* Dataset: Gazeta


Training procedure
------------------


* Training script: URL
* Config: gpt\_training\_config.json


Eval results
------------


* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v1 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



* Train dataset: Gazeta v1 train
* Test dataset: Gazeta v2 test
* Source max\_length: 600
* Target max\_length: 200
* no\_repeat\_ngram\_size: 4
* num\_beams: 5



Evaluation script: URL


Flags: --language ru --tokenize-after --lower