
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Centrum

Centrum is a pretrained model for multi-document summarization, trained with centroid-based pretraining objective on the NewSHead dataset. It is initialized from [allenai/led-base-16384](https://huggingface.co/allenai/led-base-16384). The details of the approach are mentioned in the preprint [Multi-Document Summarization with Centroid-Based Pretraining](https://arxiv.org/abs/2208.01006) (Ratish Puduppully and Mark Steedman).
It achieves the following results on the evaluation set:
- Loss: 3.5568

## Model description

The script for training and inference of Centrum is available on https://github.com/ratishsp/centrum

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 1
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- training_steps: 100000
- mixed_precision_training: Native AMP
- label_smoothing_factor: 0.1

### Training results

| Training Loss | Epoch | Step   | Validation Loss |
|:-------------:|:-----:|:------:|:---------------:|
| 4.1628        | 0.05  | 500    | 4.0732          |
| 4.0278        | 0.09  | 1000   | 3.9800          |
| 4.0008        | 0.14  | 1500   | 3.9283          |
| 3.9564        | 0.19  | 2000   | 3.8941          |
| 3.9193        | 0.23  | 2500   | 3.8780          |
| 3.9185        | 0.28  | 3000   | 3.8501          |
| 3.8881        | 0.32  | 3500   | 3.8334          |
| 3.8869        | 0.37  | 4000   | 3.8211          |
| 3.876         | 0.42  | 4500   | 3.8057          |
| 3.8552        | 0.46  | 5000   | 3.7954          |
| 3.8198        | 0.51  | 5500   | 3.7861          |
| 3.8016        | 0.56  | 6000   | 3.7750          |
| 3.8033        | 0.6   | 6500   | 3.7651          |
| 3.7927        | 0.65  | 7000   | 3.7528          |
| 3.7978        | 0.7   | 7500   | 3.7429          |
| 3.7727        | 0.74  | 8000   | 3.7367          |
| 3.7634        | 0.79  | 8500   | 3.7275          |
| 3.7395        | 0.83  | 9000   | 3.7158          |
| 3.7432        | 0.88  | 9500   | 3.7066          |
| 3.7623        | 0.93  | 10000  | 3.7039          |
| 3.7182        | 0.97  | 10500  | 3.6904          |
| 3.7146        | 1.02  | 11000  | 3.6881          |
| 3.681         | 1.07  | 11500  | 3.6797          |
| 3.6745        | 1.11  | 12000  | 3.6750          |
| 3.6794        | 1.16  | 12500  | 3.6748          |
| 3.6802        | 1.21  | 13000  | 3.6696          |
| 3.665         | 1.25  | 13500  | 3.6609          |
| 3.6516        | 1.3   | 14000  | 3.6633          |
| 3.6577        | 1.34  | 14500  | 3.6573          |
| 3.6409        | 1.39  | 15000  | 3.6519          |
| 3.6691        | 1.44  | 15500  | 3.6490          |
| 3.6521        | 1.48  | 16000  | 3.6475          |
| 3.6435        | 1.53  | 16500  | 3.6465          |
| 3.6466        | 1.58  | 17000  | 3.6392          |
| 3.644         | 1.62  | 17500  | 3.6419          |
| 3.6347        | 1.67  | 18000  | 3.6347          |
| 3.6205        | 1.71  | 18500  | 3.6328          |
| 3.6451        | 1.76  | 19000  | 3.6310          |
| 3.6327        | 1.81  | 19500  | 3.6284          |
| 3.6166        | 1.85  | 20000  | 3.6267          |
| 3.622         | 1.9   | 20500  | 3.6212          |
| 3.6164        | 1.95  | 21000  | 3.6199          |
| 3.6178        | 1.99  | 21500  | 3.6201          |
| 3.5892        | 2.04  | 22000  | 3.6201          |
| 3.5855        | 2.09  | 22500  | 3.6221          |
| 3.5658        | 2.13  | 23000  | 3.6193          |
| 3.5916        | 2.18  | 23500  | 3.6144          |
| 3.5767        | 2.22  | 24000  | 3.6101          |
| 3.5809        | 2.27  | 24500  | 3.6115          |
| 3.5561        | 2.32  | 25000  | 3.6110          |
| 3.5831        | 2.36  | 25500  | 3.6080          |
| 3.5551        | 2.41  | 26000  | 3.6121          |
| 3.5588        | 2.46  | 26500  | 3.6072          |
| 3.5645        | 2.5   | 27000  | 3.6056          |
| 3.5804        | 2.55  | 27500  | 3.6038          |
| 3.5712        | 2.6   | 28000  | 3.6052          |
| 3.5494        | 2.64  | 28500  | 3.6014          |
| 3.582         | 2.69  | 29000  | 3.5995          |
| 3.5487        | 2.73  | 29500  | 3.6051          |
| 3.5709        | 2.78  | 30000  | 3.5954          |
| 3.5546        | 2.83  | 30500  | 3.5941          |
| 3.5525        | 2.87  | 31000  | 3.5952          |
| 3.5603        | 2.92  | 31500  | 3.5972          |
| 3.5572        | 2.97  | 32000  | 3.5947          |
| 3.5106        | 3.01  | 32500  | 3.5952          |
| 3.5142        | 3.06  | 33000  | 3.5937          |
| 3.506         | 3.11  | 33500  | 3.5965          |
| 3.515         | 3.15  | 34000  | 3.5932          |
| 3.5247        | 3.2   | 34500  | 3.5951          |
| 3.5384        | 3.24  | 35000  | 3.5917          |
| 3.5165        | 3.29  | 35500  | 3.5887          |
| 3.5187        | 3.34  | 36000  | 3.5866          |
| 3.5097        | 3.38  | 36500  | 3.5895          |
| 3.5136        | 3.43  | 37000  | 3.5878          |
| 3.5095        | 3.48  | 37500  | 3.5839          |
| 3.5226        | 3.52  | 38000  | 3.5859          |
| 3.5277        | 3.57  | 38500  | 3.5827          |
| 3.4959        | 3.62  | 39000  | 3.5846          |
| 3.5003        | 3.66  | 39500  | 3.5823          |
| 3.5095        | 3.71  | 40000  | 3.5820          |
| 3.4814        | 3.75  | 40500  | 3.5854          |
| 3.5173        | 3.8   | 41000  | 3.5796          |
| 3.4968        | 3.85  | 41500  | 3.5810          |
| 3.5183        | 3.89  | 42000  | 3.5783          |
| 3.512         | 3.94  | 42500  | 3.5784          |
| 3.5069        | 3.99  | 43000  | 3.5775          |
| 3.5014        | 4.03  | 43500  | 3.5819          |
| 3.4787        | 4.08  | 44000  | 3.5836          |
| 3.4625        | 4.12  | 44500  | 3.5788          |
| 3.4902        | 4.17  | 45000  | 3.5784          |
| 3.4927        | 4.22  | 45500  | 3.5773          |
| 3.4813        | 4.26  | 46000  | 3.5769          |
| 3.4637        | 4.31  | 46500  | 3.5761          |
| 3.4731        | 4.36  | 47000  | 3.5771          |
| 3.4856        | 4.4   | 47500  | 3.5786          |
| 3.4579        | 4.45  | 48000  | 3.5790          |
| 3.5032        | 4.5   | 48500  | 3.5738          |
| 3.4826        | 4.54  | 49000  | 3.5749          |
| 3.4709        | 4.59  | 49500  | 3.5746          |
| 3.4916        | 4.63  | 50000  | 3.5745          |
| 3.4715        | 4.68  | 50500  | 3.5706          |
| 3.4926        | 4.73  | 51000  | 3.5729          |
| 3.4974        | 4.77  | 51500  | 3.5725          |
| 3.4796        | 4.82  | 52000  | 3.5683          |
| 3.4817        | 4.87  | 52500  | 3.5707          |
| 3.4683        | 4.91  | 53000  | 3.5721          |
| 3.4986        | 4.96  | 53500  | 3.5689          |
| 3.4763        | 5.01  | 54000  | 3.5716          |
| 3.4668        | 5.05  | 54500  | 3.5700          |
| 3.4274        | 5.1   | 55000  | 3.5724          |
| 3.4499        | 5.14  | 55500  | 3.5717          |
| 3.4507        | 5.19  | 56000  | 3.5706          |
| 3.4343        | 5.24  | 56500  | 3.5697          |
| 3.4151        | 5.28  | 57000  | 3.5710          |
| 3.4469        | 5.33  | 57500  | 3.5712          |
| 3.458         | 5.38  | 58000  | 3.5692          |
| 3.4559        | 5.42  | 58500  | 3.5680          |
| 3.4354        | 5.47  | 59000  | 3.5683          |
| 3.4479        | 5.52  | 59500  | 3.5703          |
| 3.4627        | 5.56  | 60000  | 3.5678          |
| 3.4478        | 5.61  | 60500  | 3.5659          |
| 3.4645        | 5.65  | 61000  | 3.5675          |
| 3.4658        | 5.7   | 61500  | 3.5666          |
| 3.4657        | 5.75  | 62000  | 3.5658          |
| 3.4618        | 5.79  | 62500  | 3.5653          |
| 3.4541        | 5.84  | 63000  | 3.5653          |
| 3.4552        | 5.89  | 63500  | 3.5648          |
| 3.4679        | 5.93  | 64000  | 3.5648          |
| 3.4423        | 5.98  | 64500  | 3.5652          |
| 3.3893        | 6.03  | 65000  | 3.5646          |
| 3.4239        | 6.07  | 65500  | 3.5668          |
| 3.4329        | 6.12  | 66000  | 3.5639          |
| 3.4151        | 6.16  | 66500  | 3.5649          |
| 3.4181        | 6.21  | 67000  | 3.5682          |
| 3.4314        | 6.26  | 67500  | 3.5669          |
| 3.4245        | 6.3   | 68000  | 3.5629          |
| 3.421         | 6.35  | 68500  | 3.5663          |
| 3.4329        | 6.4   | 69000  | 3.5660          |
| 3.4122        | 6.44  | 69500  | 3.5651          |
| 3.4362        | 6.49  | 70000  | 3.5628          |
| 3.4497        | 6.54  | 70500  | 3.5648          |
| 3.431         | 6.58  | 71000  | 3.5626          |
| 3.432         | 6.63  | 71500  | 3.5648          |
| 3.4208        | 6.67  | 72000  | 3.5635          |
| 3.4526        | 6.72  | 72500  | 3.5645          |
| 3.4139        | 6.77  | 73000  | 3.5621          |
| 3.4212        | 6.81  | 73500  | 3.5629          |
| 3.4352        | 6.86  | 74000  | 3.5597          |
| 3.4242        | 6.91  | 74500  | 3.5597          |
| 3.429         | 6.95  | 75000  | 3.5619          |
| 3.4133        | 7.0   | 75500  | 3.5592          |
| 3.4086        | 7.04  | 76000  | 3.5621          |
| 3.4056        | 7.09  | 76500  | 3.5604          |
| 3.4158        | 7.14  | 77000  | 3.5629          |
| 3.4153        | 7.18  | 77500  | 3.5609          |
| 3.4155        | 7.23  | 78000  | 3.5621          |
| 3.4117        | 7.28  | 78500  | 3.5626          |
| 3.407         | 7.32  | 79000  | 3.5638          |
| 3.3977        | 7.37  | 79500  | 3.5604          |
| 3.4134        | 7.42  | 80000  | 3.5611          |
| 3.4403        | 7.46  | 80500  | 3.5630          |
| 3.4002        | 7.51  | 81000  | 3.5601          |
| 3.4147        | 7.55  | 81500  | 3.5577          |
| 3.4068        | 7.6   | 82000  | 3.5588          |
| 3.4165        | 7.65  | 82500  | 3.5613          |
| 3.409         | 7.69  | 83000  | 3.5596          |
| 3.4213        | 7.74  | 83500  | 3.5583          |
| 3.403         | 7.79  | 84000  | 3.5601          |
| 3.3819        | 7.83  | 84500  | 3.5580          |
| 3.4182        | 7.88  | 85000  | 3.5570          |
| 3.4099        | 7.93  | 85500  | 3.5570          |
| 3.3845        | 7.97  | 86000  | 3.5582          |
| 3.411         | 8.02  | 86500  | 3.5610          |
| 3.3952        | 8.06  | 87000  | 3.5588          |
| 3.4211        | 8.11  | 87500  | 3.5588          |
| 3.4171        | 8.16  | 88000  | 3.5570          |
| 3.3825        | 8.2   | 88500  | 3.5607          |
| 3.3807        | 8.25  | 89000  | 3.5579          |
| 3.3842        | 8.3   | 89500  | 3.5583          |
| 3.3809        | 8.34  | 90000  | 3.5596          |
| 3.4033        | 8.39  | 90500  | 3.5590          |
| 3.4156        | 8.44  | 91000  | 3.5577          |
| 3.3927        | 8.48  | 91500  | 3.5585          |
| 3.4041        | 8.53  | 92000  | 3.5596          |
| 3.4006        | 8.57  | 92500  | 3.5600          |
| 3.4007        | 8.62  | 93000  | 3.5578          |
| 3.4047        | 8.67  | 93500  | 3.5572          |
| 3.3904        | 8.71  | 94000  | 3.5571          |
| 3.3888        | 8.76  | 94500  | 3.5581          |
| 3.3876        | 8.81  | 95000  | 3.5572          |
| 3.3872        | 8.85  | 95500  | 3.5575          |
| 3.3753        | 8.9   | 96000  | 3.5577          |
| 3.3961        | 8.95  | 96500  | 3.5568          |
| 3.4131        | 8.99  | 97000  | 3.5579          |
| 3.3647        | 9.04  | 97500  | 3.5573          |
| 3.3792        | 9.08  | 98000  | 3.5576          |
| 3.3755        | 9.13  | 98500  | 3.5575          |
| 3.3981        | 9.18  | 99000  | 3.5573          |
| 3.3914        | 9.22  | 99500  | 3.5573          |
| 3.4136        | 9.27  | 100000 | 3.5575          |


### Framework versions

- Transformers 4.20.0.dev0
- Pytorch 1.11.0
- Datasets 2.2.2
- Tokenizers 0.12.1

{"license": "apache-2.0", "tags": ["generated_from_trainer"], "datasets": ["NewSHead"], "model-index": [{"name": "Centrum", "results": []}]}
transformers pytorch led text2text-generation generated_from_trainer dataset:NewSHead arxiv:2208.01006 license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #led #text2text-generation #generated_from_trainer #dataset-NewSHead #arxiv-2208.01006 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

Centrum
=======


Centrum is a pretrained model for multi-document summarization, trained with centroid-based pretraining objective on the NewSHead dataset. It is initialized from allenai/led-base-16384. The details of the approach are mentioned in the preprint Multi-Document Summarization with Centroid-Based Pretraining (Ratish Puduppully and Mark Steedman).
It achieves the following results on the evaluation set:


* Loss: 3.5568


Model description
-----------------


The script for training and inference of Centrum is available on URL


Intended uses & limitations
---------------------------


More information needed


Training and evaluation data
----------------------------


More information needed


Training procedure
------------------


### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 3e-05
* train\_batch\_size: 1
* eval\_batch\_size: 4
* seed: 42
* distributed\_type: multi-GPU
* num\_devices: 4
* gradient\_accumulation\_steps: 4
* total\_train\_batch\_size: 16
* total\_eval\_batch\_size: 16
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 10000
* training\_steps: 100000
* mixed\_precision\_training: Native AMP
* label\_smoothing\_factor: 0.1


### Training results



### Framework versions


* Transformers 4.20.0.dev0
* Pytorch 1.11.0
* Datasets 2.2.2
* Tokenizers 0.12.1



### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 3e-05
* train\_batch\_size: 1
* eval\_batch\_size: 4
* seed: 42
* distributed\_type: multi-GPU
* num\_devices: 4
* gradient\_accumulation\_steps: 4
* total\_train\_batch\_size: 16
* total\_eval\_batch\_size: 16
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 10000
* training\_steps: 100000
* mixed\_precision\_training: Native AMP
* label\_smoothing\_factor: 0.1 ### Training results ### Framework versions


* Transformers 4.20.0.dev0
* Pytorch 1.11.0
* Datasets 2.2.2
* Tokenizers 0.12.1
TAGS
#transformers #pytorch #led #text2text-generation #generated_from_trainer #dataset-NewSHead #arxiv-2208.01006 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 3e-05
* train\_batch\_size: 1
* eval\_batch\_size: 4
* seed: 42
* distributed\_type: multi-GPU
* num\_devices: 4
* gradient\_accumulation\_steps: 4
* total\_train\_batch\_size: 16
* total\_eval\_batch\_size: 16
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 10000
* training\_steps: 100000
* mixed\_precision\_training: Native AMP
* label\_smoothing\_factor: 0.1 ### Training results ### Framework versions


* Transformers 4.20.0.dev0
* Pytorch 1.11.0
* Datasets 2.2.2
* Tokenizers 0.12.1
passage: TAGS
#transformers #pytorch #led #text2text-generation #generated_from_trainer #dataset-NewSHead #arxiv-2208.01006 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 3e-05
* train\_batch\_size: 1
* eval\_batch\_size: 4
* seed: 42
* distributed\_type: multi-GPU
* num\_devices: 4
* gradient\_accumulation\_steps: 4
* total\_train\_batch\_size: 16
* total\_eval\_batch\_size: 16
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 10000
* training\_steps: 100000
* mixed\_precision\_training: Native AMP
* label\_smoothing\_factor: 0.1### Training results### Framework versions


* Transformers 4.20.0.dev0
* Pytorch 1.11.0
* Datasets 2.2.2
* Tokenizers 0.12.1