pytorch longt5 text2text-generation dataset:kmfoda/booksum transformers summarization summary booksum long-document long-form license:apache-2.0 model-index autotrain_compatible
---
tags:
- summarization
- summary
- booksum
- long-document
- long-form
license: apache-2.0
datasets:
- kmfoda/booksum
metrics:
- rouge
inference: False
model-index:
- name: pszemraj/long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: kmfoda/booksum
      type: kmfoda/booksum
      config: kmfoda--booksum
      split: test
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 34.5471
      verified: true
    - name: ROUGE-2
      type: rouge
      value: 5.8126
      verified: true
    - name: ROUGE-L
      type: rouge
      value: 15.5684
      verified: true
    - name: ROUGE-LSUM
      type: rouge
      value: 31.878
      verified: true
    - name: loss
      type: loss
      value: .nan
      verified: true
    - name: gen_len
      type: gen_len
      value: 294.7317
      verified: true
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: samsum
      type: samsum
      config: samsum
      split: test
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 25.4769
      verified: true
    - name: ROUGE-2
      type: rouge
      value: 6.0336
      verified: true
    - name: ROUGE-L
      type: rouge
      value: 17.9249
      verified: true
    - name: ROUGE-LSUM
      type: rouge
      value: 22.0692
      verified: true
    - name: loss
      type: loss
      value: .nan
      verified: true
    - name: gen_len
      type: gen_len
      value: 53.8242
      verified: true
---

# long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP

> NOTE: this is still a work-in-progress (WIP) and not completed/converged by any means, but sharing to maybe save some time for others :) 

## Updates

_As I make updates to this WIP checkpoint I will post a note here._

- July 8, 2022: add checkpoint with ~4 epochs of training on A100, equating to approx 350 steps of functional batch size 128
- July 4, 2022: add checkpoint with 6 additional epochs of training with the dataset summary outputs filtered to 1024 **tokens**, resolving prior issue of short summaries.

## About

- a checkpoint of [Stancld/longt5-tglobal-large-16384-pubmed-3k_steps](https://huggingface.co/Stancld/longt5-tglobal-large-16384-pubmed-3k_steps) trained on `kmfoda/booksum` for about 26 epochs 
- max input lengths during training vary between 8192 and 16384 tokens depending on GPU availability. This checkpoint was **trained with 16384 tokens as the max input length for the final 10+ epochs**

  
 ## Comparisons
 
 - compare to [pszemraj/led-large-book-summary](https://huggingface.co/pszemraj/led-large-book-summary). 
   - **inference API has been disabled because it's too compute-intensive :/**
 