pytorch led text2text-generation dataset:billsum arxiv:2004.05150 transformers summarization license:apache-2.0 model-index autotrain_compatible
---
license: apache-2.0
datasets: billsum
tags:
- summarization
widget:
- text: "The people of the State of California do enact as follows: SECTIONHEADER Section 1170.02 is added to the Penal Code, to read: 1170.02. A prisoner is not eligible for resentence or recall pursuant to subdivision (e) of Section 1170 if he or she was convicted of first-degree murder if the victim was a peace officer, as defined in Section 830.1, 830.2, 830.3, 830.31, 830.32, 830.33, 830.34, 830.35, 830.36, 830.37, 830.4, 830.5, 830.6, 830.10, 830.11, or 830.12, who was killed while engaged in the performance of his or her duties, and the individual knew, or reasonably should have known, that the victim was a peace officer engaged in the performance of his or her duties, or the victim was a peace officer or a former peace officer under any of the above-enumerated sections, and was intentionally killed in retaliation for the performance of his or her official duties. SECTIONHEADER Section 3550 of the Penal Code is amended to read: 3550. Notwithstanding any other law, except as provided in subdivision (b), if the head physician of an institution in which a prisoner is incarcerated determines, as provided in this section, that the prisoner is permanently medically incapacitated with a medical condition that renders him or her permanently unable to perform activities of basic daily living, and results in the prisoner requiring 24-hour care, and that incapacitation did not exist at the time of sentencing, the prisoner shall be granted medical parole if the Board of Parole Hearings determines that the conditions under which he or she would be released would not reasonably pose a threat to public safety. This section does not alter or diminish the rights conferred under the Victims Bill of Rights Act of 2008 . Subdivision (a) does not apply to any of the following: A prisoner sentenced to death or life in prison without possibility of parole. A prisoner who is serving a sentence for which parole, pursuant to subdivision (a), is prohibited by any initiative statute. A prisoner who was convicted of first-degree murder if the victim was a peace officer, as defined in Section 830.1, 830.2, 830.3, 830.31, 830.32, 830.33, 830.34, 830.35, 830.36, 830.37, 830.4, 830.5, 830.6, 830.10, 830.11, or 830.12, who was killed while engaged in the performance of his or her duties, and the individual knew, or reasonably should have known, that the victim was a peace officer engaged in the performance of his or her duties, or the victim was a peace officer or a former peace officer under any of the above-enumerated sections, and was intentionally killed in retaliation for the performance of his or her official duties. When a physician employed by the Department of Corrections and Rehabilitation who is the primary care provider for a prisoner identifies a prisoner that he or she believes meets the medical criteria for medical parole specified in subdivision (a), the primary care physician shall recommend to the head physician of the institution where the prisoner is located that the prisoner be referred to the Board of Parole Hearings for consideration for medical parole. Within 30 days of receiving that recommendation, if the head physician of the institution concurs in the recommendation of the primary care physician, he or she shall refer the matter to the Board of Parole Hearings using a standardized form and format developed by the department, and if the head physician of the institution does not concur in the recommendation, he or she shall provide the primary care physician with a written explanation of the reasons for denying the referral. Notwithstanding any other provisions of this section, the prisoner or his or her family member or designee may independently request consideration for medical parole by contacting the head physician at the prison or the department. Within 30 days of receiving the request, the head physician of the institution shall, in consultation with the prisoners primary care physician, make a determination regarding whether the prisoner meets the criteria for medical parole as specified in subdivision (a) and, if the head physician of the institution determines that the prisoner satisfies the criteria set forth in subdivision (a), he or she shall refer the matter to the Board of Parole Hearings using a standardized form and format developed by the department. If the head physician of the institution does not concur in the recommendation, he or she shall provide the prisoner or his or her family member or designee with a written explanation of the reasons for denying the application. The Department of Corrections and Rehabilitation shall complete parole plans for inmates referred to the Board of Parole Hearings for medical parole consideration. The parole plans shall include, but not be limited to, the inmates plan for residency and medical care. Notwithstanding any other law, medical parole hearings shall be conducted by two-person panels consisting of at least one commissioner. In the event of a tie vote, the matter shall be referred to the full board for a decision. Medical parole hearings may be heard in absentia. Upon receiving a recommendation from the head physician of the institution where a prisoner is located for the prisoner to be granted medical parole pursuant to subdivision (c) or (d), the board, as specified in subdivision (f), shall make an independent judgment regarding whether the conditions under which the inmate would be released pose a reasonable threat to public safety, and make written findings related thereto. Notwithstanding any other law, the board or the Division of Adult Parole Operations shall have the authority to impose any reasonable conditions on prisoners subject to medical parole supervision pursuant to subdivision (a), including, but not limited to, the requirement that the parolee submit to electronic monitoring. As a further condition of medical parole, pursuant to subdivision (a), the parolee may be required to submit to an examination by a physician selected by the board for the purpose of diagnosing the parolees current medical condition. In the event such an examination takes place, a report of the examination and diagnosis shall be submitted to the board by the examining physician. If the board determines, based on that medical examination, that the persons medical condition has improved to the extent that the person no longer qualifies for medical parole, the board shall return the person to the custody of the department. Notwithstanding any other law establishing maximum periods for parole, a prisoner sentenced to a determinate term who is placed on medical parole supervision prior to the earliest possible release date and who remains eligible for medical parole, shall remain on medical parole, pursuant to subdivision (a), until that earliest possible release date, at which time the parolee shall commence serving that period of parole provided by, and under the provisions of, Chapter 8 of Title 1. Notwithstanding any other law establishing maximum periods for parole, a prisoner sentenced to an indeterminate term who is placed on medical parole supervision prior to the prisoners minimum eligible parole date, and who remains eligible for medical parole, shall remain on medical parole pursuant to subdivision (a) until that minimum eligible parole date, at which time the parolee shall be eligible for parole consideration under all other provisions of Chapter 8 of Title 1. The Department of Corrections and Rehabilitation shall, at the time a prisoner is placed on medical parole supervision pursuant to subdivision (a), ensure that the prisoner has applied for any federal entitlement programs for which the prisoner is eligible, and has in his or her possession a discharge medical summary, full medical records, parole medications, and all property belonging to the prisoner that was under the control of the department. Any additional records shall be sent to the prisoners forwarding address after release to health care-related parole supervision. The provisions for medical parole set forth in this title shall not affect an inmates eligibility for any other form of parole or release provided by law. (1) Notwithstanding any other law, the Department of Corrections and Rehabilitation shall give notice to the county of commitment and the proposed county of release, if that county is different than the county of commitment, of any medical parole hearing as described in subdivision (f), and of any medical parole release as described in subdivision (g). Notice shall be made at least 30 days, or as soon as feasible, prior to the time any medical parole hearing or medical parole release is scheduled for an inmate receiving medical parole consideration, regardless of whether the inmate is sentenced either determinately or indeterminately."
model-index:
- name: d0r1h/LEDBill
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: billsum
      type: billsum
      config: default
      split: test
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 38.6502
      verified: true
    - name: ROUGE-2
      type: rouge
      value: 18.5458
      verified: true
    - name: ROUGE-L
      type: rouge
      value: 25.6561
      verified: true
    - name: ROUGE-LSUM
      type: rouge
      value: 33.1575
      verified: true
    - name: loss
      type: loss
      value: 2.1305277347564697
      verified: true
    - name: gen_len
      type: gen_len
      value: 288.372
      verified: true
---

# Longformer Encoder-Decoder (LED) fine-tuned on Billsum



This model is a fine-tuned version of [led-base-16384](https://huggingface.co/allenai/led-base-16384) on the [billsum](https://huggingface.co/datasets/billsum) dataset.

As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from [*bart-base*](https://huggingface.co/facebook/bart-base) since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.


## How to use

```Python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("d0r1h/LEDBill")
model = AutoModelForSeq2SeqLM.from_pretrained("d0r1h/LEDBill", return_dict_in_generate=True).to(device)

case = "......."

input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
global_attention_mask = torch.zeros_like(input_ids)
global_attention_mask[:, 0] = 1

sequences = model.generate(input_ids, 
                           global_attention_mask=global_attention_mask).sequences
summary = tokenizer.batch_decode(sequences, 
                                 skip_special_tokens=True)
                                 
```


## Evaluation results

When the model is used for summarizing Billsum documents(10 sample), it achieves the following results:

| Model |            rouge1-f  |  rouge1-p  |  rouge2-f   |  rouge2-p  | rougeL-f   |  rougeL-p  |
|:-----------:|:-----:|:-----:|:------:|:-----:|:------:|:-----:|
| LEDBill           | **34** |  **37** | **15**  | **16** | **30**  | **32** | 
| led-base          | 2     |  15     | 0      | 0     | 2      | 15 |

[This notebook](https://colab.research.google.com/drive/1iEEFbWeTGUSDesmxHIU2QDsPQM85Ka1K?usp=sharing) shows how *led* can effectively be used for downstream task such summarization.

