pytorch t5 text2text-generation en dataset:asahi417/qg_squad transformers question generation question answer generation license:cc-by-4.0 autotrain_compatible
---
language: en
tags:
- question generation
- question answer generation
license: cc-by-4.0
datasets:
- asahi417/qg_squad
metrics:
- bleu
- meteor
- rouge
- bertscore
- moverscore
widget:
- text: "generate question: <hl> Beyonce <hl> further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records."
  example_title: "Question Generation Example 1"
- text: "generate question: Beyonce further expanded her acting career, starring as blues singer <hl> Etta James <hl> in the 2008 musical biopic, Cadillac Records."
  example_title: "Question Generation Example 2"
- text: "generate question: Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic,  <hl> Cadillac Records  <hl> ."
  example_title: "Question Generation Example 3"
- text: "extract answers: <hl> Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records. <hl> Her performance in the film received praise from critics, and she garnered several nominations for her portrayal of James, including a Satellite Award nomination for Best Supporting Actress, and a NAACP Image Award nomination for Outstanding Supporting Actress."
  example_title: "Answer Extraction Example 1"
- text: "extract answers: Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records. <hl> Her performance in the film received praise from critics, and she garnered several nominations for her portrayal of James, including a Satellite Award nomination for Best Supporting Actress, and a NAACP Image Award nomination for Outstanding Supporting Actress. <hl>"
  example_title: "Answer Extraction Example 2"
pipeline_tag: text2text-generation
---

# T5 SMALL fine-tuned for English Question Generation & Answer Extraction
T5 SMALL Model fine-tuned on Japanese question generation dataset (SQuAD) with an extensive hyper-parameter search.
This model is fine-tuned on question generation & answer extraction jointly.
- [Online Demo](https://autoqg.net/)
- [Project Repository](https://github.com/asahi417/lm-question-generation)


## Overview

**Language model:** t5-small   
**Language:** English (en)    
**Downstream-task:** Question Generation, Answer Extraction  
**Training data:** SQuAD  
**Eval data:** SQuAD   
**Code:**  See [our repository](https://github.com/asahi417/lm-question-generation)

## Usage
### In Transformers
```python
from transformers import pipeline

model_path = 'asahi417/lmqg-t5-small-squad-multitask'
pipe = pipeline("text2text-generation", model_path)

# Question Genration
paragraph = 'Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.'
# highlight an answer in the paragraph to generate question
answer = 'Etta James'
highlight_token = '<hl>'
input_text = paragraph.replace(answer, '{0} {1} {0}'.format(highlight_token, answer))
input_text = 'generate question: {}'.format(input_text)  # add task specific prefix
generation = pipe(input_text)
print(generation)
>>> [{'generated_text': 'What is the name of the biopic that Beyonce starred in?'}]

# Answer Extraction
paragraph = 'Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records. Her performance in the film received praise from critics, and she garnered several nominations for her portrayal of James, including a Satellite Award nomination for Best Supporting Actress, and a NAACP Image Award nomination for Outstanding Supporting Actress.'
# highlight a sentence where the answer should be extracted
sentence = 'Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.'
input_text = paragraph.replace(sentence, '{0} {1} {0}'.format(highlight_token, sentence))
input_text = 'extract answer: <hl> {} <hl>'.format(input_text)  # add task specific prefix
generation = pipe(input_text)
print(generation)
>>> [{'generated_text': 'Etta James'}]
```

## Evaluations

Evaluation on the test set of [SQuAD QG dataset](https://huggingface.co/datasets/asahi417/qg_squad).
The results are comparable with the [leaderboard](https://paperswithcode.com/sota/question-generation-on-squad11) and previous works.
All evaluations were done using our [evaluation script](https://github.com/asahi417/lm-question-generation).


| BLEU 4 | ROUGE L  | METEOR | BERTScore | MoverScore |
| ------ | -------- | ------ | --------- | ---------- |
| 24.17  | 51.11    | 25.58  | 90.17     | 63.71      |

- [metric file](https://huggingface.co/asahi417/lmqg-t5-small-squad-multitask/raw/main/eval/metric.first.sentence.paragraph_answer.question.asahi417_qg_squad.default.json)

## Fine-tuning Parameters
We ran grid search to find the best hyper-parameters and continued fine-tuning until the validation metric decrease.
The best hyper-parameters can be found [here](https://huggingface.co/asahi417/lmqg-t5-small-squad-multitask/raw/main/trainer_config.json), and fine-tuning script is released in [our  repository](https://github.com/asahi417/lm-question-generation).

## Citation
TBA


