Model Card for Charlenator-T5-IELTS-Grammar

This fine-tune is specifically tailored to correcting and evaluating grammar and vocabulary in the context of IELTS interviews.

Model Details

Model Description

This model was trained on a synthetic dataset of IELTS test answers. While there are numerous high-quality grammar correction tools and vocabulary datasets available, most of them are optimized for contexts like business writing or academic essays, which don’t fully capture the unique requirements of this use case. My goal is to create a fine-tune that bridges this gap—delivering a model specifically tailored to correcting and evaluating spoken grammar and vocabulary in the context of IELTS interviews. This specialized approach ensures that the model not only provides relevant corrections but also maintains the conversational tone and meaning essential for authentic IELTS responses.

Direct Use

The fine-tune relies on a specific prompt structure that was present across the entire training dataset:

"Correct any grammatical errors in this paragraph. Respond only with the corrected sentence or with the original sentence if no errors were found. Original paragraph: "

Out-of-Scope Use

The fine-tune should not be used in a conversational context, but rather for one-off evaluations.

Bias, Risks, and Limitations

The training dataset was injected with a limited set of error types, which leads to potential blind spots.

• SUBJECT_VERB_AGREEMENT_ERROR
• ARTICLE_ERROR
• PREPOSITION_ERROR
• TENSE_ERROR

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

Evaluation

Performance was measured using Errant.

Overall Precision: 0.8807 Overall Recall: 0.6454 Overall F0.5 Score: 0.8209

================================================== Error Type: R:VERB:SVA Precision: 0.9713 Recall: 0.9185 F0.5 Score: 0.9602

Error Type: R:MORPH Precision: 0.9083 Recall: 0.7388 F0.5 Score: 0.8684

Error Type: R:ORTH Precision: 0.9211 Recall: 0.8805 F0.5 Score: 0.9126

Error Type: R:NOUN:NUM Precision: 0.9312 Recall: 0.7492 F0.5 Score: 0.8880

Error Type: U:VERB Precision: 0.7857 Recall: 0.5789 F0.5 Score: 0.7333

Error Type: R:OTHER Precision: 0.7169 Recall: 0.3900 F0.5 Score: 0.6140

Error Type: R:SPELL Precision: 0.9302 Recall: 0.7240 F0.5 Score: 0.8801

Error Type: R:NOUN Precision: 0.8122 Recall: 0.4336 F0.5 Score: 0.6914

Error Type: R:VERB:FORM Precision: 0.9429 Recall: 0.8105 F0.5 Score: 0.9130

Error Type: R:VERB:TENSE Precision: 0.8879 Recall: 0.5655 F0.5 Score: 0.7970

Error Type: R:DET Precision: 0.7333 Recall: 0.3667 F0.5 Score: 0.6111

Error Type: R:ADJ Precision: 0.6154 Recall: 0.3721 F0.5 Score: 0.5442

Error Type: M:OTHER Precision: 0.4000 Recall: 0.0444 F0.5 Score: 0.1538

Error Type: U:CONJ Precision: 0.3333 Recall: 0.5000 F0.5 Score: 0.3571

Error Type: R:PREP Precision: 0.8000 Recall: 0.1538 F0.5 Score: 0.4348

Error Type: R:PUNCT Precision: 0.5000 Recall: 0.3750 F0.5 Score: 0.4688

Error Type: R:PRON Precision: 0.9565 Recall: 0.5946 F0.5 Score: 0.8527

Error Type: R:VERB Precision: 0.7879 Recall: 0.4262 F0.5 Score: 0.6736

Error Type: M:NOUN Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: U:OTHER Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:DET Precision: 0.2857 Recall: 0.0741 F0.5 Score: 0.1818

Error Type: U:PREP Precision: 0.7500 Recall: 0.2727 F0.5 Score: 0.5556

Error Type: U:PART Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:VERB Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: U:NOUN Precision: 0.6667 Recall: 0.5000 F0.5 Score: 0.6250

Error Type: U:VERB:TENSE Precision: 0.6000 Recall: 0.5000 F0.5 Score: 0.5769

Error Type: R:ADJ:FORM Precision: 1.0000 Recall: 0.4000 F0.5 Score: 0.7692

Error Type: U:DET Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: R:NOUN:INFL Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: R:ADV Precision: 0.7143 Recall: 0.2778 F0.5 Score: 0.5435

Error Type: M:ADJ Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:PUNCT Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: U:PUNCT Precision: 1.0000 Recall: 0.5000 F0.5 Score: 0.8333

Error Type: U:PRON Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:ADV Precision: 1.0000 Recall: 0.0714 F0.5 Score: 0.2778

Error Type: R:VERB:INFL Precision: 0.9524 Recall: 0.9524 F0.5 Score: 0.9524

Error Type: R:PART Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: R:CONJ Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: U:VERB:FORM Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:CONJ Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:PREP Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: M:PRON Precision: 0.6667 Recall: 0.2857 F0.5 Score: 0.5263

Error Type: R:WO Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: U:ADJ Precision: 1.0000 Recall: 0.5000 F0.5 Score: 0.8333

Error Type: M:VERB:FORM Precision: 0.5000 Recall: 0.4286 F0.5 Score: 0.4839

Error Type: M:PART Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000

Error Type: U:ADV Precision: 0.0000 Recall: 0.0000 F0.5 Score: 0.0000