| # FLAN-T5-Base fine-tuned on Historical Content Completion | |
| This model is a fine-tuned version of [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) on a history dataset, focused on paragraph completion tasks. | |
| ## Model description | |
| This model is designed to complete historical paragraphs, either by providing the ending or the beginning of a given text. | |
| ## Intended uses & limitations | |
| This model is intended for educational purposes and to assist in creating history-related content. It's specifically trained on completing historical paragraphs and should not be used for question generation or answering. | |
| ## Training and evaluation data | |
| The model was trained on the dataset [ambrosfitz/just_history_xl](https://huggingface.co/datasets/ambrosfitz/just_history_xl). | |
| ## Training procedure | |
| The model was trained using the following key strategies: | |
| 1. Varied Task Formulation: Randomly choosing between completing the end or beginning of paragraphs | |
| 2. Task Instruction: Including varied task instructions to prevent overfitting | |
| 3. Full Dataset Usage: Utilizing the entire dataset to maximize historical knowledge | |
| 4. Controlled Training: Using a single epoch with a low learning rate to prevent overfitting | |
| Hyperparameters: | |
| - Number of epochs: 1 | |
| - Learning rate: 0.0003 | |
| - Batch size: 4 | |
| - Gradient Accumulation Steps: 4 | |
| ## Results | |
| Test set results: {'eval_loss': 0.29231172800064087, 'eval_runtime': 119.2142, 'eval_samples_per_second': 78.9, 'eval_steps_per_second': 19.729, 'epoch': 0.9998405188453564} | |