--- library_name: transformers base_model: ai-forever/rugpt3small_based_on_gpt2 tags: - generated_from_trainer metrics: - accuracy model-index: - name: aristototle_interface results: [] --- # aristototle_interface This model is a fine-tuned version of [ai-forever/rugpt3small_based_on_gpt2](https://huggingface.co/ai-forever/rugpt3small_based_on_gpt2) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 3.0259 - Accuracy: 0.4040 Model Description This model is a fine-tuned version of ai-forever/rugpt3small_based_on_gpt2, designed for causal language modeling tasks. It has been trained on a custom dataset to generate coherent and contextually relevant text. Training Details Training Epochs: 29.86 Total FLOPs: 8,153,103 GF Training Loss: 3.8147 Training Runtime: 35 minutes and 43.75 seconds Number of Training Samples: 291 Training Samples per Second: 4.072 Training Steps per Second: 0.056 Evaluation Metrics Evaluation Epoch: 29.86 Evaluation Accuracy: 40.4% Evaluation Loss: 3.0259 Evaluation Runtime: 0.12 seconds Number of Evaluation Samples: 1 Evaluation Samples per Second: 8.08 Evaluation Steps per Second: 8.08 Perplexity: 20.6125 Intended Use This model is intended for text generation tasks where coherent and contextually appropriate responses are required. It can be used in applications such as chatbots, content creation, and more. Limitations The model has been trained on a limited dataset (291 samples), which may affect its generalization capabilities. The evaluation accuracy of approximately 40% indicates that the model may not perform optimally across all contexts. The perplexity score suggests room for improvement in generating more confident predictions. Future Work To enhance the performance of this model, consider the following: Increase the size and diversity of the training dataset. Experiment with additional training epochs or different hyperparameters. Evaluate the model on a broader set of examples to better assess its capabilities. ## Training procedure ## [Training Procedure](pplx://action/followup) The model was trained using the `transformers` library and the `run_clm.py` script. Here's a summary of the training process: * **[Model](pplx://action/followup):** `ai-forever/rugpt3small_based_on_gpt2` (a Russian language GPT-2 model). * **[Objective](pplx://action/followup):** Causal Language Modeling (text generation). * **[Hardware](pplx://action/followup):** Google Colab with a single CUDA-enabled GPU. * **[Mixed Precision](pplx://action/followup):** FP16 training was enabled to reduce memory footprint and potentially improve training speed. * **[Optimizer](pplx://action/followup):** AdamW (`adamw_torch`) was used as the optimizer. * **[Learning Rate](pplx://action/followup):** The learning rate was set to `3e-5`. * **[Warmup](pplx://action/followup):** A linear warmup schedule with `500` warmup steps was used. * **[Training Data](pplx://action/followup):** Custom text dataset loaded from ` The model was trained on a custom text dataset loaded from the following sources using the `plain_text` dataset configuration: * **Training set:** Aristotle's major works. (32,835 examples) * Аристотель. Категории * Аристотель. Никомахова этика * Аристотель. Физика * Аристотель. Метафизика * Аристотель. Риторика * Аристотель. Поэтика * **[Validation Data](pplx://action/followup):** Custom text dataset loaded from `- Аристотель. Никомахова этика ttps://lib.ru/POEEAST/ARISTOTEL/nikomah.txt` using the `plain_text` dataset configuration. The validation set contained 111 examples. * **Validation set:** Aristotle. Никомахова этика (111 examples) * * **[Batch Size](pplx://action/followup):** A per-device batch size of `8` was used with a gradient accumulation size of `8`, resulting in an effective batch size of 64. * **[Sequence Length](pplx://action/followup):** The maximum sequence length (block size) was set to `2048`. * **[Gradient Checkpointing](pplx://action/followup):** Enabled to reduce memory consumption. * **[Epochs](pplx://action/followup):** Trained for `30` epochs. The training data was passed over 30 times. * **[Evaluation](pplx://action/followup):** Evaluation was performed every `1000` steps using the validation dataset. * **[Logging](pplx://action/followup):** Training progress and metrics were logged every `100` steps to TensorBoard and Weights & Biases (WandB). * **[Checkpoints](pplx://action/followup):** Model checkpoints were saved every `1000` steps, with a limit of `3` saved checkpoints. ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - gradient_accumulation_steps: 8 - total_train_batch_size: 64 - optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 500 - num_epochs: 30.0 - mixed_precision_training: Native AMP ### Training results ### Framework versions - Transformers 4.49.0.dev0 - Pytorch 2.5.1+cu124 - Datasets 3.3.0 - Tokenizers 0.21.0