Spaces:
Sleeping
Sleeping
File size: 1,481 Bytes
840b176 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
# Shakespeare GPT
A GPT-2 model fine-tuned on Shakespeare's works, capable of generating Shakespeare-style text.
## Project Overview
This project implements a GPT-2 architecture trained on Shakespeare's works to generate Shakespeare-style text. The model uses a context window of 1024 tokens and implements various optimizations including gradient accumulation and learning rate scheduling.
## Model Architecture
- Base Architecture: GPT-2 (124M parameters)
- Layers: 12
- Heads: 12
- Embedding Dimension: 768
- Context Length: 1024 tokens
- Total Parameters: ~124M
## Training Details
- Dataset: Shakespeare's complete works
- Training Device: GPU/MPS (Apple Silicon)
- Batch Size: 16 (Effective batch size: 64 with gradient accumulation)
- Learning Rate: 6e-4 with cosine decay
- Weight Decay: 0.1
- Training Steps: 10,000
## Performance
- Best Validation Loss: [Insert your best validation loss]
- Training Time: [Insert your training time]
## Requirements
- bash
- pip install -r requirements.txt
## Project Structure
βββ src/
β βββ train_shakespeare.py # Training script
β βββ app.py # Gradio interface
β βββ input.txt # Training data
βββ requirements.txt
βββ README.md
## Usage
### Training
To train the model:
bash
python src/train_shakespeare.py
### Inference
- To run the Gradio interface locally:
- bash
- python src/app.py
bash
python src/app.py
|