|
--- |
|
title: SmoLLMv2 |
|
emoji: 🐢 |
|
colorFrom: indigo |
|
colorTo: blue |
|
sdk: gradio |
|
sdk_version: 5.13.1 |
|
app_file: app.py |
|
pinned: false |
|
license: mit |
|
short_description: Text generation using smollmv2-135M model |
|
--- |
|
|
|
# SmoLLMv2: A Small but Efficient Language Model |
|
|
|
[Training Repo Link](https://github.com/Shilpaj1994/SmoLLMv2) |
|
[Gradio App Link](https://huggingface.co/spaces/Shilpaj/SmoLLMv2) |
|
|
|
|
|
SmoLLMv2 is a 135M parameter language model designed for efficient text generation. It incorporates several modern architectural improvements while maintaining a small footprint. |
|
|
|
|
|
|
|
## Features |
|
|
|
- **Efficient Architecture**: |
|
- 30 transformer layers |
|
- 9 attention heads |
|
- 576 embedding dimension |
|
- Memory-efficient attention with reduced KV dimensions |
|
- Rotary Position Embeddings (RoPE) |
|
- SwiGLU activation function |
|
|
|
- **Training Optimizations**: |
|
- Mixed precision training (16-bit) |
|
- Gradient accumulation |
|
- OneCycleLR scheduler |
|
- Streaming dataset support |
|
- Automatic model compilation (with PyTorch 2.0+) |
|
|
|
|
|
|
|
## Model Architecture |
|
|
|
SmoLLMv2 incorporates several efficiency improvements: |
|
|
|
1. **Reduced KV Dimensions**: Uses 189-dimensional key/value projections (instead of full 576) to save memory and computation. |
|
2. **RoPE Attention**: Implements Rotary Position Embeddings for better handling of sequential information. |
|
3. **SwiGLU Activation**: Uses the SwiGLU activation function in the MLP layers for better performance. |
|
4. **Weight Sharing**: Shares weights between input embeddings and output projection. |
|
|
|
|
|
|
|
## Configuration |
|
|
|
The model's behavior can be customized through various configuration classes in `config.py`: |
|
|
|
- `SmollmConfig`: Core model architecture and training parameters |
|
- `RoPEConfig`: Rotary Position Embedding settings |
|
- `OptimizerConfig`: Optimization and learning rate settings |
|
- `DataConfig`: Dataset and tokenizer configuration |
|
- `TrainerConfig`: Training infrastructure settings |
|
|
|
|
|
|
|
## Dataset |
|
|
|
The model is trained on the Cosmopedia dataset, which is streamed during training to handle large-scale data efficiently. |
|
|
|
|
|
|
|
## Requirements |
|
|
|
See `requirements.txt` for full dependencies. Key requirements: |
|
|
|
- PyTorch ≥ 2.0.0 |
|
- Transformers ≥ 4.30.0 |
|
- Lightning ≥ 2.0.0 |
|
- Gradio ≥ 5.13.1 |
|
|
|
|
