README.md

---
library_name: transformers
license: mit
datasets:
- openslr/librispeech_asr
- slprl/SpokenSwag
- slprl/sTinyStories
base_model:
- Qwen/Qwen2.5-0.5B
pipeline_tag: audio-to-audio
---

# Model Card for SLAM

This is a Speech Language Model trained for generating speech continuations over discrete [Hubert tokens](https://huggingface.co/slprl/mhubert-base-25hz).


## Model Details

### Model Description
This is a Speech Language Model, introduced in "[_Slamming_: Training a Speech Language Model on One GPU in a Day](https://arxiv.org/abs/2502.15814)", focusing on efficient training. 
It was fine-tuned from [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) over a vocabulary of 500 speech tokens extracted from 
the 11-th layer of [mhubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz). For a stronger version of the model trained with 
slightly more compute - 2*A100 for 2 days, see [slam_scaled](https://huggingface.co/slprl/slam_scaled).

The model was trained by next-token prediction over a subset of LibriSpeech, Libri-Light and a synthetic data 
[sTinyStories](https://huggingface.co/datasets/slprl/sTinyStories). It was then trained with DPO over 
[SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).

- **Developed by:** [SLP-RL](https://huggingface.co/slprl)
- **Model type:** SpeechLM
- **License:** MIT
- **Finetuned from model:** [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)

### Model Sources

- **Repository:** [https://github.com/slp-rl/slamkit](https://github.com/slp-rl/slamkit)
- **Paper:** [https://arxiv.org/abs/2502.15814](https://arxiv.org/abs/2502.15814)
- **Demo** [https://pages.cs.huji.ac.il/adiyoss-lab/slamming/](https://pages.cs.huji.ac.il/adiyoss-lab/slamming/)

## Uses
This is a base SpeechLM and as such can be used to generate continuations for speech segments, or as base for further tuning. See the _SlamKit_
[codebase](https://github.com/slp-rl/slamkit) for more details on usage, and checkout the [demo page](https://pages.cs.huji.ac.il/adiyoss-lab/slamming/) for some generation examples

### Out-of-Scope Use
This model was trained on curated speech datasets which contain mainly audio-books and stories, as such the outputs should not be treated as factual in any way.



## How to Get Started with the Model
We refer users to the official repository for full usage explanations - [github](https://github.com/slp-rl/slamkit).


## Training Details
We highly encourage users to read the full [paper](https://arxiv.org/abs/2502.15814), for full training details, a brief overview is provided below.


### Training Data
This model was trained on a subset of [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) train, 
[Libri-Light](https://ai.meta.com/tools/libri-light/) and the synthetic dataset
[sTinyStories](https://huggingface.co/datasets/slprl/sTinyStories) for the pre-training phase. It was also trained with DPO on the synthetic
dataset [SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).

### Training Procedure
This model was trained by next token prediction over several datasets, and then trained with DPO over [SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).
Please refer to the [paper]() or [code](https://github.com/slp-rl/slamkit) for the full training recipes.

#### Preprocessing
Speech tokens are extracted from the audio using [Hubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz), and quantised using the 
official kmeans released with the model in [textlesslib](https://github.com/facebookresearch/textlesslib/tree/main). Units are de-duplicated.
We encourage you to explore the official repository for full details - [github](https://github.com/slp-rl/slamkit).


## Evaluation
The paper provides full results, we do give here some results and also refer to the [demo page]() to listen to some samples.
| Model                                    | Compute (GPU days) | Parameters | sBLIMP ↑ | sStoryCloze ↑ | tStoryCloze ↑ | GenPPL ↓ | Auto-BLEU ↓ |
|------------------------------------------|--------------------|------------|----------|--------------|--------------|---------|------------|
| [TWIST-1.3B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                      | 160xV100          | 1B         | 57.00    | 52.4         | 70.6         | 131.8   | 3.20       |
| [TWIST-7B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                        | ?                 | 7B         | 59.00    | 55.3         | 74.1         | 93.7    | 3.06       |
| [TWIST-13B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                       | ?                 | 13B        | 59.20    | 55.4         | 76.4         | -       | -          |
| [Scaled Optimal](https://arxiv.org/abs/2404.00685)      | ?                 | 823M       | **61.3** | **56.7**     | **78.0**     | -       | -          |
| [Predicted Optimal]((https://arxiv.org/abs/2404.00685))   | 1xA5000           | 78M        | 56.85    | 54.09        | 70.49        | -       | -          |
| TWIST-350M (Original recipe)            | 1xA5000           | 305M       | 51.52 ± .19 | 53.65 ± .57 | 68.80 ± .47 | 259.2 ± 6.7 | 3.26 ± .46 |
| *Slam (-DPO) (ours)*                    | 1xA5000           | 358M       | *56.45* ± .17 | *55.59* ± .30 | *78.01* ± .27 | *88.3* ± 1.0 | 3.47 ± .17 |
| **Slam (ours)**                       | 1xA5000           | 358M       | **58.86** ± .20 | **58.04** ± .51 | **82.04** ± .21 | **62.8** ± 4.1 | 3.88 ± .11 |



### Compute Infrastructure
This model was trained as part of ["*Slamming*: Training a Speech Language Model on One GPU in a Day"](https://arxiv.org/abs/2502.15814), focusing on efficient training.

#### Hardware
This model was trained using **only a single Nvidia A5000 GPU**, 16 CPU cores and 24 GB of RAM for **24 hours**.

#### Software
The model wastrained using the [*SlamKit*](https://github.com/slp-rl/slamkit) codebase which builds upon 🤗transformers extending it to support
easy and efficient training of Speech Language Models.

## Citation

**BibTeX:**
```
@misc{maimon2025slamming,
      title={Slamming: Training a Speech Language Model on One GPU in a Day}, 
      author={Gallil Maimon and Avishai Elmakies and Yossi Adi},
      year={2025},
      eprint={2502.15814},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2502.15814}, 
}
```