pico-decoder-tiny-1 trained to 125k steps

README.md

@@ -0,0 +1,57 @@
+---
+datasets:
+- pico-lm/pretokenized-dolma
+language:
+- en
+license: apache-2.0
+metrics:
+- pico-lm/perplexity
+pipeline_tag: text-generation
+---
+
+# Pico Decoder Tiny
+
+**pico-decoder-tiny** is the smallest (11M) model in the `pico-decoder` suite — a lightweight, LLaMA-style decoder-only transformer trained from scratch using [`pico-train`](https://github.com/pico-lm/pico-train). It is designed for transparent and reproducible research into the learning dynamics of language models, and is fully compatible with the `pico-analyze` toolkit for detailed interpretability analysis.
+
+> NOTE: The `pico-decoder-tiny-1` branch contains the full commit history for the training run. 
+
+## 🔧 Model Details
+
+| Field               | Value                              |
+|---------------------|------------------------------------|
+| **Architecture**     | Decoder-only transformer (LLaMA-style) |
+| **Parameters**       | 11M                               |
+| **Layers**           | 12                                |
+| **Hidden Size**      | 96                               |
+| **Feed Foward Size** | 384                                |
+| **Attention Heads**  | 12                                 |
+| **Key/Value Heads**  | 4                                  |
+
+## 📚 Training
+
+- **Dataset**: [`pretokenized-dolma`](https://huggingface.co/datasets/pico-lm/pretokenized-dolma), English-only
+- **Training steps**: 200,000
+- **Batch size**: 1024
+- **Sequence length**: 2048
+- **Optimizer**: AdamW
+- **Learning rate schedule**: Linear decay with warmup
+- **Compute**: 16 A100-SXM4-80GB GPUs
+
+## 📈 Evaluation and Analysis
+
+This model supports fine-grained analysis using [`pico-analyze`](https://github.com/pico-lm/pico-analyze). This tool enables researchers to understand how learning unfolds over training, even at very small scales.
+
+We also evaluate perplexity of the model on the [`pico-paloma-tinsy`](https://huggingface.co/datasets/pico-lm/pretokenized-paloma-tinsy) dataset.
+
+## 📄 Citation
+
+If you use `pico-tiny` or any other `pico-decoder` model in your research, please cite:
+
+```bibtex
+@software{pico2025,
+    author = {Diehl Martinez, Richard},
+    title = {Pico: A Lightweight Framework for Studying Language Model Learning Dynamics},
+    year = {2025,
+    url = {https://github.com/pico-lm}
+}
+```

config.json

@@ -0,0 +1,22 @@
+{
+  "activation_hidden_dim": 384,
+  "architectures": [
+    "PicoDecoderHF"
+  ],
+  "attention_n_heads": 12,
+  "attention_n_kv_heads": 4,
+  "auto_map": {
+    "AutoConfig": "pico_decoder.PicoDecoderHFConfig",
+    "AutoModelForCausalLM": "pico_decoder.PicoDecoderHF"
+  },
+  "batch_size": 1024,
+  "d_model": 96,
+  "max_seq_len": 2048,
+  "model_type": "pico_decoder",
+  "n_layers": 12,
+  "norm_eps": 1e-06,
+  "position_emb_theta": 10000.0,
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.3",
+  "vocab_size": 50304
+}

eval_results/step_0.json

@@ -0,0 +1 @@
+{"paloma": 59435.05139917247}

eval_results/step_1000.json

@@ -0,0 +1 @@
+{"paloma": 2176.422658291594}

eval_results/step_10000.json

@@ -0,0 +1 @@
+{"paloma": 313.8343901737226}

eval_results/step_100000.json

@@ -0,0 +1 @@
+{"paloma": 142.78638106821307}

eval_results/step_101000.json

@@ -0,0 +1 @@
+{"paloma": 141.94935619208042}

eval_results/step_102000.json

@@ -0,0 +1 @@
+{"paloma": 141.70286827486152}

eval_results/step_103000.json

@@ -0,0 +1 @@
+{"paloma": 142.04302229000717}

eval_results/step_104000.json

@@ -0,0 +1 @@
+{"paloma": 141.28717064840868}

eval_results/step_105000.json

@@ -0,0 +1 @@
+{"paloma": 140.51475878293505}

eval_results/step_106000.json

@@ -0,0 +1 @@
+{"paloma": 140.16921514750356}

eval_results/step_107000.json

@@ -0,0 +1 @@
+{"paloma": 140.26420981211115}

eval_results/step_108000.json

@@ -0,0 +1 @@
+{"paloma": 140.04452455683452}

eval_results/step_109000.json

@@ -0,0 +1 @@
+{"paloma": 139.86317522019044}

eval_results/step_11000.json

@@ -0,0 +1 @@
+{"paloma": 268.332413412885}

eval_results/step_110000.json

@@ -0,0 +1 @@
+{"paloma": 139.03501056627945}

eval_results/step_111000.json

@@ -0,0 +1 @@
+{"paloma": 138.82995247192915}

eval_results/step_112000.json

@@ -0,0 +1 @@
+{"paloma": 139.3511911510175}

eval_results/step_113000.json

@@ -0,0 +1 @@
+{"paloma": 138.58265911295024}

eval_results/step_114000.json

@@ -0,0 +1 @@
+{"paloma": 137.69777231083515}

eval_results/step_115000.json

@@ -0,0 +1 @@
+{"paloma": 138.02265500158384}

eval_results/step_116000.json

@@ -0,0 +1 @@
+{"paloma": 137.61472352954985}

eval_results/step_117000.json

@@ -0,0 +1 @@
+{"paloma": 137.60625675962362}

eval_results/step_118000.json

@@ -0,0 +1 @@
+{"paloma": 138.34565367748513}

eval_results/step_119000.json

@@ -0,0 +1 @@
+{"paloma": 136.82007657393345}

eval_results/step_12000.json

@@ -0,0 +1 @@
+{"paloma": 254.46162965488767}

eval_results/step_120000.json

@@ -0,0 +1 @@
+{"paloma": 136.73713672285712}

eval_results/step_121000.json

@@ -0,0 +1 @@
+{"paloma": 136.92190282984478}

eval_results/step_122000.json

@@ -0,0 +1 @@
+{"paloma": 136.2517988980855}

eval_results/step_123000.json

@@ -0,0 +1 @@
+{"paloma": 137.01620536614794}

eval_results/step_124000.json

@@ -0,0 +1 @@
+{"paloma": 136.65228910047418}

eval_results/step_125000.json

@@ -0,0 +1 @@
+{"paloma": 136.16973869742418}

eval_results/step_13000.json

@@ -0,0 +1 @@
+{"paloma": 249.49274518747362}

eval_results/step_14000.json

@@ -0,0 +1 @@
+{"paloma": 260.20426084006704}

eval_results/step_15000.json

@@ -0,0 +1 @@
+{"paloma": 239.41960436525244}

eval_results/step_16000.json

@@ -0,0 +1 @@
+{"paloma": 229.52608692687562}

eval_results/step_17000.json

@@ -0,0 +1 @@
+{"paloma": 225.62338353731906}

eval_results/step_18000.json

@@ -0,0 +1 @@
+{"paloma": 212.63083450470236}

eval_results/step_19000.json

@@ -0,0 +1 @@
+{"paloma": 208.58393890899234}

eval_results/step_2000.json

@@ -0,0 +1 @@
+{"paloma": 884.8345587062504}

eval_results/step_20000.json

@@ -0,0 +1 @@
+{"paloma": 205.5417249480191}

eval_results/step_21000.json

@@ -0,0 +1 @@
+{"paloma": 201.91932611332538}

eval_results/step_22000.json

@@ -0,0 +1 @@
+{"paloma": 199.08593051392026}

eval_results/step_23000.json

@@ -0,0 +1 @@
+{"paloma": 198.37421456945066}

eval_results/step_24000.json

@@ -0,0 +1 @@
+{"paloma": 193.6218826051373}

eval_results/step_25000.json

@@ -0,0 +1 @@
+{"paloma": 192.3463352126942}

eval_results/step_26000.json

@@ -0,0 +1 @@
+{"paloma": 190.8057739201323}

eval_results/step_27000.json

@@ -0,0 +1 @@
+{"paloma": 192.40458190831572}

eval_results/step_28000.json

@@ -0,0 +1 @@
+{"paloma": 188.59788890599373}