When training a simple LORA on a homogenous dataset, the dynamics can truly be beautiful. Nearly perfect logarithmic monotonic descent.

(This is training Llama 3B to generate synthetic PHISH concert setlists and associated metadata). 

What's remarkable is how this adaptation can occur so quickly - the adapters only have 2M parameters, target only the Q and V matrices of each layer, run 300+tok/sec on M3 Max, and work beautifully. The model learns the format, the context, and creates meaningful synthetic data.

Endless intelligence most beautiful.