Super cool paper (rStar-Math) from MSFT on improving reasoning in SLMs to rival o1 on math benchmarks. The authors use a "self-evolution" approach that iteratively fine-tunes on self-generated reasoning traces via MCTS and reward models.

Big takeaway for me is the importance of the process preference model (see chart). The authors found that collecting explicit process scores is difficult and MCTS-generated Q-values are too noisy for direct training. Instead, they cleverly use these Q-values as preference signals to train a process preference model!

My thought - this suggests process preferences can be directly annotated to guide MCTS-driven reasoning. While this paper used Python execution to validate steps and narrow the search space, the preference-based approach could generalize well to domains like textual reasoning where such immediate validation isn't possible.