I spent the weekend reading some recent great math+reasoning papers:
1. AceMath (2. rStar-Math (3. PRIME (Here are some of my naive thoughts! It could be wrong.

All of these papers are showing possible ways to reach o1. The secret source is pretty much the same thing: 
**high-quality/difficult prompt with verifiable answer**

1. AceMath takes a simple approach (rejection-fine-tuning -> RFT) to scale up all the SFT dataset to massive size based on the verifiable answer matching. No RM is necessary, but you can still use outcome RM to help boost the performance.
2. rStarMath uses self-evolving SFT approach to gradually boost the data quality and process preference model (PPM) performance. rStarMath is still an RFT, where the samples are coming from MCTS guided with PPM. Still, it requires strong supervision from the verifiable reward in the end. rStarMath also scales up inference compute by utilizing the PPM at each step.
3. PRIME takes a very different angle! PRIME actually uses PPO to train the model, but the major contribution is on how to assign the outcome's reward to each intermediate steps. It also relies heavily on using the verifiable answer to obtain the "correct" on-policy model outputs. 

The results are quite interesting. It seems that all these approaches are reaching similar results. Eurus-2 might seem weaker due to its smaller training set size. These results are all somewhat on par with o1-mini already/ Given some leakage that o1-mini is ~20B, it basically says that on there is no gap with o1 at least on math problems now. However, o1-mini might win significantly in other broader reasoning tasks, like physics, puzzles, etc. 

These results might reveal that reaching o1 is more of a data or infra problem than an algorithm problem. As we find great ways to scale up the (good and difficult prompt, verifiable answer) pairs from different domains, the actual algorithm might not influence too much. Some algorithms are more data efficient than the others, but many of them will take us to o1 or even o3.