autogen/website/blog/2023-04-21-LLM-tuning-math/index.md
Experiment Results  The first figure in this blog post shows the average accuracy and average inference cost of each configuration on the level 2 Algebra test set.  Surprisingly, the tuned gpt-3.5-turbo model is selected as a better model and it vastly outperforms untuned gpt-4 in accuracy (92% vs. 70%) with equal or 2.5 times higher inference budget. The same observation can be obtained on the level 3 Algebra test set.  ![level 3 algebra](img/level3algebra.png)  However, the selected model changes on level 4 Algebra.  ![level 4 algebra](img/level4algebra.png)  This time gpt-4 is selected as the best model. The tuned gpt-4 achieves much higher accuracy (56% vs. 44%) and lower cost than the untuned gpt-4. On level 5 the result is similar.  ![level 5 algebra](img/level5algebra.png)  We can see that AutoGen has found different optimal model and inference parameters for each subset of a particular level, which shows that these parameters matter in cost-sensitive LLM applications and need to be carefully tuned or adapted.  An example notebook to run these experiments can be found at: https://github.com/microsoft/FLAML/blob/v1.2.1/notebook/autogen_chatgpt.ipynb. The experiments were run when AutoGen was a subpackage in FLAML. 