autogen/samples/tools/autogenbench/scenarios/HumanEval/README.md
# HumanEval Benchmark  This scenario implements a modified version of the [HumanEval](https://arxiv.org/abs/2107.03374) benchmark. Compared to the original benchmark, there are **two key differences** here:  - A chat model rather than a completion model is used. - The agents get pass/fail feedback about their implementations, and can keep trying until they succeed or run out of tokens or turns. 