autogen/samples/tools/autogenbench/CONTRIBUTING.md
Implementing and Running Benchmark Tasks At the core of any benchmark is a set of tasks. To implement tasks that are runnable by AutoGenBench, you must adhere to AutoGenBench's templating and scenario expansion algorithms, as outlined below.  ### Task Definitions  All tasks are stored in JSONL files (in subdirectories under `./Tasks`). Each line of a tasks file is a JSON object with the following schema:  ``` {    "id": string,    "template": dirname,    "substitutions" {        "filename1": {        	   "find_string1_1": replace_string1_1,            "find_string1_2": replace_string1_2,            ...            "find_string1_M": replace_string1_N        }        "filename2": {        	   "find_string2_1": replace_string2_1,            "find_string2_2": replace_string2_2,            ...            "find_string2_N": replace_string2_N        }    } } ```  For example:  ``` {     "id": "two_agent_stocks_gpt4",     "template": "default_two_agents",     "substitutions": { 	"scenario.py": {             "__MODEL__": "gpt-4", 	}, 	"prompt.txt": {             "__PROMPT__": "Plot and save to disk a chart of NVDA and TESLA stock price YTD."         }     } } ```  In this example, the string `__MODEL__` will be replaced in the file `scenarios.py`, while the string `__PROMPT__` will be replaced in the `prompt.txt` file.  The `template` field can also take on a list value, but this usage is considered advanced and is not described here. See the `autogenbench/run_cmd.py` code, or the `GAIA` benchmark tasks files for additional information about this option.  