autogen/samples/tools/autogenbench/CONTRIBUTING.md
Integrating with the `tabulate` and `clone` commands.  The above details are sufficient for defining and running tasks, but if you wish to support the `autogenbench tabulate` and `autogenbench clone` commands, a few additional steps are required.  ### Tabulations  If you wish to leverage the default tabulation logic, it is as simple as arranging your `scenario.py` file to output the string "ALL TESTS PASSED !#!#" to the console in the event that a task was solved correctly.  If you wish to implement your own tabulation logic, simply create the file `Scripts/custom_tabulate.py` and include a `main(args)` method. Here, the `args` parameter will be provided by AutoGenBench, and is a drop-in replacement for `sys.argv`. In particular, `args[0]` will be the invocation command (similar to the executable or script name in `sys.argv`), and the remaining values (`args[1:]`) are the command line parameters.  Should you provide a custom tabulation script, please implement `--help` and `-h` options for documenting your interface.  The `scenarios/GAIA/Scripts/custom_tabulate.py` is a great example of custom tabulation. It also shows how you can reuse some components of the default tabulator to speed up development.   ### Cloning  If you wish your benchmark to be available via the `autogenbench clone` command, you will need to take three additional steps:  #### Manifest First, provide a `MANIFEST.json` file in the root of your benchmark. An example is provided below, from which you can see the schema:  ```json {     "files": {         "Templates/TwoAgents/prompt.txt": "Templates/TwoAgents/prompt.txt",         "Templates/TwoAgents/coding/my_tests.py": "Templates/TwoAgents/coding/my_tests.py",         "Templates/TwoAgents/scenario.py": "Templates/TwoAgents/scenario.py",         "README.md": "README.md", 	"Scripts/init_tasks.py": "Scripts/init_tasks.py", 	"Scripts/custom_tabulate.py": "Scripts/custom_tabulate.py"     } } ```  The keys of the `files` dictionary are local paths, relative to your benchmark's root directory. The values are relative paths in the AutoGen GitHub repository (relative to the folder where the MANIFEST.json file is located). In most cases, the keys and values will be identical.  #### SCENARIOS dictionary Second, you must add an entry to the `scenarios` dictionary in `autogen/samples/tools/autogenbench/scenarios/MANIFEST.json`.  #### Scripts/init_tasks.py Finally, you should provide an `Scripts/init_tasks.py` file, in your benchmark folder, and include a `main()` method therein. This method will be loaded and called automatically by `autogenbench clone` after all manifest files have been downloaded.  This `init_tasks.py` script is a great place to download benchmarks from their original sources and convert them to the JSONL format required by AutoGenBench: - See `HumanEval/Scripts/init_tasks.py` for an example of how to expand a benchmark from an original GitHub repository. - See `GAIA/Scripts/init_tasks.py` for an example of how to expand a benchmark from `Hugging Face Hub`. - See `MATH/SCripts/init_tasks.py` for an example of how to expand a benchmark from an author-hosted website. 