evaluation/regression/README.md

# OpenHands - Regression Test Framework

OpenHands project is an open-source software engineering AI that can solve various software engineering tasks. This repository contains the regression test framework for OpenHands project.

## Running the Tests

To run the tests for OpenHands project, you can use the provided test runner script. Follow these steps:

1. Ensure you have Python 3.6 or higher installed on your system.
2. Install the required dependencies by running the following command in your terminal:
   ```
   pip install -r requirements.txt
   ```
3. Navigate to the root directory of the project.
4. Run the test suite using the test runner script with the required arguments:
   ```
   python evaluation/regression/run_tests.py --OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxx --model=gpt-4o
   ```
   Replace `sk-xxxxxxxxxxxxxxxxxxxxxx` with your actual OpenAI API key. The default model is `gpt-4o`, but you can specify a different model if needed.

The test runner will discover and execute all the test cases in the `cases/` directory, and display the results of the test suite, including the status of each individual test case and the overall summary.

## Test Case Structure

The test cases for OpenHands project are organized in the `cases/` directory. Each test case has the following structure:

```
cases/
├── hello-world/
│   ├── task.txt
│   ├── outputs/
│   │   └── codeact_agent/
│   │       └── workspace/
│   │           ├── hello_world.sh
│   └── test_hello_world.py
├── create_web_app/
│   ├── task.txt
│   ├── outputs/
│   │   └── codeact_agent/
│   │       └── workspace/
│   │           ├── app.py
│   │           ├── requirements.txt
│   │           ├── static/
│   │           └── templates/
│   └── test_create_web_app.py
└── ...
```

- `task.txt`: This file contains the task description provided by the user.
- `outputs/`: This directory contains the output generated by OpenHands for each agent.
- `outputs/*/workspace/`: This directory contains the actual output files generated by OpenHands.
- `test_*.py`: These are the test scripts that validate the output of OpenHands.

## Adding New Test Cases

To add a new test case to the regression test framework, follow the same steps as described in the previous sections.

## Customizing the Test Cases

The test cases can be customized by modifying the fixtures defined in the `conftest.py` file. The available fixtures are:

- `test_cases_dir`: The directory containing the test cases.
- `task_file`: The path to the `task.txt` file for the current test case.
- `workspace_dir`: The path to the `workspace/` directory for the current test case.
- `model`: The model selected start the generation.
- `run_test_case`: A fixture that runs OpenHands and generates the workspace for the current test case.

You can modify these fixtures to change the behavior of the test cases or add new ones as needed.

If you have any questions or need further assistance, feel free to reach out to the project maintainers.