# Test-suite Reduction

## Preperation Work

As test-suite reduction relies on the results of evaluation, make sure that you've run the evaluation script and an `eval_results.json` has been generated for each model under test.

Use the following command to install necessary dependencies:

```bash
# in $EVALPLUS_ROOT
pip install -r requirements-tsr.txt
```

## Usage

```bash
python3 run.py \
  --dataset DATASET \
  --sample_eval_dir SAMPLE_DIR \
  --model MODEL \
  [--report_dir REPORT_DIR]

# Example
python3 run.py --dataset humaneval --sample_eval_dir $HOME/HumanEval --model ALL
```

Parameter descriptions:
* `--dataset`: currently, `humaneval` and `mbpp` are supported.
* `--sample_eval_dir` is the directory containing all the LLM evaluation results. We require the directory be structured as
    ```bash
    SAMPLE_EVAL_DIR
    ├── LLM_1
    │   ├── ...
    │   └── eval_results.json
    ├── LLM_2
    │   ├── ...
    ├── ...
    ```
* `--report_dir` is the directory where we store intermediate files, pass@k results, and reduced dataset. If not specified, `REPORT_DIR=./tsr_info` by default.
* If `MODEL` is a specific LLM name, the cross-validation results will be generated in `REPORT_DIR`; if `MODEL == ALL`, a reduced dataset will be generated in `REPORT_DIR`.

## Known Issues

If you find the program stuck at the mutant generation step, try removing the line
```python
assert len(completion_id) == len(problems), "Missing problems in samples"
```
in `evalplus/evaluate.py`.