Update src/tasks_content.py
Browse files- src/tasks_content.py +2 -1
src/tasks_content.py
CHANGED
@@ -29,7 +29,8 @@ TASKS_DESCRIPTIONS = {
|
|
29 |
|
30 |
The benchmark clones the repo to the local directory, the model fixes the issue according to logs and the local repo state,
|
31 |
and then the benchmark pushes the repo to GitHub and requests the result of the GitHub CI.
|
32 |
-
We use the `Pass@1` rate metric to measure CI repair, indicating the ratio of data points, for which the build passed successfully after the generated fix.
|
|
|
33 |
Models can be evaluated in three settings:
|
34 |
* `full` – **no** ground truth diffs are used for model evaluation;
|
35 |
* `oracle: files` – ground truth diffs are used to select files that should be corrected to fix the issue;
|
|
|
29 |
|
30 |
The benchmark clones the repo to the local directory, the model fixes the issue according to logs and the local repo state,
|
31 |
and then the benchmark pushes the repo to GitHub and requests the result of the GitHub CI.
|
32 |
+
We use the `Pass@1` rate metric to measure CI repair, indicating the ratio of data points, for which the build passed successfully after the generated fix.
|
33 |
+
|
34 |
Models can be evaluated in three settings:
|
35 |
* `full` – **no** ground truth diffs are used for model evaluation;
|
36 |
* `oracle: files` – ground truth diffs are used to select files that should be corrected to fix the issue;
|