galtimur commited on
Commit
84dc4f2
·
verified ·
1 Parent(s): 8314e15

Update src/tasks_content.py

Browse files
Files changed (1) hide show
  1. src/tasks_content.py +2 -1
src/tasks_content.py CHANGED
@@ -29,7 +29,8 @@ TASKS_DESCRIPTIONS = {
29
 
30
  The benchmark clones the repo to the local directory, the model fixes the issue according to logs and the local repo state,
31
  and then the benchmark pushes the repo to GitHub and requests the result of the GitHub CI.
32
- We use the `Pass@1` rate metric to measure CI repair, indicating the ratio of data points, for which the build passed successfully after the generated fix.
 
33
  Models can be evaluated in three settings:
34
  * `full` – **no** ground truth diffs are used for model evaluation;
35
  * `oracle: files` – ground truth diffs are used to select files that should be corrected to fix the issue;
 
29
 
30
  The benchmark clones the repo to the local directory, the model fixes the issue according to logs and the local repo state,
31
  and then the benchmark pushes the repo to GitHub and requests the result of the GitHub CI.
32
+ We use the `Pass@1` rate metric to measure CI repair, indicating the ratio of data points, for which the build passed successfully after the generated fix.
33
+
34
  Models can be evaluated in three settings:
35
  * `full` – **no** ground truth diffs are used for model evaluation;
36
  * `oracle: files` – ground truth diffs are used to select files that should be corrected to fix the issue;