Spaces:
Sleeping
Sleeping
Minor readme cleanup
Browse files
README.md
CHANGED
@@ -36,23 +36,23 @@ os.environ["HF_ALLOW_CODE_EVAL"] = "1"`
|
|
36 |
### Inputs
|
37 |
The following arguments are inherited from the basic `code_eval`:
|
38 |
|
39 |
-
|
40 |
|
41 |
-
|
42 |
|
43 |
-
|
44 |
|
45 |
-
|
46 |
|
47 |
-
|
48 |
|
49 |
In addition, this metric supports three additional arguments, specifying which default imports should be made available:
|
50 |
|
51 |
-
|
52 |
|
53 |
-
|
54 |
|
55 |
-
|
56 |
|
57 |
As the additional arguments are optional, this could be used as a drop-in replacement for `code_eval`.
|
58 |
|
|
|
36 |
### Inputs
|
37 |
The following arguments are inherited from the basic `code_eval`:
|
38 |
|
39 |
+
**`predictions`** (`List[List[str]]`): a list of candidates to evaluate. Each candidate should be a list of strings with several code candidates to solve the problem.
|
40 |
|
41 |
+
**`references`** (`List[str]`): a list with a test for each prediction. Each test should evaluate the correctness of a code candidate.
|
42 |
|
43 |
+
**`k`** (`List[int]`): number of code candidates to consider in the evaluation. The default value is `[1, 10, 100]`.
|
44 |
|
45 |
+
**`num_workers`** (`int`): the number of workers used to evaluate the candidate programs (The default value is `4`).
|
46 |
|
47 |
+
**`timeout`** (`float`): The maximum time taken to produce a prediction before it is considered a "timeout". The default value is `3.0` (i.e. 3 seconds).
|
48 |
|
49 |
In addition, this metric supports three additional arguments, specifying which default imports should be made available:
|
50 |
|
51 |
+
**`use_safe_builtins`** (`bool`): Whether or not to allow the usage of [`RestrictedPython.safe_builtins`](https://github.com/zopefoundation/RestrictedPython/blob/c31c133844ac2308f5cc930e934a7227a2a6a77b/src/RestrictedPython/Guards.py#L23), defaults to True
|
52 |
|
53 |
+
**`use_limited_builtins`** (`bool`): Whether or not to allow the usage of [`RestrictedPython.limited_builtins`](https://github.com/zopefoundation/RestrictedPython/blob/c31c133844ac2308f5cc930e934a7227a2a6a77b/src/RestrictedPython/Limits.py#L14), which provides limited implementations of `range`, `list`, and `tuple`. defaults to True.
|
54 |
|
55 |
+
**`use_utility_builtins`** (`bool`): Whether or not to allow the usage of [`RestrictedPython.utility_builtins`](https://github.com/zopefoundation/RestrictedPython/blob/c31c133844ac2308f5cc930e934a7227a2a6a77b/src/RestrictedPython/Utilities.py#L19), which includes the `string`, `math`, `random`, and `set` packages, among others. Defaults to True.
|
56 |
|
57 |
As the additional arguments are optional, this could be used as a drop-in replacement for `code_eval`.
|
58 |
|