We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
hf_argparser doesn't set the required flag on non-defaulted enums
## Environment info

- `transformers` version: 3.0.0-4.3.3
- Platform: macOS
- Python version: 3.9
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?): n/a
- Using GPU in script?: n/a
- Using distributed or parallel set-up in script?: n/a

### Who can help

I'm not sure who the owner is of hf_argparser.

## Information

We're using hf_argparser to parse our experiment config into dataclasses before training.

## To reproduce

Steps to reproduce the behavior:

1. Add an enum argument without a default to a dataclass
2. Parse the command line arguments without supplying the enum argument
3. Should have raised an exception and printed the usage, instead defaults the value to `None`.

## Expected behavior

It should raise an exception. The issue is on https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L100, the if statement should have an else which sets `kwargs["required"]=True`, the same way line [134](https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L134) does.

I can work up a patch if you agree this is an issue. I think it will also occur with anything that falls into [this branch](https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L118) of the if too.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/transformers/hf_argparser.py`
Content:
```
1 # Copyright 2020 The HuggingFace Team. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 
15 import dataclasses
16 import json
17 import re
18 import sys
19 from argparse import ArgumentParser, ArgumentTypeError
20 from enum import Enum
21 from pathlib import Path
22 from typing import Any, Iterable, List, NewType, Optional, Tuple, Union
23 
24 
25 DataClass = NewType("DataClass", Any)
26 DataClassType = NewType("DataClassType", Any)
27 
28 
29 # From https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse
30 def string_to_bool(v):
31     if isinstance(v, bool):
32         return v
33     if v.lower() in ("yes", "true", "t", "y", "1"):
34         return True
35     elif v.lower() in ("no", "false", "f", "n", "0"):
36         return False
37     else:
38         raise ArgumentTypeError(
39             f"Truthy value expected: got {v} but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive)."
40         )
41 
42 
43 class HfArgumentParser(ArgumentParser):
44     """
45     This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.
46 
47     The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)
48     arguments to the parser after initialization and you'll get the output back after parsing as an additional
49     namespace.
50     """
51 
52     dataclass_types: Iterable[DataClassType]
53 
54     def __init__(self, dataclass_types: Union[DataClassType, Iterable[DataClassType]], **kwargs):
55         """
56         Args:
57             dataclass_types:
58                 Dataclass type, or list of dataclass types for which we will "fill" instances with the parsed args.
59             kwargs:
60                 (Optional) Passed to `argparse.ArgumentParser()` in the regular way.
61         """
62         super().__init__(**kwargs)
63         if dataclasses.is_dataclass(dataclass_types):
64             dataclass_types = [dataclass_types]
65         self.dataclass_types = dataclass_types
66         for dtype in self.dataclass_types:
67             self._add_dataclass_arguments(dtype)
68 
69     def _add_dataclass_arguments(self, dtype: DataClassType):
70         for field in dataclasses.fields(dtype):
71             if not field.init:
72                 continue
73             field_name = f"--{field.name}"
74             kwargs = field.metadata.copy()
75             # field.metadata is not used at all by Data Classes,
76             # it is provided as a third-party extension mechanism.
77             if isinstance(field.type, str):
78                 raise ImportError(
79                     "This implementation is not compatible with Postponed Evaluation of Annotations (PEP 563),"
80                     "which can be opted in from Python 3.7 with `from __future__ import annotations`."
81                     "We will add compatibility when Python 3.9 is released."
82                 )
83             typestring = str(field.type)
84             for prim_type in (int, float, str):
85                 for collection in (List,):
86                     if (
87                         typestring == f"typing.Union[{collection[prim_type]}, NoneType]"
88                         or typestring == f"typing.Optional[{collection[prim_type]}]"
89                     ):
90                         field.type = collection[prim_type]
91                 if (
92                     typestring == f"typing.Union[{prim_type.__name__}, NoneType]"
93                     or typestring == f"typing.Optional[{prim_type.__name__}]"
94                 ):
95                     field.type = prim_type
96 
97             if isinstance(field.type, type) and issubclass(field.type, Enum):
98                 kwargs["choices"] = [x.value for x in field.type]
99                 kwargs["type"] = type(kwargs["choices"][0])
100                 if field.default is not dataclasses.MISSING:
101                     kwargs["default"] = field.default
102             elif field.type is bool or field.type == Optional[bool]:
103                 if field.default is True:
104                     self.add_argument(f"--no_{field.name}", action="store_false", dest=field.name, **kwargs)
105 
106                 # Hack because type=bool in argparse does not behave as we want.
107                 kwargs["type"] = string_to_bool
108                 if field.type is bool or (field.default is not None and field.default is not dataclasses.MISSING):
109                     # Default value is True if we have no default when of type bool.
110                     default = True if field.default is dataclasses.MISSING else field.default
111                     # This is the value that will get picked if we don't include --field_name in any way
112                     kwargs["default"] = default
113                     # This tells argparse we accept 0 or 1 value after --field_name
114                     kwargs["nargs"] = "?"
115                     # This is the value that will get picked if we do --field_name (without value)
116                     kwargs["const"] = True
117             elif (
118                 hasattr(field.type, "__origin__") and re.search(r"^typing\.List\[(.*)\]$", str(field.type)) is not None
119             ):
120                 kwargs["nargs"] = "+"
121                 kwargs["type"] = field.type.__args__[0]
122                 assert all(
123                     x == kwargs["type"] for x in field.type.__args__
124                 ), "{} cannot be a List of mixed types".format(field.name)
125                 if field.default_factory is not dataclasses.MISSING:
126                     kwargs["default"] = field.default_factory()
127             else:
128                 kwargs["type"] = field.type
129                 if field.default is not dataclasses.MISSING:
130                     kwargs["default"] = field.default
131                 elif field.default_factory is not dataclasses.MISSING:
132                     kwargs["default"] = field.default_factory()
133                 else:
134                     kwargs["required"] = True
135             self.add_argument(field_name, **kwargs)
136 
137     def parse_args_into_dataclasses(
138         self, args=None, return_remaining_strings=False, look_for_args_file=True, args_filename=None
139     ) -> Tuple[DataClass, ...]:
140         """
141         Parse command-line args into instances of the specified dataclass types.
142 
143         This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:
144         docs.python.org/3.7/library/argparse.html#argparse.ArgumentParser.parse_args
145 
146         Args:
147             args:
148                 List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)
149             return_remaining_strings:
150                 If true, also return a list of remaining argument strings.
151             look_for_args_file:
152                 If true, will look for a ".args" file with the same base name as the entry point script for this
153                 process, and will append its potential content to the command line args.
154             args_filename:
155                 If not None, will uses this file instead of the ".args" file specified in the previous argument.
156 
157         Returns:
158             Tuple consisting of:
159 
160                 - the dataclass instances in the same order as they were passed to the initializer.abspath
161                 - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser
162                   after initialization.
163                 - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)
164         """
165         if args_filename or (look_for_args_file and len(sys.argv)):
166             if args_filename:
167                 args_file = Path(args_filename)
168             else:
169                 args_file = Path(sys.argv[0]).with_suffix(".args")
170 
171             if args_file.exists():
172                 fargs = args_file.read_text().split()
173                 args = fargs + args if args is not None else fargs + sys.argv[1:]
174                 # in case of duplicate arguments the first one has precedence
175                 # so we append rather than prepend.
176         namespace, remaining_args = self.parse_known_args(args=args)
177         outputs = []
178         for dtype in self.dataclass_types:
179             keys = {f.name for f in dataclasses.fields(dtype) if f.init}
180             inputs = {k: v for k, v in vars(namespace).items() if k in keys}
181             for k in keys:
182                 delattr(namespace, k)
183             obj = dtype(**inputs)
184             outputs.append(obj)
185         if len(namespace.__dict__) > 0:
186             # additional namespace.
187             outputs.append(namespace)
188         if return_remaining_strings:
189             return (*outputs, remaining_args)
190         else:
191             if remaining_args:
192                 raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")
193 
194             return (*outputs,)
195 
196     def parse_json_file(self, json_file: str) -> Tuple[DataClass, ...]:
197         """
198         Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the
199         dataclass types.
200         """
201         data = json.loads(Path(json_file).read_text())
202         outputs = []
203         for dtype in self.dataclass_types:
204             keys = {f.name for f in dataclasses.fields(dtype) if f.init}
205             inputs = {k: v for k, v in data.items() if k in keys}
206             obj = dtype(**inputs)
207             outputs.append(obj)
208         return (*outputs,)
209 
210     def parse_dict(self, args: dict) -> Tuple[DataClass, ...]:
211         """
212         Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass
213         types.
214         """
215         outputs = []
216         for dtype in self.dataclass_types:
217             keys = {f.name for f in dataclasses.fields(dtype) if f.init}
218             inputs = {k: v for k, v in args.items() if k in keys}
219             obj = dtype(**inputs)
220             outputs.append(obj)
221         return (*outputs,)
222
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/transformers/hf_argparser.py b/src/transformers/hf_argparser.py
--- a/src/transformers/hf_argparser.py
+++ b/src/transformers/hf_argparser.py
@@ -99,6 +99,8 @@
                 kwargs["type"] = type(kwargs["choices"][0])
                 if field.default is not dataclasses.MISSING:
                     kwargs["default"] = field.default
+                else:
+                    kwargs["required"] = True
             elif field.type is bool or field.type == Optional[bool]:
                 if field.default is True:
                     self.add_argument(f"--no_{field.name}", action="store_false", dest=field.name, **kwargs)
@@ -124,6 +126,8 @@
                 ), "{} cannot be a List of mixed types".format(field.name)
                 if field.default_factory is not dataclasses.MISSING:
                     kwargs["default"] = field.default_factory()
+                elif field.default is dataclasses.MISSING:
+                    kwargs["required"] = True
             else:
                 kwargs["type"] = field.type
                 if field.default is not dataclasses.MISSING:

{"golden_diff": "diff --git a/src/transformers/hf_argparser.py b/src/transformers/hf_argparser.py\n--- a/src/transformers/hf_argparser.py\n+++ b/src/transformers/hf_argparser.py\n@@ -99,6 +99,8 @@\n                 kwargs[\"type\"] = type(kwargs[\"choices\"][0])\n                 if field.default is not dataclasses.MISSING:\n                     kwargs[\"default\"] = field.default\n+                else:\n+                    kwargs[\"required\"] = True\n             elif field.type is bool or field.type == Optional[bool]:\n                 if field.default is True:\n                     self.add_argument(f\"--no_{field.name}\", action=\"store_false\", dest=field.name, **kwargs)\n@@ -124,6 +126,8 @@\n                 ), \"{} cannot be a List of mixed types\".format(field.name)\n                 if field.default_factory is not dataclasses.MISSING:\n                     kwargs[\"default\"] = field.default_factory()\n+                elif field.default is dataclasses.MISSING:\n+                    kwargs[\"required\"] = True\n             else:\n                 kwargs[\"type\"] = field.type\n                 if field.default is not dataclasses.MISSING:\n", "issue": "hf_argparser doesn't set the required flag on non-defaulted enums\n## Environment info\r\n\r\n- `transformers` version: 3.0.0-4.3.3\r\n- Platform: macOS\r\n- Python version: 3.9\r\n- PyTorch version (GPU?): n/a\r\n- Tensorflow version (GPU?): n/a\r\n- Using GPU in script?: n/a\r\n- Using distributed or parallel set-up in script?: n/a\r\n\r\n### Who can help\r\n\r\nI'm not sure who the owner is of hf_argparser.\r\n\r\n## Information\r\n\r\nWe're using hf_argparser to parse our experiment config into dataclasses before training.\r\n\r\n## To reproduce\r\n\r\nSteps to reproduce the behavior:\r\n\r\n1. Add an enum argument without a default to a dataclass\r\n2. Parse the command line arguments without supplying the enum argument\r\n3. Should have raised an exception and printed the usage, instead defaults the value to `None`.\r\n\r\n## Expected behavior\r\n\r\nIt should raise an exception. The issue is on https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L100, the if statement should have an else which sets `kwargs[\"required\"]=True`, the same way line [134](https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L134) does.\r\n\r\nI can work up a patch if you agree this is an issue. I think it will also occur with anything that falls into [this branch](https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L118) of the if too.\r\n\n", "before_files": [{"content": "# Copyright 2020 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport dataclasses\nimport json\nimport re\nimport sys\nfrom argparse import ArgumentParser, ArgumentTypeError\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing import Any, Iterable, List, NewType, Optional, Tuple, Union\n\n\nDataClass = NewType(\"DataClass\", Any)\nDataClassType = NewType(\"DataClassType\", Any)\n\n\n# From https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse\ndef string_to_bool(v):\n    if isinstance(v, bool):\n        return v\n    if v.lower() in (\"yes\", \"true\", \"t\", \"y\", \"1\"):\n        return True\n    elif v.lower() in (\"no\", \"false\", \"f\", \"n\", \"0\"):\n        return False\n    else:\n        raise ArgumentTypeError(\n            f\"Truthy value expected: got {v} but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive).\"\n        )\n\n\nclass HfArgumentParser(ArgumentParser):\n    \"\"\"\n    This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.\n\n    The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)\n    arguments to the parser after initialization and you'll get the output back after parsing as an additional\n    namespace.\n    \"\"\"\n\n    dataclass_types: Iterable[DataClassType]\n\n    def __init__(self, dataclass_types: Union[DataClassType, Iterable[DataClassType]], **kwargs):\n        \"\"\"\n        Args:\n            dataclass_types:\n                Dataclass type, or list of dataclass types for which we will \"fill\" instances with the parsed args.\n            kwargs:\n                (Optional) Passed to `argparse.ArgumentParser()` in the regular way.\n        \"\"\"\n        super().__init__(**kwargs)\n        if dataclasses.is_dataclass(dataclass_types):\n            dataclass_types = [dataclass_types]\n        self.dataclass_types = dataclass_types\n        for dtype in self.dataclass_types:\n            self._add_dataclass_arguments(dtype)\n\n    def _add_dataclass_arguments(self, dtype: DataClassType):\n        for field in dataclasses.fields(dtype):\n            if not field.init:\n                continue\n            field_name = f\"--{field.name}\"\n            kwargs = field.metadata.copy()\n            # field.metadata is not used at all by Data Classes,\n            # it is provided as a third-party extension mechanism.\n            if isinstance(field.type, str):\n                raise ImportError(\n                    \"This implementation is not compatible with Postponed Evaluation of Annotations (PEP 563),\"\n                    \"which can be opted in from Python 3.7 with `from __future__ import annotations`.\"\n                    \"We will add compatibility when Python 3.9 is released.\"\n                )\n            typestring = str(field.type)\n            for prim_type in (int, float, str):\n                for collection in (List,):\n                    if (\n                        typestring == f\"typing.Union[{collection[prim_type]}, NoneType]\"\n                        or typestring == f\"typing.Optional[{collection[prim_type]}]\"\n                    ):\n                        field.type = collection[prim_type]\n                if (\n                    typestring == f\"typing.Union[{prim_type.__name__}, NoneType]\"\n                    or typestring == f\"typing.Optional[{prim_type.__name__}]\"\n                ):\n                    field.type = prim_type\n\n            if isinstance(field.type, type) and issubclass(field.type, Enum):\n                kwargs[\"choices\"] = [x.value for x in field.type]\n                kwargs[\"type\"] = type(kwargs[\"choices\"][0])\n                if field.default is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default\n            elif field.type is bool or field.type == Optional[bool]:\n                if field.default is True:\n                    self.add_argument(f\"--no_{field.name}\", action=\"store_false\", dest=field.name, **kwargs)\n\n                # Hack because type=bool in argparse does not behave as we want.\n                kwargs[\"type\"] = string_to_bool\n                if field.type is bool or (field.default is not None and field.default is not dataclasses.MISSING):\n                    # Default value is True if we have no default when of type bool.\n                    default = True if field.default is dataclasses.MISSING else field.default\n                    # This is the value that will get picked if we don't include --field_name in any way\n                    kwargs[\"default\"] = default\n                    # This tells argparse we accept 0 or 1 value after --field_name\n                    kwargs[\"nargs\"] = \"?\"\n                    # This is the value that will get picked if we do --field_name (without value)\n                    kwargs[\"const\"] = True\n            elif (\n                hasattr(field.type, \"__origin__\") and re.search(r\"^typing\\.List\\[(.*)\\]$\", str(field.type)) is not None\n            ):\n                kwargs[\"nargs\"] = \"+\"\n                kwargs[\"type\"] = field.type.__args__[0]\n                assert all(\n                    x == kwargs[\"type\"] for x in field.type.__args__\n                ), \"{} cannot be a List of mixed types\".format(field.name)\n                if field.default_factory is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default_factory()\n            else:\n                kwargs[\"type\"] = field.type\n                if field.default is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default\n                elif field.default_factory is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default_factory()\n                else:\n                    kwargs[\"required\"] = True\n            self.add_argument(field_name, **kwargs)\n\n    def parse_args_into_dataclasses(\n        self, args=None, return_remaining_strings=False, look_for_args_file=True, args_filename=None\n    ) -> Tuple[DataClass, ...]:\n        \"\"\"\n        Parse command-line args into instances of the specified dataclass types.\n\n        This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:\n        docs.python.org/3.7/library/argparse.html#argparse.ArgumentParser.parse_args\n\n        Args:\n            args:\n                List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)\n            return_remaining_strings:\n                If true, also return a list of remaining argument strings.\n            look_for_args_file:\n                If true, will look for a \".args\" file with the same base name as the entry point script for this\n                process, and will append its potential content to the command line args.\n            args_filename:\n                If not None, will uses this file instead of the \".args\" file specified in the previous argument.\n\n        Returns:\n            Tuple consisting of:\n\n                - the dataclass instances in the same order as they were passed to the initializer.abspath\n                - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser\n                  after initialization.\n                - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)\n        \"\"\"\n        if args_filename or (look_for_args_file and len(sys.argv)):\n            if args_filename:\n                args_file = Path(args_filename)\n            else:\n                args_file = Path(sys.argv[0]).with_suffix(\".args\")\n\n            if args_file.exists():\n                fargs = args_file.read_text().split()\n                args = fargs + args if args is not None else fargs + sys.argv[1:]\n                # in case of duplicate arguments the first one has precedence\n                # so we append rather than prepend.\n        namespace, remaining_args = self.parse_known_args(args=args)\n        outputs = []\n        for dtype in self.dataclass_types:\n            keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n            inputs = {k: v for k, v in vars(namespace).items() if k in keys}\n            for k in keys:\n                delattr(namespace, k)\n            obj = dtype(**inputs)\n            outputs.append(obj)\n        if len(namespace.__dict__) > 0:\n            # additional namespace.\n            outputs.append(namespace)\n        if return_remaining_strings:\n            return (*outputs, remaining_args)\n        else:\n            if remaining_args:\n                raise ValueError(f\"Some specified arguments are not used by the HfArgumentParser: {remaining_args}\")\n\n            return (*outputs,)\n\n    def parse_json_file(self, json_file: str) -> Tuple[DataClass, ...]:\n        \"\"\"\n        Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the\n        dataclass types.\n        \"\"\"\n        data = json.loads(Path(json_file).read_text())\n        outputs = []\n        for dtype in self.dataclass_types:\n            keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n            inputs = {k: v for k, v in data.items() if k in keys}\n            obj = dtype(**inputs)\n            outputs.append(obj)\n        return (*outputs,)\n\n    def parse_dict(self, args: dict) -> Tuple[DataClass, ...]:\n        \"\"\"\n        Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass\n        types.\n        \"\"\"\n        outputs = []\n        for dtype in self.dataclass_types:\n            keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n            inputs = {k: v for k, v in args.items() if k in keys}\n            obj = dtype(**inputs)\n            outputs.append(obj)\n        return (*outputs,)\n", "path": "src/transformers/hf_argparser.py"}], "after_files": [{"content": "# Copyright 2020 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport dataclasses\nimport json\nimport re\nimport sys\nfrom argparse import ArgumentParser, ArgumentTypeError\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing import Any, Iterable, List, NewType, Optional, Tuple, Union\n\n\nDataClass = NewType(\"DataClass\", Any)\nDataClassType = NewType(\"DataClassType\", Any)\n\n\n# From https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse\ndef string_to_bool(v):\n    if isinstance(v, bool):\n        return v\n    if v.lower() in (\"yes\", \"true\", \"t\", \"y\", \"1\"):\n        return True\n    elif v.lower() in (\"no\", \"false\", \"f\", \"n\", \"0\"):\n        return False\n    else:\n        raise ArgumentTypeError(\n            f\"Truthy value expected: got {v} but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive).\"\n        )\n\n\nclass HfArgumentParser(ArgumentParser):\n    \"\"\"\n    This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.\n\n    The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)\n    arguments to the parser after initialization and you'll get the output back after parsing as an additional\n    namespace.\n    \"\"\"\n\n    dataclass_types: Iterable[DataClassType]\n\n    def __init__(self, dataclass_types: Union[DataClassType, Iterable[DataClassType]], **kwargs):\n        \"\"\"\n        Args:\n            dataclass_types:\n                Dataclass type, or list of dataclass types for which we will \"fill\" instances with the parsed args.\n            kwargs:\n                (Optional) Passed to `argparse.ArgumentParser()` in the regular way.\n        \"\"\"\n        super().__init__(**kwargs)\n        if dataclasses.is_dataclass(dataclass_types):\n            dataclass_types = [dataclass_types]\n        self.dataclass_types = dataclass_types\n        for dtype in self.dataclass_types:\n            self._add_dataclass_arguments(dtype)\n\n    def _add_dataclass_arguments(self, dtype: DataClassType):\n        for field in dataclasses.fields(dtype):\n            if not field.init:\n                continue\n            field_name = f\"--{field.name}\"\n            kwargs = field.metadata.copy()\n            # field.metadata is not used at all by Data Classes,\n            # it is provided as a third-party extension mechanism.\n            if isinstance(field.type, str):\n                raise ImportError(\n                    \"This implementation is not compatible with Postponed Evaluation of Annotations (PEP 563),\"\n                    \"which can be opted in from Python 3.7 with `from __future__ import annotations`.\"\n                    \"We will add compatibility when Python 3.9 is released.\"\n                )\n            typestring = str(field.type)\n            for prim_type in (int, float, str):\n                for collection in (List,):\n                    if (\n                        typestring == f\"typing.Union[{collection[prim_type]}, NoneType]\"\n                        or typestring == f\"typing.Optional[{collection[prim_type]}]\"\n                    ):\n                        field.type = collection[prim_type]\n                if (\n                    typestring == f\"typing.Union[{prim_type.__name__}, NoneType]\"\n                    or typestring == f\"typing.Optional[{prim_type.__name__}]\"\n                ):\n                    field.type = prim_type\n\n            if isinstance(field.type, type) and issubclass(field.type, Enum):\n                kwargs[\"choices\"] = [x.value for x in field.type]\n                kwargs[\"type\"] = type(kwargs[\"choices\"][0])\n                if field.default is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default\n                else:\n                    kwargs[\"required\"] = True\n            elif field.type is bool or field.type == Optional[bool]:\n                if field.default is True:\n                    self.add_argument(f\"--no_{field.name}\", action=\"store_false\", dest=field.name, **kwargs)\n\n                # Hack because type=bool in argparse does not behave as we want.\n                kwargs[\"type\"] = string_to_bool\n                if field.type is bool or (field.default is not None and field.default is not dataclasses.MISSING):\n                    # Default value is True if we have no default when of type bool.\n                    default = True if field.default is dataclasses.MISSING else field.default\n                    # This is the value that will get picked if we don't include --field_name in any way\n                    kwargs[\"default\"] = default\n                    # This tells argparse we accept 0 or 1 value after --field_name\n                    kwargs[\"nargs\"] = \"?\"\n                    # This is the value that will get picked if we do --field_name (without value)\n                    kwargs[\"const\"] = True\n            elif (\n                hasattr(field.type, \"__origin__\") and re.search(r\"^typing\\.List\\[(.*)\\]$\", str(field.type)) is not None\n            ):\n                kwargs[\"nargs\"] = \"+\"\n                kwargs[\"type\"] = field.type.__args__[0]\n                assert all(\n                    x == kwargs[\"type\"] for x in field.type.__args__\n                ), \"{} cannot be a List of mixed types\".format(field.name)\n                if field.default_factory is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default_factory()\n                elif field.default is dataclasses.MISSING:\n                    kwargs[\"required\"] = True\n            else:\n                kwargs[\"type\"] = field.type\n                if field.default is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default\n                elif field.default_factory is not dataclasses.MISSING:\n                    kwargs[\"default\"] = field.default_factory()\n                else:\n                    kwargs[\"required\"] = True\n            self.add_argument(field_name, **kwargs)\n\n    def parse_args_into_dataclasses(\n        self, args=None, return_remaining_strings=False, look_for_args_file=True, args_filename=None\n    ) -> Tuple[DataClass, ...]:\n        \"\"\"\n        Parse command-line args into instances of the specified dataclass types.\n\n        This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:\n        docs.python.org/3.7/library/argparse.html#argparse.ArgumentParser.parse_args\n\n        Args:\n            args:\n                List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)\n            return_remaining_strings:\n                If true, also return a list of remaining argument strings.\n            look_for_args_file:\n                If true, will look for a \".args\" file with the same base name as the entry point script for this\n                process, and will append its potential content to the command line args.\n            args_filename:\n                If not None, will uses this file instead of the \".args\" file specified in the previous argument.\n\n        Returns:\n            Tuple consisting of:\n\n                - the dataclass instances in the same order as they were passed to the initializer.abspath\n                - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser\n                  after initialization.\n                - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)\n        \"\"\"\n        if args_filename or (look_for_args_file and len(sys.argv)):\n            if args_filename:\n                args_file = Path(args_filename)\n            else:\n                args_file = Path(sys.argv[0]).with_suffix(\".args\")\n\n            if args_file.exists():\n                fargs = args_file.read_text().split()\n                args = fargs + args if args is not None else fargs + sys.argv[1:]\n                # in case of duplicate arguments the first one has precedence\n                # so we append rather than prepend.\n        namespace, remaining_args = self.parse_known_args(args=args)\n        outputs = []\n        for dtype in self.dataclass_types:\n            keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n            inputs = {k: v for k, v in vars(namespace).items() if k in keys}\n            for k in keys:\n                delattr(namespace, k)\n            obj = dtype(**inputs)\n            outputs.append(obj)\n        if len(namespace.__dict__) > 0:\n            # additional namespace.\n            outputs.append(namespace)\n        if return_remaining_strings:\n            return (*outputs, remaining_args)\n        else:\n            if remaining_args:\n                raise ValueError(f\"Some specified arguments are not used by the HfArgumentParser: {remaining_args}\")\n\n            return (*outputs,)\n\n    def parse_json_file(self, json_file: str) -> Tuple[DataClass, ...]:\n        \"\"\"\n        Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the\n        dataclass types.\n        \"\"\"\n        data = json.loads(Path(json_file).read_text())\n        outputs = []\n        for dtype in self.dataclass_types:\n            keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n            inputs = {k: v for k, v in data.items() if k in keys}\n            obj = dtype(**inputs)\n            outputs.append(obj)\n        return (*outputs,)\n\n    def parse_dict(self, args: dict) -> Tuple[DataClass, ...]:\n        \"\"\"\n        Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass\n        types.\n        \"\"\"\n        outputs = []\n        for dtype in self.dataclass_types:\n            keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n            inputs = {k: v for k, v in args.items() if k in keys}\n            obj = dtype(**inputs)\n            outputs.append(obj)\n        return (*outputs,)\n", "path": "src/transformers/hf_argparser.py"}]}