We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Provide a more useful error when parsing fails during AST safety checks
### Summary & why

Currently when Black encounters a SyntaxError while parsing during its AST safety check, it tries older versions of Python to see if it will parse then under its grammar (i.e. the syntax - which is an oversimplification but generally words). In other words, if you have a file that is valid Python 2.7 code and are running on Python 3.8, Black will first try will try Python 3.8 which fails, then 3.7 which also fails, then 3.6, 3.5, and finally 2.7. Since 2.7 is last version it tried parsing with, the error message from that version is the one ultimately emitted to the user.

The behaviour described above logically makes sense but is of limited use and sometimes even annoyance of the end-user since chances are that their file is supposed to be valid Python 3, so a message for Python 2 ain't useful (and can be downright confusing). 

### Solutions & technical details 

We should change Black so when a SyntaxError is encountered while parsing during the AST safety checks it reparses with the latest Python it can try. Then we can take its error message and emit that one to the user, and voilà, a much more useful error message.  

Another option (from @ichard26 personally) is that we could record the error message of the latest Python parse attempt when it fails and then if all of the parse attempts fail, emit the recorded error message instead of freshest error message (probably Python 2). 

One slight issue is that we emit an error telling users if they need Python 2 support they should reinstall Black with the `python2` extra when we try Python 2 parsing (which fails due to missing dependencies). This could be downgraded to a warning so the rest of this enhancement can still do its thing. Or it could just be eliminated which is probably fine since Python 2 is basically dead. 

To potential contributors: 

- [Checkout our contributing docs to get started](https://github.com/psf/black/blob/master/CONTRIBUTING.md). If you need help, feel free to reach out!
- You'll need to definitely touch this part of the codebase: https://github.com/psf/black/blob/14c76e89716b5b53c97ece80bb935ea956b7dd89/src/black/__init__.py#L6422-L6443
- You'll probably also need to add / modify some tests in `tests/` 
- And finally add a changelog entry

\- Richard (@ichard26) (added for anyone looking to contribute)

---

## Original Report

This one had me scratching my head for a while.

**How to reproduce**

Feed the following into black with -t py37:

```
reported_error = f""

e=""
# Actual error
malformed_fstring = f"{e}  ({type(e).__name__], {e.args})"

```
Please note the typo in `malformed_fstring`: I put a square closing bracket "]" in place of a curly closing bracket "}".
Please also note that if the e variable is not set, the error does not show up.

**Expected result**

I expected that black would form a usable error message about "malformed_fstring having unmatched parenteses" or some such thing.

**Actual result**

```
black -l 999 -t py37 "test.py"
error: cannot format /tmp/test.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 1)
Oh no! 💥 💔 💥
1 files left unchanged, 1 file failed to reformat.
```

Please note that it reports an error on line 1, even if the error is on line 5.

**Environment**

```
$ black --version
black, version 20.8b1
$ python --version
Python 3.7.4
$ uname -a
Linux y8 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

```
**Possible causes**

After digging for a bit it it seems this happens when black fails to use the AST for python 3.x for some reason and falls back to trying the AST for 2.x which does not support f-strings at all and then barfs on the very first instance of an f-string. In the future, I will take note that whenever black barfs on an f-string, it is because it is parsing thje file as 2.x because it completely failed to parse as 3.x, and instead of looking at the f-string mentioned, start going all out Newton on the rest of the file.

As for why this exact string produces this outcome I can only speculate it has something to do with the "type" keyword being misinterpreted somehow.




--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/black/parsing.py`
Content:
```
1 """
2 Parse Python code and perform AST validation.
3 """
4 import ast
5 import sys
6 from typing import Iterable, Iterator, List, Set, Union
7 
8 # lib2to3 fork
9 from blib2to3.pytree import Node, Leaf
10 from blib2to3 import pygram, pytree
11 from blib2to3.pgen2 import driver
12 from blib2to3.pgen2.grammar import Grammar
13 from blib2to3.pgen2.parse import ParseError
14 
15 from black.mode import TargetVersion, Feature, supports_feature
16 from black.nodes import syms
17 
18 try:
19     from typed_ast import ast3, ast27
20 except ImportError:
21     if sys.version_info < (3, 8):
22         print(
23             "The typed_ast package is required but not installed.\n"
24             "You can upgrade to Python 3.8+ or install typed_ast with\n"
25             "`python3 -m pip install typed-ast`.",
26             file=sys.stderr,
27         )
28         sys.exit(1)
29     else:
30         ast3 = ast27 = ast
31 
32 
33 class InvalidInput(ValueError):
34     """Raised when input source code fails all parse attempts."""
35 
36 
37 def get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:
38     if not target_versions:
39         # No target_version specified, so try all grammars.
40         return [
41             # Python 3.7+
42             pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords,
43             # Python 3.0-3.6
44             pygram.python_grammar_no_print_statement_no_exec_statement,
45             # Python 2.7 with future print_function import
46             pygram.python_grammar_no_print_statement,
47             # Python 2.7
48             pygram.python_grammar,
49         ]
50 
51     if all(version.is_python2() for version in target_versions):
52         # Python 2-only code, so try Python 2 grammars.
53         return [
54             # Python 2.7 with future print_function import
55             pygram.python_grammar_no_print_statement,
56             # Python 2.7
57             pygram.python_grammar,
58         ]
59 
60     # Python 3-compatible code, so only try Python 3 grammar.
61     grammars = []
62     # If we have to parse both, try to parse async as a keyword first
63     if not supports_feature(target_versions, Feature.ASYNC_IDENTIFIERS):
64         # Python 3.7+
65         grammars.append(
66             pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords
67         )
68     if not supports_feature(target_versions, Feature.ASYNC_KEYWORDS):
69         # Python 3.0-3.6
70         grammars.append(pygram.python_grammar_no_print_statement_no_exec_statement)
71     # At least one of the above branches must have been taken, because every Python
72     # version has exactly one of the two 'ASYNC_*' flags
73     return grammars
74 
75 
76 def lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:
77     """Given a string with source, return the lib2to3 Node."""
78     if not src_txt.endswith("\n"):
79         src_txt += "\n"
80 
81     for grammar in get_grammars(set(target_versions)):
82         drv = driver.Driver(grammar, pytree.convert)
83         try:
84             result = drv.parse_string(src_txt, True)
85             break
86 
87         except ParseError as pe:
88             lineno, column = pe.context[1]
89             lines = src_txt.splitlines()
90             try:
91                 faulty_line = lines[lineno - 1]
92             except IndexError:
93                 faulty_line = "<line number missing in source>"
94             exc = InvalidInput(f"Cannot parse: {lineno}:{column}: {faulty_line}")
95     else:
96         raise exc from None
97 
98     if isinstance(result, Leaf):
99         result = Node(syms.file_input, [result])
100     return result
101 
102 
103 def lib2to3_unparse(node: Node) -> str:
104     """Given a lib2to3 node, return its string representation."""
105     code = str(node)
106     return code
107 
108 
109 def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:
110     filename = "<unknown>"
111     if sys.version_info >= (3, 8):
112         # TODO: support Python 4+ ;)
113         for minor_version in range(sys.version_info[1], 4, -1):
114             try:
115                 return ast.parse(src, filename, feature_version=(3, minor_version))
116             except SyntaxError:
117                 continue
118     else:
119         for feature_version in (7, 6):
120             try:
121                 return ast3.parse(src, filename, feature_version=feature_version)
122             except SyntaxError:
123                 continue
124     if ast27.__name__ == "ast":
125         raise SyntaxError(
126             "The requested source code has invalid Python 3 syntax.\n"
127             "If you are trying to format Python 2 files please reinstall Black"
128             " with the 'python2' extra: `python3 -m pip install black[python2]`."
129         )
130     return ast27.parse(src)
131 
132 
133 def stringify_ast(
134     node: Union[ast.AST, ast3.AST, ast27.AST], depth: int = 0
135 ) -> Iterator[str]:
136     """Simple visitor generating strings to compare ASTs by content."""
137 
138     node = fixup_ast_constants(node)
139 
140     yield f"{'  ' * depth}{node.__class__.__name__}("
141 
142     for field in sorted(node._fields):  # noqa: F402
143         # TypeIgnore has only one field 'lineno' which breaks this comparison
144         type_ignore_classes = (ast3.TypeIgnore, ast27.TypeIgnore)
145         if sys.version_info >= (3, 8):
146             type_ignore_classes += (ast.TypeIgnore,)
147         if isinstance(node, type_ignore_classes):
148             break
149 
150         try:
151             value = getattr(node, field)
152         except AttributeError:
153             continue
154 
155         yield f"{'  ' * (depth+1)}{field}="
156 
157         if isinstance(value, list):
158             for item in value:
159                 # Ignore nested tuples within del statements, because we may insert
160                 # parentheses and they change the AST.
161                 if (
162                     field == "targets"
163                     and isinstance(node, (ast.Delete, ast3.Delete, ast27.Delete))
164                     and isinstance(item, (ast.Tuple, ast3.Tuple, ast27.Tuple))
165                 ):
166                     for item in item.elts:
167                         yield from stringify_ast(item, depth + 2)
168 
169                 elif isinstance(item, (ast.AST, ast3.AST, ast27.AST)):
170                     yield from stringify_ast(item, depth + 2)
171 
172         elif isinstance(value, (ast.AST, ast3.AST, ast27.AST)):
173             yield from stringify_ast(value, depth + 2)
174 
175         else:
176             # Constant strings may be indented across newlines, if they are
177             # docstrings; fold spaces after newlines when comparing. Similarly,
178             # trailing and leading space may be removed.
179             # Note that when formatting Python 2 code, at least with Windows
180             # line-endings, docstrings can end up here as bytes instead of
181             # str so make sure that we handle both cases.
182             if (
183                 isinstance(node, ast.Constant)
184                 and field == "value"
185                 and isinstance(value, (str, bytes))
186             ):
187                 lineend = "\n" if isinstance(value, str) else b"\n"
188                 # To normalize, we strip any leading and trailing space from
189                 # each line...
190                 stripped = [line.strip() for line in value.splitlines()]
191                 normalized = lineend.join(stripped)  # type: ignore[attr-defined]
192                 # ...and remove any blank lines at the beginning and end of
193                 # the whole string
194                 normalized = normalized.strip()
195             else:
196                 normalized = value
197             yield f"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}"
198 
199     yield f"{'  ' * depth})  # /{node.__class__.__name__}"
200 
201 
202 def fixup_ast_constants(
203     node: Union[ast.AST, ast3.AST, ast27.AST]
204 ) -> Union[ast.AST, ast3.AST, ast27.AST]:
205     """Map ast nodes deprecated in 3.8 to Constant."""
206     if isinstance(node, (ast.Str, ast3.Str, ast27.Str, ast.Bytes, ast3.Bytes)):
207         return ast.Constant(value=node.s)
208 
209     if isinstance(node, (ast.Num, ast3.Num, ast27.Num)):
210         return ast.Constant(value=node.n)
211 
212     if isinstance(node, (ast.NameConstant, ast3.NameConstant)):
213         return ast.Constant(value=node.value)
214 
215     return node
216
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/black/parsing.py b/src/black/parsing.py
--- a/src/black/parsing.py
+++ b/src/black/parsing.py
@@ -3,7 +3,7 @@
 """
 import ast
 import sys
-from typing import Iterable, Iterator, List, Set, Union
+from typing import Iterable, Iterator, List, Set, Union, Tuple
 
 # lib2to3 fork
 from blib2to3.pytree import Node, Leaf
@@ -106,28 +106,36 @@
     return code
 
 
-def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:
+def parse_single_version(
+    src: str, version: Tuple[int, int]
+) -> Union[ast.AST, ast3.AST, ast27.AST]:
     filename = "<unknown>"
-    if sys.version_info >= (3, 8):
-        # TODO: support Python 4+ ;)
-        for minor_version in range(sys.version_info[1], 4, -1):
-            try:
-                return ast.parse(src, filename, feature_version=(3, minor_version))
-            except SyntaxError:
-                continue
-    else:
-        for feature_version in (7, 6):
-            try:
-                return ast3.parse(src, filename, feature_version=feature_version)
-            except SyntaxError:
-                continue
-    if ast27.__name__ == "ast":
-        raise SyntaxError(
-            "The requested source code has invalid Python 3 syntax.\n"
-            "If you are trying to format Python 2 files please reinstall Black"
-            " with the 'python2' extra: `python3 -m pip install black[python2]`."
-        )
-    return ast27.parse(src)
+    # typed_ast is needed because of feature version limitations in the builtin ast
+    if sys.version_info >= (3, 8) and version >= (3,):
+        return ast.parse(src, filename, feature_version=version)
+    elif version >= (3,):
+        return ast3.parse(src, filename, feature_version=version[1])
+    elif version == (2, 7):
+        return ast27.parse(src)
+    raise AssertionError("INTERNAL ERROR: Tried parsing unsupported Python version!")
+
+
+def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:
+    # TODO: support Python 4+ ;)
+    versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]
+
+    if ast27.__name__ != "ast":
+        versions.append((2, 7))
+
+    first_error = ""
+    for version in sorted(versions, reverse=True):
+        try:
+            return parse_single_version(src, version)
+        except SyntaxError as e:
+            if not first_error:
+                first_error = str(e)
+
+    raise SyntaxError(first_error)
 
 
 def stringify_ast(

{"golden_diff": "diff --git a/src/black/parsing.py b/src/black/parsing.py\n--- a/src/black/parsing.py\n+++ b/src/black/parsing.py\n@@ -3,7 +3,7 @@\n \"\"\"\n import ast\n import sys\n-from typing import Iterable, Iterator, List, Set, Union\n+from typing import Iterable, Iterator, List, Set, Union, Tuple\n \n # lib2to3 fork\n from blib2to3.pytree import Node, Leaf\n@@ -106,28 +106,36 @@\n     return code\n \n \n-def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:\n+def parse_single_version(\n+    src: str, version: Tuple[int, int]\n+) -> Union[ast.AST, ast3.AST, ast27.AST]:\n     filename = \"<unknown>\"\n-    if sys.version_info >= (3, 8):\n-        # TODO: support Python 4+ ;)\n-        for minor_version in range(sys.version_info[1], 4, -1):\n-            try:\n-                return ast.parse(src, filename, feature_version=(3, minor_version))\n-            except SyntaxError:\n-                continue\n-    else:\n-        for feature_version in (7, 6):\n-            try:\n-                return ast3.parse(src, filename, feature_version=feature_version)\n-            except SyntaxError:\n-                continue\n-    if ast27.__name__ == \"ast\":\n-        raise SyntaxError(\n-            \"The requested source code has invalid Python 3 syntax.\\n\"\n-            \"If you are trying to format Python 2 files please reinstall Black\"\n-            \" with the 'python2' extra: `python3 -m pip install black[python2]`.\"\n-        )\n-    return ast27.parse(src)\n+    # typed_ast is needed because of feature version limitations in the builtin ast\n+    if sys.version_info >= (3, 8) and version >= (3,):\n+        return ast.parse(src, filename, feature_version=version)\n+    elif version >= (3,):\n+        return ast3.parse(src, filename, feature_version=version[1])\n+    elif version == (2, 7):\n+        return ast27.parse(src)\n+    raise AssertionError(\"INTERNAL ERROR: Tried parsing unsupported Python version!\")\n+\n+\n+def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:\n+    # TODO: support Python 4+ ;)\n+    versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]\n+\n+    if ast27.__name__ != \"ast\":\n+        versions.append((2, 7))\n+\n+    first_error = \"\"\n+    for version in sorted(versions, reverse=True):\n+        try:\n+            return parse_single_version(src, version)\n+        except SyntaxError as e:\n+            if not first_error:\n+                first_error = str(e)\n+\n+    raise SyntaxError(first_error)\n \n \n def stringify_ast(\n", "issue": "Provide a more useful error when parsing fails during AST safety checks\n### Summary & why\r\n\r\nCurrently when Black encounters a SyntaxError while parsing during its AST safety check, it tries older versions of Python to see if it will parse then under its grammar (i.e. the syntax - which is an oversimplification but generally words). In other words, if you have a file that is valid Python 2.7 code and are running on Python 3.8, Black will first try will try Python 3.8 which fails, then 3.7 which also fails, then 3.6, 3.5, and finally 2.7. Since 2.7 is last version it tried parsing with, the error message from that version is the one ultimately emitted to the user.\r\n\r\nThe behaviour described above logically makes sense but is of limited use and sometimes even annoyance of the end-user since chances are that their file is supposed to be valid Python 3, so a message for Python 2 ain't useful (and can be downright confusing). \r\n\r\n### Solutions & technical details \r\n\r\nWe should change Black so when a SyntaxError is encountered while parsing during the AST safety checks it reparses with the latest Python it can try. Then we can take its error message and emit that one to the user, and voil\u00e0, a much more useful error message.  \r\n\r\nAnother option (from @ichard26 personally) is that we could record the error message of the latest Python parse attempt when it fails and then if all of the parse attempts fail, emit the recorded error message instead of freshest error message (probably Python 2). \r\n\r\nOne slight issue is that we emit an error telling users if they need Python 2 support they should reinstall Black with the `python2` extra when we try Python 2 parsing (which fails due to missing dependencies). This could be downgraded to a warning so the rest of this enhancement can still do its thing. Or it could just be eliminated which is probably fine since Python 2 is basically dead. \r\n\r\nTo potential contributors: \r\n\r\n- [Checkout our contributing docs to get started](https://github.com/psf/black/blob/master/CONTRIBUTING.md). If you need help, feel free to reach out!\r\n- You'll need to definitely touch this part of the codebase: https://github.com/psf/black/blob/14c76e89716b5b53c97ece80bb935ea956b7dd89/src/black/__init__.py#L6422-L6443\r\n- You'll probably also need to add / modify some tests in `tests/` \r\n- And finally add a changelog entry\r\n\r\n\\- Richard (@ichard26) (added for anyone looking to contribute)\r\n\r\n---\r\n\r\n## Original Report\r\n\r\nThis one had me scratching my head for a while.\r\n\r\n**How to reproduce**\r\n\r\nFeed the following into black with -t py37:\r\n\r\n```\r\nreported_error = f\"\"\r\n\r\ne=\"\"\r\n# Actual error\r\nmalformed_fstring = f\"{e}  ({type(e).__name__], {e.args})\"\r\n\r\n```\r\nPlease note the typo in `malformed_fstring`: I put a square closing bracket \"]\" in place of a curly closing bracket \"}\".\r\nPlease also note that if the e variable is not set, the error does not show up.\r\n\r\n**Expected result**\r\n\r\nI expected that black would form a usable error message about \"malformed_fstring having unmatched parenteses\" or some such thing.\r\n\r\n**Actual result**\r\n\r\n```\r\nblack -l 999 -t py37 \"test.py\"\r\nerror: cannot format /tmp/test.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 1)\r\nOh no! \ud83d\udca5 \ud83d\udc94 \ud83d\udca5\r\n1 files left unchanged, 1 file failed to reformat.\r\n```\r\n\r\nPlease note that it reports an error on line 1, even if the error is on line 5.\r\n\r\n**Environment**\r\n\r\n```\r\n$ black --version\r\nblack, version 20.8b1\r\n$ python --version\r\nPython 3.7.4\r\n$ uname -a\r\nLinux y8 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux\r\n\r\n```\r\n**Possible causes**\r\n\r\nAfter digging for a bit it it seems this happens when black fails to use the AST for python 3.x for some reason and falls back to trying the AST for 2.x which does not support f-strings at all and then barfs on the very first instance of an f-string. In the future, I will take note that whenever black barfs on an f-string, it is because it is parsing thje file as 2.x because it completely failed to parse as 3.x, and instead of looking at the f-string mentioned, start going all out Newton on the rest of the file.\r\n\r\nAs for why this exact string produces this outcome I can only speculate it has something to do with the \"type\" keyword being misinterpreted somehow.\r\n\r\n\r\n\n", "before_files": [{"content": "\"\"\"\nParse Python code and perform AST validation.\n\"\"\"\nimport ast\nimport sys\nfrom typing import Iterable, Iterator, List, Set, Union\n\n# lib2to3 fork\nfrom blib2to3.pytree import Node, Leaf\nfrom blib2to3 import pygram, pytree\nfrom blib2to3.pgen2 import driver\nfrom blib2to3.pgen2.grammar import Grammar\nfrom blib2to3.pgen2.parse import ParseError\n\nfrom black.mode import TargetVersion, Feature, supports_feature\nfrom black.nodes import syms\n\ntry:\n    from typed_ast import ast3, ast27\nexcept ImportError:\n    if sys.version_info < (3, 8):\n        print(\n            \"The typed_ast package is required but not installed.\\n\"\n            \"You can upgrade to Python 3.8+ or install typed_ast with\\n\"\n            \"`python3 -m pip install typed-ast`.\",\n            file=sys.stderr,\n        )\n        sys.exit(1)\n    else:\n        ast3 = ast27 = ast\n\n\nclass InvalidInput(ValueError):\n    \"\"\"Raised when input source code fails all parse attempts.\"\"\"\n\n\ndef get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:\n    if not target_versions:\n        # No target_version specified, so try all grammars.\n        return [\n            # Python 3.7+\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords,\n            # Python 3.0-3.6\n            pygram.python_grammar_no_print_statement_no_exec_statement,\n            # Python 2.7 with future print_function import\n            pygram.python_grammar_no_print_statement,\n            # Python 2.7\n            pygram.python_grammar,\n        ]\n\n    if all(version.is_python2() for version in target_versions):\n        # Python 2-only code, so try Python 2 grammars.\n        return [\n            # Python 2.7 with future print_function import\n            pygram.python_grammar_no_print_statement,\n            # Python 2.7\n            pygram.python_grammar,\n        ]\n\n    # Python 3-compatible code, so only try Python 3 grammar.\n    grammars = []\n    # If we have to parse both, try to parse async as a keyword first\n    if not supports_feature(target_versions, Feature.ASYNC_IDENTIFIERS):\n        # Python 3.7+\n        grammars.append(\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords\n        )\n    if not supports_feature(target_versions, Feature.ASYNC_KEYWORDS):\n        # Python 3.0-3.6\n        grammars.append(pygram.python_grammar_no_print_statement_no_exec_statement)\n    # At least one of the above branches must have been taken, because every Python\n    # version has exactly one of the two 'ASYNC_*' flags\n    return grammars\n\n\ndef lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:\n    \"\"\"Given a string with source, return the lib2to3 Node.\"\"\"\n    if not src_txt.endswith(\"\\n\"):\n        src_txt += \"\\n\"\n\n    for grammar in get_grammars(set(target_versions)):\n        drv = driver.Driver(grammar, pytree.convert)\n        try:\n            result = drv.parse_string(src_txt, True)\n            break\n\n        except ParseError as pe:\n            lineno, column = pe.context[1]\n            lines = src_txt.splitlines()\n            try:\n                faulty_line = lines[lineno - 1]\n            except IndexError:\n                faulty_line = \"<line number missing in source>\"\n            exc = InvalidInput(f\"Cannot parse: {lineno}:{column}: {faulty_line}\")\n    else:\n        raise exc from None\n\n    if isinstance(result, Leaf):\n        result = Node(syms.file_input, [result])\n    return result\n\n\ndef lib2to3_unparse(node: Node) -> str:\n    \"\"\"Given a lib2to3 node, return its string representation.\"\"\"\n    code = str(node)\n    return code\n\n\ndef parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:\n    filename = \"<unknown>\"\n    if sys.version_info >= (3, 8):\n        # TODO: support Python 4+ ;)\n        for minor_version in range(sys.version_info[1], 4, -1):\n            try:\n                return ast.parse(src, filename, feature_version=(3, minor_version))\n            except SyntaxError:\n                continue\n    else:\n        for feature_version in (7, 6):\n            try:\n                return ast3.parse(src, filename, feature_version=feature_version)\n            except SyntaxError:\n                continue\n    if ast27.__name__ == \"ast\":\n        raise SyntaxError(\n            \"The requested source code has invalid Python 3 syntax.\\n\"\n            \"If you are trying to format Python 2 files please reinstall Black\"\n            \" with the 'python2' extra: `python3 -m pip install black[python2]`.\"\n        )\n    return ast27.parse(src)\n\n\ndef stringify_ast(\n    node: Union[ast.AST, ast3.AST, ast27.AST], depth: int = 0\n) -> Iterator[str]:\n    \"\"\"Simple visitor generating strings to compare ASTs by content.\"\"\"\n\n    node = fixup_ast_constants(node)\n\n    yield f\"{'  ' * depth}{node.__class__.__name__}(\"\n\n    for field in sorted(node._fields):  # noqa: F402\n        # TypeIgnore has only one field 'lineno' which breaks this comparison\n        type_ignore_classes = (ast3.TypeIgnore, ast27.TypeIgnore)\n        if sys.version_info >= (3, 8):\n            type_ignore_classes += (ast.TypeIgnore,)\n        if isinstance(node, type_ignore_classes):\n            break\n\n        try:\n            value = getattr(node, field)\n        except AttributeError:\n            continue\n\n        yield f\"{'  ' * (depth+1)}{field}=\"\n\n        if isinstance(value, list):\n            for item in value:\n                # Ignore nested tuples within del statements, because we may insert\n                # parentheses and they change the AST.\n                if (\n                    field == \"targets\"\n                    and isinstance(node, (ast.Delete, ast3.Delete, ast27.Delete))\n                    and isinstance(item, (ast.Tuple, ast3.Tuple, ast27.Tuple))\n                ):\n                    for item in item.elts:\n                        yield from stringify_ast(item, depth + 2)\n\n                elif isinstance(item, (ast.AST, ast3.AST, ast27.AST)):\n                    yield from stringify_ast(item, depth + 2)\n\n        elif isinstance(value, (ast.AST, ast3.AST, ast27.AST)):\n            yield from stringify_ast(value, depth + 2)\n\n        else:\n            # Constant strings may be indented across newlines, if they are\n            # docstrings; fold spaces after newlines when comparing. Similarly,\n            # trailing and leading space may be removed.\n            # Note that when formatting Python 2 code, at least with Windows\n            # line-endings, docstrings can end up here as bytes instead of\n            # str so make sure that we handle both cases.\n            if (\n                isinstance(node, ast.Constant)\n                and field == \"value\"\n                and isinstance(value, (str, bytes))\n            ):\n                lineend = \"\\n\" if isinstance(value, str) else b\"\\n\"\n                # To normalize, we strip any leading and trailing space from\n                # each line...\n                stripped = [line.strip() for line in value.splitlines()]\n                normalized = lineend.join(stripped)  # type: ignore[attr-defined]\n                # ...and remove any blank lines at the beginning and end of\n                # the whole string\n                normalized = normalized.strip()\n            else:\n                normalized = value\n            yield f\"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}\"\n\n    yield f\"{'  ' * depth})  # /{node.__class__.__name__}\"\n\n\ndef fixup_ast_constants(\n    node: Union[ast.AST, ast3.AST, ast27.AST]\n) -> Union[ast.AST, ast3.AST, ast27.AST]:\n    \"\"\"Map ast nodes deprecated in 3.8 to Constant.\"\"\"\n    if isinstance(node, (ast.Str, ast3.Str, ast27.Str, ast.Bytes, ast3.Bytes)):\n        return ast.Constant(value=node.s)\n\n    if isinstance(node, (ast.Num, ast3.Num, ast27.Num)):\n        return ast.Constant(value=node.n)\n\n    if isinstance(node, (ast.NameConstant, ast3.NameConstant)):\n        return ast.Constant(value=node.value)\n\n    return node\n", "path": "src/black/parsing.py"}], "after_files": [{"content": "\"\"\"\nParse Python code and perform AST validation.\n\"\"\"\nimport ast\nimport sys\nfrom typing import Iterable, Iterator, List, Set, Union, Tuple\n\n# lib2to3 fork\nfrom blib2to3.pytree import Node, Leaf\nfrom blib2to3 import pygram, pytree\nfrom blib2to3.pgen2 import driver\nfrom blib2to3.pgen2.grammar import Grammar\nfrom blib2to3.pgen2.parse import ParseError\n\nfrom black.mode import TargetVersion, Feature, supports_feature\nfrom black.nodes import syms\n\ntry:\n    from typed_ast import ast3, ast27\nexcept ImportError:\n    if sys.version_info < (3, 8):\n        print(\n            \"The typed_ast package is required but not installed.\\n\"\n            \"You can upgrade to Python 3.8+ or install typed_ast with\\n\"\n            \"`python3 -m pip install typed-ast`.\",\n            file=sys.stderr,\n        )\n        sys.exit(1)\n    else:\n        ast3 = ast27 = ast\n\n\nclass InvalidInput(ValueError):\n    \"\"\"Raised when input source code fails all parse attempts.\"\"\"\n\n\ndef get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:\n    if not target_versions:\n        # No target_version specified, so try all grammars.\n        return [\n            # Python 3.7+\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords,\n            # Python 3.0-3.6\n            pygram.python_grammar_no_print_statement_no_exec_statement,\n            # Python 2.7 with future print_function import\n            pygram.python_grammar_no_print_statement,\n            # Python 2.7\n            pygram.python_grammar,\n        ]\n\n    if all(version.is_python2() for version in target_versions):\n        # Python 2-only code, so try Python 2 grammars.\n        return [\n            # Python 2.7 with future print_function import\n            pygram.python_grammar_no_print_statement,\n            # Python 2.7\n            pygram.python_grammar,\n        ]\n\n    # Python 3-compatible code, so only try Python 3 grammar.\n    grammars = []\n    # If we have to parse both, try to parse async as a keyword first\n    if not supports_feature(target_versions, Feature.ASYNC_IDENTIFIERS):\n        # Python 3.7+\n        grammars.append(\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords\n        )\n    if not supports_feature(target_versions, Feature.ASYNC_KEYWORDS):\n        # Python 3.0-3.6\n        grammars.append(pygram.python_grammar_no_print_statement_no_exec_statement)\n    # At least one of the above branches must have been taken, because every Python\n    # version has exactly one of the two 'ASYNC_*' flags\n    return grammars\n\n\ndef lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:\n    \"\"\"Given a string with source, return the lib2to3 Node.\"\"\"\n    if not src_txt.endswith(\"\\n\"):\n        src_txt += \"\\n\"\n\n    for grammar in get_grammars(set(target_versions)):\n        drv = driver.Driver(grammar, pytree.convert)\n        try:\n            result = drv.parse_string(src_txt, True)\n            break\n\n        except ParseError as pe:\n            lineno, column = pe.context[1]\n            lines = src_txt.splitlines()\n            try:\n                faulty_line = lines[lineno - 1]\n            except IndexError:\n                faulty_line = \"<line number missing in source>\"\n            exc = InvalidInput(f\"Cannot parse: {lineno}:{column}: {faulty_line}\")\n    else:\n        raise exc from None\n\n    if isinstance(result, Leaf):\n        result = Node(syms.file_input, [result])\n    return result\n\n\ndef lib2to3_unparse(node: Node) -> str:\n    \"\"\"Given a lib2to3 node, return its string representation.\"\"\"\n    code = str(node)\n    return code\n\n\ndef parse_single_version(\n    src: str, version: Tuple[int, int]\n) -> Union[ast.AST, ast3.AST, ast27.AST]:\n    filename = \"<unknown>\"\n    # typed_ast is needed because of feature version limitations in the builtin ast\n    if sys.version_info >= (3, 8) and version >= (3,):\n        return ast.parse(src, filename, feature_version=version)\n    elif version >= (3,):\n        return ast3.parse(src, filename, feature_version=version[1])\n    elif version == (2, 7):\n        return ast27.parse(src)\n    raise AssertionError(\"INTERNAL ERROR: Tried parsing unsupported Python version!\")\n\n\ndef parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:\n    # TODO: support Python 4+ ;)\n    versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]\n\n    if ast27.__name__ != \"ast\":\n        versions.append((2, 7))\n\n    first_error = \"\"\n    for version in sorted(versions, reverse=True):\n        try:\n            return parse_single_version(src, version)\n        except SyntaxError as e:\n            if not first_error:\n                first_error = str(e)\n\n    raise SyntaxError(first_error)\n\n\ndef stringify_ast(\n    node: Union[ast.AST, ast3.AST, ast27.AST], depth: int = 0\n) -> Iterator[str]:\n    \"\"\"Simple visitor generating strings to compare ASTs by content.\"\"\"\n\n    node = fixup_ast_constants(node)\n\n    yield f\"{'  ' * depth}{node.__class__.__name__}(\"\n\n    for field in sorted(node._fields):  # noqa: F402\n        # TypeIgnore has only one field 'lineno' which breaks this comparison\n        type_ignore_classes = (ast3.TypeIgnore, ast27.TypeIgnore)\n        if sys.version_info >= (3, 8):\n            type_ignore_classes += (ast.TypeIgnore,)\n        if isinstance(node, type_ignore_classes):\n            break\n\n        try:\n            value = getattr(node, field)\n        except AttributeError:\n            continue\n\n        yield f\"{'  ' * (depth+1)}{field}=\"\n\n        if isinstance(value, list):\n            for item in value:\n                # Ignore nested tuples within del statements, because we may insert\n                # parentheses and they change the AST.\n                if (\n                    field == \"targets\"\n                    and isinstance(node, (ast.Delete, ast3.Delete, ast27.Delete))\n                    and isinstance(item, (ast.Tuple, ast3.Tuple, ast27.Tuple))\n                ):\n                    for item in item.elts:\n                        yield from stringify_ast(item, depth + 2)\n\n                elif isinstance(item, (ast.AST, ast3.AST, ast27.AST)):\n                    yield from stringify_ast(item, depth + 2)\n\n        elif isinstance(value, (ast.AST, ast3.AST, ast27.AST)):\n            yield from stringify_ast(value, depth + 2)\n\n        else:\n            # Constant strings may be indented across newlines, if they are\n            # docstrings; fold spaces after newlines when comparing. Similarly,\n            # trailing and leading space may be removed.\n            # Note that when formatting Python 2 code, at least with Windows\n            # line-endings, docstrings can end up here as bytes instead of\n            # str so make sure that we handle both cases.\n            if (\n                isinstance(node, ast.Constant)\n                and field == \"value\"\n                and isinstance(value, (str, bytes))\n            ):\n                lineend = \"\\n\" if isinstance(value, str) else b\"\\n\"\n                # To normalize, we strip any leading and trailing space from\n                # each line...\n                stripped = [line.strip() for line in value.splitlines()]\n                normalized = lineend.join(stripped)  # type: ignore[attr-defined]\n                # ...and remove any blank lines at the beginning and end of\n                # the whole string\n                normalized = normalized.strip()\n            else:\n                normalized = value\n            yield f\"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}\"\n\n    yield f\"{'  ' * depth})  # /{node.__class__.__name__}\"\n\n\ndef fixup_ast_constants(\n    node: Union[ast.AST, ast3.AST, ast27.AST]\n) -> Union[ast.AST, ast3.AST, ast27.AST]:\n    \"\"\"Map ast nodes deprecated in 3.8 to Constant.\"\"\"\n    if isinstance(node, (ast.Str, ast3.Str, ast27.Str, ast.Bytes, ast3.Bytes)):\n        return ast.Constant(value=node.s)\n\n    if isinstance(node, (ast.Num, ast3.Num, ast27.Num)):\n        return ast.Constant(value=node.n)\n\n    if isinstance(node, (ast.NameConstant, ast3.NameConstant)):\n        return ast.Constant(value=node.value)\n\n    return node\n", "path": "src/black/parsing.py"}]}