We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Warning: Unable to determine whether or not callable is a generator with a return value [unable to parse decorated class methods]
### Description

If you have a decorated spider method, `scrapy.utils.misc` [throws a warning](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/misc.py#L240-L264) saying that it cannot determine if the callable is a generator with a return value.

`ast.parse()` fails [here](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/misc.py#L228-L230) when called with a decorated method.

### Steps to Reproduce

I just copied the logic from `misc.py` and used it to analyze a class with the same overall code structure:

```python
import re
import ast
import inspect

class Foo:
    @classmethod
    def func(self):
        """
        Description of func
        """  
        return

code = re.sub(r"^[\t ]+", "", inspect.getsource(Foo.func))
tree = ast.parse(code)
```

```
python3 test.py
> IndentationError: unexpected indent
```

The regex replacement isn't accounting for a possible decorator, so the code ends up looking like:

```
@classmethod
    def func(self):
        """
        Description of func
        """  
        return
```

**Expected behavior:** I'd like to be able to use decorated methods without dealing with noisy logs.

**Actual behavior:** My container logs are filled with tons of warning messages. The only workaround is to avoid the usage of decorators.

**Reproduces how often:** 100% of the time to my knowledge

### Versions

```
$ scrapy version --verbose

Scrapy       : 2.5.1
lxml         : 4.6.4.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.8.2 (default, Dec 21 2020, 15:06:04) - [Clang 12.0.0 (clang-1200.0.32.29)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-10.15.7-x86_64-i386-64bit
```

### Additional context

This is my first time filing a Scrapy issue. I'm happy to add more context if necessary, and apologies in advance if this has already been discussed elsewhere (fwiw I couldn't find anything).

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `scrapy/utils/misc.py`
Content:
```
1 """Helper functions which don't fit anywhere else"""
2 import ast
3 import inspect
4 import os
5 import re
6 import hashlib
7 import warnings
8 from collections import deque
9 from contextlib import contextmanager
10 from importlib import import_module
11 from pkgutil import iter_modules
12 
13 from w3lib.html import replace_entities
14 
15 from scrapy.utils.datatypes import LocalWeakReferencedCache
16 from scrapy.utils.python import flatten, to_unicode
17 from scrapy.item import Item
18 from scrapy.utils.deprecate import ScrapyDeprecationWarning
19 
20 
21 _ITERABLE_SINGLE_VALUES = dict, Item, str, bytes
22 
23 
24 def arg_to_iter(arg):
25     """Convert an argument to an iterable. The argument can be a None, single
26     value, or an iterable.
27 
28     Exception: if arg is a dict, [arg] will be returned
29     """
30     if arg is None:
31         return []
32     elif not isinstance(arg, _ITERABLE_SINGLE_VALUES) and hasattr(arg, '__iter__'):
33         return arg
34     else:
35         return [arg]
36 
37 
38 def load_object(path):
39     """Load an object given its absolute object path, and return it.
40 
41     The object can be the import path of a class, function, variable or an
42     instance, e.g. 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware'.
43 
44     If ``path`` is not a string, but is a callable object, such as a class or
45     a function, then return it as is.
46     """
47 
48     if not isinstance(path, str):
49         if callable(path):
50             return path
51         else:
52             raise TypeError("Unexpected argument type, expected string "
53                             f"or object, got: {type(path)}")
54 
55     try:
56         dot = path.rindex('.')
57     except ValueError:
58         raise ValueError(f"Error loading object '{path}': not a full path")
59 
60     module, name = path[:dot], path[dot + 1:]
61     mod = import_module(module)
62 
63     try:
64         obj = getattr(mod, name)
65     except AttributeError:
66         raise NameError(f"Module '{module}' doesn't define any object named '{name}'")
67 
68     return obj
69 
70 
71 def walk_modules(path):
72     """Loads a module and all its submodules from the given module path and
73     returns them. If *any* module throws an exception while importing, that
74     exception is thrown back.
75 
76     For example: walk_modules('scrapy.utils')
77     """
78 
79     mods = []
80     mod = import_module(path)
81     mods.append(mod)
82     if hasattr(mod, '__path__'):
83         for _, subpath, ispkg in iter_modules(mod.__path__):
84             fullpath = path + '.' + subpath
85             if ispkg:
86                 mods += walk_modules(fullpath)
87             else:
88                 submod = import_module(fullpath)
89                 mods.append(submod)
90     return mods
91 
92 
93 def extract_regex(regex, text, encoding='utf-8'):
94     """Extract a list of unicode strings from the given text/encoding using the following policies:
95 
96     * if the regex contains a named group called "extract" that will be returned
97     * if the regex contains multiple numbered groups, all those will be returned (flattened)
98     * if the regex doesn't contain any group the entire regex matching is returned
99     """
100     warnings.warn(
101         "scrapy.utils.misc.extract_regex has moved to parsel.utils.extract_regex.",
102         ScrapyDeprecationWarning,
103         stacklevel=2
104     )
105 
106     if isinstance(regex, str):
107         regex = re.compile(regex, re.UNICODE)
108 
109     try:
110         strings = [regex.search(text).group('extract')]   # named group
111     except Exception:
112         strings = regex.findall(text)    # full regex or numbered groups
113     strings = flatten(strings)
114 
115     if isinstance(text, str):
116         return [replace_entities(s, keep=['lt', 'amp']) for s in strings]
117     else:
118         return [replace_entities(to_unicode(s, encoding), keep=['lt', 'amp'])
119                 for s in strings]
120 
121 
122 def md5sum(file):
123     """Calculate the md5 checksum of a file-like object without reading its
124     whole content in memory.
125 
126     >>> from io import BytesIO
127     >>> md5sum(BytesIO(b'file content to hash'))
128     '784406af91dd5a54fbb9c84c2236595a'
129     """
130     m = hashlib.md5()
131     while True:
132         d = file.read(8096)
133         if not d:
134             break
135         m.update(d)
136     return m.hexdigest()
137 
138 
139 def rel_has_nofollow(rel):
140     """Return True if link rel attribute has nofollow type"""
141     return rel is not None and 'nofollow' in rel.replace(',', ' ').split()
142 
143 
144 def create_instance(objcls, settings, crawler, *args, **kwargs):
145     """Construct a class instance using its ``from_crawler`` or
146     ``from_settings`` constructors, if available.
147 
148     At least one of ``settings`` and ``crawler`` needs to be different from
149     ``None``. If ``settings `` is ``None``, ``crawler.settings`` will be used.
150     If ``crawler`` is ``None``, only the ``from_settings`` constructor will be
151     tried.
152 
153     ``*args`` and ``**kwargs`` are forwarded to the constructors.
154 
155     Raises ``ValueError`` if both ``settings`` and ``crawler`` are ``None``.
156 
157     .. versionchanged:: 2.2
158        Raises ``TypeError`` if the resulting instance is ``None`` (e.g. if an
159        extension has not been implemented correctly).
160     """
161     if settings is None:
162         if crawler is None:
163             raise ValueError("Specify at least one of settings and crawler.")
164         settings = crawler.settings
165     if crawler and hasattr(objcls, 'from_crawler'):
166         instance = objcls.from_crawler(crawler, *args, **kwargs)
167         method_name = 'from_crawler'
168     elif hasattr(objcls, 'from_settings'):
169         instance = objcls.from_settings(settings, *args, **kwargs)
170         method_name = 'from_settings'
171     else:
172         instance = objcls(*args, **kwargs)
173         method_name = '__new__'
174     if instance is None:
175         raise TypeError(f"{objcls.__qualname__}.{method_name} returned None")
176     return instance
177 
178 
179 @contextmanager
180 def set_environ(**kwargs):
181     """Temporarily set environment variables inside the context manager and
182     fully restore previous environment afterwards
183     """
184 
185     original_env = {k: os.environ.get(k) for k in kwargs}
186     os.environ.update(kwargs)
187     try:
188         yield
189     finally:
190         for k, v in original_env.items():
191             if v is None:
192                 del os.environ[k]
193             else:
194                 os.environ[k] = v
195 
196 
197 def walk_callable(node):
198     """Similar to ``ast.walk``, but walks only function body and skips nested
199     functions defined within the node.
200     """
201     todo = deque([node])
202     walked_func_def = False
203     while todo:
204         node = todo.popleft()
205         if isinstance(node, ast.FunctionDef):
206             if walked_func_def:
207                 continue
208             walked_func_def = True
209         todo.extend(ast.iter_child_nodes(node))
210         yield node
211 
212 
213 _generator_callbacks_cache = LocalWeakReferencedCache(limit=128)
214 
215 
216 def is_generator_with_return_value(callable):
217     """
218     Returns True if a callable is a generator function which includes a
219     'return' statement with a value different than None, False otherwise
220     """
221     if callable in _generator_callbacks_cache:
222         return _generator_callbacks_cache[callable]
223 
224     def returns_none(return_node):
225         value = return_node.value
226         return value is None or isinstance(value, ast.NameConstant) and value.value is None
227 
228     if inspect.isgeneratorfunction(callable):
229         code = re.sub(r"^[\t ]+", "", inspect.getsource(callable))
230         tree = ast.parse(code)
231         for node in walk_callable(tree):
232             if isinstance(node, ast.Return) and not returns_none(node):
233                 _generator_callbacks_cache[callable] = True
234                 return _generator_callbacks_cache[callable]
235 
236     _generator_callbacks_cache[callable] = False
237     return _generator_callbacks_cache[callable]
238 
239 
240 def warn_on_generator_with_return_value(spider, callable):
241     """
242     Logs a warning if a callable is a generator function and includes
243     a 'return' statement with a value different than None
244     """
245     try:
246         if is_generator_with_return_value(callable):
247             warnings.warn(
248                 f'The "{spider.__class__.__name__}.{callable.__name__}" method is '
249                 'a generator and includes a "return" statement with a value '
250                 'different than None. This could lead to unexpected behaviour. Please see '
251                 'https://docs.python.org/3/reference/simple_stmts.html#the-return-statement '
252                 'for details about the semantics of the "return" statement within generators',
253                 stacklevel=2,
254             )
255     except IndentationError:
256         callable_name = spider.__class__.__name__ + "." + callable.__name__
257         warnings.warn(
258             f'Unable to determine whether or not "{callable_name}" is a generator with a return value. '
259             'This will not prevent your code from working, but it prevents Scrapy from detecting '
260             f'potential issues in your implementation of "{callable_name}". Please, report this in the '
261             'Scrapy issue tracker (https://github.com/scrapy/scrapy/issues), '
262             f'including the code of "{callable_name}"',
263             stacklevel=2,
264         )
265
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -226,7 +226,14 @@
         return value is None or isinstance(value, ast.NameConstant) and value.value is None
 
     if inspect.isgeneratorfunction(callable):
-        code = re.sub(r"^[\t ]+", "", inspect.getsource(callable))
+        src = inspect.getsource(callable)
+        pattern = re.compile(r"(^[\t ]+)")
+        code = pattern.sub("", src)
+
+        match = pattern.match(src)  # finds indentation
+        if match:
+            code = re.sub(f"\n{match.group(0)}", "\n", code)  # remove indentation
+
         tree = ast.parse(code)
         for node in walk_callable(tree):
             if isinstance(node, ast.Return) and not returns_none(node):

{"golden_diff": "diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py\n--- a/scrapy/utils/misc.py\n+++ b/scrapy/utils/misc.py\n@@ -226,7 +226,14 @@\n         return value is None or isinstance(value, ast.NameConstant) and value.value is None\n \n     if inspect.isgeneratorfunction(callable):\n-        code = re.sub(r\"^[\\t ]+\", \"\", inspect.getsource(callable))\n+        src = inspect.getsource(callable)\n+        pattern = re.compile(r\"(^[\\t ]+)\")\n+        code = pattern.sub(\"\", src)\n+\n+        match = pattern.match(src)  # finds indentation\n+        if match:\n+            code = re.sub(f\"\\n{match.group(0)}\", \"\\n\", code)  # remove indentation\n+\n         tree = ast.parse(code)\n         for node in walk_callable(tree):\n             if isinstance(node, ast.Return) and not returns_none(node):\n", "issue": "Warning: Unable to determine whether or not callable is a generator with a return value [unable to parse decorated class methods]\n### Description\r\n\r\nIf you have a decorated spider method, `scrapy.utils.misc` [throws a warning](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/misc.py#L240-L264) saying that it cannot determine if the callable is a generator with a return value.\r\n\r\n`ast.parse()` fails [here](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/misc.py#L228-L230) when called with a decorated method.\r\n\r\n### Steps to Reproduce\r\n\r\nI just copied the logic from `misc.py` and used it to analyze a class with the same overall code structure:\r\n\r\n```python\r\nimport re\r\nimport ast\r\nimport inspect\r\n\r\nclass Foo:\r\n    @classmethod\r\n    def func(self):\r\n        \"\"\"\r\n        Description of func\r\n        \"\"\"  \r\n        return\r\n\r\ncode = re.sub(r\"^[\\t ]+\", \"\", inspect.getsource(Foo.func))\r\ntree = ast.parse(code)\r\n```\r\n\r\n```\r\npython3 test.py\r\n> IndentationError: unexpected indent\r\n```\r\n\r\nThe regex replacement isn't accounting for a possible decorator, so the code ends up looking like:\r\n\r\n```\r\n@classmethod\r\n    def func(self):\r\n        \"\"\"\r\n        Description of func\r\n        \"\"\"  \r\n        return\r\n```\r\n\r\n**Expected behavior:** I'd like to be able to use decorated methods without dealing with noisy logs.\r\n\r\n**Actual behavior:** My container logs are filled with tons of warning messages. The only workaround is to avoid the usage of decorators.\r\n\r\n**Reproduces how often:** 100% of the time to my knowledge\r\n\r\n### Versions\r\n\r\n```\r\n$ scrapy version --verbose\r\n\r\nScrapy       : 2.5.1\r\nlxml         : 4.6.4.0\r\nlibxml2      : 2.9.10\r\ncssselect    : 1.1.0\r\nparsel       : 1.6.0\r\nw3lib        : 1.22.0\r\nTwisted      : 21.7.0\r\nPython       : 3.8.2 (default, Dec 21 2020, 15:06:04) - [Clang 12.0.0 (clang-1200.0.32.29)]\r\npyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)\r\ncryptography : 35.0.0\r\nPlatform     : macOS-10.15.7-x86_64-i386-64bit\r\n```\r\n\r\n### Additional context\r\n\r\nThis is my first time filing a Scrapy issue. I'm happy to add more context if necessary, and apologies in advance if this has already been discussed elsewhere (fwiw I couldn't find anything).\n", "before_files": [{"content": "\"\"\"Helper functions which don't fit anywhere else\"\"\"\nimport ast\nimport inspect\nimport os\nimport re\nimport hashlib\nimport warnings\nfrom collections import deque\nfrom contextlib import contextmanager\nfrom importlib import import_module\nfrom pkgutil import iter_modules\n\nfrom w3lib.html import replace_entities\n\nfrom scrapy.utils.datatypes import LocalWeakReferencedCache\nfrom scrapy.utils.python import flatten, to_unicode\nfrom scrapy.item import Item\nfrom scrapy.utils.deprecate import ScrapyDeprecationWarning\n\n\n_ITERABLE_SINGLE_VALUES = dict, Item, str, bytes\n\n\ndef arg_to_iter(arg):\n    \"\"\"Convert an argument to an iterable. The argument can be a None, single\n    value, or an iterable.\n\n    Exception: if arg is a dict, [arg] will be returned\n    \"\"\"\n    if arg is None:\n        return []\n    elif not isinstance(arg, _ITERABLE_SINGLE_VALUES) and hasattr(arg, '__iter__'):\n        return arg\n    else:\n        return [arg]\n\n\ndef load_object(path):\n    \"\"\"Load an object given its absolute object path, and return it.\n\n    The object can be the import path of a class, function, variable or an\n    instance, e.g. 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware'.\n\n    If ``path`` is not a string, but is a callable object, such as a class or\n    a function, then return it as is.\n    \"\"\"\n\n    if not isinstance(path, str):\n        if callable(path):\n            return path\n        else:\n            raise TypeError(\"Unexpected argument type, expected string \"\n                            f\"or object, got: {type(path)}\")\n\n    try:\n        dot = path.rindex('.')\n    except ValueError:\n        raise ValueError(f\"Error loading object '{path}': not a full path\")\n\n    module, name = path[:dot], path[dot + 1:]\n    mod = import_module(module)\n\n    try:\n        obj = getattr(mod, name)\n    except AttributeError:\n        raise NameError(f\"Module '{module}' doesn't define any object named '{name}'\")\n\n    return obj\n\n\ndef walk_modules(path):\n    \"\"\"Loads a module and all its submodules from the given module path and\n    returns them. If *any* module throws an exception while importing, that\n    exception is thrown back.\n\n    For example: walk_modules('scrapy.utils')\n    \"\"\"\n\n    mods = []\n    mod = import_module(path)\n    mods.append(mod)\n    if hasattr(mod, '__path__'):\n        for _, subpath, ispkg in iter_modules(mod.__path__):\n            fullpath = path + '.' + subpath\n            if ispkg:\n                mods += walk_modules(fullpath)\n            else:\n                submod = import_module(fullpath)\n                mods.append(submod)\n    return mods\n\n\ndef extract_regex(regex, text, encoding='utf-8'):\n    \"\"\"Extract a list of unicode strings from the given text/encoding using the following policies:\n\n    * if the regex contains a named group called \"extract\" that will be returned\n    * if the regex contains multiple numbered groups, all those will be returned (flattened)\n    * if the regex doesn't contain any group the entire regex matching is returned\n    \"\"\"\n    warnings.warn(\n        \"scrapy.utils.misc.extract_regex has moved to parsel.utils.extract_regex.\",\n        ScrapyDeprecationWarning,\n        stacklevel=2\n    )\n\n    if isinstance(regex, str):\n        regex = re.compile(regex, re.UNICODE)\n\n    try:\n        strings = [regex.search(text).group('extract')]   # named group\n    except Exception:\n        strings = regex.findall(text)    # full regex or numbered groups\n    strings = flatten(strings)\n\n    if isinstance(text, str):\n        return [replace_entities(s, keep=['lt', 'amp']) for s in strings]\n    else:\n        return [replace_entities(to_unicode(s, encoding), keep=['lt', 'amp'])\n                for s in strings]\n\n\ndef md5sum(file):\n    \"\"\"Calculate the md5 checksum of a file-like object without reading its\n    whole content in memory.\n\n    >>> from io import BytesIO\n    >>> md5sum(BytesIO(b'file content to hash'))\n    '784406af91dd5a54fbb9c84c2236595a'\n    \"\"\"\n    m = hashlib.md5()\n    while True:\n        d = file.read(8096)\n        if not d:\n            break\n        m.update(d)\n    return m.hexdigest()\n\n\ndef rel_has_nofollow(rel):\n    \"\"\"Return True if link rel attribute has nofollow type\"\"\"\n    return rel is not None and 'nofollow' in rel.replace(',', ' ').split()\n\n\ndef create_instance(objcls, settings, crawler, *args, **kwargs):\n    \"\"\"Construct a class instance using its ``from_crawler`` or\n    ``from_settings`` constructors, if available.\n\n    At least one of ``settings`` and ``crawler`` needs to be different from\n    ``None``. If ``settings `` is ``None``, ``crawler.settings`` will be used.\n    If ``crawler`` is ``None``, only the ``from_settings`` constructor will be\n    tried.\n\n    ``*args`` and ``**kwargs`` are forwarded to the constructors.\n\n    Raises ``ValueError`` if both ``settings`` and ``crawler`` are ``None``.\n\n    .. versionchanged:: 2.2\n       Raises ``TypeError`` if the resulting instance is ``None`` (e.g. if an\n       extension has not been implemented correctly).\n    \"\"\"\n    if settings is None:\n        if crawler is None:\n            raise ValueError(\"Specify at least one of settings and crawler.\")\n        settings = crawler.settings\n    if crawler and hasattr(objcls, 'from_crawler'):\n        instance = objcls.from_crawler(crawler, *args, **kwargs)\n        method_name = 'from_crawler'\n    elif hasattr(objcls, 'from_settings'):\n        instance = objcls.from_settings(settings, *args, **kwargs)\n        method_name = 'from_settings'\n    else:\n        instance = objcls(*args, **kwargs)\n        method_name = '__new__'\n    if instance is None:\n        raise TypeError(f\"{objcls.__qualname__}.{method_name} returned None\")\n    return instance\n\n\n@contextmanager\ndef set_environ(**kwargs):\n    \"\"\"Temporarily set environment variables inside the context manager and\n    fully restore previous environment afterwards\n    \"\"\"\n\n    original_env = {k: os.environ.get(k) for k in kwargs}\n    os.environ.update(kwargs)\n    try:\n        yield\n    finally:\n        for k, v in original_env.items():\n            if v is None:\n                del os.environ[k]\n            else:\n                os.environ[k] = v\n\n\ndef walk_callable(node):\n    \"\"\"Similar to ``ast.walk``, but walks only function body and skips nested\n    functions defined within the node.\n    \"\"\"\n    todo = deque([node])\n    walked_func_def = False\n    while todo:\n        node = todo.popleft()\n        if isinstance(node, ast.FunctionDef):\n            if walked_func_def:\n                continue\n            walked_func_def = True\n        todo.extend(ast.iter_child_nodes(node))\n        yield node\n\n\n_generator_callbacks_cache = LocalWeakReferencedCache(limit=128)\n\n\ndef is_generator_with_return_value(callable):\n    \"\"\"\n    Returns True if a callable is a generator function which includes a\n    'return' statement with a value different than None, False otherwise\n    \"\"\"\n    if callable in _generator_callbacks_cache:\n        return _generator_callbacks_cache[callable]\n\n    def returns_none(return_node):\n        value = return_node.value\n        return value is None or isinstance(value, ast.NameConstant) and value.value is None\n\n    if inspect.isgeneratorfunction(callable):\n        code = re.sub(r\"^[\\t ]+\", \"\", inspect.getsource(callable))\n        tree = ast.parse(code)\n        for node in walk_callable(tree):\n            if isinstance(node, ast.Return) and not returns_none(node):\n                _generator_callbacks_cache[callable] = True\n                return _generator_callbacks_cache[callable]\n\n    _generator_callbacks_cache[callable] = False\n    return _generator_callbacks_cache[callable]\n\n\ndef warn_on_generator_with_return_value(spider, callable):\n    \"\"\"\n    Logs a warning if a callable is a generator function and includes\n    a 'return' statement with a value different than None\n    \"\"\"\n    try:\n        if is_generator_with_return_value(callable):\n            warnings.warn(\n                f'The \"{spider.__class__.__name__}.{callable.__name__}\" method is '\n                'a generator and includes a \"return\" statement with a value '\n                'different than None. This could lead to unexpected behaviour. Please see '\n                'https://docs.python.org/3/reference/simple_stmts.html#the-return-statement '\n                'for details about the semantics of the \"return\" statement within generators',\n                stacklevel=2,\n            )\n    except IndentationError:\n        callable_name = spider.__class__.__name__ + \".\" + callable.__name__\n        warnings.warn(\n            f'Unable to determine whether or not \"{callable_name}\" is a generator with a return value. '\n            'This will not prevent your code from working, but it prevents Scrapy from detecting '\n            f'potential issues in your implementation of \"{callable_name}\". Please, report this in the '\n            'Scrapy issue tracker (https://github.com/scrapy/scrapy/issues), '\n            f'including the code of \"{callable_name}\"',\n            stacklevel=2,\n        )\n", "path": "scrapy/utils/misc.py"}], "after_files": [{"content": "\"\"\"Helper functions which don't fit anywhere else\"\"\"\nimport ast\nimport inspect\nimport os\nimport re\nimport hashlib\nimport warnings\nfrom collections import deque\nfrom contextlib import contextmanager\nfrom importlib import import_module\nfrom pkgutil import iter_modules\n\nfrom w3lib.html import replace_entities\n\nfrom scrapy.utils.datatypes import LocalWeakReferencedCache\nfrom scrapy.utils.python import flatten, to_unicode\nfrom scrapy.item import Item\nfrom scrapy.utils.deprecate import ScrapyDeprecationWarning\n\n\n_ITERABLE_SINGLE_VALUES = dict, Item, str, bytes\n\n\ndef arg_to_iter(arg):\n    \"\"\"Convert an argument to an iterable. The argument can be a None, single\n    value, or an iterable.\n\n    Exception: if arg is a dict, [arg] will be returned\n    \"\"\"\n    if arg is None:\n        return []\n    elif not isinstance(arg, _ITERABLE_SINGLE_VALUES) and hasattr(arg, '__iter__'):\n        return arg\n    else:\n        return [arg]\n\n\ndef load_object(path):\n    \"\"\"Load an object given its absolute object path, and return it.\n\n    The object can be the import path of a class, function, variable or an\n    instance, e.g. 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware'.\n\n    If ``path`` is not a string, but is a callable object, such as a class or\n    a function, then return it as is.\n    \"\"\"\n\n    if not isinstance(path, str):\n        if callable(path):\n            return path\n        else:\n            raise TypeError(\"Unexpected argument type, expected string \"\n                            f\"or object, got: {type(path)}\")\n\n    try:\n        dot = path.rindex('.')\n    except ValueError:\n        raise ValueError(f\"Error loading object '{path}': not a full path\")\n\n    module, name = path[:dot], path[dot + 1:]\n    mod = import_module(module)\n\n    try:\n        obj = getattr(mod, name)\n    except AttributeError:\n        raise NameError(f\"Module '{module}' doesn't define any object named '{name}'\")\n\n    return obj\n\n\ndef walk_modules(path):\n    \"\"\"Loads a module and all its submodules from the given module path and\n    returns them. If *any* module throws an exception while importing, that\n    exception is thrown back.\n\n    For example: walk_modules('scrapy.utils')\n    \"\"\"\n\n    mods = []\n    mod = import_module(path)\n    mods.append(mod)\n    if hasattr(mod, '__path__'):\n        for _, subpath, ispkg in iter_modules(mod.__path__):\n            fullpath = path + '.' + subpath\n            if ispkg:\n                mods += walk_modules(fullpath)\n            else:\n                submod = import_module(fullpath)\n                mods.append(submod)\n    return mods\n\n\ndef extract_regex(regex, text, encoding='utf-8'):\n    \"\"\"Extract a list of unicode strings from the given text/encoding using the following policies:\n\n    * if the regex contains a named group called \"extract\" that will be returned\n    * if the regex contains multiple numbered groups, all those will be returned (flattened)\n    * if the regex doesn't contain any group the entire regex matching is returned\n    \"\"\"\n    warnings.warn(\n        \"scrapy.utils.misc.extract_regex has moved to parsel.utils.extract_regex.\",\n        ScrapyDeprecationWarning,\n        stacklevel=2\n    )\n\n    if isinstance(regex, str):\n        regex = re.compile(regex, re.UNICODE)\n\n    try:\n        strings = [regex.search(text).group('extract')]   # named group\n    except Exception:\n        strings = regex.findall(text)    # full regex or numbered groups\n    strings = flatten(strings)\n\n    if isinstance(text, str):\n        return [replace_entities(s, keep=['lt', 'amp']) for s in strings]\n    else:\n        return [replace_entities(to_unicode(s, encoding), keep=['lt', 'amp'])\n                for s in strings]\n\n\ndef md5sum(file):\n    \"\"\"Calculate the md5 checksum of a file-like object without reading its\n    whole content in memory.\n\n    >>> from io import BytesIO\n    >>> md5sum(BytesIO(b'file content to hash'))\n    '784406af91dd5a54fbb9c84c2236595a'\n    \"\"\"\n    m = hashlib.md5()\n    while True:\n        d = file.read(8096)\n        if not d:\n            break\n        m.update(d)\n    return m.hexdigest()\n\n\ndef rel_has_nofollow(rel):\n    \"\"\"Return True if link rel attribute has nofollow type\"\"\"\n    return rel is not None and 'nofollow' in rel.replace(',', ' ').split()\n\n\ndef create_instance(objcls, settings, crawler, *args, **kwargs):\n    \"\"\"Construct a class instance using its ``from_crawler`` or\n    ``from_settings`` constructors, if available.\n\n    At least one of ``settings`` and ``crawler`` needs to be different from\n    ``None``. If ``settings `` is ``None``, ``crawler.settings`` will be used.\n    If ``crawler`` is ``None``, only the ``from_settings`` constructor will be\n    tried.\n\n    ``*args`` and ``**kwargs`` are forwarded to the constructors.\n\n    Raises ``ValueError`` if both ``settings`` and ``crawler`` are ``None``.\n\n    .. versionchanged:: 2.2\n       Raises ``TypeError`` if the resulting instance is ``None`` (e.g. if an\n       extension has not been implemented correctly).\n    \"\"\"\n    if settings is None:\n        if crawler is None:\n            raise ValueError(\"Specify at least one of settings and crawler.\")\n        settings = crawler.settings\n    if crawler and hasattr(objcls, 'from_crawler'):\n        instance = objcls.from_crawler(crawler, *args, **kwargs)\n        method_name = 'from_crawler'\n    elif hasattr(objcls, 'from_settings'):\n        instance = objcls.from_settings(settings, *args, **kwargs)\n        method_name = 'from_settings'\n    else:\n        instance = objcls(*args, **kwargs)\n        method_name = '__new__'\n    if instance is None:\n        raise TypeError(f\"{objcls.__qualname__}.{method_name} returned None\")\n    return instance\n\n\n@contextmanager\ndef set_environ(**kwargs):\n    \"\"\"Temporarily set environment variables inside the context manager and\n    fully restore previous environment afterwards\n    \"\"\"\n\n    original_env = {k: os.environ.get(k) for k in kwargs}\n    os.environ.update(kwargs)\n    try:\n        yield\n    finally:\n        for k, v in original_env.items():\n            if v is None:\n                del os.environ[k]\n            else:\n                os.environ[k] = v\n\n\ndef walk_callable(node):\n    \"\"\"Similar to ``ast.walk``, but walks only function body and skips nested\n    functions defined within the node.\n    \"\"\"\n    todo = deque([node])\n    walked_func_def = False\n    while todo:\n        node = todo.popleft()\n        if isinstance(node, ast.FunctionDef):\n            if walked_func_def:\n                continue\n            walked_func_def = True\n        todo.extend(ast.iter_child_nodes(node))\n        yield node\n\n\n_generator_callbacks_cache = LocalWeakReferencedCache(limit=128)\n\n\ndef is_generator_with_return_value(callable):\n    \"\"\"\n    Returns True if a callable is a generator function which includes a\n    'return' statement with a value different than None, False otherwise\n    \"\"\"\n    if callable in _generator_callbacks_cache:\n        return _generator_callbacks_cache[callable]\n\n    def returns_none(return_node):\n        value = return_node.value\n        return value is None or isinstance(value, ast.NameConstant) and value.value is None\n\n    if inspect.isgeneratorfunction(callable):\n        src = inspect.getsource(callable)\n        pattern = re.compile(r\"(^[\\t ]+)\")\n        code = pattern.sub(\"\", src)\n\n        match = pattern.match(src)  # finds indentation\n        if match:\n            code = re.sub(f\"\\n{match.group(0)}\", \"\\n\", code)  # remove indentation\n\n        tree = ast.parse(code)\n        for node in walk_callable(tree):\n            if isinstance(node, ast.Return) and not returns_none(node):\n                _generator_callbacks_cache[callable] = True\n                return _generator_callbacks_cache[callable]\n\n    _generator_callbacks_cache[callable] = False\n    return _generator_callbacks_cache[callable]\n\n\ndef warn_on_generator_with_return_value(spider, callable):\n    \"\"\"\n    Logs a warning if a callable is a generator function and includes\n    a 'return' statement with a value different than None\n    \"\"\"\n    try:\n        if is_generator_with_return_value(callable):\n            warnings.warn(\n                f'The \"{spider.__class__.__name__}.{callable.__name__}\" method is '\n                'a generator and includes a \"return\" statement with a value '\n                'different than None. This could lead to unexpected behaviour. Please see '\n                'https://docs.python.org/3/reference/simple_stmts.html#the-return-statement '\n                'for details about the semantics of the \"return\" statement within generators',\n                stacklevel=2,\n            )\n    except IndentationError:\n        callable_name = spider.__class__.__name__ + \".\" + callable.__name__\n        warnings.warn(\n            f'Unable to determine whether or not \"{callable_name}\" is a generator with a return value. '\n            'This will not prevent your code from working, but it prevents Scrapy from detecting '\n            f'potential issues in your implementation of \"{callable_name}\". Please, report this in the '\n            'Scrapy issue tracker (https://github.com/scrapy/scrapy/issues), '\n            f'including the code of \"{callable_name}\"',\n            stacklevel=2,\n        )\n", "path": "scrapy/utils/misc.py"}]}