prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  pkgutil.iter_modules with arbitrary path
  ## Description of the issue
  The iter_modules patch implemented in #5959 has a bug where the path must start with the _MEIPASS or it will throw an assertion error.
  
  The normal iter_modules function can take any valid path. Your code first calls that:
  https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py#L37
  
  and later asserts it starts with _MEIPASS
  https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py#L59
  
  which means that a path outside of the executable will throw the assertion error.
  
  I think when implementing it was overlooked that this function could be used to look at a path outside the executable path.
  
  ### Context information (for bug reports)
  
  * PyInstaller Version 4.8
  * All OS and python versions
  
  I will have a look into creating a pull request to fix this issue.
  I think the solution is to change the assertion to an if statement to only run the code below that if it starts with _MEIPASS and thus could be bundled in the executable.
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py`
  Content:
  ```
  1 #-----------------------------------------------------------------------------
  2 # Copyright (c) 2021, PyInstaller Development Team.
  3 #
  4 # Licensed under the Apache License, Version 2.0 (the "License");
  5 # you may not use this file except in compliance with the License.
  6 #
  7 # The full license is in the file COPYING.txt, distributed with this software.
  8 #
  9 # SPDX-License-Identifier: Apache-2.0
  10 #-----------------------------------------------------------------------------
  11 #
  12 # This rthook overrides pkgutil.iter_modules with custom implementation that uses PyInstaller's FrozenImporter to list
  13 # sub-modules embedded in the PYZ archive. The non-embedded modules (binary extensions, or .pyc modules in noarchive
  14 # build) are handled by original pkgutil iter_modules implementation (and consequently, python's FileFinder).
  15 #
  16 # The preferred way of adding support for iter_modules would be adding non-standard iter_modules() method to
  17 # FrozenImporter itself. However, that seems to work only for path entry finders (for use with sys.path_hooks), while
  18 # PyInstaller's FrozenImporter is registered as meta path finders (for use with sys.meta_path). Turning FrozenImporter
  19 # into path entry finder, would seemingly require the latter to support on-filesystem resources (e.g., extension
  20 # modules) in addition to PYZ-embedded ones.
  21 #
  22 # Therefore, we instead opt for overriding pkgutil.iter_modules with custom implementation that augments the output of
  23 # original implementation with contents of PYZ archive from FrozenImporter's TOC.
  24 
  25 import os
  26 import pkgutil
  27 import sys
  28 
  29 from pyimod03_importers import FrozenImporter
  30 
  31 _orig_pkgutil_iter_modules = pkgutil.iter_modules
  32 
  33 
  34 def _pyi_pkgutil_iter_modules(path=None, prefix=''):
  35     # Use original implementation to discover on-filesystem modules (binary extensions in regular builds, or both binary
  36     # extensions and compiled pyc modules in noarchive debug builds).
  37     yield from _orig_pkgutil_iter_modules(path, prefix)
  38 
  39     # Find the instance of PyInstaller's FrozenImporter.
  40     for importer in pkgutil.iter_importers():
  41         if isinstance(importer, FrozenImporter):
  42             break
  43     else:
  44         return
  45 
  46     if not path:
  47         # Search for all top-level packages/modules. These will have no dots in their entry names.
  48         for entry in importer.toc:
  49             if entry.count('.') != 0:
  50                 continue
  51             is_pkg = importer.is_package(entry)
  52             yield pkgutil.ModuleInfo(importer, prefix + entry, is_pkg)
  53     else:
  54         # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.
  55         SYS_PREFIX = sys._MEIPASS + os.path.sep
  56         SYS_PREFIXLEN = len(SYS_PREFIX)
  57         # Only single path is supported, and it must start with sys._MEIPASS.
  58         pkg_path = os.path.normpath(path[0])
  59         assert pkg_path.startswith(SYS_PREFIX)
  60         # Construct package prefix from path...
  61         pkg_prefix = pkg_path[SYS_PREFIXLEN:]
  62         pkg_prefix = pkg_prefix.replace(os.path.sep, '.')
  63         # ... and ensure it ends with a dot (so we can directly filter out the package itself).
  64         if not pkg_prefix.endswith('.'):
  65             pkg_prefix += '.'
  66         pkg_prefix_len = len(pkg_prefix)
  67 
  68         for entry in importer.toc:
  69             if not entry.startswith(pkg_prefix):
  70                 continue
  71             name = entry[pkg_prefix_len:]
  72             if name.count('.') != 0:
  73                 continue
  74             is_pkg = importer.is_package(entry)
  75             yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)
  76 
  77 
  78 pkgutil.iter_modules = _pyi_pkgutil_iter_modules
  79
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
  --- a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
  +++ b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
  @@ -43,7 +43,7 @@
       else:
           return
   
  -    if not path:
  +    if path is None:
           # Search for all top-level packages/modules. These will have no dots in their entry names.
           for entry in importer.toc:
               if entry.count('.') != 0:
  @@ -54,25 +54,28 @@
           # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.
           SYS_PREFIX = sys._MEIPASS + os.path.sep
           SYS_PREFIXLEN = len(SYS_PREFIX)
  -        # Only single path is supported, and it must start with sys._MEIPASS.
  -        pkg_path = os.path.normpath(path[0])
  -        assert pkg_path.startswith(SYS_PREFIX)
  -        # Construct package prefix from path...
  -        pkg_prefix = pkg_path[SYS_PREFIXLEN:]
  -        pkg_prefix = pkg_prefix.replace(os.path.sep, '.')
  -        # ... and ensure it ends with a dot (so we can directly filter out the package itself).
  -        if not pkg_prefix.endswith('.'):
  -            pkg_prefix += '.'
  -        pkg_prefix_len = len(pkg_prefix)
   
  -        for entry in importer.toc:
  -            if not entry.startswith(pkg_prefix):
  -                continue
  -            name = entry[pkg_prefix_len:]
  -            if name.count('.') != 0:
  +        for pkg_path in path:
  +            pkg_path = os.path.normpath(pkg_path)
  +            if not pkg_path.startswith(SYS_PREFIX):
  +                # if the path does not start with sys._MEIPASS then it cannot be a bundled package.
                   continue
  -            is_pkg = importer.is_package(entry)
  -            yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)
  +            # Construct package prefix from path...
  +            pkg_prefix = pkg_path[SYS_PREFIXLEN:]
  +            pkg_prefix = pkg_prefix.replace(os.path.sep, '.')
  +            # ... and ensure it ends with a dot (so we can directly filter out the package itself).
  +            if not pkg_prefix.endswith('.'):
  +                pkg_prefix += '.'
  +            pkg_prefix_len = len(pkg_prefix)
  +
  +            for entry in importer.toc:
  +                if not entry.startswith(pkg_prefix):
  +                    continue
  +                name = entry[pkg_prefix_len:]
  +                if name.count('.') != 0:
  +                    continue
  +                is_pkg = importer.is_package(entry)
  +                yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)
   
   
   pkgutil.iter_modules = _pyi_pkgutil_iter_modules
  
verification_info: {"golden_diff": "diff --git a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py\n--- a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py\n+++ b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py\n@@ -43,7 +43,7 @@\n     else:\n         return\n \n-    if not path:\n+    if path is None:\n         # Search for all top-level packages/modules. These will have no dots in their entry names.\n         for entry in importer.toc:\n             if entry.count('.') != 0:\n@@ -54,25 +54,28 @@\n         # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.\n         SYS_PREFIX = sys._MEIPASS + os.path.sep\n         SYS_PREFIXLEN = len(SYS_PREFIX)\n-        # Only single path is supported, and it must start with sys._MEIPASS.\n-        pkg_path = os.path.normpath(path[0])\n-        assert pkg_path.startswith(SYS_PREFIX)\n-        # Construct package prefix from path...\n-        pkg_prefix = pkg_path[SYS_PREFIXLEN:]\n-        pkg_prefix = pkg_prefix.replace(os.path.sep, '.')\n-        # ... and ensure it ends with a dot (so we can directly filter out the package itself).\n-        if not pkg_prefix.endswith('.'):\n-            pkg_prefix += '.'\n-        pkg_prefix_len = len(pkg_prefix)\n \n-        for entry in importer.toc:\n-            if not entry.startswith(pkg_prefix):\n-                continue\n-            name = entry[pkg_prefix_len:]\n-            if name.count('.') != 0:\n+        for pkg_path in path:\n+            pkg_path = os.path.normpath(pkg_path)\n+            if not pkg_path.startswith(SYS_PREFIX):\n+                # if the path does not start with sys._MEIPASS then it cannot be a bundled package.\n                 continue\n-            is_pkg = importer.is_package(entry)\n-            yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)\n+            # Construct package prefix from path...\n+            pkg_prefix = pkg_path[SYS_PREFIXLEN:]\n+            pkg_prefix = pkg_prefix.replace(os.path.sep, '.')\n+            # ... and ensure it ends with a dot (so we can directly filter out the package itself).\n+            if not pkg_prefix.endswith('.'):\n+                pkg_prefix += '.'\n+            pkg_prefix_len = len(pkg_prefix)\n+\n+            for entry in importer.toc:\n+                if not entry.startswith(pkg_prefix):\n+                    continue\n+                name = entry[pkg_prefix_len:]\n+                if name.count('.') != 0:\n+                    continue\n+                is_pkg = importer.is_package(entry)\n+                yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)\n \n \n pkgutil.iter_modules = _pyi_pkgutil_iter_modules\n", "issue": "pkgutil.iter_modules with arbitrary path\n## Description of the issue\r\nThe iter_modules patch implemented in #5959 has a bug where the path must start with the _MEIPASS or it will throw an assertion error.\r\n\r\nThe normal iter_modules function can take any valid path. Your code first calls that:\r\nhttps://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py#L37\r\n\r\nand later asserts it starts with _MEIPASS\r\nhttps://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py#L59\r\n\r\nwhich means that a path outside of the executable will throw the assertion error.\r\n\r\nI think when implementing it was overlooked that this function could be used to look at a path outside the executable path.\r\n\r\n### Context information (for bug reports)\r\n\r\n* PyInstaller Version 4.8\r\n* All OS and python versions\r\n\r\nI will have a look into creating a pull request to fix this issue.\r\nI think the solution is to change the assertion to an if statement to only run the code below that if it starts with _MEIPASS and thus could be bundled in the executable.\n", "before_files": [{"content": "#-----------------------------------------------------------------------------\n# Copyright (c) 2021, PyInstaller Development Team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n#\n# The full license is in the file COPYING.txt, distributed with this software.\n#\n# SPDX-License-Identifier: Apache-2.0\n#-----------------------------------------------------------------------------\n#\n# This rthook overrides pkgutil.iter_modules with custom implementation that uses PyInstaller's FrozenImporter to list\n# sub-modules embedded in the PYZ archive. The non-embedded modules (binary extensions, or .pyc modules in noarchive\n# build) are handled by original pkgutil iter_modules implementation (and consequently, python's FileFinder).\n#\n# The preferred way of adding support for iter_modules would be adding non-standard iter_modules() method to\n# FrozenImporter itself. However, that seems to work only for path entry finders (for use with sys.path_hooks), while\n# PyInstaller's FrozenImporter is registered as meta path finders (for use with sys.meta_path). Turning FrozenImporter\n# into path entry finder, would seemingly require the latter to support on-filesystem resources (e.g., extension\n# modules) in addition to PYZ-embedded ones.\n#\n# Therefore, we instead opt for overriding pkgutil.iter_modules with custom implementation that augments the output of\n# original implementation with contents of PYZ archive from FrozenImporter's TOC.\n\nimport os\nimport pkgutil\nimport sys\n\nfrom pyimod03_importers import FrozenImporter\n\n_orig_pkgutil_iter_modules = pkgutil.iter_modules\n\n\ndef _pyi_pkgutil_iter_modules(path=None, prefix=''):\n    # Use original implementation to discover on-filesystem modules (binary extensions in regular builds, or both binary\n    # extensions and compiled pyc modules in noarchive debug builds).\n    yield from _orig_pkgutil_iter_modules(path, prefix)\n\n    # Find the instance of PyInstaller's FrozenImporter.\n    for importer in pkgutil.iter_importers():\n        if isinstance(importer, FrozenImporter):\n            break\n    else:\n        return\n\n    if not path:\n        # Search for all top-level packages/modules. These will have no dots in their entry names.\n        for entry in importer.toc:\n            if entry.count('.') != 0:\n                continue\n            is_pkg = importer.is_package(entry)\n            yield pkgutil.ModuleInfo(importer, prefix + entry, is_pkg)\n    else:\n        # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.\n        SYS_PREFIX = sys._MEIPASS + os.path.sep\n        SYS_PREFIXLEN = len(SYS_PREFIX)\n        # Only single path is supported, and it must start with sys._MEIPASS.\n        pkg_path = os.path.normpath(path[0])\n        assert pkg_path.startswith(SYS_PREFIX)\n        # Construct package prefix from path...\n        pkg_prefix = pkg_path[SYS_PREFIXLEN:]\n        pkg_prefix = pkg_prefix.replace(os.path.sep, '.')\n        # ... and ensure it ends with a dot (so we can directly filter out the package itself).\n        if not pkg_prefix.endswith('.'):\n            pkg_prefix += '.'\n        pkg_prefix_len = len(pkg_prefix)\n\n        for entry in importer.toc:\n            if not entry.startswith(pkg_prefix):\n                continue\n            name = entry[pkg_prefix_len:]\n            if name.count('.') != 0:\n                continue\n            is_pkg = importer.is_package(entry)\n            yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)\n\n\npkgutil.iter_modules = _pyi_pkgutil_iter_modules\n", "path": "PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py"}], "after_files": [{"content": "#-----------------------------------------------------------------------------\n# Copyright (c) 2021, PyInstaller Development Team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n#\n# The full license is in the file COPYING.txt, distributed with this software.\n#\n# SPDX-License-Identifier: Apache-2.0\n#-----------------------------------------------------------------------------\n#\n# This rthook overrides pkgutil.iter_modules with custom implementation that uses PyInstaller's FrozenImporter to list\n# sub-modules embedded in the PYZ archive. The non-embedded modules (binary extensions, or .pyc modules in noarchive\n# build) are handled by original pkgutil iter_modules implementation (and consequently, python's FileFinder).\n#\n# The preferred way of adding support for iter_modules would be adding non-standard iter_modules() method to\n# FrozenImporter itself. However, that seems to work only for path entry finders (for use with sys.path_hooks), while\n# PyInstaller's FrozenImporter is registered as meta path finders (for use with sys.meta_path). Turning FrozenImporter\n# into path entry finder, would seemingly require the latter to support on-filesystem resources (e.g., extension\n# modules) in addition to PYZ-embedded ones.\n#\n# Therefore, we instead opt for overriding pkgutil.iter_modules with custom implementation that augments the output of\n# original implementation with contents of PYZ archive from FrozenImporter's TOC.\n\nimport os\nimport pkgutil\nimport sys\n\nfrom pyimod03_importers import FrozenImporter\n\n_orig_pkgutil_iter_modules = pkgutil.iter_modules\n\n\ndef _pyi_pkgutil_iter_modules(path=None, prefix=''):\n    # Use original implementation to discover on-filesystem modules (binary extensions in regular builds, or both binary\n    # extensions and compiled pyc modules in noarchive debug builds).\n    yield from _orig_pkgutil_iter_modules(path, prefix)\n\n    # Find the instance of PyInstaller's FrozenImporter.\n    for importer in pkgutil.iter_importers():\n        if isinstance(importer, FrozenImporter):\n            break\n    else:\n        return\n\n    if path is None:\n        # Search for all top-level packages/modules. These will have no dots in their entry names.\n        for entry in importer.toc:\n            if entry.count('.') != 0:\n                continue\n            is_pkg = importer.is_package(entry)\n            yield pkgutil.ModuleInfo(importer, prefix + entry, is_pkg)\n    else:\n        # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.\n        SYS_PREFIX = sys._MEIPASS + os.path.sep\n        SYS_PREFIXLEN = len(SYS_PREFIX)\n\n        for pkg_path in path:\n            pkg_path = os.path.normpath(pkg_path)\n            if not pkg_path.startswith(SYS_PREFIX):\n                # if the path does not start with sys._MEIPASS then it cannot be a bundled package.\n                continue\n            # Construct package prefix from path...\n            pkg_prefix = pkg_path[SYS_PREFIXLEN:]\n            pkg_prefix = pkg_prefix.replace(os.path.sep, '.')\n            # ... and ensure it ends with a dot (so we can directly filter out the package itself).\n            if not pkg_prefix.endswith('.'):\n                pkg_prefix += '.'\n            pkg_prefix_len = len(pkg_prefix)\n\n            for entry in importer.toc:\n                if not entry.startswith(pkg_prefix):\n                    continue\n                name = entry[pkg_prefix_len:]\n                if name.count('.') != 0:\n                    continue\n                is_pkg = importer.is_package(entry)\n                yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)\n\n\npkgutil.iter_modules = _pyi_pkgutil_iter_modules\n", "path": "PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py"}]}