We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Path scrubbing not working for python 3.8 package installed with homebrew
Hi there, I got an issue with a pex file embedding a dependency (protobuf) under OSX. I guess it could be related to the protobuf package and how the package is installed with brew but maybe also in the way the scrubbing is done by PEX.

If I install `protobuf` through `pip3.8 install protobuf` the scrubbing works fine because `protobuf` is inside the path `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//google/protobuf/`.

But if you install the `protobuf` brew package through `brew install protobuf`, it is installed under `/usr/local/Cellar/protobuf/` and the python package is registered inside `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//homebrew-protobuf.pth` with the following line:
```
import site; site.addsitedir('/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages')
```

I put some logs to print `module.__path__` and `site_libs` before `cls._tainted_path(...)` is [called](https://github.com/pantsbuild/pex/blob/master/pex/pex.py#L209). Here is the difference between python3.7 and 3.8.

Python 3.7:
```
Module path: ['/Users/XXX/repack.pex/google', '/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google']
Site_libs : {'/usr/local/lib/python3.7/site-packages', '/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages'}
pex: Scrubbing google.__path__: /usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google
```

Python 3.8:
```
Module path: ['/Users/XXX/repack.pex/google', '/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages/google']
Site_libs : {'/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages', '/usr/local/lib/python3.8/site-packages'}

```

Notice there is no "Scrubbing" for python 3.8.  It is due to the change in where python package is located. Instead of being inside the python 3.8 directory, it is put as a standard `brew` package. The import is indirectly managed inside the `homebrew-protofbu.pth` file (see above for the path and content)

The `cls._tainted_path(...)` is then not able to to identify that the path matches in python 3.8 since they are completely different.

python 3.7 match
Module path : **/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages**/google
Site_libs : **/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages**

python 3.8 no match
Module path : /usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages/google
Site_libs : /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages


Any solution ?

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `pex/util.py`
Content:
```
1 # Copyright 2014 Pants project contributors (see CONTRIBUTORS.md).
2 # Licensed under the Apache License, Version 2.0 (see LICENSE).
3 
4 from __future__ import absolute_import
5 
6 import contextlib
7 import os
8 import tempfile
9 from hashlib import sha1
10 from site import makepath
11 
12 from pex.common import atomic_directory, safe_mkdir, safe_mkdtemp
13 from pex.compatibility import PY2, exec_function
14 from pex.third_party.pkg_resources import (
15     find_distributions,
16     resource_isdir,
17     resource_listdir,
18     resource_string,
19 )
20 from pex.tracer import TRACER
21 
22 
23 class DistributionHelper(object):
24     @classmethod
25     def access_zipped_assets(cls, static_module_name, static_path, dir_location=None):
26         """Create a copy of static resource files as we can't serve them from within the pex file.
27 
28         :param static_module_name: Module name containing module to cache in a tempdir
29         :type static_module_name: string, for example 'twitter.common.zookeeper' or similar
30         :param static_path: Module name, for example 'serverset'
31         :param dir_location: create a new temporary directory inside, or None to have one created
32         :returns temp_dir: Temporary directory with the zipped assets inside
33         :rtype: str
34         """
35         # asset_path is initially a module name that's the same as the static_path, but will be
36         # changed to walk the directory tree
37         # TODO(John Sirois): Unify with `pex.third_party.isolated(recursive_copy)`.
38         def walk_zipped_assets(static_module_name, static_path, asset_path, temp_dir):
39             for asset in resource_listdir(static_module_name, asset_path):
40                 if not asset:
41                     # The `resource_listdir` function returns a '' asset for the directory entry
42                     # itself if it is either present on the filesystem or present as an explicit
43                     # zip entry. Since we only care about files and subdirectories at this point,
44                     # skip these assets.
45                     continue
46                 asset_target = os.path.normpath(
47                     os.path.join(os.path.relpath(asset_path, static_path), asset)
48                 )
49                 if resource_isdir(static_module_name, os.path.join(asset_path, asset)):
50                     safe_mkdir(os.path.join(temp_dir, asset_target))
51                     walk_zipped_assets(
52                         static_module_name, static_path, os.path.join(asset_path, asset), temp_dir
53                     )
54                 else:
55                     with open(os.path.join(temp_dir, asset_target), "wb") as fp:
56                         path = os.path.join(static_path, asset_target)
57                         file_data = resource_string(static_module_name, path)
58                         fp.write(file_data)
59 
60         if dir_location is None:
61             temp_dir = safe_mkdtemp()
62         else:
63             temp_dir = dir_location
64 
65         walk_zipped_assets(static_module_name, static_path, static_path, temp_dir)
66 
67         return temp_dir
68 
69     @classmethod
70     def distribution_from_path(cls, path, name=None):
71         """Return a distribution from a path.
72 
73         If name is provided, find the distribution.  If none is found matching the name, return
74         None.  If name is not provided and there is unambiguously a single distribution, return that
75         distribution otherwise None.
76         """
77         if name is None:
78             distributions = set(find_distributions(path))
79             if len(distributions) == 1:
80                 return distributions.pop()
81         else:
82             for dist in find_distributions(path):
83                 if dist.project_name == name:
84                     return dist
85 
86 
87 class CacheHelper(object):
88     @classmethod
89     def update_hash(cls, filelike, digest):
90         """Update the digest of a single file in a memory-efficient manner."""
91         block_size = digest.block_size * 1024
92         for chunk in iter(lambda: filelike.read(block_size), b""):
93             digest.update(chunk)
94 
95     @classmethod
96     def hash(cls, path, digest=None, hasher=sha1):
97         """Return the digest of a single file in a memory-efficient manner."""
98         if digest is None:
99             digest = hasher()
100         with open(path, "rb") as fh:
101             cls.update_hash(fh, digest)
102         return digest.hexdigest()
103 
104     @classmethod
105     def _compute_hash(cls, names, stream_factory):
106         digest = sha1()
107         # Always use / as the path separator, since that's what zip uses.
108         hashed_names = [n.replace(os.sep, "/") for n in names]
109         digest.update("".join(hashed_names).encode("utf-8"))
110         for name in names:
111             with contextlib.closing(stream_factory(name)) as fp:
112                 cls.update_hash(fp, digest)
113         return digest.hexdigest()
114 
115     @classmethod
116     def _iter_files(cls, directory):
117         normpath = os.path.realpath(os.path.normpath(directory))
118         for root, _, files in os.walk(normpath):
119             for f in files:
120                 yield os.path.relpath(os.path.join(root, f), normpath)
121 
122     @classmethod
123     def pex_hash(cls, d):
124         """Return a reproducible hash of the contents of a directory."""
125         names = sorted(
126             f for f in cls._iter_files(d) if not (f.endswith(".pyc") or f.startswith("."))
127         )
128 
129         def stream_factory(name):
130             return open(os.path.join(d, name), "rb")  # noqa: T802
131 
132         return cls._compute_hash(names, stream_factory)
133 
134     @classmethod
135     def dir_hash(cls, d):
136         """Return a reproducible hash of the contents of a directory."""
137         names = sorted(f for f in cls._iter_files(d) if not f.endswith(".pyc"))
138 
139         def stream_factory(name):
140             return open(os.path.join(d, name), "rb")  # noqa: T802
141 
142         return cls._compute_hash(names, stream_factory)
143 
144     @classmethod
145     def cache_distribution(cls, zf, source, target_dir):
146         """Possibly cache a wheel from within a zipfile into `target_dir`.
147 
148         Given a zipfile handle and a source path prefix corresponding to a wheel install embedded within
149         that zip, maybe extract the wheel install into the target cache and then return a distribution
150         from the cache.
151 
152         :param zf: An open zip file (a zipped pex).
153         :type zf: :class:`zipfile.ZipFile`
154         :param str source: The path prefix of a wheel install embedded in the zip file.
155         :param str target_dir: The directory to cache the distribution in if not already cached.
156         :returns: The cached distribution.
157         :rtype: :class:`pex.third_party.pkg_resources.Distribution`
158         """
159         with atomic_directory(target_dir, source=source) as target_dir_tmp:
160             if target_dir_tmp is None:
161                 TRACER.log("Using cached {}".format(target_dir))
162             else:
163                 with TRACER.timed("Caching {}:{} in {}".format(zf.filename, source, target_dir)):
164                     for name in zf.namelist():
165                         if name.startswith(source) and not name.endswith("/"):
166                             zf.extract(name, target_dir_tmp)
167 
168         dist = DistributionHelper.distribution_from_path(target_dir)
169         assert dist is not None, "Failed to cache distribution ".format(source)
170         return dist
171 
172 
173 @contextlib.contextmanager
174 def named_temporary_file(*args, **kwargs):
175     """Due to a bug in python (https://bugs.python.org/issue14243), we need this to be able to use
176     the temporary file without deleting it."""
177     assert "delete" not in kwargs
178     kwargs["delete"] = False
179     fp = tempfile.NamedTemporaryFile(*args, **kwargs)
180     try:
181         with fp:
182             yield fp
183     finally:
184         os.remove(fp.name)
185 
186 
187 def iter_pth_paths(filename):
188     """Given a .pth file, extract and yield all inner paths without honoring imports.
189 
190     This shadows Python's site.py behavior, which is invoked at interpreter startup.
191     """
192     try:
193         f = open(filename, "rU" if PY2 else "r")  # noqa
194     except IOError:
195         return
196 
197     dirname = os.path.dirname(filename)
198     known_paths = set()
199 
200     with f:
201         for line in f:
202             line = line.rstrip()
203             if not line or line.startswith("#"):
204                 continue
205             elif line.startswith(("import ", "import\t")):
206                 try:
207                     exec_function(line, globals_map={})
208                     continue
209                 except Exception:
210                     # NB: import lines are routinely abused with extra code appended using `;` so the class of
211                     # exceptions that might be raised in broader than ImportError. As such we cacth broadly
212                     # here.
213 
214                     # Defer error handling to the higher level site.py logic invoked at startup.
215                     return
216             else:
217                 extras_dir, extras_dir_case_insensitive = makepath(dirname, line)
218                 if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):
219                     yield extras_dir
220                     known_paths.add(extras_dir_case_insensitive)
221
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/pex/util.py b/pex/util.py
--- a/pex/util.py
+++ b/pex/util.py
@@ -5,6 +5,7 @@
 
 import contextlib
 import os
+import sys
 import tempfile
 from hashlib import sha1
 from site import makepath
@@ -198,21 +199,42 @@
     known_paths = set()
 
     with f:
-        for line in f:
+        for i, line in enumerate(f, start=1):
             line = line.rstrip()
             if not line or line.startswith("#"):
                 continue
             elif line.startswith(("import ", "import\t")):
+                # One important side effect of executing import lines can be alteration of the
+                # sys.path directly or indirectly as a programmatic way to add sys.path entries
+                # in contrast to the standard .pth mechanism of including fixed paths as
+                # individual lines in the file. Here we capture all such programmatic attempts
+                # to expand the sys.path and report the additions.
+                original_sys_path = sys.path[:]
                 try:
+                    # N.B.: Setting sys.path to empty is ok since all the .pth files we find and
+                    # execute have already been found and executed by our ambient sys.executable
+                    # when it started up before running this PEX file. As such, all symbols imported
+                    # by the .pth files then will still be available now as cached in sys.modules.
+                    sys.path = []
                     exec_function(line, globals_map={})
-                    continue
-                except Exception:
-                    # NB: import lines are routinely abused with extra code appended using `;` so the class of
-                    # exceptions that might be raised in broader than ImportError. As such we cacth broadly
-                    # here.
+                    for path in sys.path:
+                        yield path
+                except Exception as e:
+                    # NB: import lines are routinely abused with extra code appended using `;` so
+                    # the class of exceptions that might be raised in broader than ImportError. As
+                    # such we catch broadly here.
+                    TRACER.log(
+                        "Error executing line {linenumber} of {pth_file} with content:\n"
+                        "{content}\n"
+                        "Error was:\n"
+                        "{error}".format(linenumber=i, pth_file=filename, content=line, error=e),
+                        V=9,
+                    )
 
                     # Defer error handling to the higher level site.py logic invoked at startup.
                     return
+                finally:
+                    sys.path = original_sys_path
             else:
                 extras_dir, extras_dir_case_insensitive = makepath(dirname, line)
                 if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):

{"golden_diff": "diff --git a/pex/util.py b/pex/util.py\n--- a/pex/util.py\n+++ b/pex/util.py\n@@ -5,6 +5,7 @@\n \n import contextlib\n import os\n+import sys\n import tempfile\n from hashlib import sha1\n from site import makepath\n@@ -198,21 +199,42 @@\n     known_paths = set()\n \n     with f:\n-        for line in f:\n+        for i, line in enumerate(f, start=1):\n             line = line.rstrip()\n             if not line or line.startswith(\"#\"):\n                 continue\n             elif line.startswith((\"import \", \"import\\t\")):\n+                # One important side effect of executing import lines can be alteration of the\n+                # sys.path directly or indirectly as a programmatic way to add sys.path entries\n+                # in contrast to the standard .pth mechanism of including fixed paths as\n+                # individual lines in the file. Here we capture all such programmatic attempts\n+                # to expand the sys.path and report the additions.\n+                original_sys_path = sys.path[:]\n                 try:\n+                    # N.B.: Setting sys.path to empty is ok since all the .pth files we find and\n+                    # execute have already been found and executed by our ambient sys.executable\n+                    # when it started up before running this PEX file. As such, all symbols imported\n+                    # by the .pth files then will still be available now as cached in sys.modules.\n+                    sys.path = []\n                     exec_function(line, globals_map={})\n-                    continue\n-                except Exception:\n-                    # NB: import lines are routinely abused with extra code appended using `;` so the class of\n-                    # exceptions that might be raised in broader than ImportError. As such we cacth broadly\n-                    # here.\n+                    for path in sys.path:\n+                        yield path\n+                except Exception as e:\n+                    # NB: import lines are routinely abused with extra code appended using `;` so\n+                    # the class of exceptions that might be raised in broader than ImportError. As\n+                    # such we catch broadly here.\n+                    TRACER.log(\n+                        \"Error executing line {linenumber} of {pth_file} with content:\\n\"\n+                        \"{content}\\n\"\n+                        \"Error was:\\n\"\n+                        \"{error}\".format(linenumber=i, pth_file=filename, content=line, error=e),\n+                        V=9,\n+                    )\n \n                     # Defer error handling to the higher level site.py logic invoked at startup.\n                     return\n+                finally:\n+                    sys.path = original_sys_path\n             else:\n                 extras_dir, extras_dir_case_insensitive = makepath(dirname, line)\n                 if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):\n", "issue": "Path scrubbing not working for python 3.8 package installed with homebrew\nHi there, I got an issue with a pex file embedding a dependency (protobuf) under OSX. I guess it could be related to the protobuf package and how the package is installed with brew but maybe also in the way the scrubbing is done by PEX.\r\n\r\nIf I install `protobuf` through `pip3.8 install protobuf` the scrubbing works fine because `protobuf` is inside the path `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//google/protobuf/`.\r\n\r\nBut if you install the `protobuf` brew package through `brew install protobuf`, it is installed under `/usr/local/Cellar/protobuf/` and the python package is registered inside `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//homebrew-protobuf.pth` with the following line:\r\n```\r\nimport site; site.addsitedir('/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages')\r\n```\r\n\r\nI put some logs to print `module.__path__` and `site_libs` before `cls._tainted_path(...)` is [called](https://github.com/pantsbuild/pex/blob/master/pex/pex.py#L209). Here is the difference between python3.7 and 3.8.\r\n\r\nPython 3.7:\r\n```\r\nModule path: ['/Users/XXX/repack.pex/google', '/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google']\r\nSite_libs : {'/usr/local/lib/python3.7/site-packages', '/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages'}\r\npex: Scrubbing google.__path__: /usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google\r\n```\r\n\r\nPython 3.8:\r\n```\r\nModule path: ['/Users/XXX/repack.pex/google', '/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages/google']\r\nSite_libs : {'/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages', '/usr/local/lib/python3.8/site-packages'}\r\n\r\n```\r\n\r\nNotice there is no \"Scrubbing\" for python 3.8.  It is due to the change in where python package is located. Instead of being inside the python 3.8 directory, it is put as a standard `brew` package. The import is indirectly managed inside the `homebrew-protofbu.pth` file (see above for the path and content)\r\n\r\nThe `cls._tainted_path(...)` is then not able to to identify that the path matches in python 3.8 since they are completely different.\r\n\r\npython 3.7 match\r\nModule path : **/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages**/google\r\nSite_libs : **/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages**\r\n\r\npython 3.8 no match\r\nModule path : /usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages/google\r\nSite_libs : /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages\r\n\r\n\r\nAny solution ?\n", "before_files": [{"content": "# Copyright 2014 Pants project contributors (see CONTRIBUTORS.md).\n# Licensed under the Apache License, Version 2.0 (see LICENSE).\n\nfrom __future__ import absolute_import\n\nimport contextlib\nimport os\nimport tempfile\nfrom hashlib import sha1\nfrom site import makepath\n\nfrom pex.common import atomic_directory, safe_mkdir, safe_mkdtemp\nfrom pex.compatibility import PY2, exec_function\nfrom pex.third_party.pkg_resources import (\n    find_distributions,\n    resource_isdir,\n    resource_listdir,\n    resource_string,\n)\nfrom pex.tracer import TRACER\n\n\nclass DistributionHelper(object):\n    @classmethod\n    def access_zipped_assets(cls, static_module_name, static_path, dir_location=None):\n        \"\"\"Create a copy of static resource files as we can't serve them from within the pex file.\n\n        :param static_module_name: Module name containing module to cache in a tempdir\n        :type static_module_name: string, for example 'twitter.common.zookeeper' or similar\n        :param static_path: Module name, for example 'serverset'\n        :param dir_location: create a new temporary directory inside, or None to have one created\n        :returns temp_dir: Temporary directory with the zipped assets inside\n        :rtype: str\n        \"\"\"\n        # asset_path is initially a module name that's the same as the static_path, but will be\n        # changed to walk the directory tree\n        # TODO(John Sirois): Unify with `pex.third_party.isolated(recursive_copy)`.\n        def walk_zipped_assets(static_module_name, static_path, asset_path, temp_dir):\n            for asset in resource_listdir(static_module_name, asset_path):\n                if not asset:\n                    # The `resource_listdir` function returns a '' asset for the directory entry\n                    # itself if it is either present on the filesystem or present as an explicit\n                    # zip entry. Since we only care about files and subdirectories at this point,\n                    # skip these assets.\n                    continue\n                asset_target = os.path.normpath(\n                    os.path.join(os.path.relpath(asset_path, static_path), asset)\n                )\n                if resource_isdir(static_module_name, os.path.join(asset_path, asset)):\n                    safe_mkdir(os.path.join(temp_dir, asset_target))\n                    walk_zipped_assets(\n                        static_module_name, static_path, os.path.join(asset_path, asset), temp_dir\n                    )\n                else:\n                    with open(os.path.join(temp_dir, asset_target), \"wb\") as fp:\n                        path = os.path.join(static_path, asset_target)\n                        file_data = resource_string(static_module_name, path)\n                        fp.write(file_data)\n\n        if dir_location is None:\n            temp_dir = safe_mkdtemp()\n        else:\n            temp_dir = dir_location\n\n        walk_zipped_assets(static_module_name, static_path, static_path, temp_dir)\n\n        return temp_dir\n\n    @classmethod\n    def distribution_from_path(cls, path, name=None):\n        \"\"\"Return a distribution from a path.\n\n        If name is provided, find the distribution.  If none is found matching the name, return\n        None.  If name is not provided and there is unambiguously a single distribution, return that\n        distribution otherwise None.\n        \"\"\"\n        if name is None:\n            distributions = set(find_distributions(path))\n            if len(distributions) == 1:\n                return distributions.pop()\n        else:\n            for dist in find_distributions(path):\n                if dist.project_name == name:\n                    return dist\n\n\nclass CacheHelper(object):\n    @classmethod\n    def update_hash(cls, filelike, digest):\n        \"\"\"Update the digest of a single file in a memory-efficient manner.\"\"\"\n        block_size = digest.block_size * 1024\n        for chunk in iter(lambda: filelike.read(block_size), b\"\"):\n            digest.update(chunk)\n\n    @classmethod\n    def hash(cls, path, digest=None, hasher=sha1):\n        \"\"\"Return the digest of a single file in a memory-efficient manner.\"\"\"\n        if digest is None:\n            digest = hasher()\n        with open(path, \"rb\") as fh:\n            cls.update_hash(fh, digest)\n        return digest.hexdigest()\n\n    @classmethod\n    def _compute_hash(cls, names, stream_factory):\n        digest = sha1()\n        # Always use / as the path separator, since that's what zip uses.\n        hashed_names = [n.replace(os.sep, \"/\") for n in names]\n        digest.update(\"\".join(hashed_names).encode(\"utf-8\"))\n        for name in names:\n            with contextlib.closing(stream_factory(name)) as fp:\n                cls.update_hash(fp, digest)\n        return digest.hexdigest()\n\n    @classmethod\n    def _iter_files(cls, directory):\n        normpath = os.path.realpath(os.path.normpath(directory))\n        for root, _, files in os.walk(normpath):\n            for f in files:\n                yield os.path.relpath(os.path.join(root, f), normpath)\n\n    @classmethod\n    def pex_hash(cls, d):\n        \"\"\"Return a reproducible hash of the contents of a directory.\"\"\"\n        names = sorted(\n            f for f in cls._iter_files(d) if not (f.endswith(\".pyc\") or f.startswith(\".\"))\n        )\n\n        def stream_factory(name):\n            return open(os.path.join(d, name), \"rb\")  # noqa: T802\n\n        return cls._compute_hash(names, stream_factory)\n\n    @classmethod\n    def dir_hash(cls, d):\n        \"\"\"Return a reproducible hash of the contents of a directory.\"\"\"\n        names = sorted(f for f in cls._iter_files(d) if not f.endswith(\".pyc\"))\n\n        def stream_factory(name):\n            return open(os.path.join(d, name), \"rb\")  # noqa: T802\n\n        return cls._compute_hash(names, stream_factory)\n\n    @classmethod\n    def cache_distribution(cls, zf, source, target_dir):\n        \"\"\"Possibly cache a wheel from within a zipfile into `target_dir`.\n\n        Given a zipfile handle and a source path prefix corresponding to a wheel install embedded within\n        that zip, maybe extract the wheel install into the target cache and then return a distribution\n        from the cache.\n\n        :param zf: An open zip file (a zipped pex).\n        :type zf: :class:`zipfile.ZipFile`\n        :param str source: The path prefix of a wheel install embedded in the zip file.\n        :param str target_dir: The directory to cache the distribution in if not already cached.\n        :returns: The cached distribution.\n        :rtype: :class:`pex.third_party.pkg_resources.Distribution`\n        \"\"\"\n        with atomic_directory(target_dir, source=source) as target_dir_tmp:\n            if target_dir_tmp is None:\n                TRACER.log(\"Using cached {}\".format(target_dir))\n            else:\n                with TRACER.timed(\"Caching {}:{} in {}\".format(zf.filename, source, target_dir)):\n                    for name in zf.namelist():\n                        if name.startswith(source) and not name.endswith(\"/\"):\n                            zf.extract(name, target_dir_tmp)\n\n        dist = DistributionHelper.distribution_from_path(target_dir)\n        assert dist is not None, \"Failed to cache distribution \".format(source)\n        return dist\n\n\n@contextlib.contextmanager\ndef named_temporary_file(*args, **kwargs):\n    \"\"\"Due to a bug in python (https://bugs.python.org/issue14243), we need this to be able to use\n    the temporary file without deleting it.\"\"\"\n    assert \"delete\" not in kwargs\n    kwargs[\"delete\"] = False\n    fp = tempfile.NamedTemporaryFile(*args, **kwargs)\n    try:\n        with fp:\n            yield fp\n    finally:\n        os.remove(fp.name)\n\n\ndef iter_pth_paths(filename):\n    \"\"\"Given a .pth file, extract and yield all inner paths without honoring imports.\n\n    This shadows Python's site.py behavior, which is invoked at interpreter startup.\n    \"\"\"\n    try:\n        f = open(filename, \"rU\" if PY2 else \"r\")  # noqa\n    except IOError:\n        return\n\n    dirname = os.path.dirname(filename)\n    known_paths = set()\n\n    with f:\n        for line in f:\n            line = line.rstrip()\n            if not line or line.startswith(\"#\"):\n                continue\n            elif line.startswith((\"import \", \"import\\t\")):\n                try:\n                    exec_function(line, globals_map={})\n                    continue\n                except Exception:\n                    # NB: import lines are routinely abused with extra code appended using `;` so the class of\n                    # exceptions that might be raised in broader than ImportError. As such we cacth broadly\n                    # here.\n\n                    # Defer error handling to the higher level site.py logic invoked at startup.\n                    return\n            else:\n                extras_dir, extras_dir_case_insensitive = makepath(dirname, line)\n                if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):\n                    yield extras_dir\n                    known_paths.add(extras_dir_case_insensitive)\n", "path": "pex/util.py"}], "after_files": [{"content": "# Copyright 2014 Pants project contributors (see CONTRIBUTORS.md).\n# Licensed under the Apache License, Version 2.0 (see LICENSE).\n\nfrom __future__ import absolute_import\n\nimport contextlib\nimport os\nimport sys\nimport tempfile\nfrom hashlib import sha1\nfrom site import makepath\n\nfrom pex.common import atomic_directory, safe_mkdir, safe_mkdtemp\nfrom pex.compatibility import PY2, exec_function\nfrom pex.third_party.pkg_resources import (\n    find_distributions,\n    resource_isdir,\n    resource_listdir,\n    resource_string,\n)\nfrom pex.tracer import TRACER\n\n\nclass DistributionHelper(object):\n    @classmethod\n    def access_zipped_assets(cls, static_module_name, static_path, dir_location=None):\n        \"\"\"Create a copy of static resource files as we can't serve them from within the pex file.\n\n        :param static_module_name: Module name containing module to cache in a tempdir\n        :type static_module_name: string, for example 'twitter.common.zookeeper' or similar\n        :param static_path: Module name, for example 'serverset'\n        :param dir_location: create a new temporary directory inside, or None to have one created\n        :returns temp_dir: Temporary directory with the zipped assets inside\n        :rtype: str\n        \"\"\"\n        # asset_path is initially a module name that's the same as the static_path, but will be\n        # changed to walk the directory tree\n        # TODO(John Sirois): Unify with `pex.third_party.isolated(recursive_copy)`.\n        def walk_zipped_assets(static_module_name, static_path, asset_path, temp_dir):\n            for asset in resource_listdir(static_module_name, asset_path):\n                if not asset:\n                    # The `resource_listdir` function returns a '' asset for the directory entry\n                    # itself if it is either present on the filesystem or present as an explicit\n                    # zip entry. Since we only care about files and subdirectories at this point,\n                    # skip these assets.\n                    continue\n                asset_target = os.path.normpath(\n                    os.path.join(os.path.relpath(asset_path, static_path), asset)\n                )\n                if resource_isdir(static_module_name, os.path.join(asset_path, asset)):\n                    safe_mkdir(os.path.join(temp_dir, asset_target))\n                    walk_zipped_assets(\n                        static_module_name, static_path, os.path.join(asset_path, asset), temp_dir\n                    )\n                else:\n                    with open(os.path.join(temp_dir, asset_target), \"wb\") as fp:\n                        path = os.path.join(static_path, asset_target)\n                        file_data = resource_string(static_module_name, path)\n                        fp.write(file_data)\n\n        if dir_location is None:\n            temp_dir = safe_mkdtemp()\n        else:\n            temp_dir = dir_location\n\n        walk_zipped_assets(static_module_name, static_path, static_path, temp_dir)\n\n        return temp_dir\n\n    @classmethod\n    def distribution_from_path(cls, path, name=None):\n        \"\"\"Return a distribution from a path.\n\n        If name is provided, find the distribution.  If none is found matching the name, return\n        None.  If name is not provided and there is unambiguously a single distribution, return that\n        distribution otherwise None.\n        \"\"\"\n        if name is None:\n            distributions = set(find_distributions(path))\n            if len(distributions) == 1:\n                return distributions.pop()\n        else:\n            for dist in find_distributions(path):\n                if dist.project_name == name:\n                    return dist\n\n\nclass CacheHelper(object):\n    @classmethod\n    def update_hash(cls, filelike, digest):\n        \"\"\"Update the digest of a single file in a memory-efficient manner.\"\"\"\n        block_size = digest.block_size * 1024\n        for chunk in iter(lambda: filelike.read(block_size), b\"\"):\n            digest.update(chunk)\n\n    @classmethod\n    def hash(cls, path, digest=None, hasher=sha1):\n        \"\"\"Return the digest of a single file in a memory-efficient manner.\"\"\"\n        if digest is None:\n            digest = hasher()\n        with open(path, \"rb\") as fh:\n            cls.update_hash(fh, digest)\n        return digest.hexdigest()\n\n    @classmethod\n    def _compute_hash(cls, names, stream_factory):\n        digest = sha1()\n        # Always use / as the path separator, since that's what zip uses.\n        hashed_names = [n.replace(os.sep, \"/\") for n in names]\n        digest.update(\"\".join(hashed_names).encode(\"utf-8\"))\n        for name in names:\n            with contextlib.closing(stream_factory(name)) as fp:\n                cls.update_hash(fp, digest)\n        return digest.hexdigest()\n\n    @classmethod\n    def _iter_files(cls, directory):\n        normpath = os.path.realpath(os.path.normpath(directory))\n        for root, _, files in os.walk(normpath):\n            for f in files:\n                yield os.path.relpath(os.path.join(root, f), normpath)\n\n    @classmethod\n    def pex_hash(cls, d):\n        \"\"\"Return a reproducible hash of the contents of a directory.\"\"\"\n        names = sorted(\n            f for f in cls._iter_files(d) if not (f.endswith(\".pyc\") or f.startswith(\".\"))\n        )\n\n        def stream_factory(name):\n            return open(os.path.join(d, name), \"rb\")  # noqa: T802\n\n        return cls._compute_hash(names, stream_factory)\n\n    @classmethod\n    def dir_hash(cls, d):\n        \"\"\"Return a reproducible hash of the contents of a directory.\"\"\"\n        names = sorted(f for f in cls._iter_files(d) if not f.endswith(\".pyc\"))\n\n        def stream_factory(name):\n            return open(os.path.join(d, name), \"rb\")  # noqa: T802\n\n        return cls._compute_hash(names, stream_factory)\n\n    @classmethod\n    def cache_distribution(cls, zf, source, target_dir):\n        \"\"\"Possibly cache a wheel from within a zipfile into `target_dir`.\n\n        Given a zipfile handle and a source path prefix corresponding to a wheel install embedded within\n        that zip, maybe extract the wheel install into the target cache and then return a distribution\n        from the cache.\n\n        :param zf: An open zip file (a zipped pex).\n        :type zf: :class:`zipfile.ZipFile`\n        :param str source: The path prefix of a wheel install embedded in the zip file.\n        :param str target_dir: The directory to cache the distribution in if not already cached.\n        :returns: The cached distribution.\n        :rtype: :class:`pex.third_party.pkg_resources.Distribution`\n        \"\"\"\n        with atomic_directory(target_dir, source=source) as target_dir_tmp:\n            if target_dir_tmp is None:\n                TRACER.log(\"Using cached {}\".format(target_dir))\n            else:\n                with TRACER.timed(\"Caching {}:{} in {}\".format(zf.filename, source, target_dir)):\n                    for name in zf.namelist():\n                        if name.startswith(source) and not name.endswith(\"/\"):\n                            zf.extract(name, target_dir_tmp)\n\n        dist = DistributionHelper.distribution_from_path(target_dir)\n        assert dist is not None, \"Failed to cache distribution \".format(source)\n        return dist\n\n\n@contextlib.contextmanager\ndef named_temporary_file(*args, **kwargs):\n    \"\"\"Due to a bug in python (https://bugs.python.org/issue14243), we need this to be able to use\n    the temporary file without deleting it.\"\"\"\n    assert \"delete\" not in kwargs\n    kwargs[\"delete\"] = False\n    fp = tempfile.NamedTemporaryFile(*args, **kwargs)\n    try:\n        with fp:\n            yield fp\n    finally:\n        os.remove(fp.name)\n\n\ndef iter_pth_paths(filename):\n    \"\"\"Given a .pth file, extract and yield all inner paths without honoring imports.\n\n    This shadows Python's site.py behavior, which is invoked at interpreter startup.\n    \"\"\"\n    try:\n        f = open(filename, \"rU\" if PY2 else \"r\")  # noqa\n    except IOError:\n        return\n\n    dirname = os.path.dirname(filename)\n    known_paths = set()\n\n    with f:\n        for i, line in enumerate(f, start=1):\n            line = line.rstrip()\n            if not line or line.startswith(\"#\"):\n                continue\n            elif line.startswith((\"import \", \"import\\t\")):\n                # One important side effect of executing import lines can be alteration of the\n                # sys.path directly or indirectly as a programmatic way to add sys.path entries\n                # in contrast to the standard .pth mechanism of including fixed paths as\n                # individual lines in the file. Here we capture all such programmatic attempts\n                # to expand the sys.path and report the additions.\n                original_sys_path = sys.path[:]\n                try:\n                    # N.B.: Setting sys.path to empty is ok since all the .pth files we find and\n                    # execute have already been found and executed by our ambient sys.executable\n                    # when it started up before running this PEX file. As such, all symbols imported\n                    # by the .pth files then will still be available now as cached in sys.modules.\n                    sys.path = []\n                    exec_function(line, globals_map={})\n                    for path in sys.path:\n                        yield path\n                except Exception as e:\n                    # NB: import lines are routinely abused with extra code appended using `;` so\n                    # the class of exceptions that might be raised in broader than ImportError. As\n                    # such we catch broadly here.\n                    TRACER.log(\n                        \"Error executing line {linenumber} of {pth_file} with content:\\n\"\n                        \"{content}\\n\"\n                        \"Error was:\\n\"\n                        \"{error}\".format(linenumber=i, pth_file=filename, content=line, error=e),\n                        V=9,\n                    )\n\n                    # Defer error handling to the higher level site.py logic invoked at startup.\n                    return\n                finally:\n                    sys.path = original_sys_path\n            else:\n                extras_dir, extras_dir_case_insensitive = makepath(dirname, line)\n                if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):\n                    yield extras_dir\n                    known_paths.add(extras_dir_case_insensitive)\n", "path": "pex/util.py"}]}