We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Local clusterfuzz instance >=v1.9.0 fails to install properly on Ubuntu 18.04
As requested by @inferno-chromium :)

## Faulty behavior
Local clusterfuzz instance >=v1.9.0 crashes when starting the server via `python butler.py run_server --bootstrap`

## Error
```
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
```

## Related

#1328 

## Reproduction
Following the *'Prerequisites'* steps [here](https://google.github.io/clusterfuzz/getting-started/prerequisites/) with:

 * Host: stock Ubuntu 18.04
 * Clusterfuzz: `git clone https://github.com/google/clusterfuzz && git checkout tags/v1.9.0`
 * Python: 3.7 via `sudo add-apt-repository ppa:deadsnakes/ppa && sudo apt install -y python3.7`
 * Golang: v1.14.2, Install as described in [docs](https://golang.org/doc/install?download=go1.14.2.linux-amd64.tar.gz)

Next we run `local/install_deps.bash`. This runs in a couple of errors as shown below:


### Missing dependencies:

* curl (because Ubuntu...)
* pipenv (install_deps.bash tries to install it via *apt* but there is no such package on 18.04)
  * Workaround here is fix the installer (`local/install_deps_linux.bash`) using pip insead. E.g. replace the `apt install pipenv` with `sudo -EH pip3 install -U pipenv`

### Failed package installs

* psutil from Pipfile.lock fails to build/install due to missing `#include <Python.h>`
  * Fix here is to also install `libpython3.7` and `libpython3.7-dev`

When doing these fixes the installer finishes even though it still prints a bunch of errors regarding incompatible requirements:

```
| ERROR: grpcio-tools 1.17.0 has requirement grpcio>=1.17.0, but you'll have grpcio 1.15.0 which is incompatible.
| ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.
| ERROR: google-api-core 1.17.0 has requirement google-auth<2.0dev,>=1.14.0, but you'll have google-auth 1.8.1 which is incompatible.
| ERROR: google-cloud-datastore 1.7.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 1.3.0 which is incompatible.
| ERROR: google-cloud-storage 1.13.2 has requirement google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.3.0 which is incompatible.
| ERROR: google-cloud-storage 1.13.2 has requirement google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.3.0 which is incompatible.
| ERROR: google-cloud-datastore 1.7.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 1.3.0 which is incompatible.
| ERROR: google-api-core 1.17.0 has requirement google-auth<2.0dev,>=1.14.0, but you'll have google-auth 1.8.1 which is incompatible.
| ERROR: grpcio-tools 1.17.0 has requirement grpcio>=1.17.0, but you'll have grpcio 1.15.0 which is incompatible.
| ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.
```

### Server bootstrap

Once the dependency installer is done we can run:

```
pipenv shell
# python butler.py --help
python butler.py run_server --bootstrap
```

This crashes right at the end.

```
Running: python polymer_bundler.py (cwd='local')
| Building templates for App Engine...
| App Engine templates built successfully.
Created symlink: source: /home/toor/clusterfuzz/local/storage/local_gcs, target /home/toor/clusterfuzz/src/appengine/local_gcs.
Running: gunicorn -b :9000 main:app (cwd='src/appengine')
| [2020-05-14 21:56:30 +0200] [15318] [INFO] Starting gunicorn 20.0.4
| [2020-05-14 21:56:30 +0200] [15318] [INFO] Listening at: http://0.0.0.0:9000 (15318)
| [2020-05-14 21:56:30 +0200] [15318] [INFO] Using worker: sync
| [2020-05-14 21:56:30 +0200] [15321] [INFO] Booting worker with pid: 15321
| [2020-05-14 21:56:36 +0200] [15321] [ERROR] Exception in worker process
| Traceback (most recent call last):
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
|     worker.init_process()
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
|     self.load_wsgi()
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
|     self.wsgi = self.app.wsgi()
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
|     self.callable = self.load()
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
|     return self.load_wsgiapp()
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
|     return util.import_app(self.app_uri)
|   File "/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
|     mod = importlib.import_module(module)
|   File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
|     return _bootstrap._gcd_import(name[level:], package, level)
|   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
|   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
|   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
|   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
|   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
|   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
|   File "/home/toor/clusterfuzz/src/appengine/main.py", line 32, in <module>
|     firebase_admin.initialize_app()
|   File "third_party/firebase_admin/__init__.py", line 65, in initialize_app
|     credential = credentials.ApplicationDefault()
|   File "third_party/firebase_admin/credentials.py", line 133, in __init__
|     self._g_credential, self._project_id = google.auth.default(scopes=_scopes)
|   File "/home/toor/clusterfuzz/src/third_party/google/auth/_default.py", line 321, in default
|     raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
| google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
| [2020-05-14 21:56:36 +0200] [15321] [INFO] Worker exiting (pid: 15321)
| [2020-05-14 21:56:36 +0200] [15318] [INFO] Shutting down: Master
| [2020-05-14 21:56:36 +0200] [15318] [INFO] Reason: Worker failed to boot.
| Return code is non-zero (3).
| Exit.
Bootstrapping datastore...
Running: python butler.py run setup --non-dry-run --local --config-dir=configs/test
| Creating config
| Creating fuzzer afl
| Creating fuzzer libFuzzer
| Creating fuzzer honggfuzz
| Creating fuzzer syzkaller
| Creating template afl
| Creating template engine_asan
| Creating template engine_msan
| Creating template engine_ubsan
| Creating template honggfuzz
| Creating template libfuzzer
| Creating template syzkaller
| Creating template prune
| Done

```

## Expected behavior

Local instance runs fine when following the steps in the docs as they explicitly state that there is no need for gauth.

## Scope
This is not host OS related but must have been introduced post `tag v1.8.0` as this one runs fine. However, tag v1.9.0+ shows the above behavior.



--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/appengine/main.py`
Content:
```
1 # Copyright 2020 Google LLC
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #      http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 """Python 3 entrypoint."""
15 import importlib
16 import os
17 import sys
18 
19 # Add necessary directories to path.
20 sys.path.append('python')
21 sys.path.append('third_party')
22 
23 config_modules_path = os.path.join('config', 'modules')
24 if os.path.exists(config_modules_path):
25   sys.path.append(config_modules_path)
26 
27 if os.environ.get('GAE_ENV'):
28   import pkg_resources
29   importlib.reload(pkg_resources)
30 
31   import firebase_admin
32   firebase_admin.initialize_app()
33 
34 try:
35   # Run any module initialization code.
36   import module_init
37   module_init.appengine()
38 except ImportError:
39   pass
40 
41 import server
42 app = server.app
43
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/appengine/main.py b/src/appengine/main.py
--- a/src/appengine/main.py
+++ b/src/appengine/main.py
@@ -24,12 +24,14 @@
 if os.path.exists(config_modules_path):
   sys.path.append(config_modules_path)
 
-if os.environ.get('GAE_ENV'):
+gae_env = os.environ.get('GAE_ENV')
+if gae_env:
   import pkg_resources
   importlib.reload(pkg_resources)
 
-  import firebase_admin
-  firebase_admin.initialize_app()
+  if gae_env != 'dev':
+    import firebase_admin
+    firebase_admin.initialize_app()
 
 try:
   # Run any module initialization code.

{"golden_diff": "diff --git a/src/appengine/main.py b/src/appengine/main.py\n--- a/src/appengine/main.py\n+++ b/src/appengine/main.py\n@@ -24,12 +24,14 @@\n if os.path.exists(config_modules_path):\n   sys.path.append(config_modules_path)\n \n-if os.environ.get('GAE_ENV'):\n+gae_env = os.environ.get('GAE_ENV')\n+if gae_env:\n   import pkg_resources\n   importlib.reload(pkg_resources)\n \n-  import firebase_admin\n-  firebase_admin.initialize_app()\n+  if gae_env != 'dev':\n+    import firebase_admin\n+    firebase_admin.initialize_app()\n \n try:\n   # Run any module initialization code.\n", "issue": "Local clusterfuzz instance >=v1.9.0 fails to install properly on Ubuntu 18.04\nAs requested by @inferno-chromium :)\r\n\r\n## Faulty behavior\r\nLocal clusterfuzz instance >=v1.9.0 crashes when starting the server via `python butler.py run_server --bootstrap`\r\n\r\n## Error\r\n```\r\ngoogle.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started\r\n```\r\n\r\n## Related\r\n\r\n#1328 \r\n\r\n## Reproduction\r\nFollowing the *'Prerequisites'* steps [here](https://google.github.io/clusterfuzz/getting-started/prerequisites/) with:\r\n\r\n * Host: stock Ubuntu 18.04\r\n * Clusterfuzz: `git clone https://github.com/google/clusterfuzz && git checkout tags/v1.9.0`\r\n * Python: 3.7 via `sudo add-apt-repository ppa:deadsnakes/ppa && sudo apt install -y python3.7`\r\n * Golang: v1.14.2, Install as described in [docs](https://golang.org/doc/install?download=go1.14.2.linux-amd64.tar.gz)\r\n\r\nNext we run `local/install_deps.bash`. This runs in a couple of errors as shown below:\r\n\r\n\r\n### Missing dependencies:\r\n\r\n* curl (because Ubuntu...)\r\n* pipenv (install_deps.bash tries to install it via *apt* but there is no such package on 18.04)\r\n  * Workaround here is fix the installer (`local/install_deps_linux.bash`) using pip insead. E.g. replace the `apt install pipenv` with `sudo -EH pip3 install -U pipenv`\r\n\r\n### Failed package installs\r\n\r\n* psutil from Pipfile.lock fails to build/install due to missing `#include <Python.h>`\r\n  * Fix here is to also install `libpython3.7` and `libpython3.7-dev`\r\n\r\nWhen doing these fixes the installer finishes even though it still prints a bunch of errors regarding incompatible requirements:\r\n\r\n```\r\n| ERROR: grpcio-tools 1.17.0 has requirement grpcio>=1.17.0, but you'll have grpcio 1.15.0 which is incompatible.\r\n| ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.\r\n| ERROR: google-api-core 1.17.0 has requirement google-auth<2.0dev,>=1.14.0, but you'll have google-auth 1.8.1 which is incompatible.\r\n| ERROR: google-cloud-datastore 1.7.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 1.3.0 which is incompatible.\r\n| ERROR: google-cloud-storage 1.13.2 has requirement google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.3.0 which is incompatible.\r\n| ERROR: google-cloud-storage 1.13.2 has requirement google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.3.0 which is incompatible.\r\n| ERROR: google-cloud-datastore 1.7.0 has requirement google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 1.3.0 which is incompatible.\r\n| ERROR: google-api-core 1.17.0 has requirement google-auth<2.0dev,>=1.14.0, but you'll have google-auth 1.8.1 which is incompatible.\r\n| ERROR: grpcio-tools 1.17.0 has requirement grpcio>=1.17.0, but you'll have grpcio 1.15.0 which is incompatible.\r\n| ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.\r\n```\r\n\r\n### Server bootstrap\r\n\r\nOnce the dependency installer is done we can run:\r\n\r\n```\r\npipenv shell\r\n# python butler.py --help\r\npython butler.py run_server --bootstrap\r\n```\r\n\r\nThis crashes right at the end.\r\n\r\n```\r\nRunning: python polymer_bundler.py (cwd='local')\r\n| Building templates for App Engine...\r\n| App Engine templates built successfully.\r\nCreated symlink: source: /home/toor/clusterfuzz/local/storage/local_gcs, target /home/toor/clusterfuzz/src/appengine/local_gcs.\r\nRunning: gunicorn -b :9000 main:app (cwd='src/appengine')\r\n| [2020-05-14 21:56:30 +0200] [15318] [INFO] Starting gunicorn 20.0.4\r\n| [2020-05-14 21:56:30 +0200] [15318] [INFO] Listening at: http://0.0.0.0:9000 (15318)\r\n| [2020-05-14 21:56:30 +0200] [15318] [INFO] Using worker: sync\r\n| [2020-05-14 21:56:30 +0200] [15321] [INFO] Booting worker with pid: 15321\r\n| [2020-05-14 21:56:36 +0200] [15321] [ERROR] Exception in worker process\r\n| Traceback (most recent call last):\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/arbiter.py\", line 583, in spawn_worker\r\n|     worker.init_process()\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/workers/base.py\", line 119, in init_process\r\n|     self.load_wsgi()\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/workers/base.py\", line 144, in load_wsgi\r\n|     self.wsgi = self.app.wsgi()\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/app/base.py\", line 67, in wsgi\r\n|     self.callable = self.load()\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py\", line 49, in load\r\n|     return self.load_wsgiapp()\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py\", line 39, in load_wsgiapp\r\n|     return util.import_app(self.app_uri)\r\n|   File \"/home/toor/.local/share/virtualenvs/clusterfuzz-2lmaEd3m/lib/python3.7/site-packages/gunicorn/util.py\", line 358, in import_app\r\n|     mod = importlib.import_module(module)\r\n|   File \"/usr/lib/python3.7/importlib/__init__.py\", line 127, in import_module\r\n|     return _bootstrap._gcd_import(name[level:], package, level)\r\n|   File \"<frozen importlib._bootstrap>\", line 1006, in _gcd_import\r\n|   File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\r\n|   File \"<frozen importlib._bootstrap>\", line 967, in _find_and_load_unlocked\r\n|   File \"<frozen importlib._bootstrap>\", line 677, in _load_unlocked\r\n|   File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\r\n|   File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\r\n|   File \"/home/toor/clusterfuzz/src/appengine/main.py\", line 32, in <module>\r\n|     firebase_admin.initialize_app()\r\n|   File \"third_party/firebase_admin/__init__.py\", line 65, in initialize_app\r\n|     credential = credentials.ApplicationDefault()\r\n|   File \"third_party/firebase_admin/credentials.py\", line 133, in __init__\r\n|     self._g_credential, self._project_id = google.auth.default(scopes=_scopes)\r\n|   File \"/home/toor/clusterfuzz/src/third_party/google/auth/_default.py\", line 321, in default\r\n|     raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)\r\n| google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started\r\n| [2020-05-14 21:56:36 +0200] [15321] [INFO] Worker exiting (pid: 15321)\r\n| [2020-05-14 21:56:36 +0200] [15318] [INFO] Shutting down: Master\r\n| [2020-05-14 21:56:36 +0200] [15318] [INFO] Reason: Worker failed to boot.\r\n| Return code is non-zero (3).\r\n| Exit.\r\nBootstrapping datastore...\r\nRunning: python butler.py run setup --non-dry-run --local --config-dir=configs/test\r\n| Creating config\r\n| Creating fuzzer afl\r\n| Creating fuzzer libFuzzer\r\n| Creating fuzzer honggfuzz\r\n| Creating fuzzer syzkaller\r\n| Creating template afl\r\n| Creating template engine_asan\r\n| Creating template engine_msan\r\n| Creating template engine_ubsan\r\n| Creating template honggfuzz\r\n| Creating template libfuzzer\r\n| Creating template syzkaller\r\n| Creating template prune\r\n| Done\r\n\r\n```\r\n\r\n## Expected behavior\r\n\r\nLocal instance runs fine when following the steps in the docs as they explicitly state that there is no need for gauth.\r\n\r\n## Scope\r\nThis is not host OS related but must have been introduced post `tag v1.8.0` as this one runs fine. However, tag v1.9.0+ shows the above behavior.\r\n\r\n\n", "before_files": [{"content": "# Copyright 2020 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Python 3 entrypoint.\"\"\"\nimport importlib\nimport os\nimport sys\n\n# Add necessary directories to path.\nsys.path.append('python')\nsys.path.append('third_party')\n\nconfig_modules_path = os.path.join('config', 'modules')\nif os.path.exists(config_modules_path):\n  sys.path.append(config_modules_path)\n\nif os.environ.get('GAE_ENV'):\n  import pkg_resources\n  importlib.reload(pkg_resources)\n\n  import firebase_admin\n  firebase_admin.initialize_app()\n\ntry:\n  # Run any module initialization code.\n  import module_init\n  module_init.appengine()\nexcept ImportError:\n  pass\n\nimport server\napp = server.app\n", "path": "src/appengine/main.py"}], "after_files": [{"content": "# Copyright 2020 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Python 3 entrypoint.\"\"\"\nimport importlib\nimport os\nimport sys\n\n# Add necessary directories to path.\nsys.path.append('python')\nsys.path.append('third_party')\n\nconfig_modules_path = os.path.join('config', 'modules')\nif os.path.exists(config_modules_path):\n  sys.path.append(config_modules_path)\n\ngae_env = os.environ.get('GAE_ENV')\nif gae_env:\n  import pkg_resources\n  importlib.reload(pkg_resources)\n\n  if gae_env != 'dev':\n    import firebase_admin\n    firebase_admin.initialize_app()\n\ntry:\n  # Run any module initialization code.\n  import module_init\n  module_init.appengine()\nexcept ImportError:\n  pass\n\nimport server\napp = server.app\n", "path": "src/appengine/main.py"}]}