prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  [CT-3107] [Bug] nested dependencies not installed when package is a tarball
  ### Is this a new bug in dbt-core?
  
  - [X] I believe this is a new bug in dbt-core
  - [X] I have searched the existing issues, and I could not find an existing issue for this bug
  
  ### Current Behavior
  
  when running `dbt deps` to install a package specified as a tarball, dbt doesn't install nested dependencies (i.e. packages specified in the imported package's `packages.yml` file) as it does when installing a package from local, git or the dbt hub.
  
  ### Expected Behavior
  
  consistent behaviour across import methods regarding nested dependencies. dbt should install any dependencies specified in the tarball project's packages.yml file.
  
  
  ### Steps To Reproduce
  
  this can be reproduced by importing the tarball of a package with nested dependencies. In this case, importing dbt_expectations should cause dbt_date to be installed, as its included in the package's dependencies here: https://github.com/calogica/dbt-expectations/blob/0.9.0/packages.yml
  
  Steps:
  1. create a `packages.yml` file in a project with the following structure:
  ``` yaml
  packages:
    - tarball: "https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz"
      name: "dbt_expectations"
  ```
  2. run `dbt deps`
  
  
  running dbt deps will only install dbt_expectations:
  ```
  20:08:55  Running with dbt=1.5.6
  20:08:55  Installing dbt_expectations
  20:08:56  Installed from tarball (url: https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz)
  ```
  compare this to installing the same package from dbt hub, with the following `packages.yml`:
  ``` yaml
  packages:
    - package: calogica/dbt_expectations
      version: "0.9.0"
  ```
  ```
  20:14:24  Running with dbt=1.5.6
  20:14:24  Installing calogica/dbt_expectations
  20:14:25  Installed from version 0.9.0
  20:14:25  Up to date!
  20:14:25  Installing calogica/dbt_date
  20:14:25  Installed from version 0.8.1
  20:14:25  Updated version available: 0.9.1
  20:14:25  
  20:14:25  Updates available for packages: ['calogica/dbt_date']                 
  Update your versions in packages.yml, then run dbt deps
  ```
  
  ### Relevant log output
  
  _No response_
  
  ### Environment
  
  ```markdown
  - OS: Mac OS 13.5.2 (22G91)
  - Python: 3.9
  - dbt: 1.5.6
  ```
  
  
  ### Which database adapter are you using with dbt?
  
  snowflake
  
  ### Additional Context
  
  _No response_
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `core/dbt/deps/tarball.py`
  Content:
  ```
  1 from typing import Dict
  2 
  3 from dbt.contracts.project import RegistryPackageMetadata, TarballPackage
  4 from dbt.deps.base import PinnedPackage, UnpinnedPackage
  5 
  6 
  7 class TarballPackageMixin:
  8     def __init__(self, tarball: str) -> None:
  9         super().__init__()
  10         self.tarball = tarball
  11 
  12     @property
  13     def name(self):
  14         return self.tarball
  15 
  16     def source_type(self) -> str:
  17         return "tarball"
  18 
  19 
  20 class TarballPinnedPackage(TarballPackageMixin, PinnedPackage):
  21     def __init__(self, tarball: str, package: str) -> None:
  22         super().__init__(tarball)
  23         # setup to recycle RegistryPinnedPackage fns
  24         self.package = package
  25         self.version = "tarball"
  26 
  27     @property
  28     def name(self):
  29         return self.package
  30 
  31     def to_dict(self) -> Dict[str, str]:
  32         return {
  33             "tarball": self.tarball,
  34             "version": self.version,
  35             "package": self.package,
  36         }
  37 
  38     def get_version(self):
  39         return self.version
  40 
  41     def nice_version_name(self):
  42         return f"tarball (url: {self.tarball})"
  43 
  44     def _fetch_metadata(self, project, renderer):
  45         """
  46         recycle RegistryPackageMetadata so that we can use the install and
  47         download_and_untar from RegistryPinnedPackage next.
  48         build RegistryPackageMetadata from info passed via packages.yml since no
  49         'metadata' service exists in this case.
  50         """
  51 
  52         dct = {
  53             "name": self.package,
  54             "packages": [],  # note: required by RegistryPackageMetadata
  55             "downloads": {"tarball": self.tarball},
  56         }
  57 
  58         return RegistryPackageMetadata.from_dict(dct)
  59 
  60     def install(self, project, renderer):
  61         self._install(project, renderer)
  62 
  63 
  64 class TarballUnpinnedPackage(TarballPackageMixin, UnpinnedPackage[TarballPinnedPackage]):
  65     def __init__(
  66         self,
  67         tarball: str,
  68         package: str,
  69     ) -> None:
  70         super().__init__(tarball)
  71         # setup to recycle RegistryPinnedPackage fns
  72         self.package = package
  73         self.version = "tarball"
  74 
  75     @classmethod
  76     def from_contract(cls, contract: TarballPackage) -> "TarballUnpinnedPackage":
  77         return cls(tarball=contract.tarball, package=contract.name)
  78 
  79     def incorporate(self, other: "TarballUnpinnedPackage") -> "TarballUnpinnedPackage":
  80         return TarballUnpinnedPackage(tarball=self.tarball, package=self.package)
  81 
  82     def resolved(self) -> TarballPinnedPackage:
  83         return TarballPinnedPackage(tarball=self.tarball, package=self.package)
  84
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/core/dbt/deps/tarball.py b/core/dbt/deps/tarball.py
  --- a/core/dbt/deps/tarball.py
  +++ b/core/dbt/deps/tarball.py
  @@ -1,7 +1,14 @@
  +import functools
  +import os
  +from pathlib import Path
   from typing import Dict
   
  -from dbt.contracts.project import RegistryPackageMetadata, TarballPackage
  -from dbt.deps.base import PinnedPackage, UnpinnedPackage
  +from dbt.clients import system
  +from dbt.config.project import PartialProject
  +from dbt.contracts.project import TarballPackage
  +from dbt.deps.base import PinnedPackage, UnpinnedPackage, get_downloads_path
  +from dbt.exceptions import DependencyError
  +from dbt.utils import _connection_exception_retry as connection_exception_retry
   
   
   class TarballPackageMixin:
  @@ -20,9 +27,10 @@
   class TarballPinnedPackage(TarballPackageMixin, PinnedPackage):
       def __init__(self, tarball: str, package: str) -> None:
           super().__init__(tarball)
  -        # setup to recycle RegistryPinnedPackage fns
           self.package = package
           self.version = "tarball"
  +        self.tar_path = os.path.join(Path(get_downloads_path()), self.package)
  +        self.untarred_path = f"{self.tar_path}_untarred"
   
       @property
       def name(self):
  @@ -31,8 +39,7 @@
       def to_dict(self) -> Dict[str, str]:
           return {
               "tarball": self.tarball,
  -            "version": self.version,
  -            "package": self.package,
  +            "name": self.package,
           }
   
       def get_version(self):
  @@ -42,23 +49,38 @@
           return f"tarball (url: {self.tarball})"
   
       def _fetch_metadata(self, project, renderer):
  -        """
  -        recycle RegistryPackageMetadata so that we can use the install and
  -        download_and_untar from RegistryPinnedPackage next.
  -        build RegistryPackageMetadata from info passed via packages.yml since no
  -        'metadata' service exists in this case.
  -        """
  -
  -        dct = {
  -            "name": self.package,
  -            "packages": [],  # note: required by RegistryPackageMetadata
  -            "downloads": {"tarball": self.tarball},
  -        }
  -
  -        return RegistryPackageMetadata.from_dict(dct)
  +        """Download and untar the project and parse metadata from the project folder."""
  +        download_untar_fn = functools.partial(
  +            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name
  +        )
  +        connection_exception_retry(download_untar_fn, 5)
  +
  +        tar_contents = os.listdir(self.untarred_path)
  +        if len(tar_contents) != 1:
  +            raise DependencyError(
  +                f"Incorrect structure for package extracted from {self.tarball}."
  +                f"The extracted package needs to follow the structure {self.name}/<package_contents>."
  +            )
  +        child_folder = os.listdir(self.untarred_path)[0]
  +
  +        self.untarred_path = os.path.join(self.untarred_path, child_folder)
  +        partial = PartialProject.from_project_root(self.untarred_path)
  +        metadata = partial.render_package_metadata(renderer)
  +        metadata.name = self.package if self.package else metadata.name
  +        return metadata
   
       def install(self, project, renderer):
  -        self._install(project, renderer)
  +        download_untar_fn = functools.partial(
  +            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name
  +        )
  +        connection_exception_retry(download_untar_fn, 5)
  +        dest_path = self.get_installation_path(project, renderer)
  +        if os.path.exists(dest_path):
  +            if system.path_is_symlink(dest_path):
  +                system.remove_file(dest_path)
  +            else:
  +                system.rmdir(dest_path)
  +        system.move(self.untarred_path, dest_path)
   
   
   class TarballUnpinnedPackage(TarballPackageMixin, UnpinnedPackage[TarballPinnedPackage]):
  
verification_info: {"golden_diff": "diff --git a/core/dbt/deps/tarball.py b/core/dbt/deps/tarball.py\n--- a/core/dbt/deps/tarball.py\n+++ b/core/dbt/deps/tarball.py\n@@ -1,7 +1,14 @@\n+import functools\n+import os\n+from pathlib import Path\n from typing import Dict\n \n-from dbt.contracts.project import RegistryPackageMetadata, TarballPackage\n-from dbt.deps.base import PinnedPackage, UnpinnedPackage\n+from dbt.clients import system\n+from dbt.config.project import PartialProject\n+from dbt.contracts.project import TarballPackage\n+from dbt.deps.base import PinnedPackage, UnpinnedPackage, get_downloads_path\n+from dbt.exceptions import DependencyError\n+from dbt.utils import _connection_exception_retry as connection_exception_retry\n \n \n class TarballPackageMixin:\n@@ -20,9 +27,10 @@\n class TarballPinnedPackage(TarballPackageMixin, PinnedPackage):\n     def __init__(self, tarball: str, package: str) -> None:\n         super().__init__(tarball)\n-        # setup to recycle RegistryPinnedPackage fns\n         self.package = package\n         self.version = \"tarball\"\n+        self.tar_path = os.path.join(Path(get_downloads_path()), self.package)\n+        self.untarred_path = f\"{self.tar_path}_untarred\"\n \n     @property\n     def name(self):\n@@ -31,8 +39,7 @@\n     def to_dict(self) -> Dict[str, str]:\n         return {\n             \"tarball\": self.tarball,\n-            \"version\": self.version,\n-            \"package\": self.package,\n+            \"name\": self.package,\n         }\n \n     def get_version(self):\n@@ -42,23 +49,38 @@\n         return f\"tarball (url: {self.tarball})\"\n \n     def _fetch_metadata(self, project, renderer):\n-        \"\"\"\n-        recycle RegistryPackageMetadata so that we can use the install and\n-        download_and_untar from RegistryPinnedPackage next.\n-        build RegistryPackageMetadata from info passed via packages.yml since no\n-        'metadata' service exists in this case.\n-        \"\"\"\n-\n-        dct = {\n-            \"name\": self.package,\n-            \"packages\": [],  # note: required by RegistryPackageMetadata\n-            \"downloads\": {\"tarball\": self.tarball},\n-        }\n-\n-        return RegistryPackageMetadata.from_dict(dct)\n+        \"\"\"Download and untar the project and parse metadata from the project folder.\"\"\"\n+        download_untar_fn = functools.partial(\n+            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name\n+        )\n+        connection_exception_retry(download_untar_fn, 5)\n+\n+        tar_contents = os.listdir(self.untarred_path)\n+        if len(tar_contents) != 1:\n+            raise DependencyError(\n+                f\"Incorrect structure for package extracted from {self.tarball}.\"\n+                f\"The extracted package needs to follow the structure {self.name}/<package_contents>.\"\n+            )\n+        child_folder = os.listdir(self.untarred_path)[0]\n+\n+        self.untarred_path = os.path.join(self.untarred_path, child_folder)\n+        partial = PartialProject.from_project_root(self.untarred_path)\n+        metadata = partial.render_package_metadata(renderer)\n+        metadata.name = self.package if self.package else metadata.name\n+        return metadata\n \n     def install(self, project, renderer):\n-        self._install(project, renderer)\n+        download_untar_fn = functools.partial(\n+            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name\n+        )\n+        connection_exception_retry(download_untar_fn, 5)\n+        dest_path = self.get_installation_path(project, renderer)\n+        if os.path.exists(dest_path):\n+            if system.path_is_symlink(dest_path):\n+                system.remove_file(dest_path)\n+            else:\n+                system.rmdir(dest_path)\n+        system.move(self.untarred_path, dest_path)\n \n \n class TarballUnpinnedPackage(TarballPackageMixin, UnpinnedPackage[TarballPinnedPackage]):\n", "issue": "[CT-3107] [Bug] nested dependencies not installed when package is a tarball\n### Is this a new bug in dbt-core?\r\n\r\n- [X] I believe this is a new bug in dbt-core\r\n- [X] I have searched the existing issues, and I could not find an existing issue for this bug\r\n\r\n### Current Behavior\r\n\r\nwhen running `dbt deps` to install a package specified as a tarball, dbt doesn't install nested dependencies (i.e. packages specified in the imported package's `packages.yml` file) as it does when installing a package from local, git or the dbt hub.\r\n\r\n### Expected Behavior\r\n\r\nconsistent behaviour across import methods regarding nested dependencies. dbt should install any dependencies specified in the tarball project's packages.yml file.\r\n\r\n\r\n### Steps To Reproduce\r\n\r\nthis can be reproduced by importing the tarball of a package with nested dependencies. In this case, importing dbt_expectations should cause dbt_date to be installed, as its included in the package's dependencies here: https://github.com/calogica/dbt-expectations/blob/0.9.0/packages.yml\r\n\r\nSteps:\r\n1. create a `packages.yml` file in a project with the following structure:\r\n``` yaml\r\npackages:\r\n  - tarball: \"https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz\"\r\n    name: \"dbt_expectations\"\r\n```\r\n2. run `dbt deps`\r\n\r\n\r\nrunning dbt deps will only install dbt_expectations:\r\n```\r\n20:08:55  Running with dbt=1.5.6\r\n20:08:55  Installing dbt_expectations\r\n20:08:56  Installed from tarball (url: https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz)\r\n```\r\ncompare this to installing the same package from dbt hub, with the following `packages.yml`:\r\n``` yaml\r\npackages:\r\n  - package: calogica/dbt_expectations\r\n    version: \"0.9.0\"\r\n```\r\n```\r\n20:14:24  Running with dbt=1.5.6\r\n20:14:24  Installing calogica/dbt_expectations\r\n20:14:25  Installed from version 0.9.0\r\n20:14:25  Up to date!\r\n20:14:25  Installing calogica/dbt_date\r\n20:14:25  Installed from version 0.8.1\r\n20:14:25  Updated version available: 0.9.1\r\n20:14:25  \r\n20:14:25  Updates available for packages: ['calogica/dbt_date']                 \r\nUpdate your versions in packages.yml, then run dbt deps\r\n```\r\n\r\n### Relevant log output\r\n\r\n_No response_\r\n\r\n### Environment\r\n\r\n```markdown\r\n- OS: Mac OS 13.5.2 (22G91)\r\n- Python: 3.9\r\n- dbt: 1.5.6\r\n```\r\n\r\n\r\n### Which database adapter are you using with dbt?\r\n\r\nsnowflake\r\n\r\n### Additional Context\r\n\r\n_No response_\n", "before_files": [{"content": "from typing import Dict\n\nfrom dbt.contracts.project import RegistryPackageMetadata, TarballPackage\nfrom dbt.deps.base import PinnedPackage, UnpinnedPackage\n\n\nclass TarballPackageMixin:\n    def __init__(self, tarball: str) -> None:\n        super().__init__()\n        self.tarball = tarball\n\n    @property\n    def name(self):\n        return self.tarball\n\n    def source_type(self) -> str:\n        return \"tarball\"\n\n\nclass TarballPinnedPackage(TarballPackageMixin, PinnedPackage):\n    def __init__(self, tarball: str, package: str) -> None:\n        super().__init__(tarball)\n        # setup to recycle RegistryPinnedPackage fns\n        self.package = package\n        self.version = \"tarball\"\n\n    @property\n    def name(self):\n        return self.package\n\n    def to_dict(self) -> Dict[str, str]:\n        return {\n            \"tarball\": self.tarball,\n            \"version\": self.version,\n            \"package\": self.package,\n        }\n\n    def get_version(self):\n        return self.version\n\n    def nice_version_name(self):\n        return f\"tarball (url: {self.tarball})\"\n\n    def _fetch_metadata(self, project, renderer):\n        \"\"\"\n        recycle RegistryPackageMetadata so that we can use the install and\n        download_and_untar from RegistryPinnedPackage next.\n        build RegistryPackageMetadata from info passed via packages.yml since no\n        'metadata' service exists in this case.\n        \"\"\"\n\n        dct = {\n            \"name\": self.package,\n            \"packages\": [],  # note: required by RegistryPackageMetadata\n            \"downloads\": {\"tarball\": self.tarball},\n        }\n\n        return RegistryPackageMetadata.from_dict(dct)\n\n    def install(self, project, renderer):\n        self._install(project, renderer)\n\n\nclass TarballUnpinnedPackage(TarballPackageMixin, UnpinnedPackage[TarballPinnedPackage]):\n    def __init__(\n        self,\n        tarball: str,\n        package: str,\n    ) -> None:\n        super().__init__(tarball)\n        # setup to recycle RegistryPinnedPackage fns\n        self.package = package\n        self.version = \"tarball\"\n\n    @classmethod\n    def from_contract(cls, contract: TarballPackage) -> \"TarballUnpinnedPackage\":\n        return cls(tarball=contract.tarball, package=contract.name)\n\n    def incorporate(self, other: \"TarballUnpinnedPackage\") -> \"TarballUnpinnedPackage\":\n        return TarballUnpinnedPackage(tarball=self.tarball, package=self.package)\n\n    def resolved(self) -> TarballPinnedPackage:\n        return TarballPinnedPackage(tarball=self.tarball, package=self.package)\n", "path": "core/dbt/deps/tarball.py"}], "after_files": [{"content": "import functools\nimport os\nfrom pathlib import Path\nfrom typing import Dict\n\nfrom dbt.clients import system\nfrom dbt.config.project import PartialProject\nfrom dbt.contracts.project import TarballPackage\nfrom dbt.deps.base import PinnedPackage, UnpinnedPackage, get_downloads_path\nfrom dbt.exceptions import DependencyError\nfrom dbt.utils import _connection_exception_retry as connection_exception_retry\n\n\nclass TarballPackageMixin:\n    def __init__(self, tarball: str) -> None:\n        super().__init__()\n        self.tarball = tarball\n\n    @property\n    def name(self):\n        return self.tarball\n\n    def source_type(self) -> str:\n        return \"tarball\"\n\n\nclass TarballPinnedPackage(TarballPackageMixin, PinnedPackage):\n    def __init__(self, tarball: str, package: str) -> None:\n        super().__init__(tarball)\n        self.package = package\n        self.version = \"tarball\"\n        self.tar_path = os.path.join(Path(get_downloads_path()), self.package)\n        self.untarred_path = f\"{self.tar_path}_untarred\"\n\n    @property\n    def name(self):\n        return self.package\n\n    def to_dict(self) -> Dict[str, str]:\n        return {\n            \"tarball\": self.tarball,\n            \"name\": self.package,\n        }\n\n    def get_version(self):\n        return self.version\n\n    def nice_version_name(self):\n        return f\"tarball (url: {self.tarball})\"\n\n    def _fetch_metadata(self, project, renderer):\n        \"\"\"Download and untar the project and parse metadata from the project folder.\"\"\"\n        download_untar_fn = functools.partial(\n            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name\n        )\n        connection_exception_retry(download_untar_fn, 5)\n\n        tar_contents = os.listdir(self.untarred_path)\n        if len(tar_contents) != 1:\n            raise DependencyError(\n                f\"Incorrect structure for package extracted from {self.tarball}.\"\n                f\"The extracted package needs to follow the structure {self.name}/<package_contents>.\"\n            )\n        child_folder = os.listdir(self.untarred_path)[0]\n\n        self.untarred_path = os.path.join(self.untarred_path, child_folder)\n        partial = PartialProject.from_project_root(self.untarred_path)\n        metadata = partial.render_package_metadata(renderer)\n        metadata.name = self.package if self.package else metadata.name\n        return metadata\n\n    def install(self, project, renderer):\n        download_untar_fn = functools.partial(\n            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name\n        )\n        connection_exception_retry(download_untar_fn, 5)\n        dest_path = self.get_installation_path(project, renderer)\n        if os.path.exists(dest_path):\n            if system.path_is_symlink(dest_path):\n                system.remove_file(dest_path)\n            else:\n                system.rmdir(dest_path)\n        system.move(self.untarred_path, dest_path)\n\n\nclass TarballUnpinnedPackage(TarballPackageMixin, UnpinnedPackage[TarballPinnedPackage]):\n    def __init__(\n        self,\n        tarball: str,\n        package: str,\n    ) -> None:\n        super().__init__(tarball)\n        # setup to recycle RegistryPinnedPackage fns\n        self.package = package\n        self.version = \"tarball\"\n\n    @classmethod\n    def from_contract(cls, contract: TarballPackage) -> \"TarballUnpinnedPackage\":\n        return cls(tarball=contract.tarball, package=contract.name)\n\n    def incorporate(self, other: \"TarballUnpinnedPackage\") -> \"TarballUnpinnedPackage\":\n        return TarballUnpinnedPackage(tarball=self.tarball, package=self.package)\n\n    def resolved(self) -> TarballPinnedPackage:\n        return TarballPinnedPackage(tarball=self.tarball, package=self.package)\n", "path": "core/dbt/deps/tarball.py"}]}