We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Issues with dvc get and dvc get-url after 1.0
Hi,

I've been excited to explore DVC as I think it can offer a lot for some of the things I'm working on. I've had a couple of issues with `dvc get` and `dvc get-url` that I *think* are probably bugs. I've put them together to avoid creating lots of issues; apologies if you'd rather they were separate, I can move them into different threads.

I am testing as of v1.1.1.

## dvc get does not recreate directories correctly

After v1.0 `dvc get` pointed to a directory does not result in correctly placed files. All of the files are placed in the root, rathe than subdirectories. I've only tested this with a git repository that is not a DVC repository. It worked in v0.94, but does not work correctly in v1.0 or v1.1. `dvc import` works correctly.

The following example gets [this directory](https://github.com/explosion/projects/tree/master/nel-emerson/scripts):

```
$ dvc get https://github.com/explosion/projects nel-emerson
$ tree nel-emerson/
nel-emerson/
├── README.md
├── __init__.py
├── el_recipe.py
├── el_tutorial.py
├── emerson_annotated_text.jsonl
├── emerson_input_text.txt
├── entities.csv
├── nel_schema.png
├── notebook_video.ipynb
└── requirements.txt
0 directories, 10 files
```

And here's `dvc import`, which works correctly:

```
$ dvc import https://github.com/explosion/projects nel-emerson
$ tree nel-emerson/
nel-emerson/
├── README.md
├── __init__.py
├── input
│   └── entities.csv
├── prodigy
│   ├── emerson_annotated_text.jsonl
│   └── emerson_input_text.txt
├── requirements.txt
└── scripts
    ├── el_recipe.py
    ├── el_tutorial.py
    ├── nel_schema.png
    └── notebook_video.ipynb
3 directories, 10 files
```

## dvc get-url fails on URLs with weak ETAGs

I understand why `import-url` needs the strong ETag, but I don't think it makes sense for `get-url`, so I suspect this is a bug. If so it's a regression after v1.0, as it didn't happen in v0.94.

```
$ dvc get-url https://raw.githubusercontent.com/explosion/projects/master/nel-emerson/requirements.txt
ERROR: failed to get 'https://raw.githubusercontent.com/explosion/projects/master/nel-emerson/requirements.txt' - Weak ETags are not supported. (Etag: 'W/"24c3b7f31292b04ac8fa9938c6e14baed94a089944652e8abf7dbb9459ae5b56"', URL: 'https://raw.githubusercontent.com/explosion/projects/master/nel-emerson/requirements.txt')
```

## dvc get-url does not follow redirects

This is maybe more of a feature request than a bug report, although if it's expected behaviour, I think at least the error message needs to be different. I also think it's an important feature, as it's pretty common to have an https URL sitting in front of an AWS bucket or other file storage, as direct storage URLs don't allow much flexibility for moving data.

There's also a specific, very common scenario where this comes up: binary data such as trained models are often attached to Github releases as release artefacts. These artefacts are addressed as URLs that redirect to S3. For instance:

```
$ dvc get-url https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz
ERROR: failed to get 'https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz' - dependency 'https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz' does not exist
```

While `wget` works fine:

```
wget https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz
--2020-06-29 19:29:31--  https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz
Resolving github.com (github.com)... 140.82.118.4
Connecting to github.com (github.com)|140.82.118.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
...
Saving to: ‘ro_core_news_sm-2.3.1.tar.gz
```



--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `dvc/tree/http.py`
Content:
```
1 import logging
2 import os.path
3 import threading
4 
5 from funcy import cached_property, memoize, wrap_prop, wrap_with
6 
7 import dvc.prompt as prompt
8 from dvc.exceptions import DvcException, HTTPError
9 from dvc.path_info import HTTPURLInfo
10 from dvc.progress import Tqdm
11 from dvc.scheme import Schemes
12 
13 from .base import BaseTree
14 
15 logger = logging.getLogger(__name__)
16 
17 
18 @wrap_with(threading.Lock())
19 @memoize
20 def ask_password(host, user):
21     return prompt.password(
22         "Enter a password for "
23         "host '{host}' user '{user}'".format(host=host, user=user)
24     )
25 
26 
27 class HTTPTree(BaseTree):  # pylint:disable=abstract-method
28     scheme = Schemes.HTTP
29     PATH_CLS = HTTPURLInfo
30     PARAM_CHECKSUM = "etag"
31     CAN_TRAVERSE = False
32 
33     SESSION_RETRIES = 5
34     SESSION_BACKOFF_FACTOR = 0.1
35     REQUEST_TIMEOUT = 10
36     CHUNK_SIZE = 2 ** 16
37 
38     def __init__(self, repo, config):
39         super().__init__(repo, config)
40 
41         url = config.get("url")
42         if url:
43             self.path_info = self.PATH_CLS(url)
44             user = config.get("user", None)
45             if user:
46                 self.path_info.user = user
47         else:
48             self.path_info = None
49 
50         self.auth = config.get("auth", None)
51         self.custom_auth_header = config.get("custom_auth_header", None)
52         self.password = config.get("password", None)
53         self.ask_password = config.get("ask_password", False)
54         self.headers = {}
55 
56     def _auth_method(self, path_info=None):
57         from requests.auth import HTTPBasicAuth, HTTPDigestAuth
58 
59         if path_info is None:
60             path_info = self.path_info
61 
62         if self.auth:
63             if self.ask_password and self.password is None:
64                 host, user = path_info.host, path_info.user
65                 self.password = ask_password(host, user)
66             if self.auth == "basic":
67                 return HTTPBasicAuth(path_info.user, self.password)
68             if self.auth == "digest":
69                 return HTTPDigestAuth(path_info.user, self.password)
70             if self.auth == "custom" and self.custom_auth_header:
71                 self.headers.update({self.custom_auth_header: self.password})
72         return None
73 
74     @wrap_prop(threading.Lock())
75     @cached_property
76     def _session(self):
77         import requests
78         from requests.adapters import HTTPAdapter
79         from urllib3.util.retry import Retry
80 
81         session = requests.Session()
82 
83         retries = Retry(
84             total=self.SESSION_RETRIES,
85             backoff_factor=self.SESSION_BACKOFF_FACTOR,
86         )
87 
88         session.mount("http://", HTTPAdapter(max_retries=retries))
89         session.mount("https://", HTTPAdapter(max_retries=retries))
90 
91         return session
92 
93     def request(self, method, url, **kwargs):
94         import requests
95 
96         kwargs.setdefault("allow_redirects", True)
97         kwargs.setdefault("timeout", self.REQUEST_TIMEOUT)
98 
99         try:
100             res = self._session.request(
101                 method,
102                 url,
103                 auth=self._auth_method(),
104                 headers=self.headers,
105                 **kwargs,
106             )
107 
108             redirect_no_location = (
109                 kwargs["allow_redirects"]
110                 and res.status_code in (301, 302)
111                 and "location" not in res.headers
112             )
113 
114             if redirect_no_location:
115                 # AWS s3 doesn't like to add a location header to its redirects
116                 # from https://s3.amazonaws.com/<bucket name>/* type URLs.
117                 # This should be treated as an error
118                 raise requests.exceptions.RequestException
119 
120             return res
121 
122         except requests.exceptions.RequestException:
123             raise DvcException(f"could not perform a {method} request")
124 
125     def exists(self, path_info, use_dvcignore=True):
126         return bool(self.request("HEAD", path_info.url))
127 
128     def get_file_hash(self, path_info):
129         url = path_info.url
130         headers = self.request("HEAD", url).headers
131         etag = headers.get("ETag") or headers.get("Content-MD5")
132 
133         if not etag:
134             raise DvcException(
135                 "could not find an ETag or "
136                 "Content-MD5 header for '{url}'".format(url=url)
137             )
138 
139         return etag
140 
141     def _download(self, from_info, to_file, name=None, no_progress_bar=False):
142         response = self.request("GET", from_info.url, stream=True)
143         if response.status_code != 200:
144             raise HTTPError(response.status_code, response.reason)
145         with open(to_file, "wb") as fd:
146             with Tqdm.wrapattr(
147                 fd,
148                 "write",
149                 total=None
150                 if no_progress_bar
151                 else self._content_length(response),
152                 leave=False,
153                 desc=from_info.url if name is None else name,
154                 disable=no_progress_bar,
155             ) as fd_wrapped:
156                 for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):
157                     fd_wrapped.write(chunk)
158 
159     def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
160         def chunks():
161             with open(from_file, "rb") as fd:
162                 with Tqdm.wrapattr(
163                     fd,
164                     "read",
165                     total=None
166                     if no_progress_bar
167                     else os.path.getsize(from_file),
168                     leave=False,
169                     desc=to_info.url if name is None else name,
170                     disable=no_progress_bar,
171                 ) as fd_wrapped:
172                     while True:
173                         chunk = fd_wrapped.read(self.CHUNK_SIZE)
174                         if not chunk:
175                             break
176                         yield chunk
177 
178         response = self.request("POST", to_info.url, data=chunks())
179         if response.status_code not in (200, 201):
180             raise HTTPError(response.status_code, response.reason)
181 
182     @staticmethod
183     def _content_length(response):
184         res = response.headers.get("Content-Length")
185         return int(res) if res else None
186
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/dvc/tree/http.py b/dvc/tree/http.py
--- a/dvc/tree/http.py
+++ b/dvc/tree/http.py
@@ -122,12 +122,27 @@
         except requests.exceptions.RequestException:
             raise DvcException(f"could not perform a {method} request")
 
+    def _head(self, url):
+        response = self.request("HEAD", url)
+        if response.ok:
+            return response
+
+        # Sometimes servers are configured to forbid HEAD requests
+        # Context: https://github.com/iterative/dvc/issues/4131
+        with self.request("GET", url, stream=True) as r:
+            if r.ok:
+                return r
+
+        return response
+
     def exists(self, path_info, use_dvcignore=True):
-        return bool(self.request("HEAD", path_info.url))
+        return bool(self._head(path_info.url))
 
     def get_file_hash(self, path_info):
         url = path_info.url
-        headers = self.request("HEAD", url).headers
+
+        headers = self._head(url).headers
+
         etag = headers.get("ETag") or headers.get("Content-MD5")
 
         if not etag:

{"golden_diff": "diff --git a/dvc/tree/http.py b/dvc/tree/http.py\n--- a/dvc/tree/http.py\n+++ b/dvc/tree/http.py\n@@ -122,12 +122,27 @@\n         except requests.exceptions.RequestException:\n             raise DvcException(f\"could not perform a {method} request\")\n \n+    def _head(self, url):\n+        response = self.request(\"HEAD\", url)\n+        if response.ok:\n+            return response\n+\n+        # Sometimes servers are configured to forbid HEAD requests\n+        # Context: https://github.com/iterative/dvc/issues/4131\n+        with self.request(\"GET\", url, stream=True) as r:\n+            if r.ok:\n+                return r\n+\n+        return response\n+\n     def exists(self, path_info, use_dvcignore=True):\n-        return bool(self.request(\"HEAD\", path_info.url))\n+        return bool(self._head(path_info.url))\n \n     def get_file_hash(self, path_info):\n         url = path_info.url\n-        headers = self.request(\"HEAD\", url).headers\n+\n+        headers = self._head(url).headers\n+\n         etag = headers.get(\"ETag\") or headers.get(\"Content-MD5\")\n \n         if not etag:\n", "issue": "Issues with dvc get and dvc get-url after 1.0\nHi,\r\n\r\nI've been excited to explore DVC as I think it can offer a lot for some of the things I'm working on. I've had a couple of issues with `dvc get` and `dvc get-url` that I *think* are probably bugs. I've put them together to avoid creating lots of issues; apologies if you'd rather they were separate, I can move them into different threads.\r\n\r\nI am testing as of v1.1.1.\r\n\r\n## dvc get does not recreate directories correctly\r\n\r\nAfter v1.0 `dvc get` pointed to a directory does not result in correctly placed files. All of the files are placed in the root, rathe than subdirectories. I've only tested this with a git repository that is not a DVC repository. It worked in v0.94, but does not work correctly in v1.0 or v1.1. `dvc import` works correctly.\r\n\r\nThe following example gets [this directory](https://github.com/explosion/projects/tree/master/nel-emerson/scripts):\r\n\r\n```\r\n$ dvc get https://github.com/explosion/projects nel-emerson\r\n$ tree nel-emerson/\r\nnel-emerson/\r\n\u251c\u2500\u2500 README.md\r\n\u251c\u2500\u2500 __init__.py\r\n\u251c\u2500\u2500 el_recipe.py\r\n\u251c\u2500\u2500 el_tutorial.py\r\n\u251c\u2500\u2500 emerson_annotated_text.jsonl\r\n\u251c\u2500\u2500 emerson_input_text.txt\r\n\u251c\u2500\u2500 entities.csv\r\n\u251c\u2500\u2500 nel_schema.png\r\n\u251c\u2500\u2500 notebook_video.ipynb\r\n\u2514\u2500\u2500 requirements.txt\r\n0 directories, 10 files\r\n```\r\n\r\nAnd here's `dvc import`, which works correctly:\r\n\r\n```\r\n$ dvc import https://github.com/explosion/projects nel-emerson\r\n$ tree nel-emerson/\r\nnel-emerson/\r\n\u251c\u2500\u2500 README.md\r\n\u251c\u2500\u2500 __init__.py\r\n\u251c\u2500\u2500 input\r\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 entities.csv\r\n\u251c\u2500\u2500 prodigy\r\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 emerson_annotated_text.jsonl\r\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 emerson_input_text.txt\r\n\u251c\u2500\u2500 requirements.txt\r\n\u2514\u2500\u2500 scripts\r\n    \u251c\u2500\u2500 el_recipe.py\r\n    \u251c\u2500\u2500 el_tutorial.py\r\n    \u251c\u2500\u2500 nel_schema.png\r\n    \u2514\u2500\u2500 notebook_video.ipynb\r\n3 directories, 10 files\r\n```\r\n\r\n## dvc get-url fails on URLs with weak ETAGs\r\n\r\nI understand why `import-url` needs the strong ETag, but I don't think it makes sense for `get-url`, so I suspect this is a bug. If so it's a regression after v1.0, as it didn't happen in v0.94.\r\n\r\n```\r\n$ dvc get-url https://raw.githubusercontent.com/explosion/projects/master/nel-emerson/requirements.txt\r\nERROR: failed to get 'https://raw.githubusercontent.com/explosion/projects/master/nel-emerson/requirements.txt' - Weak ETags are not supported. (Etag: 'W/\"24c3b7f31292b04ac8fa9938c6e14baed94a089944652e8abf7dbb9459ae5b56\"', URL: 'https://raw.githubusercontent.com/explosion/projects/master/nel-emerson/requirements.txt')\r\n```\r\n\r\n## dvc get-url does not follow redirects\r\n\r\nThis is maybe more of a feature request than a bug report, although if it's expected behaviour, I think at least the error message needs to be different. I also think it's an important feature, as it's pretty common to have an https URL sitting in front of an AWS bucket or other file storage, as direct storage URLs don't allow much flexibility for moving data.\r\n\r\nThere's also a specific, very common scenario where this comes up: binary data such as trained models are often attached to Github releases as release artefacts. These artefacts are addressed as URLs that redirect to S3. For instance:\r\n\r\n```\r\n$ dvc get-url https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz\r\nERROR: failed to get 'https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz' - dependency 'https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz' does not exist\r\n```\r\n\r\nWhile `wget` works fine:\r\n\r\n```\r\nwget https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz\r\n--2020-06-29 19:29:31--  https://github.com/explosion/spacy-models/releases/download/ro_core_news_sm-2.3.1/ro_core_news_sm-2.3.1.tar.gz\r\nResolving github.com (github.com)... 140.82.118.4\r\nConnecting to github.com (github.com)|140.82.118.4|:443... connected.\r\nHTTP request sent, awaiting response... 302 Found\r\n...\r\nSaving to: \u2018ro_core_news_sm-2.3.1.tar.gz\r\n```\r\n\r\n\n", "before_files": [{"content": "import logging\nimport os.path\nimport threading\n\nfrom funcy import cached_property, memoize, wrap_prop, wrap_with\n\nimport dvc.prompt as prompt\nfrom dvc.exceptions import DvcException, HTTPError\nfrom dvc.path_info import HTTPURLInfo\nfrom dvc.progress import Tqdm\nfrom dvc.scheme import Schemes\n\nfrom .base import BaseTree\n\nlogger = logging.getLogger(__name__)\n\n\n@wrap_with(threading.Lock())\n@memoize\ndef ask_password(host, user):\n    return prompt.password(\n        \"Enter a password for \"\n        \"host '{host}' user '{user}'\".format(host=host, user=user)\n    )\n\n\nclass HTTPTree(BaseTree):  # pylint:disable=abstract-method\n    scheme = Schemes.HTTP\n    PATH_CLS = HTTPURLInfo\n    PARAM_CHECKSUM = \"etag\"\n    CAN_TRAVERSE = False\n\n    SESSION_RETRIES = 5\n    SESSION_BACKOFF_FACTOR = 0.1\n    REQUEST_TIMEOUT = 10\n    CHUNK_SIZE = 2 ** 16\n\n    def __init__(self, repo, config):\n        super().__init__(repo, config)\n\n        url = config.get(\"url\")\n        if url:\n            self.path_info = self.PATH_CLS(url)\n            user = config.get(\"user\", None)\n            if user:\n                self.path_info.user = user\n        else:\n            self.path_info = None\n\n        self.auth = config.get(\"auth\", None)\n        self.custom_auth_header = config.get(\"custom_auth_header\", None)\n        self.password = config.get(\"password\", None)\n        self.ask_password = config.get(\"ask_password\", False)\n        self.headers = {}\n\n    def _auth_method(self, path_info=None):\n        from requests.auth import HTTPBasicAuth, HTTPDigestAuth\n\n        if path_info is None:\n            path_info = self.path_info\n\n        if self.auth:\n            if self.ask_password and self.password is None:\n                host, user = path_info.host, path_info.user\n                self.password = ask_password(host, user)\n            if self.auth == \"basic\":\n                return HTTPBasicAuth(path_info.user, self.password)\n            if self.auth == \"digest\":\n                return HTTPDigestAuth(path_info.user, self.password)\n            if self.auth == \"custom\" and self.custom_auth_header:\n                self.headers.update({self.custom_auth_header: self.password})\n        return None\n\n    @wrap_prop(threading.Lock())\n    @cached_property\n    def _session(self):\n        import requests\n        from requests.adapters import HTTPAdapter\n        from urllib3.util.retry import Retry\n\n        session = requests.Session()\n\n        retries = Retry(\n            total=self.SESSION_RETRIES,\n            backoff_factor=self.SESSION_BACKOFF_FACTOR,\n        )\n\n        session.mount(\"http://\", HTTPAdapter(max_retries=retries))\n        session.mount(\"https://\", HTTPAdapter(max_retries=retries))\n\n        return session\n\n    def request(self, method, url, **kwargs):\n        import requests\n\n        kwargs.setdefault(\"allow_redirects\", True)\n        kwargs.setdefault(\"timeout\", self.REQUEST_TIMEOUT)\n\n        try:\n            res = self._session.request(\n                method,\n                url,\n                auth=self._auth_method(),\n                headers=self.headers,\n                **kwargs,\n            )\n\n            redirect_no_location = (\n                kwargs[\"allow_redirects\"]\n                and res.status_code in (301, 302)\n                and \"location\" not in res.headers\n            )\n\n            if redirect_no_location:\n                # AWS s3 doesn't like to add a location header to its redirects\n                # from https://s3.amazonaws.com/<bucket name>/* type URLs.\n                # This should be treated as an error\n                raise requests.exceptions.RequestException\n\n            return res\n\n        except requests.exceptions.RequestException:\n            raise DvcException(f\"could not perform a {method} request\")\n\n    def exists(self, path_info, use_dvcignore=True):\n        return bool(self.request(\"HEAD\", path_info.url))\n\n    def get_file_hash(self, path_info):\n        url = path_info.url\n        headers = self.request(\"HEAD\", url).headers\n        etag = headers.get(\"ETag\") or headers.get(\"Content-MD5\")\n\n        if not etag:\n            raise DvcException(\n                \"could not find an ETag or \"\n                \"Content-MD5 header for '{url}'\".format(url=url)\n            )\n\n        return etag\n\n    def _download(self, from_info, to_file, name=None, no_progress_bar=False):\n        response = self.request(\"GET\", from_info.url, stream=True)\n        if response.status_code != 200:\n            raise HTTPError(response.status_code, response.reason)\n        with open(to_file, \"wb\") as fd:\n            with Tqdm.wrapattr(\n                fd,\n                \"write\",\n                total=None\n                if no_progress_bar\n                else self._content_length(response),\n                leave=False,\n                desc=from_info.url if name is None else name,\n                disable=no_progress_bar,\n            ) as fd_wrapped:\n                for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):\n                    fd_wrapped.write(chunk)\n\n    def _upload(self, from_file, to_info, name=None, no_progress_bar=False):\n        def chunks():\n            with open(from_file, \"rb\") as fd:\n                with Tqdm.wrapattr(\n                    fd,\n                    \"read\",\n                    total=None\n                    if no_progress_bar\n                    else os.path.getsize(from_file),\n                    leave=False,\n                    desc=to_info.url if name is None else name,\n                    disable=no_progress_bar,\n                ) as fd_wrapped:\n                    while True:\n                        chunk = fd_wrapped.read(self.CHUNK_SIZE)\n                        if not chunk:\n                            break\n                        yield chunk\n\n        response = self.request(\"POST\", to_info.url, data=chunks())\n        if response.status_code not in (200, 201):\n            raise HTTPError(response.status_code, response.reason)\n\n    @staticmethod\n    def _content_length(response):\n        res = response.headers.get(\"Content-Length\")\n        return int(res) if res else None\n", "path": "dvc/tree/http.py"}], "after_files": [{"content": "import logging\nimport os.path\nimport threading\n\nfrom funcy import cached_property, memoize, wrap_prop, wrap_with\n\nimport dvc.prompt as prompt\nfrom dvc.exceptions import DvcException, HTTPError\nfrom dvc.path_info import HTTPURLInfo\nfrom dvc.progress import Tqdm\nfrom dvc.scheme import Schemes\n\nfrom .base import BaseTree\n\nlogger = logging.getLogger(__name__)\n\n\n@wrap_with(threading.Lock())\n@memoize\ndef ask_password(host, user):\n    return prompt.password(\n        \"Enter a password for \"\n        \"host '{host}' user '{user}'\".format(host=host, user=user)\n    )\n\n\nclass HTTPTree(BaseTree):  # pylint:disable=abstract-method\n    scheme = Schemes.HTTP\n    PATH_CLS = HTTPURLInfo\n    PARAM_CHECKSUM = \"etag\"\n    CAN_TRAVERSE = False\n\n    SESSION_RETRIES = 5\n    SESSION_BACKOFF_FACTOR = 0.1\n    REQUEST_TIMEOUT = 10\n    CHUNK_SIZE = 2 ** 16\n\n    def __init__(self, repo, config):\n        super().__init__(repo, config)\n\n        url = config.get(\"url\")\n        if url:\n            self.path_info = self.PATH_CLS(url)\n            user = config.get(\"user\", None)\n            if user:\n                self.path_info.user = user\n        else:\n            self.path_info = None\n\n        self.auth = config.get(\"auth\", None)\n        self.custom_auth_header = config.get(\"custom_auth_header\", None)\n        self.password = config.get(\"password\", None)\n        self.ask_password = config.get(\"ask_password\", False)\n        self.headers = {}\n\n    def _auth_method(self, path_info=None):\n        from requests.auth import HTTPBasicAuth, HTTPDigestAuth\n\n        if path_info is None:\n            path_info = self.path_info\n\n        if self.auth:\n            if self.ask_password and self.password is None:\n                host, user = path_info.host, path_info.user\n                self.password = ask_password(host, user)\n            if self.auth == \"basic\":\n                return HTTPBasicAuth(path_info.user, self.password)\n            if self.auth == \"digest\":\n                return HTTPDigestAuth(path_info.user, self.password)\n            if self.auth == \"custom\" and self.custom_auth_header:\n                self.headers.update({self.custom_auth_header: self.password})\n        return None\n\n    @wrap_prop(threading.Lock())\n    @cached_property\n    def _session(self):\n        import requests\n        from requests.adapters import HTTPAdapter\n        from urllib3.util.retry import Retry\n\n        session = requests.Session()\n\n        retries = Retry(\n            total=self.SESSION_RETRIES,\n            backoff_factor=self.SESSION_BACKOFF_FACTOR,\n        )\n\n        session.mount(\"http://\", HTTPAdapter(max_retries=retries))\n        session.mount(\"https://\", HTTPAdapter(max_retries=retries))\n\n        return session\n\n    def request(self, method, url, **kwargs):\n        import requests\n\n        kwargs.setdefault(\"allow_redirects\", True)\n        kwargs.setdefault(\"timeout\", self.REQUEST_TIMEOUT)\n\n        try:\n            res = self._session.request(\n                method,\n                url,\n                auth=self._auth_method(),\n                headers=self.headers,\n                **kwargs,\n            )\n\n            redirect_no_location = (\n                kwargs[\"allow_redirects\"]\n                and res.status_code in (301, 302)\n                and \"location\" not in res.headers\n            )\n\n            if redirect_no_location:\n                # AWS s3 doesn't like to add a location header to its redirects\n                # from https://s3.amazonaws.com/<bucket name>/* type URLs.\n                # This should be treated as an error\n                raise requests.exceptions.RequestException\n\n            return res\n\n        except requests.exceptions.RequestException:\n            raise DvcException(f\"could not perform a {method} request\")\n\n    def _head(self, url):\n        response = self.request(\"HEAD\", url)\n        if response.ok:\n            return response\n\n        # Sometimes servers are configured to forbid HEAD requests\n        # Context: https://github.com/iterative/dvc/issues/4131\n        with self.request(\"GET\", url, stream=True) as r:\n            if r.ok:\n                return r\n\n        return response\n\n    def exists(self, path_info, use_dvcignore=True):\n        return bool(self._head(path_info.url))\n\n    def get_file_hash(self, path_info):\n        url = path_info.url\n\n        headers = self._head(url).headers\n\n        etag = headers.get(\"ETag\") or headers.get(\"Content-MD5\")\n\n        if not etag:\n            raise DvcException(\n                \"could not find an ETag or \"\n                \"Content-MD5 header for '{url}'\".format(url=url)\n            )\n\n        return etag\n\n    def _download(self, from_info, to_file, name=None, no_progress_bar=False):\n        response = self.request(\"GET\", from_info.url, stream=True)\n        if response.status_code != 200:\n            raise HTTPError(response.status_code, response.reason)\n        with open(to_file, \"wb\") as fd:\n            with Tqdm.wrapattr(\n                fd,\n                \"write\",\n                total=None\n                if no_progress_bar\n                else self._content_length(response),\n                leave=False,\n                desc=from_info.url if name is None else name,\n                disable=no_progress_bar,\n            ) as fd_wrapped:\n                for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):\n                    fd_wrapped.write(chunk)\n\n    def _upload(self, from_file, to_info, name=None, no_progress_bar=False):\n        def chunks():\n            with open(from_file, \"rb\") as fd:\n                with Tqdm.wrapattr(\n                    fd,\n                    \"read\",\n                    total=None\n                    if no_progress_bar\n                    else os.path.getsize(from_file),\n                    leave=False,\n                    desc=to_info.url if name is None else name,\n                    disable=no_progress_bar,\n                ) as fd_wrapped:\n                    while True:\n                        chunk = fd_wrapped.read(self.CHUNK_SIZE)\n                        if not chunk:\n                            break\n                        yield chunk\n\n        response = self.request(\"POST\", to_info.url, data=chunks())\n        if response.status_code not in (200, 201):\n            raise HTTPError(response.status_code, response.reason)\n\n    @staticmethod\n    def _content_length(response):\n        res = response.headers.get(\"Content-Length\")\n        return int(res) if res else None\n", "path": "dvc/tree/http.py"}]}