prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  `dvc pull` with wrong S3 remote failed but user wasn't informed
  System
  ----------
  
  My `dvc` version is 0.77.3, I installed & upgraded `dvc` through `pip3` and I'm using Fedora 31.
  
  Problem
  -----------
  
  I followed the 'Getting Started' guide and ran the following commands
  
  ```
  $ dvc init
  $ git add .dvc/* && git commit -m 'Initialized dvc'
  $ dvc remote add -d s3remote https://s3.amazonaws.com/<bucket-name>
  $ dvc add <some-file>
  $ git add .gitignore <some-file>.dvc && git commit -m 'Added <some-file'
  $ dvc push -r s3remote
  ```
  The last command finished with `Everything is up to date.` which is weird since the S3 bucket was empty when checking it. I went on to delete `<some-file>` and run `dvc pull -r s3remote` which would restore `<some-file>` without throwing any errors. This was surprising since it did not get clear where `dvc` is storing my file backups and why my S3 bucket was still empty without `dvc` communicating any errors.
  
  Only when I removed the `.dvc/cache` folder and ran `dvc pull -r s3remote` it would complain with the following:
  
  ```
  ERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/31/69f7ce4ebb503afca037d35b7eb3a9' to '.dvc/cache/31/69f7ce4ebb503afca037d35b7eb3a9' - '301 Moved Permanently'
  
  ERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/a3/04afb96060aad90176268345e10355' to '.dvc/cache/a3/04afb96060aad90176268345e10355' - '301 Moved Permanently'
  
  ERROR: failed to pull data from the cloud - 2 files failed to download
  
  
  Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
  ```
  
  The fix
  ---------
  
  On my end, it was an easy fix. After consulting the documentation in more detail, I realized that I set up my S3 remote incorrectly using the `https://s3.amazonaws.com/<bucket-name>` URL to the bucket whereas I should have used the `s3://<bucket-name>` URL. Hence, running
  
  ```
  $ dvc remote modify s3remote url s3://<bucket-name>
  ```
  
  did the job.
  
  It still remains weird why I was never warned about the remote being incorrectly set up and why `dvc pull -r s3remote` worked without any problems even though there were no files in the S3 bucket (it seems that it restored them from local cache as a fall-back) but it should have given me a headsup!
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `dvc/remote/http.py`
  Content:
  ```
  1 import logging
  2 import threading
  3 
  4 from funcy import cached_property, wrap_prop
  5 
  6 from dvc.config import Config
  7 from dvc.config import ConfigError
  8 from dvc.exceptions import DvcException, HTTPError
  9 from dvc.progress import Tqdm
  10 from dvc.remote.base import RemoteBASE
  11 from dvc.scheme import Schemes
  12 
  13 logger = logging.getLogger(__name__)
  14 
  15 
  16 class RemoteHTTP(RemoteBASE):
  17     scheme = Schemes.HTTP
  18     SESSION_RETRIES = 5
  19     SESSION_BACKOFF_FACTOR = 0.1
  20     REQUEST_TIMEOUT = 10
  21     CHUNK_SIZE = 2 ** 16
  22     PARAM_CHECKSUM = "etag"
  23 
  24     def __init__(self, repo, config):
  25         super().__init__(repo, config)
  26 
  27         url = config.get(Config.SECTION_REMOTE_URL)
  28         self.path_info = self.path_cls(url) if url else None
  29 
  30         if not self.no_traverse:
  31             raise ConfigError(
  32                 "HTTP doesn't support traversing the remote to list existing "
  33                 "files. Use: `dvc remote modify <name> no_traverse true`"
  34             )
  35 
  36     def _download(self, from_info, to_file, name=None, no_progress_bar=False):
  37         response = self._request("GET", from_info.url, stream=True)
  38         if response.status_code != 200:
  39             raise HTTPError(response.status_code, response.reason)
  40         with Tqdm(
  41             total=None if no_progress_bar else self._content_length(response),
  42             leave=False,
  43             bytes=True,
  44             desc=from_info.url if name is None else name,
  45             disable=no_progress_bar,
  46         ) as pbar:
  47             with open(to_file, "wb") as fd:
  48                 for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):
  49                     fd.write(chunk)
  50                     pbar.update(len(chunk))
  51 
  52     def exists(self, path_info):
  53         return bool(self._request("HEAD", path_info.url))
  54 
  55     def _content_length(self, response):
  56         res = response.headers.get("Content-Length")
  57         return int(res) if res else None
  58 
  59     def get_file_checksum(self, path_info):
  60         url = path_info.url
  61         headers = self._request("HEAD", url).headers
  62         etag = headers.get("ETag") or headers.get("Content-MD5")
  63 
  64         if not etag:
  65             raise DvcException(
  66                 "could not find an ETag or "
  67                 "Content-MD5 header for '{url}'".format(url=url)
  68             )
  69 
  70         if etag.startswith("W/"):
  71             raise DvcException(
  72                 "Weak ETags are not supported."
  73                 " (Etag: '{etag}', URL: '{url}')".format(etag=etag, url=url)
  74             )
  75 
  76         return etag
  77 
  78     @wrap_prop(threading.Lock())
  79     @cached_property
  80     def _session(self):
  81         import requests
  82         from requests.adapters import HTTPAdapter
  83         from urllib3.util.retry import Retry
  84 
  85         session = requests.Session()
  86 
  87         retries = Retry(
  88             total=self.SESSION_RETRIES,
  89             backoff_factor=self.SESSION_BACKOFF_FACTOR,
  90         )
  91 
  92         session.mount("http://", HTTPAdapter(max_retries=retries))
  93         session.mount("https://", HTTPAdapter(max_retries=retries))
  94 
  95         return session
  96 
  97     def _request(self, method, url, **kwargs):
  98         import requests
  99 
  100         kwargs.setdefault("allow_redirects", True)
  101         kwargs.setdefault("timeout", self.REQUEST_TIMEOUT)
  102 
  103         try:
  104             return self._session.request(method, url, **kwargs)
  105         except requests.exceptions.RequestException:
  106             raise DvcException("could not perform a {} request".format(method))
  107 
  108     def gc(self):
  109         raise NotImplementedError
  110
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/dvc/remote/http.py b/dvc/remote/http.py
  --- a/dvc/remote/http.py
  +++ b/dvc/remote/http.py
  @@ -101,7 +101,22 @@
           kwargs.setdefault("timeout", self.REQUEST_TIMEOUT)
   
           try:
  -            return self._session.request(method, url, **kwargs)
  +            res = self._session.request(method, url, **kwargs)
  +
  +            redirect_no_location = (
  +                kwargs["allow_redirects"]
  +                and res.status_code in (301, 302)
  +                and "location" not in res.headers
  +            )
  +
  +            if redirect_no_location:
  +                # AWS s3 doesn't like to add a location header to its redirects
  +                # from https://s3.amazonaws.com/<bucket name>/* type URLs.
  +                # This should be treated as an error
  +                raise requests.exceptions.RequestException
  +
  +            return res
  +
           except requests.exceptions.RequestException:
               raise DvcException("could not perform a {} request".format(method))
  
verification_info: {"golden_diff": "diff --git a/dvc/remote/http.py b/dvc/remote/http.py\n--- a/dvc/remote/http.py\n+++ b/dvc/remote/http.py\n@@ -101,7 +101,22 @@\n         kwargs.setdefault(\"timeout\", self.REQUEST_TIMEOUT)\n \n         try:\n-            return self._session.request(method, url, **kwargs)\n+            res = self._session.request(method, url, **kwargs)\n+\n+            redirect_no_location = (\n+                kwargs[\"allow_redirects\"]\n+                and res.status_code in (301, 302)\n+                and \"location\" not in res.headers\n+            )\n+\n+            if redirect_no_location:\n+                # AWS s3 doesn't like to add a location header to its redirects\n+                # from https://s3.amazonaws.com/<bucket name>/* type URLs.\n+                # This should be treated as an error\n+                raise requests.exceptions.RequestException\n+\n+            return res\n+\n         except requests.exceptions.RequestException:\n             raise DvcException(\"could not perform a {} request\".format(method))\n", "issue": "`dvc pull` with wrong S3 remote failed but user wasn't informed\nSystem\r\n----------\r\n\r\nMy `dvc` version is 0.77.3, I installed & upgraded `dvc` through `pip3` and I'm using Fedora 31.\r\n\r\nProblem\r\n-----------\r\n\r\nI followed the 'Getting Started' guide and ran the following commands\r\n\r\n```\r\n$ dvc init\r\n$ git add .dvc/* && git commit -m 'Initialized dvc'\r\n$ dvc remote add -d s3remote https://s3.amazonaws.com/<bucket-name>\r\n$ dvc add <some-file>\r\n$ git add .gitignore <some-file>.dvc && git commit -m 'Added <some-file'\r\n$ dvc push -r s3remote\r\n```\r\nThe last command finished with `Everything is up to date.` which is weird since the S3 bucket was empty when checking it. I went on to delete `<some-file>` and run `dvc pull -r s3remote` which would restore `<some-file>` without throwing any errors. This was surprising since it did not get clear where `dvc` is storing my file backups and why my S3 bucket was still empty without `dvc` communicating any errors.\r\n\r\nOnly when I removed the `.dvc/cache` folder and ran `dvc pull -r s3remote` it would complain with the following:\r\n\r\n```\r\nERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/31/69f7ce4ebb503afca037d35b7eb3a9' to '.dvc/cache/31/69f7ce4ebb503afca037d35b7eb3a9' - '301 Moved Permanently'\r\n\r\nERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/a3/04afb96060aad90176268345e10355' to '.dvc/cache/a3/04afb96060aad90176268345e10355' - '301 Moved Permanently'\r\n\r\nERROR: failed to pull data from the cloud - 2 files failed to download\r\n\r\n\r\nHaving any troubles? Hit us up at https://dvc.org/support, we are always happy to help!\r\n```\r\n\r\nThe fix\r\n---------\r\n\r\nOn my end, it was an easy fix. After consulting the documentation in more detail, I realized that I set up my S3 remote incorrectly using the `https://s3.amazonaws.com/<bucket-name>` URL to the bucket whereas I should have used the `s3://<bucket-name>` URL. Hence, running\r\n\r\n```\r\n$ dvc remote modify s3remote url s3://<bucket-name>\r\n```\r\n\r\ndid the job.\r\n\r\nIt still remains weird why I was never warned about the remote being incorrectly set up and why `dvc pull -r s3remote` worked without any problems even though there were no files in the S3 bucket (it seems that it restored them from local cache as a fall-back) but it should have given me a headsup!\n", "before_files": [{"content": "import logging\nimport threading\n\nfrom funcy import cached_property, wrap_prop\n\nfrom dvc.config import Config\nfrom dvc.config import ConfigError\nfrom dvc.exceptions import DvcException, HTTPError\nfrom dvc.progress import Tqdm\nfrom dvc.remote.base import RemoteBASE\nfrom dvc.scheme import Schemes\n\nlogger = logging.getLogger(__name__)\n\n\nclass RemoteHTTP(RemoteBASE):\n    scheme = Schemes.HTTP\n    SESSION_RETRIES = 5\n    SESSION_BACKOFF_FACTOR = 0.1\n    REQUEST_TIMEOUT = 10\n    CHUNK_SIZE = 2 ** 16\n    PARAM_CHECKSUM = \"etag\"\n\n    def __init__(self, repo, config):\n        super().__init__(repo, config)\n\n        url = config.get(Config.SECTION_REMOTE_URL)\n        self.path_info = self.path_cls(url) if url else None\n\n        if not self.no_traverse:\n            raise ConfigError(\n                \"HTTP doesn't support traversing the remote to list existing \"\n                \"files. Use: `dvc remote modify <name> no_traverse true`\"\n            )\n\n    def _download(self, from_info, to_file, name=None, no_progress_bar=False):\n        response = self._request(\"GET\", from_info.url, stream=True)\n        if response.status_code != 200:\n            raise HTTPError(response.status_code, response.reason)\n        with Tqdm(\n            total=None if no_progress_bar else self._content_length(response),\n            leave=False,\n            bytes=True,\n            desc=from_info.url if name is None else name,\n            disable=no_progress_bar,\n        ) as pbar:\n            with open(to_file, \"wb\") as fd:\n                for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):\n                    fd.write(chunk)\n                    pbar.update(len(chunk))\n\n    def exists(self, path_info):\n        return bool(self._request(\"HEAD\", path_info.url))\n\n    def _content_length(self, response):\n        res = response.headers.get(\"Content-Length\")\n        return int(res) if res else None\n\n    def get_file_checksum(self, path_info):\n        url = path_info.url\n        headers = self._request(\"HEAD\", url).headers\n        etag = headers.get(\"ETag\") or headers.get(\"Content-MD5\")\n\n        if not etag:\n            raise DvcException(\n                \"could not find an ETag or \"\n                \"Content-MD5 header for '{url}'\".format(url=url)\n            )\n\n        if etag.startswith(\"W/\"):\n            raise DvcException(\n                \"Weak ETags are not supported.\"\n                \" (Etag: '{etag}', URL: '{url}')\".format(etag=etag, url=url)\n            )\n\n        return etag\n\n    @wrap_prop(threading.Lock())\n    @cached_property\n    def _session(self):\n        import requests\n        from requests.adapters import HTTPAdapter\n        from urllib3.util.retry import Retry\n\n        session = requests.Session()\n\n        retries = Retry(\n            total=self.SESSION_RETRIES,\n            backoff_factor=self.SESSION_BACKOFF_FACTOR,\n        )\n\n        session.mount(\"http://\", HTTPAdapter(max_retries=retries))\n        session.mount(\"https://\", HTTPAdapter(max_retries=retries))\n\n        return session\n\n    def _request(self, method, url, **kwargs):\n        import requests\n\n        kwargs.setdefault(\"allow_redirects\", True)\n        kwargs.setdefault(\"timeout\", self.REQUEST_TIMEOUT)\n\n        try:\n            return self._session.request(method, url, **kwargs)\n        except requests.exceptions.RequestException:\n            raise DvcException(\"could not perform a {} request\".format(method))\n\n    def gc(self):\n        raise NotImplementedError\n", "path": "dvc/remote/http.py"}], "after_files": [{"content": "import logging\nimport threading\n\nfrom funcy import cached_property, wrap_prop\n\nfrom dvc.config import Config\nfrom dvc.config import ConfigError\nfrom dvc.exceptions import DvcException, HTTPError\nfrom dvc.progress import Tqdm\nfrom dvc.remote.base import RemoteBASE\nfrom dvc.scheme import Schemes\n\nlogger = logging.getLogger(__name__)\n\n\nclass RemoteHTTP(RemoteBASE):\n    scheme = Schemes.HTTP\n    SESSION_RETRIES = 5\n    SESSION_BACKOFF_FACTOR = 0.1\n    REQUEST_TIMEOUT = 10\n    CHUNK_SIZE = 2 ** 16\n    PARAM_CHECKSUM = \"etag\"\n\n    def __init__(self, repo, config):\n        super().__init__(repo, config)\n\n        url = config.get(Config.SECTION_REMOTE_URL)\n        self.path_info = self.path_cls(url) if url else None\n\n        if not self.no_traverse:\n            raise ConfigError(\n                \"HTTP doesn't support traversing the remote to list existing \"\n                \"files. Use: `dvc remote modify <name> no_traverse true`\"\n            )\n\n    def _download(self, from_info, to_file, name=None, no_progress_bar=False):\n        response = self._request(\"GET\", from_info.url, stream=True)\n        if response.status_code != 200:\n            raise HTTPError(response.status_code, response.reason)\n        with Tqdm(\n            total=None if no_progress_bar else self._content_length(response),\n            leave=False,\n            bytes=True,\n            desc=from_info.url if name is None else name,\n            disable=no_progress_bar,\n        ) as pbar:\n            with open(to_file, \"wb\") as fd:\n                for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):\n                    fd.write(chunk)\n                    pbar.update(len(chunk))\n\n    def exists(self, path_info):\n        return bool(self._request(\"HEAD\", path_info.url))\n\n    def _content_length(self, response):\n        res = response.headers.get(\"Content-Length\")\n        return int(res) if res else None\n\n    def get_file_checksum(self, path_info):\n        url = path_info.url\n        headers = self._request(\"HEAD\", url).headers\n        etag = headers.get(\"ETag\") or headers.get(\"Content-MD5\")\n\n        if not etag:\n            raise DvcException(\n                \"could not find an ETag or \"\n                \"Content-MD5 header for '{url}'\".format(url=url)\n            )\n\n        if etag.startswith(\"W/\"):\n            raise DvcException(\n                \"Weak ETags are not supported.\"\n                \" (Etag: '{etag}', URL: '{url}')\".format(etag=etag, url=url)\n            )\n\n        return etag\n\n    @wrap_prop(threading.Lock())\n    @cached_property\n    def _session(self):\n        import requests\n        from requests.adapters import HTTPAdapter\n        from urllib3.util.retry import Retry\n\n        session = requests.Session()\n\n        retries = Retry(\n            total=self.SESSION_RETRIES,\n            backoff_factor=self.SESSION_BACKOFF_FACTOR,\n        )\n\n        session.mount(\"http://\", HTTPAdapter(max_retries=retries))\n        session.mount(\"https://\", HTTPAdapter(max_retries=retries))\n\n        return session\n\n    def _request(self, method, url, **kwargs):\n        import requests\n\n        kwargs.setdefault(\"allow_redirects\", True)\n        kwargs.setdefault(\"timeout\", self.REQUEST_TIMEOUT)\n\n        try:\n            res = self._session.request(method, url, **kwargs)\n\n            redirect_no_location = (\n                kwargs[\"allow_redirects\"]\n                and res.status_code in (301, 302)\n                and \"location\" not in res.headers\n            )\n\n            if redirect_no_location:\n                # AWS s3 doesn't like to add a location header to its redirects\n                # from https://s3.amazonaws.com/<bucket name>/* type URLs.\n                # This should be treated as an error\n                raise requests.exceptions.RequestException\n\n            return res\n\n        except requests.exceptions.RequestException:\n            raise DvcException(\"could not perform a {} request\".format(method))\n\n    def gc(self):\n        raise NotImplementedError\n", "path": "dvc/remote/http.py"}]}