We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
FileNotFoundError when using an s3 bucket as the model_dir with HuggingFace model server
/kind bug

First of all, I'd like to say thank you for the work on KServe! It's been delightful so far playing around with KServe. But we found a small bug while testing out the HuggingFace model server (which we're aware is a very new addition as well).

**What steps did you take and what happened:**
1. Created an InferenceService using the HuggingFace model server (yaml pasted below)
2. Specified an s3 bucket as the `model_dir` (I suspect this might happen for anything that's not a local dir)
3. Observed that the model is succesfully downloaded to a tmp directory and loaded, but then encountered the `FileNotFoundError` right after

Logs:
```
% k logs huggingface-predictor-00003-deployment-8659bb8b9-m945b
Defaulted container "kserve-container" out of: kserve-container, queue-proxy
INFO:root:Copying contents of s3://kserve-test-models/classifier to local
INFO:root:Downloaded object classifier/config.json to /tmp/tmpckx_trr1/config.json
...
INFO:root:Successfully copied s3://kserve-test-models/classifier to /tmp/tmpckx_trr1
INFO:kserve:successfully loaded tokenizer for task: 4
INFO:kserve:successfully loaded huggingface model from path /tmp/tmpckx_trr1
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/huggingfaceserver/huggingfaceserver/__main__.py", line 69, in <module>
    kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(
  File "/huggingfaceserver/huggingfaceserver/huggingface_model_repository.py", line 24, in __init__
    self.load_models()
  File "/kserve/kserve/model_repository.py", line 37, in load_models
    for name in os.listdir(self.models_dir):
FileNotFoundError: [Errno 2] No such file or directory: 's3://kserve-test-models/spam-classifier'
```


**What did you expect to happen:**

I expected that this would work, as the model was successfully downloaded and loaded. But I did find a tmp workaround below and I think I know where the issue is!

**What's the InferenceService yaml:**
```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface
spec:
  predictor:
    serviceAccountName: huggingface-sa
    containers:
    - args:
      - --model_name=spam-classifier
      # - --model_id=xyz (see workaround below)
      - --model_dir=s3://kserve-test-models/classifier
      - --tensor_input_names=input_ids
      image: kserve/huggingfaceserver:latest
      name: kserve-container
```

**Anything else you would like to add:**

A temporary workaround I found is to supply the `model_id` argument. It can have any value, as the `model_dir` will override it anyway during loading:

https://github.com/kserve/kserve/blob/5172dc80b0a23d2263757de70a02eeef08b18811/python/huggingfaceserver/huggingfaceserver/model.py#L91-L94

<details>
  <summary>I have verified that this workaround works (expand to see logs).</summary>

```
% k logs huggingface-predictor-00004-deployment-946b4d6c8-pk5nj -f
Defaulted container "kserve-container" out of: kserve-container, queue-proxy
INFO:root:Copying contents of s3://kserve-test-models/classifier to local
INFO:root:Downloaded object classifier/config.json to /tmp/tmppwjsica7/config.json
...
INFO:kserve:successfully loaded tokenizer for task: 4
INFO:kserve:successfully loaded huggingface model from path /tmp/tmppwjsica7
INFO:kserve:Registering model: classifier
INFO:kserve:Setting max asyncio worker threads as 5
INFO:kserve:Starting uvicorn with 1 workers
2024-02-09 18:57:33.228 uvicorn.error INFO:     Started server process [1]
2024-02-09 18:57:33.229 uvicorn.error INFO:     Waiting for application startup.
2024-02-09 18:57:33.234 1 kserve INFO [start():62] Starting gRPC server on [::]:8081
2024-02-09 18:57:33.234 uvicorn.error INFO:     Application startup complete.
2024-02-09 18:57:33.235 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
```
</details>

I think the issue is here:
https://github.com/kserve/kserve/blob/5172dc80b0a23d2263757de70a02eeef08b18811/python/huggingfaceserver/huggingfaceserver/__main__.py#L63-L72

1. `model.load()` will succeed, so we jump to line 68
2. It checks for `args.model_id`, which is empty, so we go inside the if block
3. It will try to instantiate `HuggingfaceModelRepository` with `model_dir`, which is pointing to an s3 bucket and not a local directory, thus causing the `FileNotFoundError`
4. This is how I came up with the workaround of passing `model_id`, so that the else block is executed instead (because the model did load succesfully, so doing `kserve.ModelServer().start([model] if model.ready else [])` won't be a problem)

**Environment:**

- Cloud Environment: aws
- Kubernetes version: (use `kubectl version`): v1.27.9-eks-5e0fdde
- OS (e.g. from `/etc/os-release`): Ubuntu 22.04.3 LTS


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `python/huggingfaceserver/huggingfaceserver/__main__.py`
Content:
```
1 # Copyright 2024 The KServe Authors.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #    http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 
15 import argparse
16 import logging
17 
18 from kserve.model import PredictorConfig
19 from . import HuggingfaceModel, HuggingfaceModelRepository
20 import kserve
21 from kserve.errors import ModelMissingError
22 
23 
24 def list_of_strings(arg):
25     return arg.split(',')
26 
27 
28 parser = argparse.ArgumentParser(parents=[kserve.model_server.parser])
29 
30 parser.add_argument('--model_dir', required=False, default=None,
31                     help='A URI pointer to the model binary')
32 parser.add_argument('--model_id', required=False,
33                     help='Huggingface model id')
34 parser.add_argument('--tensor_parallel_degree', type=int, default=-1,
35                     help='tensor parallel degree')
36 parser.add_argument('--max_length', type=int, default=None,
37                     help='max sequence length for the tokenizer')
38 parser.add_argument('--do_lower_case', type=bool, default=True,
39                     help='do lower case for the tokenizer')
40 parser.add_argument('--add_special_tokens', type=bool, default=True,
41                     help='the sequences will be encoded with the special tokens relative to their model')
42 parser.add_argument('--tensor_input_names', type=list_of_strings, default=None,
43                     help='the tensor input names passed to the model')
44 parser.add_argument('--task', required=False, help="The ML task name")
45 
46 try:
47     from vllm.engine.arg_utils import AsyncEngineArgs
48 
49     parser = AsyncEngineArgs.add_cli_args(parser)
50     _vllm = True
51 except ImportError:
52     _vllm = False
53 args, _ = parser.parse_known_args()
54 
55 if __name__ == "__main__":
56     engine_args = AsyncEngineArgs.from_cli_args(args) if _vllm else None
57     predictor_config = PredictorConfig(args.predictor_host, args.predictor_protocol,
58                                        args.predictor_use_ssl,
59                                        args.predictor_request_timeout_seconds)
60     model = HuggingfaceModel(args.model_name,
61                              predictor_config=predictor_config,
62                              kwargs=vars(args), engine_args=engine_args)
63     try:
64         model.load()
65     except ModelMissingError:
66         logging.error(f"fail to locate model file for model {args.model_name} under dir {args.model_dir},"
67                       f"trying loading from model repository.")
68     if not args.model_id:
69         kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(
70             [model] if model.ready else [])
71     else:
72         kserve.ModelServer().start([model] if model.ready else [])
73
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/python/huggingfaceserver/huggingfaceserver/__main__.py b/python/huggingfaceserver/huggingfaceserver/__main__.py
--- a/python/huggingfaceserver/huggingfaceserver/__main__.py
+++ b/python/huggingfaceserver/huggingfaceserver/__main__.py
@@ -62,11 +62,9 @@
                              kwargs=vars(args), engine_args=engine_args)
     try:
         model.load()
+        kserve.ModelServer().start([model] if model.ready else [])
     except ModelMissingError:
         logging.error(f"fail to locate model file for model {args.model_name} under dir {args.model_dir},"
                       f"trying loading from model repository.")
-    if not args.model_id:
         kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(
             [model] if model.ready else [])
-    else:
-        kserve.ModelServer().start([model] if model.ready else [])

{"golden_diff": "diff --git a/python/huggingfaceserver/huggingfaceserver/__main__.py b/python/huggingfaceserver/huggingfaceserver/__main__.py\n--- a/python/huggingfaceserver/huggingfaceserver/__main__.py\n+++ b/python/huggingfaceserver/huggingfaceserver/__main__.py\n@@ -62,11 +62,9 @@\n                              kwargs=vars(args), engine_args=engine_args)\n     try:\n         model.load()\n+        kserve.ModelServer().start([model] if model.ready else [])\n     except ModelMissingError:\n         logging.error(f\"fail to locate model file for model {args.model_name} under dir {args.model_dir},\"\n                       f\"trying loading from model repository.\")\n-    if not args.model_id:\n         kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(\n             [model] if model.ready else [])\n-    else:\n-        kserve.ModelServer().start([model] if model.ready else [])\n", "issue": "FileNotFoundError when using an s3 bucket as the model_dir with HuggingFace model server\n/kind bug\r\n\r\nFirst of all, I'd like to say thank you for the work on KServe! It's been delightful so far playing around with KServe. But we found a small bug while testing out the HuggingFace model server (which we're aware is a very new addition as well).\r\n\r\n**What steps did you take and what happened:**\r\n1. Created an InferenceService using the HuggingFace model server (yaml pasted below)\r\n2. Specified an s3 bucket as the `model_dir` (I suspect this might happen for anything that's not a local dir)\r\n3. Observed that the model is succesfully downloaded to a tmp directory and loaded, but then encountered the `FileNotFoundError` right after\r\n\r\nLogs:\r\n```\r\n% k logs huggingface-predictor-00003-deployment-8659bb8b9-m945b\r\nDefaulted container \"kserve-container\" out of: kserve-container, queue-proxy\r\nINFO:root:Copying contents of s3://kserve-test-models/classifier to local\r\nINFO:root:Downloaded object classifier/config.json to /tmp/tmpckx_trr1/config.json\r\n...\r\nINFO:root:Successfully copied s3://kserve-test-models/classifier to /tmp/tmpckx_trr1\r\nINFO:kserve:successfully loaded tokenizer for task: 4\r\nINFO:kserve:successfully loaded huggingface model from path /tmp/tmpckx_trr1\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\r\n    return _run_code(code, main_globals, None,\r\n  File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/huggingfaceserver/huggingfaceserver/__main__.py\", line 69, in <module>\r\n    kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(\r\n  File \"/huggingfaceserver/huggingfaceserver/huggingface_model_repository.py\", line 24, in __init__\r\n    self.load_models()\r\n  File \"/kserve/kserve/model_repository.py\", line 37, in load_models\r\n    for name in os.listdir(self.models_dir):\r\nFileNotFoundError: [Errno 2] No such file or directory: 's3://kserve-test-models/spam-classifier'\r\n```\r\n\r\n\r\n**What did you expect to happen:**\r\n\r\nI expected that this would work, as the model was successfully downloaded and loaded. But I did find a tmp workaround below and I think I know where the issue is!\r\n\r\n**What's the InferenceService yaml:**\r\n```yaml\r\napiVersion: serving.kserve.io/v1beta1\r\nkind: InferenceService\r\nmetadata:\r\n  name: huggingface\r\nspec:\r\n  predictor:\r\n    serviceAccountName: huggingface-sa\r\n    containers:\r\n    - args:\r\n      - --model_name=spam-classifier\r\n      # - --model_id=xyz (see workaround below)\r\n      - --model_dir=s3://kserve-test-models/classifier\r\n      - --tensor_input_names=input_ids\r\n      image: kserve/huggingfaceserver:latest\r\n      name: kserve-container\r\n```\r\n\r\n**Anything else you would like to add:**\r\n\r\nA temporary workaround I found is to supply the `model_id` argument. It can have any value, as the `model_dir` will override it anyway during loading:\r\n\r\nhttps://github.com/kserve/kserve/blob/5172dc80b0a23d2263757de70a02eeef08b18811/python/huggingfaceserver/huggingfaceserver/model.py#L91-L94\r\n\r\n<details>\r\n  <summary>I have verified that this workaround works (expand to see logs).</summary>\r\n\r\n```\r\n% k logs huggingface-predictor-00004-deployment-946b4d6c8-pk5nj -f\r\nDefaulted container \"kserve-container\" out of: kserve-container, queue-proxy\r\nINFO:root:Copying contents of s3://kserve-test-models/classifier to local\r\nINFO:root:Downloaded object classifier/config.json to /tmp/tmppwjsica7/config.json\r\n...\r\nINFO:kserve:successfully loaded tokenizer for task: 4\r\nINFO:kserve:successfully loaded huggingface model from path /tmp/tmppwjsica7\r\nINFO:kserve:Registering model: classifier\r\nINFO:kserve:Setting max asyncio worker threads as 5\r\nINFO:kserve:Starting uvicorn with 1 workers\r\n2024-02-09 18:57:33.228 uvicorn.error INFO:     Started server process [1]\r\n2024-02-09 18:57:33.229 uvicorn.error INFO:     Waiting for application startup.\r\n2024-02-09 18:57:33.234 1 kserve INFO [start():62] Starting gRPC server on [::]:8081\r\n2024-02-09 18:57:33.234 uvicorn.error INFO:     Application startup complete.\r\n2024-02-09 18:57:33.235 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)\r\n```\r\n</details>\r\n\r\nI think the issue is here:\r\nhttps://github.com/kserve/kserve/blob/5172dc80b0a23d2263757de70a02eeef08b18811/python/huggingfaceserver/huggingfaceserver/__main__.py#L63-L72\r\n\r\n1. `model.load()` will succeed, so we jump to line 68\r\n2. It checks for `args.model_id`, which is empty, so we go inside the if block\r\n3. It will try to instantiate `HuggingfaceModelRepository` with `model_dir`, which is pointing to an s3 bucket and not a local directory, thus causing the `FileNotFoundError`\r\n4. This is how I came up with the workaround of passing `model_id`, so that the else block is executed instead (because the model did load succesfully, so doing `kserve.ModelServer().start([model] if model.ready else [])` won't be a problem)\r\n\r\n**Environment:**\r\n\r\n- Cloud Environment: aws\r\n- Kubernetes version: (use `kubectl version`): v1.27.9-eks-5e0fdde\r\n- OS (e.g. from `/etc/os-release`): Ubuntu 22.04.3 LTS\r\n\n", "before_files": [{"content": "# Copyright 2024 The KServe Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#    http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport logging\n\nfrom kserve.model import PredictorConfig\nfrom . import HuggingfaceModel, HuggingfaceModelRepository\nimport kserve\nfrom kserve.errors import ModelMissingError\n\n\ndef list_of_strings(arg):\n    return arg.split(',')\n\n\nparser = argparse.ArgumentParser(parents=[kserve.model_server.parser])\n\nparser.add_argument('--model_dir', required=False, default=None,\n                    help='A URI pointer to the model binary')\nparser.add_argument('--model_id', required=False,\n                    help='Huggingface model id')\nparser.add_argument('--tensor_parallel_degree', type=int, default=-1,\n                    help='tensor parallel degree')\nparser.add_argument('--max_length', type=int, default=None,\n                    help='max sequence length for the tokenizer')\nparser.add_argument('--do_lower_case', type=bool, default=True,\n                    help='do lower case for the tokenizer')\nparser.add_argument('--add_special_tokens', type=bool, default=True,\n                    help='the sequences will be encoded with the special tokens relative to their model')\nparser.add_argument('--tensor_input_names', type=list_of_strings, default=None,\n                    help='the tensor input names passed to the model')\nparser.add_argument('--task', required=False, help=\"The ML task name\")\n\ntry:\n    from vllm.engine.arg_utils import AsyncEngineArgs\n\n    parser = AsyncEngineArgs.add_cli_args(parser)\n    _vllm = True\nexcept ImportError:\n    _vllm = False\nargs, _ = parser.parse_known_args()\n\nif __name__ == \"__main__\":\n    engine_args = AsyncEngineArgs.from_cli_args(args) if _vllm else None\n    predictor_config = PredictorConfig(args.predictor_host, args.predictor_protocol,\n                                       args.predictor_use_ssl,\n                                       args.predictor_request_timeout_seconds)\n    model = HuggingfaceModel(args.model_name,\n                             predictor_config=predictor_config,\n                             kwargs=vars(args), engine_args=engine_args)\n    try:\n        model.load()\n    except ModelMissingError:\n        logging.error(f\"fail to locate model file for model {args.model_name} under dir {args.model_dir},\"\n                      f\"trying loading from model repository.\")\n    if not args.model_id:\n        kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(\n            [model] if model.ready else [])\n    else:\n        kserve.ModelServer().start([model] if model.ready else [])\n", "path": "python/huggingfaceserver/huggingfaceserver/__main__.py"}], "after_files": [{"content": "# Copyright 2024 The KServe Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#    http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport logging\n\nfrom kserve.model import PredictorConfig\nfrom . import HuggingfaceModel, HuggingfaceModelRepository\nimport kserve\nfrom kserve.errors import ModelMissingError\n\n\ndef list_of_strings(arg):\n    return arg.split(',')\n\n\nparser = argparse.ArgumentParser(parents=[kserve.model_server.parser])\n\nparser.add_argument('--model_dir', required=False, default=None,\n                    help='A URI pointer to the model binary')\nparser.add_argument('--model_id', required=False,\n                    help='Huggingface model id')\nparser.add_argument('--tensor_parallel_degree', type=int, default=-1,\n                    help='tensor parallel degree')\nparser.add_argument('--max_length', type=int, default=None,\n                    help='max sequence length for the tokenizer')\nparser.add_argument('--do_lower_case', type=bool, default=True,\n                    help='do lower case for the tokenizer')\nparser.add_argument('--add_special_tokens', type=bool, default=True,\n                    help='the sequences will be encoded with the special tokens relative to their model')\nparser.add_argument('--tensor_input_names', type=list_of_strings, default=None,\n                    help='the tensor input names passed to the model')\nparser.add_argument('--task', required=False, help=\"The ML task name\")\n\ntry:\n    from vllm.engine.arg_utils import AsyncEngineArgs\n\n    parser = AsyncEngineArgs.add_cli_args(parser)\n    _vllm = True\nexcept ImportError:\n    _vllm = False\nargs, _ = parser.parse_known_args()\n\nif __name__ == \"__main__\":\n    engine_args = AsyncEngineArgs.from_cli_args(args) if _vllm else None\n    predictor_config = PredictorConfig(args.predictor_host, args.predictor_protocol,\n                                       args.predictor_use_ssl,\n                                       args.predictor_request_timeout_seconds)\n    model = HuggingfaceModel(args.model_name,\n                             predictor_config=predictor_config,\n                             kwargs=vars(args), engine_args=engine_args)\n    try:\n        model.load()\n        kserve.ModelServer().start([model] if model.ready else [])\n    except ModelMissingError:\n        logging.error(f\"fail to locate model file for model {args.model_name} under dir {args.model_dir},\"\n                      f\"trying loading from model repository.\")\n        kserve.ModelServer(registered_models=HuggingfaceModelRepository(args.model_dir)).start(\n            [model] if model.ready else [])\n", "path": "python/huggingfaceserver/huggingfaceserver/__main__.py"}]}