We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Bug in LLaMA fast tokenizer
In my environment, using the LLaMA fast tokenizer raises an error about protobuf:
```
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/convert_slow_tokenizer.py", line 445, in __init__
    from .utils import sentencepiece_model_pb2 as model_pb2
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/utils/sentencepiece_model_pb2.py", line 91, in <module>
    _descriptor.EnumValueDescriptor(
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
```

While downgrading the protobuf version removed the error, it slowed down the initialization time by ~8x.
* Initialization with fast tokenizer & protobuf==3.20.3 
```
real    4m18.476s
user    3m52.706s
sys     0m27.644s
```
* Initialization with slow tokenizer
```
real    0m27.620s
user    0m8.011s
sys     0m19.237s
```


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `cacheflow/sampling_params.py`
Content:
```
1 """Sampling parameters for text generation."""
2 from typing import List, Optional, Union
3 
4 
5 class SamplingParams:
6     """Sampling parameters for text generation.
7 
8     Overall, we follow the sampling parameters from the OpenAI text completion
9     API (https://platform.openai.com/docs/api-reference/completions/create).
10     In addition, we support beam search, which is not supported by OpenAI.
11 
12     Args:
13         n: Number of output sequences to return for the given prompt.
14         best_of: Number of output sequences that are generated from the prompt.
15             From these `best_of` sequences, the top `n` sequences are returned.
16             `best_of` must be greater than or equal to `n`. This is treated as
17             the beam width when `use_beam_search` is True. By default, `best_of`
18             is set to `n`.
19         presence_penalty: Float that penalizes new tokens based on whether they
20             appear in the generated text so far. Values > 0 encourage the model
21             to use new tokens, while values < 0 encourage the model to repeat
22             tokens.
23         frequency_penalty: Float that penalizes new tokens based on their
24             frequency in the generated text so far. Values > 0 encourage the
25             model to use new tokens, while values < 0 encourage the model to
26             repeat tokens.
27         temperature: Float that controls the randomness of the sampling. Lower
28             values make the model more deterministic, while higher values make
29             the model more random. Zero means greedy sampling.
30         top_p: Float that controls the cumulative probability of the top tokens
31             to consider. Must be in (0, 1]. Set to 1 to consider all tokens.
32         top_k: Integer that controls the number of top tokens to consider. Set
33             to -1 to consider all tokens.
34         use_beam_search: Whether to use beam search instead of sampling.
35         stop: List of strings that stop the generation when they are generated.
36             The returned output will not contain the stop strings.
37         ignore_eos: Whether to ignore the EOS token and continue generating
38             tokens after the EOS token is generated.
39         max_tokens: Maximum number of tokens to generate per output sequence.
40         logprobs: Number of log probabilities to return per output token.
41     """
42 
43     def __init__(
44         self,
45         n: int = 1,
46         best_of: Optional[int] = None,
47         presence_penalty: float = 0.0,
48         frequency_penalty: float = 0.0,
49         temperature: float = 1.0,
50         top_p: float = 1.0,
51         top_k: int = -1,
52         use_beam_search: bool = False,
53         stop: Union[str, List[str]] = [],
54         ignore_eos: bool = False,
55         max_tokens: int = 16,
56         logprobs: Optional[int] = None,
57     ) -> None:
58         self.n = n
59         self.best_of = best_of if best_of is not None else n
60         self.presence_penalty = presence_penalty
61         self.frequency_penalty = frequency_penalty
62         self.temperature = temperature
63         self.top_p = top_p
64         self.top_k = top_k
65         self.use_beam_search = use_beam_search
66         self.stop = [stop] if isinstance(stop, str) else list(stop)
67         self.ignore_eos = ignore_eos
68         self.max_tokens = max_tokens
69         self.logprobs = logprobs
70 
71         self._verify_args()
72         if self.use_beam_search:
73             self._verity_beam_search()
74         elif self.temperature == 0.0:
75             # Zero temperature means greedy sampling.
76             self._verify_greedy_sampling()
77 
78     def _verify_args(self) -> None:
79         if self.n < 1:
80             raise ValueError(f"n must be at least 1, got {self.n}.")
81         if self.best_of < self.n:
82             raise ValueError(f"best_of must be greater than or equal to n, "
83                              f"got n={self.n} and best_of={self.best_of}.")
84         if not -2.0 <= self.presence_penalty <= 2.0:
85             raise ValueError("presence_penalty must be in [-2, 2], got "
86                              f"{self.presence_penalty}.")
87         if not -2.0 <= self.frequency_penalty <= 2.0:
88             raise ValueError("frequency_penalty must be in [-2, 2], got "
89                              f"{self.frequency_penalty}.")
90         if self.temperature < 0.0:
91             raise ValueError(
92                 f"temperature must be non-negative, got {self.temperature}.")
93         if not 0.0 < self.top_p <= 1.0:
94             raise ValueError(f"top_p must be in (0, 1], got {self.top_p}.")
95         if self.top_k < -1 or self.top_k == 0:
96             raise ValueError(f"top_k must be -1 (disable), or at least 1, "
97                              f"got {self.top_k}.")
98         if self.max_tokens < 1:
99             raise ValueError(
100                 f"max_tokens must be at least 1, got {self.max_tokens}.")
101         if self.logprobs is not None and self.logprobs < 0:
102             raise ValueError(
103                 f"logprobs must be non-negative, got {self.logprobs}.")
104 
105     def _verity_beam_search(self) -> None:
106         if self.best_of == 1:
107             raise ValueError("best_of must be greater than 1 when using beam "
108                              f"search. Got {self.best_of}.")
109         if self.temperature > 0.0:
110             raise ValueError("temperature must be 0 when using beam search.")
111         if self.top_p < 1.0:
112             raise ValueError("top_p must be 1 when using beam search.")
113         if self.top_k != -1:
114             raise ValueError("top_k must be -1 when using beam search.")
115 
116     def _verify_greedy_sampling(self) -> None:
117         if self.best_of > 1:
118             raise ValueError("best_of must be 1 when using greedy sampling."
119                              f"Got {self.best_of}.")
120         if self.top_p < 1.0:
121             raise ValueError("top_p must be 1 when using greedy sampling.")
122         if self.top_k != -1:
123             raise ValueError("top_k must be -1 when using greedy sampling.")
124 
125     def __repr__(self) -> str:
126         return (f"SamplingParams(n={self.n}, "
127                 f"best_of={self.best_of}, "
128                 f"presence_penalty={self.presence_penalty}, "
129                 f"frequency_penalty={self.frequency_penalty}, "
130                 f"temperature={self.temperature}, "
131                 f"top_p={self.top_p}, "
132                 f"top_k={self.top_k},"
133                 f"use_beam_search={self.use_beam_search}, "
134                 f"stop={self.stop}, "
135                 f"ignore_eos={self.ignore_eos}, "
136                 f"max_tokens={self.max_tokens}, "
137                 f"logprobs={self.logprobs})")
138
```

Path: `cacheflow/server/tokenizer_utils.py`
Content:
```
1 from typing import List, Tuple, Union
2 
3 from transformers import (AutoConfig, AutoTokenizer, PreTrainedTokenizer,
4                           PreTrainedTokenizerFast)
5 
6 from cacheflow.logger import init_logger
7 
8 logger = init_logger(__name__)
9 
10 _MODEL_TYPES_WITH_SLOW_TOKENIZER = [
11     # LLaMA fast tokenizer has a bug related to protobuf.
12     # See https://github.com/WoosukKwon/cacheflow/issues/80#issue-1698550554
13     "llama",
14 ]
15 
16 
17 def get_tokenizer(
18     model_name: str,
19     *args,
20     **kwargs,
21 ) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:
22     config = AutoConfig.from_pretrained(model_name)
23     if config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:
24         if getattr(kwargs, "use_fast", False) == True:
25             raise ValueError(
26                 f"Cannot use the fast tokenizer for {config.model_type} due to "
27                 "bugs in the fast tokenizer.")
28         logger.info(
29             f"Using the slow tokenizer for {config.model_type} due to bugs in "
30             "the fast tokenizer. This could potentially lead to performance "
31             "degradation.")
32         kwargs["use_fast"] = False
33     return AutoTokenizer.from_pretrained(model_name, *args, **kwargs)
34 
35 
36 def detokenize_incrementally(
37     tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast],
38     prev_output_tokens: List[str],
39     new_token_id: int,
40     skip_special_tokens: bool,
41 ) -> Tuple[str, str]:
42     """Detokenizes the new token in conjuction with the previous output tokens.
43 
44     NOTE: This function does not update prev_output_tokens.
45 
46     Returns:
47         new_token: The new token as a string.
48         output_text: The new output text as a string.
49     """
50     new_token = tokenizer.convert_ids_to_tokens(
51         new_token_id, skip_special_tokens=skip_special_tokens)
52     output_tokens = prev_output_tokens + [new_token]
53 
54     # Convert the tokens to a string.
55     # Optimization: If the tokenizer does not have `added_tokens_encoder`,
56     # then we can directly use `convert_tokens_to_string`.
57     if not getattr(tokenizer, "added_tokens_encoder", {}):
58         output_text = tokenizer.convert_tokens_to_string(output_tokens)
59         return new_token, output_text
60 
61     # Adapted from https://github.com/huggingface/transformers/blob/v4.28.0/src/transformers/tokenization_utils.py#L921
62     # NOTE(woosuk): The following code is slow because it runs a for loop over
63     # the output_tokens. In Python, running a for loop over a list can be slow
64     # even when the loop body is very simple.
65     sub_texts = []
66     current_sub_text = []
67     for token in output_tokens:
68         if skip_special_tokens and token in tokenizer.all_special_ids:
69             continue
70         if token in tokenizer.added_tokens_encoder:
71             if current_sub_text:
72                 sub_text = tokenizer.convert_tokens_to_string(current_sub_text)
73                 sub_texts.append(sub_text)
74                 current_sub_text = []
75             sub_texts.append(token)
76         else:
77             current_sub_text.append(token)
78     if current_sub_text:
79         sub_text = tokenizer.convert_tokens_to_string(current_sub_text)
80         sub_texts.append(sub_text)
81     output_text = " ".join(sub_texts)
82     return new_token, output_text
83
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/cacheflow/sampling_params.py b/cacheflow/sampling_params.py
--- a/cacheflow/sampling_params.py
+++ b/cacheflow/sampling_params.py
@@ -129,7 +129,7 @@
                 f"frequency_penalty={self.frequency_penalty}, "
                 f"temperature={self.temperature}, "
                 f"top_p={self.top_p}, "
-                f"top_k={self.top_k},"
+                f"top_k={self.top_k}, "
                 f"use_beam_search={self.use_beam_search}, "
                 f"stop={self.stop}, "
                 f"ignore_eos={self.ignore_eos}, "
diff --git a/cacheflow/server/tokenizer_utils.py b/cacheflow/server/tokenizer_utils.py
--- a/cacheflow/server/tokenizer_utils.py
+++ b/cacheflow/server/tokenizer_utils.py
@@ -7,11 +7,7 @@
 
 logger = init_logger(__name__)
 
-_MODEL_TYPES_WITH_SLOW_TOKENIZER = [
-    # LLaMA fast tokenizer has a bug related to protobuf.
-    # See https://github.com/WoosukKwon/cacheflow/issues/80#issue-1698550554
-    "llama",
-]
+_MODEL_TYPES_WITH_SLOW_TOKENIZER = []
 
 
 def get_tokenizer(
@@ -20,7 +16,15 @@
     **kwargs,
 ) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:
     config = AutoConfig.from_pretrained(model_name)
-    if config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:
+    if config.model_type == "llama" and getattr(kwargs, "use_fast", True):
+        # LLaMA fast tokenizer causes protobuf errors in some environments.
+        # However, we found that the below LLaMA fast tokenizer works well in
+        # most environments.
+        model_name = "hf-internal-testing/llama-tokenizer"
+        logger.info(
+            f"Using the LLaMA fast tokenizer in '{model_name}' to avoid "
+            "potential protobuf errors.")
+    elif config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:
         if getattr(kwargs, "use_fast", False) == True:
             raise ValueError(
                 f"Cannot use the fast tokenizer for {config.model_type} due to "

{"golden_diff": "diff --git a/cacheflow/sampling_params.py b/cacheflow/sampling_params.py\n--- a/cacheflow/sampling_params.py\n+++ b/cacheflow/sampling_params.py\n@@ -129,7 +129,7 @@\n                 f\"frequency_penalty={self.frequency_penalty}, \"\n                 f\"temperature={self.temperature}, \"\n                 f\"top_p={self.top_p}, \"\n-                f\"top_k={self.top_k},\"\n+                f\"top_k={self.top_k}, \"\n                 f\"use_beam_search={self.use_beam_search}, \"\n                 f\"stop={self.stop}, \"\n                 f\"ignore_eos={self.ignore_eos}, \"\ndiff --git a/cacheflow/server/tokenizer_utils.py b/cacheflow/server/tokenizer_utils.py\n--- a/cacheflow/server/tokenizer_utils.py\n+++ b/cacheflow/server/tokenizer_utils.py\n@@ -7,11 +7,7 @@\n \n logger = init_logger(__name__)\n \n-_MODEL_TYPES_WITH_SLOW_TOKENIZER = [\n-    # LLaMA fast tokenizer has a bug related to protobuf.\n-    # See https://github.com/WoosukKwon/cacheflow/issues/80#issue-1698550554\n-    \"llama\",\n-]\n+_MODEL_TYPES_WITH_SLOW_TOKENIZER = []\n \n \n def get_tokenizer(\n@@ -20,7 +16,15 @@\n     **kwargs,\n ) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:\n     config = AutoConfig.from_pretrained(model_name)\n-    if config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:\n+    if config.model_type == \"llama\" and getattr(kwargs, \"use_fast\", True):\n+        # LLaMA fast tokenizer causes protobuf errors in some environments.\n+        # However, we found that the below LLaMA fast tokenizer works well in\n+        # most environments.\n+        model_name = \"hf-internal-testing/llama-tokenizer\"\n+        logger.info(\n+            f\"Using the LLaMA fast tokenizer in '{model_name}' to avoid \"\n+            \"potential protobuf errors.\")\n+    elif config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:\n         if getattr(kwargs, \"use_fast\", False) == True:\n             raise ValueError(\n                 f\"Cannot use the fast tokenizer for {config.model_type} due to \"\n", "issue": "Bug in LLaMA fast tokenizer\nIn my environment, using the LLaMA fast tokenizer raises an error about protobuf:\r\n```\r\n  File \"/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/convert_slow_tokenizer.py\", line 445, in __init__\r\n    from .utils import sentencepiece_model_pb2 as model_pb2\r\n  File \"/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/utils/sentencepiece_model_pb2.py\", line 91, in <module>\r\n    _descriptor.EnumValueDescriptor(\r\n  File \"/opt/conda/envs/dev/lib/python3.9/site-packages/google/protobuf/descriptor.py\", line 796, in __new__\r\n    _message.Message._CheckCalledFromGeneratedFile()\r\nTypeError: Descriptors cannot not be created directly.\r\nIf this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.\r\nIf you cannot immediately regenerate your protos, some other possible workarounds are:\r\n 1. Downgrade the protobuf package to 3.20.x or lower.\r\n 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).\r\n```\r\n\r\nWhile downgrading the protobuf version removed the error, it slowed down the initialization time by ~8x.\r\n* Initialization with fast tokenizer & protobuf==3.20.3 \r\n```\r\nreal    4m18.476s\r\nuser    3m52.706s\r\nsys     0m27.644s\r\n```\r\n* Initialization with slow tokenizer\r\n```\r\nreal    0m27.620s\r\nuser    0m8.011s\r\nsys     0m19.237s\r\n```\r\n\n", "before_files": [{"content": "\"\"\"Sampling parameters for text generation.\"\"\"\nfrom typing import List, Optional, Union\n\n\nclass SamplingParams:\n    \"\"\"Sampling parameters for text generation.\n\n    Overall, we follow the sampling parameters from the OpenAI text completion\n    API (https://platform.openai.com/docs/api-reference/completions/create).\n    In addition, we support beam search, which is not supported by OpenAI.\n\n    Args:\n        n: Number of output sequences to return for the given prompt.\n        best_of: Number of output sequences that are generated from the prompt.\n            From these `best_of` sequences, the top `n` sequences are returned.\n            `best_of` must be greater than or equal to `n`. This is treated as\n            the beam width when `use_beam_search` is True. By default, `best_of`\n            is set to `n`.\n        presence_penalty: Float that penalizes new tokens based on whether they\n            appear in the generated text so far. Values > 0 encourage the model\n            to use new tokens, while values < 0 encourage the model to repeat\n            tokens.\n        frequency_penalty: Float that penalizes new tokens based on their\n            frequency in the generated text so far. Values > 0 encourage the\n            model to use new tokens, while values < 0 encourage the model to\n            repeat tokens.\n        temperature: Float that controls the randomness of the sampling. Lower\n            values make the model more deterministic, while higher values make\n            the model more random. Zero means greedy sampling.\n        top_p: Float that controls the cumulative probability of the top tokens\n            to consider. Must be in (0, 1]. Set to 1 to consider all tokens.\n        top_k: Integer that controls the number of top tokens to consider. Set\n            to -1 to consider all tokens.\n        use_beam_search: Whether to use beam search instead of sampling.\n        stop: List of strings that stop the generation when they are generated.\n            The returned output will not contain the stop strings.\n        ignore_eos: Whether to ignore the EOS token and continue generating\n            tokens after the EOS token is generated.\n        max_tokens: Maximum number of tokens to generate per output sequence.\n        logprobs: Number of log probabilities to return per output token.\n    \"\"\"\n\n    def __init__(\n        self,\n        n: int = 1,\n        best_of: Optional[int] = None,\n        presence_penalty: float = 0.0,\n        frequency_penalty: float = 0.0,\n        temperature: float = 1.0,\n        top_p: float = 1.0,\n        top_k: int = -1,\n        use_beam_search: bool = False,\n        stop: Union[str, List[str]] = [],\n        ignore_eos: bool = False,\n        max_tokens: int = 16,\n        logprobs: Optional[int] = None,\n    ) -> None:\n        self.n = n\n        self.best_of = best_of if best_of is not None else n\n        self.presence_penalty = presence_penalty\n        self.frequency_penalty = frequency_penalty\n        self.temperature = temperature\n        self.top_p = top_p\n        self.top_k = top_k\n        self.use_beam_search = use_beam_search\n        self.stop = [stop] if isinstance(stop, str) else list(stop)\n        self.ignore_eos = ignore_eos\n        self.max_tokens = max_tokens\n        self.logprobs = logprobs\n\n        self._verify_args()\n        if self.use_beam_search:\n            self._verity_beam_search()\n        elif self.temperature == 0.0:\n            # Zero temperature means greedy sampling.\n            self._verify_greedy_sampling()\n\n    def _verify_args(self) -> None:\n        if self.n < 1:\n            raise ValueError(f\"n must be at least 1, got {self.n}.\")\n        if self.best_of < self.n:\n            raise ValueError(f\"best_of must be greater than or equal to n, \"\n                             f\"got n={self.n} and best_of={self.best_of}.\")\n        if not -2.0 <= self.presence_penalty <= 2.0:\n            raise ValueError(\"presence_penalty must be in [-2, 2], got \"\n                             f\"{self.presence_penalty}.\")\n        if not -2.0 <= self.frequency_penalty <= 2.0:\n            raise ValueError(\"frequency_penalty must be in [-2, 2], got \"\n                             f\"{self.frequency_penalty}.\")\n        if self.temperature < 0.0:\n            raise ValueError(\n                f\"temperature must be non-negative, got {self.temperature}.\")\n        if not 0.0 < self.top_p <= 1.0:\n            raise ValueError(f\"top_p must be in (0, 1], got {self.top_p}.\")\n        if self.top_k < -1 or self.top_k == 0:\n            raise ValueError(f\"top_k must be -1 (disable), or at least 1, \"\n                             f\"got {self.top_k}.\")\n        if self.max_tokens < 1:\n            raise ValueError(\n                f\"max_tokens must be at least 1, got {self.max_tokens}.\")\n        if self.logprobs is not None and self.logprobs < 0:\n            raise ValueError(\n                f\"logprobs must be non-negative, got {self.logprobs}.\")\n\n    def _verity_beam_search(self) -> None:\n        if self.best_of == 1:\n            raise ValueError(\"best_of must be greater than 1 when using beam \"\n                             f\"search. Got {self.best_of}.\")\n        if self.temperature > 0.0:\n            raise ValueError(\"temperature must be 0 when using beam search.\")\n        if self.top_p < 1.0:\n            raise ValueError(\"top_p must be 1 when using beam search.\")\n        if self.top_k != -1:\n            raise ValueError(\"top_k must be -1 when using beam search.\")\n\n    def _verify_greedy_sampling(self) -> None:\n        if self.best_of > 1:\n            raise ValueError(\"best_of must be 1 when using greedy sampling.\"\n                             f\"Got {self.best_of}.\")\n        if self.top_p < 1.0:\n            raise ValueError(\"top_p must be 1 when using greedy sampling.\")\n        if self.top_k != -1:\n            raise ValueError(\"top_k must be -1 when using greedy sampling.\")\n\n    def __repr__(self) -> str:\n        return (f\"SamplingParams(n={self.n}, \"\n                f\"best_of={self.best_of}, \"\n                f\"presence_penalty={self.presence_penalty}, \"\n                f\"frequency_penalty={self.frequency_penalty}, \"\n                f\"temperature={self.temperature}, \"\n                f\"top_p={self.top_p}, \"\n                f\"top_k={self.top_k},\"\n                f\"use_beam_search={self.use_beam_search}, \"\n                f\"stop={self.stop}, \"\n                f\"ignore_eos={self.ignore_eos}, \"\n                f\"max_tokens={self.max_tokens}, \"\n                f\"logprobs={self.logprobs})\")\n", "path": "cacheflow/sampling_params.py"}, {"content": "from typing import List, Tuple, Union\n\nfrom transformers import (AutoConfig, AutoTokenizer, PreTrainedTokenizer,\n                          PreTrainedTokenizerFast)\n\nfrom cacheflow.logger import init_logger\n\nlogger = init_logger(__name__)\n\n_MODEL_TYPES_WITH_SLOW_TOKENIZER = [\n    # LLaMA fast tokenizer has a bug related to protobuf.\n    # See https://github.com/WoosukKwon/cacheflow/issues/80#issue-1698550554\n    \"llama\",\n]\n\n\ndef get_tokenizer(\n    model_name: str,\n    *args,\n    **kwargs,\n) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:\n    config = AutoConfig.from_pretrained(model_name)\n    if config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:\n        if getattr(kwargs, \"use_fast\", False) == True:\n            raise ValueError(\n                f\"Cannot use the fast tokenizer for {config.model_type} due to \"\n                \"bugs in the fast tokenizer.\")\n        logger.info(\n            f\"Using the slow tokenizer for {config.model_type} due to bugs in \"\n            \"the fast tokenizer. This could potentially lead to performance \"\n            \"degradation.\")\n        kwargs[\"use_fast\"] = False\n    return AutoTokenizer.from_pretrained(model_name, *args, **kwargs)\n\n\ndef detokenize_incrementally(\n    tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast],\n    prev_output_tokens: List[str],\n    new_token_id: int,\n    skip_special_tokens: bool,\n) -> Tuple[str, str]:\n    \"\"\"Detokenizes the new token in conjuction with the previous output tokens.\n\n    NOTE: This function does not update prev_output_tokens.\n\n    Returns:\n        new_token: The new token as a string.\n        output_text: The new output text as a string.\n    \"\"\"\n    new_token = tokenizer.convert_ids_to_tokens(\n        new_token_id, skip_special_tokens=skip_special_tokens)\n    output_tokens = prev_output_tokens + [new_token]\n\n    # Convert the tokens to a string.\n    # Optimization: If the tokenizer does not have `added_tokens_encoder`,\n    # then we can directly use `convert_tokens_to_string`.\n    if not getattr(tokenizer, \"added_tokens_encoder\", {}):\n        output_text = tokenizer.convert_tokens_to_string(output_tokens)\n        return new_token, output_text\n\n    # Adapted from https://github.com/huggingface/transformers/blob/v4.28.0/src/transformers/tokenization_utils.py#L921\n    # NOTE(woosuk): The following code is slow because it runs a for loop over\n    # the output_tokens. In Python, running a for loop over a list can be slow\n    # even when the loop body is very simple.\n    sub_texts = []\n    current_sub_text = []\n    for token in output_tokens:\n        if skip_special_tokens and token in tokenizer.all_special_ids:\n            continue\n        if token in tokenizer.added_tokens_encoder:\n            if current_sub_text:\n                sub_text = tokenizer.convert_tokens_to_string(current_sub_text)\n                sub_texts.append(sub_text)\n                current_sub_text = []\n            sub_texts.append(token)\n        else:\n            current_sub_text.append(token)\n    if current_sub_text:\n        sub_text = tokenizer.convert_tokens_to_string(current_sub_text)\n        sub_texts.append(sub_text)\n    output_text = \" \".join(sub_texts)\n    return new_token, output_text\n", "path": "cacheflow/server/tokenizer_utils.py"}], "after_files": [{"content": "\"\"\"Sampling parameters for text generation.\"\"\"\nfrom typing import List, Optional, Union\n\n\nclass SamplingParams:\n    \"\"\"Sampling parameters for text generation.\n\n    Overall, we follow the sampling parameters from the OpenAI text completion\n    API (https://platform.openai.com/docs/api-reference/completions/create).\n    In addition, we support beam search, which is not supported by OpenAI.\n\n    Args:\n        n: Number of output sequences to return for the given prompt.\n        best_of: Number of output sequences that are generated from the prompt.\n            From these `best_of` sequences, the top `n` sequences are returned.\n            `best_of` must be greater than or equal to `n`. This is treated as\n            the beam width when `use_beam_search` is True. By default, `best_of`\n            is set to `n`.\n        presence_penalty: Float that penalizes new tokens based on whether they\n            appear in the generated text so far. Values > 0 encourage the model\n            to use new tokens, while values < 0 encourage the model to repeat\n            tokens.\n        frequency_penalty: Float that penalizes new tokens based on their\n            frequency in the generated text so far. Values > 0 encourage the\n            model to use new tokens, while values < 0 encourage the model to\n            repeat tokens.\n        temperature: Float that controls the randomness of the sampling. Lower\n            values make the model more deterministic, while higher values make\n            the model more random. Zero means greedy sampling.\n        top_p: Float that controls the cumulative probability of the top tokens\n            to consider. Must be in (0, 1]. Set to 1 to consider all tokens.\n        top_k: Integer that controls the number of top tokens to consider. Set\n            to -1 to consider all tokens.\n        use_beam_search: Whether to use beam search instead of sampling.\n        stop: List of strings that stop the generation when they are generated.\n            The returned output will not contain the stop strings.\n        ignore_eos: Whether to ignore the EOS token and continue generating\n            tokens after the EOS token is generated.\n        max_tokens: Maximum number of tokens to generate per output sequence.\n        logprobs: Number of log probabilities to return per output token.\n    \"\"\"\n\n    def __init__(\n        self,\n        n: int = 1,\n        best_of: Optional[int] = None,\n        presence_penalty: float = 0.0,\n        frequency_penalty: float = 0.0,\n        temperature: float = 1.0,\n        top_p: float = 1.0,\n        top_k: int = -1,\n        use_beam_search: bool = False,\n        stop: Union[str, List[str]] = [],\n        ignore_eos: bool = False,\n        max_tokens: int = 16,\n        logprobs: Optional[int] = None,\n    ) -> None:\n        self.n = n\n        self.best_of = best_of if best_of is not None else n\n        self.presence_penalty = presence_penalty\n        self.frequency_penalty = frequency_penalty\n        self.temperature = temperature\n        self.top_p = top_p\n        self.top_k = top_k\n        self.use_beam_search = use_beam_search\n        self.stop = [stop] if isinstance(stop, str) else list(stop)\n        self.ignore_eos = ignore_eos\n        self.max_tokens = max_tokens\n        self.logprobs = logprobs\n\n        self._verify_args()\n        if self.use_beam_search:\n            self._verity_beam_search()\n        elif self.temperature == 0.0:\n            # Zero temperature means greedy sampling.\n            self._verify_greedy_sampling()\n\n    def _verify_args(self) -> None:\n        if self.n < 1:\n            raise ValueError(f\"n must be at least 1, got {self.n}.\")\n        if self.best_of < self.n:\n            raise ValueError(f\"best_of must be greater than or equal to n, \"\n                             f\"got n={self.n} and best_of={self.best_of}.\")\n        if not -2.0 <= self.presence_penalty <= 2.0:\n            raise ValueError(\"presence_penalty must be in [-2, 2], got \"\n                             f\"{self.presence_penalty}.\")\n        if not -2.0 <= self.frequency_penalty <= 2.0:\n            raise ValueError(\"frequency_penalty must be in [-2, 2], got \"\n                             f\"{self.frequency_penalty}.\")\n        if self.temperature < 0.0:\n            raise ValueError(\n                f\"temperature must be non-negative, got {self.temperature}.\")\n        if not 0.0 < self.top_p <= 1.0:\n            raise ValueError(f\"top_p must be in (0, 1], got {self.top_p}.\")\n        if self.top_k < -1 or self.top_k == 0:\n            raise ValueError(f\"top_k must be -1 (disable), or at least 1, \"\n                             f\"got {self.top_k}.\")\n        if self.max_tokens < 1:\n            raise ValueError(\n                f\"max_tokens must be at least 1, got {self.max_tokens}.\")\n        if self.logprobs is not None and self.logprobs < 0:\n            raise ValueError(\n                f\"logprobs must be non-negative, got {self.logprobs}.\")\n\n    def _verity_beam_search(self) -> None:\n        if self.best_of == 1:\n            raise ValueError(\"best_of must be greater than 1 when using beam \"\n                             f\"search. Got {self.best_of}.\")\n        if self.temperature > 0.0:\n            raise ValueError(\"temperature must be 0 when using beam search.\")\n        if self.top_p < 1.0:\n            raise ValueError(\"top_p must be 1 when using beam search.\")\n        if self.top_k != -1:\n            raise ValueError(\"top_k must be -1 when using beam search.\")\n\n    def _verify_greedy_sampling(self) -> None:\n        if self.best_of > 1:\n            raise ValueError(\"best_of must be 1 when using greedy sampling.\"\n                             f\"Got {self.best_of}.\")\n        if self.top_p < 1.0:\n            raise ValueError(\"top_p must be 1 when using greedy sampling.\")\n        if self.top_k != -1:\n            raise ValueError(\"top_k must be -1 when using greedy sampling.\")\n\n    def __repr__(self) -> str:\n        return (f\"SamplingParams(n={self.n}, \"\n                f\"best_of={self.best_of}, \"\n                f\"presence_penalty={self.presence_penalty}, \"\n                f\"frequency_penalty={self.frequency_penalty}, \"\n                f\"temperature={self.temperature}, \"\n                f\"top_p={self.top_p}, \"\n                f\"top_k={self.top_k}, \"\n                f\"use_beam_search={self.use_beam_search}, \"\n                f\"stop={self.stop}, \"\n                f\"ignore_eos={self.ignore_eos}, \"\n                f\"max_tokens={self.max_tokens}, \"\n                f\"logprobs={self.logprobs})\")\n", "path": "cacheflow/sampling_params.py"}, {"content": "from typing import List, Tuple, Union\n\nfrom transformers import (AutoConfig, AutoTokenizer, PreTrainedTokenizer,\n                          PreTrainedTokenizerFast)\n\nfrom cacheflow.logger import init_logger\n\nlogger = init_logger(__name__)\n\n_MODEL_TYPES_WITH_SLOW_TOKENIZER = []\n\n\ndef get_tokenizer(\n    model_name: str,\n    *args,\n    **kwargs,\n) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:\n    config = AutoConfig.from_pretrained(model_name)\n    if config.model_type == \"llama\" and getattr(kwargs, \"use_fast\", True):\n        # LLaMA fast tokenizer causes protobuf errors in some environments.\n        # However, we found that the below LLaMA fast tokenizer works well in\n        # most environments.\n        model_name = \"hf-internal-testing/llama-tokenizer\"\n        logger.info(\n            f\"Using the LLaMA fast tokenizer in '{model_name}' to avoid \"\n            \"potential protobuf errors.\")\n    elif config.model_type in _MODEL_TYPES_WITH_SLOW_TOKENIZER:\n        if getattr(kwargs, \"use_fast\", False) == True:\n            raise ValueError(\n                f\"Cannot use the fast tokenizer for {config.model_type} due to \"\n                \"bugs in the fast tokenizer.\")\n        logger.info(\n            f\"Using the slow tokenizer for {config.model_type} due to bugs in \"\n            \"the fast tokenizer. This could potentially lead to performance \"\n            \"degradation.\")\n        kwargs[\"use_fast\"] = False\n    return AutoTokenizer.from_pretrained(model_name, *args, **kwargs)\n\n\ndef detokenize_incrementally(\n    tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast],\n    prev_output_tokens: List[str],\n    new_token_id: int,\n    skip_special_tokens: bool,\n) -> Tuple[str, str]:\n    \"\"\"Detokenizes the new token in conjuction with the previous output tokens.\n\n    NOTE: This function does not update prev_output_tokens.\n\n    Returns:\n        new_token: The new token as a string.\n        output_text: The new output text as a string.\n    \"\"\"\n    new_token = tokenizer.convert_ids_to_tokens(\n        new_token_id, skip_special_tokens=skip_special_tokens)\n    output_tokens = prev_output_tokens + [new_token]\n\n    # Convert the tokens to a string.\n    # Optimization: If the tokenizer does not have `added_tokens_encoder`,\n    # then we can directly use `convert_tokens_to_string`.\n    if not getattr(tokenizer, \"added_tokens_encoder\", {}):\n        output_text = tokenizer.convert_tokens_to_string(output_tokens)\n        return new_token, output_text\n\n    # Adapted from https://github.com/huggingface/transformers/blob/v4.28.0/src/transformers/tokenization_utils.py#L921\n    # NOTE(woosuk): The following code is slow because it runs a for loop over\n    # the output_tokens. In Python, running a for loop over a list can be slow\n    # even when the loop body is very simple.\n    sub_texts = []\n    current_sub_text = []\n    for token in output_tokens:\n        if skip_special_tokens and token in tokenizer.all_special_ids:\n            continue\n        if token in tokenizer.added_tokens_encoder:\n            if current_sub_text:\n                sub_text = tokenizer.convert_tokens_to_string(current_sub_text)\n                sub_texts.append(sub_text)\n                current_sub_text = []\n            sub_texts.append(token)\n        else:\n            current_sub_text.append(token)\n    if current_sub_text:\n        sub_text = tokenizer.convert_tokens_to_string(current_sub_text)\n        sub_texts.append(sub_text)\n    output_text = \" \".join(sub_texts)\n    return new_token, output_text\n", "path": "cacheflow/server/tokenizer_utils.py"}]}