
tokenizer (PreTrainedTokenizer) —
The tokenizer responsible for encoding the prompt into input tokens.


Returns

str or list of str



The converted prompt


Maybe convert a prompt into a “multi vector”-compatible prompt. If the prompt includes a token that corresponds
to a multi-vector textual inversion embedding, this function will process the prompt so that the special token
is replaced with multiple special tokens each corresponding to on