paddlenlp/transformers/blenderbot/tokenizer.py
def _tokenize(self, text):
        """
        End-to-end tokenization for Blenderbot models.
        Args:
            text (str): The text to be tokenized.

        Returns:
            list: A list of string representing converted tokens.
        """
        bpe_tokens = []
        re = try_import("regex")
        for token in re.findall(self.pat, text):
            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
            bpe_tokens.extend(
                bpe_token for bpe_token in self.bpe(token).split(' '))
        return bpe_tokens

        End-to-end tokenization for Blenderbot models.
        Args:
            text (str): The text to be tokenized.

        Returns:
            list: A list of string representing converted tokens.
        
End-to-end tokenization for Blenderbot models.
End - to - end tokenization for Blenderbot models .
def _tokenize(self, text):
        bpe_tokens = []
        re = try_import("regex")
        for token in re.findall(self.pat, text):
            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
            bpe_tokens.extend(
                bpe_token for bpe_token in self.bpe(token).split(' '))
        return bpe_tokens
def _tokenize ( self , text ) : bpe_tokens = [ ] re = try_import ( "regex" ) for token in re . findall ( self . pat , text ) : token = '' . join ( self . byte_encoder [ b ] for b in token . encode ( 'utf-8' ) ) bpe_tokens . extend ( bpe_token for bpe_token in self . bpe ( token ) . split ( ' ' ) ) return bpe_tokens
End-to-end tokenization for Blenderbot models.
End - to - end tokenization for Blenderbot models .
"""
        End-to-end tokenization for Blenderbot models.
        Args:
            text (str): The text to be tokenized.

        Returns:
            list: A list of string representing converted tokens.
        """
A list of string representing converted tokens.
A list of string representing converted tokens .
list
self
text
The text to be tokenized.
The text to be tokenized .