paddlenlp/transformers/t5/tokenizer.py
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1):
        """
        Build model inputs from a sequence or a pair of sequence.

        An Reformer sequence has the following format:

        - single sequence:      ``X </s>``
        - pair of sequences:        ``A </s> B </s>``

        Args:
            token_ids_0 (List[int]):
                List of IDs to which the special tokens will be added.
            token_ids_1 (List[int], optional):
                Optional second list of IDs for sequence pairs. Defaults to None.

        Returns:
            List[int]: List of input_id with the appropriate special tokens.

        """
        token_ids_0 = self._add_eos_if_not_present(token_ids_0)
        if token_ids_1 is None:
            return token_ids_0
        else:
            token_ids_1 = self._add_eos_if_not_present(token_ids_1)
            return token_ids_0 + token_ids_1

        Build model inputs from a sequence or a pair of sequence.

        An Reformer sequence has the following format:

        - single sequence:      ``X </s>``
        - pair of sequences:        ``A </s> B </s>``

        Args:
            token_ids_0 (List[int]):
                List of IDs to which the special tokens will be added.
            token_ids_1 (List[int], optional):
                Optional second list of IDs for sequence pairs. Defaults to None.

        Returns:
            List[int]: List of input_id with the appropriate special tokens.

        
Build model inputs from a sequence or a pair of sequence.
An Reformer sequence has the following format.


Build model inputs from a sequence or a pair of sequence . An Reformer sequence has the following format .
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1):
        token_ids_0 = self._add_eos_if_not_present(token_ids_0)
        if token_ids_1 is None:
            return token_ids_0
        else:
            token_ids_1 = self._add_eos_if_not_present(token_ids_1)
            return token_ids_0 + token_ids_1
def build_inputs_with_special_tokens ( self , token_ids_0 , token_ids_1 ) : token_ids_0 = self . _add_eos_if_not_present ( token_ids_0 ) if token_ids_1 is None : return token_ids_0 else : token_ids_1 = self . _add_eos_if_not_present ( token_ids_1 ) return token_ids_0 + token_ids_1
Build model inputs from a sequence or a pair of sequence.
Build model inputs from a sequence or a pair of sequence .
"""
        Build model inputs from a sequence or a pair of sequence.

        An Reformer sequence has the following format:

        - single sequence:      ``X </s>``
        - pair of sequences:        ``A </s> B </s>``

        Args:
            token_ids_0 (List[int]):
                List of IDs to which the special tokens will be added.
            token_ids_1 (List[int], optional):
                Optional second list of IDs for sequence pairs. Defaults to None.

        Returns:
            List[int]: List of input_id with the appropriate special tokens.

        """
List of input_id with the appropriate special tokens.
List of input_id with the appropriate special tokens .
List[int]
self
token_ids_0
List of IDs to which the special tokens will be added.
List of IDs to which the special tokens will be added .
token_ids_1
Optional second list of IDs for sequence pairs. Defaults to None.
Optional second list of IDs for sequence pairs . Defaults to None .
None