examples/sentiment_analysis/skep/train_opinion.py
def convert_example_to_feature(example,
                               tokenizer,
                               max_seq_len=512,
                               no_entity_label="O",
                               is_test=False):
    """
    Builds model inputs from a sequence or a pair of sequence for sequence classification tasks
    by concatenating and adding special tokens. And creates a mask from the two sequences passed 
    to be used in a sequence-pair classification task.
        
    A skep_ernie_1.0_large_ch/skep_ernie_2.0_large_en sequence has the following format:
    ::
        - single sequence: ``[CLS] X [SEP]``
        - pair of sequences: ``[CLS] A [SEP] B [SEP]``

    A skep_ernie_1.0_large_ch/skep_ernie_2.0_large_en sequence pair mask has the following format:
    ::

        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
        | first sequence    | second sequence |

    If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

    Args:
        example(obj:`list[str]`): List of input data, containing text and label if it have label.
        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization.
            Sequences longer than this will be truncated, sequences shorter will be padded.
        no_entity_label(obj:`str`, defaults to "O"): The label represents that the token isn't an entity. 
        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.

    Returns:
        input_ids(obj:`list[int]`): The list of token ids.
        token_type_ids(obj: `list[int]`): List of sequence pair mask.
        label(obj:`list[int]`, optional): The input label if not test data.
    """
    tokens = example['tokens']
    labels = example['labels']
    tokenized_input = tokenizer(
        tokens,
        return_length=True,
        is_split_into_words=True,
        max_seq_len=max_seq_len)

    input_ids = np.array(tokenized_input['input_ids'], dtype="int64")
    token_type_ids = np.array(tokenized_input['token_type_ids'], dtype="int64")
    seq_len = np.array(tokenized_input['seq_len'], dtype="int64")

    if is_test:
        return input_ids, token_type_ids, seq_len
    else:
        labels = labels[:(max_seq_len - 2)]
        encoded_label = np.array(
            [no_entity_label] + labels + [no_entity_label], dtype="int64")

        return input_ids, token_type_ids, seq_len, encoded_label

    Builds model inputs from a sequence or a pair of sequence for sequence classification tasks
    by concatenating and adding special tokens. And creates a mask from the two sequences passed 
    to be used in a sequence-pair classification task.
        
    A skep_ernie_1.0_large_ch/skep_ernie_2.0_large_en sequence has the following format:
    ::
        - single sequence: ``[CLS] X [SEP]``
        - pair of sequences: ``[CLS] A [SEP] B [SEP]``

    A skep_ernie_1.0_large_ch/skep_ernie_2.0_large_en sequence pair mask has the following format:
    ::

        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
        | first sequence    | second sequence |

    If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

    Args:
        example(obj:`list[str]`): List of input data, containing text and label if it have label.
        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization.
            Sequences longer than this will be truncated, sequences shorter will be padded.
        no_entity_label(obj:`str`, defaults to "O"): The label represents that the token isn't an entity. 
        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.

    Returns:
        input_ids(obj:`list[int]`): The list of token ids.
        token_type_ids(obj: `list[int]`): List of sequence pair mask.
        label(obj:`list[int]`, optional): The input label if not test data.
    
Builds model inputs from a sequence or a pair of sequence for sequence classification tasks
by concatenating and adding special tokens. And creates a mask from the two sequences passed
to be used in a sequence-pair classification task.







If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).
Builds model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens . And creates a mask from the two sequences passed to be used in a sequence - pair classification task . If ` token_ids_1 ` is ` None ` this method only returns the first portion of the mask ( 0s ) .
def convert_example_to_feature(example,
                               tokenizer,
                               max_seq_len=512,
                               no_entity_label="O",
                               is_test=False):
    tokens = example['tokens']
    labels = example['labels']
    tokenized_input = tokenizer(
        tokens,
        return_length=True,
        is_split_into_words=True,
        max_seq_len=max_seq_len)
    input_ids = np.array(tokenized_input['input_ids'], dtype="int64")
    token_type_ids = np.array(tokenized_input['token_type_ids'], dtype="int64")
    seq_len = np.array(tokenized_input['seq_len'], dtype="int64")
    if is_test:
        return input_ids, token_type_ids, seq_len
    else:
        labels = labels[:(max_seq_len - 2)]
        encoded_label = np.array(
            [no_entity_label] + labels + [no_entity_label], dtype="int64")
        return input_ids, token_type_ids, seq_len, encoded_label
def convert_example_to_feature ( example , tokenizer , max_seq_len = 512 , no_entity_label = "O" , is_test = False ) : tokens = example [ 'tokens' ] labels = example [ 'labels' ] tokenized_input = tokenizer ( tokens , return_length = True , is_split_into_words = True , max_seq_len = max_seq_len ) input_ids = np . array ( tokenized_input [ 'input_ids' ] , dtype = "int64" ) token_type_ids = np . array ( tokenized_input [ 'token_type_ids' ] , dtype = "int64" ) seq_len = np . array ( tokenized_input [ 'seq_len' ] , dtype = "int64" ) if is_test : return input_ids , token_type_ids , seq_len else : labels = labels [ : ( max_seq_len - 2 ) ] encoded_label = np . array ( [ no_entity_label ] + labels + [ no_entity_label ] , dtype = "int64" ) return input_ids , token_type_ids , seq_len , encoded_label
Builds model inputs from a sequence or a pair of sequence for sequence classification tasks
by concatenating and adding special tokens.
Builds model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens .
"""
    Builds model inputs from a sequence or a pair of sequence for sequence classification tasks
    by concatenating and adding special tokens. And creates a mask from the two sequences passed 
    to be used in a sequence-pair classification task.
        
    A skep_ernie_1.0_large_ch/skep_ernie_2.0_large_en sequence has the following format:
    ::
        - single sequence: ``[CLS] X [SEP]``
        - pair of sequences: ``[CLS] A [SEP] B [SEP]``

    A skep_ernie_1.0_large_ch/skep_ernie_2.0_large_en sequence pair mask has the following format:
    ::

        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
        | first sequence    | second sequence |

    If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

    Args:
        example(obj:`list[str]`): List of input data, containing text and label if it have label.
        tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer` 
            which contains most of the methods. Users should refer to the superclass for more information regarding methods.
        max_seq_len(obj:`int`): The maximum total input sequence length after tokenization.
            Sequences longer than this will be truncated, sequences shorter will be padded.
        no_entity_label(obj:`str`, defaults to "O"): The label represents that the token isn't an entity. 
        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.

    Returns:
        input_ids(obj:`list[int]`): The list of token ids.
        token_type_ids(obj: `list[int]`): List of sequence pair mask.
        label(obj:`list[int]`, optional): The input label if not test data.
    """
`list[int]`): The list of token ids.
token_type_ids(obj: `list[int]`): List of sequence pair mask.
label(obj:`list[int]`, optional): The input label if not test data.
` list [ int ] ` ) : The list of token ids . token_type_ids ( obj : ` list [ int ] ` ) : List of sequence pair mask . label ( obj : ` list [ int ] ` optional ) : The input label if not test data .
input_ids(obj
example
tokenizer
max_seq_len
no_entity_label
is_test
example(obj
`list[str]`): List of input data, containing text and label if it have label.
` list [ str ] ` ) : List of input data containing text and label if it have label .
tokenizer(obj
`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
which contains most of the methods. Users should refer to the superclass for more information regarding methods.
` PretrainedTokenizer ` ) : This tokenizer inherits from : class : ` ~paddlenlp . transformers . PretrainedTokenizer ` which contains most of the methods . Users should refer to the superclass for more information regarding methods .
max_seq_len(obj
`int`): The maximum total input sequence length after tokenization.
Sequences longer than this will be truncated, sequences shorter will be padded.
` int ` ) : The maximum total input sequence length after tokenization . Sequences longer than this will be truncated sequences shorter will be padded .
no_entity_label(obj
`str`, defaults to "O"): The label represents that the token isn't an entity.
` str ` defaults to " O " ) : The label represents that the token isn ' t an entity .
is_test(obj
`False`, defaults to `False`): Whether the example contains label or not.
` False ` defaults to ` False ` ) : Whether the example contains label or not .