examples/language_model/chinesebert/utils.py
def convert_example(example, tokenizer, is_test=False):
    """
    Builds model inputs from a sequence for sequence classification tasks. 
    It use `jieba.cut` to tokenize text.

    Args:
        example(obj:`list[str]`): List of input data, containing text and label if it have label.
        tokenizer(obj: paddlenlp.data.JiebaTokenizer): It use jieba to cut the chinese string.
        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.

    Returns:
        input_ids(obj:`list[int]`): The list of token ids.
        valid_length(obj:`int`): The input sequence valid length.
        label(obj:`numpy.array`, data type of int64, optional): The input label if not is_test.
    """

    input_ids = tokenizer.encode(example["text"])
    input_ids = np.array(input_ids, dtype='int64')

    if not is_test:
        label = np.array(example["label"], dtype="int64")
        return input_ids, label
    else:
        return input_ids

    Builds model inputs from a sequence for sequence classification tasks. 
    It use `jieba.cut` to tokenize text.

    Args:
        example(obj:`list[str]`): List of input data, containing text and label if it have label.
        tokenizer(obj: paddlenlp.data.JiebaTokenizer): It use jieba to cut the chinese string.
        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.

    Returns:
        input_ids(obj:`list[int]`): The list of token ids.
        valid_length(obj:`int`): The input sequence valid length.
        label(obj:`numpy.array`, data type of int64, optional): The input label if not is_test.
    
Builds model inputs from a sequence for sequence classification tasks.
It use `jieba.cut` to tokenize text.
Builds model inputs from a sequence for sequence classification tasks . It use ` jieba . cut ` to tokenize text .
def convert_example(example, tokenizer, is_test=False):
    input_ids = tokenizer.encode(example["text"])
    input_ids = np.array(input_ids, dtype='int64')
    if not is_test:
        label = np.array(example["label"], dtype="int64")
        return input_ids, label
    else:
        return input_ids
def convert_example ( example , tokenizer , is_test = False ) : input_ids = tokenizer . encode ( example [ "text" ] ) input_ids = np . array ( input_ids , dtype = 'int64' ) if not is_test : label = np . array ( example [ "label" ] , dtype = "int64" ) return input_ids , label else : return input_ids
Builds model inputs from a sequence for sequence classification tasks.
Builds model inputs from a sequence for sequence classification tasks .
"""
    Builds model inputs from a sequence for sequence classification tasks. 
    It use `jieba.cut` to tokenize text.

    Args:
        example(obj:`list[str]`): List of input data, containing text and label if it have label.
        tokenizer(obj: paddlenlp.data.JiebaTokenizer): It use jieba to cut the chinese string.
        is_test(obj:`False`, defaults to `False`): Whether the example contains label or not.

    Returns:
        input_ids(obj:`list[int]`): The list of token ids.
        valid_length(obj:`int`): The input sequence valid length.
        label(obj:`numpy.array`, data type of int64, optional): The input label if not is_test.
    """
`list[int]`): The list of token ids.
valid_length(obj:`int`): The input sequence valid length.
label(obj:`numpy.array`, data type of int64, optional): The input label if not is_test.
` list [ int ] ` ) : The list of token ids . valid_length ( obj : ` int ` ) : The input sequence valid length . label ( obj : ` numpy . array ` data type of int64 optional ) : The input label if not is_test .
input_ids(obj
example
tokenizer
is_test
example(obj
`list[str]`): List of input data, containing text and label if it have label.
` list [ str ] ` ) : List of input data containing text and label if it have label .
tokenizer(obj
paddlenlp.data.JiebaTokenizer): It use jieba to cut the chinese string.
paddlenlp . data . JiebaTokenizer ) : It use jieba to cut the chinese string .
is_test(obj
`False`, defaults to `False`): Whether the example contains label or not.
` False ` defaults to ` False ` ) : Whether the example contains label or not .