ktrain/text/data.py
def texts_from_array(x_train, y_train, x_test=None, y_test=None, 
                   class_names = [],
                   max_features=MAX_FEATURES, maxlen=MAXLEN, 
                   val_pct=0.1, ngram_range=1, preprocess_mode='standard', 
                   lang=None, # auto-detected
                   random_state=None,
                   verbose=1):
    """
    Loads and preprocesses text data from arrays.
    texts_from_array can handle data for both text classification
    and text regression.  If class_names is empty, a regression task is assumed.
    Args:
        x_train(list): list of training texts 
        y_train(list): labels in one of the following forms:
                       1. list of integers representing classes (class_names is required)
                       2. list of strings representing classes (class_names is required)
                       3. a one or multi hot encoded array representing classes (class_names is required)
                       4. numerical values for text regresssion (class_names should be left empty)
        x_val(list): list of training texts 
        y_val(list): labels in one of the following forms:
                      1. list of integers representing classes (class_names is required)
                      2. list of strings representing classes (class_names is required)
                      3. a one or multi hot encoded array representing classes (class_names is required)
                      4. numerical values for text regression (class_names should be left empty)
        class_names (list): list of strings representing class labels
                            shape should be (num_examples,1) or (num_examples,)
        max_features(int): max num of words to consider in vocabulary
        maxlen(int): each document can be of most <maxlen> words. 0 is used as padding ID.
        ngram_range(int): size of multi-word phrases to consider
                          e.g., 2 will consider both 1-word phrases and 2-word phrases
                               limited by max_features
        val_pct(float): Proportion of training to use for validation.
                        Has no effect if x_val and  y_val is supplied.
        preprocess_mode (str):  Either 'standard' (normal tokenization) or 'bert'
                                tokenization and preprocessing for use with 
                                BERT text classification model.
        lang (str):            language.  Auto-detected if None.
        random_state(int):      If integer is supplied, train/test split is reproducible.
                                If None, train/test split will be random.
        verbose (boolean): verbosity
    """

    if not class_names and verbose:
        #classes =  list(set(y_train))
        #classes.sort()
        #class_names = ["%s" % (c) for c in classes]
        print('task: text regression (supply class_names argument if this is supposed to be classification task)')
    else:
        print('task: text classification')

    if x_test is None or y_test is None:
        x_train, x_test, y_train, y_test = train_test_split(x_train, y_train, 
                                                            test_size=val_pct,
                                                            random_state=random_state)

    # convert string labels to integers, if necessary
    if isinstance(y_train[0], str):
        if not isinstance(y_test[0], str): 
            raise ValueError('y_train contains strings, but y_test does not')
        from sklearn.preprocessing import LabelEncoder
        encoder = LabelEncoder()
        encoder.fit(y_train)
        y_train = encoder.transform(y_train)
        y_test = encoder.transform(y_test)


    # detect language
    if lang is None: lang = tpp.detect_lang(x_train)
    check_unsupported_lang(lang, preprocess_mode)

    # return preprocessed the texts
    preproc_type = tpp.TEXT_PREPROCESSORS.get(preprocess_mode, None)
    if None: raise ValueError('unsupported preprocess_mode')
    preproc = preproc_type(maxlen,
                           max_features,
                           classes = class_names,
                           lang=lang, ngram_range=ngram_range)
    trn = preproc.preprocess_train(x_train, y_train, verbose=verbose)
    val = preproc.preprocess_test(x_test,  y_test, verbose=verbose)
    return (trn, val, preproc)

    Loads and preprocesses text data from arrays.
    texts_from_array can handle data for both text classification
    and text regression.  If class_names is empty, a regression task is assumed.
    Args:
        x_train(list): list of training texts 
        y_train(list): labels in one of the following forms:
                       1. list of integers representing classes (class_names is required)
                       2. list of strings representing classes (class_names is required)
                       3. a one or multi hot encoded array representing classes (class_names is required)
                       4. numerical values for text regresssion (class_names should be left empty)
        x_val(list): list of training texts 
        y_val(list): labels in one of the following forms:
                      1. list of integers representing classes (class_names is required)
                      2. list of strings representing classes (class_names is required)
                      3. a one or multi hot encoded array representing classes (class_names is required)
                      4. numerical values for text regression (class_names should be left empty)
        class_names (list): list of strings representing class labels
                            shape should be (num_examples,1) or (num_examples,)
        max_features(int): max num of words to consider in vocabulary
        maxlen(int): each document can be of most <maxlen> words. 0 is used as padding ID.
        ngram_range(int): size of multi-word phrases to consider
                          e.g., 2 will consider both 1-word phrases and 2-word phrases
                               limited by max_features
        val_pct(float): Proportion of training to use for validation.
                        Has no effect if x_val and  y_val is supplied.
        preprocess_mode (str):  Either 'standard' (normal tokenization) or 'bert'
                                tokenization and preprocessing for use with 
                                BERT text classification model.
        lang (str):            language.  Auto-detected if None.
        random_state(int):      If integer is supplied, train/test split is reproducible.
                                If None, train/test split will be random.
        verbose (boolean): verbosity
    
Loads and preprocesses text data from arrays.
texts_from_array can handle data for both text classification
and text regression.  If class_names is empty, a regression task is assumed.
Loads and preprocesses text data from arrays . texts_from_array can handle data for both text classification and text regression . If class_names is empty a regression task is assumed .
def texts_from_array(x_train, y_train, x_test=None, y_test=None, 
                   class_names = [],
                   max_features=MAX_FEATURES, maxlen=MAXLEN, 
                   val_pct=0.1, ngram_range=1, preprocess_mode='standard', 
                   lang=None, 
                   random_state=None,
                   verbose=1):
    if not class_names and verbose:
        print('task: text regression (supply class_names argument if this is supposed to be classification task)')
    else:
        print('task: text classification')
    if x_test is None or y_test is None:
        x_train, x_test, y_train, y_test = train_test_split(x_train, y_train, 
                                                            test_size=val_pct,
                                                            random_state=random_state)
    if isinstance(y_train[0], str):
        if not isinstance(y_test[0], str): 
            raise ValueError('y_train contains strings, but y_test does not')
        from sklearn.preprocessing import LabelEncoder
        encoder = LabelEncoder()
        encoder.fit(y_train)
        y_train = encoder.transform(y_train)
        y_test = encoder.transform(y_test)
    if lang is None: lang = tpp.detect_lang(x_train)
    check_unsupported_lang(lang, preprocess_mode)
    preproc_type = tpp.TEXT_PREPROCESSORS.get(preprocess_mode, None)
    if None: raise ValueError('unsupported preprocess_mode')
    preproc = preproc_type(maxlen,
                           max_features,
                           classes = class_names,
                           lang=lang, ngram_range=ngram_range)
    trn = preproc.preprocess_train(x_train, y_train, verbose=verbose)
    val = preproc.preprocess_test(x_test,  y_test, verbose=verbose)
    return (trn, val, preproc)
def texts_from_array ( x_train , y_train , x_test = None , y_test = None , class_names = [ ] , max_features = MAX_FEATURES , maxlen = MAXLEN , val_pct = 0.1 , ngram_range = 1 , preprocess_mode = 'standard' , lang = None , random_state = None , verbose = 1 ) : if not class_names and verbose : print ( 'task: text regression (supply class_names argument if this is supposed to be classification task)' ) else : print ( 'task: text classification' ) if x_test is None or y_test is None : x_train , x_test , y_train , y_test = train_test_split ( x_train , y_train , test_size = val_pct , random_state = random_state ) if isinstance ( y_train [ 0 ] , str ) : if not isinstance ( y_test [ 0 ] , str ) : raise ValueError ( 'y_train contains strings, but y_test does not' ) from sklearn . preprocessing import LabelEncoder encoder = LabelEncoder ( ) encoder . fit ( y_train ) y_train = encoder . transform ( y_train ) y_test = encoder . transform ( y_test ) if lang is None : lang = tpp . detect_lang ( x_train ) check_unsupported_lang ( lang , preprocess_mode ) preproc_type = tpp . TEXT_PREPROCESSORS . get ( preprocess_mode , None ) if None : raise ValueError ( 'unsupported preprocess_mode' ) preproc = preproc_type ( maxlen , max_features , classes = class_names , lang = lang , ngram_range = ngram_range ) trn = preproc . preprocess_train ( x_train , y_train , verbose = verbose ) val = preproc . preprocess_test ( x_test , y_test , verbose = verbose ) return ( trn , val , preproc )
Loads and preprocesses text data from arrays.
Loads and preprocesses text data from arrays .
# auto-detected """
    Loads and preprocesses text data from arrays.
    texts_from_array can handle data for both text classification
    and text regression.  If class_names is empty, a regression task is assumed.
    Args:
        x_train(list): list of training texts 
        y_train(list): labels in one of the following forms:
                       1. list of integers representing classes (class_names is required)
                       2. list of strings representing classes (class_names is required)
                       3. a one or multi hot encoded array representing classes (class_names is required)
                       4. numerical values for text regresssion (class_names should be left empty)
        x_val(list): list of training texts 
        y_val(list): labels in one of the following forms:
                      1. list of integers representing classes (class_names is required)
                      2. list of strings representing classes (class_names is required)
                      3. a one or multi hot encoded array representing classes (class_names is required)
                      4. numerical values for text regression (class_names should be left empty)
        class_names (list): list of strings representing class labels
                            shape should be (num_examples,1) or (num_examples,)
        max_features(int): max num of words to consider in vocabulary
        maxlen(int): each document can be of most <maxlen> words. 0 is used as padding ID.
        ngram_range(int): size of multi-word phrases to consider
                          e.g., 2 will consider both 1-word phrases and 2-word phrases
                               limited by max_features
        val_pct(float): Proportion of training to use for validation.
                        Has no effect if x_val and  y_val is supplied.
        preprocess_mode (str):  Either 'standard' (normal tokenization) or 'bert'
                                tokenization and preprocessing for use with 
                                BERT text classification model.
        lang (str):            language.  Auto-detected if None.
        random_state(int):      If integer is supplied, train/test split is reproducible.
                                If None, train/test split will be random.
        verbose (boolean): verbosity
    """ #classes =  list(set(y_train)) #classes.sort() #class_names = ["%s" % (c) for c in classes] # convert string labels to integers, if necessary # detect language # return preprocessed the texts
x_train
list of training texts
list of training texts
y_train
labels in one of the following forms:
1. list of integers representing classes (class_names is required)
2. list of strings representing classes (class_names is required)
3. a one or multi hot encoded array representing classes (class_names is required)
4. numerical values for text regresssion (class_names should be left empty)
labels in one of the following forms : 1 . list of integers representing classes ( class_names is required ) 2 . list of strings representing classes ( class_names is required ) 3 . a one or multi hot encoded array representing classes ( class_names is required ) 4 . numerical values for text regresssion ( class_names should be left empty )
x_test
y_test
class_names
list of strings representing class labels
shape should be (num_examples,1) or (num_examples,)
list of strings representing class labels shape should be ( num_examples 1 ) or ( num_examples )
max_features
max num of words to consider in vocabulary
max num of words to consider in vocabulary
maxlen
each document can be of most  words. 0 is used as padding ID.
each document can be of most words . 0 is used as padding ID .
val_pct
Proportion of training to use for validation.
Has no effect if x_val and  y_val is supplied.
Proportion of training to use for validation . Has no effect if x_val and y_val is supplied .
ngram_range
size of multi-word phrases to consider
e.g., 2 will consider both 1-word phrases and 2-word phrases
limited by max_features
size of multi - word phrases to consider e . g . 2 will consider both 1 - word phrases and 2 - word phrases limited by max_features
preprocess_mode
Either 'standard' (normal tokenization) or 'bert'
tokenization and preprocessing for use with
BERT text classification model.
Either ' standard ' ( normal tokenization ) or ' bert ' tokenization and preprocessing for use with BERT text classification model .
lang
language.  Auto-detected if None.
language . Auto - detected if None .
random_state
If integer is supplied, train/test split is reproducible.
If None, train/test split will be random.
If integer is supplied train / test split is reproducible . If None train / test split will be random .
verbose
None
x_val
list of training texts
list of training texts
y_val
labels in one of the following forms:
1. list of integers representing classes (class_names is required)
2. list of strings representing classes (class_names is required)
3. a one or multi hot encoded array representing classes (class_names is required)
4. numerical values for text regression (class_names should be left empty)
labels in one of the following forms : 1 . list of integers representing classes ( class_names is required ) 2 . list of strings representing classes ( class_names is required ) 3 . a one or multi hot encoded array representing classes ( class_names is required ) 4 . numerical values for text regression ( class_names should be left empty )