models/train_classifier.py
def tokenize(text):
    """The tokenize function will form tokenization for the text messages
        to use for model training and testing

    Args:
        text (DataFrame): the DataFrame with text column for tokenization


    Returns:
        DataFrame: The DataFrame with words tokens and values for modelling

    """

    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()
    wordporter = SnowballStemmer("english")

    stopword = set(stopwords.words("english"))
    
    # Retain alphabetic words: alpha_only
    alpha_only = [t.lower() for t in tokens if t.isalpha()]

    # Remove all stop words: no_stops
    _tokens = [t for t in alpha_only if t not in stopword]

    no_stop_tokens = [wordporter.stem(word) for word in _tokens]

    clean_tokens = []

    # for tok in tokens:
    for tok in no_stop_tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens
The tokenize function will form tokenization for the text messages
        to use for model training and testing

    Args:
        text (DataFrame): the DataFrame with text column for tokenization


    Returns:
        DataFrame: The DataFrame with words tokens and values for modelling

    
The tokenize function will form tokenization for the text messages
to use for model training and testing
The tokenize function will form tokenization for the text messages to use for model training and testing
def tokenize(text):
    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()
    wordporter = SnowballStemmer("english")
    stopword = set(stopwords.words("english"))
    alpha_only = [t.lower() for t in tokens if t.isalpha()]
    _tokens = [t for t in alpha_only if t not in stopword]
    no_stop_tokens = [wordporter.stem(word) for word in _tokens]
    clean_tokens = []
    for tok in no_stop_tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)
    return clean_tokens
def tokenize ( text ) : tokens = word_tokenize ( text ) lemmatizer = WordNetLemmatizer ( ) wordporter = SnowballStemmer ( "english" ) stopword = set ( stopwords . words ( "english" ) ) alpha_only = [ t . lower ( ) for t in tokens if t . isalpha ( ) ] _tokens = [ t for t in alpha_only if t not in stopword ] no_stop_tokens = [ wordporter . stem ( word ) for word in _tokens ] clean_tokens = [ ] for tok in no_stop_tokens : clean_tok = lemmatizer . lemmatize ( tok ) . lower ( ) . strip ( ) clean_tokens . append ( clean_tok ) return clean_tokens
The tokenize function will form tokenization for the text messages
to use for model training and testing
The tokenize function will form tokenization for the text messages to use for model training and testing
"""The tokenize function will form tokenization for the text messages
        to use for model training and testing

    Args:
        text (DataFrame): the DataFrame with text column for tokenization


    Returns:
        DataFrame: The DataFrame with words tokens and values for modelling

    """ # Retain alphabetic words: alpha_only # Remove all stop words: no_stops # for tok in tokens:
The DataFrame with words tokens and values for modelling
The DataFrame with words tokens and values for modelling
DataFrame
text
the DataFrame with text column for tokenization
the DataFrame with text column for tokenization