models/train_classifier.py
def tokenize(text):
    ''' This function breaks the texts into tokens, replaces URLs as well as lemmatizes the tokens. 
    
    If the input is invalid or the data does not exist, this function will raise an error.
    
    INPUT:
    text --> defined sentences
    
    OUTPUT:
    clean_tokens --> list of generated tokens
    '''
    
    urls_detected = re.findall(url_regex, text)
    for url in urls_detected:
        text = text.replace(url, "urlplaceholder")

    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()

    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens
 This function breaks the texts into tokens, replaces URLs as well as lemmatizes the tokens. 
    
    If the input is invalid or the data does not exist, this function will raise an error.
    
    INPUT:
    text --> defined sentences
    
    OUTPUT:
    clean_tokens --> list of generated tokens
    
This function breaks the texts into tokens, replaces URLs as well as lemmatizes the tokens.
If the input is invalid or the data does not exist, this function will raise an error.

-> defined sentences

-> list of generated tokens
This function breaks the texts into tokens replaces URLs as well as lemmatizes the tokens . If the input is invalid or the data does not exist this function will raise an error . - > defined sentences - > list of generated tokens
def tokenize(text):
    urls_detected = re.findall(url_regex, text)
    for url in urls_detected:
        text = text.replace(url, "urlplaceholder")
    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()
    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)
    return clean_tokens
def tokenize ( text ) : urls_detected = re . findall ( url_regex , text ) for url in urls_detected : text = text . replace ( url , "urlplaceholder" ) tokens = word_tokenize ( text ) lemmatizer = WordNetLemmatizer ( ) clean_tokens = [ ] for tok in tokens : clean_tok = lemmatizer . lemmatize ( tok ) . lower ( ) . strip ( ) clean_tokens . append ( clean_tok ) return clean_tokens
This function breaks the texts into tokens, replaces URLs as well as lemmatizes the tokens.
This function breaks the texts into tokens replaces URLs as well as lemmatizes the tokens .
''' This function breaks the texts into tokens, replaces URLs as well as lemmatizes the tokens. 
    
    If the input is invalid or the data does not exist, this function will raise an error.
    
    INPUT:
    text --> defined sentences
    
    OUTPUT:
    clean_tokens --> list of generated tokens
    '''
text