utils4text.py
def compute_tokens(subreddit_convs_concise):
    """
    Given the text data from a subreddit, lemmatize and compute the word tokens using the defined function, text_lemmatize.
    Before that, remove the newline tag and expanding the English contraction.
    The reason why the progress_bar is set to false is because of Google Colab's memory limitation.
    If it's not the problem in your local machine, you could simply convert it to be true to check the processing status.
    
    Arg:
        subreddit_convs_concise: A conversation dataframe from a subreddit.
    Return:
        subreddit_tokens: A series with each row containing a list of word tokens from either post or comment.
    """
    # copy the text column from original dataframe
    subreddit_text = subreddit_convs_concise['text'].copy()

    # expanding contraction
    subreddit_text = subreddit_text.swifter.progress_bar(False).apply(lambda text: text.replace('\n', ' '))\
                                   .swifter.progress_bar(False).apply(lambda text: ' '.join([contractions.fix(word) for word in text.split()]))

    # lemmatize
    lemmatizer = WordNetLemmatizer()
    subreddit_tokens = subreddit_text.swifter.progress_bar(False).apply(lambda text: text_lemmatize(text, lemmatizer))

    return subreddit_tokens

    Given the text data from a subreddit, lemmatize and compute the word tokens using the defined function, text_lemmatize.
    Before that, remove the newline tag and expanding the English contraction.
    The reason why the progress_bar is set to false is because of Google Colab's memory limitation.
    If it's not the problem in your local machine, you could simply convert it to be true to check the processing status.
    
    Arg:
        subreddit_convs_concise: A conversation dataframe from a subreddit.
    Return:
        subreddit_tokens: A series with each row containing a list of word tokens from either post or comment.
    
Given the text data from a subreddit, lemmatize and compute the word tokens using the defined function, text_lemmatize.
Before that, remove the newline tag and expanding the English contraction.
The reason why the progress_bar is set to false is because of Google Colab's memory limitation.
If it's not the problem in your local machine, you could simply convert it to be true to check the processing status.

A conversation dataframe from a subreddit.
Return:
subreddit_tokens: A series with each row containing a list of word tokens from either post or comment.
Given the text data from a subreddit lemmatize and compute the word tokens using the defined function text_lemmatize . Before that remove the newline tag and expanding the English contraction . The reason why the progress_bar is set to false is because of Google Colab ' s memory limitation . If it ' s not the problem in your local machine you could simply convert it to be true to check the processing status . A conversation dataframe from a subreddit . Return : subreddit_tokens : A series with each row containing a list of word tokens from either post or comment .
def compute_tokens(subreddit_convs_concise):
    subreddit_text = subreddit_convs_concise['text'].copy()
    subreddit_text = subreddit_text.swifter.progress_bar(False).apply(lambda text: text.replace('\n', ' '))\
                                   .swifter.progress_bar(False).apply(lambda text: ' '.join([contractions.fix(word) for word in text.split()]))
    lemmatizer = WordNetLemmatizer()
    subreddit_tokens = subreddit_text.swifter.progress_bar(False).apply(lambda text: text_lemmatize(text, lemmatizer))
    return subreddit_tokens
def compute_tokens ( subreddit_convs_concise ) : subreddit_text = subreddit_convs_concise [ 'text' ] . copy ( ) subreddit_text = subreddit_text . swifter . progress_bar ( False ) . apply ( lambda text : text . replace ( '\n' , ' ' ) ) . swifter . progress_bar ( False ) . apply ( lambda text : ' ' . join ( [ contractions . fix ( word ) for word in text . split ( ) ] ) ) lemmatizer = WordNetLemmatizer ( ) subreddit_tokens = subreddit_text . swifter . progress_bar ( False ) . apply ( lambda text : text_lemmatize ( text , lemmatizer ) ) return subreddit_tokens
Given the text data from a subreddit, lemmatize and compute the word tokens using the defined function, text_lemmatize.
Given the text data from a subreddit lemmatize and compute the word tokens using the defined function text_lemmatize .
"""
    Given the text data from a subreddit, lemmatize and compute the word tokens using the defined function, text_lemmatize.
    Before that, remove the newline tag and expanding the English contraction.
    The reason why the progress_bar is set to false is because of Google Colab's memory limitation.
    If it's not the problem in your local machine, you could simply convert it to be true to check the processing status.
    
    Arg:
        subreddit_convs_concise: A conversation dataframe from a subreddit.
    Return:
        subreddit_tokens: A series with each row containing a list of word tokens from either post or comment.
    """ # copy the text column from original dataframe # expanding contraction # lemmatize
subreddit_convs_concise