Utility functions for processing tweets or texts I have 4 functions that process texts or tweepy.models.Status object (this object stores text information, author information, retweets &amp; likes information of a specific tweet).

The first function is filter_unique, which is used to exclude duplicate tweets from a list of tweets. The result is a generator.

The function clean_text, is used to remove punctuations, undefined characters, and whitespaces from a string.

The function split_texts has two inputs, the list of tweets/strings and the mode of the splits (naive=True means it only use text.split(), if set to False then it will clean the text first before splitting)

The function total_rts calculates the total number of retweets. string_inclusion='word' means that it will calculate the total retweets of tweets that contain 'word'.

Question: How to make this module more efficient, and perhaps advanced?

We can use the sample tweets testfile.npy which contains objects tweepy.models.Status.



import tweepy
import string

def filter_unique(tweets):

    uniques = set()

    for tweet in tweets:

        if not isinstance(tweet, tweepy.Status):
            raise TypeError('Each element must be of tweepy.Status object')

        try:
            tweet = tweet.retweeted_status
        except:
            pass

        if tweet.id not in uniques:

            uniques.add(tweet.id)

            yield tweet

def clean_text(text):

    punct = string.punctuation
    printable = string.printable
    whitespace = string.whitespace

    table = text.maketrans({key: None for key in printable})
    undef = set(text.translate(table))

    table = text.maketrans({key: None for key in undef})
    removed_undef = text.translate(table)

    table = text.maketrans({key: None for key in punct})
    cleaned = removed_undef.translate(table)

    table = text.maketrans({key: ' ' for key in whitespace})
    cleaned = cleaned.translate(table)

    return cleaned

def split_texts(texts, naive):

    if naive:
        for text in texts:
            yield text.split()
    else:
        for text in texts:
            yield clean_text(text).split()

def total_rts(tweets, string_inclusion = False, naive = True):

    result = 0

    if not string_inclusion :

        result = sum([tweet.retweet_count for tweet in tweets]);

    else:

        if naive:
            try:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in tweet.full_text.split()])
            except:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in tweet.text.split()])
        else:
            try:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in clean_text(tweet.full_text).split()])
            except:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in clean_text(tweet.text).split()])

    return result
A few obligatory PEP-8 rules:


Imports should be grouped in the following order:


standard library imports
related third party imports
local application/library specific imports

Use blank lines in functions, sparingly, to indicate logical sections.
Two blank lines between the import statements and other code.
Two blank lines between each function.
Don't use spaces around the = sign when used to indicate a keyword argument or a default parameter value.


If this is written for python-3.x, I'd also suggest going for Type Hinting.



In the split_texts definition, I'd suggest having naive=False as the default (or naive=True) depending on the more obvious choice of the two.



In the function total_rts (could be renamed to total_retweets), you can avoid one entire level of nesting by returning early in the first if not string_inclusion clause.



The function clean_text makes several transition tables, which could be extracted out of the function and made a constant. More importantly, what I understood from the definition, I think you're (in order):


cleaning the text of all printable characters
cleaning the text of all characters leftover from step 1
cleaning all punctuation from result of step 2
replacing all kinds of whitespace characters with a single ' ' in the result of step 3.


You can define global transition tables as follows:

WHITESPACE = ' ' * len(string.whitespace)
CLEAN_TRANSLATION = str.maketrans(string.whitespace, WHITESPACE, string.punctuation)


Alternatively, you can take the CLEAN_TRANSLATION inside the clean_text function and use the undefined non-printable characters in concatenation to string.punctuation. So, the function becomes:

def clean_text(text):
    undefined_chars = text.translate(
        str.maketrans({c: None for c in string.printable}))
    cleanup_translation = str.maketrans(
        string.whitespace, WHITESPACE, string.punctuation + undefined_chars)
    return text.translate(cleanup_translation)




And, most importantly, the description for each function that you've provided in the question, could be converted to a docstring for each of them.