Seaching big files using list in Python - How can improve the speed?
Question: I have a folder with 300+ .txt files with total size of 15GB+. These files
contain tweets. Each line is a different tweet. I have a list of keywords I'd
like to search the tweets for. I have created a script that searches each line
of every file for every item on my list. If the tweet contains the keyword,
then it writes the line into another file. This is my code:

    
    
    # Search each file for every item in keywords
    print("Searching the files of " + filename + " for the appropriate keywords...")
    for file in os.listdir(file_path):
        f = open(file_path + file, 'r')
        for line in f:
            for key in keywords:
                if re.search(key, line, re.IGNORECASE):
                    db.write(line)
    

This is the format each line has:

    
    
    {"created_at":"Wed Feb 03 06:53:42 +0000 2016","id":694775753754316801,"id_str":"694775753754316801","text":"me with Dibyabhumi Multiple College students https:\/\/t.co\/MqmDwbCDAF","source":"\u003ca href=\"http:\/\/www.facebook.com\/twitter\" rel=\"nofollow\"\u003eFacebook\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":5981342,"id_str":"5981342","name":"Lava Kafle","screen_name":"lkafle","location":"Kathmandu, Nepal","url":"http:\/\/about.me\/lavakafle","description":"@deerwalkinc 24000+ tweeps bigdata  #Team #Genomics  http:\/\/deerwalk.com #Genetic #Testing #population #health #management #BigData #Analytics #java #hadoop","protected":false,"verified":false,"followers_count":24742,"friends_count":23169,"listed_count":1481,"favourites_count":147252,"statuses_count":171880,"created_at":"Sat May 12 04:49:14 +0000 2007","utc_offset":20700,"time_zone":"Kathmandu","geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"EDECE9","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme3\/bg.gif","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme3\/bg.gif","profile_background_tile":false,"profile_link_color":"088253","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"E3E2DE","profile_text_color":"634047","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/677805092859420672\/kzoS-GZ__normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/677805092859420672\/kzoS-GZ__normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/5981342\/1416802075","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"urls":[{"url":"https:\/\/t.co\/MqmDwbCDAF","expanded_url":"http:\/\/fb.me\/Yj1JW9bJ","display_url":"fb.me\/Yj1JW9bJ","indices":[45,68]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1454482422661"}
    

The script works but **it takes a lot of time**. For ~40 keywords it needs
more than 2 hours. Obviously my code is not optimized. What can I do to
improve the speed?

p.s. I have read some relevant questions regarding searching and speed but I
suspect that the problem in my script lies in the fact that I'm using a list
for the keywords. I've tried some of the suggested solutions but to no avail.


Answer: ## 1) External library

If you're willing to lean on external libraries (and time to execute is more
important than the one-off time cost to install), you might be able to gain
some speed by loading each file into a simple Pandas DataFrame and performing
the keyword search as a vector operation. To get the matching tweets, you
would do something like:

    
    
    import pandas as pd
    dataframe_from_text = pd.read_csv("/path/to/file.txt")
    matched_tweets_index =  dataframe_from_text.str.match("keyword_a|keyword_b")
    dataframe_from_text[matched_tweets_index] # Uses the boolean search above to filter the full dataframe
    # You'd then have a mini dataframe of matching tweets in `dataframe_from_text`. 
    # You could loop through these to save them out to a file using the `.to_dict(orient="records")` format.
    

Dataframe operations within Pandas can be really quick so might be worth
investigating.

## 2) Group your regex

Looks like you're not logging which keyword you matched against. If this is
true, you could group your keywords into a single regex query like so:

    
    
    for line in f:
        keywords_combined = "|".join(keywords)
        if re.search(keywords_combined, line, re.IGNORECASE):
            db.write(line)
    

I've not tested this but by reducing the number of loops per line, that could
trim some time off.

