Python large text file searching
Question: I have a 500 MB text file that was made a long time ago. It has what looks
like html or xml tags but they are not consistent throughout the file. I am
trying to find the information between two tags that do not match. What I am
using currently works but is very slow: myDict has a list of keywords in it. I
can only guarantee the X+key and /N exist. There are no other tags that are
consistent. The Dictionary has 18000 keys.

    
    
    for key in myDict:
        start_position = 0
    
        start_position = the_whole_file.find('<X>'+key, start_position)
    
        end_position = the_whole_file.find('</N>', start_position)
    
        date =  the_whole_file[start_position:end_position]
    

Is there a way to do this faster?


Answer: reverse the way you are doing it, instead of iterating through the dictionary
and searching for potential matches. iterate through potential matches and
search the dictionary

    
    
    import re
    for part in re.findall("\<X\>(.*)\<\/N\>",the_whole_text):
        key = part.split(" ",1)[0]
        if key in my_dict:
           do_something(part)
    

since dictionary lookup is `O(1)` as opposed to string finding of `O(N)`
(searching the whole file for every key is expensive ...)

so searching your file contents is `~O(500,000,000)` and you are doing that
18,000 times

this way you only search the file once finding all potentials ... then you
look up each one to see if its in your data dictionary

