Python: fast iteration through file
Question: I need to iterate through two files many million times, counting the number of
appearances of word pairs throughout the files. (in order to build contingency
table of two words to calculate Fisher's Exact Test score)

I'm currently using

    
    
    from itertools import izip
    src=tuple(open('src.txt','r'))
    tgt=tuple(open('tgt.txt','r'))
    w1count=0
    w2count=0
    w1='someword'
    w2='anotherword'
    for x,y in izip(src,tgt):
        if w1 in x:
             w1count+=1
        if w2 in y:
             w2count+=1
        .....
    

While this is not bad, I want to know if there is any faster way to iterate
through two files, hopefully significantly faster.

I appreciate your help in advance.


Answer: I still don't quite get what exactly you are trying to do, but here's some
example code that might point you in the right direction.

We can use a dictionary or a `collections.Counter` instance to count all
occurring words and pairs _in a single pass_ through the files. After that, we
only need to query the in-memory data.

    
    
    import collections
    import itertools
    import re
    
    def find_words(line):
        for match in re.finditer("\w+", line):
            yield match.group().lower()
    
    counts1 = collections.Counter()
    counts2 = collections.Counter()
    counts_pairs = collections.Counter()
    
    with open("src.txt") as f1, open("tgt.txt") as f2:
        for line1, line2 in itertools.izip(f1, f2):
            words1 = list(find_words(line1))
            words2 = list(find_words(line2))
            counts1.update(words1)
            counts2.update(words2)
            counts_pairs.update(itertools.product(words1, words2))
    
    print counts1["someword"]
    print counts1["anotherword"]
    print counts_pairs["someword", "anotherword"]
    

