Computing frequencies fast in Python
Question: I need to count the frequencies of words in a corpus. Usually I use the
Counter class from the collections package.

    
    
    from collections import Counter
    list_of_words = ['one', 'two', 'three', 'three']
    freqs = Counter(list_of_words)
    

However, the corpus I am analysing consists of several million words, so it
would be great if there was a faster way to compute these scores?

Here is the code that reads in the words:

    
    
    from read_cg3 import read_cg3
    
    test = read_cg3('/Users/arashsaidi/Work/Corpus/DUO_Corpus/Bokmaal-tagged-random/DUO_BM_0.txt')
    count = 0
    word_list = []
    for sentence in test:
        for word in sentence:
            count += 1
            word_list.append(word)
    print count
    

The read_cg3 is a module that reads parsed files and returns a list of
sentences. Here is the module:

    
    
    import re
    
    
    def is_number(s):
        try:
            float(s)
            return True
        except ValueError:
            return False
    
    
    def read_cg3(cg3_file):
        """
        Reads a cg3 file and returns a list of each sentence with Token, parsed, and one tag
        :param cg3_file: path to file
        :return: list of words + attributes
        """
        rx_token = re.compile("^\"<(.+?)>\"$")
        rx_attributes = re.compile("^\s+\".+?\"\s+.+$")
        rx_eos = re.compile("^\s*$")
    
        curr_token = None
        curr_word = []
        curr_sentence = []
        result = []
    
        with open(cg3_file) as cg3_file:
            for line in cg3_file:
    
                if rx_token.match(line):
                    curr_token = "\"%s\"" % rx_token.match(line).group(1)
                    # print curr_token
    
                if rx_attributes.match(line):
                    curr_word = line.split()
                    # print curr_word[0], curr_word[1]
                    # print curr_word
                    if curr_token and curr_word:
                        # to get more tags uncomment this and comment below
                        # curr_sentence += [[curr_token] + curr_word]
                        if '$' not in curr_word[0] and not is_number(curr_word[0].strip('"').replace('.', '')) \
                                and len(curr_word[0]) < 30:
                            # curr_sentence += [[curr_token.strip('"')] +
                            # [curr_word[0].lower().strip('"')] + [curr_word[1]]]
                            curr_sentence += [curr_word[0].lower().strip('"')]
                        curr_token = None
                        curr_word = []
    
                if rx_eos.match(line):
                    # print curr_sentence
                    if curr_sentence:
                        result += [curr_sentence]
                    curr_sentence = []
                    curr_token = None
                    curr_word = []
    
        # cleanup if last sentence not EOL
        if curr_token and curr_word:
            print 'cg3 reached end of file and did some cleanup on file {}'.format(cg3_file)
            curr_sentence += [[curr_token] + curr_word]
    
        if curr_sentence:
            print 'cg3 reached end of file and did some cleanup on file {}'.format(cg3_file)
            result += curr_sentence
    
        return result
    

Here is the way the files read by the read_cg3 look like:

    
    
    "<TEKNOLOGI>"
        "teknologi" subst appell mask ub ent 
    "<OG>"
        "og" konj <*> 
    "<UNDERVISNING>"
        "undervisning" subst appell fem ub ent <*> 
    "<|>"
        "$|" clb <overskrift> <<< 
    
    "<En>"
        "en" det mask ent kvant 
    "<intervjuunders¯kelse>"
        "intervjuunders¯kelse" subst appell mask ub ent 
    "<av>"
        "av" prep 
    "<musikklÊreres>"
        "musikklÊrer" subst appell mask ub fl gen 
    "<didaktiske>"
        "didaktisk" adj fl pos 
    "<bruk>"
        "bruk" subst appell mask ub ent 
    "<av>"
        "av" prep 
    "<digitale>"
        "digital" adj fl pos 
    "<verkt¯y>"
        "verkt¯y" subst appell n¯yt ub fl <*¯y> 
    "<i>"
        "i" prep 
    "<undervisningsfaget>"
        "undervisningsfag" subst appell n¯yt be ent 
    "<komposisjon>"
        "komposisjon" subst appell mask ub ent 
    "<i>"
        "i" prep 
    "<videregÂende>"
        "videregÂende" adj ub m/f ent pos 
    "<skole>"
        "skole" subst appell mask ub ent 
    "<|>"
        "$|" clb <overskrift> <<< 
    
    "<Markus>"
        "Markus" subst prop mask 
    "<A.>"
        "A." subst prop fork <*> 
    "<SkjÊrstad>"
        "SkjÊrstad" subst prop <*stad> <*> 
    "<|>"
        "$|" clb <overskrift> <<< 
    

My method just reads in one file, this is for testing, the corpus consists of
around 30000 files.


Answer: You can sue built-in `count` function that compiled in C :

    
    
    dict((i,test_list.count(i)) for i in set(test_list))
    

for better understanding you can see the following benchmarking :

    
    
    from timeit import timeit
    
    s1="""l=[1, 1, 1, 2, 3, 4, 1, 2, 5, 7, 2, 3]
    from collections import Counter
    Counter(l)"""
    s2="""l=[1, 1, 1, 2, 3, 4, 1, 2, 5, 7, 2, 3]
    dict((i,l.count(i)) for i in set(l))"""
    
    print 'using Counter : ' ,timeit(stmt=s1, number=1000000)
    print 'using built-in : ',timeit(stmt=s2, number=1000000)
    

result :

    
    
    using Counter :  8.78281712532
    using built-in :  2.91788387299
    

