Python, dictionaries, and chi-square contingency table
Question: This is a problem I've been racking my brains on for a long time, so any help
would be great. I have a file which contains several lines in the following
format (word, time that the word occurred in, and frequency of documents
containing the given word within the given instance in time). Below is an
example of what the inputfile looks like.

    
    
    #inputfile
    <word, time, frequency>
    apple, 1, 3
    banana, 1, 2
    apple, 2, 1
    banana, 2, 4
    orange, 3, 1
    

I have Python class below that I used to create 2-D dictionaries to store the
above file using as the key, and frequency as the value:

    
    
    class Ddict(dict):
        '''
        2D dictionary class
        '''
        def __init__(self, default=None):
                self.default = default
    
        def __getitem__(self, key):
                if not self.has_key(key):
                    self[key] = self.default()
                return dict.__getitem__(self, key)
    
    
    wordtime=Ddict(dict) # Store each inputfile entry with a <word,time> key
    timeword=Ddict(dict) # Store each inputfile entry with a <time,word> key
    
    # Loop over every line of the inputfile
    for line in open('inputfile'):
        word,time,count=line.split(',')
    
        # If <word,time> already a key, increment count
        try:
            wordtime[word][time]+=count
        # Otherwise, create the key
        except KeyError:
            wordtime[word][time]=count
    
        # If <time,word> already a key, increment count     
        try:
            timeword[time][word]+=count
        # Otherwise, create the key
        except KeyError:
            timeword[time][word]=count
    

The question that I have pertains to calculating certain things while
iterating over the entries in this 2D dictionary. For each word 'w' at each
time 't', calculate:

  1. The number of documents **with** word 'w' **within** time 't'. (a)
  2. The number of documents **without** word 'w' **within** time 't'. (b)
  3. The number of documents **with** word 'w' **outside** time 't'. (c)
  4. The number of documents **without** word 'w' **outside** time 't'. (d)

Each of the items above represents one of the cells of a chi-square
contingency table for each word and time. Can all of these be calculated
within a single loop or do they need to be done one at a time?

Ideally, I would like the output to be what's below, where a,b,c,d are all the
items calculated above:

    
    
    print "%s, %s, %s, %s" %(a,b,c,d)
    

In the case of the input file above, the result of trying to find the
contingency table for the word 'apple' at time '1' would be `(3,2,1,6)`. I'll
explain how each cell is calculated:

  * '3' documents contain 'apple' within time '1'.
  * There are '2' documents within time '1' that don't contain 'apple'.
  * There is '1' document containing 'apple' outside time '1'.
  * There are 6 documents outside time '1' that don't contain the word 'apple' (1+4+1).


Answer: Your 4 numbers for apple/1 add up to 12, more than the total number of
observations (11)! There are only 5 documents outside time '1' that don't
contain the word 'apple'.

You need to partition the observations into 4 disjoint subsets:  
a: apple and 1 => 3  
b: not-apple and 1 => 2  
c: apple and not-1 => 1  
d: not-apple and not-1 => 5

Here is some code that shows one way of doing it:

    
    
    from collections import defaultdict
    
    class Crosstab(object):
    
        def __init__(self):
            self.count = defaultdict(lambda: defaultdict(int))
            self.row_tot = defaultdict(int)
            self.col_tot = defaultdict(int)
            self.grand_tot = 0
    
        def add(self, r, c, n):
            self.count[r][c] += n
            self.row_tot[r] += n
            self.col_tot[c] += n
            self.grand_tot += n
    
    def load_data(line_iterator, conv_funcs):
        ct = Crosstab()
        for line in line_iterator:
            r, c, n = [func(s) for func, s in zip(conv_funcs, line.split(','))]
            ct.add(r, c, n)
        return ct
    
    def display_all_2x2_tables(crosstab):
        for rx in crosstab.row_tot:
            for cx in crosstab.col_tot:
                a = crosstab.count[rx][cx]
                b = crosstab.col_tot[cx] - a
                c = crosstab.row_tot[rx] - a
                d = crosstab.grand_tot - a - b - c
                assert all(x >= 0 for x in (a, b, c, d))
                print ",".join(str(x) for x in (rx, cx, a, b, c, d))
    
    if __name__ == "__main__":
    
        # inputfile
        # <word, time, frequency>
        lines = """\
        apple, 1, 3
        banana, 1, 2
        apple, 2, 1
        banana, 2, 4
        orange, 3, 1""".splitlines()
    
        ct = load_data(lines, (str.strip, int, int))
        display_all_2x2_tables(ct)
    

and here is the output:

    
    
    orange,1,0,5,1,5
    orange,2,0,5,1,5
    orange,3,1,0,0,10
    apple,1,3,2,1,5
    apple,2,1,4,3,3
    apple,3,0,1,4,6
    banana,1,2,3,4,2
    banana,2,4,1,2,4
    banana,3,0,1,6,4
    

