How to efficiently process a large file with a grouping variable in Python
Question: I've got a dataset that looks something like the following:

    
    
    ID   Group
    1001 2
    1006 2
    1008 1
    1027 2
    1013 1
    1014 4
    

So basically, a long list of unsorted IDs with a grouping variable as well.

At the moment, I want to take subsets of this list based on the generation of
a random number (imagine they're being drafted, or won the lottery, etc.).
Right now, this is the code I'm using to just process it row-by-row, by ID.

    
    
    reader = csv.reader(open(inputname), delimiter=' ')
    out1 = open(output1name,'wb')
    out2 = open(output2name,'wb')
    
    for row in reader:
        assignment = gcd(1,p,marg_rate,rho)
        if assignment[0,0]==1:
            out1.write(row[0])
            out1.write("\n")
        if assignment[0,1]==1:
            out2.write(row[0])
            out2.write("\n")
    

Basically, i the gcd() function goes one way, you get written to one file,
another way to a second, and then some get tossed out. The issue is I'd now
like to do this by _Group_ rather than ID - basically, I'd like to assign
values to the first time a member of the group appears, and then apply it to
all members of that group (so for example, if 1001 goes to File 2, so does
1006 and 1027).

Is there an efficient way to do this in Python? The file's large enough that
I'm a little wary of my first thought, which was to do the assignments in a
dictionary or list and then have the program look it up for each line.


Answer: I used
[`random.randint`](https://docs.python.org/2/library/random.html#random.randint)
to generate a random number, but this can be easily replaced.

The idea is to use a
[`defaultdict`](https://docs.python.org/2/library/collections.html#collections.defaultdict)
to have **single** score (`dict` keys are unique) for a group from the moment
it's created:

    
    
    import csv
    import random
    from collections import defaultdict
    
    reader = csv.DictReader(open(inputname), delimiter=' ')
    out1 = open(output1name,'wb')
    out2 = open(output2name,'wb')
    
    # create a dictionary with a random default integer value [0, 1] for
    # keys that are accessed for the first time
    group_scores = defaultdict(lambda: random.randint(0,1))
    
    for row in reader:
        # set a score for current row according to it's group
        # if none found - defaultdict will call it's lambda for new keys
        # and create a score for this row and all who follow
        score = group_scores[row['Group']]
        if score==0:
            out1.write(row['ID'])
            out1.write("\n")
        if score==1:
            out2.write(row['ID'])
            out2.write("\n")
    
    out1.close()
    out2.close()
    

I've also used
[`DictReader`](https://docs.python.org/2/library/csv.html#csv.DictReader)
which I find nicer for `csv` files with headers.

**Tip:** you may want to use the [`with` context
manager](http://effbot.org/zone/python-with-statement.htm) to open files.

Example output:

    
    
    reut@sharabani:~/python/ran$ cat out1.txt 
    1001
    1006
    1008
    1027
    1013
    reut@sharabani:~/python/ran$ cat out2.txt 
    1014
    

