How do I count exoplanets per system in a file with over 10,000 lines in Python?
Question: I am working with astronomical data and I need help summarizing it.

My data contains ~10,000 lines, where each line represents a system.

The input file is tab delimited like this: exo sys_planet_count

    
    
    0   1   
    0   0   
    3   4   
    0   1   
    2   5   
    0   0   
    

Note that exo planet count is usually 0 or 1, but NOT Always.

**Each line represents a system** and there are two columns, one for the
exo_planets found in that system and one for the total number of planets
found.

I need the data summarized like this by increasing sys_planet_count:

system_planet_count exo system_hits system_misses

    
    
    5 3500 3000 1000
    6 4500 4000 1500
    

The **number of exo planets must be greater or equal than system_hits** ,
because there could be only one exo planet per system or several, it depends.

system_planet_count is how the the table is organized.

For each line (system) that matches a particular system_planet_count, it adds
the number of exos found. If there were exos found, it adds +1 to the
system_hits category because that line found exo planets, a hit. If there were
NO exos found in that line, it adds one to the system_misses category because
there were no lines in a planet.

NOTE that system_misses and system_hits category is specific to that
system_planet count, i.e. 3000 and 1000 for system_planet_count of 5 but 4000
and 1500 for a system_planet_count of 6

The problem is that the data is NOT ordered in ascending order of
sys_planet_counts.

To summarize the data, I came up with the following code. What should I do to
summarize the data in a quick manner that doesn't take 10 or 15 minutes?

I was thinking about using a dictionary, since each system_planet_count could
act as key

    
    
    while open('data.txt','r') as input:
        for line in input:
            system_planet_count = 0
            exo_count = 0
            system_hits = 0
            system_misses = 0
    
            foo
        output.write(str(system_planet_count) + '\t' + str(exo_count) + '\t' + str(system_hits) + '\t' + str(system_misses) + '\')
    

Input example:

exo sys_planet_count

    
    
     2 1
     0 1
     1 1
     0 5
     1 5
     0 5
     0 5
     2 5
     0 5
     0 4
    

Output:

system_planet_count exo system_hits system_misses

    
    
     1 3 2 1
     4 0 0 1
     5 3 2 4
    


Answer: This should do the summary you want:

    
    
    from collections import defaultdict
    
    def summarize(file_name):
        exo, hit, miss = 0, 1, 2  # indexes of according counts
        d = defaultdict(lambda: [0, 0, 0])  # keep all counts for each type of system
        with open(file_name, 'r') as input:
            for line in input:
                exos, planets = map(int, line.strip().split())  # split, cast to int
                if exos:
                    d[planets][exo] += exos
                    d[planets][hit] += 1
                else:
                    d[planets][miss] += 1
    
        for key in sorted(d.keys()):
            print('{} {} {} {}'.format(key, d[key][exo], d[key][hit], d[key][miss]))
    
    summarize('data.txt')
    

