Using Comprehensions to Handle a Large Dataset in Python 2.7 I have a Python 2.7.6 script parsing large files (~60MB to ~2GB) containing lines of the following format:


  componentA componentB &lt; floating point value >


My goal is to sum the floating point values for duplicate pairs of components and reflected pairs, reducing each pair to a single entry in the output file. For example, if I have the following three lines in the input file:


  componentA componentB 1.5
  componentB componentA 2.0
  componentA componentB 0.5


The output file should combine them into one entry:


  componentA componentB 4.0


Other notes:  


The output also needs to be sorted by the lesser of each component pair.
I am not very concerned about the memory footprint of the routine because it will be run on a server with substantial amounts of RAM available. I am optimizing for speed and code maintainability.


The code I currently have accomplishes this by storing component names in nested dictionaries. The top-level dictionary maps component names to a second dictionary. The second-level dictionary maps the second half of each component pair to the floating-point value corresponding to the pair.

Example:

base_dict = {componentA: {componentB: 1.2,
                          componentC: 0.7}}


Output from this example would look like this:


  componentA componentB 1.2
  componentA componentC 0.7


Here is my code:

base_dict = {}

with open(input_file_name, 'r') as fi:
    for line in fi:
        component1, component2, value = line.split()
        (min, max) = (component1, component2) if (component1 &lt; component2) else (component2, component1)

        if not min in base_dict:
            base_dict[min] = {max:float(value)}

        elif not max in base_dict[min]:
            base_dict[min][max] = float(value)

        else:
            base_dict[min][max] += float(value)

out_list = []
for component1, nested_dict in base_dict.iteritems():
    for component2, value in nested_dict.iteritems():
        out_list.append("{0}\t{1}\t{2}\n".format(component1, component2, value))

with open(output_file_name), 'w') as fo:
    fo.writelines(sorted(out_list))


In addition to a general code review for this snippet, there are two sections that I would like to convert to comprehensions if possible:


Constructing the nested dictionaries - I may be dealing with close to 10 million lines in the input file, and I would like to utilize implicit loops to do this.
Constructing the output list for sorting


If anyone has any completely different approaches to the problem of summing duplicate and reflected pairs, I'm all ears.
As requested:

Sort the list according to the component names. To achieve a correct order, either swap component names when necessary, or supply the custom comparator (which will account for (componentA, componentB) being the same thing as (componentB, componentA)).

Once the list is sorted, the entries to be tallied are adjacent, and an actual tallies could be computed in a single pass over the list.