Reducing duplicates in Python list of lists
Question: I am writing a program that reads in a number of files and then indexes the
terms in them. I am able to read in the files into a 2d array (list) in
python, but then I need to remove the duplicates in the first column and store
the index in a new column with the first appearance of the duplicated word.

For example:

    
    
    ['when', 1]
    ['yes', 1]
    ['', 1]
    ['greg', 1]
    ['17', 1]
    ['when',2]
    

the first column is the term, and the second is the DocID that it came from i
want to be able to change this to:

    
    
    ['when', 1, 2]
    ['yes', 1]
    ['', 1]
    ['greg', 1]
    ['17', 1]
    

removing the duplicate.

This is what I have so far:

    
    
    for j in range(0,len(index)):
            for r in range(1,len(index)):
                    if index[j][0] == index[r][0]:
                            index[j].append(index[r][1])
                            index.remove(index[r])
    

i keep getting an out of range error at

    
    
    if index[j][0] == index[r][0]:
    

and i think it is because I'm removing an object from the index so it is
becoming smaller. any ideas would be much appreciated (and yes, I know I
shouldn't modify the original, but this is just testing it on a small scale)


Answer: Wouldn't be more appropiate to build a
[`dict`](http://docs.python.org/py3k/library/stdtypes.html#mapping-types-
dict)/[`defaultdict`](http://docs.python.org/py3k/library/collections.html#collections.defaultdict)?

Something like:

    
    
    from collections import defaultdict
    
    ar = [['when', 1],
          ['yes', 1],
          ['', 1],
          ['greg', 1],
          ['17', 1],
          ['when',2]] 
    
    result = defaultdict(list)
    for lst in ar:
        result[lst[0]].append(lst[1])
    

Output:

    
    
    >>> for k,v in result.items():
    ...     print(repr(k),v)
    '' [1]
    'yes' [1]
    'greg' [1]
    'when' [1, 2]
    '17' [1]
    

