Python list reduction
Question: I'm working on a continuously learning focused web crawler to find news
articles related to specific crisis and tragedy events that happen around the
world. I am currently working on making the data model as lean and efficient
as possible considering its constant growth as the crawl continues.

**I am storing the data model in a list** (to do TFIDF comparisons to the page
being crawled) **and I want to reduce the size of the list but not lose the
relative counts of each item in the list**.

This is a sample model from 2 crawled webpages:

    
    
    [[u'remark', u'special', u'agent', u'richard', u'deslauri', u'press', u'investig', u'crime', u'terror', u'crime', u'inform', u'servic', u'inform', u'laboratori', u'servic', u'want', u'want', u'want', u'terror', u'crime', u'want', u'news', u'news', u'press', u'news', u'servic', u'crime', u'inform', u'servic', u'laboratori', u'servic', u'servic', u'crime', u'crime', u'crime', u'terror', u'boston', u'press', u'remark', u'special', u'agent', u'richard', u'deslauri', u'press', u'investig', u'remark', u'special', u'agent', u'richard', u'deslauri', u'press', u'investig', u'boston', u'special', u'agent', u'remark', u'richard', u'deslauri', u'boston', u'investig', u'time', u'time', u'investig', u'boston', u'terror', u'law', u'enforc', u'boston', u'polic', u'polic', u'alreadi', u'alreadi', u'law', u'enforc', u'around', u'evid', u'boston', u'polic', u'evid', u'laboratori', u'evid', u'laboratori', u'may', u'alreadi', u'laboratori', u'investig', u'boston', u'polic', u'law', u'enforc', u'investig', u'around', u'alreadi', u'around', u'investig', u'law', u'enforc', u'evid', u'may', u'time', u'may', u'may', u'investig', u'may', u'around', u'time', u'investig', u'investig', u'boston', u'boston', u'news', u'press', u'boston', u'want', u'boston', u'want', u'news', u'servic', u'inform'], [u'2011', u'request', u'inform', u'tamerlan', u'tsarnaev', u'foreign', u'govern', u'crime', u'crime', u'inform', u'servic', u'inform', u'servic', u'nation', u'want', u'ten', u'want', u'want', u'crime', u'want', u'news', u'news', u'press', u'releas', u'news', u'stori', u'servic', u'crime', u'inform', u'servic', u'servic', u'servic', u'crime', u'crime', u'crime', u'news', u'press', u'press', u'releas', u'2011', u'request', u'inform', u'tamerlan', u'tsarnaev', u'foreign', u'govern', u'2011', u'request', u'inform', u'tamerlan', u'tsarnaev', u'foreign', u'govern', u'2013', u'nation', u'press', u'tamerlan', u'tsarnaev', u'dzhokhar', u'tsarnaev', u'tamerlan', u'tsarnaev', u'dzhokhar', u'tsarnaev', u'dzhokhar', u'tsarnaev', u'tamerlan', u'tsarnaev', u'dzhokhar', u'tsarnaev', u'2011', u'foreign', u'govern', u'inform', u'tamerlan', u'tsarnaev', u'inform', u'2011', u'govern', u'inform', u'tamerlan', u'tsarnaev', u'foreign', u'foreign', u'govern', u'2011', u'inform', u'foreign', u'govern', u'nation', u'press', u'releas', u'crime', u'releas', u'ten', u'news', u'stori', u'2013', u'ten', u'news', u'stori', u'2013', u'ten', u'news', u'stori', u'2013', u'2011', u'request', u'inform', u'tamerlan', u'tsarnaev', u'foreign', u'govern', u'nation', u'press', u'releas', u'want', u'news', u'servic', u'inform', u'govern']]
    

I want to maintain the list of words and not embed the count into the list
itself. I would like the list to go from:

[Boston, Boston,Boston,Bombings,Bombings,Tsarnaev,Tsarnaev,Time] to
[Boston,Boston,Bombings,Tsarnaev]

_Basically,_ if I had a list [a,a,a,b,b,c], I would want to reduce it to
[a,a,b]

**EDIT:** Sorry for not being clear, but I will try again. I do **not** want a
set. The number of occurrences is very important because it is a weighted list
so "Boston" should appear more times than "time" or another similar term. What
I am trying to accomplish is to _minimize_ the data model while removing the
insignificant terms from the model. So in the above example, I purposely left
out C because it adds to much "fat" to the model. I want to keep the
relativity in that A appeared 1 more time than B and 2 more times than C but
since C only appeared once in the original model, it is being removed from the
_lean_ model.


Answer: 
    from collections import defaultdict
    d = defaultdict(int)
    for w in words[0]:
        d[w] += 1
    mmin = min(d[p] for p in d)
    

then you can subtract this mmin from each word and create a new list. But
perhaps the dict is compact enough. To preserve the order, you can use the
info from the dict and devise some smart way to filter your initial word list.

For example, for the word list `[a,a,a,b,b,c]`, the dictionary will contain
`{a:3, b:2, c:1}` and the `mmin=1`. You can use this information to have a
slimmer dictionary by subtracting 1 from all items to get `{a:2, b:1}` and
since `c` is `0` it is removed.

Complete code:

    
    
    from collections import defaultdict
    d = defaultdict(int)
    words = ['a','a','a','b','b','c']
    for w in words:
        d[w] += 1
    mmin = min(d[p] for p in d)
    slim=[]
    for w in words:
        if d[w] > mmin:
            slim.append(w)
            d[w] -= 1
    print slim
    

