How to replace the words that appear once in a sentence in python
Question: I want to replace the words that appear once in a sentence with `'<unk>'`.
Like for a sentence: `hello hello world my world`, I want the output to be
`hello hello world <unk> world`, how to do that?

Right now I'm doing like this:

    
    
     wordlist1 = trainfiles.split(None)
            wordlist2 = []
            for word1 in wordlist1:
                lastchar = word1[-1:]
                if lastchar in [",",".","!","?",";"]:
                    word2 = word1.rstrip(lastchar)
                else:
                    word2 = word1
                wordlist2.append(word2)
            freq = {}
            for word2 in wordlist2:
                freq[word2] = freq.get(word2,0)+1
            keylist = freq.keys()
            keylist.sort()
    
        for key2 in keylist:
            if freq[key2] == 1:
                print "%-10s %d" % ('<unk>', freq[key2])
            else:
                print "%-10s %d" % (key2, freq[key2])
    

Which gives me an output like:

    
    
    hello   2
    <unk>   1
    world   2
    

But, I need the output like :

    
    
    hello hello world <unk> world
    

How to do that?


Answer: Use `collections.Counter` to count the frequency the words in your sentence

    
    
    from collections import Counter
    s = 'hello hello world my world'
    counts = Counter(s.split())
    

Then use a generator expression to replace any word that has a count of 1, and
join the result a space character.

    
    
    replaced = ' '.join(i if counts[i] > 1 else '<unk>' for i in s.split())
    

Result

    
    
    'hello hello world <unk> world'
    

