Removing duplicate records in a file
<blockquote>
  <p><strong>Possible Duplicate:</strong><br>
  <a href="http://stackoverflow.com/questions/1215208/how-might-i-remove-duplicate-lines-from-a-file">How might I remove duplicate lines from a file?</a>  </p>
</blockquote>



<p>I have a file with duplicated records that I want to remove. This is what I have tried  </p>

<pre><code>import sys  

for line in sys.stdin:  
    line = line.rstrip()  
    line = line.split()  
    idlist = []   
    if idlist == []:  
        idlist = line[1]  
    else:  
    idlist.append(line[1])  
    print line[0], idlist  
</code></pre>

<h2>#did not work</h2>

<h1>and this</h1>

<pre><code>for line in sys.stdin:  
    line = line.rstrip()  
    line = line.split()  
    lines_seen = set()  
    dup = line[1]  
    if dup not in lines_seen:  
        lines_seen = dup  
    else:  
        lines_seen.append(dup)  
    print line[0], lines_seen  

sys.stdin.close()
</code></pre>

<h2>#did not work either!</h2>

<p>This is what the input looks like</p>

<p>BLE 1234<br>
BLE 1223<br>
LLE 3456<br>
ELE 1223<br>
BLE 4444<br>
ELE 5555<br>
BLE 4444  </p>

<h1>And this is what I want the output to look like</h1>

<p>BLE 1234<br>
BLE 1223<br>
LLE 3456<br>
BLE 4444<br>
ELE 5555  </p>

<p>Thanks!
edg</p>
