How to delete repeating lines starting with specific word in python
Question: I have an inputfile of the form

All tests start with the word "Test" and all errors start with the word
"error"

    
    
    Test1
    Error1
    Error1 
    Error2
    Test1
    Error3
    
    Test2
    Error1
    Error4 
    
    Test2
    Error5
    Error1
    
    Test3
    Error1
    
    I want it in the format:
    Test1
    Error1
    Error1
    Error2
    Error3 // Removed test1 
    
    Test2
    Error1
    Error4
    Error5
    Error1
    
    Test3
    Error1 
    

Basically while going through the file, it should delete repeated testnames
and write it in the same order to an output file. Following is my code

    
    
    import os
    import sys
    import optparse
    
    def delete_duplicate(inputfile,outputfile): 
        output = open(outputfile, "w")
        from collections import OrderedDict
        input = open(inputfile, "r")
        lines = (line.strip() for line in input)
        unique_lines = OrderedDict.fromkeys((line for line in lines if line))
        for unique_line in unique_lines:
            output.write(unique_line)
            output.write("\n") 
    
    My code removes duplicate lines and gives result as below: 
    Test1
    Error1
    Error2
    Error3 
    
    Test2
    Error4
    Error5
    
    Test3 
    

It is working fine with testnames but not with errors. Can anybody help?


Answer: All you need is to preserve the lines that starts with `Test` in a set and
check if you have it already just don't write it in output file :

    
    
    def delete_duplicate(inputfile,outputfile,seen={}):
        with open(outputfile, "w") as output,open(inputfile, "r") as input: 
          for line in input:
            if line not in seen:
                 output.write(line+'\n')
            if line.startswith('Test'):
                seen.add(line)
    

The advantage of `set` is that its order is O(1) for check the membership and
adding items.

