Python - Formatting specific data within text files
Question: I have a number of log files I need extract and format data. some of these log
files are very big with over 10,000 lines.

can anyone suggest a code sample to help me read a text file, remove unwanted
lines, then edit the remaining lines into a particular format. i haven't been
able to find any previous threads that have what I'm after.

An example of the data I need to edit is below:

    
    
    136: add student 50000000 35011 / Y01T :Unknown id in field 3 - ignoring line
    
    137: add student 50000000 5031 / Y01S :Unknown id in field 3 - ignoring line
    
    138: add student 50000000 881 / Y01S :Unknown course idnumber in field 4 - ignoring line
    
    139: add student 50000000 5732 / Y01S :Unknown id in field 3 - ignoring line
    
    134: add student 50000000 W250 / Y02S :OK
    
    135: add student 50000000 35033 / Y01T :OK
    

I need to search the file and delete any line that is suffixed with :OK. Then,
i need to edit the remaining lines in to a CSV format such as:

    
    
    add,student,50000000,1234 / abcd
    

Any tips, code snippets, etc would be very helpful and i'll be most grateful.
i'd try it first before asking, but I have little time to self teach python
file access/string formatting. So please allow me to apologise in advance for
not attempting it prior to asking


Answer: This could be a solution:

    
    
    import sys
    
    if len(sys.argv) != 2:
        print 'Add an input file as parameter'
        sys.exit(1)
    
    print 'opening file: %s' % sys.argv[1]
    
    with open(sys.argv[1]) as input, open('output', 'w+') as output:
        for line in input:
            if line is not None:
                if line == '\n':
                    pass
                elif 'OK' in line:
                    pass
                else:
                    new_line = line.split(' ', 7)
                    output.write('%s,%s,%s,%s / %s\n' % (new_line[1], new_line[2], new_line[3], new_line[4], new_line[6]))
                    # just for checking purposes let's print the lines
                    print '%s,%s,%s,%s / %s' % (new_line[1], new_line[2], new_line[3], new_line[4], new_line[6])
    

Watch out for the output file name!

