filter a Linux log file using Python
Question: I want to filter a log file to keep all lines matching a certain pattern. I
want to do this with Python.

Here's my first attempt:

    
    
    #!/usr/bin/env python
    
    from sys import argv 
    
    script, filename = argv
    with open(filename) as f:
        for line in f:
            try:
                e = line.index("some_term_I_want_to_match")
            except: 
                pass
            else:
                print(line)
    

How can I improve this to:

  * save the result to a new file of similar name (i.e., a different extension)
  * use regex to make it more flexible/powerful.

(I'm just learning Python. This question is as much about learning Python as
it is about accomplishing this particular result.)

**OK, here's what I came up with so far...** But how do you do the equivalent
of prepending an `r` as in the following line

    
    
    re.compile(r"\s*")
    

where the string is **not** a string literal, as in the next line?

    
    
    re.compile(a_string_variable)
    

Other than that, I think this updated version does the job:

    
    
    #!/usr/bin/env python
    
    from sys import argv 
    import re
    import os
    import argparse #requires Python 2.7 or above
    
    parser = argparse.ArgumentParser(description='filters a text file on the search phrase')
    parser.add_argument('-s','--search', help='search phrase or keyword to match',required=True)
    parser.add_argument('-f','--filename', help='input file name',required=True)
    parser.add_argument('-v','--verbose', help='display output to the screen too', required=False, action="store_true")
    args = parser.parse_args()
    
    keyword = args.search
    original_file = args.filename
    verbose = args.verbose
    
    base_file, ext = os.path.splitext(original_file)
    new_file = base_file + ".filtered" + ext
    
    regex_c = re.compile(keyword)
    
    with open(original_file) as fi:
        with open(new_file, 'w') as fo:
            for line in fi:
                result = regex_c.search(line)
                if(result):
                    fo.write(line)
                    if(verbose):
                        print(line)
    

Can this be easily improved?


Answer: Well, you know, you have answered most of your questions yourself already :)

For regular expression matching use [`re`
module](http://docs.python.org/3/library/re.html) (the doc has pretty
explanatory examples).

You already have made use `open()` function for opening a file. Use the same
function for open files for writing, just provide a corresponding `mode`
parameter ("w" or "a" combined with "+" if you need, see `help(open)` in the
Python interactive shell). That's it.

