Python program and regex
Question: Would you be so kind to help me with some regex and Python code? I have
recently received a lot of help for a Python script to go through some
firewall logs. I am almost there, but still need a little help.

Here some log output from a firewall:

    
    
    Nov 11 00:00:09 firewall %ASA-3-710003: TCP access denied by ACL from 1.1.1.1/50624 to internet:2.2.2.2/80
    Nov  6 12:42:23 firewall %ASA-4-106023: Deny tcp src inside:3.3.3.3/42059 dst internet:4.4.4.4/389 by access-group "access_out" [0x0, 0x0]
    

The logs can both have IPv4 and IPv6 addresses. I need to extract the protocol
(tcp|udp|icmp), source_ip, the destination_ip, the destination_port and a
counter for each unique line. But as you can see from the two above lines, the
log output is different in the two lines, e.g. in one line the protocol is
written in uppercase, the other in lowercase, and also different text in
between the IP addresses. I am able to get the correct output from one of the
lines but not both.

    
    
    prot   source                    destination               port   hitcnt
    ------ ------------------        ------------------        ------ ------------------
    tcp    1111:2222:0:abcd::12      2222:3333:0:efab::101     389    180
    tcp    2222:3333:0:efab::50      1111:2222:0:abcd::12      389    29
    ump    1111:4444:0:1111::2       1111:2222:0:abcd::12      123    4
    tcp    1.1.1.1                   3.3.3.3                   23     4
    imp    2.2.2.2                   4.4.4.4                          4
    

The code from python script looks like this (thanks to holroy)

    
    
    import re       #for regular expressions - to match ip's
    import sys      #for parsing command line opts
    from collections import Counter
    
    DENY_PATTERN = re.compile(r'Deny\s(?P<protocol>.+?)\ssrc.*?:(?P<src>[0-9a-f\.]*)/?.*?\s.*?dst.*?:(?P<dst>[0-9a-f\.]*)((/(?P<dst_port>[0-9]*)\s)|\s)')
    
    LINE_FORMAT='{0:<6.6} {1:<25.25} {2:<25.25} {3:<6.6} {4}'
    
    def process_log_file(log file):
        """Reads through the log_file, and returns a counter based on Deny-lines."""
    
        # Process file line by line
        with open(logfile, 'r') as data :
            seen = Counter()
    
            # find all Deny line and append them in a list
            for line in data :
                # If line has 'Deny ' in it, then check it some more
                if 'Deny ' in line :
                    seen.update(DENY_PATTERN.findall(line)) 
        return seen
    
    def print_counter(counter):
        """Pretty print the result of the counter."""
    
        print(LINE_FORMAT.format('prot', 'source', 'destination',  'port', 'hitcnt'))
        print(LINE_FORMAT.format(*tuple(('------------------',) * 5)))
        for (protocol, src, dst, _, _, dst_port), count in counter.most_common():
            print(LINE_FORMAT.format(protocol, src, dst, dst_port, count))
    
    if __name__ == '__main__':
        # if file is specified on command line, parse, else ask for file
        if sys.argv[1:] : 
            print "File: %s" % (sys.argv[1])
            logfile = sys.argv[1]
        else:
            logfile = raw_input("Please enter a file to parse, e.g /var/log/secure: ")
    
        denial_counter = process_log_file(logfile)
        print_counter(denial_counter)
    

I've tried a lot of regex combinations (e.g.
<https://regex101.com/r/wC6gS7/2>, but am not able to get output right.

Also if I try with an or (|) between two regex, I am not able to call it
`(?P<protocol>)` in both the regex since it has already been used. If I use a
DENY_PATTERN and DENY_PATTERN2 with each regex and call it like this it also
doesn't work:

    
    
    for line in data :
        if 'Deny ' in line :
            seen.update(DENY_PATTERN.findall(line)) 
    
    for line in data :
        if 'ASA-3 ' in line :
            seen.update(DENY_PATTERN2.findall(line))
    

Thank you in advance.


Answer: ## Testing multiple patterns

To test for multiple complex pattern, you'll want to add those tests within
the one and only loop through your data file. That is you want the following
code:

    
    
        for line in data :
    
            # If line has 'Deny ' in it, then check it some more
            if 'Deny ' in line:
                seen.update(DENY_PATTERN.findall(line)) 
    
            # If line has 'ASA-3' in it, then check for the other pattern
            if 'ASA-3' in line:
                seen.update(DENIED_PATTERN.findall(line))
    

This way you read through the file one time, and only do the heavier pattern
matching if the basic test against `Deny` or `ASA-3` has already occured.

## Regexp comments

In your regexp you have a few extra groups, which are not needed, and you test
against `[A-Z].*?`, which is _one_ uppercase character, followed by a non-
greedy match of anything. Unless that is specific what you are after, it would
be better to use `[A-Z]*?` (or `[A-Za-z]*?`) to match for a sequence of
characters. I took the liberty of saving a few extra version, ending with the
following (see <https://regex101.com/r/wC6gS7/8>):

    
    
    DENIED_PATTERN = re.compile(r'''
        :\s+?                                   # Start with leading white space
        (?P<protocol>[A-Z]*?)                   # Capture "<protocol>" start
        \saccess\s+denied\s+by\s+ACL\s+from\s+  # Matches literal string
        (?P<src>[0-9a-f\.:]*)/.*?:              # Captures "<src>" plus extra text
        (?P<dst>[0-9a-f\.:]*)                   # Captures "<dst>"
        (?:/(?P<dst_port>[0-9]*?))?\s           # Optional captures "/<dst_port>" 
        ''', re.X)
    

Note also that using the `re.X` (or `re.VERBOSE`) can drastically increase the
understanding of your patterns. Using this flag you can include whitespace and
end of line comments using `#`.

Using `(?: ... )` is a nice construct to create a group, which can be made
optional by adding a `?`, like in `(?:/(?P<dst_port>[0-9]+))?`. This would
optionally capture stuff like "/80". Note however that since it is now made
optional, I added a `\s` to satisify the greadiness of earlier constructs, as
it otherwise could be left out.

Regarding alternate groups, they are often useful to construct using the non-capturing group construct, like in `(?: ... | ... | ...)`. But be aware that you if combined with named capture groups, the names needs to be unique.

## Changes to `print_counter()`

As the number of captured groups varies in the two patterns, there is a need
to modify the print out, as well. To differentiate between the two patterns
you could either depend upon the count of captured groups, or you could
introduce an identifier into your match groups. I.e. you could extend the
regexp's to include `Deny` and `ASA-3`.

I chose to use the number of capture groups. As now the number of capture
groups varies in the for loop this poses a slight issue with expanding the
tuple directly as you either have too many or too few elements to unpack. The
solution to this is to delay the expansion a little. This ends up with the
following code:

    
    
    def print_counter(counter):
        """Pretty print the result of the counter."""
    
        print(LINE_FORMAT.format('prot', 'source', 'destination',  'port', 'hitcnt'))
        print(LINE_FORMAT.format(*tuple(('------------------',) * 5)))
        for re_match, count in counter.most_common():
    
            re_match_length = len(re_match)
    
            # Pick elements based on match from DENY_PATTERN        
            if re_match_length == 8:
                protocol, src, src_port, dst, dst_port, icmp_spec, icmp_type, icmp_code =  re_match
                if icmp_code or icmp_type:
                    dst_port = '{}, {}'.format(icmp_code, icmp_type)
    
            # Pick elements based on match from DENIED_PATTERN
            if re_match_length == 4:
                protocol, src, dst, dst_port = re_match
    
    
            print(LINE_FORMAT.format(protocol.lower(), src, dst, dst_port, count))
    

To beautify the output a bit, I added a `protocol.lower()` so that they are
always presented in lowercase. This could also have been done within the
corresponding `if` statement, along side with other needed tweaking before
outputting the line.

