title: Parsing an IP adress in a logfile in python 3
body: <p>I'm quite new to python and I'm trying to (using python 3) go through a number of large custom log files to extract parameters from certain <code>GET</code> requests an try to collect some statistics from them. I got kind of far but I got stuck with two questions that me and my colleague could not figure out why they were giving us such headache. I will post the two questions separately to not confuse you.</p>
  
  <p>My log file looks like this:</p>
  
  <pre><code>80 172.23.131.149 "2018-07-05 13:08:25 860" "POST /bios/servlet/bios.servlets.sso.WaffleLoginServlet HTTP/1.1" 401 5 891 891 "-" "Java/1.8.0_171"
  8080 172.23.131.251 "2018-07-05 13:08:26 594" "HEAD /bios/servlet/bios.servlets.web.Ping?level=3 HTTP/1.0" 200 - 1953 1953 "-" "-"
  8080 172.23.131.252 "2018-07-05 13:08:26 594" "HEAD /bios/servlet/bios.servlets.web.Ping?level=3 HTTP/1.0" 200 - 953 953 "-" "-"
  80 172.23.131.149 "2018-07-05 13:08:28 188" "GET /bios/wms/app/baggis/web/WMS_STHLM_STOCKHOLMSKARTA_HYBRID_INTERN?TILED=TRUE&amp;SERVICE=WMS&amp;VERSION=1.1.1&amp;REQUEST=GetMap&amp;FORMAT=image%2Fpng&amp;TRANSPARENT=false&amp;LAYERS=p_1002095&amp;SRS=EPSG%3A3011&amp;STYLES=&amp;r=n2q&amp;WIDTH=256&amp;HEIGHT=256&amp;BBOX=156240.234375%2C6576777.34375%2C156269.53125%2C6576806.640625 HTTP/1.1" 200 133210 3547 3516 "http://tkkarta3.stockholm.se/astolmap/v3/kopplet/tkkarta.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
  80 172.23.131.149 "2018-07-05 13:08:28 188" "GET /bios/wms/app/baggis/web/WMS_STHLM_STOCKHOLMSKARTA_HYBRID_INTERN?TILED=TRUE&amp;SERVICE=WMS&amp;VERSION=1.1.1&amp;REQUEST=GetMap&amp;FORMAT=image%2Fpng&amp;TRANSPARENT=false&amp;LAYERS=p_1002095&amp;SRS=EPSG%3A3011&amp;STYLES=&amp;r=n2q&amp;WIDTH=256&amp;HEIGHT=256&amp;BBOX=156240.234375%2C6576748.046875%2C156269.53125%2C6576777.34375 HTTP/1.1" 200 108066 3547 3532 "http://tkkarta3.stockholm.se/astolmap/v3/kopplet/tkkarta.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
  80 172.23.131.149 "2018-07-05 13:08:28 188" "POST /bios/servlet/bios.servlets.GetGeometryComponents HTTP/1.1" 401 4 2484 2484 "-" "Java/1.8.0_171"
  80 172.23.131.149 "2018-07-05 13:08:28 204" "GET /bios/wms/app/baggis/web/WMS_STHLM_STOCKHOLMSKARTA_HYBRID_INTERN?TILED=TRUE&amp;SERVICE=WMS&amp;VERSION=1.1.1&amp;REQUEST=GetMap&amp;FORMAT=image%2Fpng&amp;TRANSPARENT=false&amp;LAYERS=p_1002095&amp;SRS=EPSG%3A3011&amp;STYLES=&amp;r=n2q&amp;WIDTH=256&amp;HEIGHT=256&amp;BBOX=156210.9375%2C6576806.640625%2C156240.234375%2C6576835.9375 HTTP/1.1" 200 123953 3563 3547 "http://tkkarta3.stockholm.se/astolmap/v3/kopplet/tkkarta.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
  80 172.23.131.149 "2018-07-05 13:08:28 204" "GET /bios/wms/app/baggis/web/WMS_STHLM_STOCKHOLMSKARTA_HYBRID_INTERN?TILED=TRUE&amp;SERVICE=WMS&amp;VERSION=1.1.1&amp;REQUEST=GetMap&amp;FORMAT=image%2Fpng&amp;TRANSPARENT=false&amp;LAYERS=p_1002095&amp;SRS=EPSG%3A3011&amp;STYLES=&amp;r=n2q&amp;WIDTH=256&amp;HEIGHT=256&amp;BBOX=156210.9375%2C6576777.34375%2C156240.234375%2C6576806.640625 HTTP/1.1" 200 147132 3563 3547 "http://tkkarta3.stockholm.se/astolmap/v3/kopplet/tkkarta.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
  80 172.23.131.149 "2018-07-05 13:08:28 204" "GET /bios/wms/app/baggis/web/WMS_STHLM_STOCKHOLMSKARTA_HYBRID_INTERN?TILED=TRUE&amp;SERVICE=WMS&amp;VERSION=1.1.1&amp;REQUEST=GetMap&amp;FORMAT=image%2Fpng&amp;TRANSPARENT=false&amp;LAYERS=p_1002095&amp;SRS=EPSG%3A3011&amp;STYLES=&amp;r=n2q&amp;WIDTH=256&amp;HEIGHT=256&amp;BBOX=156269.53125%2C6576777.34375%2C156298.828125%2C6576806.640625 HTTP/1.1" 200 145701 3563 3547 "http://tkkarta3.stockholm.se/astolmap/v3/kopplet/tkkarta.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
  80 172.23.137.120 "2018-07-06 10:04:32 856" "GET /bios/wms/app/baggis/web/WMS_STHLM_STOCKHOLMSKARTA_GRA?FORMAT=image%2Fpng&amp;TILED=TRUE&amp;SERVICE=WMS&amp;VERSION=1.1.1&amp;REQUEST=GetMap&amp;STYLES=&amp;SRS=EPSG%3A5850&amp;BBOX=150000,6580000,151875,6581875&amp;WIDTH=256&amp;HEIGHT=256 HTTP/1.1" 200 58443 0 0 "https://iservice.stockholm.se/open/TyckTill/Pages/TyckTill.aspx?systemId=synpunktsportalen" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"
  80 172.23.137.120 "2018-07-06 10:04:25 400" "GET /bios/dpwebmap/cust_sth/slk/tycktill/app.htmlclient.gwt.DPWebApp.nocache.js HTTP/1.1" 200 3924 0 0 "https://iservice.stockholm.se/open/TyckTill/Pages/TyckTill.aspx?systemId=synpunktsportalen" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"
  </code></pre>
  
  <p>What I am trying to do is to extract the IP address for all lines with a string <code>REQUEST=GetMap</code>. The regex I am using is:</p>
  
  <pre><code>rexp_ip = r"(?P&lt;ip&gt;(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))"
  </code></pre>
  
  <p>I'm using the key <code>ip</code> to count the number of occurrences of all the IP-addresses in the log files further down in my code. </p>
  
  <p>I've been staring at the regex and tried to change it a bit back and forth but it still does not work. 
  <a href="https://regex101.com/r/yS3lY3/53" rel="nofollow noreferrer">But it works in Regex101 which is very confusing</a></p>
  
  <p>The full code for the task is:</p>
  
  <pre><code>#!/usr/bin/env python3
  
  import os
  import re
  from collections import Counter
  
  # regular expression
  
  #rexp = [r'(?P&lt;timestamp&gt;\d{1,2}-\w{3}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}) client (?P&lt;client&gt;(?:\d{1,3}\.){3}\d{1,3}).+query: (?P&lt;domain&gt;.+) IN (?P&lt;qtype&gt;[A-Z]+) \+.+\({2}(?P&lt;server&gt;(?:\d{1,3}\.){3}\d{1,3})\){2}'
  #rexp = r"(^.+layers=(?P&lt;domain&gt;.*?)&amp;)" # sök efter LAYERS= eller layers=
  
  rexp_layer = r"(^.+layers=(?P&lt;domain&gt;.*?)[&amp;\s])"                # search for the name of the requested layer (between the string 'LAYERS=' or 'layers=' and a ampersand '&amp;' or blankspace ' ') in each line and give it the key 'domain'
  rexp_port = r"(?P&lt;port&gt;\d{2,4} )"                               # search for the 2 or 4 digit value in the beginning of each line
  rexp_ip = r"(?P&lt;ip&gt;(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))"
  
  rexp_date = r"(?P&lt;datum&gt;\d{4}\-\d{2}\-\d{2})"                   # search for the date in format XXXX-XX-XX and give it the key 'datum'
  rexp_time = r"(?P&lt;tid&gt;\d{2}\:\d{2}\:\d{2})"                     # search for the timestamp XX:XX:XX and give it the key 'tid'
  
  rexp_name = r"(^.+/bios/wms/app/(?P&lt;name&gt;.+?)\?)"     # search for the name of the called WMS-service (are between the string '/bios/wms/app/' and a '?') and give it the key 'name'to the FIRST occurrence of "?", "+?" makes the "+" non-greedy
  
  rexp_coordsys = r"(^.+&amp;\wRS=(?P&lt;koordsys&gt;.*?)[&amp;\s])"             # search for the coordinate system between the string '&amp;SRS=' or '&amp;CRS=' and a ampersand '&amp;' and give it the key 'koordsys'
  
  rexp_width = r"(^.+WIDTH=(?P&lt;width&gt;.*?)&amp;)"                      # search for the width of the requested picture (are between the string 'WIDTH=' and a ampersand '&amp;') and give it the key 'width'
  rexp_height = r"(^.+HEIGHT=(?P&lt;height&gt;.*?)[&amp;\s])"               # search for the height of the requested picture (are between the string 'HEIGHT=' and a ampersand '&amp;') and give it the key 'height'
  
  # rexp_bbox = r"(((?P&lt;bbox_xmin&gt;-?\d+\.?\d*)%2C)((?P&lt;bbox_ymin&gt;-?\d+\.?\d*)%2C)((?P&lt;bbox_xmax&gt;-?\d+\.?\d*)%2C)((?P&lt;bbox_ymax&gt;-?\d+\.?\d*)[\s&amp;]))"  # FUNKAR INTE ÄNNU HÄR KAN MAN FORTSÄTTA
  
  # create counter dictionary
  cnt_domains = Counter()                 # for counting the occurrances of a certain layer
  cnt_port = Counter()                    # for counting the occurrances of a certain layer
  cnt_ip = Counter()                      # for counting the occurrances of a IP-adress
  #cnt_date = Counter()                    # for counting the occurrances of a certain date  -- i probably will not use that
  
  cnt_name = Counter()                    # for counting the occurrances of a certain service
  cnt_coordsys = Counter()                # for counting the occurrances of a certain coordinate system
  cnt_width = Counter()                   # for counting the occurrances of a certain requested width
  cnt_height = Counter()                  # for counting the occurrances of a certain requested height
  cnt_bbox = Counter()
  
  # Compile regular expression for faster computing
  rexp_layer_compile = re.compile(rexp_layer, re.IGNORECASE)      # get the regex to look for occurrences of LAYERS or layers - seems to work
  rexp_port_compile = re.compile(rexp_port)
  rexp_ip_compile = re.compile(rexp_ip)
  rexp_name_compile = re.compile(rexp_name, re.IGNORECASE)        # No diffenence with re.IGNORECASE
  rexp_coordsys_compile = re.compile(rexp_coordsys)               # mixes in regex for layers
  rexp_width_compile = re.compile(rexp_width, re.IGNORECASE)
  rexp_height_compile = re.compile(rexp_height, re.IGNORECASE)
  # rexp_bbox_compile = re.compile(rexp_bbox)
  
  # Path to folder with log files
  #path = '/home/uwestephan/Logg-file-parsing/ws00848'
  # path = '/home/uwestephan/Logg-file-parsing/ws00524'
  # path = '/home/uwestephan/Logg-file-parsing/ws00524_test'
  
  path = '/home/uwestephan/Logg-file-parsing/ws00848_test'
  
  # setting the line counters to zero
  matchedGETMAP = 0
  failedGETMAP = 0
  failed = 0
  failedLAYER = 0
  
  # open file
  for filename in os.listdir(path):
      filmedsokvag = (path+"/"+filename)
      print (filmedsokvag)
  
      # read file / gather data
      f = open(filmedsokvag, 'r')
  
      # exclude all lines that do not have the string 'GetMap' in it
      for line in f:
          if re.findall('GetMap',line):                   # check if there is a string 'GetMap' in the line in the log file
  
              m = re.match(rexp_layer_compile, line)      # match the name of the requested layer
              p = re.match(rexp_port_compile, line)       # match the port
              i = re.match(rexp_ip_compile, line)                 # match the IP-adress
              n = re.match(rexp_name_compile, line)       # match the name of the WMS-service thats requested
              c = re.match(rexp_coordsys_compile, line)   # match the coordinate system
              w = re.match(rexp_width_compile, line)      # match the width of the requested picture that the WMS-service is sending
              h = re.match(rexp_height_compile, line)     # match the height of the requested picture that the WMS-service is sending
  #            b = re.match(rexp_bbox_compile, line)
  
              if m:
                  cnt_domains.update([m.group('domain')])     # here I try to count the occurrences of a the layer names
                  # matchedGETMAP += 1                          # add 1 to the line counter that count processed lines in the file (as i do not process all lines in this if sentence)
              else:
  #                failedGETMAP += 1
                  failedLAYER += 1                        # Counts the number of lines with a getmap request who do NOT have the parameter LAYER called
  
  
              if p:
                  cnt_port.update([p.group('port')])          # here I try to count the occurrences of a the differnt ports
  #            else:
  #               continue
  
              if i:
                  cnt_ip.update([i.group('ip')])              # here I try to count the occurrences of the IP-adresses - THAT ONE DOES NOT WORK
  
              #For debugging only - the regular expression for the IP adress seems not to work
              else:
                  print("Cannot find IP address")
  
              if n:
                  cnt_name.update([n.group('name')])          # here I try to count the occurrences of a the names of the WMS-services
                  matchedGETMAP += 1                          # add 1 to the line counter that count processed lines in the file (as i do not process all lines in this if sentence)
              else:
                  failedGETMAP += 1
  
              if c:
                  cnt_coordsys.update([c.group('koordsys')])  # here I try to count the occurrences of a coordinate systems
  #            else:
  #                continue
  
              if w:
                  cnt_width.update([w.group('width')])        # here I try to count the occurrences of the widths of the requested pictures that the WMS-service is sending
  #            else:
  #                continue
  
              if h:
                  cnt_height.update([h.group('height')])        # here I try to count the occurrences of the heights of the requested pictures that the WMS-service is sending
  #            else:
  #                continue
  
  #            if b:
  #                cnt_bbox.update([b.group('bbox_xmin')])        # here I try to count the occurrences of the heights of the requested pictures that the WMS-service is sending
  #            else:
  #                continue
  
  
  
          else:
              failed += 1         # add 1 to the counter that counts the lines that NOT processed by the if sentence above
              continue
  
  
  # Remove hyphon from the cnt_domains dictionary - not realy neccesarry -&gt; IT CREATES NOT A COUNTER DICTIONARY BUT A NORMAL DICTIONARY
  # cnt_domains = {key.replace('"',''): val for key,val in cnt_domains.items()}
  
  
  # Create an empty dictionary for my replace values
  f100 = open('Oversattningstabell_for_lagernamn_csv.csv', 'r')
  DictionaryReplaceValues = {}
  for line in f100:
      x = line.split(",")
      a = x[0]
      b = x[1]
      c = len(b)-1        # Removes the \n from the end of each line by counting the lenght of the line b and the reassigning a shorter string back to b
      b = b[0:c]          # Removes the \n from the end of each line by counting the lenght of the line b and the reassigning a shorter string back to b
      DictionaryReplaceValues[a]=b
  
  print("\n\nDet här är min Replacement dictionary")
  for key in DictionaryReplaceValues.keys():
      print (key, " = ", DictionaryReplaceValues[key])
  
  
  # Create an empty dictionary for the translated dictionary - Not really neccesarry
  cnt_domains_newname = {}
  
  # Replace the old dictionary with an new one using the translating dictionary DictionaryReplaceValues
  cnt_domains_newname = dict((DictionaryReplaceValues.get(key, key), value) for (key, value) in cnt_domains.items())
  
  # Make a counter out of the dictionary created above
  new_counter_cnt_domains_newname = Counter(cnt_domains_newname)
  
  
  
  # Output Results
  print('[*] %d Number of GetMap request that matched the regular expression' % (matchedGETMAP))
  print('[*] %d Number of GetMap request that failed to match the regular expression' % (failedGETMAP), end='\n\n')
  print('[*] %d Number of other request in the log files ' % (failed), end='\n\n')
  print('[*] %d Number of GetMap requests that request the Top layer of the WMS' % (failedLAYER), end='\n\n')
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring Layer Queried')
  print('[*] ============================================')
  #for domain, count in cnt_domains_newname.most_common(100):
  for domain, count in new_counter_cnt_domains_newname.most_common(100):
      print('[*] %60s: %d' % (domain, count))
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring Port Queried')
  print('[*] ============================================')
  for port, count in cnt_port.most_common(100):
      print('[*] %60s: %d' % (port, count))
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring IP-adresses Queried')
  print('[*] ============================================')
  for ip, count in cnt_ip.most_common(100):
      print('[*] %60s: %d' % (ip, count))
  #    print(ip, count)
  print('[*] ============================================')
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring WMS-name Queried')
  print('[*] ============================================')
  for name, count in cnt_name.most_common(100):
      print('[*] %60s: %d' % (name, count))
  print('[*] ============================================')
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring Coordinate Systemes Queried')
  print('[*] ============================================')
  for koordsys, count in cnt_coordsys.most_common(100):
      print('[*] %60s: %d' % (koordsys, count))
  print('[*] ============================================')
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring Picture Widths Queried')
  print('[*] ============================================')
  for width, count in cnt_width.most_common(100):
      print('[*] %60s: %d' % (width, count))
  print('[*] ============================================')
  print('[*] ============================================')
  print('[*] 100 Most Frequently Occurring Picture Heights Queried')
  print('[*] ============================================')
  for height, count in cnt_height.most_common(100):
      print('[*] %60s: %d' % (height, count))
  print('[*] ============================================')
  #print('[*] ============================================')
  #print('[*] 100 Most Frequently Occurring BBOX_xmin Queried')
  #print('[*] ============================================')
  #for bbox_xmin, count in cnt_bbox.most_common(100):
  #    print('[*] %30s: %d' % (bbox_xmin, count))
  #print('[*] ============================================')
  
  # Output results to file
  with open('parseroutput.txt', 'w') as fd:
      print('[*] %d Number of GetMap request that matched the regular expression' % (matchedGETMAP), file=fd)
      print('[*] %d Number of GetMap request that failed to match the regular expression' % (failedGETMAP), end='\n\n', file=fd)
      print('[*] %d Number of other request in the log files ' % (failed), end='\n\n', file=fd)
      print('[*] %d Number of GetMap requests that request the Top layer of the WMS' % (failedLAYER), end='\n\n', file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring Layer Queried', file=fd)
      print('[*] ============================================', file=fd)
      for domain, count in new_counter_cnt_domains_newname.most_common(100):
          print('%s: %d' % (domain, count), file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring Port Queried', file=fd)
      print('[*] ============================================', file=fd)
      for port, count in cnt_port.most_common(100):
          print('%s: %d' % (port, count), file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring IP-adresses Queried', file=fd)
      print('[*] ============================================', file=fd)
      for ip, count in cnt_ip.most_common(100):
          print('%s: %d' % (ip, count), file=fd)
          print(ip, count)
      print('[*] ============================================', file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring WMS-name Queried', file=fd)
      print('[*] ============================================', file=fd)
      for name, count in cnt_name.most_common(100):
          print('%s: %d' % (name, count), file=fd)
      print('[*] ============================================', file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring Coordinate Systemes Queried', file=fd)
      print('[*] ============================================', file=fd)
      for koordsys, count in cnt_coordsys.most_common(100):
          print('%s: %d' % (koordsys, count), file=fd)
      print('[*] ============================================', file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring Picture Widths Queried', file=fd)
      print('[*] ============================================', file=fd)
      for width, count in cnt_width.most_common(100):
          print('%s: %d' % (width, count), file=fd)
      print('[*] ============================================', file=fd)
      print('[*] ============================================', file=fd)
      print('[*] 100 Most Frequently Occurring Picture Heights Queried', file=fd)
      print('[*] ============================================', file=fd)
      for height, count in cnt_height.most_common(100):
          print('%s: %d' % (height, count), file=fd)
      print('[*] ============================================', file=fd)
  </code></pre>
  
  <p>Do you have any ideas how to accomplish that the regex extracts the IP-addresses?</p>