title: python script to merge and consolidate logfiles
body: <p>I've 2 logfiles (sample_1.log and sample_2.log) in different formats. Both logs have the same attributes like:</p>
  <ul>
  <li>timestamp (2020-07-22 07:22:37.822863)</li>
  <li>error category (Exception or ERROR)</li>
  <li>error message (Error1,Error2..)</li>
  <li>user (oracle,administrator...)</li>
  <li>id (3521,2294...)</li>
  </ul>
  <p><strong>sample_1.log</strong> has a JSON like pattern and looks like this:</p>
  <pre><code>2020-07-22 07:22:37.822863: { &quot;level&quot;: &quot;Exception&quot;, &quot;message&quot;: &quot;Error4&quot;, &quot;user&quot;: &quot;oracle&quot;, &quot;id&quot;: &quot;3521&quot; }  
  2020-09-22 12:31:44.789319: { &quot;level&quot;: &quot;Exception&quot;, &quot;message&quot;: &quot;Error1&quot;, &quot;user&quot;: &quot;administrator&quot;, &quot;id&quot;: &quot;4371&quot; }  
  2021-04-06 22:51:10.999642: { &quot;level&quot;: &quot;ERROR&quot;, &quot;message&quot;: &quot;Error3&quot;, &quot;user&quot;: &quot;azureuser&quot;, &quot;id&quot;: &quot;2294&quot; }  
  2020-07-19 15:45:58.576940: { &quot;level&quot;: &quot;Exception&quot;, &quot;message&quot;: &quot;Error2&quot;, &quot;user&quot;: &quot;hacker40&quot;, &quot;id&quot;: &quot;8677&quot; }  
  2021-01-23 14:07:18.922480: { &quot;level&quot;: &quot;ERROR&quot;, &quot;message&quot;: &quot;Error5&quot;, &quot;user&quot;: &quot;hacker40&quot;, &quot;id&quot;: &quot;1865&quot; }  
  2020-08-19 05:47:46.983299: { &quot;level&quot;: &quot;Exception&quot;, &quot;message&quot;: &quot;Error4&quot;, &quot;user&quot;: &quot;pi&quot;, &quot;id&quot;: &quot;8993&quot; }  
  2021-03-25 13:13:06.012237: { &quot;level&quot;: &quot;ERROR&quot;, &quot;message&quot;: &quot;Error5&quot;, &quot;user&quot;: &quot;mysql&quot;, &quot;id&quot;: &quot;3561&quot; }  
  2020-05-05 06:37:50.976402: { &quot;level&quot;: &quot;ERROR&quot;, &quot;message&quot;: &quot;Error5&quot;, &quot;user&quot;: &quot;pi&quot;, &quot;id&quot;: &quot;1754&quot; } 
  </code></pre>
  <p><strong>sample_2.log</strong> has a more general format and looks like this:</p>
  <pre><code>ERROR at 19:30:13 on Thu, 02/07/2020  - Error1 - tracking id is 4126 - user is puppet.  
  ERROR at 12:08:30 on Mon, 26/10/2020  - Error1 - tracking id is 5567 - user is ftp.  
  Exception at 21:35:12 on Sun, 28/06/2020  - Error5 - tracking id is 8077 - user is vagrant.  
  ERROR at 06:36:05 on Sat, 11/07/2020  - Error1 - tracking id is 5218 - user is puppet.  
  Exception at 17:40:33 on Fri, 01/01/2021  - Error3 - tracking id is 8252 - user is mysql.  
  Exception at 13:49:18 on Wed, 06/05/2020  - Error2 - tracking id is 8369 - user is hacker15.  
  Exception at 21:09:05 on Sat, 16/05/2020  - Error1 - tracking id is 5091 - user is adm.  
  ERROR at 10:39:13 on Thu, 29/04/2021  - Error2 - tracking id is 4225 - user is oracle.  
  </code></pre>
  <p>Is it possible to write a python script that grab both logfiles, consolidate them to the same format and sort the events by date?</p>
  <p>The consolidation script should have two import parameters and one output parameter:</p>
  <pre><code>consolidate.py sample_1.log sample_2.log output.log
  </code></pre>
  <p>Output format can be like in sample_2.log</p>
  <p>I'm thankful for every suggestion</p>
  <p><code>Edit</code></p>
  <p>Someone helped me to transform the content of <em><strong>sample_1.log</strong></em> to the same format like <em><strong>sample_2.log</strong></em></p>
  <pre class="lang-py prettyprint-override"><code>
  import re
  from datetime import datetime, date
  
  file = open(&quot;input1.log&quot;,&quot;r&quot;)
  log = file.read().splitlines()
  file.close()
  
  for e in log:
      '''                  2020-06-24 19:07:54.153862: { &quot;level&quot;: &quot;Exception&quot;, &quot;message&quot;: &quot;Error4&quot;, &quot;user&quot;: &quot;oracle&quot;, &quot;id&quot;: &quot;9293&quot; }'''
      pattern = r'(\d{4}-\d{2}-\d{2}) (\d\d:\d\d:\d\d.\d+): { &quot;level&quot;: &quot;(.+)&quot;, &quot;message&quot;: &quot;(.+)&quot;, &quot;user&quot;: &quot;(.+)&quot;, &quot;id&quot;: &quot;(.+)&quot; }'
      match = re.search(pattern, e)
  
      DAY_OF_WEEK = {
          0: &quot;Mon&quot;,
          1: &quot;Tue&quot;,
          2: &quot;Wed&quot;,
          3: &quot;Thu&quot;,
          4: &quot;Fri&quot;,
          5: &quot;Sat&quot;,
          6: &quot;Sun&quot;,
      }
  
      date_ = match.group(1)
      time = match.group(2)
      level = match.group(3)
      message = match.group(4)
      user = match.group(5)
      id = match.group(6)
  
      day = date.fromisoformat(date_).weekday()
      day = DAY_OF_WEEK[day]
  
      time = time.split('.')[0]
  
      dt = re.sub(r'-', '/', date_)
  
      newFormat = f'{level} at {time} on {day}, {dt}  - {message} - tracking id is {id} - user is {user}'
      print(newFormat)
  
  </code></pre>
  <p>How can I improve the script or append the output of the python script above to sample_2.log and sort them by date?</p>
  <p>Hopefully someone can help me now</p>