title: python 2.7 how to manipulate big file data extraction
body: <p>here is how looks my datafiles  (from directory)
  all is a huge (mess) log file pit containing different datas</p>
  <p>so I have to get from that some data</p>
  <p>here how it looks if I concatenate a few data</p>
  <pre><code>2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_DDD;RRRR_SSDDDD
  2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_III;FT_RECEPTION
  2022-08-31 23:51:05,560; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;&lt;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae&gt;;;&lt;Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD&gt;; END TRANSFER FROM SOURCE
  2022-08-31 23:51:07,233; example_data;;;example_data;TYPE_DDD;CON_CheckMeshArea_Output
  2022-08-31 23:51:07,233; example_data;;;example_data;TYPE_III;APP_REQload rate = 171.142578125 KB/s ;
  2022-08-31 23:51:07,521; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;;&lt;Source: PPPPP-CON;CON_CheckMeshArea_Output&gt;; ;&lt;Destination:PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea_Output&gt;;;&lt;7740db78-8
  05ad9427ab7&gt;; BEGIN UPLOAD TO DESTINATION
  2022-08-31 23:51:07,523; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;example_data;TYPE_DDD;CON_CheckMeshArea_Output
  2022-08-31 23:50:51,990; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;&lt;cd0fea11-de93-482c-b745-4336688e13ed&gt;;;&lt;Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea&gt;; BEGIN TRANSFER FROM SOURCE
  2022-08-31 23:50:52,118; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;&lt;Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea&gt;; ;&lt;Destination:PPPPP-CON;CON_CheckMeshArea&gt;;;&lt;f13b8416-1a13-4212-aac8-
  BEGIN UPLOAD TO DESTINATION
  2022-08-31 23:50:52,359; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;Uploaded size = 1 KB ; Upload duration = 0.008 seconds ; Upload rate = 245.4833984375 KB/s ;
  2022-08-31 23:50:52,359; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;&lt;Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea&gt;; ;&lt;Destination:PPPPP-CON;CON_CheckMeshArea&gt;;;&lt;f13b8416-1a13-4212-aac8-
  END UPLOAD TO DESTINATION
  2022-08-31 23:50:52,377; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;&lt;cd0fea11-de93-482c-b745-4336688e13ed&gt;;;&lt;Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea&gt;; END TRANSFER FROM SOURCE
  2022-08-31 23:51:01,920; example_data;;;example_data;TYPE_DDD;RRRR_SSDDDD
  2022-08-31 23:51:01,920; example_data;;;example_data;TYPE_III;APP_REQ
  2022-08-31 23:51:01,965; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;&lt;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae&gt;;;&lt;Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD&gt;; BEGIN TRANSFER FROM SOURCE
  2022-08-31 23:51:01,966;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP input data;server=p-APPSRV-82.SOMEDOMAIN port=990 user=service-p-PPPPPlication-0015 binaryTransfer=true passiveMode=true bufferSiz
  cit=true protocol=TLS
  2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply code=226
  2022-08-31 23:51:03,575; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;Downloaded size = 135895 KB ; Download duration = 1.045 seconds ; Download rate = 130043.72009569379 KB/s ;
  2022-08-31 23:51:03,647; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;&lt;Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD&gt;; ;&lt;Destination:PPPPP-GGGIP-MMMM;RRRR_SSDDDD&gt;;;&lt;7b19f9ba-203d-4479-96b2-cc6d664bffcd&gt;; 
  2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply string=226 Directory send OK.
  2022-08-31 23:51:07,279; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;;&lt;8841c2be-ee3a-4508-9c50-f817e5c7dfc8&gt;;;&lt;Source: PPPPP-CON;CON_CheckMeshArea_Output&gt;; BEGIN TRANSFER FROM SOURCE
  2022-08-31 23:51:07,479; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;Downloaded size = 0 KB ; Download duration = 0.004 seconds ; Down
  </code></pre>
  <p>from here I wanna extract for each Source/Destination couple the upload/download rate/size/duration &amp; id's (the hexadecimal parts)  So I had in my idea to</p>
  <ul>
  <li>sort by ID</li>
  <li>sort each by date</li>
  <li>make a specific output file for each</li>
  <li>then read each file for correlation to
  date/source/destination/size/rate/duration</li>
  </ul>
  <p>Si my idea was to run a loop to extract that from a regexp &amp; made it readable as a csv
  but I had memory errors</p>
  <p><em>(the host has only 4Go)</em></p>
  <p>I tryied many things like</p>
  <p>like</p>
  <pre><code> import os
  import re
  import string
  import sys
  import datetime
  
  datalogpath = sys.argv[1]               #  get the path of log files to extract datas
  startdatestamp = sys.argv[2]            #  get a chronologic start marker
  enddatestamp = sys.argv[3]              #  get a chronologic start marker
  outputfilename = sys.argv[4]            #  output file will be csv formated results
  outputfilename2 = 'testsort.txt'        #  output file will be csv formated results
  searchpattern = re.compile('linked to the consume|size|rate|type_|source|dest')    # regexp to filter from logs directory
  err_occur = []                          # just to manage error on empty line while reading the logs files
  t_fmt = '%Y-%M-%D %H:%m:%S,%f'          # timestamp format
  line_array = []                         # line buffer
  
  for filename in os.listdir(datalogpath):
          with open(os.path.join(datalogpath, filename)) as in_file:
                  for line in in_file:
                          #found = searchpattern.match(line)
                          found = searchpattern.search(line)
                          if found :
                                  # STEP2
                                  line = line.replace('|', ' - ')
                                  line = line.replace(' - ', ';')
                                  line = line.replace(' : ', ';')
                                  line = line.replace(' INFO ', ';')
                                  line = line.replace(' DEBUG ', ';')
                                  line = line.replace('&gt;', '&gt;;')
                                  line = line.replace('&lt;', ';&lt;')
                                  line_array += line # keep line
          # end open
  sorted_line_array = sorted(line_array,key=lambda line: datetime.strptime(line_array.split(' ')[2], t_fmt))
  </code></pre>
  <p>It might be run by : <code>python script.py ../data/ 20220810 20220813 out.txt</code></p>
  <p>I guess a better method would be to
  make it read/grep all of a date as a dictionary -&gt; then sort it -&gt; then write specific file for it ? if so how can I do that ?  (<em>I m more or less new to python ; an short Example would be appreciated</em>)</p>
  <p>Currently I got &quot;memory error&quot; whatever I change my algo/method
  The ideal result I would get is separated files for each ID within all sorted by date lines  matching the ID <code>(as very final result)</code></p>
  <p>so  expectation is to get separated files like :</p>
  <pre><code>7d4e2260-bf7e-4d81-9cd3-be7e4decaaae
  cd0fea11-de93-482c-b745-4336688e13ed
  8841c2be-ee3a-4508-9c50-f817e5c7dfc8
  etc...
  </code></pre>
  <p>each containing
  all matching lines about that ID sorted by date/time</p>
  <p>so for example for the 7d4e2260-bf7e-4d81-9cd3-be7e4decaaae file it will be</p>
  <pre><code>2022-08-31 23:51:01,965; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;&lt;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae&gt;;;&lt;Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD&gt;; BEGIN TRANSFER FROM SOURCE
  2022-08-31 23:51:01,966;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP input data;server=p-APPSRV-82.SOMEDOMAIN port=990 user=service-p-PPPPPlication-0015 binaryTransfer=true passiveMode=true bufferSiz
  2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply code=226
  2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply string=226 Directory send OK.
  2022-08-31 23:51:03,575; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;Downloaded size = 135895 KB ; Download duration = 1.045 seconds ; Download rate = 130043.72009569379 KB/s ;
  2022-08-31 23:51:03,647; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;&lt;Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD&gt;; ;&lt;Destination:PPPPP-GGGIP-MMMM;RRRR_SSDDDD&gt;;;&lt;7b19f9ba-203d-4479-96b2-cc6d664bffcd&gt;;
  2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_DDD;RRRR_SSDDDD
  2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_III;FT_RECEPTION
  2022-08-31 23:51:05,560; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;&lt;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae&gt;;;&lt;Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD&gt;; END TRANSFER FROM SOURCE
  </code></pre>
  <p>currently here is what I got is looking like :</p>
  <pre><code>[workdir]$ python loganalysze.fpu.py 2022-08-10 2022-08-13 ../data_Z2/
  Traceback (most recent call last):
    File &quot;loganalysze.fpu.py&quot;, line 34, in &lt;module&gt;
    found = searchpattern.search(line)
  MemoryError
  [workdir]$
  </code></pre>