title: Reading WARC Files Efficiently
body: <p>I am reading a WARC file with python's 'warc' library. Current file that I am using, is around 4.50 GB. The thing is ;</p>
  
  <pre><code>file = warc.open("random.warc")
  html_lists = [line for line in file]
  </code></pre>
  
  <p>Executing these 2 lines takes up to 40 seconds. Since there will be 64000 more files like this one, it is not acceptable that it takes 40 seconds per file. Do you guys have any tips to improve performance or any different approaches?</p>
  
  <p>Edit : I found out that Beautifulsoup operations take some time. So I removed it and wrote the necessary stuff myself. It is 100x faster now. It takes +- 60 seconds to read and process 4.50 GB data. With this line of code I remove the scripts from data;</p>
  
  <pre><code>clean = re.sub(r"&lt;script.*?&lt;/script&gt;", "", string=text)
  </code></pre>
  
  <p>And with this one I split the text and remove the stamp which I don't need</p>
  
  <pre><code>warc_stamp = str(soup).split(r"\r\n\r\n")
  </code></pre>
  
  <p>As I said it is faster but 60 seconds are not that good in this case. Any suggestions ?</p>