title: Parsing large log files in Haskell
body: <p>Suppose I have several 200mb+ files that I want to grep through. How would I do this in Haskell?</p>
  
  <p>Here's my initial program:</p>
  
  <pre><code>import Data.List
  import Control.Monad
  import System.IO
  import System.Environment
  
  main = do
    filename &lt;- liftM head getArgs
    contents &lt;- liftM lines $ readFile filename
    putStrLn . unlines . filter (isPrefixOf "import") $ contents
  </code></pre>
  
  <p>This reads the whole file into memory before parsing through it.
  Then I went with this:</p>
  
  <pre><code>import Data.List
  import Control.Monad
  import System.IO
  import System.Environment
  
  main = do
    filename &lt;- liftM head getArgs
    file &lt;- (openFile filename ReadMode)
    contents &lt;- liftM lines $ hGetContents file
    putStrLn . unlines . filter (isPrefixOf "import") $ contents
  </code></pre>
  
  <p>I thought since <code>hGetContents</code> is lazy, <a href="http://book.realworldhaskell.org/read/io.html#io.lazy" rel="nofollow">it will avoid reading the whole file into memory</a>. But running both scripts under <code>valgrind</code> showed similar memory usage for both. So either my script is wrong, or <code>valgrind</code> is wrong. I compile the scripts using</p>
  
  <pre><code>ghc --make test.hs -prof
  </code></pre>
  
  <p>What am I missing? Bonus question: I see a lot of mentions on SO of how Lazy IO in Haskell is actually a bad thing. How / why would I use strict IO?</p>
  
  <p><strong>Update:</strong></p>
  
  <p>So it looks like I was wrong in my reading of valgrind. Using <code>+RTS -s</code>, here's what I get:</p>
  
  <pre><code> 7,807,461,968 bytes allocated in the heap
   1,563,351,416 bytes copied during GC
         101,888 bytes maximum residency (1150 sample(s))
          45,576 bytes maximum slop
               2 MB total memory in use (0 MB lost due to fragmentation)
  
  Generation 0: 13739 collections,     0 parallel,  2.91s,  2.95s elapsed
  Generation 1:  1150 collections,     0 parallel,  0.18s,  0.18s elapsed
  
  INIT  time    0.00s  (  0.00s elapsed)
  MUT   time    2.07s  (  2.28s elapsed)
  GC    time    3.09s  (  3.13s elapsed)
  EXIT  time    0.00s  (  0.00s elapsed)
  Total time    5.16s  (  5.41s elapsed)
  </code></pre>
  
  <p>The important line is <code>101,888 bytes maximum residency</code>, which says that at any given point my script was using 101 kb of memory at most. The file I was grepping through was 44 mb. So I think the verdict is: <code>readFile</code> and <code>hGetContents</code> are both lazy.</p>
  
  <p><strong>Follow-up question:</strong></p>
  
  <p>Why do I see 7gb of memory allocated on the heap? That seems really high for a script that's reading in a 44 mb file.</p>
  
  <p><strong>Update to follow-up question</strong></p>
  
  <p>Looks like a few gb of memory allocated on the heap is not atypical for Haskell, so no cause for concern. Using <code>ByteString</code>s instead of <code>String</code>s takes the memory usage down a lot:</p>
  
  <pre><code>  81,617,024 bytes allocated in the heap
        35,072 bytes copied during GC
        78,832 bytes maximum residency (1 sample(s))
        26,960 bytes maximum slop
             2 MB total memory in use (0 MB lost due to fragmentation)
  </code></pre>