title: How to read big csv file?
body: <p>I trying to read a big csv file by haskell, and generate the word count by each column.</p>
  
  <p>This more than 4M rows in the file. </p>
  
  <p>So I choice read a block and get the word count each time(5k rows one block).
  And than sum it together.</p>
  
  <p>When I test the function with 12000 rows and 120000 rows the time increase almost linear.
  But When read 180000 rows,  run time exceeds more than four times.</p>
  
  <p>I think it because the memory is not enough，swap with disk make the function much slower.</p>
  
  <p>I had write my code as map/reduce style,But how to make the haskell don't hold all data in memory?</p>
  
  <p>The blow is my code and profiling result.</p>
  
  <pre><code>import Data.Ord
  import Text.CSV.Lazy.String
  import Data.List
  import System.IO
  import Data.Function (on)
  import System.Environment
  
  splitLength = 5000
  
  
  mySplit' [] = []
  mySplit' xs = [x] ++  mySplit' t 
      where
      x = take splitLength xs 
      t = drop splitLength xs                
  
  getBlockCount::Ord a =&gt; [[a]] -&gt; [[(a,Int)]]
  getBlockCount t =   map 
      (map (\x -&gt; ((head x),length x))) $ 
      map group $ map sort $ transpose t
  
  foldData::Ord a=&gt; [(a,Int)]-&gt;[(a,Int)]-&gt;[(a,Int)]
  foldData lxs rxs = map combind wlist
      where
          wlist = groupBy ((==) `on` fst) $ sortBy (comparing fst) $ lxs ++ rxs
          combind xs 
           | 1==(length xs) = head xs
           | 2 ==(length xs) = (((fst . head) xs ), ((snd . head) xs)+((snd . last) xs))
  
  
  loadTestData datalen = do
      testFile &lt;- readFile "data/test_csv"
      let cfile = fromCSVTable $ csvTable $ parseCSV testFile
      let column = head cfile
      let body = take datalen $ tail cfile
      let countData = foldl1' (zipWith  foldData)  $ map  getBlockCount  $ mySplit' body 
      let output =  zip column $ map ( reverse . sortBy (comparing snd) ) countData
      appendFile "testdata" $ foldl1 (\x y -&gt; x ++"\n"++y)$ map show $tail output
  
  main = do 
      s&lt;-getArgs
      loadTestData $ read  $ last s
  </code></pre>
  
  <p>profiling result </p>
  
  <pre><code>loadData +RTS -p -RTS 12000
  
  total time  =        1.02 secs   (1025 ticks @ 1000 us, 1 processor)
  total alloc = 991,266,560 bytes  (excludes profiling overheads)
  
   loadData +RTS -p -RTS 120000
  
  total time  =       17.28 secs   (17284 ticks @ 1000 us, 1 processor)
  total alloc = 9,202,259,064 bytes  (excludes profiling overheads)
  
  
  
     loadData +RTS -p -RTS 180000
  
  total time  =       85.06 secs   (85059 ticks @ 1000 us, 1 processor)
  total alloc = 13,760,818,848 bytes  (excludes profiling overheads)
  </code></pre>