title: Loading a CSV in memory with Cassava
body: <p>I am trying to load a CSV into memory as a Vector of Vector with Cassava. My program does work but uses a huge amount of memory for a 50MB csv file and I don't understand why.</p>
  
  <p>I know that working with Data.Csv.Streaming should be better for big files, but I thought 50MB would still be ok. I tried both Data.Csv and Data.Csv.Streaming with more or less canonical examples from the github project page, I also tried to implement my own parser that outputs Vector of Vector (I based my code on attoparsec-csv <a href="https://hackage.haskell.org/package/attoparsec-csv" rel="noreferrer">https://hackage.haskell.org/package/attoparsec-csv</a>), and all these solutions use about 2000MB of memory! I am sure there is something wrong in what I am doing. What is the right way of doing this?</p>
  
  <p>My final goal is to have the data fully loaded into memory for further processing later on. For example, I could split my data into interesting matrices and work with those using Hmatrix.</p>
  
  <p>Here are the 2 programs I tried with Cassava:</p>
  
  <p>1/ Using Data.Csv</p>
  
  <pre><code>import qualified Data.ByteString.Lazy as BL
  import qualified Data.Vector as V
  import Data.Csv
  import Data.Foldable
  
  
  main = do
     csv &lt;- BL.readFile "train.csv"
     let Right res = decode HasHeader csv :: Either String (V.Vector(V.Vector(BL.ByteString)))
     print $ res V.! 0
  </code></pre>
  
  <p>2/ Using Data.Csv.Streaming</p>
  
  <pre><code>{-# LANGUAGE BangPatterns #-}
  
  import qualified Data.ByteString.Lazy as BL
  import qualified Data.Vector as V
  import Data.Csv.Streaming
  import Data.Foldable
  
  
  main = do
     csv &lt;- BL.readFile "train.csv"
     let !a = decode HasHeader csv :: Records(V.Vector(BL.ByteString))
     let !res = V.fromList $ Data.Foldable.toList a
     print $ res V.! 0
  </code></pre>
  
  <p>Note that I don't give you the program I made based on attoparsec-csv because it is almost exactly the same with Vector instead of List. The memory usage of that solution is still quite poor.</p>
  
  <p>Interestingly, in the Data.Csv.Streaming solution, if I simply print my data using a Data.Foldable.for_, everything is super fast with a 2MB memory usage. This made me think that my problem is linked to the way I construct my Vector. Probably accumulating thunks instead of stacking raw data into a compact data structure.</p>
  
  <p>Thank you for your help,</p>
  
  <p>Antoine</p>