title: How do I prevent scotty from eating up memory for large text output?
body: <p>I have a Scotty/WAI application and one of the endpoints sends a large <code>Text</code> output built from a list of elements. Here is the relevant code:</p>
  
  <pre><code>  import Data.Text.Lazy as L
    import Data.Text.Lazy.Encoding as E
  
    class (Show csv) =&gt; ToCSV csv where
      toCSV :: csv -&gt; L.Text
      toCSV = pack . show
  
    instance (ToCSV c) =&gt; ToCSV [c] where
      toCSV []     = empty
      toCSV (c:cs) = toCSV c &lt;&gt; "\n" &lt;&gt; toCSV cs
  
  
    get "/api/transactions" $ accept "text/csv" $ do
      purp &lt;- selectPurpose
      txs &lt;- allEntries &lt;$&gt; inWeb (listTransactions purp)
      setHeader "Content-Type" "text/csv"
      raw $ E.encodeUtf8 $ toCSV txs
  </code></pre>
  
  <p>As I understand <a href="http://hackage.haskell.org/package/scotty-0.10.1/docs/Web-Scotty.html" rel="nofollow">Scotty's documentation</a> the output should be lazily built and sent over the wire without the need to build the whole text/bytestring in memory. However this is not the behaviour that I observe: when I call this endpoint the server starts to eat up memory and I infer it is building the whole string, before sending it in one go.</p>
  
  <p>Am I missing something?</p>
  
  <p><em>Edit 1</em>:</p>
  
  <p>I have written a <code>doStream</code> function that's supposed to send chunks of resulting BS one by one:</p>
  
  <pre><code>doStream :: Text -&gt; W.StreamingBody   
  doStream t build flush = do
    let bs = E.encodeUtf8 t
    mapM_ (\ chunk -&gt; build (B.fromByteString chunk)) (BS.toChunks bs)
    flush
  </code></pre>
  
  <p>but actually it still builds the whole output in memory... </p>
  
  <p><em>Edit 2</em>:</p>
  
  <p>Actually, streaming this way works fine. The server process still eats up a lot of memory though, which might be actually garbage collectible upon sending each chunk. I will try to analyze memory usage more deeply to see where this consumption comes from. </p>
  
  <p><em>Edit 3</em>:</p>
  
  <p>I tried to limit heap to 2GB but this makes the process crash. Some memory is retained during the whole transformation process...</p>