title: How to implement data streaming with the Snap framework?
body: <p>I'd like to implement streaming of large data (in both directions) with the Snap server. To explore the possibilities I created a sample program that has two endpoints - reading and writing. There is a very simple internal buffer that holds one <code>ByteString</code> and whatever is written to the writing endpoint appears in the reading one. (Currently there is no way how to terminate the stream, but that's fine for this purpose.)</p>
  
  <pre><code>{-# LANGUAGE OverloadedStrings #-}                                           
  import Control.Applicative                                                   
  import Control.Concurrent.MVar.Lifted                                        
  import Control.Monad                                                         
  import Data.ByteString (ByteString)                                          
  import Blaze.ByteString.Builder (Builder, fromByteString)                    
  import Data.Enumerator                                                       
  import qualified Data.Enumerator.List as E                                   
  import Data.Enumerator.Binary (enumFile, iterHandle)                         
  import Snap.Core                                                             
  import Snap.Http.Server                                                      
  
  main :: IO ()                                                                
  main = do                                                                    
    buf &lt;- newEmptyMVar                                                        
    quickHttpServe (site buf)                                                  
  
  site :: MVar ByteString -&gt; Snap ()                                           
  site buf =                                                                   
      route [ ("read", modifyResponse (setBufferingMode False                  
                                       . setResponseBody (fromBuf buf)))       
            , ("write", runRequestBody (toBuf buf))                            
            ]                                                                  
  
  fromBuf :: MVar ByteString -&gt; Enumerator Builder IO a                        
  fromBuf buf = E.repeatM (liftM fromByteString $ takeMVar buf)                
  
  toBuf :: MVar ByteString -&gt; Iteratee ByteString IO ()                        
  toBuf buf = E.mapM_ (putMVar buf)
  </code></pre>
  
  <p>Then I run in different terminals</p>
  
  <pre class="lang-sh prettyprint-override"><code>curl http://localhost:8000/read &gt;/dev/nul
  </code></pre>
  
  <p>and</p>
  
  <pre class="lang-sh prettyprint-override"><code>dd if=/dev/zero bs=1M count=100 | \
    curl --data-binary @- http://localhost:8000/write
  </code></pre>
  
  <p>But the writing part fails with <em>an exception escaped to toplevel: Too many bytes read</em>. This is obviously an instance of <code>TooManyBytesReadException</code>, but I couldn't find where it's thrown. Writing smaller amount of data like 1MB works as expected.</p>
  
  <p>My questions are:</p>
  
  <ol>
  <li>Where/how to fix the reading limit?</li>
  <li>Will this stream data, without loading the whole POST request in memory? If not, how to fix it?</li>
  </ol>