title: Storing large structured binary data with Haskell
body: <p>I'm writing an application that interacts with a large (10-1000 GB) memory-mapped binary file, holding essentially a bunch of objects that refer to each other. I've come up with a mechanism to read/write this data which is effective, but ugly and verbose (imo).</p>
  
  <p><strong>Q: Is there a more elegant way to achieve what I've done?</strong></p>
  
  <p>I have a typeclass for structured data, with one method that reads the structure into a Haskell datatype (<code>DataOp</code> is a <code>ReaderT</code> around <code>IO</code>).</p>
  
  <pre><code>class DBStruct a where
      structRead :: Addr a -&gt; DataOp a
  </code></pre>
  
  <p>To make this more readable, I have another typeclass that defines what structure members go where:</p>
  
  <pre><code>class DBStruct st =&gt; StructMem structTy valTy name | structTy name -&gt; valTy where
      offset :: structTy -&gt; valTy -&gt; name -&gt; Int64
  </code></pre>
  
  <p>I have a few helper functions that use the offset method for reading/writing structure elements, for reading structures from stored references, and for lazily deferring structure reads (to allow lazy reading of the entire file).</p>
  
  <p>The problem with this is that it involves a lot of repetition to use. For one structure, I first have to define the Haskell type:</p>
  
  <pre><code>data RowBlock = RowBlock {rbNext :: Maybe RowBlock
                           ,rbPrev :: Maybe RowBlock
                           ,rbRows :: [RowTy]
                           }
  </code></pre>
  
  <p>Then the <code>name</code> types:</p>
  
  <pre><code>data Next = Next
  data Prev = Prev
  data Count = Count
  newtype Row = Row Int64
  </code></pre>
  
  <p>Then the instances for each structure member:</p>
  
  <pre><code>instance StructMem RowBlock (Maybe (Addr RowBlock)) Next where offset _ _ _ = 0
  instance StructMem RowBlock (Maybe (Addr RowBlock)) Prev where offset _ _ _ = 8
  instance StructMem RowBlock Int64 Count where offset _ _ _ = 16
  instance StructMem RowBlock RowTy Row where offset _ _ (Row n) = 24 + n * 8
  </code></pre>
  
  <p>Then the structure read method:</p>
  
  <pre><code>instance DBStruct RowBlock where
      structRead a = do
          n &lt;- elemMaybePtr a Next
          p &lt;- elemMaybePtr a Prev
          c &lt;- elemRead a Count
          rs &lt;- mapM (elemRead a . Row) [0 .. c-1]
          return $ RowBlock n p rs
  </code></pre>
  
  <p>So all I've really accomplished is to re-implement C structs in a much more verbose (and slow) way. I would be happier if this were more concise while preserving type safety. Surely this is a commonly encountered problem.</p>
  
  <p>A few possible alternatives I can think of are:</p>
  
  <ul>
  <li>Ditch memory-mapped files and use <code>Data.Binary</code>, writing the <code>ByteStrings</code> to disk the normal way.</li>
  <li>Use <code>deriving Generic</code> to create generic read and write functions</li>
  <li><a href="http://twanvl.nl/blog/haskell/overloading-functional-references" rel="nofollow">Overload Functional References</a></li>
  <li>Do something magical with monadic lenses.</li>
  </ul>
  
  <p>EDIT: <a href="http://pastebin.com/Hc3qjz0Q" rel="nofollow">SSCCE as requested</a></p>