title: Refactoring large data object
body: <p><strong>What are some common strategies for refactoring large "state-only" objects?</strong></p>
  
  <p>I am working on a specific soft-real-time decision support system which does online modeling/simulation of the national airspace.  This piece of software consumes a number of live data feeds, and produces a once-per-minute estimate of the "state" of a large number of entities in the airspace.  The problem breaks down neatly until we hit what is currently the lowest-level entity.</p>
  
  <p>Our mathematical model estimates/predicts upwards of 50 parameters for a timeline of several hours into the past and future for each of these entities, roughly once per minute.  Currently, these records are encoded as a single Java class with a lot of fields (some get collapsed into an <code>ArrayList</code>).  Our model is evolving, and the dependencies among the fields are not yet set in stone, so each instance wanders through a convoluted model, accumulating settings as it goes along.  </p>
  
  <p>Currently we have something like the following, which uses a builder pattern approach to build up the contents of the record, and enforce what the known dependencies are (as a check against programmer error as evolve the mode.)  Once the estimate is done, we convert the below into an immutable form using a <code>.build()</code> type method.</p>
  
  <pre><code>final class OneMinuteEstimate {
  
    enum EstimateState { INFANT, HEADER, INDEPENDENT, ... };
    EstimateState state = EstimateState.INFANT; 
  
    // "header" stuff
    DateTime estimatedAtTime = null;
    DateTime stamp = null;
    EntityId id = null;
  
    // independent fields
    int status1 = -1;
    ...
  
    // dependent/complex fields...
    ... goes on for 40+ more fields... 
  
    void setHeaderFields(...)
    {
       if (!EstimateState.INFANT.equals(state)) {
          throw new IllegalStateException("Must be in INFANT state to set header");
       }
  
       ... 
    }
  
  }
  </code></pre>
  
  <p>Once a very large number of these estimates are complete, they are assembled into timelines where aggregate patterns/trends are analyzed.  We have looked at using an embedded database but have struggled with performance issues; we'd rather get this sorted out in terms of data modeling and then incrementally move portions of the soft-real-time code into an embedded data store.</p>
  
  <p>Once the "time sensitive" pieces of this are done, the products are flushed to flat files and a database.</p>
  
  <p><strong>Problems:</strong></p>
  
  <ul>
  <li>It's a giant class, with way too many fields.  </li>
  <li>There is very little behavior encoded in the class; it's mostly a holder for data fields.</li>
  <li>Maintaining the <code>build()</code> method is extremely cumbersome.</li>
  <li>It feels clumsy to manually maintain a "state machine" abstraction merely for the purpose of ensuring that a large number of dependent modeling components are properly populating a data object, but it has saved us a lot of frustration as the model evolves.</li>
  <li>There is a lot of duplication, particularly when the records described above are aggregated into very similar "rollups" which amount to rolling sums/averages or other statistical products of the above structure in time series.</li>
  <li>While some of the fields could be clumped together, they are all logically "peers" of one another, and any breakdown we've tried has resulted in having behavior/logic artificially split and needing to reach two levels deep in indirection.</li>
  </ul>
  
  <p>Out of the box ideas entertained, but this is something we need to evolve incrementally.  Before anyone else says it, I'll note that one could suggest that our mathematical model is insufficiently crisp if the data representation for that model is this hard to get ahold of.  Fair point, and we're working that, but I think that's a side-effect of an R&amp;D environment with a lot of contributors, and a lot of concurrent hypotheses in play.</p>
  
  <p>(Not that it matters, but this is implemented in Java.  We use HSQLDB or Postgres for output products.  We don't use any persistence framework, partly out of a lack of familiarity, partly because we have enough performance trouble with just the database alone and hand-coded storage routines... we're skeptical of moving towards additional abstraction.)</p>