title: C++ Designing for SIMD: Making an SoA less of a PiTA
body: <h2>Bittersweet SoAs</h2>
  
  <p>I've recently come to see the delights of using hand-written SIMD intrinsics with the SoA (structure of array) representation.</p>
  
  <p>The speed improvements over my former AoS (array of structures) code, at least for straightforward sequential-type streaming operations, was little short of amazing with doubled to tripled speed ups. As a bonus, it simplified the logic to exclude those tricky horizontal operations and shuffling components around in addition to reducing memory use.</p>
  
  <p>Yet then there's this bittersweet sense afterwards where I realize what a PITA they are to work with in code, especially interface design.</p>
  
  <h2>Mid-Level Interface Design</h2>
  
  <p>I'm often dealing with designing mid-level interfaces. They're higher-level than say, <code>std::vector</code>, but lower-level than say, a <code>Monster</code> class in a video game. These are always some of the most awkward interfaces for me to design and keep stable, because they're not low-level enough to provide a simple read/write interface as with a standard C++ container. Yet they're not high-level enough (lacking sufficient logic in the entry points in the interface) to completely hide and abstract away the underlying representation and only provide high-level operations.</p>
  
  <p>An example of what I consider a mid-level design is a programmable particle system API which desires to be as efficient and scalable as possible for certain scenarios while being convenient for casual scenarios (ex: to scripters). Such a design has to offer particle access, and unless it's going to have a method for every possible algorithm related to particles imaginable, it'll have to expose some of those raw SoA details somewhere, somewhere, to let clients benefit from them.</p>
  
  <p>The design also shouldn't necessarily be made to require SoA type code to be written all the time. The more daily usage still does not demand utmost efficiency so much as convenience, simplicity, productivity. It's only for those rarer, performance-critical scenarios where the underlying SoA representation comes in handy.</p>
  
  <p>So how do you API/lib designers and large-scale system guys deal with balancing these types of needs?</p>
  
  <h2>Balancing Multiple Access Patterns</h2>
  
  <p>Since the SoA obliterates away any per-element structure, might it be a decent idea to instantiate structs/classes on the fly as the user accesses the <code>nth</code> element using the more convenient, random-access portions of the interface? Perhaps a structure containing pointers/references to the nth entries of multiple SoA arrays for mutable access?</p>
  
  <p>Also if the more common usage patterns are more random-access scalar logic rather than sequential-access SIMD vector logic, but the SIMD portions are triggered enough to still make it better to just use one data structure for it all, might this kind of hybrid SoA representation balance all the needs better?</p>
  
  <pre><code>struct AoSoA
  {
      ALIGN16 float x[4];
      ALIGN16 float y[4];
      ALIGN16 float z[4];
  };
  ALIGN16 AoSoA elements[n/4];
  </code></pre>
  
  <p>I don't understand the nature of cache lines that well to know if this kind of representation is worthwhile. I have noticed it doesn't help so much for the sequential SIMD cases where we can devote the full resources to one bulky algorithm, but it seems like it might be helpful for cases that need a lot of horizontal logic across components or random-access scalar logic cases where the system might be doing a lot of other things at the same time.</p>
  
  <p>Anyway, I'm generally looking for insight into how to effectively design middle-level data structure interfaces with SoA backend representations as implementation details without transferring the complexity to the client unless they really want it.</p>
  
  <p>I really want to avoid forcing clients to always write SoA-type code in every place that uses the interface unless they really need that efficiency, and I'm curious as to how to balance those more daily, random-access scalar usage scenarios vs. the rarer but not too uncommon scenarios that take advantage of the SoA representation.</p>