title: Javascript Transducers for a Data Flow project
body: <p>with my current project, I am dealing with large streams of numerical data and transformations that have to take place on them in a data-flow-programmable fashion.</p>
  
  <p>I stumbled upon the idea of transducers, which promised to solve the difficulties to handle multiple transformations on large arrays. It seems that transducers don't suit exactly for what I'm trying to solve here.</p>
  
  <p>I am looking for a pattern / concept for transducers which only collect a needed amount of lookback to then process out a result. Similar to the browser version of tensorflow, reaktor, max-msp (input outputs, flow-graphs, node-based, visual-programming)</p>
  
  <p>Most of these modules, should be connected to a source, but should also be able to act as a source to chain those to other modules </p>
  
  <pre><code>source ( a stream ) =[new-value]|=&gt; module1 =&gt; module2 =&gt; ...
                                  |=&gt; module3 =&gt; module4 // branch off here to a new chain
  </code></pre>
  
  <p>From my understanding, the transducers as explained in most blogs takes the whole array, and feeds each individual values trough chosen transformers.</p>
  
  <p>Yet my modules/transformers don't require so much data to work, say the example of a simple moving average with a look back of 4 steps.</p>
  
  <p>I imagine that module to collect enough data until it starts it's output.
  I also don't need to hold the whole array in memory, I should only deal with the exact amounts needed. Results/Outputs would be optionally stored in a database.</p>
  
  <pre><code>stream =[sends-1-value]=&gt; module[collects-values-until-processing-starts] =[sends-one-value]=&gt;...
  </code></pre>
  
  <p>It should also be possible to connect multiple sources into a module (which transducers didn't seem to provide.</p>
  
  <p>Would the transducer pattern here still apply or is something else out there?</p>
  
  <p>To be honest, every programmer would have an idea to make this work, yet I am asking for some established way of doing it, just like transducers came to be. </p>