title: Scala: Iterate over 20k images once, crop 100 ways to make 100 Iterators
body: <h3>requirement</h3>
  
  <p>I need to iterate over images, splitting each into 100 blocks (ROIs) and calculating something independently per block. I can't store anything other than the file paths in a list in memory, and I can't perform disk IO more than once. Performance is more important than simplicity here. How do I build 100 iterators while iterating over images?</p>
  
  <h3>code</h3>
  
  <p>I've written this a few ways but always get a StackOverflowError after ~5 hours (should finish in under 20 minutes).
  The following is the way that made the most sense to me: Iterate over an in-memory list of paths and build a map of iterators.</p>
  
  <pre><code>def calcAll(run: ImageBase, rois: Traversable[Roi]): Map[Roi, TraversableOnce[T]] = {
      val results: mutable.Map[Roi, Iterator[T]] = emptyMutableMap(rois)
  
      // calculate feature between every two frames
      var prevImage: RichImage = null // it'll be ok, I promise
      for (frame &lt;- ImageStore.walk(run) { // iterates over nio Path objects
          val image = RichImages.of(frame)
          if (prevImage != null) for (roi &lt;- rois) {
              val next: Iterator[T] = calc(Iterator(prevImage.crop(roi), image.crop(roi)))
              results(roi) = results(roi) ++ next // StackOverflowError!!
          }
          prevImage = image
      }
  
      results.toMap // immutable
  }
  </code></pre>
  
  <h3>background</h3>
  
  <p>I have a directory of 20k grayscale frames from a video. The video has a set of 100 Regions of Interest (ROIs), non-overlapping rectangles that we care about. I need to calculate features between consecutive images, but independently for each ROI. The amount of data and number of ROIs prohibits reading an image more than once.</p>