title: Can yield produce multiple consecutive generators?
body: <p>Here are two functions that split iterable items to sub-lists. I believe that this type of task is programmed many times. I use them to parse log files that consist of <code>repr</code> lines like ('result', 'case', 123, 4.56) and ('dump', ..) and so on.</p>
  
  <p>I would like to change these so that they will yield iterators rather than lists. Because the list may grow pretty large, but I may be able to decide to take it or skip it based on first few items. Also, if the iter version is available I would like to nest them, but with these list versions that would waste some memory by duplicating parts.</p>
  
  <p>But deriving multiple generators from an iterable source wan't easy for me, so I ask for help. If possible, I wish to avoid introducing new classes.</p>
  
  <p>Also, if you know a better title for this question, please tell me.</p>
  
  <p>Thank you!</p>
  
  <pre><code>def cleave_by_mark (stream, key_fn, end_with_mark=False):
      '''[f f t][t][f f] (true) [f f][t][t f f](false)'''
      buf = []
      for item in stream:
          if key_fn(item):
              if end_with_mark: buf.append(item)
              if buf: yield buf
              buf = []
              if end_with_mark: continue
          buf.append(item)
      if buf: yield buf
  
  def cleave_by_change (stream, key_fn):
      '''[1 1 1][2 2][3][2 2 2 2]'''
      prev = None
      buf = []
      for item in stream:
          iden = key_fn(item)
          if prev is None: prev = iden
          if prev != iden:
              yield buf
              buf = []
              prev = iden
          buf.append(item)
      if buf: yield buf
  </code></pre>
  
  <hr>
  
  <h1>edit: my own answer</h1>
  
  <p>Thanks to everyone's answer, I could write what I asked for! Of course, as for the "cleave_for_change" function I could also use <code>itertools.groupby</code>.</p>
  
  <pre><code>def cleave_by_mark (stream, key_fn, end_with_mark=False):
      hand = []
      def gen ():
          key = key_fn(hand[0])
          yield hand.pop(0)
          while 1:
              if end_with_mark and key: break
              hand.append(stream.next())
              key = key_fn(hand[0])
              if (not end_with_mark) and key: break
              yield hand.pop(0)
      while 1:
          # allow StopIteration in the main loop
          if not hand: hand.append(stream.next())
          yield gen()
  
  for cl in cleave_by_mark (iter((1,0,0,1,1,0)), lambda x:x):
      print list(cl),  # start with 1
  # -&gt; [1, 0, 0] [1] [1, 0]
  for cl in cleave_by_mark (iter((0,1,0,0,1,1,0)), lambda x:x):
      print list(cl),
  # -&gt; [0] [1, 0, 0] [1] [1, 0]
  for cl in cleave_by_mark (iter((1,0,0,1,1,0)), lambda x:x, True):
      print list(cl),  # end with 1
  # -&gt; [1] [0, 0, 1] [1] [0]
  for cl in cleave_by_mark (iter((0,1,0,0,1,1,0)), lambda x:x, True):
      print list(cl),
  # -&gt; [0, 1] [0, 0, 1] [1] [0]
  </code></pre>
  
  <p>/</p>
  
  <pre><code>def cleave_by_change (stream, key_fn):
      '''[1 1 1][2 2][3][2 2 2 2]'''
      hand = []
      def gen ():
          headkey = key_fn(hand[0])
          yield hand.pop(0)
          while 1:
              hand.append(stream.next())
              key = key_fn(hand[0])
              if key != headkey: break
              yield hand.pop(0)
      while 1:
          # allow StopIteration in the main loop
          if not hand: hand.append(stream.next())
          yield gen()
  
  for cl in cleave_by_change (iter((1,1,1,2,2,2,3,2)), lambda x:x):
      print list(cl),
  # -&gt; [1, 1, 1] [2, 2, 2] [3] [2]
  </code></pre>
  
  <p><strong>CAUTION:</strong> If anyone's going to use these, be sure to <strong>exhaust the generators</strong> at every level, as Andrew pointed out. Because otherwise the outer generator-yielding loop will restart right where the inner generator left instead of where the next "block" begins.</p>
  
  <pre><code>stream = itertools.product('abc','1234', 'ABCD')
  for a in iters.cleave_by_change(stream, lambda x:x[0]):
      for b in iters.cleave_by_change(a, lambda x:x[1]):
          print b.next()
          for sink in b: pass
      for sink in a: pass
  
  ('a', '1', 'A')
  ('b', '1', 'A')
  ('c', '1', 'A')
  </code></pre>