title: How do I fairly assign tasks to workers in Python? - Splitting iterable into similarly sized chunks
body: <p>I have workers and tasks to do:</p>
  
  <pre><code>workers = ['peter', 'paul', 'mary']
  tasks = range(13)
  </code></pre>
  
  <p>Now I want to split the tasks into chunks or batches of work, so that each worker can work on one batch and does about the same amount of work as everybody else. In my real life I want to schedule batch jobs to a compute farm. The batch jobs are supposed to run in parallel. The actual schedule&amp;dispatch is done by a commercial grade tool such as lsf or grid.</p>
  
  <p>Some examples of what I would expect:</p>
  
  <pre><code>&gt;&gt;&gt; distribute_work(['peter', 'paul', 'mary'], range(3))
  [('peter', [0]), ('paul', [1]), ('mary', [2])]
  &gt;&gt;&gt; distribute_work(['peter', 'paul', 'mary'], range(6))
  [('peter', [0, 3]), ('paul', [1, 4]), ('mary', [2, 5])]
  &gt;&gt;&gt; distribute_work(['peter', 'paul', 'mary'], range(5))
  [('peter', [0, 3]), ('paul', [1, 4]), ('mary', [2])]
  </code></pre>
  
  <p>This question is very similar to the questions <a href="https://stackoverflow.com/q/12285115/923794">here</a>, <a href="https://stackoverflow.com/q/5850536/923794">here</a>, and <a href="https://stackoverflow.com/q/434287/923794">here</a></p>
  
  <p>The difference is that I want these features, in the order or precedence:</p>
  
  <ol>
  <li>No use of <code>len</code>, if possible no build-up of long data structures internally</li>
  <li>Accept a generator</li>
  <li>Return generators</li>
  <li>As much use of stdlib components as possible</li>
  </ol>
  
  <p>Some side notes on requirements:</p>
  
  <ul>
  <li>No dicts on purpose: I've workers with the same name that can do multiple batches (unix hostnames). If your solution uses dicts, that's fine because we can always do worker lookup by a batch enumeration.</li>
  <li>Arbitrary length: Both workers and tasks can be iterables of any length >= 1. And they do not have to split evenly as shown in the example above where Mary only gets one task.</li>
  <li>Order: To me is not important. I guess others may prefer some order like [0,1], [2,3], [5], but I don't care. If your solution can keep or switch the order, maybe that's worthwhile pointing out to others.</li>
  </ul>
  
  <p>I have tried to wrap my head around <code>itertools</code> and this particular problem and came up with the following code to illustrate the question:</p>
  
  <pre><code>from itertools import *
  
  def distribute_work(workers, tasks):
      batches = range(len(workers))
      return [ ( workers[k],
                 [t[1] for t in i]
                 )   for (k,i) in groupby(sorted(zip(cycle(batches),
                                                     tasks),
                                                 key=lambda t: t[0]),
                                          lambda t: t[0]) ]
  </code></pre>
  
  <p>This satisfies 4., but the sort very likely violates 1.. And 2./3. are not even thought about.</p>
  
  <p>Probably there's some easy solution to this, combining some stdlib components in a way I haven't thought of. But maybe not. Any takers?</p>