title: How can I get my Luigi scheduler to utilize multiple cores with the parallel-scheduling flag?
body: <p>I have the following line in my <code>luigi.cfg</code> file (on all nodes, scheduler and workers):</p>
  
  <pre><code>[core]
  parallel-scheduling: true
  </code></pre>
  
  <p>However, when I monitor CPU utilization on my luigi scheduler (with a graph of around ~4000 tasks, handling requests from ~100 workers), it is only utilizing a single core on the scheduler, with the single <code>luigid</code> thread often hitting 100% CPU utilization. My understanding is that this configuration variable should parallelize scheduling of tasks. </p>
  
  <p>The source suggests that this flag should indeed use multiple cores on the scheduler. In <a href="https://github.com/spotify/luigi/blob/master/luigi/interface.py#L194" rel="noreferrer">https://github.com/spotify/luigi/blob/master/luigi/interface.py#L194</a>, a call is made to <a href="https://github.com/spotify/luigi/blob/master/luigi/worker.py#L498" rel="noreferrer">https://github.com/spotify/luigi/blob/master/luigi/worker.py#L498</a> to check the <code>.complete()</code> state of the task in parallel.</p>
  
  <p>What am I missing to get my Luigi scheduler to utilize all of its cores?</p>