title: Jobs running on driver node databrick
body: <p>I have notebook that runs a machine learning jobs in databrick, i'm using dbutils to accept variables and pass that to notebook.</p>
  <p>I have created another notebook as parent and pass variables via this notebook and run multiple notebook with.</p>
  <pre><code>ThreadPoolExecutor, ProcessPoolExecutor
  def processAnIntegerNumber(id):
  
  dbutils.notebook.run(path = &quot;/Users/child_notebook&quot;,
                                      timeout_seconds = 3600,
                                      arguments = {&quot;id&quot;:id})
  </code></pre>
  <p>this will create multiple jobs, the problem that I have is all these 10-30 jobs that I pass as variable runs in Driver nodes and does not use worker nodes and as result it's extremely slow.</p>
  <p>anyway to run python notebook in parallel, without using scala?</p>
  <p>regards</p>