title: How to run pySpark code on multiple workers in AWS glue
body: <p>I have a Glue Table. The table has been created by crawling an S3 bucket that contains parquet files. The data is partitioned by year/month/day.</p>
  <p>I fetch the Dynamic Frame of a specific partition first. That’s easy.</p>
  <pre><code>datasource0 = glueContext.create_dynamic_frame.from_catalog(database = &quot;testdata&quot;, table_name = &quot;sampletable&quot;, transformation_ctx = &quot;datasource0&quot;,push_down_predicate = &quot;(year == '2019' and month == '08' and day == '02')&quot;)
  </code></pre>
  <p>I want to find the max value of a specific column. That’s straightforward as well:</p>
  <pre><code>max = datasource0.toDF().agg({“my-col”:”max”}).first()[0]
  </code></pre>
  <p>When I launched this job, I selected 10 workers (the default).</p>
  <ul>
  <li><p>Is the <code>create_dynamic_frame</code> being processed on 10 worker?</p>
  </li>
  <li><p>Is the <code>agg</code> function being processed on 10 workers?</p>
  </li>
  </ul>
  <p>If the answer is yes, how can I get this intuition? If the answer is no, then, how can I make it so?</p>
  <p>My objective is to speed up the execution of the above code, which runs today in 5 min.</p>