title: Running AWS Glue ETL Job (Spark) for large data
body: <p>Currently, I have a GLUE ETL Script in Scala.</p>
  <p>Following are my GLUE script settings:</p>
  <ul>
  <li>Spark 2.4, Scala 2 (Glue Version 2.0)</li>
  <li>Worker type : G1.X (Recommended for memory intensive job)</li>
  <li>Number of workers : 10</li>
  </ul>
  <p>I am reading 60 GB data in the database that I am reading in the dataframe like this</p>
  <pre class="lang-scala prettyprint-override"><code>val largeDF = glueContext.getCatalogSource(database = &quot;&quot;, tableName = &quot;&quot;).getDynamicFrame().toDF()
  
  val smallDF = glueContext.getCatalogSource(database = &quot;&quot;, tableName = &quot;&quot;).getDynamicFrame().toDF()
  
  val result = largeDF.join(broadcast(smallDF) , smalldf(&quot;col&quot;) === largeDF(&quot;col&quot;), &quot;leftsemi&quot;)
  
  result.show(false)
  </code></pre>
  <p>However this is running for 6 hours, and fails with the following error</p>
  <pre class="lang-sh prettyprint-override"><code>Exception in task 0.0 in stage 3.0 (TID 77)
  java.io.IOException: No space left on device
  </code></pre>
  <p>Do I need to increase the number of workers ? What's the best way to calculate ideal setting for reading large data in AWS Glue?</p>