title: pyspark take a long time to execute list of query
body: <p>I work with pyspark in standalone. My dataframe that I want enriched contains 440,000 rows. I have a constant &quot;_temp_ok_calculer&quot; column with a &quot;ko&quot; value. I want to update this column according to a filter list.
  my filter list is retrieved from another file. the list contains 36 items. The 10 top elemnts:</p>
  <pre><code>[None, None, '`Operator level 3` like &quot;POI&quot; AND `Auto Cleaning criteria` not like&quot;POI&quot; and `Auto Cleaning criteria` not like &quot;TRY&quot;  AND `Major cost` = null', '`Operator level 3` &lt;&gt; &quot;UP&quot; AND `Operator level 3` &lt;&gt; &quot;LL&quot; AND `Operator level 3` &lt;&gt; &quot;RTY&quot; AND `Auto Cleaning criteria` &lt;&gt;&quot;uy&quot; AND `Auto Cleaning criteria` &lt;&gt; &quot;rr&quot;  AND `Major cost` = null', None, '`Operator level 2` =&quot;RRR&quot; OR  `Operator level 2`= &quot;AAA&quot; AND `Auto Cleaning criteria` !=&quot;IO&quot; AND `Auto Cleaning criteria` &lt;&gt; &quot;TY&quot;  AND `Workstation Type` = &quot;A&quot;', '`Operator level 2`  =&quot;UU&quot; OR  `Operator level 2` = &quot;TYU&quot; AND `Auto Cleaning criteria`  != &quot;UI&quot; AND `Auto Cleaning criteria`  != &quot;TY&quot; AND `Workstation Type` = &quot;YH A&quot;', '`Operator level 2` =&quot;YT&quot; OR  `Operator level 2`= &quot;QE&quot; AND `Auto Cleaning criteria` != &quot;TY&quot; AND `Auto Cleaning criteria` != &quot;TT&quot; AND `Workstation Type` = &quot;BB&quot;', '`Operator level 2` =&quot;UYT&quot; OR  `Operator level 2`= &quot;ORE&quot; AND `Auto Cleaning criteria` != &quot;SSE&quot; AND `Auto Cleaning criteria` != &quot;TT&quot; AND `Workstation Type` = &quot;A&quot;', '`Operator level 2` =&quot;AR&quot; OR  `Operator level 2`= &quot;DA&quot; AND `Auto Cleaning criteria` != &quot;TS&quot; AND `Auto Cleaning criteria` != &quot;TT&quot; AND `Workstation Type` = &quot;AA&quot;']
  </code></pre>
  <p>I use this code to update the column &quot;_temp_ok_calculer&quot; according to the filter it works but it is too slow. it takes 45 minutes to complete.
  How can I optimize the execution time please?</p>
  <pre><code>def valo_filtre_sql(self, df_valo, filtre):
      #df_valo: my dataframe that i want to update
      #filtre: my sql list of filtre
      #Function which allows to define the lines which must be assigned according to the applied filters
      #apply valuation when the value of temp_ok_filtre is equal to 'ok'
      i = 0
      filtre_no_none=list(filter(None, filtre))
      filtre_none=list(filter(lambda x:x==None, filtre))
      if len(list(filter(None, filtre_no_none)))&gt;0 : 
          while i &lt; len(filtre_no_none):
              # _temp_ok_calculer = 'no' is the indicator which allows you to check if the condition of the &quot;Filters on Drivers databases&quot; column
              #is accomplished. if the result is ok, valuation is possible. 
              request = &quot;CASE WHEN _temp_ok_calculer='non' AND &quot; + filtre_no_none[i] + &quot; THEN 'ok' ELSE CASE WHEN _temp_ok_calculer='ok' THEN 'ok' ELSE 'ko' END END &quot;
              sqlDF = df_valo.withColumn(&quot;_temp_ok_calculer&quot;, F.expr(request))
              df_valo= sqlDF
              df_valo=df_valo.when(F.col(&quot;_temp_ok_calculer&quot;)==&quot;non&quot; &amp; filtre_no_none[i],F.lit(&quot;ok&quot;))
              i = i + 1 
      else : 
          while i &lt; len(filtre_none):
              # _temp_ok_calculer = 'no' is the indicator which allows you to check if the condition of the &quot;Filters on Drivers databases&quot; column
              #is accomplished. if the result is ok, valuation is possible.
              request = &quot;CASE WHEN _temp_ok_calculer='non'&quot;  + &quot; THEN 'ok' ELSE CASE WHEN _temp_ok_calculer='ok' THEN 'ok' ELSE 'ko' END END &quot;
              sqlDF = df_valo.withColumn(&quot;_temp_ok_calculer&quot;, F.expr(request))
              df_valo= sqlDF
              i = i + 1 
      return df_valo
  </code></pre>
  <p>I try to change the config spark but it doesn't work.</p>
  <pre><code>conf = pyspark.SparkConf().setAll([('spark.executor.memory', '14g'), ('spark.executor.cores', '7'), ('spark.cores.max', '8'), ('spark.driver.memory','2g'),(&quot;spark.sql.crossJoin.enabled&quot; , &quot;true&quot; ),(&quot;spark.jars.packages&quot;, &quot;com.crealytics:spark-excel_2.12:0.13.5&quot;),('spark.sql.codegen.wholeStage', 'false'),(&quot;spark.network.timeout&quot;, &quot;600s&quot;)])
  spark = SparkSession.builder.config(conf=conf).getOrCreate()
  </code></pre>