title: Is there a better way to write this SparkSQL statement?
body: <p>What are the best ways to write SparkSQL statements?</p>
  
  <p>Any suggestions on such statements:</p>
  
  <pre><code>import pyspark.sql.functions as psf
  
  df_final = df_01\
              .select(df_01['*'],
                      psf.when(df_01.record_two&gt;\
                               df_01.record_three,
                               df_01.record_two)
                         .otherwise(df_01.record_three)\
                         .alias("some_new_alias"))\
              .where("some_field &gt; 1000")\
              .where((df_01.record_one.isin(df_red01.record_one)==False) |\
                     (df_01.record_one.isin(df_blue01.record_one)==False) |\
                     (df_01.record_one.isin(df_violet01.record_one)==False) |\
                     (df_01.record_one.isin(df_green01.record_one)==False) |\
                     (df_01.record_one.isin(df_black01.record_one)==False) |\
                     (df_01.record_one.isin(df_white01.record_one)==False) |\
                     (df_01.record_one.isin(df_red02.record_one)==False) |\
                     (df_01.record_one.isin(df_blue02.record_one)==False) |\
                     (df_01.record_one.isin(df_violet02.record_one)==False) |\
                     (df_01.record_one.isin(df_green02.record_one)==False) |\
                     (df_01.record_one.isin(df_black02.record_one)==False) |\
                     (df_01.record_one.isin(df_white02.record_one)==False) |\
                     (df_01.record_one.isin(df_blue03.record_one)==False) |\
                     (df_01.record_one.isin(df_violet03.record_one)==False) |\
                     (df_01.record_one.isin(df_green03.record_one)==False) |\
                     (df_01.record_one.isin(df_black03.record_one)==False) |\
                     (df_01.record_one.isin(df_violet04.record_one)==False) |\
                     (df_01.record_one.isin(df_green04.record_one)==False) |\
                     (df_01.record_one.isin(df_violet04.record_one)==False))\
              .select("record_one", "some_new_alias")
  
  
  df_another_test_frame = df_jibber01\
                           .select(df_jibber01.field01,
                                   df_jibber01.field02,
                                   df_jibber01.field03,
                                   df_jibber01.field04,
                                   df_jibber01.field05,
                                   df_jibber01.field06,
                                   df_jibber01.field07,
                                   df_jibber01.field08,
                                   df_jibber01.field09,
                                   psf.when(df_jibber01.field04 &lt;= 100,
                                            psf.round(2000*df_jibber01.field10/59, 10))\
                                      .when(df_jibber01.field05 &gt;= 1,
                                            psf.round(2000*df_jibber01.field10/59, 10))                                     
                                      .when(df_jibber01.field06 &gt;= 2,
                                            psf.round(2000*df_jibber01.field10/59, 10))                                     
                                      .when(df_jibber01.field04 &lt;= 3,
                                            psf.round(20*df_jibber01.field10/59, 10))                                     
                                      .when(df_jibber01.field05 &gt;= 4,
                                            psf.round(20*df_jibber01.field10/59, 10))       
                                      .when(df_jibber01.field06 &gt;= 5,
                                            psf.round(20*df_jibber01.field10/59, 10))
                                      .when(df_jibber01.field04 &lt;= 6,
                                            psf.round(9999*df_jibber01.field10/59, 10))
                                      .when(df_jibber01.field05 &gt;= 7,
                                            psf.round(9999*df_jibber01.field10/59, 10))
                                      .when(df_jibber01.field06 &gt;= 8,
                                            psf.round(9999*df_jibber01.field10/59, 10))                                     
                                      .otherwise(psf.round(9999*df_jibber01.field10/59, 10))\
                                      .alias("field11")
                                  )
  </code></pre>
  
  <p>There are multiple "where" and "when" conditions. Is there a cleaner way to write? I have 100s of such statements.</p>
  
  <p>Any suggestion would be helpful.</p>