title: pyspark join with null conditions
body: <p>I am trying to join two pyspark dataframes as below based on &quot;Year&quot; and &quot;invoice&quot; columns. But if &quot;Year&quot; is missing in df1, then I need to join just based on &quot;&quot;invoice&quot; alone</p>
  <p>df1:</p>
  <pre><code>Year    invoice    Status   Item
  2020    262        YES      bag
  2019    252        YES      ball
  2018    240        YES      pen
  2017    228        YES      ink
  2016    216        NO       headphone
  2015    213        NO       bicycle
          198        NO       ribbon
          175        YES      phone
      
  </code></pre>
  <p>df2:</p>
  <pre><code>Year    invoice
  2020    262
  2016    216
  2014    175
  2013    198
  2019    252
  </code></pre>
  <p>Expected output:</p>
  <pre><code>Year    invoice    Status   Item
  2020    262        YES      bag
  2016    216        NO       headphone
  2014    175        YES      phone
  2013    198        NO       ribbon
  2019    252        YES      ball
  </code></pre>
  <p>I am able to join df1 and df2 as below (only based on Year and invoice&quot; column. If year is missing in df1, I need to add the logic of joining two columns based on invoice alone.</p>
  <pre><code>df_results = df1.join(df2, on=['Year', 'invoice'], how='left') \
                  .drop(df2.Year) \
                  .drop(df2.invoice)
  </code></pre>
  <p>Please let me know how to join if &quot;Year&quot; is not available in the df1, and dataframes should be joined based on &quot;invoice&quot; alone. Thanks.</p>