title: How to seperate CSV values with in a cell in pyspark dataframe to new columns and their values respectively
body: <p>The current spark data frame has CSV values in cell level of one column, I trying to explode it to new columns. The example dataframe</p>
  <pre><code>    a_id                                    features
  1   2020     &quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant1&quot;,&quot;1&quot;,&quot;0.1&quot;,&quot;aa&quot;
  2   2021     &quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant2&quot;,&quot;1&quot;,&quot;0.2&quot;,&quot;ab&quot;
  3   2022     &quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant3&quot;,&quot;1&quot;,&quot;0.3&quot;,&quot;ac&quot;,&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant3&quot;,&quot;1.1&quot;,&quot;3.3&quot;,&quot;acx&quot;
  4   2023     &quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant4&quot;,&quot;1&quot;,&quot;0.4&quot;,&quot;ad&quot;
  5   2024     &quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant5&quot;,&quot;1&quot;,&quot;0.5&quot;,&quot;ae&quot;,&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant5&quot;,&quot;1.2&quot;,&quot;6.3&quot;,&quot;xwy&quot;,&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant5&quot;,&quot;2.2&quot;,&quot;8.3&quot;,&quot;bunr&quot;
  6   2025     &quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;constant6&quot;,&quot;1&quot;,&quot;0.6&quot;,&quot;af&quot;
  </code></pre>
  <p>The features column has multiple csv values, and in it (a, b, c, d) act as header and they get repeated in some cells (in row 3 and 5), I want to extract only one header and its respective values. The output of the expected dataframe is as shown</p>
  <p><strong>Output spark dataframe</strong></p>
  <pre><code>    a_id       a        d
  1   2020   constant1   [&quot;aa&quot;]
  2   2021   constant2   [&quot;ab&quot;]
  3   2022   constant3   [&quot;ac&quot;,&quot;acx&quot;]
  4   2023   constant4   [&quot;ad&quot;]
  5   2024   constant5   [&quot;ae&quot;,&quot;xwy&quot;,&quot;bunr&quot;]
  6   2025   constant6   [&quot;af&quot;]
  </code></pre>
  <p>As shown, I would like to extract only a and d headers as new columns, where a is constant and d has multiple values, where its values made as list.</p>
  <p>Please help how to convert this in pysaprk. The above dataframe is streaming dataframe in real time.</p>