title: pyspark | transforming list of numpy arrays into columns in dataframe
body: <p>I am trying to take an rdd that looks like:</p>
  
  <p>[&lt;1x24000 sparse matrix of type ''
          with 10 stored elements in Compressed Sparse Row format>, . . . ]</p>
  
  <p>and ideally turn it into a dataframe that looks like:</p>
  
  <pre><code>&lt;code&gt;
     +-----------------+
     |  A  |  B  |   C |
     +-----------------+
     | 1.0 | 0.0 | 0.0 |
     +-----+-----+-----+
     | 1.0 | 1.0 | 0.0 |
     +-----+-----+-----+
  &lt;/code&gt;
  </code></pre>
  
  <p>However, I keep getting this:</p>
  
  <pre><code>&lt;code&gt;
  +---------------+
  |             _1|
  +---------------+
  |[1.0, 0.0, 0.0]|
  +---------------+
  |[1.0, 1.0, 0.0]|
  +---------------+
  &lt;/code&gt;
  </code></pre>
  
  <p>I am having the darnedest time because each row is filled with numpy arrays. </p>
  
  <p>I used this code to create the dataframe from the rdd: </p>
  
  <pre><code>&lt;code&gt;res.flatMap(lambda x: np.array(x.todense())).map(list).map(lambda l : Row([float(x) for x in l])).toDF()&lt;/code&gt;
  </code></pre>
  
  <p>**Explode does not help (it puts everything into the same column)</p>
  
  <p>** I tried using a UDF on the resulting dataframe but I cannot seem to separate the numpy array into individual values. </p>
  
  <p>Please help!</p>