title: Generate a nested nested structure in pyspark
body: <p>I have the following DF:</p>
  <pre><code>+----+------+--------+----+----+----+----+----------+
  | ID | Name |  Vl1   | Vl2| Vl3| Vl4|Vl5 | Vl6      |
  +----+------+--------+----+----+----+----+----------+
  |1   |John  |    1.5 |null|null|null|   A|2022-01-01|
  |1   |John  |    1   |null|null|null|   A|2022-01-01|
  |1   |John  |    3   |null|1   |null|   A|2022-01-01|
  |1   |John  |    4   |null|1   |null|   A|2022-01-01|
  |2   |Ana   |    2.5 |null|null|null|   A|2022-01-01|
  |2   |Ana   |    0   |null|null|null|   A|2022-01-01|
  |2   |Ana   |    null|null|null|null|   A|2022-01-01|
  |2   |Ana   |    2   |null|null|null|   A|2022-01-01|
  |2   |Ana   |    2   |2   |null|null|   A|2022-01-01|
  |2   |Ana   |    1   |null|null|null|   A|2022-01-01|
  |3   |Paul  |    5   |null|null|null|   A|2022-01-01|
  |3   |Paul  |    null|2   |null|null|   A|2022-01-01|
  |3   |Paul  |    2.5 |null|2   |null|   A|2022-01-01|
  |3   |Paul  |    null|null|3   |null|   A|2022-01-01|
  +----+------+--------+----+----+----+----+----------+
  </code></pre>
  <p>How can I generate the following nested structure:</p>
  <pre><code> |-- Title: string (nullable = true)
   |-- Company: string (nullable = true)
   |-- Client: array (nullable = false)
   |    |-- element: struct (containsNull = false)
   |    |    |-- Vl1: double (nullable = true)
   |    |    |-- Vl2: double (nullable = true)
   |    |    |-- Prch: array (nullable = false)
   |    |    |    |-- element: struct (containsNull = false)
   |    |    |    |    |-- Vl3: string (nullable = true)
   |    |    |    |    |-- Detail: array (nullable = true)
   |    |    |    |    |    |-- element: struct (containsNull = false)
   |    |    |    |    |    |    |-- Vl4: date (nullable = true)
   |    |    |    |    |-- Bs: array (nullable = true)
   |    |    |    |    |    |-- element: struct (containsNull = false)
   |    |    |    |    |    |    |-- Vl5: string (nullable = true)
   |    |    |    |    |    |    |-- Vl6: date (nullable = true)
  </code></pre>
  <p>Until now, I used withcolumn to generate the columnns <code>Tiltle</code> and <code>Company</code>. Then I used <code>groupby</code> to group by <code>clients</code>  and then I used <code>collect_list</code> to generate the first level of the nested structure, but how can I generate the other levels of the strucuture (<code>Prch</code>, <code>Detail</code>, <code>Bs</code>).</p>
  <p>Just to know, my DF has 1 billion lines. I would like to know the best way to generate this structure. My spark environment has 4 workers with 5 cores by worker.</p>
  <p><strong>MVCE:</strong></p>
  <pre><code>  data = [
   (&quot;1&quot;,&quot;John&quot;,1.5,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;1&quot;,&quot;John&quot;,1.0,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;1&quot;,&quot;John&quot;,3.0,None,1.0,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;1&quot;,&quot;John&quot;,4.0,None,1.0,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;2&quot;,&quot;Ana&quot;,2.5,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;2&quot;,&quot;Ana&quot;,0.0,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;2&quot;,&quot;Ana&quot;,None,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;2&quot;,&quot;Ana&quot;,2.0,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;2&quot;,&quot;Ana&quot;,2.0,2.0,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;2&quot;,&quot;Ana&quot;,1.0,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;3&quot;,&quot;Paul&quot;,5.0,None,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;3&quot;,&quot;Paul&quot;,None,2.0,None,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;3&quot;,&quot;Paul&quot;,2.5,None,2.0,None,&quot;A&quot;, &quot;2022-01-01&quot;),
   (&quot;3&quot;,&quot;Paul&quot;,None,None,3.0,None,&quot;A&quot;, &quot;2022-01-01&quot;)
   ]
  
  schema = StructType([
      StructField(&quot;Id&quot;, StringType(),True),
      StructField(&quot;Name&quot;, StringType(),True),
      StructField(&quot;Vl1&quot;, DoubleType(),True),
      StructField(&quot;Vl2&quot;, DoubleType(), True),
      StructField(&quot;Vl3&quot;, DoubleType(), True),
      StructField(&quot;Vl4&quot;, DateType(), True),
      StructField(&quot;Vl5&quot;, StringType(), True),
      StructField(&quot;Vl6&quot;, StringType(), True)
    ])
  
  df = spark.createDataFrame(data=data,schema=schema)
  df.printSchema()
  df.show()
  </code></pre>