title: Adding Columns to a pyspark dataframe based on the length of an array in another column
body: <p>I have some JSON data that I'm loading into a pyspark dataframe that originally looks something like this:</p>
  <pre><code>`{&quot;timestamp&quot;: &quot;2022-07-02T23:59:22.393458Z&quot;, &quot;version&quot;: 2, &quot;payload&quot;: {&quot;links&quot;: [{&quot;down&quot; : {&quot;db&quot;:&quot;0:0&quot;, &quot;maker&quot;:&quot;gg&quot;, &quot;dev&quot;:&quot;51&quot;}, &quot;up&quot; : {&quot;db&quot;:&quot;0:1&quot;, &quot;maker&quot;:&quot;03&quot;, &quot;dev&quot;:&quot;52&quot;}, &quot;max_w&quot; : 3, &quot;max_s&quot; : 3, &quot;curr_w&quot; : 8, &quot;curr_s&quot; : 2},{&quot;down&quot; : {&quot;db&quot;:&quot;0:0&quot;, &quot;maker&quot;:&quot;tr&quot;, &quot;dev&quot;:&quot;20&quot;}, &quot;up&quot; : {&quot;db&quot;:&quot;0:2:1&quot;, &quot;maker&quot;:&quot;pr&quot;, &quot;dev&quot;:&quot;1022&quot;}, &quot;max_w&quot; : 8, &quot;max_s&quot; : 2, &quot;curr_w&quot; : 7, &quot;curr_s&quot; : 4}]}}}`
  </code></pre>
  <p>The file I am ingesting into my dataframe has multiple JSONs in a single file like the above. The array that is contained in payload.links varies in size within the json statements. I'm needing to get the data from the arrays into columns to make the dataframe look like:</p>
  <p>timestamp | version | payload_links_0_down_db | payload_links_0_down_maker | payload_links_0_down_dev | payload_links_0_up_db | payload_links_0_up_maker | payload_links_0_up_dev | payload_links_0_max_w | payload_links_0_max_s | payload_links_0_curr_w | payload_links_0_curr_s | payload_links_1_down_db | payload_links_1_down_maker | payload_links_1_down_dev | payload_links_1_up_db | payload_links_1_up_maker | payload_links_1_up_dev | payload_links_1_max_w | payload_links_1_max_s | payload_links_1_curr_w | payload_links_1_curr_s</p>
  <p>etc...</p>
  <p>I understand how to bring the data into a dataframe using <code>spark_df = spark.read.option(&quot;multiline&quot;, &quot;true&quot;).json(source_s3_path)</code></p>
  <p>and I also know how to access each element of the array like this: <code>spark_df = spark_df.withColumn(&quot;max_width&quot;, spark_df.payload.links[0].max_w)</code></p>
  <p>But due to the array size changing from json to json, I'm struggling with how to create the correct number of columns in the dataframe as well as handling populating the columns without throwing index out of bounds errors if for instance there is a max array length of 20, but the data also includes arrays of length 3.</p>
  <p>How can I get the data down to the final schema that I need?</p>