title: How to query/extract array elements from within a pyspark dataframe
body: <p>I am trying to build a data frame that will incorporate data in an array.   The array is a 2 value array with a index and value pair.   Not every index value is present for every row in the data frame:array.   Here is what the schema looks like</p>
  
  <pre><code>root
  |-- visitNumber: string (nullable = true)
  |-- visitId: string (nullable = true)
  |-- customDimensions: array (nullable = true)
  |    |-- element: struct (containsNull = true)
  |    |    |-- index: string (nullable = true)
  |    |    |-- value: string (nullable = true)
  </code></pre>
  
  <p>There are many other columns, but those are not involved in the question.</p>
  
  <p>Here is an example of the customDimensions array in terms of data:</p>
  
  <pre><code>[[1, ],[2, ],[3,"apple"],[6,"1-111-32"],[42, ],[5, ]]
  </code></pre>
  
  <p>What I am trying to accomplish is to incorporate columns that hold the value for a particular index.  For instance:</p>
  
  <pre><code>df = df.withColumn("index6", *stuff to get the value at index 6*)
  </code></pre>
  
  <p>This would be a repeatable iteration as there is data throughout the 'customDimensions' that holds required data that we can "flatten" and express as separate columns.</p>