title: Get element of Struct Type for a row in scala using any function
body: <p>I am doing some calculations on row level in Scala/Spark. I have a dataframe created using JSON below-</p>
  
  <pre><code>{"available":false,"createTime":"2016-01-08","dataValue":{"names_source":{"first_names":["abc", "def"],"last_names_id":[123,456]},"another_source_array":[{"first":"1.1","last":"ONE"}],"another_source":"TableSources","location":"GMP", "timestamp":"2018-02-11"},"deleteTime":"2016-01-08"}
  </code></pre>
  
  <p>You can create dataframe using this JSON directly. My schema looks like below-</p>
  
  <pre><code>root
   |-- available: boolean (nullable = true)
   |-- createTime: string (nullable = true)
   |-- dataValue: struct (nullable = true)
   |    |-- another_source: string (nullable = true)
   |    |-- another_source_array: array (nullable = true)
   |    |    |-- element: struct (containsNull = true)
   |    |    |    |-- first: string (nullable = true)
   |    |    |    |-- last: string (nullable = true)
   |    |-- location: string (nullable = true)
   |    |-- names_source: struct (nullable = true)
   |    |    |-- first_names: array (nullable = true)
   |    |    |    |-- element: string (containsNull = true)
   |    |    |-- last_names_id: array (nullable = true)
   |    |    |    |-- element: long (containsNull = true)
   |    |-- timestamp: string (nullable = true)
   |-- deleteTime: string (nullable = true)
  </code></pre>
  
  <p>I am reading all columns separately with readSchema and writing with writeSchema. Out of two complex columns, I am able to process one but not other.</p>
  
  <p>Below is a part of my read schema-</p>
  
  <pre><code>  .add("names_source", StructType(
        StructField("first_names", ArrayType.apply(StringType)) ::
          StructField("last_names_id", ArrayType.apply(DoubleType)) ::
          Nil
  ))
  .add("another_source_array", ArrayType(StructType(
        StructField("first", StringType) ::
          StructField("last", StringType) ::
          Nil
  )))
  </code></pre>
  
  <p>Here is a part of my write schema-</p>
  
  <pre><code>.add("names_source", StructType.apply(Seq(
             StructField("first_names", StringType),
              StructField("last_names_id", DoubleType))
    ))
    .add("another_source_array", ArrayType(StructType.apply(Seq(
             StructField("first", StringType),
              StructField("last", StringType))
     )))
  </code></pre>
  
  <p>In processing, I am using a method to index all columns. below is my piece of code for the function-</p>
  
  <pre><code>   def myMapRedFunction(df: DataFrame, spark: SparkSession): DataFrame = {
  
      val columnIndex = dataIndexingSchema.fieldNames.zipWithIndex.toMap
  
      val myRDD = df.rdd
        .map(row =&gt; {
        Row(
          row.getAs[Boolean](columnIndex("available")),
          parseDate(row.getAs[String](columnIndex("create_time"))),
          ??I Need help here??
          row.getAs[String](columnIndex("another_source")),
          anotherSourceArrayFunction(row.getSeq[Row](columnIndex("another_source_array"))),
          row.getAs[String](columnIndex("location")),
          row.getAs[String](columnIndex("timestamp")),
          parseDate(row.getAs[String](columnIndex("delete_time")))
        )
      }).distinct
  
      spark.createDataFrame(myRDD, dataWriteSchema)
    }
  </code></pre>
  
  <p><code>another_source_array</code> column is being processed by <code>anotherSourceArrayFunction</code> method to make sure we get schema as per the requirements. I need a similar function to get <code>names_source</code> column. Below is the function that I am using for <code>another_source_array</code> column.</p>
  
  <pre><code>    def anotherSourceArrayFunction(data: Seq[Row]): Seq[Row] = {
      if (data == null) {
        data
      } else {
        data.map(r =&gt; {
          val first = r.getAs[String]("first").ToUpperCase()
          val last = r.getAs[String]("last")
          new GenericRowWithSchema(Array(first,last), StructType(
            StructField("first", StringType) ::
              StructField("last", StringType) ::
              Nil
          ))
        })
      }
    }
  </code></pre>
  
  <p>Probably in short, I need something like <a href="https://stackoverflow.com/questions/40526054/get-elements-of-type-structure-of-row-by-name-in-spark-scala">this</a>, where I can get my <code>names_source</code> column structure as a struct.</p>
  
  <pre><code>names_source:struct&lt;first_names:array&lt;string&gt;,last_names_id:array&lt;bigint&gt;&gt;
  another_source_array:array&lt;struct&lt;first:string,last:string&gt;&gt;
  </code></pre>
  
  <p>Above are the column schema required finally. I am able to get <code>another_source_array</code>  properly and need help in <code>names_source</code>. I think my write schema for this column is correct but I am not sure. But I need finally <code>names_source:struct&lt;first_names:array&lt;string&gt;,last_names_id:array&lt;bigint&gt;&gt;</code> as column schema. </p>
  
  <p>Note: I am able to get <code>another_source_array</code> column carefully without any problem. I kept here that function to make it better understanding. </p>