title: How to compare two structypes in Scala and change datatype of columns in Scala?
body: <p>I am trying to move data from GP to HDFS using Scala &amp; Spark.</p>
  
  <pre><code>val execQuery    = "select * from schema.tablename"
  val yearDF       = spark.read.format("jdbc").option("url", connectionUrl).option("dbtable", s"(${execQuery}) as year2016").option("user", devUserName).option("password", devPassword).option("partitionColumn","header_id").option("lowerBound", 19919927).option("upperBound", 28684058).option("numPartitions",30).load()
  val yearDFSchema = yearDF.schema
  </code></pre>
  
  <p>The schema for yearDF is:</p>
  
  <pre><code>root
   |-- source_system_name: string (nullable = true)
   |-- table_refresh_delay_min: decimal(38,30) (nullable = true)
   |-- release_number: decimal(38,30) (nullable = true)
   |-- change_number: decimal(38,30) (nullable = true)
   |-- interface_queue_enabled_flag: string (nullable = true)
   |-- rework_enabled_flag: string (nullable = true)
   |-- fdm_application_id: decimal(15,0) (nullable = true)
   |-- history_enabled_flag: string (nullable = true)
  </code></pre>
  
  <p>The schema of same table on hive which is given by our project:</p>
  
  <pre><code>val hiveColumns = source_system_name:String|description:String|creation_date:Timestamp|status:String|status_date:Timestamp|table_refresh_delay_min:Timestamp|release_number:Double|change_number:Double|interface_queue_enabled_flag:String|rework_enabled_flag:String|fdm_application_id:Bigint|history_enabled_flag:String
  </code></pre>
  
  <p>So I took hiveColumns and created a new StructType as given below:</p>
  
  <pre><code>def convertDatatype(datatype: String): DataType = {
    val convert = datatype match {
      case "string"     =&gt; StringType
      case "bigint"     =&gt; LongType
      case "int"        =&gt; IntegerType
      case "double"     =&gt; DoubleType
      case "date"       =&gt; TimestampType
      case "boolean"    =&gt; BooleanType
      case "timestamp"  =&gt; TimestampType
    }
    convert
  }
  
  
  val schemaList = hiveColumns.split("\\|")
  val newSchema  = new StructType(schemaList.map(col =&gt; col.split(":")).map(e =&gt; StructField(e(0), convertDatatype(e(1)), true)))
  newSchema.printTreeString()
  </code></pre>
  
  
  
  <pre><code>root
   |-- source_system_name: string (nullable = true)
   |-- table_refresh_delay_min: double (nullable = true)
   |-- release_number: double (nullable = true)
   |-- change_number: double (nullable = true)
   |-- interface_queue_enabled_flag: string (nullable = true)
   |-- rework_enabled_flag: string (nullable = true)
   |-- fdm_application_id: long (nullable = true)
   |-- history_enabled_flag: string (nullable = true)
  </code></pre>
  
  <p>When I try to apply my new schema: schemaStructType on yearDF as below, I get the exception:</p>
  
  <pre><code> Caused by: java.lang.RuntimeException: java.math.BigDecimal is not a valid external type for schema of double
  </code></pre>
  
  <p>The exception occurs due to conversion of decimal to double. 
  What I don't understand is how can I convert the datatype of columns: <code>table_refresh_delay_min, release_number, change_number, fdm_application_id</code> in the StructType: <code>newSchema</code> from <code>DoubleType</code> to their corresponding datatypes present in yearDF's Schema. i.e.</p>
  
  <p>If the column in <code>yearDFSchema</code> has a decimal datatype with precision more than zero, in this case decimal(38,30), I need to convert the same column's datatype in <code>newSchema</code> to <code>DecimalType(38,30)</code></p>
  
  <p>Could anyone let me know how can I achieve it ?</p>