title: Is casting necessary for string columns which contain Int value for doing arithmetic comparison?
body: <p>I am using Apache Spark 2.1.1. </p>
  
  <p>I have a Dataset  as follows.</p>
  
  <pre><code>final case class TestModel(id: String,
                         code: String, 
                         measure: String, 
                         value: String) 
  </code></pre>
  
  <p>I am loading it properly from a csv file. For different <code>measure</code> values, the <code>value</code> datatype could be different. E.g. if measure is 'Age', then <code>value</code> would be age in string. Now to do some processing, I am casting the <code>value</code> to IntegerType and then doing the comparison with the age range specified in another dataset.
  Is the following correct way to do it ?</p>
  
  <pre><code>val testData = spark.read.schema(testSchema).option("header", "false").csv(dataPath).as[TestModel]
  val ageBasedTestData = testData.filter($"measure" === "Age")
  
  
  var ageBasedData = ageBasedTestData.join(anotherDS, ageBasedTestData("code") === anotherDS("code") &amp;&amp;
                                 anotherDS("ages").getItem(0) &lt;= ageBasedTestData("value").cast(IntegerType) &amp;&amp;
                                 anotherDS("ages").getItem(1) &gt; ageBasedTestData("value").cast(IntegerType))
                                 .select( some column names)
  </code></pre>
  
  <p>Is the above casting of <code>value</code> column to Interger type before comparison with age range correct way of doing this? Basically I am converting the string to Int just for the comparison purposes then don't care about the datatype. I have ran the code both with the cast and without the cast and both give me same results. So I am not sure what is happening behind the scenes without the cast. Does it automatically cast the string to Int and do the comparison. If it matters, the datatype of "ages" array from anotherDS dataset is Integer.  </p>