title: Apache Spark SQL can not SELECT Cassandra timestamp columns
body: <p>I created Docker containers in which I installed Apache Spark 3.1.2 (Hadoop 3.2) that host a ThriftServer which is configured to access Cassandra via the <a href="https://github.com/datastax/spark-cassandra-connector" rel="nofollow noreferrer">spark-cassandra-connector(3.1.0)</a>. Each of these services is running in it's own container. So I got 5 containers up (1x spark master, 2x spark worker, 1x spark thriftserver, 1x cassandra) which are configured to live in the same network via docker-compose.
  I use the beeline client from Apache Hive(1.2.1) to query the database. Everything works fine, except for querying a field in Cassandra with the type <code>timestamp</code>.</p>
  <pre><code>org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 0.0 failed 4 times, most recent failure: Lost task 9.3 in stage 0.0 (TID 53) (192.168.80.5 executor 0): java.lang.ClassCastException: java.sql.Timestamp cannot be cast to java.time.Instant
  </code></pre>
  <p>I checked the Spark/spark-cassandra-connector documentations but didn't find much except for a <a href="https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration" rel="nofollow noreferrer">configuration property called spark.sql.datetime.java8API.enabled</a> which</p>
  <pre><code>If the configuration property is set to true, java.time.Instant and java.time.LocalDate classes of Java 8 API are used as external types for Catalyst's TimestampType and DateType. If it is set to false, java.sql.Timestamp and java.sql.Date are used for the same purpose.
  </code></pre>
  <p>I think this property could maybe help in my case. Despite saying in the docs that the default value is false, the value is always <code>true</code> in my case. I don't set it anywhere and I've tried overwriting it with <code>false</code> in the <code>$SPARK_HOME/conf/spark-defaults.conf</code> file and via the <code>--conf</code> commandline parameters when starting up ThriftServer (and master/worker instances), but the environment tab (at <code>localhost:4040</code>) always shows it as <code>true</code>.</p>
  <p>Is there a way to make Spark convert the timestamps in a way that doesn't lead to an exception? It would be important to do that in SQL since I want to connect software for data visualization later on.</p>