title: Timezone problem with spark.sql time functions
body: <p>i'm writing some code on a jupyter notebook using spark 2.4.7 and pyspark running in Standalone Mode.<br />
  I need to convert some timestamps to unix time to do some operations however i noticed a strange behavior,following the code i'm running:</p>
  <pre><code>import pyspark
  from pyspark.sql import SparkSession
  import pyspark.sql.functions as F
  from datetime import datetime, timedelta, date
  
  spark = SparkSession.builder \
          .appName(&quot;test&quot;) \
          .master(n_spark_master)\
          .config(&quot;spark.total.executor.cores&quot;,n_spark_cores_max)\
          .config(&quot;spark.cores.max&quot;, n_spark_cores_max)\
          .config(&quot;spark.executor.memory&quot;,n_spark_executor_memory)\
          .config(&quot;spark.executor.cores&quot;,n_spark_executor_cores)\
          .enableHiveSupport() \
          .getOrCreate()
  
  print(datetime.now().astimezone().tzinfo)
  
  df = spark.createDataFrame([
      (1, &quot;a&quot;),
      (2, &quot;b&quot;),
      (3, &quot;c&quot;), ], [&quot;dummy1&quot;, &quot;dummy2&quot;])
  
  epoch = datetime.utcfromtimestamp(0) df=df.withColumn('epoch',lit(epoch))  
  timeFmt = '%Y-%m-%dT%H:%M:%S'  
  df= df.withColumn('unix_time_epoch',F.unix_timestamp('epoch', format=timeFmt)) df.show()
  </code></pre>
  <p>output:</p>
  <pre><code>CET
  +------+------+-------------------+---------------+
  |dummy1|dummy2|              epoch|unix_time_epoch|
  +------+------+-------------------+---------------+
  |     1|     a|1970-01-01 00:00:00|          -3600|
  |     2|     b|1970-01-01 00:00:00|          -3600|
  |     3|     c|1970-01-01 00:00:00|          -3600|
  +------+------+-------------------+---------------+
  </code></pre>
  <p>from documentation of spark 2.4.7:</p>
  <blockquote>
  <p>pyspark.sql.functions.unix_timestamp(timestamp=None, format='yyyy-MM-dd HH:mm:ss')[source]<br />
  Convert time string with given pattern (‘yyyy-MM-dd HH:mm:ss’, by default) to Unix time stamp (in seconds), using <strong>the default timezone and the default locale</strong>, return null if fail.</p>
  </blockquote>
  <p>The previous command <code>print(datetime.now().astimezone().tzinfo)</code> which outputs <code>CET</code> should give me my local timezone, which is indeed the correct one on the machine since i'm in UTC+1.<br />
  On the UI of Spark also i can clearly see <code>user.timezone=Europe/Rome</code>.<br />
  Still it looks like spark is trying to convert from UTC+1 to UTC and therefore i get in the output <code>unix_time_epoch = -3600</code>, which i instead would expect it to be <code>unix_time_epoch = 0</code>.<br />
  I've tried to change to UTC as other thread suggest like this:</p>
  <pre><code>import pyspark
  from pyspark.sql import SparkSession
  import pyspark.sql.functions as F
  from datetime import datetime, timedelta, date
  import time
  
  os.environ['TZ'] = 'Europe/London'
  time.tzset()
  
  spark = SparkSession.builder \
          .appName(&quot;test&quot;) \
          .master(n_spark_master)\
          .config(&quot;spark.total.executor.cores&quot;,n_spark_cores_max)\
          .config(&quot;spark.cores.max&quot;, n_spark_cores_max)\
          .config(&quot;spark.executor.memory&quot;,n_spark_executor_memory)\
          .config(&quot;spark.executor.cores&quot;,n_spark_executor_cores)\
          .config('spark.driver.extraJavaOptions', '-Duser.timezone=UTC') \
          .config('spark.executor.extraJavaOptions', '-Duser.timezone=UTC') \
          .config('spark.sql.session.timeZone', 'UTC') \
          .enableHiveSupport() \
          .getOrCreate()
  
  print(datetime.now().astimezone().tzinfo)
  
  df = spark.createDataFrame([
      (1, &quot;a&quot;),
      (2, &quot;b&quot;),
      (3, &quot;c&quot;),
  ], [&quot;dummy1&quot;, &quot;dummy2&quot;])
  
  epoch = datetime.utcfromtimestamp(0)
  df=df.withColumn('epoch',lit(epoch))
  timeFmt = '%Y-%m-%dT%H:%M:%S'
  df = df.withColumn('unix_time_epoch',F.unix_timestamp('epoch', format=timeFmt))
  df.show()
  </code></pre>
  <p>but the output is:</p>
  <pre><code>GMT
  +------+------+-------------------+---------------+
  |dummy1|dummy2|              epoch|unix_time_epoch|
  +------+------+-------------------+---------------+
  |     1|     a|1969-12-31 23:00:00|          -3600|
  |     2|     b|1969-12-31 23:00:00|          -3600|
  |     3|     c|1969-12-31 23:00:00|          -3600|
  +------+------+-------------------+---------------+
  </code></pre>
  <p>What i want to achieve is to evalueate everything in UTC regardless with not timezone offset since in Rome where i'm located local timezone shifts during the year between UTC+1 and UTC+2, expected output should be like this:</p>
  <pre><code>+------+------+-------------------+---------------+
  |dummy1|dummy2|              epoch|unix_time_epoch|
  +------+------+-------------------+---------------+
  |     1|     a|1970-01-01 00:00:00|              0|
  |     2|     b|1970-01-01 00:00:00|              0|
  |     3|     c|1970-01-01 00:00:00|              0|
  +------+------+-------------------+---------------+
  </code></pre>