title: Converting epoch to datetime in PySpark data frame using udf
body: <p>I have a PySpark dataframe with this schema:</p>
  
  <pre><code>root
   |-- epoch: double (nullable = true)
   |-- var1: double (nullable = true)
   |-- var2: double (nullable = true)
  </code></pre>
  
  <p>Where epoch is in seconds and should be converted to date time. In order to do so, I define a user defined function (udf) as follows:</p>
  
  <pre><code>from pyspark.sql.functions import udf    
  import time
  def epoch_to_datetime(x):
      return time.localtime(x)
      # return time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(x))
      # return x * 0 + 1
  
  epoch_to_datetime_udf = udf(epoch_to_datetime, DoubleType())
  df.withColumn("datetime", epoch_to_datetime(df2.epoch)).show()
  </code></pre>
  
  <p>I get this error:</p>
  
  <pre><code>---&gt; 21     return time.localtime(x)
      22     # return x * 0 + 1
      23 
      TypeError: a float is required
  </code></pre>
  
  <p>If I simply return <code>x + 1</code> in the function, it works. Trying <code>float(x)</code> or <code>float(str(x))</code> or <code>numpy.float(x)</code> in <code>time.localtime(x)</code> does not help and I still get an error. Outside of <code>udf</code>, <code>time.localtime(1.514687216E9)</code> or other numbers works fine. Using <code>datetime</code> package to convert <code>epoch</code> to datetim results in similar errors. </p>
  
  <p>It seems that <code>time</code> and <code>datetime</code> packages do not like to fed with <code>DoubleType</code> from PySpark. Any ideas how I can solve this issue? Thanks.</p>