title: Duplicated timestamps when reading data from a CSV with PySpark
body: <p>I want to read data from a cvs file with the following format</p>
  
  <pre><code>HIERARCHYELEMENTID, REFERENCETIMESTAMP, VALUE
  LOUTHNMA,"2014-12-03 00:00:00.0",0.004433333289
  LOUTHNMA,"2014-12-03 00:15:00.0",0.004022222182
  LOUTHNMA,"2014-12-03 00:30:00.0",0.0037666666289999998
  LOUTHNMA,"2014-12-03 00:45:00.0",0.003522222187
  LOUTHNMA,"2014-12-03 01:00:00.0",0.0033333332999999996   
  </code></pre>
  
  <p>I am using the following PySpark function to read from this file</p>
  
  <pre><code># Define a specific function to load  flow data with schema
  def load_flow_data(sqlContext, filename, timeFormat):
  
      # Columns we're interested in
      flow_columns = ['DMAID','TimeStamp', 'Value']
      df = load_data(sqlContext, filename, flow_schema, flow_columns)
  
      # convert type of timestamp column from string to timestamp
      col = unix_timestamp(df['TimeStamp'], timeFormat).cast("timestamp")    
      df = df.withColumn('realTimeStamp', col)
  
      return df
  </code></pre>
  
  <p>with the following schema and auxiliary function</p>
  
  <pre><code>flow_schema = StructType([
      StructField('DMAID', StringType(), True),
      StructField('TimeStamp', StringType(), True),
      StructField('Value', FloatType(), True)
  ])
  
  def load_data(sqlContext, filename, schema=None, columns=None):
  # If no schema is specified, then infer the schema automatically
  if schema is None:
      df = sqlContext.read.format('com.databricks.spark.csv'). \
          option('header', 'true').option('inferschema', 'true'). \
          option('mode', 'DROPMALFORMED'). \
          load(filename)
  else:
      df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load(filename, schema=schema)
  
  # If no columns are specified, then select all columns
  if columns is None:
      columns = schema.names
  df = df.select(columns)
  
  return df   
  </code></pre>
  
  <p>I load the data from the cvs file using these commands</p>
  
  <pre><code>timeFormat = "yyyy-MM-dd HH:mm:SS"
  
  df_flow_DMA = load_flow_data(sqlContext, flow_file, timeFormat)
  </code></pre>
  
  <p>Then I convert this data frame to Pandas for visualisation purposes. </p>
  
  <p>However, I find that <code>col = unix_timestamp(df['TimeStamp'], timeFormat).cast("timestamp")</code> is mapping different date &amp; time strings in the cvs file (found in the field 'TimeStamp') to the same 'realTimeStamp' field, as shown in the attached screenshot.  </p>
  
  <p><a href="https://i.stack.imgur.com/2AsAz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2AsAz.png" alt="enter image description here"></a></p>
  
  <p>I suspect that the problem is related to the date time string format that I am passing to <code>load_flow_data</code>; I have tried several variations but nothing seems to work.</p>
  
  <p>Could someone please provide a hist at what is wrong with my code? I use Python 2.7 and Spark 1.6.</p>
  
  <p>Cheers</p>