title: Elastic search and PySpark date_format_parsing error
body: <p>I am currently working on a school project using spark streaming and elastic search.
  I am collecting some reddit informations using a kafka connector. Spark is then connected to this topic and streams informations to elastic search.</p>
  
  <p>The problem that I am facing is some weird elastic search error due to the date format.</p>
  
  <pre><code>date_time_parse_exception: Failed to parse with all enclosed parsers
  </code></pre>
  
  <p>My data contains a timestamp in the format</p>
  
  <pre><code>yyyy-MM-dd HH:mm:ss
  </code></pre>
  
  <p>after some researches online, I understood that the problem was with the elastic search mapping (since it usese Jova-time). So I modified my mapping as follows</p>
  
  <pre><code>"mappings": {
          "properties": {
              "created_utc": {
                  "type": "date",
                  "format": "yyyy-MM-dd'T'HH:mm:ssZZ"
              }
  </code></pre>
  
  <p>(this is just a part of my mapping)</p>
  
  <p>Unfortunately, I've not solved the problem and, honestly, at the moment I don't really know what the problem could be.</p>
  
  <p>Thank you very much !</p>
  
  <p>At the origin, my date is a unix timestamp. 
  Since I'd like to work with a normal timestamp, I've converted it as follows using spark's functions:</p>
  
  <p>unix timestamp to timestamp string:</p>
  
  <pre><code>wordsDataFrame = wordsDataFrame.withColumn('created_utc', f.date_format(wordsDataFrame.created_utc.cast(dataType=tp.TimestampType()), "yyyy-MM-dd HH:mm:ss"))
  </code></pre>
  
  <p>string to timestamp type:</p>
  
  <pre><code>wordsDataFrame = wordsDataFrame.withColumn('created_utc', f.to_timestamp('created_utc', "yyyy-MM-dd HH:mm:ss"))
  </code></pre>
  
  <p>I've also tryed to modify the conversion using the same format required by elastic search:</p>
  
  <pre><code>wordsDataFrame = wordsDataFrame.withColumn('created_utc', f.to_timestamp('created_utc', "yyyy-MM-dd'T'HH:mm:ss"))
  </code></pre>
  
  <p>but this gives me null dates.</p>