title: Converting from DynamicFrame to Pyspark Dataframe results in null timestamp and date values
body: <p>Here's the code loading data from a AWS Glue Data Catalog table:</p>
  <pre><code>         cleanseQueue = self.glueContext.create_dynamic_frame.from_catalog(
  
                              database=&quot;my_db&quot;,
  
                              table_name=&quot;cleanse_queue_table&quot;,
  
                              transformation_ctx=&quot;cleanseQueue&quot;
  
                          )
  
          # Script generated for node ApplyMapping
  
          cleanseQueue = ApplyMapping.apply(
  
              frame=cleanseQueue,
  
              mappings=[
  
                  (&quot;cleanseinprocess&quot;, &quot;boolean&quot;, &quot;cleanseinprocess&quot;, &quot;boolean&quot;),
  
                  (&quot;ingestedfilepath&quot;, &quot;string&quot;, &quot;ingestedfilepath&quot;, &quot;string&quot;),
  
                  (&quot;ingesteddate&quot;, &quot;date&quot;, &quot;ingesteddate&quot;, &quot;date&quot;),
  
                  (&quot;ingestedtime&quot;, &quot;timestamp&quot;, &quot;ingestedtime&quot;, &quot;timestamp&quot;),
  
              ],
  
              transformation_ctx=&quot;cleanseQueue&quot;,
  
          )
  
          cleanseQueueResolved = ResolveChoice.apply(frame=cleanseQueue, choice=&quot;make_cols&quot;, transformation_ctx=&quot;cleanseQueueResolved&quot;)
  
  
          cleanseQueue_DF = cleanseQueueResolved.toDF()
  
          cleanseQueue_DF = cleanseQueue_DF.withColumn(&quot;ingesteddate&quot;, to_date(F.col(&quot;ingesteddate&quot;),'dd-MM-yyyy').cast(DateType()))\
  
                                   .withColumn(&quot;ingestedtime&quot;, to_timestamp(F.col(&quot;ingestedtime&quot;),'dd-MM-yyyy HH:mm').cast(TimestampType()))
  
          cleanseQueue_DF.show()
  </code></pre>
  <p><em>Output from spark dataframe:</em></p>
  <p><a href="https://i.stack.imgur.com/SwzwV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SwzwV.png" alt="enter image description here" /></a></p>
  <pre><code>cleanseQueueResolved.show()
  </code></pre>
  <p><em>Output from Dynamic Frame:</em></p>
  <p>{&quot;ingestedfilepath&quot;: &quot;PrimeBrokerData/inS3/file/path/jpmavail_.20210726.csv&quot;, &quot;ingesteddate&quot;: &quot;2021-12-10&quot;, &quot;ingestedtime&quot;: &quot;2021-12-10 19:27:46.0&quot;, &quot;cleanseinprocess&quot;: true}</p>
  <p>{&quot;ingestedfilepath&quot;: &quot;PrimeBrokerData/inS3/file/path/jpmavail_.20210726.csv&quot;, &quot;ingesteddate&quot;: &quot;2021-12-15&quot;, &quot;ingestedtime&quot;: &quot;2021-12-15 18:09:36&quot;, &quot;cleanseinprocess&quot;: false}</p>
  <p>I would like to know what I'm missing here... I've tried with and without the explicit date and timestamp cast to no avail, the result is the same.</p>
  <p>Thanks</p>
  <p>Hasanain</p>