title: Data Type validation in pyspark
body: <p>We are building a data ingestion framework in pyspark and wondering what the best way is to handle datatype exceptions.  Basically, we want to have a reject table capturing all the data that does not confirm to the schema.</p>
  
  <pre class="lang-python prettyprint-override"><code>stringDf = sparkSession.createDataFrame(
      [
          ("11/25/1991","1"),
          ("11/24/1991", None),
          ("11/30/1991","a")
      ], 
      ['dateAsString','intAsString']
  )
  </code></pre>
  
  <p>Here is my <code>stringDf</code> with two columns.</p>
  
  <pre class="lang-python prettyprint-override"><code>+------------+-----------+
  |dateAsString|intAsString|
  +------------+-----------+
  |  11/25/1991|          1|
  |  11/24/1991|       null|
  |  11/30/1991|          a|
  +------------+-----------+
  </code></pre>
  
  <p>I would like to create a new column to the data frame called <code>dataTypeValidationErrors</code> to capture all the errors that might be present in this dataset.  What is the best way to achieve this using pyspark?</p>
  
  <pre class="lang-python prettyprint-override"><code>+------------+-----------+------------------------+
  |dateAsString|intAsString|dataTypeValidationErrors|
  +------------+-----------+------------------------+
  |  11/25/1991|          1|None                    |
  |  11/24/1991|       null|None                    |
  |  11/30/1991|          a|Not a valid Number      |
  +------------+-----------+------------------------+
  </code></pre>