title: Pyspark, find all the data types of a column
body: <p>I am doing some data cleaning and data profiling work. So the given data could be quite messy. </p>
  
  <p>I want to get all the potential data types of a column using pyspark.
  Data types like: </p>
  
  <ol>
  <li>Integer</li>
  <li>Real number</li>
  <li>Date/Time</li>
  <li>String (Text)</li>
  <li>etc</li>
  </ol>
  
  <p>I will need to do more processing to generate the respective metadata based on what types do the column has.</p>
  
  <p>A column can contain more than one type. I don't mean the built-in data types. The given data are all of type <code>string</code>, but some are in the form of <code>"1234"</code> which is actually an <code>int</code>, and some are in the form of <code>"2019/11/19"</code>, which is actually a <code>date</code>.</p>
  
  <p>For example, the column <code>number</code> could contain values like </p>
  
  <ul>
  <li><code>"123"</code></li>
  <li><code>"123.456"</code></li>
  <li><code>"123,456.789"</code></li>
  <li><code>"NUMBER 123"</code></li>
  </ul>
  
  <p>In the above example, the data types would be <code>INTEGER, REAL NUMBER, STRING</code>.</p>
  
  <p>If I used <code>df.schema[col].dataType</code>, it just simply give me StringType.</p>
  
  <p>I was thinking that I can iterate through the entire column, and use <code>regex</code> to see which type does the row belong to, but I am curious if there is some better way to do it, since it's a relatively large dataset.</p>
  
  <hr>
  
  <p>Now I kind of solve the issue by iterating through the column, and do some type checking: </p>
  
  <ul>
  <li><code>df = spark.sql('SELECT col as _col FROM table GROUP BY _col')</code></li>
  <li><code>df.rdd.map(lambda s: typeChecker(s))</code> </li>
  </ul>
  
  <p>where in typeChecker I just check which type does <code>s._col</code> belongs to.</p>
  
  <p>Thanks.</p>