title: How to convert floats to IntegerType using StructType in PySpark?
body: <p>I am trying to create a Spark data frame from a Pandas data frame, where I specify the column data types using StructType class. I've saved the pandas data frame as df and the spark data frame as data.</p>
  
  <p>Before I get into any of that, there's an error somewhere in the csv file, I made use of pandas' read_csv method's parameter, error_bad_lines. I'm unfamiliar with a spark equivalent.</p>
  
  <pre class="lang-py prettyprint-override"><code>df = pd.read_csv('Amazon_Responded_Oct05.csv',error_bad_lines=False)
  df.head()
  &gt;&gt;&gt;&gt;
      user_id_str user_followers_count    text_
  0   143515471.0 1503    @AmazonHelp Can you please DM me? A product I ...
  1   85741735.0  149569  @SeanEPanjab I'm sorry, we're unable to DM you...
  2   143515471.0 1503    @AmazonHelp It was purchased on... 
  3   143515471.0 1503    @AmazonHelp I am following you now, if it help...
  4   85741735.0  149569  @SeanEPanjab Please give us a call/chat so we ...
  </code></pre>
  
  <p>Notice how the user_id_str column is populated with float values, ie 143515471.0 Below is where the error is raised.</p>
  
  <pre class="lang-py prettyprint-override"><code>data_schema = [StructField('user_followers_count',IntegerType(),True),
                 StructField('user_id_str',StringType(),True),
                 StructField('text',StringType(),True)]
  final_struc = StructType(fields=data_schema)
  
  data = spark.createDataFrame(df,schema=final_struc)
  &gt;&gt;&gt;&gt;
  TypeError: field user_followers_count: IntegerType can not accept object 143515471.0 in type &lt;class 'float'&gt;
  </code></pre>
  
  <p>I've tried to remedy this from the pandas end without success</p>
  
  <pre class="lang-py prettyprint-override"><code>df.astype({'user_id_str': 'int','user_followers_count':'int','text_':'str'}).dtypes
  df.head(1)
  &gt;&gt;&gt;&gt;
      user_id_str user_followers_count    text_
  0   143515471.0 1503    @AmazonHelp Can you please DM me? A product I ...
  </code></pre>
  
  <p>In summary, I've taken various approaches to my goal, creating a Spark data frame with column data types, IntegerType, IntegerType, StringType without success. I'd much appreciate a way to force this data conversion.</p>
  
  <p>Edit:</p>
  
  <p>Lastly, I've tried simply starting with Spark; but that was also unfruitful.</p>
  
  <pre class="lang-py prettyprint-override"><code>
  data_1 = spark.read.csv('Amazon_Responded_Oct05.csv',schema=final_struc,enforceSchema=True)
  data_1.head(5)
  &gt;&gt;&gt;&gt;
  +--------------------+-----------+----+
  |user_followers_count|user_id_str|text|
  +--------------------+-----------+----+
  |                null|       null|null|
  |                null|       null|null|
  |                null|       null|null|
  |                null|       null|null|
  |                null|       null|null|
  +--------------------+-----------+----+
  only showing top 5 rows
  </code></pre>