Using keras tokenizer on Spark DataFrame. Column is not iterable I am having trouble using the Tokenizer

The code is

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer()
tokenizer.fit_on_texts(df.descricao_despesa)



X = tokenizer.texts_to_sequences(df.descricao_despesa)


maxlen = 100

X= pad_sequences(df.descricao_despesa, padding='post', maxlen=maxlen)


I tried to cast column descricao_despesa to String but still doest work.

If i try to cast to StringType i receive 

from pyspark.sql.functions import col
df = df.withColumn("descricao_despesa", col("descricao_despesa").cast('StringType'))

ParseException: u'\nDataType stringtype() is not supported.(line 1, pos 0)\n\n== SQL ==\nStringType\n^^^\n


I am using spark 2.0 and Python 2.7
I'm not sure that one can use Keras on Spark to process data in parallel (using multiple workers). 

But if your data can fit in RAM on a single node, that you can easily create a Pandas DataFrame from a Spark DataFrame and pass that DF to Keras. You can also convert result back to Spark DF if you need.

Spark DF --> Pandas DF:

pdf = df.toPandas() # df - is a Spark DataFrame


Pandas DF --> Spark DF:

sdf = spark.createDataFrame(pdf)