Using keras tokenizer on Spark DataFrame. Column is not iterable
I'm not sure that one can use Keras on Spark to process data in parallel (using multiple workers). 

But if your data can fit in RAM on a single node, that you can easily create a Pandas DataFrame from a Spark DataFrame and pass that DF to Keras. You can also convert result back to Spark DF if you need.

Spark DF --> Pandas DF:

pdf = df.toPandas() # df - is a Spark DataFrame


Pandas DF --> Spark DF:

sdf = spark.createDataFrame(pdf)