How can I convert unicode to string of a dataframe column?
Since it's a string, you could remove the first and last characters:
From '[23,4,77,890,455]' to '23,4,77,890,455'
Then apply the split() function to generate an array, taking , as the delimiter.
I have a spark dataframe which has a column 'X'.The column contains elements which are in the form:

u'[23,4,77,890,455,................]'

. How can I convert this unicode to list.That is my output should be

[23,4,77,890,455...................]

.  I have apply it for each element in the 'X' column.
I have tried df.withColumn("X_new", ast.literal_eval(x)) and got the error 

"Malformed String"

I also tried 

df.withColumn("X_new", json.loads(x)) and got the error "Expected
  String or Buffer"

and 

df.withColumn("X_new", json.dumps(x)) which says JSON not
  serialisable.

and also 

df_2 = df.rdd.map(lambda x: x.encode('utf-8')) which says rdd has no
  attribute encode.

I dont want to use collect and toPandas() because its memory consuming.(But if thats the only way please do tell).I am using Pyspark
Update: cph_sto gave the answer using UDF.Though it worked well,I find that it is Slow.Can Somebody suggest any other method?