title: Spark sql query returns StringType instead of ArrayType?
body: <p>In trying to apply my UDF during my spark.sql query, instead of returning my cleaned words in array form the query simply returns one long string that looks like my array. This gives me an error when attempting to apply CountVectorizer. The error it raises is <code>'requirement failed: Column cleanedWords must be of type equal to one of the following types: [ArrayType(StringType,true), ArrayType(StringType,false)] but was actually of type StringType.'</code></p>
  
  <p>This is my code:</p>
  
  <pre><code>from string import punctuation
  from hebrew import stop_words
  hebrew_stopwords = stop_words()
  
  def removepuncandstopwords(listofwords):
      newlistofwords = []
      for word in listofwords:
          if word not in hebrew_stopwords:
              for punc in punctuation:
                  word = word.strip(punc)
              newlistofwords.append(word)
      return newlistofwords
  
  from pyspark.ml.feature import CountVectorizer, IDF, Tokenizer, Normalizer
  from pyspark.sql.types import ArrayType, StringType
  
  sqlctx.udf.register("removepuncandstopwords", removepuncandstopwords, ArrayType(StringType()))
  
  sentenceData = spark.createDataFrame([
      (0, "Hello my friend; i am sam"),
      (1, "Hello, my name is sam")
  ], ["label", "sentence"])
  
  tokenizer = Tokenizer(inputCol="sentence", outputCol="words")
  wordsData = tokenizer.transform(sentenceData)
  wordsData.registerTempTable("wordsData")
  wordsDataCleaned = spark.sql("select label, sentence, words, removepuncandstopwords(words) as cleanedWords from wordsData")
  
  
  
  wordsDataCleaned[['cleanedWords']].rdd.take(2)[0]
  Out[163]:
  Row(cleanedWords='[hello, my, friend, i, am, sam]')
  </code></pre>
  
  <p>How can i resolve this issue?</p>