title: How to return data in function passed to flatmap
body: <p>I have a file of data in the following schema:</p>
  
  <pre><code>+--------------------+-------------------+
  |           full_text|                 id|
  +--------------------+-------------------+
  </code></pre>
  
  <p>I want to change each record to pairs of (token, id). For example for record <code>Hi, how are you, 1010</code> I want it to transform to:</p>
  
  <pre><code>(hi, 1010)
  (how, 1010)
  (are, 1010)
  (you, 1010)
  </code></pre>
  
  <p>So I did it using SparkSql in the following way:</p>
  
  <pre><code>data = spark.read.json("../input/Spark_tweets.json").select("full_text", "id")
  data.rdd.map(lambda data : (data[0], data[1]))\
      .flatMap(lambda row: generateWordPlusId(row[0], row[1])).saveAsTextFile("out")
  </code></pre>
  
  <p>and the <code>generateWordPlusId</code> function does:</p>
  
  <pre><code>def generateWordPlusId(sentence, id):
      t = []
      for i in [word for word in sentence.lower().split()]:
          t.append((i, id))
      return tuple(t)
  </code></pre>
  
  <p>and it works as desired. My question is the function that we passed to a <code>flatMap</code> method, like <code>generateWordPlusId</code> in my case, what exactly should returns? I insert data into a 2-D array and at last convert it to the tuple to return. Is this typical approach and optimized or there is a better way to do that? </p>