title: Tokenizing strings in a "String RDD" returning another RDD
body: <p>I have a Spark RDD of individual string values, where each string is formed out of words separated by <code>|</code> symbols.</p>
  
  <p>This RDD was generated by a SparkSQL query, NOT by a <code>.textFile(...)</code> load operation.</p>
  
  <p>I can't (unless I'm miss-understanding something fundamental) use a <code>.flatMap(_.split("|"))</code> operation as this flattens each string to individual characters before applying the <code>.split()</code>.</p>
  
  <p>However, I do need to do something like a <code>.flatMap()</code> in that I need a 1 to many mapping. As my data set is potentially very large I need this operation to parallelize, hence the use of RDDs and related operations.</p>
  
  <p>Interestingly when processing strings from RDDs loaded using <code>.textFile(...)</code>, then the <code>.flatMap(...)</code> operation does exactly what I want! So I'm guessing that there must be a way...</p>
  
  <p>Any help or suggestions would be greatly appreciated!</p>
  
  <p>Thanks!</p>