title: Creating RDD of words and their index in the original text
body: <p>I want to process a large text using spark and the order of the words in the text is important and should be kept.</p>
  
  <p>I tried the following approach but it was not successful for large texts. It has out of memory issue.(obviously!)</p>
  
  <pre><code>hadoopConf.set("textinputformat.record.delimiter", "$$$$$");//read whole text
  
  JavaRDD&lt;String&gt; texts = sparkContext.newAPIHadoopFile(inputFile, TextInputFormat.class, LongWritable.class, Text.class, hadoopConf).values().map((x) -&gt; x.toString());
  JavaRDD&lt;Tuple2&lt;String, Integer&gt;&gt; lines = texts.flatMap(new readDocuments());
  
  
  public class readDocuments implements FlatMapFunction&lt;String, Tuple2&lt;String,  Integer&gt;&gt; {
      private static final long serialVersionUID = 1L;
  
      // line , index in original text
      @Override
      public Iterator&lt;Tuple2&lt;String, Integer&gt;&gt; call(String text) throws Exception {
          List&lt;Tuple2&lt;String, Integer&gt;&gt; lines = new ArrayList&lt;Tuple2&lt;String, Integer&gt;&gt;();
          String[] tempLines = text.split("\n");
          for (int i = 0; i &lt; tempLines.length; i++) {
              if (tempLines[i].length() &gt; 0)
                  lines.add(new Tuple2&lt;String, Integer&gt;(tempLines[i], i));
          }
          return lines.iterator();
      }
  }
  </code></pre>
  
  <p>I also tried to read files using <code>sparkContext.wholeTextFiles(inputFile)</code> but the same problem! </p>
  
  <p>Any idea or hint will be really appreciated.</p>