title: efficient way to reformat/shift time series data using Spark
body: <p>I want to build some time series models using spark. The first step is to reformat the sequence data into training samples. The idea is:</p>
  
  <p>original sequential data (each t* is a number)</p>
  
  <pre><code>t1  t2  t3  t4  t5  t6  t7  t8  t9  t10
  </code></pre>
  
  <p>desired output</p>
  
  <pre><code>t1  t2  t3  t4  t5  t6
  t2  t3  t4  t5  t6  t7
  t3  t4  t5  t6  t7  t8
  ..................
  </code></pre>
  
  <p>how to write a function in spark to do this.
  The function signature should be like</p>
  
  <p>reformat(Array[Integer], n: Integer)    </p>
  
  <p>return type is Dataframe or Vector</p>
  
  <p>==========The code I tried on Spark 1.6.1 =========</p>
  
  <pre><code>val arraydata=Array[Double](1,2,3,4,5,6,7,8,9,10)
  val slideddata = arraydata.sliding(4).toSeq
  val rows = arraydata.sliding(4).map{x=&gt;Row(x:_*)}
  sc.parallelize(arraydata.sliding(4).toSeq).toDF("Values")
  </code></pre>
  
  <p>The final line can not go through with error: </p>
  
  <pre><code>Error:(52, 48) value toDF is not a member of org.apache.spark.rdd.RDD[Array[Double]]
      sc.parallelize(arraydata.sliding(4).toSeq).toDF("Values")
  </code></pre>