title: Spark and SparkSQL: How to imitate window function?
body: <h1>Description</h1>
  
  <p>Given a dataframe <code>df</code></p>
  
  <pre><code>id |       date
  ---------------
   1 | 2015-09-01
   2 | 2015-09-01
   1 | 2015-09-03
   1 | 2015-09-04
   2 | 2015-09-04
  </code></pre>
  
  <p>I want to create a running counter or index, </p>
  
  <ul>
  <li>grouped by the same id and</li>
  <li>sorted by date in that group, </li>
  </ul>
  
  <p>thus</p>
  
  <pre><code>id |       date |  counter
  --------------------------
   1 | 2015-09-01 |        1
   1 | 2015-09-03 |        2
   1 | 2015-09-04 |        3
   2 | 2015-09-01 |        1
   2 | 2015-09-04 |        2
  </code></pre>
  
  <p>This is something I can achieve with window function, e.g.</p>
  
  <pre><code>val w = Window.partitionBy("id").orderBy("date")
  val resultDF = df.select( df("id"), rowNumber().over(w) )
  </code></pre>
  
  <p>Unfortunately, Spark 1.4.1 does not support window functions for regular dataframes:</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Could not resolve window function 'row_number'. Note that, using window functions currently requires a HiveContext;
  </code></pre>
  
  <h1>Questions</h1>
  
  <ul>
  <li>How can I achieve the above computation on current Spark 1.4.1 without using window functions? </li>
  <li>When will window functions for regular dataframes be supported in Spark?</li>
  </ul>
  
  <p>Thanks!</p>