title: Why Spark SQL considers the support of indexes unimportant?
body: <p>Quoting the Spark DataFrames, <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#unsupported-hive-functionality">Datasets and SQL manual</a>:</p>
  
  <blockquote>
    <p>A handful of Hive optimizations are not yet included in Spark. Some of
    these (such as indexes) are less important due to Spark SQL’s
    in-memory computational model. Others are slotted for future releases
    of Spark SQL.</p>
  </blockquote>
  
  <p>Being new to Spark, I'm a bit baffled by this for two reasons:</p>
  
  <ol>
  <li><p>Spark SQL is designed to process Big Data, and at least in my use
  case the data size far exceeds the size of available memory.
  Assuming this is not uncommon, what is meant by "Spark SQL’s
  in-memory computational model"? Is Spark SQL recommended only for
  cases where the data fits in memory?</p></li>
  <li><p>Even assuming the data fits in memory, a full scan over a very large
      dataset can take a long time. I read <a href="http://lemire.me/blog/2010/12/20/for-your-in-memory-databases-do-you-really-need-an-index/">this argument</a> against
      indexing in in-memory database, but I was not convinced. The example
      there discusses a scan of a 10,000,000 records table, but that's not
      really big data. Scanning a table with billions of records can cause
      simple queries of the "SELECT x WHERE y=z" type take forever instead
      of returning immediately.</p></li>
  </ol>
  
  <p>I understand that Indexes have disadvantages like slower INSERT/UPDATE, space requirements, etc. But in my use case, I first process and load a large batch of data into Spark SQL, and then explore this data as a whole, without further modifications. Spark SQL is useful for the initial distributed processing and loading of the data, but the lack of indexing makes interactive exploration slower and more cumbersome than I expected it to be. </p>
  
  <p>I'm wondering then why the Spark SQL team considers indexes unimportant to a degree that it's off their road map. Is there a different usage pattern that can provide the benefits of indexing without resorting to implementing something equivalent independently?</p>