title: Does spark predicate pushdown work with JDBC?
body: <p>According to <a href="https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html">this</a></p>
  
  <blockquote>
    <p>Catalyst applies logical optimizations such as predicate pushdown. The
    optimizer can push filter predicates down into the data source,
    enabling the physical execution to skip irrelevant data.</p>
  </blockquote>
  
  <p>Spark supports push down of predicates to the data source.
  Is this feature also available / expected for JDBC?</p>
  
  <p>(From inspecting the DB logs I can see it's not the default behavior right now - the full query is passed to the DB, even if it's later limited by spark filters)</p>
  
  <p><strong>MORE DETAILS</strong></p>
  
  <p>Running Spark 1.5 with PostgreSQL 9.4</p>
  
  <p>code snippet:</p>
  
  <pre><code>from pyspark import SQLContext, SparkContext, Row, SparkConf
  from data_access.data_access_db import REMOTE_CONNECTION
  
  sc = SparkContext()
  sqlContext = SQLContext(sc)
  
  url = 'jdbc:postgresql://{host}/{database}?user={user}&amp;password={password}'.format(**REMOTE_CONNECTION)
  sql = "dummy"
  
  df = sqlContext.read.jdbc(url=url, table=sql)
  df = df.limit(1)
  df.show()
  </code></pre>
  
  <p>SQL Trace:</p>
  
  <pre><code>&lt; 2015-09-15 07:11:37.718 EDT &gt;LOG:  execute &lt;unnamed&gt;: SET extra_float_digits = 3                                                                                                                      
  &lt; 2015-09-15 07:11:37.771 EDT &gt;LOG:  execute &lt;unnamed&gt;: SELECT * FROM dummy WHERE 1=0                                                                                                                   
  &lt; 2015-09-15 07:11:37.830 EDT &gt;LOG:  execute &lt;unnamed&gt;: SELECT c.oid, a.attnum, a.attname, c.relname, n.nspname, a.attnotnull OR (t.typtype = 'd' AND t.typnotnull), pg_catalog.pg_get_expr(d.adbin, d.a
  drelid) LIKE '%nextval(%' FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON (c.relnamespace = n.oid) JOIN pg_catalog.pg_attribute a ON (c.oid = a.attrelid) JOIN pg_catalog.pg_type t ON (a.a
  tttypid = t.oid) LEFT JOIN pg_catalog.pg_attrdef d ON (d.adrelid = a.attrelid AND d.adnum = a.attnum) JOIN (SELECT 15218474 AS oid , 1 AS attnum UNION ALL SELECT 15218474, 3) vals ON (c.oid = vals.oid
   AND a.attnum = vals.attnum)                                                                                                                                                                            
  &lt; 2015-09-15 07:11:40.936 EDT &gt;LOG:  execute &lt;unnamed&gt;: SET extra_float_digits = 3                                                                                                                      
  &lt; 2015-09-15 07:11:40.964 EDT &gt;LOG:  execute &lt;unnamed&gt;: SELECT "id","name" FROM dummy                                                                                                                   
  </code></pre>
  
  <p>I would expect that the last select will include a <code>limit 1</code> clause - but it doesn't                                                                                                                                                                                                        </p>