title: Spark doesn't use predicate pushdown for ADLS Gen2
body: <p><em>Why Spark doesn't do predicate pushdown for Spark and ADLS Gen 2 (blob storage) in my case?</em> From articles <a href="https://www.blue-granite.com/blog/10-things-to-know-about-azure-data-lake-storage-gen2" rel="nofollow noreferrer">like this</a> I know that ADLS Gen 2 support predicate pushdown. <em>But does Spark works with ADLS Gen 2 API properly to ensure predicat pushdown?</em></p>
  
  <pre><code># Folder structure example `&lt;base_path&gt;/NAME=ABC/YEAR=2019/MONTH=05/part-xxxx-.snappy.parquet`
  spark_session.read \
      .parquet('&lt;base_path&gt;') \
      .filter((F.col('NAME') == 'ABC') &amp; (F.col('YEAR') == 2019) &amp; (F.col('MONTH') == 5)) \
      .explain()
  </code></pre>
  
  <p>From output you can see PySpark use simple <code>RemoteServiceExec filter</code> instead of <code>PushedFilter</code>. AFAIK, it means Spark doesn't use predicate pushdown for the query.</p>
  
  <p>P.S. I'm running the job in client deploy mode through databricks-connect</p>
  
  <p>Output:</p>
  
  <pre><code>RemoteServiceExec filter {
    condition {
      sql_expr {
        sql_repr: "((('#non-sql-expr-183d4706-d0e7-476a-823c-efe0de41914a' = 'ABC') AND ('#non-sql-expr-823ed3f2-b08f-4d00-88c1-29f45d08c43e' = 2019)) AND ('#non-sql-expr-a361bb1a-085f-4948-8fa5-7e1e14f835ee' = 5))"
        embedded_non_sql_exprs {
          uuid: "#non-sql-expr-823ed3f2-b08f-4d00-88c1-29f45d08c43e"
          expr {
            named_expr {
              attr {
                attr_ref {
                  name: "YEAR"
                  dataType {
                    sql_repr: "INT"
                  }
                  nullable: true
                  metadata {
                    json_repr: "{}"
                  }
                  expr_id {
                    id: 39
                    jvm_id: "929051d7-3030-4e2c-bf57-923aa610f40e"
                  }
                }
              }
            }
          }
        }
        embedded_non_sql_exprs {
          uuid: "#non-sql-expr-183d4706-d0e7-476a-823c-efe0de41914a"
          expr {
            named_expr {
              attr {
                attr_ref {
                  name: "NAME"
                  dataType {
                    sql_repr: "STRING"
                  }
                  nullable: true
                  metadata {
                    json_repr: "{}"
                  }
                  expr_id {
                    id: 38
                    jvm_id: "929051d7-3030-4e2c-bf57-923aa610f40e"
                  }
                }
              }
            }
          }
        }
        embedded_non_sql_exprs {
          uuid: "#non-sql-expr-a361bb1a-085f-4948-8fa5-7e1e14f835ee"
          expr {
            named_expr {
              attr {
                attr_ref {
                  name: "MONTH"
                  dataType {
                    sql_repr: "INT"
                  }
                  nullable: true
                  metadata {
                    json_repr: "{}"
                  }
                  expr_id {
                    id: 40
                    jvm_id: "929051d7-3030-4e2c-bf57-923aa610f40e"
                  }
                }
              }
            }
          }
        }
      }
    }
    child {
      logical_relation {
        relation {
          data_frame_read_spec {
            user_specified_source: "parquet"
            paths: "abfss://&lt;base_path&gt;"
          }
        }
        schema {
          ...
        }
        uuid: "da53053d-5e34-4c8d-8bab-f7a30dfefde4"
      }
    }
  }
  serialization_context {
  }
  </code></pre>