title: filter spark dataframe with row field that is an array of strings
body: <p>Using Spark 1.5 and Scala 2.10.6</p>
  
  <p>I'm trying to filter a dataframe via a field "tags" that is an array of strings.  Looking for all rows that have the tag 'private'.</p>
  
  <pre><code>val report = df.select("*")
    .where(df("tags").contains("private"))
  </code></pre>
  
  <p>getting:</p>
  
  <blockquote>
    <p>Exception in thread "main" org.apache.spark.sql.AnalysisException:
    cannot resolve 'Contains(tags, private)' due to data type mismatch:
    argument 1 requires string type, however, 'tags' is of array
    type.;</p>
  </blockquote>
  
  <p>Is the filter method better suited?  </p>
  
  <p>UPDATED:</p>
  
  <p>the data is coming from cassandra adapter but a minimal example that shows what I'm trying to do and also gets the above error is:</p>
  
  <pre><code>  def testData (sc: SparkContext): DataFrame = {
      val stringRDD = sc.parallelize(Seq("""
        { "name": "ed",
          "tags": ["red", "private"]
        }""",
        """{ "name": "fred",
          "tags": ["public", "blue"]
        }""")
      )
      val sqlContext = new org.apache.spark.sql.SQLContext(sc)
      import sqlContext.implicits._
      sqlContext.read.json(stringRDD)
    }
    def run(sc: SparkContext) {
      val df1 = testData(sc)
      df1.show()
      val report = df1.select("*")
        .where(df1("tags").contains("private"))
      report.show()
    }
  </code></pre>
  
  <p>UPDATED: the tags array can be any length and the 'private' tag can be in any position</p>
  
  <p>UPDATED: one solution that works: UDF</p>
  
  <pre><code>val filterPriv = udf {(tags: mutable.WrappedArray[String]) =&gt; tags.contains("private")}
  val report = df1.filter(filterPriv(df1("tags")))
  </code></pre>