title: spark filter string column by contains one of list of strings
body: <p>I need to achieve something similar to: <a href="https://stackoverflow.com/questions/16046146/checking-if-values-in-list-is-part-of-string">Checking if values in List is part of String</a> in spark. I.e. there is a dataframe of:</p>
  
  <pre><code>abcd_some long strings
  goo bar baz
  </code></pre>
  
  <p>and an Array of desired words like <code>["some", "bar"]</code>.</p>
  
  <p>An UDF with this code would work just fine, however, I would like to have something more efficient. Is there a way to express the <code>FILTER my_col CONTAINS ONE OF [items]</code> using the SQL DSL? Perhaps by dynamically constructing a REGEX?</p>
  
  <blockquote>
    <p>NOTE: it is not an exat match rather a regular 'CONTAINS' / LIKE '%thing%'. I.e. not an exact match. Otherwise the <code>isIn</code> operator would work.</p>
  </blockquote>
  
  <h1>edit</h1>
  
  <p>probably generating some SQL code dynamically is the most efficient way.</p>
  
  <pre><code>def orFilterGeneratorMultiContains(filterPredicates:Seq[String], column:String):Column = {
  
      col(column).contains(filterPredicates(0)) or col(column).contains(filterPredicates(1)) // TODO iterate
    }
    def filterToDesiredApps(filterPredicates:Seq[String], column:String)(df:DataFrame):DataFrame={
        df.filter(orFilterGeneratorMultiContains(filterPredicates, column))
    }
  </code></pre>
  
  <p>So is need to still figure out how to properly iterate the expression.</p>
  
  <h1>edit 2</h1>
  
  <p>However, this turns out to be a bit tricky:</p>
  
  <pre><code>import org.apache.spark.sql.functions.col
  
  val column = col("foo")
  val interstingTHings = Seq("bar", "baz", "thing3")
  
  interstingTHings.foldLeft(column) { (filteredOrColumnExpression, predicateItem) =&gt;
    // TODO how to properly nest the OR operator?
    // filteredOrColumnExpression.contains(predicateItem) // generates: Contains(Contains(Contains('foo, bar), baz), thing3)
    filteredOrColumnExpression or filteredOrColumnExpression.contains(predicateItem) // generates: ((('foo || Contains('foo, bar)) || Contains(('foo || Contains('foo, bar)), baz)) || Contains((('foo || Contains('foo, bar)) || Contains(('foo || Contains('foo, bar)), baz)), thing3)) 
    //     TODO but what y really would need is:
    //      col(column).contains("bar") or col(column).contains("baz") or col(column).contains("thing3")
  }.explain(true)
  </code></pre>
  
  <p>as it does not generate the correct <code>OR</code> nested filter conditions.</p>