title: Comparison operator in PySpark (not equal/ !=)
body: <p>I am trying to obtain all rows in a dataframe where two flags are set to '1' and subsequently all those that where only one of two is set to '1' and the other <strong>NOT EQUAL</strong> to '1'</p>
  
  <p>With the following schema (three columns),</p>
  
  <pre><code>df = sqlContext.createDataFrame([('a',1,'null'),('b',1,1),('c',1,'null'),('d','null',1),('e',1,1)], #,('f',1,'NaN'),('g','bla',1)],
                              schema=('id', 'foo', 'bar')
                              )
  </code></pre>
  
  <p>I obtain the following dataframe:</p>
  
  <pre><code>+---+----+----+
  | id| foo| bar|
  +---+----+----+
  |  a|   1|null|
  |  b|   1|   1|
  |  c|   1|null|
  |  d|null|   1|
  |  e|   1|   1|
  +---+----+----+
  </code></pre>
  
  <p>When I apply the desired filters, the first filter (foo=1 AND bar=1) works, but not the other (foo=1 AND NOT bar=1)</p>
  
  <pre><code>foobar_df = df.filter( (df.foo==1) &amp; (df.bar==1) )
  </code></pre>
  
  <p>yields:</p>
  
  <pre><code>+---+---+---+
  | id|foo|bar|
  +---+---+---+
  |  b|  1|  1|
  |  e|  1|  1|
  +---+---+---+
  </code></pre>
  
  <p><strong>Here is the non-behaving filter:</strong></p>
  
  <pre><code>foo_df = df.filter( (df.foo==1) &amp; (df.bar!=1) )
  foo_df.show()
  +---+---+---+
  | id|foo|bar|
  +---+---+---+
  +---+---+---+
  </code></pre>
  
  <p>Why is it not filtering? How can I get the columns where only foo is equal to '1'?</p>