title: Pandas: efficiently exclude rows basing on criteria spanning all columns
body: <p>I have a DataFrame, which, simplified, can look like this:</p>
  
  <pre><code>ID     A    B    C
  ID_1   0    1    1
  ID_2   0    0    0
  ID_3   1    1    0
  </code></pre>
  
  <p>And so on. What I need to do is to <em>remove</em> rows where "1" is present in more than 50% of the columns.</p>
  
  <p>I came up with this function, using <code>apply</code>:</p>
  
  <pre><code>def thresh_filter(value):
      counts = value.value_counts(True) # Fraction based counts
      if (counts.index == 1).any():
          # Some rows are all 0s
          return counts[1] &gt; 0.50
      else:
          return False
  </code></pre>
  
  <p>In an effort to make it faster, I also tried this solution:</p>
  
  <pre><code>def thresh_filter(value):
      counts = value.value_counts(True)
      max_idx = counts.idxmax()
      max_percentage = counts.max()
      return max_idx == 1 and max_percentage &gt; 0.50
  </code></pre>
  
  <p>And then use it with</p>
  
  <pre><code>result = mydf.apply(thresh_filter, axis=1)
  </code></pre>
  
  <p>The problem is that it's <strong>very</strong> slow. On a 50K line DataFrame with 24 columns it takes several seconds, which makes it quite not viable. </p>
  
  <p>What can be done to improve its efficiency?</p>