title: Pandas compare value with previous row with filtration condition
body: <p>I have a DataFrame with information about employee salary. It's about 900000+ rows.</p>
  
  <p>Sample:</p>
  
  <pre><code>+----+-------------+---------------+----------+
  |    |   table_num | name          |   salary |
  |----+-------------+---------------+----------|
  |  0 |      001234 | John Johnson  |     1200 |
  |  1 |      001234 | John Johnson  |     1000 |
  |  2 |      001235 | John Johnson  |     1000 |
  |  3 |      001235 | John Johnson  |     1200 |
  |  4 |      001235 | John Johnson  |     1000 |
  |  5 |      001235 | Steve Stevens |     1000 |
  |  6 |      001236 | Steve Stevens |     1200 |
  |  7 |      001236 | Steve Stevens |     1200 |
  |  8 |      001236 | Steve Stevens |     1200 |
  +----+-------------+---------------+----------+
  </code></pre>
  
  <p>dtypes:</p>
  
  <pre><code>table_num: string
  name: string
  salary: float
  </code></pre>
  
  <p>I need to add a column with information about increased\decreased salary level.
  I'm using the <code>shift()</code> function to compare value in rows.</p>
  
  <p>Main problem is in filtration and iteration over all unique employees over the whole dataset. </p>
  
  <p><strong>It takes about 3 and half hour</strong> in my script.</p>
  
  <p><strong>How to do it faster?</strong></p>
  
  <p>My script:</p>
  
  <pre><code># giving us only unique combination of 'table_num' and 'name'
      # since there can be same 'table_num' for different 'name'
      # and same names with different 'table_num' appears sometimes
  
  names_df = df[['table_num', 'name']].drop_duplicates()
  
  # then extracting particular name and table_num from Series
  for i in range(len(names_df)):    ### Bottleneck of whole script ###    
      t = names_df.iloc[i,[0,1]][0]
      n = names_df.iloc[i,[0,1]][1]
  
      # using shift() and lambda to check if there difference between two rows 
      diff_sal = (df[(df['table_num']==t)
                 &amp; ((df['name']==n))]['salary'] - df[(df['table_num']==t)
                                                   &amp; ((df['name']==n))]['salary'].shift(1)).apply(lambda x: 1 if x&gt;0 else (-1 if x&lt;0 else 0))
      df.loc[diff_sal.index, 'inc'] = diff_sal.values
  </code></pre>
  
  <p>Sample input data:</p>
  
  <pre><code>df = pd.DataFrame({'table_num': ['001234','001234','001235','001235','001235','001235','001236','001236','001236'], 
                       'name': ['John Johnson','John Johnson','John Johnson','John Johnson','John Johnson', 'Steve Stevens', 'Steve Stevens', 'Steve Stevens', 'Steve Stevens'], 
                       'salary':[1200.,1000.,1000.,1200.,1000.,1000.,1200.,1200.,1200.]})
  </code></pre>
  
  <p>Sample output:</p>
  
  <pre><code>+----+-------------+---------------+----------+-------+
  |    |   table_num | name          |   salary |   inc |
  |----+-------------+---------------+----------+-------|
  |  0 |      001234 | John Johnson  |     1200 |     0 |
  |  1 |      001234 | John Johnson  |     1000 |    -1 |
  |  2 |      001235 | John Johnson  |     1000 |     0 |
  |  3 |      001235 | John Johnson  |     1200 |     1 |
  |  4 |      001235 | John Johnson  |     1000 |    -1 |
  |  5 |      001235 | Steve Stevens |     1000 |     0 |
  |  6 |      001236 | Steve Stevens |     1200 |     0 |
  |  7 |      001236 | Steve Stevens |     1200 |     0 |
  |  8 |      001236 | Steve Stevens |     1200 |     0 |
  +----+-------------+---------------+----------+-------+
  </code></pre>