Determine added, removed and kept items between two lists My code is determining the difference between two iterables in terms of items that have been addded, removed and kept. The items are assumed to be sortable and there are no duplicates.

def diff(before, after):
    """Return diff between iterables of sortable items `before` and `after` as
    tuple of three lists of items that have been added, removed and kept.
    """
    before_iter = iter(sorted(before))
    after_iter = iter(sorted(after))
    add_items = []
    rem_items = []
    keep_items = []
    try:
        new = old = None
        while True:
            if new == old:
                if new is not None:
                    keep_items.append(new)
                new = old = None
                new = next(after_iter)
                old = next(before_iter)
            elif new &lt; old:
                add_items.append(new)
                new = None
                new = next(after_iter)
            elif old &lt; new:
                rem_items.append(old)
                old = None
                old = next(before_iter)
    except StopIteration:
        if new is not None:
            add_items.append(new)
        if old is not None:
            rem_items.append(old)
        add_items += list(after_iter)
        rem_items += list(before_iter)
    return add_items, rem_items, keep_items

if __name__ == '__main__':
    assert diff([], []) == ([], [], [])
    assert diff([1, 2, 3], [4, 3, 2]) == ([4], [1], [2, 3])
    assert diff(['foo', 'bar'], ['baz', 'bar']) == (['baz'], ['foo'], ['bar'])


Main concerns:


The way I'm working with iterators seems a bit awkward and appears to unnecessarily bloat the code. I feel there must be a more elegant pattern, maybe without iterators at all?
The code should perform well on large lists. That's also why I'm not using multiple set operations instead. (add_items = set(after) - set(before), etc.) Or can I solve this efficiently with set operations? (I'm expecting both very small and very large differences between the iterables.)




Update: Unfortunately, I failed to include a key requirement into the question which therefore couldn't be considered in the answers. But I'll state it here, just for the record:


The algorithm needs to perform well when comparison between the items is expensive. Since an approach using set operations incorporates more comparisons than the algorithm above, it performs worse in my use case.
First, let's look at the naive implementation using sets:

def diff_set(before, after):
    b, a = set(before), set(after)
    return list(a - b), list(b - a), list(a &amp; b)


This is (so much more) readable and understandable than your code. The question is, is this significantly slower than your code?

Here are timings of your three example cases:

             Input                     diff             diff_set         diff_counter     
 ------------------------------ ------------------- ------------------- ------------------- 
  [], []                         1.72 µs ± 29.9 ns   1.21 µs ± 5.34 ns   9.91 µs ± 91.4 ns  
  [1,2,3], [4,3,2]               3.16 µs ± 183 ns    1.57 µs ± 28.4 ns   12 µs ± 180 ns     
  ['foo','bar'], ['baz','bar']   2.77 µs ± 83.4 ns   1.56 µs ± 37.7 ns   11.5 µs ± 240 ns


The set implementation is faster in all three cases. But what about the scaling behavior? First lets set up some random test data of increasing length. Note that I scaled the number of available values up with the size of the lists, so that set(before), set(after) does not become trivial and so that there are both added and removed values. The only way in which this might be unrealistic is that both before and after are always of the same size.

import random
random.seed(42)
random_data = [([random.randrange(2*n) for _ in range(n)], [random.randrange(2*n) for _ in range(n)]) for n in range(1, 10000, 100)]


Here also the set implementation wins:



In other words: always try the naive approach first and then test to see if the more complicated approach actually improves anything.



For reference, here the implementation of diff_counter I used:

from collections import Counter

def diff_counter(before, after):
    c = Counter(after)
    c.subtract(before)
    return list(+c), list(-c), [elem for elem, count in c.items() if count == 0]