Retain top-N elements as we loop across all elements
Question: Here is what I am trying to do. The output of a calculation on a dataframe
gives a number. I use that number to rank the different dataframes and I need
to retain the top-N (in the example below, the top 10 is chosen). The ranking
is achieved by comparing the number to the last number of a reverse sorted
list. If the current number is larger, the list is popped and the new entry
added to the list followed by reverse sorting again. The following is
structurally identical to what I have and it works, albeit slowly. I would
appreciate any suggestions to improve its speed, efficiency or Pythonicness.

    
    
    import random
    import pandas as pd
    
    def gen_df():
        return random.uniform(0.0, 1.0), pd.DataFrame()
    
    if __name__ == '__main__':
        mylist = []
        for i in range(1000):
            val, df = gen_df()
            if len(mylist) < 10:
                mylist.append((val, df))
            else:
                mylist.sort(reverse=True)
                if mylist[-1][0] < val:
                    mylist.pop()
                    mylist.append((val, df))
    

EDIT: Reduced one sort after suggestion by zondo.


Answer: The way to speed it up is to replace your list with a min-heap of size 10. Put
the first 10 frames into the heap. Then, for each item, if it's larger than
the smallest item on the heap, pop the smallest item and push the new item.

I'm not a Python programmer, so I'll present the pseudocode.

    
    
    heap = new min-heap
    for each item
        if (heap.length < 10)
            heap.push(item)
        else if (item > heap.peek())
            heap.pop(); // remove smallest item
            heap.push(item); // add new item
    

This assumes, of course, that there's a min-heap implementation that you can
use. I suspect [heapq](https://docs.python.org/2/library/heapq.html) will do
the trick.

That's going to be significantly faster than sorting the list every time you
insert a new item.

