confused by python builtin sort() returning out-of-order items
Someone recommended that I check for NaN values in score, and sure enough, there were some. Removing those now allows the sort to work properly.
TL;DR - don't allow NaNs in your sort key. If you see weird sorting behavior, check for NaNs.
I have a large (~26k) list of dicts; I call .sort(key=lambda x:x['score'], reverse=True) to sort the list by the score of each element. While it does do some amount of sorting, it's not hard to find examples of out-of-order items.
Specifically, I have a data set of the form {specimen:"Specimen Name", test_param:2.6, score:0.8649} (score is between -1 and 1, and is what I'm trying to sort on). About 26k of these entries are put into a single list (let's call it data). I call data.sort(key=lambda x:x['score'], reverse=True), and yet the resulting list contains items in the middle of the list that have a higher score than at the head; the items have no discernable (to me) pattern to their order, apart from being "generally" greatest-to-least.
I've tried several approaches to make sure it wasn't some silly type error: casting to float in the lambda and supplying a default value with key=lambda x:float(x.get('score', -100)); reformatting the list of dicts into a list of tuples (score, dict) and sorting by x[0]. The (incorrect) order that it returns is very consistent, so I'm ruling out race condition oddities.
I've resulted to hand-crafting a simple sort function, and that works fine (just slower). How is it that this seemingly simple case of sorting is exhibiting such odd behavior?