filter numpy array of datetimes by frequency of occurance
Thanks for all of your suggestions.
I ended up doing something completely different with dictionaries in the end and found it much faster for the processing that I required.
I created a dictionary with a unique set of timestamps as the keys and empty lists as the values and then looped once through the unordered list (or array) and populated the value lists with the values that I wanted to count.
Thanks again!
I have an array of over 2 million records, each record has a 10 minutes resolution timestamp in datetime.datetime format, as well as several other values in other columns.
I only want to retain the records which have timestamps that occur 20 or more times in the array.  What's the fastest way to do this?  I've got plenty of RAM, so I'm looking for processing speed.
I've tried [].count() in a list comprehension but started to lose the will to live waiting for it to finish.  I've also tried numpy.bincount() but tragically it doesn't like datetime.datetime
Any suggestions would be much appreciated.
Thanks!