Calculate mean on values in python collections.Counter
Question: I'm profiling some numeric time measurements that cluster extremely closely. I
would like to obtain mean, standard deviation, etc. Some inputs are large, so
I thought I could avoid creating lists of millions of numbers and instead use
Python collections.Counter objects as a compact representation.

Example: one of my small inputs yields a `collection.Counter` like `[(48,
4082), (49, 1146)]` which means 4,082 occurrences of the value 48 and 1,146
occurrences of the value 49. For this data set I manually calculate the mean
to be something like 48.2192042846.

Of course if I had a simple list of 4,082 + 1,146 = 5,228 integers I would
just feed it to numpy.mean().

My question: how can I calculate descriptive statistics from the values in a
`collections.Counter` object just as if I had a list of numbers? Do I have to
create the full list or is there a shortcut?


Answer: `collections.Counter()` is a subclass of `dict`. Just use `Counter().values()`
to get a list of the counts:

    
    
    counts = Counter(some_iterable_to_be_counted)
    mean = numpy.mean(counts.values())
    

Note that I did _not_ call `Counter.most_common()` here, which would produce
the list of `(key, count)` tuples you posted in your question.

If you must use the output of `Counter.most_common()` you can filter out just
the counts with a list comprehension:

    
    
    mean = numpy.mean([count for key, count in most_common_list])
    

If you are using Python 3 (where `dict.values()` returns a dictionary view),
you could either pass in `list(counts.values())`, or use the standard library
[`staticstics.mean()`
function](https://docs.python.org/3/library/statistics.html#statistics.mean),
which takes an iterable (including `dict.values()` dictionary view).

If you meant to calculate the mean _key value_ as weighted by their counts,
you'd do your own calculations directly from the counter values. In Python 2
that'd be:

    
    
    from __future__ import division
    
    mean = sum(key * count for key, count in counter.iteritems()) / sum(counter.itervalues())
    

The `from __future__` import should be at the top of your module and ensures
that you won't run into overflow issues with large floating point numbers. In
Python 3 that'd be simplified to:

    
    
    mean = sum(key * count for key, count in counter.items()) / sum(counter.values())
    

The median could be calculated with bisection; sort the `(key, count)` pairs
by key, sum the counts, and bisect the half-way point into a accumulated sum
of the counts. The index for the insertion point points to the median key in
the sorted keys list.

