title: Efficient distributed counting
body: <p>I have a series of events flowing through a system (e.g a pizza ordering system) and I want to count certain properties of each event through time. For example, I might want to see how many unique people ordered pepperoni pizza in the last 5 minutes, or how many pizzas John Doe ordered in the past week.</p>
  
  <p>It is a LOT of events, so we're using something like Cassandra or HBase because even the counts can't be stored in memory. Also, since we need to keep track of set membership (in order to count unique people ordering a particular kind of pizza, for example), it gets bigger.</p>
  
  <p>We could store a list of orders and then query to count, but this is slow. And we mostly don't care <em>who</em> ordered pepperoni pizza, just <em>how many</em> unique orders were made, and in a given time window.</p>
  
  <p>What's the best way to store this information, for example in Cassandra, such that the information can be retrieved in some time intervals?</p>
  
  <p>I tried at first to use Redis + bloom filters, but storing a bloom filter bit vector would require transactions to avoid race conditions, so then I used redis sets.</p>
  
  <p>Then I realized the whole thing was too big to just be in memory, so I decided to switch to a disk-backed store. However, there are no native sets like in redis.</p>
  
  <p>I looked at sketches / streaming algos like HyperLogLog but the conclusion was that to save the hyperloglog object, I need to store the bit array (or pickle the object or whatever)...is that kosher, and what are the best practices for this, if this is indeed the solution?</p>
  
  <p>I was tempted to save each event individually with a timestamp, then query and count on demand, but this is slow. I'm looking for something better, if it exists.</p>
  
  <h1>Example Requests:</h1>
  
  <ul>
  <li>How many unique people had a pepperoni pizza order in the past 10 minutes</li>
  <li>How many unique pepperoni pizzas were ordered by some person John Doe in the past 30 minutes</li>
  </ul>