Efficiently processing all data in a Cassandra Column Family with a MapReduce job
Question: I want to process all of the data in a column family in a MapReduce job.
Ordering is not important.

An approach is to iterate over all the row keys of the column family to use as
the input. This could be potentially a bottleneck and could replaced with a
parallel method.

I'm open to other suggestions, or for someone to tell me I'm wasting my time
with this idea. I'm currently investigating the following:

A potentially more efficient way is to assign ranges to the input instead of
iterating over all row keys (before the mapper starts). Since I am using
`RandomPartitioner`, is there a way to specify a range to query based on the
MD5?

For example, I want to split the task into 16 jobs. Since the
`RandomPartitioner` is MD5 based (from what I have read), I'd like to query
everything starting with `a` for the first range. In other words, how would I
query do a get_range on the MD5 with the start of `a` and ends before `b`.
e.g. `a0000000000000000000000000000000 - afffffffffffffffffffffffffffffff`?

I'm using the pycassa API (Python) but I'm happy to see Java examples.


Answer: I'd cheat a little:

  1. Create new rows job_(n) with each column representing each row key in the range you want
  2. Pull all columns from that specific row to indicate which rows you should pull from the CF

I do this with users. Users from a particular country get a column in the
country specific row. Users with a particular age are also added to a specific
row.

Allows me to quickly pull the rows i need based on the criteria i want and is
a little more efficient compared to pulling everything.

This is how the Mahout CassandraDataModel example functions:

  * <https://github.com/apache/mahout/blob/trunk/integration/src/main/java/org/apache/mahout/cf/taste/impl/model/cassandra/CassandraDataModel.java>

Once you have the data and can pull the rows you are interested in, you can
hand it off to your MR job(s).

Alternately, if speed isn't an issue, look into using PIG: [How to use
Cassandra's Map Reduce with or w/o
Pig?](http://stackoverflow.com/questions/2734005/how-to-use-cassandras-map-
reduce-with-or-w-o-pig)

