title: How to optimize shuffling/sorting phase in a hadoop job
body: <p>I'm doing some data preparation using a single node hadoop job. The mapper/combiner in my job outputs many keys (more than 5M or 6M) and obviously the job proceeds slowly or even fails. The mapping phase runs up to 120 mapper and there is just one reducer (these are automatically determined and I've not set any values for them). I want to optimize the job so that shuffling/sorting phase occurs more efficiently. I increased <code>mapreduce.task.io.sort.mb</code> to 300m but the job faile because its value was larger than mapper heap. I then set <code>mapred.child.java.opts</code> to -Xmx1024m but it again failed because it couldn't initialize an Output Collector. What are the best practices for these scenarios?</p>