1. A method of performing a large-scale data processing job, comprising: executing a plurality of processes on a plurality of interconnected processors, the plurality of processes including a master process for coordinating a data processing job for processing a set of input data, and a plurality of map processes and a plurality of reduce processes; in the master process, assigning input data blocks of a set of input data to respective map processes of the plurality of map processes and assigning partitions of intermediate data to respective reduce processes of the plurality of reduce processes; in each of the plurality of map processes: executing an application-independent map program to retrieve a sequence of input data blocks assigned thereto by the master process and to apply an application-specific map function to each input data block in the sequence to produce the intermediate data; and storing the intermediate data in memory of the interconnected processors; in each of the plurality of reduce processes: receiving a respective partition of the intermediate data from the memory of the interconnected processors; and applying an application-specific reduce function to the respective partition of the intermediate data to produce output values; and identifying, using predefined criteria, a partition of the intermediate data, the processing of which is determined to be likely to delay the data processing job, and taking a remedial action with respect to the identified partition.