1. A method of performing a large-scale data processing job, comprising: executing a plurality of processes on a plurality of interconnected processors, the plurality of processes including a master process for coordinating a data processing job for processing a set of input data, a plurality of map processes, and a plurality of reduce processes; in the master process, assigning input data blocks of a set of input data to respective map processes of the plurality of map processes, assigning partitions of intermediate data to respective reduce processes of the plurality of reduce processes, and determining when the data processing job has reached a predefined level of completion; in each of the plurality of map processes: executing an application-independent map program to retrieve a sequence of input data blocks assigned thereto by the master process and to apply an application-specific map function to each input data block in the sequence to produce the intermediate data; and storing the intermediate data in memory of the interconnected processors; in each of the plurality of reduce processes: receiving a respective partition of the intermediate data from the memory of the interconnected processors; and applying an application-specific reduce function to the respective partition of the intermediate data to produce output values; in accordance with a determination that processing of an identified partition of the intermediate data, identified using predefined criteria, is likely to delay the data processing job, taking a remedial action with respect to the identified partition; and in accordance with the master process determining that the data processing job has reached the predefined level of completion, scheduling a backup execution of a respective remaining task, and determining that the respective remaining task is completed when either the respective remaining task or the backup execution of the respective remaining task completes.