1. A computer implemented method for load balancing using progressive sampling to achieve a load balancing quality target over a plurality of worker tasks in a parallel processing environment, the method comprising: computing the load balancing quality target comprising at least a maximum workload variance target for the plurality of worker tasks, and a target confidence level; identifying a set of input records to be processed by the plurality of worker tasks in the parallel processing environment; sampling the set of input records to form a first set of sampled input records using a sampling engine separate from the parallel processing environment to perform: identifying the first set of sampled input records to process; forming a prospective load balancing assignment indicating how the set of input records are to be apportioned to the plurality of worker tasks, based at least in part upon the first set of sampled input records; calculating workload estimates corresponding to the plurality of worker tasks, based at least in part upon the prospective load balancing assignment and the first set of sampled input records; determining if the load balancing target is achieved, based at least in part upon the workload estimates; and if the load balancing target is not achieved, determining an additional number of input records to be sampled from the set of input records in order to achieve the load balancing target, sampling additional input records, and updating the prospective load balancing assignment based at least in part upon the additional input records; and processing the set of input records using the plurality of worker tasks in the parallel processing environment, based at least in part upon the prospective load balancing assignment.