1. A method comprising: selecting a set of one or more cloud service providers to perform a data processing job based on information about the set of one or more cloud service providers and criteria of the data processing job; designing a plan for a cloud pipeline for performing the data processing job, wherein said designing the plan for the cloud pipeline comprises decomposing the data processing job into disparate computing tasks based, at least in part, on separation of a data set into data subsets for the data processing job and information about the set of one or more cloud service providers, determining parallelism and dependencies among the disparate computing tasks; determining a plurality of processing stages for performing the data processing job based, at least in part, on the determined parallelism and dependencies among the disparate computing tasks, wherein the plurality of processing stages constitute the cloud pipeline and wherein at least one of the plurality of processing stages includes multiple of the disparate computing tasks in parallel; mapping allocated resources of the set of one or more cloud service providers to the plurality of processing stages; generating instructions and software images based, at least in part, on the plan and said mapping of the allocated resources to the plurality of processing stages, wherein the instructions and the software images implement the cloud pipeline for performing the data processing job; transmitting the instructions and the software images to machines of the set of one or more cloud service providers in accordance with the plurality of processing stages, wherein the allocated resources comprise the machines; monitoring the machines and performing of the data processing job; and if said monitoring detects a failure, then adapting the cloud pipeline to the failure.