1. A method comprising: selecting a set of one or more cloud service providers to perform a data processing job based on information about the set of one or more cloud service providers and criteria of the data processing job; designing a plan for a cloud pipeline for performing the data processing job based, at least in part, on the information about the set of one or more cloud service providers, wherein the plan comprises a set of one or more processing stages each of which indicates processing upon a subset of a data set of the data processing job; mapping allocated resources of the set of one or more cloud service providers to the plurality of processing stages; generating instructions and software images based, at least in part, on the plan, wherein the instructions and the software images implement the cloud pipeline for performing the data processing job; and transmitting the instructions and the software images to machines of the set of one or more cloud service providers, wherein the allocated resources comprise the machines. monitoring the machines and performing of the data processing job; and if said monitoring detects a failure, then adapting the cloud pipeline to the failure.