1. A system for processing a batch job, comprising: a processor configured to: receive a job name for a job submitted to execute; receive one or more job parameters; determine one or more nodes to run the job; determine one or more steps; for each step: execute a step of the one or more steps on a node of the one or more nodes using a state of data associated with a start state of the step; upon completion of executing the step, store a result to a durable storage, wherein the durable storage stores a state of data associated with a completion state of the step, and wherein the state of data associated with the start state of the step and the completion state of the step are accessible by other execution is processes as associated with either the start state of the step or the completion state of the step; a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions.