1. A method for executing jobs in a cluster computing system, the cluster computing system including a plurality of compute nodes connected by a network, the method comprising: at a first compute node of the cluster computing system, receiving via the network a job to be executed by at least one process of a distributed application executed by a processor operating under control of the first compute node; at the first compute node, receiving via the network a job state object for the job, wherein the job state object comprises a file that includes a database for storing job metadata, wherein the job metadata includes at least job status information; at the first compute node, accessing the database of the job state object to update the job metadata in connection with execution of the job by the processor of the compute node while the job state object and the job are at the first compute node; and upon completion of the execution of a portion of the job at the first compute node: transferring, from the first compute node, the job state object and the job via the network to a second compute node of the cluster computing system, at the first compute node, transmitting, via the network, an address of the job state object at the second compute node to a state tracker node of the cluster computing system, the state tracker node executing a state tracking process for storing the address of the job state object in a state object location database, wherein the second compute node further executes the job, accesses the database of the job state object to update the job metadata in connection with the further execution of the job while the job state object and the job are at the second compute node.
8. The method as recited in claim 1 , further comprising: at the first compute node, determining a plurality of tasks associated with the job based at least in part on the job state object; at the second compute node, receiving via the network a task of the plurality of tasks; and accessing the database of the job state object to update the job metadata in connection with execution of the task by a second processor of the second compute node.