1. A system for scheduling jobs, said system comprising: a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes; the NameNode including at least one processor programmed to: learn central processing unit (CPU) capabilities and disk input/output (I/O) bandwidth capabilities of compute nodes in the cluster; and schedule execution of a plurality of tasks on compute nodes of the cluster based on the learned CPU capabilities and I/O bandwidth capabilities of compute nodes of the cluster; and wherein the system further includes an optimal number of containers determined for compute nodes in the cluster; wherein the scheduling is further based on the determined optimal number of containers; and wherein the optimal number of containers is at least in part determined by use of the equation: X k * = argmax x k ⁢ ∑ i ⁢ x ki ρ ⁡ ( θ j , π k , L k ) where X k * denotes a vector of containers, ρ denotes a predicted task completion time, θ j denotes a resource requirement, π k denotes a resource capability, and L k denotes a load on a node.