1. A system for scheduling jobs, said system comprising: a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes; the NameNode including at least one processor programmed to: learn central processing unit (CPU) capabilities and disk input/output (I/O) bandwidth capabilities of compute nodes in the cluster; determine an optimal number of containers for each application running on each DataNode of the multitude of DataNodes; and schedule execution of a plurality of tasks on compute nodes of the cluster based on: (i) the learned CPU capabilities and I/O bandwidth capabilities of compute nodes of the cluster, and (ii) at least one of the determined optimal number of containers; wherein the at least one processor is further programmed to schedule the execution of the plurality of tasks on compute nodes by: computing a current under-allocation for an application; and comparing the current under-allocation to a determined optimal number of containers; wherein the at least one processor is further programmed to: determine the optimal number of containers so as to maximize throughput; and calculate the throughout based on a normalized number of completed map tasks, wherein the normalization is done based on sizes of the map tasks.