jobs and Quality of service (QoS) for production jobs. The fair scheduler has three basic concepts. Jobs are grouped into pools. Each pool is assigned a guaranteed minimum share. Excess capacity is split between jobs. By default, jobs that are uncategorized go into a default pool. Pools have to specify the minimum number of map slots, reduce slots, as well as a limit on the number of running jobs. Capacity scheduler The capacity scheduler was developed by Yahoo. The capacity scheduler supports several features that are similar to those of the fair scheduler. Queues are allocated a fraction of the total resource capacity. Free resources are allocated to queues beyond their total capacity. Within a queue, a job with a high level of priority has access to the queue's resources. There is no preemption once a job is running. Difference between Hadoop 1 and Hadoop 2 (YARN) The biggest difference between Hadoop 1 and Hadoop 2 is the addition of YARN (Yet Another Resource Negotiator), which replaced the MapReduce engine in the first version of Hadoop. YARN strives to allocate resources to various applications effectively. It runs two daemons, which take care of two different tasks: the resource manager, which does job tracking and resource allocation to applications, the application master, which monitors progress of the execution. Difference between Hadoop 2 and Hadoop 3 There are important features provided by Hadoop 3. For example, while there is one single namenode in Hadoop 2, Hadoop 3 enables having multiple name nodes, which solves the single point of failure problem. In Hadoop 3, there are containers working in principle of Docker, which reduces time spent on application development. One of the biggest changes is that Hadoop 3 decreases storage overhead with erasure coding. Also, Hadoop 3 permits usage of GPU hardware within the cluster, which is a very substantial benefit to execute deep learning algorithms on a Hadoop cluster. Other applications The HDFS is not restricted to MapReduce jobs. It can be used for other applications, many of which are under development at Apache. The list includes the HBase database, the Apache Mahout machine learning system, and the Apache Hive data warehouse. Theoretically, Hadoop could be used for any workload that is batch-oriented rather than real-time, is very data-intensive, and benefits from parallel processing. It can also be used to complement a real-time system, such as lambda architecture, Apache Storm, Flink, and