as it allowed the use of commodity hardware to store, access, and process large amounts of data by distributing the storage of data across multiple nodes. Specifically, HDFS achieves its fault tolerance by storing three or more copies of the data across different nodes, where the mapping of the data to nodes is contained in the NameNode, which is redundant itself. Nevertheless, one of the tradeoffs of HDFS is that it utilizes a write-once, read- many paradigm SO that if changes are made on a single data point, the entire file needs to be rewrit- ten. As per some reports in 2016, Yahoo has the largest Hadoop cluster in the world with 35,000 Hadoop servers, hosted across 16 countries, with a combined 600 PB of storage, that execute close to 34 million jobs per month [Dezi16]. 8.2.4.2 Data Ingestion (Kafka, Flume, Sqoop) Before data can be used, it must be ingested from the data sources, which could be structured or unstructured, and processing needs could be batch, micro-batch, or real time as described in Section 8.2.1. The Hadoop ecosystem has a variety of approaches for data ingestion such as Flume, Sqoop, and Kafka. Sqoop is particularly useful for extracting batch data from structured relational databases such as Teradata, Oracle, MySQL, SAP, etc. and Flume is particularly adept at ingest- ing data from multiple sources, which can be unstructured. A typical use for Flume is ingesting streaming log data. Kafka [Kafka], originally developed at LinkedIn, is like Flume; however, it is also a general purpose publish-subscribe model messaging system, which can also ingest data from multiple sources, and simultaneously have multiple consumers of the data. Dezyre [Dezy15] has a nice comparison of Sqoop and Flume, which is resummarized in Table 8.3. While not specifically designed for Hadoop, Kafka is used by some for its ability to add multiple consumers of the data without affecting performance. Kafka achieves this scalability since it does not track which consumers have consumed the messages; it only persists the me