efined intervals into what is called resilient distributed datasets (RDD). RDDs are immutable distributed collections of objects, which can be operated on in parallel by storing the state of mem- ory as an object, which can be shared across jobs. Consequently, this inherently allows speeding up batch processing jobs SO that MapReduce jobs can essentially be done in-memory. SPARK as such has been known to have speed ups over batch jobs by 10-100 times [Tutor16]. 8.2.4.8 Coordination and Workflow Management As with any system, administrative and coordination solutions are needed. This is especially true with a distributed system with hundreds of servers, and Zookeeper [Zoo16] is one solution for a coordination service in the Hadoop ecosystem. Zookeeper is a centralized repository where distributed applications can place and extract data, which is used to coordinate processing. For example, with a huge number of machines on a network, race conditions can otherwise occur where a machine tries to perform multiple operations at a time, or deadlocks can happen when two or more machines try to access the same data simultaneously. Zookeeper acts as the coor- dinator and can serialize operations to avoid race conditions and it provides synchronization to avoid deadlocks. In addition to Zookeeper, Oozie and Jenkins are two examples of open source workflow manag- ers that help track the progress and status of an end-to-end data processing flow. For example, it is not uncommon that a given data result that needs to be updated regularly is the result of many different processes that need to be run. Figure 8.6 provides an example workflow using a directed acyclic graph, which shows the flow of data, starting with the ingesting of data with Kafka, followed Building the Network of the Future MR job Start Kafka Pig job SPARK FIGURE 8.6 Example of workflow to be managed. by a Pig job, which is then processed in parallel by a MapReduce job, a SPARK job, and a HIVE job, after which the three are joined or summarized. The workflow man