e and Micro-Batch Processing Using STORM and SPARK STORM is a real-time distributed processing system [STORM], which allows processing of an unbounded sequence of data tuples, such as from a Twitter feed, into a new set of data tuples or event triggers. STORM defines a topology framework consisting of the following: Mapping steps Combine Reduce Baylor=(1) football=(1) is=(1) great=(1) Sub-document 1: 1=(1) Baylor football is great like =(1) at *(1) at =1 Document: like college football college=(1) attends =(1) attends 11 Baylor football is great football=(1) Baylor=(1,1,1) Baylor=3 like college football Split step Map step Combine college=(1,1) Reduce step college=2 Baylor has green football jerseys (Merge,Sort) daughter=(1) daughter=1 My daughter attends college at Baylor Sub-document 2: Baylor=(1) footballe(1,1) football=2 Baylor has green football jerseys has=(1) great=(1) great=1 My daughter attends college at Baylor green=(1) green=(1) green=1 football=(1) (has=(1) has=1 jerseys=(1) 1=(1) My=(1) is=(1) daughter=(1) jerseys=(1) jerseys=1 attends =(1) like=(1) like=1 college =(1) My=(1) at =(1) Baylor=(1) FIGURE 8.4 A simplified illustration of MapReduce to count the number of times a given word occurs in a document. Network Data and Optimization Tuple Spout Tuple Tuple Tuple Tuple Spout Tuple Tuple Tuple FIGURE 8.5 Illustration of an example of the topological data flow for STORM. Stream-the unbounded sequence of tuples Spouts-the sources of the stream (e.g., reading the API from the Twitter feed) Bolts-the process (e.g., functions, filters, aggregations, joins) that takes streams as an input and produces new streams as the output The topological data processing of these framework elements is often illustrated as shown in Figure 8.5. SPARK is an in-memory distributed data analysis and processing platform. SPARK has librar- ies for Machine Learning (MLlib), SQL, DataFrames, Graphs (GraphX), and Streaming. SPARK streaming allows micro-batches to be ingested from HDFS, Flume, Kafka, Twitter, and ZeroMQ in user-d