distributing data storage and processing with frameworks input files each line passed to mapper every key gets mapped to value of keys get sorted (or shuffledreduce (aggregatekey-value pairs collect output to file green blue greenblueblueorange greenredblueorange blue blue greenblue blue blue blueorange orange green green greenred green green red blueorange red red blue orange orange orange blue blue green red orange orange figure an example of mapreduce flow for counting the colors in input texts the whole process is described in the following six steps and depicted in figure reading the input files passing each line to mapper job the mapper job parses the colors (keysout of the file and outputs file for each color with the number of times it has been encountered (valueor more technically saidit maps key (the colorto value (the number of occurrencesthe keys get shuffled and sorted to facilitate the aggregation the reduce phase sums the number of occurrences per color and outputs one file per key with the total number of occurrences for each color the keys are collected in an output file note while hadoop makes working with big data easysetting up good working cluster still isn' trivialbut cluster managers such as apache mesos do ease the burden in realitymany (mid-sizedcompanies lack the competence to maintain healthy hadoop installation this is why we'll work with the hortonworks sandboxa pre-installed and configured hadoop ecosystem installation instructions can be found in section an introductory working example of hadoop nowkeeping the workings of hadoop in mindlet' look at spark sparkreplacing mapreduce for better performance data scientists often do interactive analysis and rely on algorithms that are inherently iterativeit can take awhile until an algorithm converges to solution as this is weak point of the mapreduce frameworkwe'll introduce the spark framework to overcome it spark improves the performance on such tasks by an order of magnitude