first steps in big data what is sparkspark is cluster computing framework similar to mapreduce sparkhoweverdoesn' handle the storage of files on the (distributedfile system itselfnor does it handle the resource management for this it relies on systems such as the hadoop file systemyarnor apache mesos hadoop and spark are thus complementary systems for testing and developmentyou can even run spark on your local system how does spark solve the problems of mapreducewhile we oversimplify things bit for the sake of clarityspark creates kind of shared ram memory between the computers of your cluster this allows the different workers to share variables (and their stateand thus eliminates the need to write the intermediate results to disk more technically and more correctly if you're into thatspark uses resilient distributed datasets (rdd)which are distributed memory abstraction that lets programmers perform in-memory computations on large clusters in faulttolerant way because it' an in-memory systemit avoids costly disk operations the different components of the spark ecosystem spark core provides nosql environment well suited for interactiveexploratory analysis spark can be run in batch and interactive mode and supports python oozie (workflow managersqoop (data exchangezookeeper (coordinationflume (log collectorpig (scriptingmahout (machine learninghcatalog hive (metadata(sql enginespark streaming spark sql mllib (machine learninghive (sql enginemapreduce (distributed processing frameworkgraphx (graph dbhbase (columnar storeyarn (cluster resource managementhdfs (hadoop file systemambari (provisioningmanagingand monitoringranger (securityfigure the spark framework when used in combination with the hadoop framework spark has four other large componentsas listed below and depicted in figure spark streaming is tool for real-time analysis spark sql provides sql interface to work with spark mllib is tool for machine learning inside the spark framework graphx is graph database for spark we'll go deeper into graph databases in see