distributing data storage and processing with frameworks hadoopa framework for storing and processing large data sets apache hadoop is framework that simplifies working with cluster of computers it aims to be all of the following things and morereliable--by automatically creating multiple copies of the data and redeploying processing logic in case of failure fault tolerant --it detects faults and applies automatic recovery scalable--data and its processing are distributed over clusters of computers (horizontal scalingportable--installable on all kinds of hardware and operating systems the core framework is composed of distributed file systema resource managerand system to run distributed programs in practice it allows you to work with the distributed file system almost as easily as with the local file system of your home computer but in the backgroundthe data can be scattered among thousands of servers the different components of hadoop at the heart of hadoop we find distributed file system (hdfsa method to execute programs on massive scale (mapreducea system to manage the cluster resources (yarnon top of thatan ecosystem of applications arose (figure )such as the databases hive and hbase and frameworks for machine learning such as mahout we'll use hive in this hive has language based on the widely used sql to interact with data stored inside the database oozie (workflow managersqoop (data exchangezookeeper (coordinationflume (log collectorpig (scriptingmahout (machine learninghcatalog (metadatamapreduce (distributed processing frameworkhive (sql enginehbase (columnar storeyarn (cluster resource managementhdfs (hadoop file systemambari (provisioningmanagingand monitoringranger (securityfigure sample from the ecosystem of applications that arose around the hadoop core framework