data science in big data world distributed programming framework once you have the data stored on the distributed file systemyou want to exploit it one important aspect of working on distributed hard disk is that you won' move your data to your programbut rather you'll move your program to the data when you start from scratch with normal general-purpose programming language such as cpythonor javayou need to deal with the complexities that come with distributed programmingsuch as restarting jobs that have failedtracking the results from the different subprocessesand so on luckilythe open source community has developed many frameworks to handle this for youand these give you much better experience working with distributed data and dealing with many of the challenges it carries data integration framework once you have distributed file system in placeyou need to add data you need to move data from one source to anotherand this is where the data integration frameworks such as apache sqoop and apache flume excel the process is similar to an extracttransformand load process in traditional data warehouse machine learning frameworks when you have the data in placeit' time to extract the coveted insights this is where you rely on the fields of machine learningstatisticsand applied mathematics before world war ii everything needed to be calculated by handwhich severely limited the possibilities of data analysis after world war ii computers and scientific computing were developed single computer could do all the counting and calculations and world of opportunities opened ever since this breakthroughpeople only need to derive the mathematical formulaswrite them in an algorithmand load their data with the enormous amount of data available nowadaysone computer can no longer handle the workload by itself in factseveral algorithms developed in the previous millennium would never terminate before the end of the universeeven if you could use every computer available on earth this has to do with time complexity (to break password by testing every possible combination an example can be found at scale well with the amount of data we need to analyze todaythis becomes problematicand specialized frameworks and libraries are required to deal with this amount of data the most popular machine-learning library for python is scikit-learn it' great machine-learning toolboxand we'll use it later in the book there areof courseother python librariespybrain for neural networks--neural networks are learning algorithms that mimic the human brain in learning mechanics and complexity neural networks are often regarded as advanced and black box