Title: A Big Data Analysis Framework Using Apache Spark and Deep Learning,
Abstract: With the spreading prevalence of Big Data, many advances have recently been
made in this field. Frameworks such as Apache Hadoop and Apache Spark have
gained a lot of traction over the past decades and have become massively
popular, especially in industries. It is becoming increasingly evident that
effective big data analysis is key to solving artificial intelligence problems.
Thus, a multi-algorithm library was implemented in the Spark framework, called
MLlib. While this library supports multiple machine learning algorithms, there
is still scope to use the Spark setup efficiently for highly time-intensive and
computationally expensive procedures like deep learning. In this paper, we
propose a novel framework that combines the distributive computational
abilities of Apache Spark and the advanced machine learning architecture of a
deep multi-layer perceptron (MLP), using the popular concept of Cascade
Learning. We conduct empirical analysis of our framework on two real world
datasets. The results are encouraging and corroborate our proposed framework,
in turn proving that it is an improvement over traditional big data analysis
methods that use either Spark or Deep learning as individual elements.