In this paper, we propose a deep learning framework based on randomized neural network. In particular, inspired by the principles of Random Vector Functional Link (RVFL) network, we present a deep RVFL network (dRVFL) with stacked layers. The parameters of the hidden layers of the dRVFL are randomly generated within a suitable range and kept fixed while the output weights are computed using the closed form solution as in a standard RVFL network. We also propose an ensemble deep network (edRVFL) that can be regarded as a marriage of ensemble learning with deep learning. Unlike traditional ensembling approaches that require training several models independently from scratch, edRVFL is obtained by training a single dRVFL network once. Both dRVFL and edRVFL frameworks are generic and can be used with any RVFL variant. To illustrate this, we integrate the deep learning networks with a recently proposed sparse-pretrained RVFL (SP-RVFL). Extensive experiments on benchmark datasets from diverse domains show the superior performance of our proposed deep RVFL networks.
@cite_23: A system architecture and a network computational approach compatible with the goal of devising a general-purpose artificial neural network computer are described. The functionalities of supervised learning and optimization are illustrated, and cluster analysis and associative recall are briefly mentioned. >
A basic framework of the standard RVFL network @cite_23 is shown in Fig. (a). The inputs to the output layer in RVFL consist of both non-linearly transformed features @math from the hidden layer and original input features @math . If @math be the input data features and @math be the number of hidden nodes, then there are total @math inputs for each output node. Since the hidden layer parameters are randomly generated and kept fixed during the training phase, only the output weights @math need to be computed. Thus, the resulting optimization problem can be mathematically represented as: where @math is the concatenation of hidden features and original features, @math is the regularization parameter and @math is the target vector.