In this paper, we propose a deep learning framework based on randomized neural network. In particular, inspired by the principles of Random Vector Functional Link (RVFL) network, we present a deep RVFL network (dRVFL) with stacked layers. The parameters of the hidden layers of the dRVFL are randomly generated within a suitable range and kept fixed while the output weights are computed using the closed form solution as in a standard RVFL network. We also propose an ensemble deep network (edRVFL) that can be regarded as a marriage of ensemble learning with deep learning. Unlike traditional ensembling approaches that require training several models independently from scratch, edRVFL is obtained by training a single dRVFL network once. Both dRVFL and edRVFL frameworks are generic and can be used with any RVFL variant. To illustrate this, we integrate the deep learning networks with a recently proposed sparse-pretrained RVFL (SP-RVFL). Extensive experiments on benchmark datasets from diverse domains show the superior performance of our proposed deep RVFL networks.
@cite_4: A long and uncertain training process is one of the most important problems for a multilayer neural network using the backpropagation algorithm. In this paper, a modified backpropagation algorithm for a certain and fast training process is presented. The modification is based on the solving of the weight matrix for the output layer using theory of equations and least squares techniques. >
Typically, Eq. can be solved via a closed form solution using either ridge regression (i.e. @math ) or Moore-Penrose pseudoinverse (i.e. @math ) @cite_4 . Using Moore-Penrose pseudoinverse, the solution is given by: @math while using the regularized least squares (or ridge regression), the closed form solution is given by: