Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive back-propagation nor differentiability. Experiments and visualizations confirmed the effectiveness of the model in terms of performance and representation learning ability.
@cite_1: Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
There is still no universal theory in explaining why a deep model works better than a shallow one. Many of the current attempts for this question are based on the conjecture that it is the hierarchical distributed representations learned from data are the driven forces behind the effectiveness of deep models. Similar works such as conjectured that better representations can be exploited to produce faster-mixing Markov chains, therefore, a deeper model always helps. Tishby and Zaslavsky @cite_1 treated the hidden layers as a successive refinement of relevant information and a deeper structure helps to speed up such process exponentially. Nevertheless, it seems for a deep model to work well, it is critical to obtain a better feature re-representation from intermediate layers.