We propose learning deep models that are monotonic with respect to a user-specified set of inputs by alternating layers of linear embeddings, ensembles of lattices, and calibrators (piecewise linear functions), with appropriate constraints for monotonicity, and jointly training the resulting network. We implement the layers and projections with new computational graph nodes in TensorFlow and use the ADAM optimizer and batched stochastic gradients. Experiments on benchmark and real-world datasets show that six-layer monotonic deep lattice networks achieve state-of-the art performance for classification and regression with monotonicity guarantees.
@cite_4: Incorporating prior knowledge of a particular task into the architecture of a learning algorithm can greatly improve generalization performance. We study here a case where we know that the function to be learned is non-decreasing in its two arguments and convex in one of them. For this purpose we propose a class of functions similar to multi-layer neural networks but (1) that has those properties, (2) is a universal approximator of Lipschitz functions with these and other properties. We apply this new class of functions to the task of modelling the price of call options. Experiments show improvements on regressing the price of call options using the new types of function classes that incorporate the a priori constraints.
@cite_7: A function estimator MSQUID is presented for fitting and bounding noisy data that are known to be monotonic. MSQUID augments a “backpropagation” neural network model with a set of constraints which restricts the model to monotonic functions. Model parameters are estimated using nonlinear, constrained optimization at little increase in computation over standard neural networks. It is proven that MSQUID can estimate any monotonic function and produces more accurate estimates than unconstrained optimization. These monotonic functions and their confidence bounds can be used in many fault detection and diagnosis systems.
@cite_8: 
@cite_9: 
@cite_1: In real world statistical applications a large class of univariate probability distributions are unimodal. This paper presents a neural network approach to probability density estimation for univariate unimodal data. This approach is superior to the traditional statistical density estimation methods in that neither of the forms of probability function (as in the parametric methods) or the kernel functions (as in the non-parametric methods) is needed to form an a prioriassumption. An example is used to demonstrate that the proposed approach is an effective method for estimating univariate unimodal probability distributions.
@cite_0: Neural network techniques are widely used in solving pattern recognition or classification problems. However, when statistical data are used in supervised training of a neural network employing the back-propagation least mean square algorithm, the behavior of the classification boundary during training is often unpredictable. This research suggests the application of monotonicity constraints to the back propagation learning algorithm. When the training sample set is preprocessed by a linear classification function, neural network performance and efficiency can be improved in classification applications where the feature vector is related monotonically to the pattern vector. Since most classification problems in business possess monotonic properties, this technique is useful in those problems where any assumptions about the properties of the data are inappropriate.
@cite_16: Neural networks applied in control loops and safety-critical domains have to meet more requirements than just the overall best function approximation. On the one hand, a small approximation error is required; on the other hand, the smoothness and the monotonicity of selected input-output relations have to be guaranteed. Otherwise, the stability of most of the control laws is lost. In this article we compare two neural network-based approaches incorporating partial monotonicity by structure, namely the Monotonic Multi-Layer Perceptron (MONMLP) network and the Monotonic MIN-MAX (MONMM) network. We show the universal approximation capabilities of both types of network for partially monotone functions. On a number of datasets, we investigate the advantages and disadvantages of these approaches related to approximation performance, training of the model and convergence.
@cite_10: 
Learning monotonic single-layer neural nets by constraining the neural net weights to be positive dates back to Archer and Wang in 1993 @cite_0 , and that basic idea has been re-visited by others @cite_1 @cite_7 @cite_4 @cite_16 , but with some negative results about the obtainable flexibility even with multiple hidden layers @cite_8 . Sill @cite_10 proposed a three-layer monotonic network that used an early form of monotonic linear embedding and max-and-min-pooling. Daniels and Velikova @cite_8 extended Sill's result to learn a partial monotonic function by combining min-max-pooling, also known as adaptive logic networks @cite_9 , with partial monotonic linear embedding, and show that their proposed architecture is an universal approximator for partial monotone functions. None of these prior neural networks were demonstrated on problems with more than @math features, nor trained on more than a few thousand examples. For our experiments we implemented a positive neural network and a min-max-pooling network with TensorFlow.