Title: Dropout as a Low-Rank Regularizer for Matrix Factorization,
Abstract: Regularization for matrix factorization (MF) and approximation problems has
been carried out in many different ways. Due to its popularity in deep
learning, dropout has been applied also for this class of problems. Despite its
solid empirical performance, the theoretical properties of dropout as a
regularizer remain quite elusive for this class of problems. In this paper, we
present a theoretical analysis of dropout for MF, where Bernoulli random
variables are used to drop columns of the factors. We demonstrate the
equivalence between dropout and a fully deterministic model for MF in which the
factors are regularized by the sum of the product of squared Euclidean norms of
the columns. Additionally, we inspect the case of a variable sized
factorization and we prove that dropout achieves the global minimum of a convex
approximation problem with (squared) nuclear norm regularization. As a result,
we conclude that dropout can be used as a low-rank regularizer with data
dependent singular-value thresholding.