Title: Decentralized Online Learning with Kernels,
Abstract: We consider multi-agent stochastic optimization problems over reproducing
kernel Hilbert spaces (RKHS). In this setting, a network of interconnected
agents aims to learn decision functions, i.e., nonlinear statistical models,
that are optimal in terms of a global convex functional that aggregates data
across the network, with only access to locally and sequentially observed
samples. We propose solving this problem by allowing each agent to learn a
local regression function while enforcing consensus constraints. We use a
penalized variant of functional stochastic gradient descent operating
simultaneously with low-dimensional subspace projections. These subspaces are
constructed greedily by applying orthogonal matching pursuit to the sequence of
kernel dictionaries and weights. By tuning the projection-induced bias, we
propose an algorithm that allows for each individual agent to learn, based upon
its locally observed data stream and message passing with its neighbors only, a
regression function that is close to the globally optimal regression function.
That is, we establish that with constant step-size selections agents' functions
converge to a neighborhood of the globally optimal one while satisfying the
consensus constraints as the penalty parameter is increased. Moreover, the
complexity of the learned regression functions is guaranteed to remain finite.
On both multi-class kernel logistic regression and multi-class kernel support
vector classification with data generated from class-dependent Gaussian mixture
models, we observe stable function estimation and state of the art performance
for distributed online multi-class classification. Experiments on the Brodatz
textures further substantiate the empirical validity of this approach.