In this paper we propose and carefully evaluate a sequence labeling framework which solely utilizes sparse indicator features derived from dense distributed word representations. The proposed model obtains (near) state-of-the art performance for both part-of-speech tagging and named entity recognition for a variety of languages. Our model relies only on a few thousand sparse coding-derived features, without applying any modification of the word representations employed for the different tasks. The proposed model has favorable generalization properties as it retains over 89.8 of its average POS tagging accuracy when trained at 1.2 of the total available training data, i.e. 150 sentences per language.
@cite_16: Sparse coding--that is, modelling data vectors as sparse linear combinations of basis elements--is widely used in machine learning, neuroscience, signal processing, and statistics. This paper focuses on the large-scale matrix factorization problem that consists of learning the basis set in order to adapt it to specific data. Variations of this problem include dictionary learning in signal processing, non-negative matrix factorization and sparse principal component analysis. In this paper, we propose to address these tasks with a new online optimization algorithm, based on stochastic approximations, which scales up gracefully to large data sets with millions of training samples, and extends naturally to various matrix factorization formulations, making it suitable for a wide range of learning problems. A proof of convergence is presented, along with experiments with natural images and genomic data demonstrating that it leads to state-of-the-art performance in terms of speed and optimization for both small and large data sets.
The general goal of sparse coding is to express signals in the form of linear combination of basis vectors and the task of finding an appropriate set of basis vectors is referred to as the dictionary learning problem @cite_16 . Generally, given a data matrix @math with its @math column @math representing the @math @math -dimensional signal, the task is to find @math and @math , such that @math . This can be formalized into an @math -regularized linear least-squares minimization problem having the form with @math being the convex set of matrices that comprise of column vectors having an @math norm at most one, matrix @math acts as the shared dictionary across the signals, and the columns of the sparse matrix @math contains the coefficients for the linear combinations of each of the @math observed signals.