We study the problem of agnostically learning halfspaces which is defined by a fixed but unknown distribution @math on @math . We define @math as the least error of a halfspace classifier for @math . A learner who can access @math has to return a hypothesis whose error is small compared to @math . Using the recently developed method of the author, Linial and Shalev-Shwartz we prove hardness of learning results under a natural assumption on the complexity of refuting random @math - @math formulas. We show that no efficient learning algorithm has non-trivial worst-case performance even under the guarantees that @math for arbitrarily small constant @math , and that @math is supported in @math . Namely, even under these favorable conditions its error must be @math for every @math . In particular, no efficient algorithm can achieve a constant approximation ratio. Under a stronger version of the assumption (where @math can be poly-logarithmic in @math ), we can take @math for arbitrarily small @math . Interestingly, this is even stronger than the best known lower bounds (Arora et. al. 1993, Feldamn et. al. 2006, Guruswami and Raghavendra 2006) for the case that the learner is restricted to return a halfspace classifier (i.e. proper learning).
@cite_61: In this paper, we study the problem of learning in the presence of classification noise in the probabilistic learning model of Valiant and its variants. In order to identify the class of “robust” learning algorithms in the most general way, we formalize a new but related model of learning from statistical queries . Intuitively, in this model a learning algorithm is forbidden to examine individual examples of the unknown target function, but is given acess to an oracle providing estimates of probabilities over the sample space of random examples. One of our main results shows that any class of functions learnable from statistical queries is in fact learnable with classification noise in Valiant's model, with a noise rate approaching the information-theoretic barrier of 1 2. We then demonstrate the generality of the statistical query model, showing that practically every class learnable in Valiant's model and its variants can also be learned in the new model (and thus can be learned in the presence of noise). A notable exception to this statement is the class of parity functions, which we prove is not learnable from statistical queries, and for which no noise-tolerant algorithm is known.
@cite_20: We give a computationally efficient algorithm that learns (under distributional assumptions) a halfspace in the difficult agnostic framework of Kearns, Schapire, and Sellie [Mach. Learn., 17 (1994), pp. 115-141], where a learner is given access to a distribution on labelled examples but where the labelling may be arbitrary (similar to malicious noise). It constructs a hypothesis whose error rate on future examples is within an additive @math of the optimal halfspace, in time poly @math for any constant @math , for the uniform distribution over @math or unit sphere in @math as well as any log-concave distribution in @math . It also agnostically learns Boolean disjunctions in time @math with respect to any distribution. Our algorithm, which performs @math polynomial regression, is a natural noise-tolerant arbitrary-distribution generalization of the well-known “low-degree” Fourier algorithm of Linial, Mansour, and Nisan. We observe that significant improvements on the running time of our algorithm would yield the fastest known algorithm for learning parity with noise, a challenging open problem in computational learning theory.
@cite_11: 
Statistical queries (SQ) algorithms @cite_61 is a class of learning algorithms whose interaction with @math is done only via statistical queries . Concretely, an SQ-algorithm can specify any function @math and an error parameter @math , and receive a number @math satisfying @math . In this model, an algorithm is efficient if it makes polynomially many queries with error parameters satisfying @math (besides that, the algorithm is not restricted). While this class is strictly smaller than the class of all algorithms, most known algorithms admit an SQ version. In addition, as opposed to general algorithms, unconditional lower bounds are known for SQ algorithms @cite_11 for several learning problems. In particular, it is known that it is hard to agnostically learn halfspaces using SQ-algorithm (e.g. @cite_20 ). We note that as with previously known lower bounds for general algorithms, these results only rule out exact algorithms.