Evolutionary Learning proceeds by evolving a population of classifiers, from which it generally returns (with some notable exceptions) the single best-of-run classifier as final result. In the meanwhile, Ensemble Learning, one of the most efficient approaches in supervised Machine Learning for the last decade, proceeds by building a population of diverse classifiers. Ensemble Learning with Evolutionary Computation thus receives increasing attention. The Evolutionary Ensemble Learning (EEL) approach presented in this paper features two contributions. First, a new fitness function, inspired by co-evolution and enforcing the classifier diversity, is presented. Further, a new selection criterion based on the classification margin is proposed. This criterion is used to extract the classifier ensemble from the final population only (Off-line) or incrementally along evolution (On-line). Experiments on a set of benchmark problems show that Off-line outperforms single-hypothesis evolutionary learning and state-of-art Boosting and generates smaller classifier ensembles.
@cite_1: In an earlier paper, we introduced a new "boosting" algorithm called AdaBoost which, theoretically, can be used to significantly reduce the error of any learning algorithm that con- sistently generates classifiers whose performance is a little better than random guessing. We also introduced the related notion of a "pseudo-loss" which is a method for forcing a learning algorithm of multi-label concepts to concentrate on the labels that are hardest to discriminate. In this paper, we describe experiments we carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems. We performed two sets of experiments. The first set compared boosting to Breiman's "bagging" method when used to aggregate various classifiers (including decision trees and single attribute- value tests). We compared the performance of the two methods on a collection of machine-learning benchmarks. In the second set of experiments, we studied in more detail the performance of boosting using a nearest-neighbor classifier on an OCR problem.
@cite_6: This article deals with an improvement for genetic programming based on a technique originating from the machine learning field: boosting. In a first part of this paper, we test the improvements offered by boosting on binary problems. Then we propose to deal with regression problems, and propose an algorithm, called GPboost, that keeps closer to the original idea of distribution in Adaboost than what has been done in previous implementation of boosting for genetic programming.
@cite_2: The decomposition of regression error into bias and variance terms provides insight into the generalization capability of modeling methods. The paper offers an introduction to bias variance decomposition of mean squared error, as well as a presentation of experimental results of the application of genetic programming. Finally ensemble methods such as bagging and boosting are discussed that can reduce the generalization error in genetic programming.
The use of Evolutionary Algorithms as weak learners with -in a standard Bagging or Boosting approach has also been investigated. Boosting approaches for GP have been applied for instance to classification @cite_6 or symbolic regression @cite_2 : each run delivers a GP tree minimizing the weighted sum of the training errors, and the weights were computed as in standard Boosting @cite_1 . While such ensembles of GP trees result, as expected, in a much lower variance of the performance, they do not fully exploit the population-based nature of GP, as independent runs are launched to learn successive classifiers.