> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <

1

Online Tool Condition Monitoring Based on
Parsimonious Ensemble+
Mahardhika Pratama, Member, IEEE, Eric Dimla, Member, IEEE, Edwin Lughofer, Witold Pedrycz,
Fellow, IEEE, Tegoeh Tjahjowidodo

Abstract— Tool condition monitoring plays a vital role to reduce
maintenance cost in manufacturing industry. It generates
information about possible unsafe states of machining process
before they occur such that faults of component can be predicted
early to support management of its life-cycle. The faults could be
isolated to minimize catastrophic effects leading to a complete
shutdown of overall production process. Accurate diagnosis of tool
wear in metal turning process remains an open challenge for both
scientists and industrial practitioners because of inhomogeneities
in work-piece material, non-stationary machining settings to suit
production requirements, and non-linear relations between
measured variables and tool wear. Common methodologies for
tool condition monitoring still rely on batch approaches which
cannot cope with a fast sampling rate of metal cutting process.
Furthermore they require a retraining process to be completed
from scratch when dealing with a new set of machining
parameters. This paper presents an online tool condition
monitoring approach based on Parsimonious Ensemble+
(pENsemble+). The unique feature of pENsemble+ lies in its highly
flexible principle where both ensemble structure and baseclassifier structure can automatically grow and shrink on the fly
based on the characteristics of data streams. Moreover, the online
feature selection scenario is integrated to actively sample relevant
input attributes. The paper presents advancement of a newly
developed ensemble learning algorithm, pENsemble, where online
active learning scenario is incorporated to reduce operator’s
labelling effort. The ensemble merging scenario is proposed which
allows reduction of ensemble complexity while retaining its
diversity.
Experimental
studies
utilising
real-world
manufacturing data streams and comparisons with well-known
algorithms were carried out. Furthermore, the efficacy of
pENsemble+ was examined using benchmark concept drift data
streams. It has been found that pENsemble+ incurs low structural
complexity and results in a significant reduction of operator’s
labelling effort.
Index Terms— Prognostic Health Management, Online
Learning, Ensemble Classifier, Lifelong Learning, Nonstationary
Environments, Concept Drifts

I. INTRODUCTION

T

OOL condition monitoring (TCM) aims to feed real-time
information of tool condition for the so-called maintenance

The first author acknowledges the support of NTU start-up grant. The third
author acknowledges the support of the Austrian COMET-K2 programme of
the Linz Center of Mechatronics (LCM), funded by the Austrian federal
government and the federal state of Upper Austria. The fourth author
acknowledges the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC) and Canada Research Chair (CRC) in
Computational Intelligence. Mahardhika Pratama is with School of Computer
Science and Engineering, Nanyang Technological University (e-mail:
mpratama@ntu.edu.sg). Eric Dimla is with the Mechanical Engineering

on-demand framework where tool is replaced at the right time
only [1]. This paradigm brings significant cost saving to the
industry because replacing sharp tools too early and too often
incurs frequent shutdown of the machining process and leads to
a dramatic increase of tool costs whereas worn tool potentially
damages the surface finishing and dimensional integrity of
work piece [2] and intensifies vibration level of the cutting
process. When the tool is no longer at desired functionality or
blunt, it entails high cutting force resulting in expensive energy
cost [3]. In a nutshell, the success of TCM allows advanced
scheduling of maintenance activities, proactive allocation of
replacement parts and enhanced fleet deployment decisions
based on the estimated progression of component life
consumption.
The TCM usually involves two tasks, namely sensing and
monitoring [5]. Sensing is a phase used to capture cutting
signals from a set of sensors in the TCM. Sensing itself can be
further classified into two types of modes, namely direct and
indirect. Indirect sensing is committed after completing the
cutting process through optical measurement, surface-finishing
measurement, and chip-size measurement, etc. The indirect
sensing is, nevertheless, not compatible with the online tool
condition monitoring which must take place while engaging the
cutting process without intermittent stoppage only because it
imposes unavoidable time loss for the sake of measurement and
does not prevent production waste [6]. The direct sensing, on
the other hand, relies on correlated process variables such as:
vibration, spindle current, acoustic emission, and force to
determine tool wear. All of which can be collected directly
through sensor and data acquisition unit installation at the
correct position [5]. Among the other three, the force signal is
well-known to be the most-correlated variable to the tool wear
because a higher cutting force is required when the tool is blunt.
The second phase, namely monitoring, aims to perform
predictive analytics from the measured signals. Existing
monitoring approaches are categorized into three groups [4],
namely first principle, data-driven and hybrid. The first

Department, Faculty of Engineering, Boulder, University Teknologi Brunei,
Brunei Darussalam (e-mail: dimla@utb.edu.bn). Edwin Lughofer is with the
Department of Knowledge-Based and Mathematical Systems, Johanes Kepler
University (email: Edwin.lughofer@jku.edu.at). Witold Pedrycz is with the
Department of Electrical and Computer Engineering, University of Alberta,
Canada (email: wpedrycz@ualberta.ca). Tegoeh Tjahjowidodo is with School
of Mechanical and Aerospace Engineering, Nanyang Technological University,
Singapore (email: ttegoeh@ntu.edu.sg).

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
principle approach usually derives the exact mathematical
model of a degrading system including its component. This
approach is, however, problem-specific and sometimes it is too
hard to be applied due to intricate and inter-related nature of a
manufacturing system. This approach is impractical because of
the fact that machining process is highly influenced by a
number of dynamic factors: temperature, cutting fluids, chip
formation, workpiece and tool materials, etc. [1]. Data-driven
approach offers an alternative of the former one where
predictive analytics is purely done using input/output data
recorded by a number of sensors and a set of data acquisition
unit. This approach makes use of intelligent techniques which
emulate dynamics of tool condition through a “learning”
process of manufacturing data. Tool wear progression is
measured by considering the plane-faced tool geometry
approach with flank wear as the underlying variable of the tool
life. The intelligent approaches feature generalization capability
where it can be deployed to monitor tool condition in the realtime mode while engaging the cutting process once a model has
been created from the learning process. The hybrid approach
combines both the first-principle and data-driven approaches
where the two are executed in parallel to take advantages of the
strength of both approaches. The downside of this approach is
obvious because of its expensive computational cost [6].
Recent progress in the TCM research has reported that the
data-driven approach has gained increasing popularity in the
community [12]-[15] because it can be deployed with a verylittle capital expenditure [12]. It is done using exclusively
sensory data and does not require complicated pre-setting
requirements or assumption and/or simplification which is
inherent in the first principle approach. The data-driven TCM
method still requires more advanced data analytics because of
at least three reasons: 1) existing approaches rely on a batch
learning approach which is not fully compatible for online realtime processing. It requires a complete dataset covering all
possible situations in the monitoring process and a complete
retraining from scratch when observing a new pattern in the
monitoring process; 2) existing approaches are constructed
under a static structure predetermined before process runs. Such
approaches are not self-adaptive, thereby being unable to adapt
to variations of machining parameters. It is also evident that the
machining process is often affected by external disturbances
which lead to previously learned concept to be invalid; 3)
existing approaches are mostly designed under crisp and certain
hidden nodes which do not cope with imprecision, uncertainty
and a noisy nature of machining processes.
As more and more industries have integrated the so-called
Internet-of-Things (IoT) in the current trend of automation and
data exchange in the so called Industry 4.0 (4 th industrial
revolution), this calls for advancements of existing predictive
maintenance to cope with online and dynamic characteristics of
machining process [12]. The concept of Evolving Intelligent
Systems (EIS) [42], [43] provides promising approach for
online predictive maintenance because it features two important
properties: online learning, dynamic and evolving structure.
This trait is appealing because it is suitable for real-time
deployment under modest computational resources. The

2

dynamic and evolving structure of EIS is capable of tracking
any variation of data streams, which prevents loss of
generalization capabilities in the presence of shift or drift in data
streams. The EIS research area has grown rapidly as manifested
by extensions and variations of EISs encountered in the
literature [44]-[50]. Vast majority of existing EISs are
constructed under a single base-model where the evolving
nature is generated from automatic partitioning of input and
output space with fuzzy rule, neuron, etc [44]-[50]. It is
understood that the ensemble paradigm is capable of improving
model’s generalization because the classification decision is
drawn from a collection of local experts. The ensemble method
handles the bias-variance dilemma better than its single model
counterpart provided that local experts exhibit good diversity.
This advantage is normally achieved when incorporating weak
local experts. Despite being already mature as reported in the
literature [51]-[53], most works in the ensemble learning
scenario utilize non-evolving or even batched base classifier.
This results in costly computational overhead and memory
burdens. The use of evolving base-classifier helps the ensemble
classifier to be more robust to deal with the local concept drift
because it offers better exploration in the local region than those
static classifiers. The local concept drift refers to a situation
where drift only occurs in some local regions only with
different intensities and speeds. Few works in the literature
[47]-[50] have incorporated evolving base classifiers under
different ensemble configurations: bagging, boosting and
stacking. These works are, however, crafted under a static
ensemble structure which cannot adapt to the concept drift.
Moreover, they suffer from the absence of drift detection
scenario which identifies the presence of concept drift. It is
worth noting that drift detection becomes vital in practice
because not only the algorithm has to adapt to the drift but also
it has to inform when the change occurs to allow in-depth
analysis of system’s behavior.
This paper presents a novel data-driven tool condition
monitoring methodology benefiting from recent progress in the
area of data stream analytics. An evolving ensemble classifier,
namely Parsimonious Classifier+ (pENsemble+) is put forward.
pENsemble+ handles aforementioned limitations because of the
fact that it works fully in the single-pass fashion where data are
directly discarded once learned without the requirement of
secondary memory or archival storage. Furthermore, it adopts a
fully evolving working scenario where both ensemble structure
and base-classifier structure can be automatically generated and
pruned from data streams. Moreover, the underlying innovation
of pENsemble+ compared to its root, pENsemble [9] is implied
by two facts: 1) pENsemble+ is equipped by the online active
learning scenario which actively selects the training samples for
model updates. This trait is vital for online tool condition
monitoring because it relieves operator’s annotation effort; 2)
pENsemble+ introduces the notion of “ensemble merging
scenario” which aims to maintain ensemble complexity in the
low level while improving diversity of the ensemble classifier.
This strategy enhances the significance-based pruning
technique existing in the literature [10] which often
compromises the model’s diversity.

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
pENsemble+ is constructed with a generalized version of
Dynamic Weighted Majority [11] which puts forward an open
ensemble structure. Unlike the original DWM [11] and its
extension in [41], pENsemble+ is equipped by an online active
learning scenario which automatically selects training samples
for model updates based on the Bayesian conflict measure
which analyses conflict level in both feature space and target
space. The online active learning scenario utilizes a dynamic
threshold mechanism which copes with rapidly changing
environments. In realm of tool condition monitoring problem,
the online active learning scenario resolves the major
bottleneck of supervised learner which happens to be overdependent on operator’s feedback. pENsemble+’s structure is
automatically generated using the drift detection method
devised with the concept of Hoeffding bound [16].
pENsemble+ adopts the penalty and reward scenario where the
base-classifier is punished when making misclassification,
whereas a reward is granted provided it returns correct
prediction. The reward scenario is an additional phase in respect
to the original DWM which is meant to retain diversity of
ensemble classifier. It is worth-noting that the strength of
ensemble classifier compared to the single classifier lies in the
diversity aspect which addresses the bias and variance dilemma
better than the single classifier and is more robust against
various forms of uncertainty such as noise, etc. This case,
however, must be interpreted with care in the data stream
context which happens to be non-stationary because outdated
or irrelevant classifier undermines final predictive decision.
Complexity reduction scenario is incorporated with the
ensemble merging scenario which focuses on the redundancy
issue. The redundancy issue is analyzed by inspecting the
mutual information of base-classifiers. Two classifiers having
significant amount of mutual information are merged. The
ensemble merging scenario offers plausible tradeoff between
diversity and simplicity since it does not scan through poor
classifiers rather focus on those redundant classifiers. Another
unique feature of pENsemble+ is shown in its online feature
selection scenario [17] which dynamically samples relevant
input attributed during the training process. This mechanism
assigns numeric weight (0 or 1) for every input attribute in
every training observation and allows to arrive at different
subsets of input attributes in the training process. pENsemble+
deploys an evolving fuzzy classifier, namely pClass as a local
expert [18] which adopts an open structure. This provides
additional flexibility in the base-classifier level. Furthermore,
pENsemble+ will be implemented under the two variants of
pClass, namely axis-parallel and multivariate. The difference
between the two lies in the rule premise where the axis-parallel
rule only exploits the Gaussian rule with diagonal inverse
covariance matrix, while the multivariate rule features more
advanced version than the former one with a non-diagonal
inverse covariance matrix.
The major contributions of this paper are outlined as follows:
1) the paper puts forward a new perspective for online tool
condition monitoring approach based on a novel evolving
ensemble classifier. The unique features of our approach are
seen in its capabilities in handling the three bottlenecks of

3

existing data-driven TCM, time and space complexity, concept
drifts, and data uncertainty;
2) A novel ensemble classifier, namely pENsemble+, is
proposed. This algorithm goes one step ahead when compared
to existing ensemble classifier where an evolving classifier is
deployed as a base-classifier. Two variants of pClass, namely
axis-parallel and multivariate, are used as the base-classifier;
3) pENsemble+ also features online feature selection which is
capable of extracting relevant input attributes on demand;
4) Another unique feature of pENsemble+ is shown in the
online active learning scenario automatically sampling relevant
samples for model updates and the ensemble merging scenario
offering complexity reduction without compromising diversity
of the ensemble classifier;
5) A real-world experiment in the metal-turning process was
carried out where real-world manufacturing data were collected
and preprocessed.
It turns out that the flank-wear is not the only factor
affecting tool life [5] and the use of chip-breaker geometry
inserts greatly affects the mechanics of machining. Our
experiment was done with two types of tool inserts with chip
breaker geometry, coated and uncoated grades. These tools
were utilized in cutting EN24T steel while predictive analytics
of tool condition were undertaken with three sensory variables,
cutting force, dynamic cutting force and vibration. The
underlying goal was to identify cutter condition on the fly. Our
experiment used a fresh insert and continued until the worn
state without artificial actions to wear the tool. The efficacy of
pENsemble+ was experimentally validated in the online tool
condition monitoring of the metal cutting process and was
compared with a number of recently published algorithms:
Learn++.NSE [19], Learn++.CDS [20], pENsemble [9] and
pClass [18]. Additional numerical results were also served
using popular concept drift problems in the literature. The
advantage of pENsemble+ was evident in our experimental
study where it attained significant improvement in time, space
and sample complexity even compared to a single classifier
without substantial compromise on accuracy.
The paper is structured as follows: Section 2 encompasses
learning policy of pENsemble+ and learning procedure of its
base classifier, pClass; Section 3 outlines experimental
procedure and machining setup; Section 4 elaborates on
numerical results; and some concluding remarks are drawn in
the last section of the paper.
II. LEARNING POLICY OF PENSEMBLE+
This section elaborates on fundamental working principle of
pENsemble+ including ensemble learning mechanism and
learning scenario of the base-classifier. An Overview of
pENsemble+ learning mechanism is depicted in Fig. 1.
pENsemble+ executes data streams on a chunk by chunk basis
where each data chunk is fed to the online active learning
scenario which is meant to shrink data chunk size and to relieve
operator’s labelling effort. The learning process continues with
the online feature selection scenario which assigns numeric
feature weights (0 or 1) for every input attributes. The
performance of base classifier is evaluated based on its

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
predictive performance reported on a new observation where
misclassification triggers a penalty reducing its voting’s weight
while reward augments it. The complexity reduction scenario is

4

implemented through the ensemble merging scenario. The drift
detection scenario determines the learning stage which governs
the stability and plasticity of the ensemble classifier,

Fig. 1 Learning Policy of pENsemble+

A. Parsimonious Classifier (pClass)
pENsemble+ deploys a newly developed evolving classifier,
namely pClass as a base-classifier in order to attains greater
flexibility in handling concept drift in individual local regions.
This phenomenon is known as the local concept drift in the
literature. It is evident that concept change applies to particular
input regions only with different rates and severities.
 Fuzzy Rule of pClass: pClass is a class of first-order evolving
fuzzy classifiers constructed with Takagi Sugeno Kang (TSK)
fuzzy system where the rule consequent implements a firstorder linear function while the rule premise is built upon the
multivariate Gaussian function generating non-axis-parallel
ellipsoidal clusters. Although the original pClass utilizes the
multivariate Gaussian function with a non-diagonal inverse
covariance matrix, the simplified version of pClass is also
realized in pENsemble+ where a diagonal covariance matrix is
used. This comparison aims to provide an overview of
ensemble performance under two different base-classifiers.
 Rule Growing Strategy of pClass: pClass makes use of three
rule growing modules, namely Datum Significance (DS), Data
Quality (DQ), volume measure. The datum significance (DS)
method is derived from the theory of statistical contribution
presented in [21], [22]. This method aims to estimate the
potential contribution of a rule or a data point during its
lifespan. This method assumes that data samples are uniformly
distributed and the statistical contribution is estimated using the
Gaussian function as the kernel function. Because the DS
method utilizes the uniform distribution assumption, it loses
spatial information of data streams. This drawback is addressed

in [23] by introducing the sliding-window-based approach
where it calculates the accumulated firing strengths across data
points in the sliding window. The size of sliding window is
often problem-dependent. The DQ method is introduced to
answer this bottleneck where it aims to extract density
information of data samples. This strategy is inspired by the
notion of recursive density estimation (RDE) in [24]. pClass
extends this method for the multivariate Gaussian function and
integrates the weighting function to cope with the outlier
drawback [25]. Furthermore, the third rule growing module
checks the volume of winning rule. It aims to limit the size of
the fuzzy rules because the over-sized rules risks on the socalled cluster delamination problem. That is, one cluster may
cover one or more distinct data distributions [26]. It must be
noted that, as with pENsemble+, pClass is also implemented
under the axis-parallel ellipsoidal rule in pENsemble+ to
analyze the effect of two different base-classifiers. The same
formulas as in the original pClass can be used except that the
diagonal covariance matrix is deployed instead of the nondiagonal version.
 Rule Pruning and Recall Strategy of pClass: pClass is
equipped by two rule pruning scenarios, namely Extended Rule
Significance (ERS) and Potential+ (P+) methods. The ERS
approach shares the same principle as the DQ method except
the statistical contribution of a fuzzy rule is estimated instead
of a data point. This component aims to scan through
inconsequential rules which do not play significant role during
its lifespan. Such rules can be pruned without compromising
generalization performance. The P+ method, on the other hand,

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
functions to capture outdated rules which are no longer relevant
to represent current data distribution. This trait is made possible
by inspecting the density evolution of fuzzy rules. The P+
method presents a modification of potential method in [27]. The
potential method is used for the rule pruning scenario in [28]
but the P+ method differs from this approach because it is based
on the inverse multi-quadratic function in lieu of the Cauchy
function. In addition, pClass incorporates the so-called rule

5

recall scenario. This scenario allows previously pruned rules to
be reactivated again in the future. This scenario refers to a case
where old concept reappears again in the future. One can
consider to introduce a new rule to overcome this situation but
this strategy catastrophically erases past learning history. This
situation is undesired because learning a local region must be
restarted from scratch again.

Algorithm 1. Pseudocode of pENsemble+
Algorithm 1: Parsimonious Ensemble+ (pENsemble+)
Given a data chunk D
factor

 ( X Pn , T PO ) where P, n, O are chunk size, the number of input dimension, the number of output dimension; set the adjustment

pi , the pruning threshold  ;

C ,  ,  1O

are global and local predictions, sum of weighted predictions for each class

D ( nO) is received
For t  1,...,  // loops over all examples in the data chunk
a data chunk

IF the ensemble network is empty

M  1 // create the first local expert, i  1 // initialize the weight of a local expert

End

 0

Execute the online active learning (1)-(4)
IF P( y0

X )input   OR P( y0 X )output  

Discard the data sample

ELSE
Accept and label the data sample for model update
Execute the feature selection mechanism (7)-(9)

 1,..., M // loop over local experts
  max ( yi , j ) // elicits the local prediction

For i

j 1,...,O

IF (

  Ct )

yi   i yi

// decreases the weight of a local expert when it predicts incorrectly

i  i p
Else

 i  min(  i (2  p ),1)

End

      yi

End

C  max(  ) // Produces the global prediction, i 
 1,...,O

For

i  1,..., M

i

// normalizes the weight

M


i 1

i

Undertake the ensemble merging procedure based on the maximum correlation index
IF (12)
Discard i-th local expert
End
End
Undertakes the drift detection method (13),(14)
IF Drift
Introduces a new base classifier
ElseIF Warning
Do nothing and prepare for possible drift in the next observation
ElseIF Stable
Train the winning Classifier
End
End

 Parameter Learning Scenario: pClass utilizes the fuzzily

weighted generalized recursive least square (FWGRLS)

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
method which presents a weight decay term to retain small and
bounded weight vector. This strategy is inspired by the concept
of Generalized Recursive Least Square [29] which incorporates
the weight decay term in the cost function of RLS [30]. The
FWGRLS can be also seen as a variation of the fuzzily weighted
recursive least square method [31] with addition of the weight
decay term. The advantage of weight decay term is to safeguard
the weight vector to keep its values small. It is worth noting that
we adopt the simplified form of GRLS method where the
second term is ignored. This leads to similar formulas of
FWRLS method except the presence of weight decay term. This
strategy is meant to improve the model’s generalization and
compactness of the rule base since a rule with a very small
weight vector can be easily detected by the ERS method. There
exist several types of weight decay term, say quadratic, quartic,
multimodal, etc. The quadratic weight decay term is selected in
the pClass since the weight vector proportionally decreases to
their initial values.
B. Ensemble Learning Scenario
pENsemble+ is developed with a generalized version of
DWM which adopts an open structure paradigm. This learning
scenario clearly differs from original DWM at least in 4
facets:1) the voting weight is given a chance to increase and this
strategy aims to retain diversity of ensemble classifier; 2) the
drift detection strategy is deployed to introduce a new baseclassifier whereas, in the original DWM, a new base-classifier
is added when the global prediction returns misclassification;
3) the ensemble merging scenario based on the maximum
information compression and the online feature selection
scenario are absent from the original DWM. Algorithm 1
illustrates the fundamental working principle of pENsemble+.
pENsemble+ works on a chunk-by-chunk basis and if no
base classifier exists, the first classifier is created using the first
data chunk. The learning procedure starts with the online active
learning scenario evaluating sample’s contribution whether it
deserves a learning process. The Bayesian conflict measure is
deployed to measure conflicts in the input and output space. A
sample is accepted for model updates provided that it satisfies
dynamic sampling criteria. If a data sample meets the dynamic
sampling criteria, the learning process continues with the
labelling process followed by the online feature selection
scenario. The online feature selection selects relevant input
features by assigning crisp weights (0 or 1) and makes possible
to arrive at different combinations of feature subsets in every
training episode. The predictive performance of each base
classifier is examined afterward where a classifier returning
misclassification is penalized by decreasing its voting weight
whereas a reward is given by increasing the voting weight when
correct prediction is made The decreasing and reward factor, p,
is selected at 0.5. The global prediction of ensemble classifier
is inferred from a weighted sum of each class. The voting
weight of each base classifier is normalized to allow
proportional voting weights among each local experts. A class
with the maximum weight is chosen as the predicted class. This
procedure is followed by the ensemble merging procedure
which is meant to capture redundant classifiers. Two classifiers
with high mutual information are coalesced. The last phase of
the training procedure is the drift detection scenario

6

categorizing dynamic of data streams into three conditions,
stable, warning and drift. When a drift is signaled, a new baseclassifier is introduced. No action is performed during the
warning condition since this phase depicts a transition period
before a drift is confirmed. Such situation usually occurs in the
presence of gradual drift. The winning classifier is updated
using the newest data chunk during the stable phase to keep upto-date with the most recent concept and to prevent over-fitting.
The winning classifier is selected from that having the lowest
predictive error – MSE. The learning components of
pENsemble+ are detailed as follows:
 Online Active Learning Strategy: online active learning
scenario is urgently required in the complex manufacturing
process because of the cost in obtaining the true class label. This
usually requires a complete shutdown in the machining process
since the flank wear has to be evaluated through visual
inspection or at least some delay is expected to receive the true
class label. pENsemble+ features an online active learning
scenario based on the extended conflict ignorance (ECI)
paradigm [32] which evaluates conflict in both feature and
target domain. This strategy was derived from the conflict and
ignorance method for the conventional TSK fuzzy classifier
[33] where the underlying difference lies in the use of a
dynamic sampling paradigm [34] and Bayesian posterior
probability estimation in both input and output space. None of
these works, however, investigate the ECI method within the
context of ensemble classifier. As a matter of fact, the ensemble
learning scenario requires an innovation for sample evaluation
strategy since it consists of a collection of local experts evolved
from different data space. One must start from the fact that a
data sample may incur different conflict degrees in different
local experts. Although the sample evaluation strategy should
take place at the local level, a centralistic sample evaluation
strategy where all base classifiers are put together under one
roof to produce the predicted class label is formed. This strategy
is chosen to suit the online feature selection module of
pENsemble+ which also adopts the centralistic feature selection
scenario. Moreover, it is found that this does not make
substantial difference since the maximum operator has to be
ultimately committed when performing local sample evaluation
to analyze the confidence of ensemble classifier.
The Bayesian conflict measure in both input and target
space is utilized to evaluate conflict level for each local expert.
The Bayesian approach is preferred over a standard distance or
firing strength measure because it encompasses the prior
probability and the joint –category and class probability. A
sample is conflicting not only because it is out of scope of a
current fuzzy rule but also if it occupies “unclean” region shared
different class samples. Moreover, the prior probability is
required to take into account the cluster’s population since a
highly populated cluster tends to be “frozen”. That is, it is no
longer responsive to accept new training stimuli due to the
characteristic of rule premise update affected by the cluster’s
support. The conflict in the output space, on the other hand, is
measured from the classifier’s truncated output. The classifier’s
output here is taken from the preference degree [32] determined
with respect to the two most dominant classes since it
intuitively informs about the degree of closeness to the decision
boundary. A sample is conflicting if it falls near two-decision

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
boundary and results in the preference degree with the value
around 0.5.
The Bayesian conflict measure formed in the input space is
expressed as follows:
R

P( yo X ) 

 P( y
o

j

Ri ) P ( X Ri ) P ( Ri )

i 1
R

 P( y

(1)
j

Ri ) P ( X Ri ) P ( Ri )

j 1 i 1

where P( y o

X ), P( X R i ), P( X R i ), P( Ri ) respectively

stand for the joint-class and category probability, the upper
likelihood function, the lower likelihood function and the prior
probability. The joint-class and category probability and the
prior probability are defined as follows:
Ni , j
N
P ( Ri )  R i , P( yo Ri )  o
(2)

N
i 1

i

N

i, j

j 1

where N i , N i , j stand for the support of i-th rule and the support
of the j-th class of the i-th cluster. (2), (3) can be softened by
adding the log operation. This approach is useful in the category
choice phase because it provides higher likelihood for a newly
created cluster to be selected as the winning rule. The joint-class
and category probability is estimated by the number of j-th class
of the i-th rule which signifies the purity degree of a fuzzy rule
[18]. The class overlapping condition is most likely to be found
in the case of unpurified cluster. The likelihood function can be
defined in the similar way as their crisp version using the
Mahalanobis distance as follows:
1
P( X Ri ) 
exp(( X  Ci )i1 ( X  Ci )T )
(3)
1/2
(2 Vi )
1

where Ci ,  i denote the Center and inverse covariance matrix
of the i-th rule while Vi stands for the volume of i-th rule. The
advantage of Bayesian approach is also clear when fuzzy rules
occupy in almost similar proximity to a data sample because of
its prior probability. The volume of fuzzy rules can be obtained
with ease with the determinant operator. If a precise estimation
is required, it can be calculated as shown in [18] where the
Gamma function and the eigenvalue are utilised. Note that the
determinant may return negative volumes as per its definition
as the signed volume.
The conflict in the output space is evaluated from the
classifier’s truncated output to guarantee that it lies in in the
interval [0,1] as follows:
y1
P( yo X )  min(max(conf , 0),1), conf 
(4)
y1  y2
where y1 , y2 stand for the most and second most dominant
classes which can be obtained from the highest and second
highest outputs of pClass. It is worth mentioning that pClass
characterizes the regression-based classifier constructed under
the MIMO architecture which scatters rule consequent for each
output. It is evident that this formula portrays the classifier’s
confusion perfectly because a significant conflict is indicated
when a classifier does not produce a conclusive prediction –

7

two outputs have about the same values. This situation may
occur when a data sample is geometrically close to the decision
boundary separating the two classes. Suppose that

P( y0 X ) input , P( y0 X ) output are

the estimate of posterior

probability in input space and output space, the condition of
sample acceptance is formalized as follows:
( P( y 0

X ) input   or P( y0 X ) output   )

(5)

where  is the conflict threshold. The higher the value of this
parameter the higher the number of training samples are
accepted for model updates, whereas the lower the value of this
parameter the fewer the number of training samples are
discarded for model updates. A sample is supposed to be a good
candidate for model updates, if it results in significant conflict
for all base classifiers. This strategy aims to enhance the
diversity of the ensemble classifier by preventing redundant
samples to be learned. Furthermore, a budget [34] controlling
the maximum labelling cost can be inserted in (5)-(7). This
approach is useful when the true class label is too expensive to
be obtained such as in the bioinformatics applications. The
online active learning can also function to address the class
imbalance issue. First, the imbalance factor is estimated to find
the minority and majority classes. The online active learning is
set loose for minority class samples up to a point where a
balanced proportion of target classes has been achieved. That
is, minority class samples are always sampled to attain equal
class distribution. Since the true class label is unknown in realm
of online active learning scenario, it is estimated with the
posterior probability in the input and output space (1), (4). Both
class imbalanced and budget scenarios are already actualized in
[12]. It is omitted here to keep the presentation concise.
 Online Feature Selection Scenario: the online feature
selection scenario is based on the OFS method in [17]. The OFS
method is generalized here to be well suited to the ensemble
working scenario since the original version only covers its
implementation to the single linear regression. The unique
feature of this approach lies in the fact that it makes possible to
arrive at different subsets of input attributes by assigning binary
weights (0 or 1). In other words, it removes the risk of
discontinuity because it provides likelihood for every input
feature to be selected in every observation. The OFS method
cannot be directly implemented in the ensemble learning
scenario because sensitivity of input attributes should be
analyzed with respect to all base classifiers. This issue leads to
carry out this scenario in a centralistic manner. That is, all fuzzy
rules of base-classifiers are put together to perform the OFS
procedure. Note that this mechanism is made possible by the
fact that pClass adopts a local learning scenario where every
rule is loosely coupled and has its own output covariance
matrix. Let’s recall the fuzzy rule of pClass comes as follows:
1
o
o
(6)
Ri : IF X n is N ( X ; Ci , i ) THEN yi  xeWi
where Ci   is the numeric center of the multivariate
Gaussian function and u is the number of input variables.
u

i1  u u is the interval-valued inverse covariance matrix,
xe  (u 1) is the extended input vector, Wio  (u 1) is the
interval-valued output weight vector.

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
The OFS procedure starts by examining the prediction of a
single pClass created by all rules of ensemble classifiers – all
fuzzy rules are put together to construct a single classifier. The
OFS method only takes place when a model returns
misclassification C  Cˆ to save computational cost because the
OFS method is meant to recover predictive quality of the model
by getting rid of the influence of poor features. The output
o
weight vector Wi is adjusted using the stochastic gradient
descent approach as follows:

Wi  Wi  Wi  
where  ,  are

learning

E
Wi

rate

and

(7)
regularization

factor,

E
can be derived with ease
W
for upper and lower output weight vector by applying the
standard MSE as the cost function. The stochastic gradient
descent is utilized to adjust the output weight rather than the
FWGRLS method as pClass, since the OFS method is
undertaken in the centralistic manner having different
optimization objective from that of the base-classifier level.
Moreover, the stochastic gradient descent is much easier to be
executed than that of the FWGRLS method because no output
covariance matrix has to be assigned when performing the OFS
scenario.
To guarantee a bounded norm, the output weight vector is
projected to the L2 ball as follows:
respectively. The gradient term

1
Wi  min(1,

Wi



)Wi

(8)

2

This strategy is also required to examine whether values of the
output weight vector is concentrated within the L2 ball and thus
pruning small values, being remote from the L2 ball center, does
not compromise the model’s generalization. The contribution
of input attribute is informed from its dominance in the output
weight vector. In realm of the TSK fuzzy system, the rule
consequent or the output weight vector steers the direction or
the tendency of a rule in the output space. In addition, the output
weight vector is more stable than the gradient information
(changing in each observation) during the sensitivity analysis.
The contribution of input attribute is expressed in the form:
R

Ij 

W

j ,i

i 1
u R

W

(9)
j ,i

j 1 i 1

where I j is the sensitivity of j-th input feature and R is a total
number of fuzzy rules across all base-classifiers. Note that the
data standardization must be performed in this context because
different input ranges obscure the true contribution of input
attributes. Suppose that B is the desired input dimensionality
and B is smaller than the original input dimension u, the input
attributes with the B largest input contributions I j are picked up
in every observation and the remainder of input attributes are
ruled out from the training process by assigning 0 weights.

8

Input attributes are not permanently forgotten and are
reactivated in the future whenever they are called for the current
data distribution – cyclic drift. The OFS method also covers the
case of partial input information required when the cost of
feature extraction is too costly. Because the partial input
information is similar to that of full input version, it is not
recounted in this paper. The sensitivity measure (9) is also used
in eTS+ [55]. Nonetheless, eTS+ adopts the hard input pruning
mechanism where superfluous features are permanently
discarded without any opportunity to be picked up again.
 Ensemble Pruning Scenario: The main bottleneck of
ensemble classifier for data stream application is found in the
issue of computational and space complexity because it incurs
considerable complexity if it consists of a large collection of
base classifiers. Nonetheless, the ensemble pruning scenario is
often counterproductive for classifier’s accuracy since it limits
the diversity of the ensemble classifier [35] – the underlying
strength of the ensemble classifier. The ensemble pruning
scenario discards superfluous classifiers – either poor classifier
or outdated classifier. Although such classifiers play little
during their lifespan, they remain important to generate diverse
output space. In realm of dynamic and evolving learning
environments, significance of base classifiers usually changes
rapidly in accordance to the context. When using the ensemble
pruning scenario, it is necessary to integrate the recall capability
because already pruned classifiers may turn out to be useful
again to cover future data distribution. The most plausible
approach to complexity reduction of ensemble classifier is by
putting forward the ensemble merging scenario. That is, it
analyses mutual information of base classifiers and base
classifiers featuring strong mutual information are merged to be
a single classifier. The mutual information is quantified by
comparing the classification output of two classifiers. The
mutual information is relatively more stable than the
significance-based criterion because it measures correlation
between two base classifiers.
An Analysis of mutual information can be performed using any
correlation measure provided that they satisfy the online
learning requirements. Nonlinear correlation measure is often
more accurate than linear correlation measure but it is not
scalable for online real-time processing and requires simplified
assumptions such as training samples follow normal
distribution [36]. pENsemble+ utilizes the maximal
compression index (MCI) to measure correlation of two base
classifiers. This approach is more robust than conventional
Pearson correlation index since it is insensitive to rotation and
translation [37]. It calculates the amount of information loss
when ignoring one of the base classifier and if no significant
difference exists or MCI is small, the information of ignored
base classifier is already covered by its pair. The MCI is
expressed as follows:
1
 ( y1 , y 2 )  (var( y1 )  var( y 2 )
2

 (var( y1 )  var( y 2 )) 2  4 var( y1 ) var( y 2 )(1   ( y1 , y 2 ) 2 )
(10)

 ( y1 , y 2 ) 

cov( y1 , y 2 )
var( y1 ) var( y 2 )

(11)

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
where cov( y1 , y 2 ), var( y1 ), var( y 2 ) ,  ( y1 , y 2 ) denote the
covariance

of

classifier’s

outputs

y1 , y 2 ,

variance

of

classifier’s output y1 , variance of classifier’s output y 2 , and
the Pearson correlation index, respectively. The variance and
covariance can be calculated recursively with ease. It is worth
mentioning that the MCI satisfies the following properties: 1)

0   ( y1 , y 2 )  0.5(var( y1 )  var( y 2 )) ; 2) the maximum
correlation
is
attained
when  ( y1 , y 2 )  0 ;
3)
 ( y1 , y 2 )   ( y 2 , y1 ) ; 4) it is insensitive against the
translation because mean expression is nowhere in (6); 5) it is
insensitive to rotation because a perpendicular distance of a
point to a line is not dependent on rotation. The ensemble
merging condition is set as follows:

 ( y1 , y 2 )  
where  is a merging

(12)

threshold. The lower the value of this
threshold implies less merging process to be performed in the
training process, whereas the higher the value of this threshold
induces more aggressive merging process is committed during
the training process. Once a merging decision is taken, one of
the two classifiers is discarded. The classifier with a lower
accuracy is selected for the pruning process while another one
is retained.
 Drift Detection Scenario: the dynamic of pENsemble+ is
controlled by a drift detection scenario, which aims to discover
abnormal patterns leading to possible change of data stream
dynamics. The drift detection is based on the Hoeffding’s
inequalities [16] which classifies dynamics of data streams into
three categories, namely normal, warning and drift. The normal
phase means no variation in data streams is found, while change
still needs for further investigation in the warning phase. The
drift phase means that change is certain in data streams. The
advantage of this method lies in assumption-free for the
probability density function. It assumes data streams as
independent and bounded random variables.
The drift detection scenario is carried out by inspecting
statistics of data streams – moving average - without any
weight. Although its weighted moving average variant does
also exist in [16], the standard moving average is deployed here
since it is more sensitive to abrupt change than the weighted
version and also is easy-to-use because it does not call for
specific tuning scenario for weight adjustment. The statistics of
data
streams
is
computed
as


X t   t X t , t  1 , X  X . Note that its recursive
t 1

version can be derived with ease. The drift detection strategy
adopts similar concept as the statistical process control except
the assumption of normal distribution is removed. The standard
deviation in the confidence interval  is replaced with the
significance level  . Two significance levels are implemented
to determine the conflict level in data streams. They correspond
to two levels of the drift detection scenario, namely the warning
level ( W ) and to the drift level (  D ).

9

This strategy partitions a data chunk into three groups,
u
namely X  [ x1 , x2 ,..., x ]   ,

Y  [ xcut 1 , xcut  2 ,..., x ]   cut 1u ,
Z  [ x1 , x2 ,..., xcut ]  cut u where
X , Y , Z are
statistics
computed from these three data groups. Each group is assigned
with a Hoeffding’s error bound  X ,  Y ,  Z to set proper conflict
levels which signify the status of data streams. The error bounds
are allocated as follows:
(m)
1
(13)
  (b  a)
ln( )
2cut (m  cut ) 
where [a, b] are the minimum and maximum values of input
attributes and  is the significance level. Note that the
significance level has a clear statistical interpretation which
corresponds to the confidence level of the Hoeffding’s bounds
1 .
The drift detection starts by finding the cutting point which
pinpoints a switch point of two data distributions and in turn
partitions the data chunk into the three groups. The switch point
does not signal directly drift condition to prevent the outlier’s
effects rather in-depth investigations must be performed to
ascertain the status of data distribution – whether a drift really
presents. It is in line with the fact of gradual drift where three
exists a transition period that depicts a mix between two
distributions. A data point is said to be a cut point given that the
following condition is met.
Zt   Z  X t   X
(14)
t

t

The next step is to determine the status of data streams by
formulating a hypothesis test. If null hypothesis is rejected with
size  D , a drift’s status is returned, whereas a warning status
is indicated, if null hypothesis is rejected with size  D , W . In
other words,  D , W correspond to the confidence level of
Hoeffding’s inequality for warning different stages:
(warning),

1  W

1   D (drift). The lower the value of  D ,W the

lower the confidence level of Hoeffding’s bound is – more
examples are considered to be a drift. This implies to more baseclassifiers to be added during the training process and vice
versa. The null hypothesis is formed as H 0 : E ( X ) 
while its alternative is defined as H1 : E ( X ) 

E (Y ) ,

E (Y ) . Since

the weight is excluded here, this hypothesis is analyzed as

X  Y    where   is found from (10) by applying a
specific significance levels  D , W which correspond to either
drift or warning. This hypothesis inspects the dynamic of data
streams after a switching point. This is meant to substantiate the
presence of drift in data streams. If the null hypothesis happens
to be maintained, the stable condition is signaled.
III. EXPERIMENTAL SET-UP & DATA COLLECTION
The set-up consisted of a Lang Swing J6 centre-lathe, onto
which a Kistler tool-post dynamometer platform (type 9263A)
was mounted to measure three mutually perpendicular
components of cutting force. A Kistler tri-axial accelerometer
(type 8730A) was used to measure three mutually perpendicular

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
components of vibration from the underside of the tool holder.
A Kistler charge amplifier (type 5006) and a Kistler power
supply/coupler (type 5134) were used to amplify and decouple
the cutting force and acceleration signals. Dry cutting was
carried out on EN24T BS 970 817M40 alloy steel work-piece
of Brinell hardness 255 of composition: 0.4% C, 0.28% Si,
0.27% Mo, 1.18% Cr, 0.5 % Mn and 1.4% Ni. The tool holder
was Sandvik SSBCR 2020 K12 and the throwaway inserts were
Sandvik Coromant of type SCMT 12 04 08 UM and material
P25 4025 and P15 4015. Three cutting speeds (m/min) and
feed-rates (mm/rev) of 275, 300, 350 and 0.1, 0.2 & 0.3
respectively were used at a constant depth of cut of 2 mm. Six
signals were recorded using an Amplicon PC-30 data
acquisition card mounted in a personal computer, at 4096 (N)
samples per channel, a sampling rate of 30 kHz, and recorded
on computer for analysis.

10

The flank wear values are determined from its effect to the
surface quality and other equipment. It is well-known that blunt
tool imposes higher cutting force which leads to costly scrap
and even damages made to the machine.
The cutting conditions were incorporated to the input vector
sets to assure that the underlying process parameters would be
less sensitive to changes in the cutting conditions (cutting
speed, feed rate and depth of cut). In total, 12 input time-domain
and frequency-domain attributes are collected to identify the
five conditions of the tool.
IV. DATA PROCESSING AND SENSOR FUSION
Any typical sensor is used to measure one desired parameter
and any other parameters influencing the measurement are
considered to be interfering with the measurement. For
example, a dynamometer is often used in measuring cutting
forces (or thereof moments) which are then correlated to the
process of interest. Because of the complex and non-linear
nature of the cutting process, sensor co-operation is not only
desired but also necessary. Essentially, sensor fusion relies on
the fact that the fusion of different source signals of probable
mediocre quality, yields better results than when only one such
a signal is used [1] [2]. A comprehensive review of the
synergistic integration of multi-sensor information can be found
in [3] with typical application scenarios in TCM found in [4-6]
[7]. Sensor fusion traditionally has been performed through
application of statistical methods such as PCS or regression
analyses [8] or a set of heuristic rules, but machine learning
approach has recently gained popularity for TCM.

Fig. 1 evolution of ensemble structure

In most metal turning processes, flank and crater wear are
usually the more prevalent forms of tool wear when cutting with
plane-faced geometry inserts, and their occurrence are
unavoidable. A judicious choice of cutting conditions can
remedy other forms of wear such as frittering, notch and nose.
In this investigation, coated (P15) and un-coated (P25) carbide
inserts with chip breaker geometry to reduce chip/tool contact
were utilized in order to minimize crater wear. The process was
interrupted occasionally to record the flank/nose wear lengths
measured with the aid of a Tool Maker’s Microscope. The ISO
3685 wear criterion was used as a guide but not strictly applied.
The decision for percentage wear on the cutting tool used as the
classification benchmark as either worn or sharp was rather
subjective. This was because each test cut began with a fresh
tool insert and cutting continued until either the tool failed or
flank/ nose wear had accumulated excessively. The following
guideline was employed in determining the tool class:
i. fresh (sharp), flank wear < 0.1 mm;
ii. nominally sharp, 0.1 mm  flanks wear < 0.1 mm;
iii. partly worn, , 0.12 mm  flanks wear < 0.15;
iv. worn, 0.15 mm  flanks wear < 0.17mm;
v. severely worn, flanks wear  0.17 mm.

Fig. 2 Fuzzy Rule Evolution

The instrumentation set-up did not allow simultaneous
recording of the static and dynamic cutting forces. The
parameters were mathematically extracted from the recorded
cutting force signals. The dynamic forces were found by
calculating the oscillatory part of the sampled force signals,
meanwhile the arithmetic mean of the sampled cutting force
signals were taken to represent the static cutting force
components. The obtained dynamic data (dynamic force and
vibration) were passed through a forward FFT, and the DC
component of the dynamic force eliminated. The sum total
power contained in the FFT spectrum (N/2) was taken for each
cut and the values from this calculation together with the static
forces formed the input data samples. In total, there were 157
data samples of which less than 10% could be said to represent

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
the worn tool – partly worn, worn, severely word, while the rest
were for a nominally sharp tool – fresh and nominally sharp.
The obtained data were normalized such that the distribution for
Algorithms
pENsemble+
(axis-parallel)
pENsemble+
(multivariate)
pENsemble
pENsemble+
(axis-parallel)
pENsemble+
(multivariate)
pENsemble

Case
Binary

Multiclass

1)

each signal was positioned within the 0.1-0.9 range, had zero
mean with an equal variance distribution for similar learning
rates and weight convergence.

CR
0.61±0.18

Table 1. Numerical Results
FR
BC
NP
2.8±1
1.1±0.3 28±10.3

IA
2

TS
46.4±1.8

RT
0.01±0.004

0.65±0.12

1.2±0.63

1.1±0.3

14.4±7.6

2

44.7±1.6

0.02±0.004

0.52±0.12
0.81±0.15

3.2±1.7
1

1.1±3
1

32±18.8
16

2
2

63
92±3.97

0.01±0.005
0.01±0.002

0.77±0.12

1

1

18

2

91.8±3.9

0.04±0.009

0.85±0.11

1

1

16

2

107

0.01±0.002

V. PREDICTION PHASE
Two cases, namely binary and multi-class, were simulated
in the prediction phase. The binary case was formed by
grouping fresh and nominally sharp as one group, while the
other three were assigned as another group. The multi-class
case consists of 4 classes which represents the following toolcondition.
 Flank wear mark value ≤ 0.15mm, tool insert
nominally sharp
 Flank wear mark value > 0.15mm, tool insert worn
(high flank)
 Nose wear length ≤ 0.2mm, nominally sharp
 Nose wear length > 0.2mm, tool worn (nose fractured
/ chipped)
The experimental procedure follows the 10-fold cross
validation (CV). The CV procedure, on the other hand, is meant
to avoid the data order dependency problem and utilizes 10-fold
process where all data are distributed into 10 mutually exclusive
bins. The first bin is used to test the generalization power of the
model, while the remainder bins were exploited to evolve the
model. This process moves to the next bin as the testing samples
while others are used as the training samples. It is repeated until
all bins have been used to examine the accuracy of the model.
Data stream environment is created by presenting the data in
small batches during the training process. Table 3 summarizes
details of our numerical study including characteristics of
datasets and the experimental procedure.
pENsemble+ was benchmarked against pENsemble which
happens to be predecessor of our proposed algorithm to exhibit
to what extent the proposed methodologies in this paper are
capable of improving numerical results of pENsemble.
pENsemble characterizes a fully evolving fuzzy classifier using
pClass [18] as a base classifier. It, however, suffers from the
absence of online active learning scenario and still utilizes the
generalization-based ensemble pruning scenarios. pENsemble+
is implemented with two types of pClass, axis-parallel and
multivariate, to perceive the effect of base-classifier to the
overall learning performance. The difference between the two
is seen in the rule premise from which two different ellipsoidal
clusters are generated automatically. Our simulation is carried
out in MATLAB under a laptop with Intel Core i7-6500U CPU
and 16 GB of RAM. The MATLAB codes of pENsemble+ are
provided in1). Two benchmarked algorithms are evaluated in 5
criteria: classification rate (CR), Fuzzy Rule (FR), base
classifier (BC), network parameters (NP), input attribute (IA),

https://www.dropbox.com/s/ve11dbgz87ydn08/pENsembleplus.zip?dl=0

11

training samples (TS), and runtime (RT). Numerical results are
tabulated in Table 1.
It is seen from Table 1 that pENsemble+ is capable of
delivering competitive accuracy with less training samples than
that pENsemble. In the context of binary classification problem,
pENsemble+ even beats pENsemble in almost all criteria. It is
also confirmed that pENsemble+ saves around 30% of total
training samples. The multivariate Gaussian function
contributes to the lowering of the fuzzy rule requirement.
VI. NUMERICAL STUDY WITH BENCHMARK PROBLEMS
This section elaborates on numerical study using three
popular concept drift problems, namely hyperplane, SEA and
SUSY. pENsemble+ is compared with the same set of
algorithms and is evaluated with the same performance metrics.
The experimental procedure follows the periodic hold-out
process where the training process is carried out using the odd
data streams, while the testing phase utilizes the even data
stream. This procedure simulates real data stream environments
where past samples are discarded once seen. Table 2 shows
numerical results of the consolidated algorithms.
A) Hyperplane Problem: this problem is obtained from the
data stream generator of massive online analysis (MOA) [38].
It features a binary classification problem where the main task
is to classify a data point into two classes with respect to the
random hyperplane in the d-dimensional feature space. A class
d

is classified as the positive class if

w x  w
i i

0

, whereas the

i 1

d

negative class is resulted from

w x  w
i i

0

. The unique

i 1

property of this problem lies in the gradual concept drift where
at first data are drawn from one distribution with probability
one. This concept gradually shifts to another data distribution
up to the point where the second concept completely replaces
the first concept. This problem consists of 120 K data samples
and the concept drift occurs after 40K-th samples. This problem
is simulated using the periodic hold-out scenario with 1,000
time stamps. Each time stamp involves 1,200 samples where
1,000 samples are used to train the model and the remainder
200 samples serve as the validation samples. The numerical
results are presented as the average of 1,000 time stamps.
pENsemble+ delivered encouraging numerical results
where it characterizes low sample consumption while
producing comparable accuracy. The online active learning
scenario is capable of significantly reducing the number of

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
training samples where pENsemble+ only attracts 30% of the
total training samples for training process. Moreover, the
ensemble merging strategy relieves computational and space
complexity by getting rid of redundant classifiers. That is, a
classifier sharing strong mutual information with other
classifiers can be discarded without substantial loss of
generalization power. It is also observed that pENsemble+ with
Numerical Example

SEA

Hyperplane

SUSY

Evaluation Criteria
Classification Rate
Fuzzy Rule
Input Attribute
Network Parameters
Execution Time
Training Samples
Ensemble Size
Classification Rate
Fuzzy Rule
Input Attribute
Network Parameters
Training Samples
Execution Time
Ensemble Size
Classification Rate
Fuzzy Rule
Input Attribute
Network Parameters
Training Samples
Execution Time
Ensemble Size

the multivariate Gaussian rule achieves comparable numerical
result as that the axis-parallel Gaussian rule. The advantage is
seen in terms of the fuzzy rule where the multivariate Gaussian
function leads to a more compact and parsimonious rule base
than the axis-parallel rule. Nonetheless, the use of multivariate
Gaussian function causes slightly higher network parameters to
be stored in the memory.

Table 2. Numerical Results of Consolidated Algorithms
pENsemble Learn++.NSE Learn++.cde pClass
0.97±0.02
4.1±1.8
2
65.2±24.7
1.14±0.2
500
2.03±0.9
0.92±0.02
3.74±0.7
2
44.8±8.2
1000
1.5±0.45
1.87±0.34
0.77±0.04
3.8±1.5
1
22.9±9.1
400
0.29±0.08
1.9±0.7

0.93±0.02
N/A
3
N/A
1804.2
500
200
0.91±0.02
N/A
4
N/A
1000
926.04
100

B)SEA problem: The SEA problem was developed by Street
and Kim [39] and is a popular benchmark problem for concept
drift. This problem presents a sudden drift and consists of two
input attributes x1 , x 2 . If the sum of the two attributes fall
below the threshold, a sample is classified to class 2, whereas a
class 1 is assigned to those higher than the threshold. The
concept drift is induced by dramatically changing the thresholds
three times during the experiment ( 4  7  4  7 ) and
data samples are uniformly drawn from the range of [0,10]. In
our experiment, we use the imbalanced version of this problem
introduced by Ditzler and Polikar [17] where the minority class
is around 25% of the total data samples. The experiment was
carried out in the chunk by chunk mode where each chunk
comprises 1000 data samples. The number of chunk is 200 and
the drift is applied in every 50 chunk. The dynamic behavior of
ensemble structure is depicted in Fig. 1, while Fig. 2 portrays
the trace of fuzzy rules. Fig. 3 visualizes the dynamic feature
selection mechanism of pENsemble+ while Fig. 4 exhibits the
trace of training samples.
SEA problem illustrates that although pENsemble+’s accuracy
is slightly more inferior than pENsemble, it attains much lower
sample consumption and comparable structural complexity.
Moreover, pENsemble+ is unlike pENsemble which fully
operates in the fully supervised manner which demands high
operator labelling efforts. pENsemble+ produces similar
performance with two different base-classifiers.
It is observed from Fig. 1 where pENsemble+ features a fully
open structure where its ensemble structure grows and shrinks
dynamically from data streams. The local expert can be
automatically generated on the fly when a concept drift is
detected in data streams. Fig. 1 also confirms that pENsemble+

12

0.93±0.02
N/A
3
N/A
2261.1
500
200
0.9
N/A
4
N/A
1000
2125.5
100

Terminated

0.89±0.1
6.6±4.2
3
157.3±101.9
0.42±0.3
500
N/A
0.91±0.02
3.8±1.7
4
114.9±52.6
1000
2.7±1.4
N/A
0.73±0.06
1.96±0.26
18
748±33.3
400
0.79±0.3
N/A

pENsemble+
(axis-parallel)
0.97±0.03
3.4±1.3
2
20.3±7.5
0.26±0.04
104.1±21.03
1.7±0.6
0.94±0.02
2
2
12
241.9±129.6
0.22±0.09
1
0.76±0.04
6.6±2.4
1
39.5±14.5
236.5±18.8
0.49±23.4
3.2±1.2

pENsemble+
(multivariate)
0.97±0.03
2.96±1.1
2
35.2±12.7
0.27±0.09
113.6±26.01
1.48±0.5
0.94±0.02
1
2
12
248.9±125.7
0.24±0.1
1
0.76±0.04
1.95±0.46
1
214.6±50.6
248.6±25.7
0.33±0.13
1.0±0.05

provides timely response to the concept drifts. It is understood
that the SEA problem characterizes three shifts (
4  7  4  7 ) and these drifts can be properly identified
by the drift detection scenario of pENsemble+ by integrating
new local experts. The efficacy of ensemble merging scenario
is also evident from Fig. 1 where a classifier with strong
redundancy are gotten rid of without compromise to the
classifier’s generalization. Unlike existing variants of ensemble
learners, pENsemble+ deploys a fully evolving local expert,
pClass. This facet is substantiated by Fig. 2 where fuzzy rules
are added and pruned on demand. This trait helps to handle
concept drift better than a static classifier since its network
structure can be expanded when a data point carries significant
novelty to the training process. Moreover, inactive rules are
pruned and this aspect is a key to alleviate the issue of
overfitting. Another unique property of pENsemble+ is found
in its dynamic feature selection scenario where different subsets
of input attributes can be fed to pENsemble+. This salient
characteristic is visualized by Fig. 3 which exhibits the number
of times input attributes have been activated. Fig. 3 also implies
that input features are activated and deactivated on demand
during the training process. The efficacy of online active
learning scenario is shown in Fig. 4. It contributes to
dramatically lower the number of training samples for model
updates. At first, pENsemble+ calls for more samples to be
labelled and learned during the training process. This figure
gradually decreases when pENsemble+ gets more mature by
seeing past training samples. Note that sample significance is
examined with the absence of true class labels.
C)Susy problem: Susy problem is a popular big dataset. It
presents a binary classification problem which aims to classify

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
a signal process that produces supersymmetric particles [40]
and a background process which does not. This problem has 18
input features in which the first 8 input features are the
kinematic properties and the last 10 features are simply the
function of the first 10 attributes. It consists of 5-millions data
samples where 4.5-millions samples are reserved for the
training samples, while the last 500 K samples are used for the
testing samples. To simulate data stream environments, data
come in batches with 10000 timestamps.
This case study demonstrates the scalability of pENsemble+ for
large-scale applications. It exhibits significant improvement
over its predecessor, pENsemble, in terms of runtime. It is
attributed by the online active learning scenario which brings
down the sample consumption to a low level. Local experts can
be added and removed on demand and on the fly. Although
pENsemble+ works on the chunk by chunk basis, it does not
revisit previously acquired data chunk. The memory demand
hence remains independent from the total number of data
chunks. Furthermore, the ensemble merging scenario reduces
the memory complexity and from our numerical examples one
can perceive that pENsemble+ has parsimonious and compact
network structure. The use of multivariate Gaussian function
Data stream
SEA
Hyperplane
Susy
TCM(binary)
TCM(multi-class)

IA
3
4
18
12
12

13

brings positive effect to alleviate computational and structural
burdens of pENsemble+. This result, however, comes at the
cost of a slight deterioration of the predictive performance.

Fig. 3 Dynamic Feature Selection

Table 3 Experimental Procedure
C
DP
TS
TRS
2
100000
200
250
2
120 K
100
1000
2
5M
10000
400
2
69
5
63
4
119
4
107

TES
250
250
100
6
12

Scenario
Holdout
Holdout
Holdout
10-fold CV
10-fold CV

IA: Input Attributes, C: Classes, DP: Data Points, TS: Time Stamps, TRS: Training Samples, TES: Testing Samples

attaining tradeoff between accuracy and complexity. Future
work will be directed to study a stacking ensemble architecture
for regression problems.
REFERENCES
1.

Fig. 4 Evolution of Training Samples
VII. CONCLUSION

This paper proposes the novel tool condition monitoring
methodology taking advantage of an evolving ensemble fuzzy
classifier, pENsemble+. pENsemble+ offers an extension of
pENsemble by integrating online active learning scenario and
ensemble merging scenario. These two learning components
improve the viability of pENsemble for real-world deployment
because it can reduce sample consumption, labelling effort and
ensemble complexity to modest level. Real-world experiments
were carried out where real-world manufacturing data from
metal turning process were collected. In addition, numerical
examples using well-known data streams are provided. It is
shown that pENsemble+ delivers encouraging performance in

Dornfeld D. A. (1990), "Neural network sensor fusion for tool condition
monitoring" Annals of the CIRP, Vol. 39/1, pp. 101-105
2. Holmbom P., pedersen O. Sandell B. and Lauber A. (1989), “Fusing sensor
systems: promises and problems”, Sensor Review, Vol. 9, No. 3, pp 143152
3. Luo R. C. and Kay M. G. (1989), “ Multisensor Integration and Fusion in
Intelligent Systems”, IEEE Trans. Systems, Man and Cybernatics, Vol. 19,
No. 5, pp 901-931
4. Balakrishnan P., Kannatey-Asibu, Jr. E., Trabelsi H. and Emel E, (1989),
“A sensor fusion approach to cutting tool monitoring”, Proc. 15th NSF
Conf. on Production Research & Technology, ASME Publication, Univ.
California at Berkely, USA, pp. 101-109
5. Dimla, Jnr. D. E., Lister P. M. and Leighton N. J., “A Multi-sensor
integration method of signals in a metal cutting operation via application
of multi-layer perceptron neural networks”
5th IEE International
Conference on Artificial Neural Networks, July 1997 Cambridge, UK,
6. Dimla, Jnr. D. E., Lister P. M. and Leighton N. J., “Sensor fusion for
cutting tool state identification in metal turning through application of
perceptron neural networks” IFAC’s SAFEPROCESS ‘97 August, Hull
UK
7. Kim J-D. and Choi I-H., (1996), "Development of a tool failure detection
system using multi-sensors", Int. J. Mach. Tools Manufact., Vol. 36, No.
8, pp. 861-870
8. Beale R and Jackson, T. (1990), "Neural Computing: An Introduction" IOP
Publishing Ltd., Bristol, UK
9. M. Pratama, W. Pedrycz, E. Lughofer, “ Evolving Ensemble Fuzzy
Classifier”, arXiv preprint arXiv:1705.06460, 2017
10. G. D. C. Cavalcanti, L. S. Oliviera, T. J. M. Moura, G. V. Calvarho, “
Combining diversity measures for ensemble pruning”, Pattern Recognition
Letters, Vol. 74, pp. 38-45, 2016
11. J. Kolter and M. Maloof. Dynamic weighted majority: An ensemble
method for drifting concepts. Journal of Machine Learning Research, Vol.
8, pp. 2755–2790, 2007

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
12. M. Pratama, E. Dimla, CY. Lai, E. Lughofer, “ Metacognitive Learning
Approach for Online Tool Condition Monitoring”, Journal of Intelligent
Manufacturing, 2017
13. S. Silva, P. Costa, M. Gouvea, A. Lacerda, F. Alves, D. Leite, “ High
impedance fault detection in power distribution systems usingwavelet
transform and evolving neural network”, Electric Power Systems Research,
Vol. 154, pp. 474-483, 2018
14. B. S. J. Costa, P. P. Angelov, L. A. Guedes, “ Fully unsupervised fault
detection and identification based on recursive density estimation and selfevolving cloud-based classifier”, Neurocomputing, Vol. 150, pp. 289-303,
2015
15. A. Lemos, W. Caminhas, F. Gomide, “ Adaptive fault detection and
diagnosis using an evolving fuzzy classifier”, Information Sciences, Vol.
220, pp. 64-85, 2013
16. I. Frias-Blanco, J. D. Campo-Avilla, G. Ramos-Jimenes, R. MoralesBueno, A. Ortiz-Diaz, Y. Caballero-Mota, “Online and Non-Parametric
Drift Detection Methods Based on Hoeffding’s Bounds”, IEEE
Transactions on Knowledge and Data Engineering, Vol. 27(3), pp. 810823, 2015
17. J. Wang, P. Zhao, S. Hoi, R. Jin, “ Online Feature Selection and Its
Applications”, IEEE Transactions on Knowledge and Data Engineering,
Vol. 26(3), pp. 698-710, 2014
18. M. Pratama, S.G. Anavatti, M.J. Er and E. Lughofer, pClass: An Effective
Classifier for Streaming Examples, IEEE Transactions on Fuzzy Systems,
vol. 23 (2), pp. 369--386, 2015
19. R. Elwell and R. Polikar. Incremental learning of concept drift in
nonstationary environments. IEEE Transactions on Neural Networks, Vol.
22(10), pp. 1517–1531, 2011
20. G. Ditzler and R. Polikar,“Incremental learning of concept drift from
streaming imbalanced data,” in IEEE Transactions on Knowledge & Data
Engineering, vol. 25(10), pp. 2283–2301, 2013
21. H. J. Rong, N. Sundararajan, G. B. Huang, and P. Saratchandran,
“Sequential adaptive fuzzy inference system (SAFIS) for nonlinear system
identiﬁcation and time series prediction,” Fuzzy Sets and Systems, vol.
157(9), pp. 1260–1275, 2006
22. G.-B. Huang, P. Saratchandran, and N. Sundararajan, “A generalized
growing and pruning RBF (GGAP-RBF) neural network for function
approximation,” IEEE Transactions on Neural Networks, vol. 16(1), pp.
57–67, 2005
23. H.-J. Rong, N. Sundarajan, G.-B. Huang, and G.-S. Zhao, “Extended
sequential adaptive fuzzy inference system for classiﬁcation problems,”
Evolving Systems, vol. 2(2), pp. 71–82, 2011
24. P. Angelov, “ Autonomous Learning Systems: From Data Streams to
Knowledge in Real-time”, John Wiley and Sons Ltd., 2012
25. L. Wang, H.-B. Ji, and Y. Jin, “Fuzzy passive-aggressive classiﬁcation: A
robust and efﬁcient algorithm for online classiﬁcation problems,”
Information Sciences, vol. 220, pp. 46–63, 2013
26. E. Lughofer, “A dynamic split-and-merge approach for evolving cluster
models,” Evolving Syst., vol. 3, no. 3, pp. 135–151, 2012
27. P.Angelov and D. Filev, "An approach to online identification of TakagiSugeno fuzzy models," IEEE Transactions on Systems, Man, and
Cybernetics, Part B. vol. 34, pp. 484-498. 2004
28. J.-C. de Barros and A. L. Dexter, “On-line identiﬁcation of computationally
undemanding evolving fuzzy models,” Fuzzy Sets and Systems, vol. 158,
pp. 1997–2012, 2007
29. Y. Xu, K. W. Wong, and C. S. Leung, “Generalized recursive least square
to the training of neural network,” IEEE Transactions on Neural Networks,
vol. 17(1), pp. 19–34, 2006
30. L. Ljung, System Identiﬁcation: Theory for the User. Upper Saddle River,
NJ: Prentice-Hall, 1999
31. P.Angelov et al, “Evolving fuzzy rule-based classiﬁers from data streams,”
IEEE Transactions on Fuzzy Systems, vol. 16(6), pp. 1462–1475, 2008
32. E. Lughofer, Single-Pass Active Learning with Conflict and Ignorance,
Evolving Systems, vol. 3,no.4, pp. 251-271, 2012
33. M.Pratama, S.Anavatti, J.Lu, Recurrent Classifier Based on an Incremental
Meta-cognitive-based Scaffolding Algorithm, IEEE Transactions on Fuzzy
Systems, Vol.23(6), pp. 2048-2066, 2015
34. I.Zliobaite, A.Bifet, B.Pfahringer, G.Holmes, ― Active Learning with
Drifting Streaming Data‖, IEEE Transactions on Neural Networks and
Learning Systems, Vol.25,no.1, pp.27-39, 2014
35. G. D. C. Cavalcanti, L. S. Oliviera, T. J. M. Moura, G. V. Calvarho, “
Combining diversity measures for ensemble pruning”, Pattern Recognition
Letters, Vol. 74, pp. 38-45, 2016

14

36. A. V. Lazo and P. Rathie, “On the entropy of continuous probability
distributions,” IEEE Trans. Inf. Theory, vol. 24, no. 1, pp. 120–122, Jan.
1978
37. P. Mitra, C. A. Murthy, and S. K. Pal, “Unsupervised feature selection
using feature similarity,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24,
no. 3, pp. 301–312, Mar. 2002
38. A. Bifet, G. Holmes, R. Kirkby, and B. P. Fahringer, “MOA: Massive
online analysis,” Journal of Machine Learning Research, vol. 11, pp. 1601–
1604, May 2010
39. W. N. Street and Y. Kim, “A streaming ensemble algorithm (SEA) for
large-scale classiﬁcation,” in Proceeding of 7th ACM SIGKDD, San
Francisco, CA, USA, 2001, pp. 377–38
40. P. Baldi, P. Sadowski, and D. Whiteson. “Searching for Exotic Particles in
High-energy Physics with Deep Learning.” Nature Communications, 5,
2014
41. A. Bouchachia, et al, DELA: A Dynamic Online Ensemble Learning
Algortihm, in European Symposium on Artificial Neural Networks,
Computational Intelligence and Machine Learning, pp. 491- 496, 2014
42. P. Angelov, R. Buswell, “ Identification of Evolving Fuzzy Rule-Based
Models”, IEEE Transactions on Fuzzy Systems, Vol. 10(5), pp. 667-677,
2002
43. N. Kasabov, and Q. Song, DENFIS: dynamic evolving neural-fuzzy
inference system and its application for time series prediction, IEEE
Transactions on Fuzzy Systems .vol10 (2).pp. 144–154. (2002)
44. P. Angelov, N. Kasabov, “Evolving Intelligent Systems, eIS”, IEEE SMC
eNewsletter, Vol.15, pp. 1-13, 2006
45. M.Pratama, S.Anavatti, E.Lughofer, GENEFIS: Towards an Effective
Localist Network, IEEE Transactions on Fuzzy Systems, Vol.2(3), pp.547562, (2014)
46. A. Lemos, W. Caminhas and F. Gomide, Multivariable Gaussian Evolving
Fuzzy Modeling System, IEEE Transactions on Fuzzy Systems, vol. 19 (1),
pp. 91--104, 2011
47. D. Dovzan, V. Logar and I. Skrjanc, Implementation of an Evolving Fuzzy
Model (eFuMo) in a Monitoring System for a Waste-Water Treatment
Process, IEEE Transactions on Fuzzy Systems, vol. 23 (5), pp. 1761--1776,
2015
48. J. A. Iglesias, A. Ledezma, A. Sanchis, “An ensemble method based on
evolving classifiers: eStacking ”, in IEEE Symposium on Evolving and
Autonomous Learning System, pp. 124-131, 2014
49. J.A Iglesias, A. Ledezma, A. Sanchiz, “Ensemble Method Based on
Individual Evolving Classifiers”, in 2013 Evolving and Adaptive
Intelligent Systems, pp. 78-83, 2013
50. L. Bueno, P. Costa, I. Mendes, E. Cruz, D. Leite, “Evolving ensemble of
fuzzy models for multivariate time series prediction”, FUZZ-IEEE 2015:
1-6
51. A. Bifet, G. Holmes, B. Pfahringer, R. Kirkby, R. Gavalda, “New ensemble
methods for evolving data streams”, Proceedings of the 15th ACM
SIGKDD international conference on Knowledge discovery and data
mining, 2009
52. A. Fern, R. Givan, “Online Ensemble Learning: An Empirical Study”,
Machine Learning, Vol. 23(1)-(3), pp. 71-109, 2003
53. E. Ikomomovska, J. Gama, S. Dzeroski, “Online tree-based ensembles and
option trees for regression on evolving data streams”, Neurocomputing,
Vol. 150, pp. 458-470, 2015

