Stochastic Configuration Networks:
Fundamentals and Algorithms

arXiv:1702.03180v4 [cs.NE] 27 Feb 2017

Dianhui Wang∗, Ming Li
Department of Computer Science and Information Technology
La Trobe University, Melbourne, VIC 3086, Australia
Email:dh.wang@latrobe.edu.au

Abstract
This paper contributes to a development of randomized methods for neural
networks. The proposed learner model is generated incrementally by stochastic
configuration (SC) algorithms, termed as Stochastic Configuration Networks
(SCNs). In contrast to the existing randomised learning algorithms for single layer
feed-forward neural networks (SLFNNs), we randomly assign the input weights
and biases of the hidden nodes in the light of a supervisory mechanism, and the
output weights are analytically evaluated in either constructive or selective manner.
As fundamentals of SCN-based data modelling techniques, we establish some
theoretical results on the universal approximation property. Three versions of
SC algorithms are presented for regression problems (applicable for classification
problems as well) in this work. Simulation results concerning both function
approximation and real world data regression indicate some remarkable merits
of our proposed SCNs in terms of less human intervention on the network size
setting, the scope adaptation of random parameters, fast learning and sound
generalization.

1 Introduction
Over the past decades, neural networks have been successfully applied for data modelling due to
its universal approximation power for nonlinear maps [3, 4, 10, 23], and learning capability from
a collection of training samples [7, 24]. In practice, however, it is quite challenging to properly
determine an appropriate architecture (here refers to the number of hidden nodes) of a neural
network so that the resulting learner model can achieve sound performance for both learning and
generalization. To resolve this problem, one turns to develop constructive approaches for building
neural networks, starting with a small sized network, followed by incrementally generating hidden
nodes (corresponding to a set of input weights and biases) and output weights until a pre-defined
termination criterion meets. Being a fundamental of manifold applications, it is essential to
ensure that the constructive neural network shares the universal approximation property. In [1],
Barron proposed a greedy learning framework based on the work reported in [13], and established
some significant results on the convergence rate. In [15], Kwok and Yeung presented a method
to construct neural networks through optimizing some objective functions. Theoretically, their
proposed algorithm can generate universal approximators for any continuous nonlinear functions
provided that the activation function meets some conditions. It has been aware that the process of
iterative searching for an appropriate set of parameters (input weights, biases and output weights) is
time-consuming and computationally intensive although the universal approximation property holds,
but very difficult to be employed for dealing with large-scale data analytics.
∗

Corresponding author.

Randomized approaches for large-scale computing are highly desirable due to their effectiveness
and efficiency [20]. In machine learning for data modelling, randomized algorithms have
demonstrated great potential in developing fast learner models and learning algorithms with much
less computational cost [19, 25, 28]. Readers are strongly recommended to refer to our survey paper
[25] for more details. From algorithm perspectives, randomized learning techniques for neural
networks received attention in later 80’s [2] and further developed in early 90’s [21, 22, 26]. A
common and basic idea behind these randomized learning algorithms is a two-step training paradigm,
that is, randomly assigning the input weights and biases of neural networks and evaluating the output
weights by solving a linear equation system using the well-known least squares method and its
regularized versions. From approximation theory viewpoint, randomized Radial Basis Function
(RBF) Networks proposed in [2] and Random Vector Functional-link (RVFL) networks proposed
in [22] can be regarded as Random Basis Approximators (RBAs) [27]. Thus, it is essential and
interesting to look into the universal approximation capability of RBAs in the sense of probability.
In [12], Igelnik and Pao proved that a RVFL network with random parameters (input weights and
biases) chosen from the uniform distribution defined over a range can be a universal approximator
with probability one for continuous functions. In [11], Husmeier revisited this significant result and
showed that the universal approximation property of RVFL networks holds as well for symmetric
interval setting of the random parameter scope if the function to be approximated meets Lipschitz
condition. In [27], Ivan et al. empirically investigated the feasibility of RBAs for data modelling
and showed that a supervisory mechanism is necessary to make RVFL networks applicable. Their
experiments clearly indicate that RVFL networks fail to approximate a target function with a very
high probability if the setting for random parameters is improper. This phenomenon was further
studied with mathematical justifications in [5]. From implementation considerations, we need to
know two key parameters involved in design of RVFL networks: the number of hidden nodes and
the scope of random parameters. Indeed, as one of the special features of this type of learner models,
the first parameter associated with the modelling accuracy must be set largely. The second parameter
is related to the approximation capability of the class of random basis functions. Obviously, these
settings play very important roles to successfully build randomized learner models for real world
applications. Intuitively, constructive or incremental RVFL (IRVFL) networks may be a possible
solution for problem solving. However, our recent work reported in [18] reveals the infeasibility of
IRVFL networks, if they are incrementally built with random input weights and biases assigned in
a fixed scope and its convergence rate satisfies some certain conditions. Thus, further researches on
supervisory mechanisms with adaptive scope setting in random assignment of the input weights and
biases to ensure the universal approximation capability of IRVFL networks are being expected.
To the development of randomized learning techniques for neural networks, our prime and original
contribution from this paper is the way of assigning the random parameters with an inequality
constraint and adaptively selecting the scope of the random parameters, ensuring the universal
approximation property of the built randomized learner models. Indeed, this work firstly touches
the base of implementation of the random basis function approximation theory. It should be
clarified that one cannot view SCNs as a specific implementation of RVFL networks due to some
remarkable distinctions in randomization of the learner model. Three algorithmic implementations
of SCNs, namely Algorithm SC-I, SC-II and SC-III, are presented with the same supervisory
mechanism for configuring the random parameters, but different methods for computing the output
weights. Concretely, SC-I employs a constructive scheme to evaluate the output weights only for
the newly added hidden node and keep all of the previously obtained output weights unchanged;
SC-II recalculates a part of the current output weights by solving a local least squares problem
with a user specified shifting window size; and SC-III finds the output weights all together through
solving a global least squares problem with the current learner model. Our experimental results on
a toy example for function approximation and real world regression tasks demonstrate remarkable
improvements on modelling performance, compared with the results obtained by existing methods
such as Modified Quickprop (MQ) [15] and IRVFL [18].
The remainder of this paper is organized as follows: Section 2 briefly reviews constructive neural
networks with a recall for the infeasibility of IRVFL networks. Section 3 details our proposed
stochastic configuration networks with both theoretical analysis and algorithmic description. Section
4 reports our simulation results, and Section 5 concludes this paper with some remarks on further
studies.

2

The following notation is used throughout this paper. Let Γ := {g1 , g2 , g3 ...} be a set of real-valued
functions, span(Γ) denote a function space spanned by Γ; L2 (D) denote the space of all Lebesgue
measurable functions f = [f1 , f2 , . . . , fm ] : Rd → Rm defined on D ⊂ Rd , with the L2 norm
defined as
!1/2
m Z
X
kf k :=
|fq (x)|2 dx
< ∞.
(1)
q=1

D

The inner product of θ = [θ1 , θ2 , . . . , θm ] : Rd → Rm and f is defined as
m Z
m
X
X
fq (x)θq (x)dx.
hfq , θq i =
hf, θi :=
q=1

q=1

(2)

D

In the special case that m = 1,R for a real-valued function ψ : Rd → R defined on D ⊂ Rd ,
its L2 normRbecomes kψk := ( D |ψ(x)|2 dx)1/2 , while the inner product of ψ1 and ψ2 becomes
hψ1 , ψ2 i = D ψ1 (x)ψ2 (x)dx.

2 Related Work

Instead of training a learner model with a fixed architecture, the process of constructive neural
networks starts with a small sized network then adds hidden nodes incrementally until an acceptable
tolerance is achieved. This approach does not require any prior knowledge about the complexity of
the network for a given task. This section briefly reviews some closely related work on constructive
neural networks. Some comments on these methods are also given.
Given a target function f : Rd → R, suppose that we have already built a SLFNN with L − 1 hidden
PL−1
nodes, i.e., fL−1 (x) = j=1 βj gj (wjT x+bj ) (L = 1, 2, . . ., f0 = 0), and the current residual error,
denoted as eL−1 = f −fL−1 , does not reach an acceptable tolerance level. The construction process
is concerned with how to incrementally add βL , gL (wL and bL ) leading to fL = fL−1 + βL gL until
the residual error falls into an expected tolerance ǫ.
2.1 Constructive Neural Networks: Deterministic Methods
In [1], Barron proposed a greedy approximation framework based on Jones’ Lemma [13]. The main
result can be stated in the following theorem.
Theorem 1 (Barron [1]). Given a Hilbert space G, suppose f ∈ conv(G) (closure of the convex
hull of the set G), and kgk ≤ bg for each g ∈ G. Let bf > cf > sup kgk + kf k. For L = 1, 2, . . .,
the sequence of approximations {fL } is deterministic and described as
fL = αL fL−1 + (1 − αL )gL ,
where
αL =

b2f
b2f + kfL−1 − f k2

,

(3)

(4)

and gL is chosen such that the following condition holds
hfL − f, gL − f i <
Then, for every L ≥ 1, kf − fL k2 ≤

′

C
L

b2f − c2f
kfL−1 − f k2 .
2b2f

(5)

′

, where C ≥ bf .

Remark 1. Some optimization techniques are needed in order to find a suitable gL to meet the
condition (5). In fact, the whole process aims to minimize kαL fL−1 + (1 − αL )gL − f k at each
iteration, which is functionally equivalent to choosing gL ∈ G that maximizes hf − fL−1 , gi. Note
that this constructive scheme is only applicable to the target function f belonging to the closure of
the convex hull of G, that means the convergence to all L2 functions cannot be guaranteed under
this framework. In addition, the new leaner model fL comes from a specific convex combination of
the previous model fL−1 and the newly added term gL , which results in a weak solution compared
against one obtained through optimizing the left hand side of (5) with respect to the coefficient αL .
3

In [15], Kwok and Yeung proposed an incremental learning strategy for building SLFNNs and
proved the universal approximation property. At each step, the input weights and biases (w and
b) of the newly added node at the hidden layer are obtained by maximizing an objective function
and the output weights are evaluated using the least squares method. This theoretical result can be
stated as follows:
Theorem 2 (Kwok and Yeung [15]). Let Γ be a set of basis function. Assume that span(Γ) is dense
in L2 space and ∀g ∈ Γ, 0 < kgk < bg for some bg ∈ R+ . If gL is selected as to maximize
heL−1 , gL i2 /kgk2 , then limL→+∞ kf − fL k = 0.

Remark 2. Theoretically, the convergence property can be guaranteed provided that we can find
gL to maximize heL−1 , gL i2 /kgk2. In practice, however, local minima occurs frequently when
performing the gradient ascent algorithm. Sometimes, during the course of incremental learning,
the optimization process in constructing a new hidden node seems meaningless as the updating of
hidden parameters is excessively slow, which means the reduction of residual error will be close to
zero. In fact, this point was discussed in [14] and a Modified Quickprop algorithm was suggested
in [15] in order to alleviate this problem. However, this weakness cannot be overcome completely
for some complex tasks. In essence, the Modified Quickprop algorithm that iteratively finds the
appropriate hidden parameters (w and b) may still face some obstacles in the optimization process
of the objective function when it is searching in a plateau of the error surface. Once w (b) at some
iterative step falls in a region that both the first and second derivatives of the objective function with
respect to w (b) are nearly zero, the learning reaches a halt and may be terminated. In a nut shell,
the derivative-based optimization processes suffer from some inherent shortcomings and have lower
probability to generate a universal learner eventually.

The constructive methods mentioned above consist of two phases: selecting the hidden parameters
(the input weights and biases) according to some certain criteria and evaluating the output
weights after adding a new hidden node. Although the universal approximation property can
be guaranteed by incrementally adding the hidden nodes, the resulting SLFNNs usually need a
quite few hidden nodes to achieve good learning performance. In practice, seeking for a basis
function through maximizing heL−1 , gL i2 /kgk2 is very time consuming. Thus, for many real world
applications, deterministic methods used in building constructive neural networks seem have no or
less applicability. As one of pathways to generate neural networks with the universal approximation
property (corresponding to the perfect learning power), randomized approaches have great potential
to access a faster and feasible solution [19, 25, 28].
2.2 Constructive Random Basis Approximators
Random vector functional-link (RVFL) networks [21, 22] can be regarded as a randomized version
of SLFNNs, where the input weights and biases are randomly assigned and fixed during the training
phase, and the output weights are analytically evaluated by the least squares method [16]. In [12],
Igelnik and Pao theoretically justified its universal approximation property based on Monte-Carlo
method with the limit-integral representation of the target function. The main result can be restated
as follows:
Theorem 3 (Igelnik and Pao [12]). For any compact set D ⊂ Rd , given f ∈ C(D) R(i.e., the set of
all continuous functions defined over D), and any activation function g that satisfies R |g(t)|2 dt <
R
′
∞ or R |g (t)|2 dt < ∞, there exist a sufficiently large L, a set of β1 , β2 , ..., βL and a probabilistic
PL
space χL , such that fL = j=1 βj g(x; wj , bj ) can approximate f with arbitrary accuracy in the
sense of probability one, if w and b are randomly assigned over χL and follow certain distribution.
Alternatively, this result can be expressed as
Z

lim E
|f (x) − fL (x)|2 dx = 0,
(6)
L→+∞

D

where E is the expectation operator with respect to the probabilistic space χL .

Remark 3. In [11], Husmeier revisited the universal approximation property of RVFL networks with
a symmetric interval setting for the random parameters. Strictly speaking, such a property holds for
only target functions satisfying Lipschitz condition. From our experience, however, most of data
modelling problems from real world applications meet Lipschitz condition. Thus, for simplicity, we
adopt the symmetric interval setting for the random parameters in this paper. It should be aware that
4

such a special setting does not limit the development of our framework at all, and it is just a matter
of implementation indeed.
The theoretical result stated in Theorem 3 is fundamental and significant for building randomized
neural networks. Similar to the case of making use of the SLFNNs in resolving real world problems,
we need to develop effective algorithms to implement such a class of randomized predictive models.
It is natural to think of an incremental implementation of RVFL (IRVFL) networks, where the model
is built incrementally with random assignment of the input weights and biases, and constructive
evaluation of its output weights. Although the construction process of IRVFL networks seems to
be computationally efficient, unfortunately the universal approximation property of the constructed
learner cannot be guaranteed. The following Theorem 4 from our recent work [18] have justified
this point.
Theorem 4 (Li and Wang [18]). Let span(Γ) be dense in L2 space and ∀g ∈ Γ, 0 < kgk < bg for
some bg ∈ R+ . Suppose that gL is randomly generated and βL is given by
βL =
For sufficiently large L, if the followings hold:

and

heL−1 , gL i
.
kgL k2

keL−1 k2 − keL k2
≤ εL < 1,
keL−1 k2
lim

L→∞

L
Y

(1 − εk ) = ε > 0.

(7)

(8)

(9)

k=1

Then, the constructive neural network with random weights, fL , has no universal approximation
capability, that is,
√
lim kf − fL k ≥ εkf k.
(10)
L→∞

Theorem 4 reveals that IRVFL networks may not share the universal approximation property, if
the output weights are taken as (7) and the residual error sequence meets conditions (8) and (9).
The consequence stated in Theorem 4 still holds if the output weights are evaluated using the least
squares method. We state this interesting result in the following Theorem 5.
Theorem 5. Let span(Γ) be dense in L2 space and ∀g ∈ Γ, 0 < kgk < bg for some bg ∈ R+ .
Suppose that gL is randomly generated and the output weights are calculated by solving the global
PL
least square problem, i.e., [β1∗ , β2∗ , . . . , βL∗ ] = arg minβ kf − j=1 βj gj k. For sufficiently large L,
if (8) and (9) hold for the residual error sequence ke∗L k (corresponding to keL k in Theorem 4), the
constructive neural network with random hidden nodes has no universal approximation capability,
that is,
lim kf −

L→∞

L
X
j=1

βj∗ gj k ≥

√

εkf k.

(11)

Proof. Simple computations can verify that the sequence ke∗L k2 is monotonically decreasing and
converges. Indeed, let β̃L = he∗L−1 , gL i/kgL k2 , we have
ke∗L k2

≤ he∗L−1 − β̃L gL , e∗L−1 − β̃L gL i
= ke∗L−1 k2 −
≤ ke∗L−1 k2 .

he∗L−1 , gL i2
kgL k2

(12)

Then, (11) can be easily obtained by following the proof of Theorem 4.
Clearly, the universal approximation property is conditional to IRVFL networks whatever the output
weights are evaluated. This happens due to multiple reasons (e.g. the scope setting and/or the
improper way to assign the random parameters) that is hard to tell mathematically. In this paper, we
propose a solution for constructing randomized learner models under a supervisory mechanism to
ensure the universal approximation property.
5

3 Stochastic Configuration Networks
Universal approximation property is fundamental to a learner model for data modelling. Logically,
one cannot expect to build a neural network with good generalization but poor learning performance.
Therefore, it is essential to share the universal approximation property for SCNs. This section
details our proposed SCNs, including proofs of the universal approximation property and algorithm
descriptions. Some remarks on algorithmic implementations are also given.
3.1 Universal Approximation Property
Given a target function f : Rd → Rm , suppose that a SCN with L − 1 hidden nodes has already
PL−1
been constructed, that is, fL−1 (x) = j=1 βj gj (wjT x + bj ) (L = 1, 2, . . ., f0 = 0), where βj =
[βj,1 , . . . , βj,m ]T . Denoted the current residual error by eL−1 = f − fL−1 = [eL−1,1 , . . . , eL−1,m ].
If keL−1 k does not reach a pre-defined tolerance level, we need to generate a new random basis
function gL (wL and bL ) and evaluate the output weights βL so that the leading model fL = fL−1 +
βL gL will have an improved residual error. In this paper, we propose a method to randomly assign
the input weights and biases with a supervisory mechanism (inequality constraint), and provide with
three ways to evaluate the output weights of SCNs with L hidden nodes (i.e., after adding the new
node). Mathematically, we can prove that the resulting randomized learner models incrementally
built based on our stochastic configuration idea are universal approximators.
Theorem 6. Suppose that span(Γ) is dense in L2 space and ∀g ∈ Γ, 0 < kgk < bg for some
bg ∈ R+ . Given 0 < r < 1 and a nonnegative real number sequence {µL } with limL→+∞ µL = 0
and µL ≤ (1 − r). For L = 1, 2, . . ., denoted by
δL =

m
X
q=1

δL,q , δL,q = (1 − r − µL )keL−1,q k2 , q = 1, 2, ..., m.

(13)

If the random basis function gL is generated to satisfy the following inequalities:
heL−1,q , gL i2 ≥ b2g δL,q , q = 1, 2, ..., m,

(14)

and the output weights are constructively evaluated by
heL−1,q , gL i
, q = 1, 2, . . . , m.
kgL k2
PL
Then, we have limL→+∞ kf − fL k = 0, where fL = j=1 βj gj , βj = [βj,1 , . . . , βj,m ]T .
βL,q =

(15)

Proof. According to (15), it is easy to verify that {ke2L k} is monotonically decreasing. Thus, the
sequence keL k is convergent as L → ∞. From (13), (14) and (15), we have

=

keL k2 − (r + µL )keL−1 k2
m
X
(heL−1,q − βL,q gL , eL−1,q − βL,q gL i − (r + µL )heL−1,q , eL−1,q i)
q=1

=

m
X
q=1

=
=
≤

((1 − r − µL )heL−1,q , eL−1,q i − 2heL−1,q , βL,q gL i + hβL,q gL , βL,q gL i)
Pm

2
q=1 heL−1,q , gL i
(1 − r − µL )keL−1 k −
kgL k2
Pm
2
q=1 heL−1,q , gL i
δL −
kgL k2
Pm
2
q=1 heL−1,q , gL i
δL −
≤ 0.
b2g
2

(16)

Therefore, the following inequality holds:
keL k2 ≤ rkeL−1 k2 + γL , (γL = µL keL−1 k2 ≥ 0).
6

(17)

Note that limL→+∞ γL = 0, by using (17), we can easily show that limL→+∞ keL k2 = 0 which
implies limL→+∞ keL k = 0. This completes the proof of Theorem 6.
Theorem 6 provides a constructive scheme, i.e., (14) and (15), that can consequently lead to a
universal approximator. Unlike the strategy that maximizes some objective functions in [15], the
supervisory mechanism (14), which aims at finding appropriate wL and bL for a new hidden node,
weakens the demanding condition as required to achieve the maximum value for heL−1 , gL i2 or
heL−1 , gL i2 /kgL k2 (for the case m = P
1). In fact, the existence of wL and bL satisfying (14) can
m
be easily deduced because Ψ(w, b) = q=1 heL−1,q , gL i2 /kgL k2 is a continuous function in the
parameter space, and δL = (1 − r − µL )keL−1 k2 can be far less than keL−1 k2 once the r is selected
to approach 1. Overall, the supervisory mechanism described in (14) not only makes it possible to
randomly assign the hidden parameters, which performs more flexibly and efficiently in generating a
new hidden node, but also enforces the residual error to be zero along with the constructive process.
Remark 4. Our proposed supervisory mechanism in (14) indicates that random assignment of wL
and bL should be constrained and data dependent. That is to say, the configuration of hidden
parameters needs to be relevant to the given training samples, rather than totally rely on the
distribution and/or scoping parameter setting of the random weights and biases. To the best of our
knowledge, our attempt on the supervisory mechanism (14) is the first time in design of randomized
learner models, and it fills the gaps between the scheme of solving a global nonlinear optimization
problem (that are usually quite demanding in both time and space since iterative solution seems
inevitable) and the strategy of freely random assignment of the hidden parameters without any
constraint. It is worth mentioning that Theorem 6 is still valid if the learning parameter r is
unfixed and set based on an increasing sequence approaching 1 (always less than 1). That makes SC
algorithm be more flexible for the implementation of randomly searching wL and bL even when the
residual error is smaller. It has been observed that finding out appropriate wL and bL for the newly
added node becomes more challenging as the residual error becomes smaller. In this case, we need
to set the value of r to be extremely close to 1.
It is obvious that βL = [βL,1 , . . . , βL,m ]T in Theorem 6 is analytically evaluated by βL,q =
heL−1,q , gL i/kgL k2 and remain fixed for further adding steps. This determination scheme, however,
may cause very slow convergence rate for the constructive process. Thus, we consider a recalculation
scheme for the output weights, that is, once gj (j = 1, 2, . . . , L) have been generated according
to (14), β1 , β2 , . . . , βL can be evaluated by minimizing the global residual error. The following
Theorem 7 gives a result on the universal approximation property if the least squares method is
applied to update the output weights in a proceeding manner.
P
PL
∗
∗
∗
∗
Let [β1∗ , β2∗ , . . . , βL∗ ] = arg minβ kf − L
j=1 βj gj k, eL = f −
j=1 βj gj = [eL,1 , . . . , eL,m ], and
2
∗
∗
define intermediate values β̃L,q = heL−1,q , gL i/kgL k for q = 1, . . . , m and ẽL = eL−1 − β̃L gL ,
where β̃L = [β̃L,1 , . . . , β̃L,m ]T and e∗0 = f .
Theorem 7. Suppose that span(Γ) is dense in L2 space and ∀g ∈ Γ, 0 < kgk < bg for some
bg ∈ R+ . Given 0 < r < 1 and a nonnegative real number sequence {µL } with limL→+∞ µL = 0
and µL ≤ (1 − r). For L = 1, 2, . . ., denoted by
∗
δL
=

m
X
q=1

∗
∗
δL,q
, δL,q
= (1 − r − µL )ke∗L−1,q k2 , q = 1, 2, ..., m.

(18)

If the random basis function gL is generated to satisfy the following inequalities:
∗
he∗L−1,q , gL i2 ≥ b2g δL,q
, q = 1, 2, ..., m,

(19)

and the output weights are evaluated by
[β1∗ , β2∗ , . . . , βL∗ ] = arg min kf −
β

Then, we have limL→+∞ kf − fL∗ k = 0, where fL∗ =
7

PL

j=1

L
X
j=1

βj gj k.

∗
∗
]T .
, . . . , βj,m
βj∗ gj , βj∗ = [βj,1

(20)

Proof. It is easy to show that ke∗L k2 ≤ kẽL k2 = ke∗L−1 − β̃L gL k2 ≤ ke∗L−1 k2 ≤ kẽL−1 k2 for
L = 1, 2, . . ., so {ke∗L k2 } is monotonically decreasing and convergent. Hence, we have
≤
=
=

ke∗L k2 − (r + µL )ke∗L−1 k2

kẽL k2 − (r + µL )ke∗L−1 k2
m 

X
he∗L−1,q − β̃L,q gL , e∗L−1,q − β̃L,q gL i − (r + µL )he∗L−1,q , e∗L−1,q i
q=1
m 
X
q=1

=


(1 − r − µL )he∗L−1,q , e∗L−1,q i − 2he∗L−1,q , β̃L,q gL i + hβ̃L,q gL , β̃L,q gL i
µL )ke∗L−1 k2

(1 − r −
Pm

=

∗
δL
−

≤

∗
δL
−

−

Pm

∗
2
q=1 heL−1,q , gL i
kgL k2
Pm
∗
2
q=1 heL−1,q , gL i

b2g

∗
2
q=1 heL−1,q , gL i
kgL k2

≤ 0.

(21)

Using the same arguments in the proof of Theorem 6, we can obtain limL→+∞ ke∗L k = 0, that
completes the proof of Theorem 7.
Evaluation of the output weights in Theorem 7 is straightforward with the use of Moore-Penrose
generalized inverse [16]. Unfortunately, this method is infeasible for large-scale data analytics. To
solve this problem, we suggest a trade-off solution with window shifting concept in the global least
squares method (i.e., we only optimize a part of the output weights after the number of the hidden
nodes exceeds a given window size). Indeed, this selective scheme for evaluating the output weights
is meaningful and significant for dealing with large-scale data processing. Similar to the proof of
Theorem 7, the resulting SCN shares the universal approximation property. Here, we state this result
in the following Theorem 8 and omit the detailed proof.
Theorem 8. Suppose that span(Γ) is dense in L2 space and ∀g ∈ Γ, 0 < kgk < bg for some
bg ∈ R+ . Given 0 < r < 1 and a nonnegative real number sequence {µL } with limL→+∞ µL = 0
and µL ≤ (1 − r). For a given window size K and L = 1, 2, . . ., denoted by
∗
=
δL

m
X
q=1

∗
∗
= (1 − r − µL )ke∗L−1,q k2 , q = 1, 2, ..., m.
δL,q
, δL,q

(22)

If the random basis function gL is generated to satisfy the following inequalities:
∗
, q = 1, 2, ..., m,
he∗L−1,q , gL i2 ≥ b2g δL,q

and, as L ≤ K, the output weights are evaluated by

[β1∗ , β2∗ , . . . , βL∗ ] = arg min kf −
β

L
X
j=1

(23)

βj gj k;

(24)

∗
Otherwise (i.e.,L > K), the output weights will be selectively evaluated (i.e., keep β1∗ , . . . , βL−K
unchanged and renew the left βL−K+1 , . . . , βL ) by
∗
∗
[βL−K+1
, βL−K+2
, . . . , βL∗ ] = arg

min

βL−K+1 ,...,βL

Then, we have limL→+∞ kf − fL∗ k = 0, where fL∗ =
3.2 Algorithm Description

kf −
PL

L−K
X

j=1

j=1

βj∗ gj −

L
X

j=L−K+1

βj gj k.

(25)

∗
∗
, . . . , βj,m
]T .
βj∗ gj , βj∗ = [βj,1

This subsection details the proposed SC algorithms, i.e., SC-I, SC-II and SC-III, which are
associated with Theorems 6, 8 and 7, respectively. In general, the main components of our proposed
SC algorithms can be summarized as follows:
8

• Configuration of Hidden Parameters : Randomly assigning the input weights and biases
to meet the constraint (14) ((19) or (23)), then generating a new hidden node and adding it
to the current learner model.
• Evaluation of Output Weights: Constructively or selectively determining the output
weights of the current learner model.
Algorithm SC-I
Given inputs X = {x1 , x2 , . . . , xN }, xi ∈ Rd and outputs T = {t1 , t2 , . . . , tN }, ti ∈ Rm ;
Set maximum number of hidden nodes Lmax , expected error tolerance ǫ, maximum times
of random configuration Tmax ; Choose a set of positive scalars Υ = {λmin : ∆λ : λmax };

1. Initialize e0 := [t1 , t2 , . . . , tN ]T , 0 < r < 1, two empty sets Ω and W ;
2. While L ≤ Lmax AND ke0 kF > ǫ, Do
Phase 1: Hidden Parameters Configuration (Step 3-17)
3.
For λ ∈ Υ, Do
4.
For k = 1, 2 . . . , Tmax , Do
5.
Randomly assign ωL and bL from [−λ, λ]d and [−λ, λ], respectively;
6.
Calculate hL , ξL,q based on Eq. (26) and (28), and µL = (1 − r)/(L + 1);
7.
If min{ξL,1, ξL,2 , ..., ξL,m } ≥ 0P
m
8.
Save wL and bL in W , ξL = q=1 ξL,q in Ω, respectively;
9.
Else go back to Step 4
10.
End If
11.
End For (corresponds to Step 4)
12.
If W is not empty
∗
13.
Find wL
, b∗L that maximize ξL in Ω, and set HL = [h∗1 , h∗2 , . . . , h∗L ];
14.
Break (go to Step 18);
15.
Else randomly take τ ∈ (0, 1 − r), renew r := r + τ , return to Step 4;
16.
End If
17. End For (corresponds to Step 3)
Phase 2: Output Weights Determination
∗
∗T
∗
18. Calculate βL,q = (eT
L−1,q · hL )/(hL · hL ), q = 1, 2, . . . , m;
T
19.
βL = [βL,1 , . . . , βL,m ] ;
20.
eL = eL−1 − βL h∗L ;
21. Renew e0 := eL ; L := L + 1;
22. End While
∗
23. Return β1 , β2 , . . . , βL , ω ∗ = [ω1∗ , . . . , ωL
] and b∗ = [b∗1 , . . . , b∗L ].

Given a training dataset with inputs X = {x1 , x2 , . . . , xN }, xi = [xi,1 , . . . , xi,d ]T ∈ Rd
and its corresponding outputs T = {t1 , t2 , . . . , tN }, where ti = [ti,1 , . . . , ti,m ]T ∈ Rm , i =
1, . . . , N . Denoted by eL−1 (X) = [eL−1,1 (X), eL−1,2 (X), . . . , eL−1,m (X)]T ∈ RN ×m as the
corresponding residual error vector before adding the L-th new hidden node, where eL−1,q (X) =
[eL−1,q (x1 ), . . . , eL−1,q (xN )] ∈ RN with q = 1, 2, . . . , m. Let

T
T
T
hL (X) = [gL (wL
x1 + bL ), gL (wL
x2 + bL ), . . . , gL (wL
xN + bL )]T ,
(26)
which stands for the activation of the L-th hidden node for the input xi , i = 1, 2, . . . , N . The
(current) hidden layer output matrix can be expressed as HL = [h1 , h2 , . . . , hL ].

In practice, the target function is presented as a collection of input-output data pairs. So βL,q =
heL−1,q , gL i/kgL k2 (q = 1, . . . , m) becomes

eL−1,q (X)T · hL (X)
, q = 1, 2, . . . , m.
(27)
hL (X)T · hL (X)
For the sake of brevity, we introduce a set of variables ξL,q , q = 1, 2, ..., m and use them in algorithm
descriptions (pseudo codes):


2
T
 eL−1,q (X) · hL (X)

ξL,q = 
− (1 − r − µL )eL−1,q (X)T eL−1,q (X) .
(28)
hL (X)T · hL (X)
βL,q =

9

Recall that for a given window size K in SC-II, as L ≤ K, the suboptimal solution β ∗ =
[β1∗ , β2∗ , . . . , βL∗ ]T ∈ RL×m can be computed using the standard least squares method, that is,
β ∗ = arg min kHL β − T k2F = HL† T,
β

(29)

where HL† is the Moore-Penrose generalized inverse [16] and k · kF represents the Frobenius norm.
As L > K, a portion of the output weights can be evaluated by

†
β window = arg min kHK β − T k2F = HK
T,
β

(30)

where β window consists of the most recent βL−K+1 , . . . , βL , HK is composed of the last K columns
of H, i.e., HK = [hL−K+1 , . . . , hL ], and the left (previous) β1 , . . . , βL−K remain unchanged. It is
easy to see that SC-II and SC-III become consistent once K ≥ Lmax .
Algorithm SC-II
Given the same items in Algorithm SC-I. Set window size K < Lmax .
1. Initialize e0 := [t1 , t2 , . . . , tN ]T , 0 < r < 1, two empty sets Ω and W ;
2. While L ≤ Lmax AND ke0 kF > ǫ, Do
3.
Proceed Phase 1 of Algorithm SC-I;
4.
Obtain HL = [h∗1 , h∗2 , . . . , h∗L ];
5.
If L ≤ K
6.
Calculate β ∗ = [β1∗ , β2∗ , . . . , βL∗ ]T = HL† T ;
7.
Else
8.
Set H̃K = HL (:, 1 : L − K), HK = HL (:, L − K + 1 : L);
∗
9.
Retrieve β previous = [β1∗ , β2∗ , . . . , βL−K
]T ;
†
window
previous
10.
Calculate β
= HK (T − H̃
β
 Kprevious
 );
β
11.
Let β ∗ = [β1∗ , β2∗ , . . . , βL∗ ]T :=
;
β window
12.
End If
13.
Calculate eL = eL−1 − βL∗ h∗L ;
14.
Renew e0 := eL ; L := L + 1;
15. End While
∗
16. Return β1∗ , β2∗ , . . . , βL∗ , ω ∗ = [ω1∗ , . . . , ωL
] and b∗ = [b∗1 , . . . , b∗L ].

Algorithm SC-III
Given the same items in Algorithm SC-I.
1.
2.
3.
4.
5.
6.
7.
8.
9.

Initialize e0 := [t1 , t2 , . . . , tN ]T , 0 < r < 1, two empty sets Ω and W ;
While L ≤ Lmax AND ke0 kF > ǫ, Do
Proceed Phase 1 of Algorithm SC-I;
Obtain HL = [h∗1 , h∗2 , . . . , h∗L ];
Calculate β ∗ = [β1∗ , β2∗ , . . . , βL∗ ]T := HL† T ;
Calculate eL = eL−1 − βL∗ h∗L ;
Renew e0 := eL ; L := L + 1;
End While
∗
Return β1∗ , β2∗ , . . . , βL∗ , ω ∗ = [ω1∗ , . . . , ωL
] and b∗ = [b∗1 , . . . , b∗L ].

Remark 5. In SC-I, we employ a scaling sigmoidal function as the hidden node activation function,
of which the parameters Υ := {λ1 : ∆λ : λmax } play important role for adaptively determining
the scope setting of the random parameters w and b. As indicated in Theorem 6, although the
randomness is beneficial for fast configuration of the newly added hidden node, the searching area
is potentially data dependent and should not be rigidly fixed. In fact, the λ ∈ Υ varies during the
course of adding new hidden nodes, that will be demonstrated later in our simulations.
10

Remark 6. Except for the expected error tolerance ǫ, termination of SC-I can be done by referring
to L
in order to prevent the over-fitting phenomenon. It is worth mentioning that a larger value
Pmax
m
of q=1 heL−1,q , gL i2 /kgL k2 would probably lead to faster decreasing in the residual error. This is
the reason behind why we conduct Tmax times of random configuration for wL and bL and finally
return a pair of w∗ and b∗ with the largest ξL , as specified between Step 3-17 in Algorithm SC-I.
Obviously, such an algorithmic operation helps in building compact SCNs.
Remark 7. As the constructive process proceeds, difficulties involved in the random configuration
for w and b become increasingly higherP
when the residual error becomes smaller. To overcome this,
m
we take time-varying value of r so that q=1 (1 − r − µL )eL−1,q (X)T eL−1,q (X) will approach to
zero and consequently create more opportunities to make ξL ≥ 0.

4 Performance Evaluation
This section reports some simulation results over four regression problems, including a function
approximation and three real-world modelling tasks. Performance evaluation goes to learning,
generalization and efficiency (computing time and the eventual number of hidden nodes required for
achieving an expected training error tolerance). Comparisons among our SC algorithms, Modified
Quickprop (MQ) [15] and IRVFL [18] are carried out. In this study, the sigmoidal activation function
g(x) = 1/(1 + exp(−x)) is used.
4.1 Data Sets
Four datasets are employed in our simulation studies, including a function approximation example,
three real world regression datasets stock, concrete and compactiv downloaded from KEEL
(Knowledge Extraction based on Evolutionary Learning) dataset repository 2 .
• DB 1 was generated by a real-valued function [27]:
2

2

2

f (x) = 0.2e−(10x−4) + 0.5e−(80x−40) + 0.3e−(80x−20) , x ∈ [0, 1].

(31)

The training dataset has 1000 points randomly generated from the uniform distribution
[0,1]. The test set, of size 300, is generated from a regularly spaced grid on [0,1].
• Data provided in stock are daily stock prices from January 1988 through October 1991,
for ten aerospace companies. The task is to approximate the price of the 10th company by
using the prices of the rest. The whole data set (DB2) contains 950 observations with nine
input variables and one output variable. We randomly selected 75% samples as the training
set while the test set consists of the left 25%.
• The third regression task (concrete) is to find the highly nonlinear functional relationship
between concrete compressive strength and the ingredients (input features) including
cement, blast furnace slag, fly ash, water, super plasticizer, coarse aggregate, fine aggregate,
and age. The whole data set (DB 3) contains 1020 instances. Both the training and test sets
are formed in the same manner as DB2.
• The fourth dataset (compactiv) comes from a real world application, that is, computer
activity dataset describing the portion of time that CPUs run in user-mode, based on 8192
computer system activities collected from a Sun SPARCstation 20/712 with 2 CPUs and
128 MB of memory running in a multi-user university department. Each system activity is
evaluated using 21 system measures (as input features). Both the training and test data sets
are formulated in the same way as done for DB2 and DB3.
Both the input and output attributes are normalized into [0,1], and all results reported in this paper
take averages over 100 independent trials. The maximum times of random configuration Tmax is set
as 200. For the other parameters such as the maximum number of hidden nodes Lmax , the expected
error tolerance ǫ and the index r, we will specify their corresponding settings later.
2

KEEL: http://www.keel.es/

11

Two metrics are used in our comparative studies, i.e., modelling accuracy and efficiency. The first
one is the widely used Root Mean Squares Error:

1/2
L
N X
X
1
RM SE = 
βj g(wjT xi + bj ) − ti ]2  ,
[
(32)
N i=1 j=1

where N is the number of samples (training or test) and L represents the number of hidden nodes.

For the sake of comparing the efficiency of different methods, we recorded both the time (measured
in seconds) spent on building the learner model and the number of hidden nodes needed to achieve
the expected error tolerance.
4.2 Results
Table 1 and Table 2 show the training and test results of MQ, IRVFL, SC-I, SC-II and SC-III on DB
1-4, in which average values and standard deviations are reported on the basis of 100 independent
trials. In particular, we set the learning rate in MQ as 0.05 and its maximum iterative number as 200
for all the regression tasks. In IRVFL, the random parameters (w and b) are taken from the uniform
distribution over [-1,1]. Clearly, from Table 1, that SC-I, SC-II and SC-III outperform other methods
in terms of both learning and generalization. For each setting of L, the training errors of MQ and
IRVFL are larger than SC algorithms, in which SC-III demonstrates the best performance compared
against SC-I and SC-II for both training and test datasets. In Table 2, both MQ and SC algorithms
lead to sound performances in comparison to the results obtained from IRVFL.
For efficiency comparisons, it can be observed from Table 3 that MQ, IRVFL and SC-I all fail in
achieving the expected training error tolerance (ǫ = 0.05) within an acceptable time period for DB1,
whilst both IRVFL and SC-I could not successfully achieve the given training error for DB2. Here,
we only report the results for DB1 and DB2 as the other two case studies share similar consequences.
First, the missing values in Table 3 (marked as ‘-’) for time cost and test performance are caused
by the extremely slow learning rate for these scenarios. With some practical consideration that if
certain patience parameter mentioned in [15] is given, the learning phase of MQ, IRVFL and SC-I
algorithm will be terminated to avoid meaningless time cost. Consequently, it seems impossible for
those methods to meet the expected error tolerance in a reasonable time slot. The proposed SC-II and
SC-III algorithms, however, perform quite well and achieve the specified training error bound with
few hidden nodes. On the other hand, compared against MQ method, SC-II and SC-III algorithms
generate less number of hidden nodes and demonstrate better generalization performance.
Table 1: Performance Comparison among MQ, IRVFL, SC-I, SC-II and SC-III on DB 1. K = 15
was used in SC-II.
Algorithms
MQ
IRVFL
SC-I
SC-II
SC-III

Training
L = 25
L = 50
0.1031±0.0001 0.1030±0.0001
0.1630±0.0008 0.1626±0.0005
0.0927±0.0020 0.0887±0.0018
0.0435±0.0061 0.0366±0.0049
0.0332±0.0065 0.0097±0.0036

Test
L = 25
L = 50
0.1011±0.0003 0.1011±0.0003
0.1622±0.0012 0.1617±0.0008
0.0912±0.0021 0.0870±0.0019
0.0411±0.0064 0.0337±0.0049
0.0308±0.0060 0.0100±0.0033

The advantages of the proposed SC algorithms are clearly visible from Figure 1 and Figure 2, in
which the curves are plotted based on the average values over 100 independent runs. The following
observations can be made from these results.
• In Figure 1, it can be observed that the error decreasing rates of our proposed SC algorithms
are quite higher than the other two methods. At the same time, their corresponding test
errors also keep decreasing when adding new hidden nodes. Furthermore, similar to the
results reported in Table 1, SC-II and SC-III perform much better than SC-I, which has
slower decreasing rate (rather than MQ and IRVFL). It should be noted that window size
K = 15 is used in SC-II, so that the error decreasing trend of SC-II and SC-III should be
consistent with each other when L ≤ 15. This fact has been clearly illustrated in Figure 1.
12

Table 2: Performance Comparison among MQ, IRVFL and SC-I, SC-II and SC-III on DB 2, DB3
and DB4. K = 15 was used in SC-II.
Data Sets

DB2

DB3

DB4

Training
L = 25
L = 50
0.0624±0.0058 0.0410±0.0014
0.2121±0.0260 0.1853±0.0248
0.0963±0.0032 0.0881±0.0026
0.0437±0.0014 0.0395±0.0008
0.0409±0.0010 0.0327±0.0007
0.1096±0.0042 0.0910±0.0014
0.2045±0.0189 0.1929±0.0135
0.1401± 0.0027 0.1358±0.0022
0.0994±0.0018 0.0943±0.0011
0.0969±0.0013 0.0835±0.0012
0.0840±0.0052 0.0600±0.0071
0.2002±0.0391 0.1924±0.0283
0.1207±0.0036 0.1137±0.0038
0.0760±0.0034 0.0579±0.0029
0.0678±0.0038 0.0394±0.0016

Algorithms
MQ
IRVFL
SC-I
SC-II
SC-III
MQ
IRVFL
SC-I
SC-II
SC-III
MQ
IRVFL
SC-I
SC-II
SC-III

Test
L = 25
L = 50
0.0611±0.0061 0.0407±0.0017
0.2057±0.0268 0.1787±0.0237
0.0925±0.0031 0.0851±0.0024
0.0427±0.0017 0.0391±0.0010
0.0403±0.0012 0.0347±0.0012
0.0999±0.0048 0.0869±0.0021
0.2109±0.0214 0.1983±0.0166
0.1315±0.0026 0.1258±0.0017
0.0925±0.0029 0.0874±0.0018
0.0898±0.0020 0.0850±0.0025
0.0848±0.0053 0.0624±0.0075
0.1958±0.0386 0.1882±0.0281
0.1169±0.0038 0.1105± 0.0040
0.0773±0.0039 0.0593±0.0036
0.0697±0.0044 0.0418±0.0021

Table 3: Efficiency comparison among MQ, IRVFL and SC-I, SC-II and SC-III on DB 1 and DB2.
Window size K = 15 was used in SC-II.
Data Sets

Algorithms
MQ
IRVFL
SC-I
SC-II
SC-III
MQ
IRVFL
SC-I
SC-II
SC-III

DB1

DB2

0.25

Test
0.0463±0.0021
0.0435±0.0047
0.0532±0.0175
0.0488±0.0016
0.0483±0.0020

0.25

IRVFL
MQ
SC-I
SC-II
SC-III

0.2

IRVFL
MQ
SC-I
SC-II
SC-III

0.2

0.15

Test RMSE

Training RMSE

Efficiency (ǫ = 0.05)
Time (s)
Nodes
0.4029±0.1690 26.6700±8.7514
0.2737±0.0719 19.9000±3.6167
1.3157±0.6429 34.1809±3.5680
0.2193±0.0454 16.7200±2.1513
0.2083±0.0317 16.3600±1.5987

0.1

0.05

0.15

0.1

0.05

0

0
0

50

100
Number of Nodes: L

150

(a)

0

50

100
Number of Nodes: L

150

(b)

Figure 1: Performance of Modified QuickProp (MQ), IRVFL, SC-I, SC-II and SC-III with 150
additive nodes on DB 1: (a) Average training RMSE and (b) Average test RMSE

• For MQ and IRVFL algorithms on DB 1, both the training and test errors stabilize at
a certain level but still unacceptable (the real training RMSE is larger than 0.1). This
extremely slow decreasing rate in learning makes it impossible to build a learner model with
13

0.6

0.5

IRVFL
MQ
SC-I
SC-II
SC-III

0.25

0.4

0.2
Test RMSE

Training RMSE

0.3

IRVFL
MQ
SC-I
SC-II
SC-III

0.3

0.15

0.2

0.1

0.1

0.05

0

0
0

50

100
Number of Nodes: L

150

0

(a)

50

100
Number of Nodes: L

150

(b)

Figure 2: Performance of Modified QuickProp (MQ), IRVFL, SC-I, SC-II and SC-III with 150
additive nodes on DB 2: (a) Average training RMSE and (b) Average test RMSE
time constraint, even when L takes a larger value. In comparison, an improved decreasing
rate of SC-I seems obviously evident compared against MQ after adding 150 hidden nodes.
• In Figure 2, MQ, SC-II, and SC-III converge faster than the other two algorithms, of which
IRVFL exhibits the slowest decreasing rate in both training and test phases. Although MQ
in this case outperforms SC-I, its error decreasing rate is slower than SC-II and SC-III as
L ≤ 50. Indeed, there is a negligible difference among the curves of SC-II, SC-III and MQ
when L > 50. As a whole, the error curves of SC-III always stay at the bottom, which
correspond to the numerical results reported in Table 3.
Table 4: Performance of SC-II with different window sizes on DB 1. L = 35
Window Size
K=5
K=10
K=15
K=20
K=25
K=30
K=35

Training
Mean
STD
0.0682 0.0049
0.0530 0.0061
0.0439 0.0062
0.0364 0.0069
0.0325 0.0073
0.0267 0.0056
0.0248 0.0069

Test
Mean
STD
0.0672 0.0049
0.0516 0.0064
0.0416 0.0068
0.0338 0.0064
0.0303 0.0067
0.0252 0.0049
0.0233 0.0062

Efficiency (ǫ = 0.05)
Time(s)
Nodes
1.36
99.99
0.72
50.85
0.34
25.07
0.28
21.21
0.26
20.29
0.25
19.62
0.25
19.75

Table 5: Performance of SC-II with different window sizes K on DB 2. L = 25
Window Size
K=5
K=10
K=15
K=20
K=25

Training
Mean
STD
0.0613 0.0039
0.0492 0.0024
0.0443 0.0014
0.0420 0.0012
0.0411 0.0010

Test
Mean
STD
0.0601 0.0037
0.0493 0.0027
0.0436 0.0020
0.0412 0.0015
0.0405 0.0013

Efficiency (ǫ = 0.05)
Time(s)
Nodes
0.88
66.42
0.31
23.48
0.21
16.62
0.20
16.16
0.20
16.35

Table 4 and 5 report some results on robustness of the modelling performance with respect to the
window size K in SC-II. It should be noted that we specified the number of hidden nodes being
added in order to perform the robustness analysis. Furthermore, for the purpose of examining the
efficiency, we set the training error tolerance for each task (the same as in Table 3) and recorded
the real time cost and number of hidden notes required for achieving that tolerance. All of the
results reported in Table 4 and 5 are averaged over 100 independent runs. For DB 1, seven cases
(K = 5, 10, 15, 20, 25, 30, 35) are considered and the number of hidden nodes is set as 35. It can
been clearly seen that both training and test performance become better along with increasing the
value of K. Importantly, small K values may lead to inferior results, for example, the training and
test RMSEs with K = 5, K = 10 and K = 15 are out of the tolerance level. On the other hand,
14

the efficiency of SC-II will become higher as the value of K increases. Apart from the scenarios of
K = 5, K = 10 and K = 15, the difference among the remaining cases can be ignored that the
number of hidden nodes is about 20 while the whole time cost is around 0.26s (see Table 4). For
DB 2, five cases (K = 5, 10, 15, 20, 25) are examined while the number of hidden nodes is set as
25. Similarly, all of the results, except for K = 5 and K = 10, are comparable. In particular, for
K = 15, 20, 25, the time cost and number of hidden nodes required for achieving the error tolerance
(ǫ = 0.05) are around 0.2s and 16, respectively.
1
Target Output
Model Output
0.8

1

1

0.8

0.8

0.6

0.6
f(x)

f(x)

f(x)

0.6
0.4

0.4

0.4
0.2
0.2
0
0

0.2

0.4

0.6

0.8

0.2

0

0

-0.2

-0.2

1

0

0.2

0.4

x

(a) IRVFL, λ = 1

0.8

1

0

(b) IRVFL, λ = 100

0.8

0.8

0.8

0.6

0.6

0.6

0.4
0.2

f(x)

1

0.4
0.2
0

0

-0.2

-0.2

0.6

0.8

1

0

0.2

0.4

x

(d) SC-I

0.8

1

0.2

0

0.4

0.6

0.4

-0.2
0.2

0.4

(c) IRVFL, λ = 200

1

0

0.2

x

1

f(x)

f(x)

0.6
x

0.6
x

(e) SC-II, K = 15

0.8

1

0

0.2

0.4

0.6

0.8

1

x

(f) SC-III

Figure 3: Approximation performance of IRVFL (with different setting of λ), SC I, SC-II and SC-III
on DB 1
Figure 3 depicts the modelling performance (for test dataset) of IRVFL, SC-I, SC-II and SC-III on
DB 1, respectively. Firstly, the importance of the scale factor λ in the activation function, which
directly determines the range of random parameters, is examined by performing different settings.
Secondly, the infeasibility of IRVFL is illustrated in comparison with the proposed SC algorithms,
where effectiveness and benefits from the supervisory mechanism of SCNs can be clearly observed.
The maximum number of the hidden nodes to be added is set as 100 for these four methods examined
here. From Figure 3(a), (b), (c), it is apparent that IRVFL networks perform poorly. In fact, we
attempted to add 20,000 hidden nodes to see the performance of IRVFL networks with different
values of λ. Unfortunately, the final result is very similar to that depicted in Figure 3(c), which is
aligned with our findings in [18]. In comparison, the test performance of SC-I shown in Figure 3(d)
is much better than IRVFL, that corresponds to the records in Table 1 and the error changing curves
in Figure 1, respectively. In addition, both SC-II and SC-III outperform SC-I as the recalculation of
βL contributes a lot in constructing a universal approximator with a much more compact structure.
4.3 Discussions
This section briefly explains why the performance of our proposed SC algorithms are better than
MQ and IRVFL algorithms. Then, further remarks are provided to highlight some characteristics of
SCNs.
Although the universal approximation property can be ensured by maximizing an objective
function (see Section 2), the Modified Quickprop (MQ) algorithm, aiming at iteratively finding
the appropriate hidden parameters (w and b) for the new hidden node, may face some obstacles
in proceeding the optimization of certain objective function. In essence, it is a gradient-ascent
algorithm with consideration of second-order information, and may be problematic when it is
searching in a plateau over the error surface. That is to say, the well-trained parameters (w and
b) at certain iterative step fall into a region where both the first and second order of derivatives
15

0.25

IRVFL

sigmiod
tanh
sin
cos
gaussian

IRVFL
0.2

0.15

Test RMSE

Training RMSE

0.2

0.25

sigmiod
tanh
sin
cos
gaussian

SC-I
0.1

0.05

0.15
SC-I
0.1

0.05

0

0
0

50

100
Number of Nodes: L

150

(a) Training

0

50

100
Number of Nodes: L

150

(b) Test

Figure 4: Comparison between SC-I and IRVFL with various types of activation function

of the objective function are nearly zero, learning basically reaches a halt and can be terminated
by a patience parameter for the purpose of time-saving. The overall speed for parameter updating
might be quite slow. This issue seems severe for regression tasks. For implementations, MQ works
less flexibly and seems very likely to fail in constructing a universal approximator, not to mention
some inherent flaws including the selection of initial weights, learning rate, as well as a reasonable
stopping criterion. Besides, there will be much computational burden in dealing with large-scale
problems. The unique feature of the proposed SCNs is the use of randomness in learner model
design with a supervisory mechanisms. Compared against Modified Quickprop algorithm, SC-II
and SC-III are computationally manageable and easy to be implemented.
∗
In addition, the output weights βL in IRVFL are analytically calculated by eL−1 (X)T ·h∗L /(h∗T
L ·hL )
and then remain fixed in further phases. This constructive approach for computing the output
weights, together with the freely random assignment of the hidden parameters, may cause very
slow decreasing rate of the residual error sequence, and as a result the learner model may fail
to approximate the target function. Although this evaluation method for the output weights is
also applied for SC-I, the whole construction process works successfully in building a universal
approximator. In the end, some features of our proposed SCNs are summarized as follows:

• The selection of r that to some extent is directly associated with the error decreasing speed
is quite important. In our experiments, its setting is unfixed and based on an increasing
sequence stating from 0.9 and approaching 1. That makes it possible for the implementation
of randomly searching wL and bL . As the constructive process proceeds, the current
residual error becomes smaller which makes the configuration task on wL and bL be more
challenging (i.e., need more searching attempts as L becomes larger). From our experience,
this difficulty can be effectively alleviated by setting r extremely close to 1.
• In SC algorithms, we used the scaling sigmoidal function in the hidden nodes, of which
λ may keep varying during the learning course. This implies that the scope setting for w
and b should not be fixed. In our simulations, λ is automatically decided from a given set
{1, 5, 15, 30, 50, 100, 150, 200}, in accompany with the setup of r. This strategy makes
it possible for the built learner to possess multi-scale random basis functions rather than
RVFL networks (as argued in [5]) or IRVFL, and can inherently improve the probability of
finding certain appropriate setting of the input weights and biases.
• In implementation, Tmax controls the pool size of the random basis function candidates. It
is related to both opportunity and efficiency, so we need to chose this parameter carefully
with trade-off mind. This operation aims at finding the most appropriate pairs of w and b
that returns the largest ξL (see Algorithm SC-I). Roughly speaking, this manipulation can
be viewed as an alternative to find a ‘suboptimal’ solution of the maximization problem
discussed in [15].
• Based on the theoretical analysis given in Section 3, many types of activation functions can
be used in SCNs, such as gaussian, sine, cosine and tanh function. Figure 4 shows both
the training and test performance of SC-I and IRVFL with different choices of activation
16

functions. It can be clearly seen that our SC-I algorithm is more efficient and effective than
IRVFL as the random assignment of w and b from [-1,1] is unworkable and useless, no
matter what kind of activation function is employed in the hidden layer.
• As a compromise between SC-I and SC-III, SC-II provides more flexibility in dealing with
large-scale data modelling tasks, by right of its distinctive window-based recalculation
for the output weights. Through optimizing a part of the output weights according to a
given window size, SC-II not only outperforms SC-I, because the evaluation manner of
the output weights in SC-I is not based on any objectives and no further adjustments are
provided throughout the construction process; but also has some advantages over SC-III
in building universal approximators for coping with large-scale data analytics. In practice,
when the number of training samples is extremely large and L > K, the computation
†
burden for calculating HK
in SC-II is far less than calculating HL† in SC-III. Besides, the
robustness of SC-II with regard to the window size is favourable once K is assigned from
a reasonable range, as shown in Table 4 and 5. It is meaningful and interesting to find out
more characteristics about SC-II in the future.

5 Conclusions
Our proposed framework in this paper provides with an alternative solution for building randomized
learner models with supervisory mechanisms. As a powerful tool for fast data modelling, SCNs
can be incrementally built by stochastically configuring the input weights and biases of each hidden
node, and determining the output weights via constructive evaluation or solving an optimization
problem for linear models. Each SCN model can be regarded as a specific implementation of
our SC algorithms, which in theory ensure the convergence provided that newly generated nodes
are kept adding to the model. From our hands-on experiences as demonstrated in the simulations,
SC-III exhibits the best performance in terms of the convergence rate, SC-I converges slowest but
still outperforms IRVFL although they evaluate the output weights in the same way. It should be
pointed that SC-II embedded with a given window size for updating the output weights makes a
good trade-off between the efficiency and the scalability (i.e., for large-scale data analytics).
In this work, we focus on SLFNN architecture with a sigmoidal basis/activation function for the
hidden nodes. Some immediate extensions to RVFL architecture (with direct links between the
inputs and the outputs) and Echo State Networks (with recurrent feedback at the hidden layer)
can be done easily. As a technical contribution to the machine learning community, our proposed
stochastic configuration idea in this paper is significant and applicable for data representation with
deep learning [6, 8, 9, 17]. Plenty of researches on SCNs and its applications can be explored in
the future, and some of them have been done by our group. For instance, deep SCNs (DSCNs) and
robust SCNs (RSCNs) have been developed for dealing with data representation and regression, and
uncertain data analytics respectively, which will be reported in our series of publications. Also, we
are extending the present algorithms to ensemble learning, online learning and distributed learning.
Except for these studies mentioned above, the following researches are being expected: robustness
analyses of the modelling performance with respect to some key parameters involved in design of
SCNs; and a guideline for selecting the basis/activation function so that the modelling performance
can be maximized.

References
[1] A. R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, IEEE
Transactions on Information Theory 39 (3) (1993) 930-945.
[2] D. S. Broomhead, D. Lowe, Multivariable functional interpolation and adaptive networks, Complex
Systems, 2 (1988) 321-355.
[3] T. Chen, H. Chen, Universal approximation to nonlinear operators by neural networks with arbitrary
activation functions and its application to dynamical systems, IEEE Transactions on Neural Networks 6(4)
(1995) 911-917.
[4] G. Cybenko, Approximations by superpositions of a sigmoidal function, Mathematics of Control, Signals
and Systems 2 (1989) 303-314.

17

[5] A. N. Gorban, I. Y. Tyukin, D. V. Prokhorov, K. I. Sofeikov, Approximation with random bases: pro et
contra, Information Sciences 364 (2016) 129-145.
[6] A. Graves, A. R. Mohamed, G. E. Hinton, Speech recognition with deep recurrent neural networks, Proc.
of IEEE International Conference on Acoustics, Speech and Signal Processing (2013) 6645-6649.
[7] R. Hecht-Nielsen, Theory of the backpropagation neural network, Neural Networks 1(1) (1988) 593-605.
[8] G. E. Hinton, S. Osindero, and Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural
Computation 18 (7) (2006) 1527-1554.
[9] G. E. Hinton, R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313
(5786) (2006) 504-507.
[10] K. Hornik, M. Stinchcombe, H. White, Multilayer feedforward networks are universal approximators,
Neural Networks 1989, 2(5):359-366
[11] D. Husmeier, Random vector functional link (RVFL) networks. Neural Networks for Conditional
Probability Estimation: Forecasting Beyond Point Predictions-Springer, (1999) Chapter 6.
[12] B. Igelnik, Y. H. Pao, Stochastic choice of basis functions in adaptive function approximation and the
functional-link net, IEEE Transactions on Neural Networks 6 (6) (1995) 1320-1329.
[13] L. K. Jones, A simple lemma on greedy approximation in Hilbert space and convergence rates for
projection pursuit regression and neural network training, The Annals of Statistics 20 (1) (1992) 608-613.
[14] T. Y. Kwok, D. Y. Yeung, Constructive neural networks: some practical considerations, Proc. of IEEE
International Conference on Neural Networks, Orlando, Florida, USA, (1994) 198-203.
[15] T. Y. Kwok, D. Y. Yeung, Objective functions for training new hidden units in constructive neural
networks, IEEE Transactions on Neural Networks 8 (5) (1997) 1131-1148.
[16] P. Lancaster, M. Tismenetsky, The Theory of Matrices: With Applications, Academic Press, 1985.
[17] Y. LeCun, Y. Bengio, G. E. Hinton, Deep learning, Nature 521 (2015) 436-444.
[18] M. Li, D. Wang, Insights into randomized algorithms for neural networks: practical issues and common
pitfalls, Information Sciences 382-383 (2017) 170-178.
[19] M. LukošEvičIus, H. Jaeger, Reservoir computing approaches to recurrent neural network training,
Computer Science Review 3(3) (2009) 127-149.
[20] M. W. Mahoney, Randomized algorithms for matrices and data, Foundations and Trends in Machine
Learning 3 (2) (2011) 123-224.
[21] Y. H. Pao, G. H. Park, D. J. Sobajic, Learning and generalization characteristics of the random vector
functional-link net, Neurocomputing 6 (2) (1994) 163-180.
[22] Y. H. Pao, Y. Takefji, Functional-link net computing: theory, system architecture, and functionalities,
IEEE Computer 25 (5) (1992) 76-79.
[23] J. Park, I. Sandberg, Universal approximation using radial basis function networks, Neural Computation
3 (1991) 246-257.
[24] D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning internal representations by backpropagating
errors, Nature 323 (1986) 533-536.
[25] S. Scardapane, D. Wang, Randomness in neural networks: An overview, WIREs Data Mining and
Knowledge Discovery (2017) e1200. doi: 10.1002/widm.1200.
[26] W. F. Schmidt, M. A. Kraaijveld, R. P. Duin, Feedforward neural networks with random weights, Proc.
of 11th IAPR International Conference on Pattern Recognition, Vol. II. Conference B: Pattern Recognition
Methodology and Systems (1992) 1-4.
[27] I. Tyukin, D. Prokhorov, Feasibility of random basis function approximators for modeling and control,
Proc. of IEEE Multi-Conference on Systems and Control, Saint Petersburg, Russia, (2009) 1391-1396.
[28] D. Wang, Editorial: Randomized algorithms for training neural networks, Information Sciences 364-365
(2016) 126-128.

18

