arXiv:1707.04035v2 [stat.ML] 23 Nov 2017

Kafnets: kernel-based non-parametric activation
functions for neural networks
Simone Scardapanea,∗, Steven Van Vaerenberghc , Simone Totarob , Aurelio
Uncinia
a

Department of Information Engineering, Electronics and Telecommunications (DIET),
Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy
b
Department of Statistical Sciences, Sapienza University of Rome, Piazzale Aldo Moro 5,
00185 Rome, Italy.
c
Department of Communications Engineering, University of Cantabria, Av. los Castros
s/n, 39005 Santander, Cantabria, Spain.

Abstract
Neural networks are generally built by interleaving (adaptable) linear layers with (fixed) nonlinear activation functions. To increase their flexibility,
several authors have proposed methods for adapting the activation functions
themselves, endowing them with varying degrees of flexibility. None of these
approaches, however, have gained wide acceptance in practice, and research
in this topic remains open. In this paper, we introduce a novel family of flexible activation functions that are based on an inexpensive kernel expansion
at every neuron. Leveraging over several properties of kernel-based models,
we propose multiple variations for designing and initializing these kernel activation functions (KAFs), including a multidimensional scheme allowing to
nonlinearly combine information from different paths in the network. The resulting KAFs can approximate any mapping defined over a subset of the real
line, either convex or nonconvex. Furthermore, they are smooth over their
entire domain, linear in their parameters, and they can be regularized using
any known scheme, including the use of `1 penalties to enforce sparseness. To
the best of our knowledge, no other known model satisfies all these properties
simultaneously. In addition, we provide a relatively complete overview on al∗

Corresponding author. Phone: +39 06 44585495, Fax: +39 06 4873300.
Email addresses: simone.scardapane@uniroma1.it (Simone Scardapane),
steven.vanvaerenbergh@unican.es (Steven Van Vaerenbergh),
aurelio.uncini@uniroma1.it (Aurelio Uncini)
Preprint submitted to Neural Networks

November 27, 2017

ternative techniques for adapting the activation functions, which is currently
lacking in the literature. A large set of experiments validates our proposal.
Keywords: Neural networks, Activation functions, Kernel methods
1. Introduction
Neural networks (NNs) are powerful approximators, which are built by interleaving linear layers with nonlinear mappings (generally called activation
functions). The latter step is usually implemented using an element-wise,
(sub-)differentiable, and fixed nonlinear function at every neuron. In particular, the current consensus has shifted from the use of contractive mappings
(e.g., sigmoids) to the use of piecewise-linear functions (e.g., rectified linear
units, ReLUs (Glorot and Bengio, 2010)), allowing a more efficient flow of
the backpropagated error (Goodfellow et al., 2016). This relatively inflexible architecture might help explaining the extreme redundancy found in the
trained parameters of modern NNs (Denil et al., 2013).
Designing ways to adapt the activation functions themselves, however,
faces several challenges. On one hand, we can parameterize a known activation function with a small number of trainable parameters, describing for
example the slope of a particular linear segment (He et al., 2015). While
immediate to implement, this only results in a small increase in flexibility
and a marginal improvement in performance in the general case (Agostinelli
et al., 2014). On the other hand, a more interesting task is to devise a scheme
allowing each activation function to model a large range of shapes, such as
any smooth function defined over a subset of the real line. In this case, the
inclusion of one (or more) hyper-parameters enables the user to trade off
between greater flexibility and a larger number of parameters per-neuron.
We refer to these schemes in general as non-parametric activation functions,
since the number of (adaptable) parameters can potentially grow without
bound.
There are three main classes of non-parametric activation functions known
in the literature: adaptive piecewise linear (APL) functions (Agostinelli et al.,
2014), maxout networks (Goodfellow et al., 2013), and spline activation functions (Guarnieri et al., 1999). These are described more in depth in Section
5. In there, we also argue that none of these approaches is fully satisfactory, meaning that each of them loses one or more desirable properties, such

2

as smoothness of the resulting functions in the APL and maxout cases (see
Table 1 in Section 6 for a schematic comparison).
In this paper, we propose a fourth class of non-parametric activation
functions, which are based on a kernel representation of the function. In particular, we define each activation function as a linear superposition of several
kernel evaluations, where the dictionary of the expansion is fixed beforehand
by sampling the real line. As we show later on, the resulting kernel activation
functions (KAFs) have a number of desirable properties, including: (i) they
can be computed cheaply using vector-matrix operations; (ii) they are linear
with respect to the trainable parameters; (iii) they are smooth over their
entire domain; (iv) using the Gaussian kernel, their parameters only have
local effects on the resulting shapes; (v) the parameters can be regularized
using any classical approach, including the possibility of enforcing sparseness
through the use of `1 norms. To the best of our knowledge, none of the known
methods possess all these properties simultaneously. We call a NN endowed
with KAFs at every neuron a Kafnet.
Importantly, framing our method as a kernel technique allows us to potentially leverage over a huge literature on kernel methods, either in statistics,
machine learning (Hofmann et al., 2008), and signal processing (Liu et al.,
2011). Here, we preliminarly demonstrate this by discussing several heuristics
to choose the kernel hyper-parameter, along with techniques for initializing
the trainable parameters. However, much more can be applied in this context, as we discuss in depth in the conclusive section. We also propose a
bi-dimensional variant of our KAF, allowing the information from multiple
linear projections to be nonlinearly combined in an adaptive fashion.
In addition, we contend that one reason for the rareness of flexible activation functions in practice can be found in the lack of a cohesive (introductory)
treatment on the topic. To this end, a further aim of this paper is to provide
a relatively comprehensive overview on the selection of a proper activation
function. In particular, we divide the discussion on the state-of-the-art in
three separate sections. Section 3 introduces the most common (fixed) activation functions used in NNs, from the classical sigmoid function up to the
recently developed self-normalizing unit (Klambauer et al., 2017) and Swish
function (Ramachandran et al., 2017). Then, we describe in Section 4 how
most of these functions can be efficiently parameterized by one or more adaptive scalar values in order to enhance their flexibility. Finally, we introduce
the three existing models for designing non-parametric activation functions
in Section 5. For each of them, we briefly discuss relative strengths and draw3

backs, which serve as a motivation for introducing the model subsequently.
The rest of the paper is composed of four additional sections. Section 6
describes the proposed KAFs, together with several practical implementation
guidelines regarding the selection of the dictionary and the choice of a proper
initialization for the weights. For completeness, Section 7 briefly describes
additional strategies to improve the activation functions, going beyond the
addition of trainable parameters in the model. A large set of experiments is
described in Section 8, and, finally, the main conclusions and a set of future
lines of research are given in Section 9.
Notation
We denote vectors using boldface lowercase letters, e.g., a; matrices are
denoted by boldface uppercase letters, e.g., A. All vectors are assumed to be
column vectors. The operator k·kp is the standard `p norm on an Euclidean
space. For p = 2, it coincides with the Euclidean norm, while for p = 1 we
obtain the Manhattan
(or taxicab) norm defined for a generic vector v ∈ RB
PB
as kvk1 = k=1 |vk |. Additional notations are introduced along the paper
when required.
2. Preliminaries
We consider training a standard feedforward NN, whose l-th layer is described by the following equation:
hl = gl (Wl hl−1 + bl ) ,

(1)

where hl−1 ∈ RNl−1 is the Nl−1 -dimensional input to the layer, Wl ∈ RNl−1 ×Nl
and bl ∈ RNl are adaptable weight matrices, and gl (·) is a nonlinear function,
called activation function, which is applied element-wise. In a NN with L
layers, x = h0 denotes the input to the network, while ŷ = hL denotes the
final output.
For training the network, we are provided with a set of I input/output
pairs S = {xi , yi }Ii=1 , and we minimize a regularized cost function given by:
J(w) =

I
X

l(yi , ŷi ) + C · r(w) ,

(2)

i=1

where w ∈ RQ collects all the trainable parameters of the network, l(·, ·) is
4

a loss function (e.g., the squared loss), r(·) is used to regularize the weights
using, e.g., `2 or `1 penalties, and the regularization factor C > 0 balances
the two terms.
In the following, we review common choices for the selection of gl , before
describing methods to adapt them based on the training data. For readability, we will drop the subscript l, and we use the letter s to denote a single
input to the function, which we call an activation. Note that, in most cases,
the activation function gL for the last layer cannot be chosen freely, as it
depends on the task and a proper scaling of the output. In particular, it is
common to select g(s) = s for regression problems, and a sigmoid function
for binary problems with yi ∈ {0, 1}:
g(s) = δ(s) =

1
.
1 + exp {−s}

(3)

For multi-class problems with dummy encodings on the output, the softmax
function generalizes the sigmoid and it ensures valid probability distributions
in the output (Bishop, 2006).
3. Fixed activation functions
We briefly review some common (fixed) activation functions for neural
networks, that are the basis for the parametric ones in the next section.
Before the current wave of deep learning, most activation functions used in
NNs were of a ‘squashing’ type, i.e., they were monotonically non-decreasing
functions satisfying:
lim g(s) = c, lim = 1 ,
(4)
s→−∞

s→∞

where c can be either 0 or −1, depending on convention. Apart from the
sigmoid in (3), another common choice is the hyperbolic tangent, defined as:
g(s) = tanh(s) =

exp {s} − exp {−s}
.
exp {s} + exp {−s}

(5)

Cybenko (1989) proved the universal approximation property for this class
of functions, and his results were later extended to a larger class of functions
in Hornik et al. (1989). In practice, squashing functions were found of limited use in deep networks (where L is large), being prone to the problem of
vanishing and exploding gradients, due to their bounded derivatives (Hochre5

iter et al., 2001). A breakthrough in modern deep learning came from the
introduction of the rectifier linear unit (ReLU) function, defined as:
g(s) = max {0, s} .

(6)

Despite being unbounded and introducing a point of non-differentiability,
the ReLU has proven to be extremely effective for deep networks (Glorot
and Bengio, 2010; Maas et al., 2013). The ReLU has two main advantages.
First, its gradient is either 0 or 1,1 making back-propagation particularly
efficient. Secondly, its activations are sparse, which is beneficial from several
points of views. A smoothed version of the ReLU, called softplus, is also
introduced in Glorot et al. (2011):
g(s) = log {1 + exp {s}} .

(7)

Despite its lack of smoothness, ReLU functions are almost always preferred
to the softplus in practice. One obvious problem of ReLUs is that, for a
wrong initialization or an unfortunate weight update, its activation can get
stuck in 0, irrespective of the input. This is referred to as the ‘dying ReLU’
condition. To circumvent this problem, Maas et al. (2013) introduced the
leaky ReLU function, defined as:
(
s
if s ≥ 0
g(s) =
,
(8)
αs otherwise
where the user-defined parameter α > 0 is generally set to a small value,
such as 0.01. While the resulting pattern of activations is not exactly sparse
anymore, the parameters cannot get stuck in a poor region. (8) can also be
written more compactly as g(s) = max {0, s} + α min {0, s}.
Another problem of activation functions having only non-negative output
values is that their mean value is always positive by definition. Motivated
by an analogy with the natural gradient, Clevert et al. (2016) introduced the
exponential linear unit (ELU) to renormalize the pattern of activations:
1

For s = 0, the function is not differentiable, but any value in [0, 1] is a valid subgradient. Most implementations of ReLU use 0 as the default choice in this case.

6

(
s
g(s) = ELU(s)
α (exp {s} − 1)

if s ≥ 0
,
otherwise

(9)

where in this case α is generally chosen as 1. The ELU modifies the negative
part of the ReLU with a function saturating at a user-defined value α. It is
computationally efficient, being smooth (differently from the ReLU and the
leaky ReLU), and with a gradient which is either 1 or g(s) + α for negative
values of s.
The recently introduced scaled ELU (SELU) generalizes the ELU to have
further control over the range of activations (Klambauer et al., 2017):
g(s) = SELU(s) = λ · ELU(s) ,

(10)

where λ > 1 is a second user-defined parameter. Particularly, it is shown in
Klambauer et al. (2017) that for λ ≈ 1.6733 and α ≈ 1.0507, the successive
application of (1) converges towards a fixed distribution with zero mean and
unit variance, leading to a self-normalizing network behavior.
Finally, Swish (Ramachandran et al., 2017) is a recently proposed activation somewhat inspired by the gating steps in a standard LSTM recurrent
cell:
g(s) = s · δ(s) ,
(11)
where δ(s) is the sigmoid in (3).
4. Parametric adaptable activation functions
An immediate way to increase the flexibility of a NN is to parameterize
one of the previously introduced activation functions with a fixed (small)
number of adaptable parameters, such that each neuron can adapt its activation function to a different shape. As long as the function remains differentiable with respect to these new parameters, it is possible to adapt them
with any numerical optimization algorithm together with the linear weights
and biases of the layer. Due to their fixed number of parameters and limited
flexibility, we call these parametric activation functions.
Historically, one of the first proposals in this sense was the generalized
hyperbolic tangent (Chen and Chang, 1996), a tanh function parameterized

7

by two additional positive scalar values a and b:
g(s) =

a (1 − exp {−bs})
.
1 + exp {−bs}

(12)

Note that the parameters a, b are initialized randomly and are adapted independently for every neuron. Specifically, a determines the range of the
output (which is called the amplitude of the function), while b controls the
slope of the curve. Trentin (2001) provides empirical evidence that learning
the amplitude for each neuron is beneficial (either in terms of generalization error, or speed of convergence) with respect to having unit amplitude
for all activation functions. Similar results were also obtained for recurrent
networks (Goh and Mandic, 2003).
More recently, He et al. (2015) consider a parametric version of the leaky
ReLU in (8), where the coefficient α is initialized at α = 0.25 everywhere and
then adapted for every neuron. The resulting activation function is called
parametric ReLU (PReLU), and it has a very simple derivative with respect
to the new parameter:
(
0 if s ≥ 0
dg(s)
=
.
(13)
dα
s otherwise
For a layer with N hidden neurons, this introduces only N additional parameters, compared to 2N parameters for the generalized tanh. Importantly, in
the case of `p regularization, the user has to be careful not to regularize the
α parameters, which would bias the optimization process towards classical
ReLU / leaky ReLU activation functions.
Similarly, Trottier et al. (2016) propose a modification of the ELU function in (9) with an additional scalar parameter β, called parametric ELU
(PELU):
α

if s ≥ 0
β s




,
(14)
g(s) =
s

α exp
−1
otherwise
β
where both α and β are initialized randomly and adapted during the training
process. Based on the analysis in Jin et al. (2016), there always exists a
setting for the linear weights and α, β which avoids the vanishing gradient
problem. Differently from the PReLU, however, the two parameters should
8

be regularized in order to avoid a degenerate behavior with respect to the
linear weights, where extremely small linear weights are coupled with very
large values for the parameters of the activation functions.
A more flexible proposal is the S-shaped ReLU
 (SReLU) (Jin et al., 2016),
which is parameterized by four scalar values tr , ar , tl , al ∈ R4 :

r
r
r

t + a (s − t )
g(s) = s


l
t + al s − tl

if s ≥ tr
if tr > s > tl .
otherwise

(15)

The SReLU is composed by three linear segments, the middle of which is the
identity. Differently from the PReLU, however, the cut-off points between
the three segments can also be adapted. Additionally, the function can have
both convex and nonconvex shapes, depending on the orientation of the left
and right segments, making it more flexible than previous proposals. Similar
to the PReLU, the four parameters should not be regularized (Jin et al.,
2016).
Finally, a parametric version of the Swish function is the β-swish (Ramachandran et al., 2017), which includes a tunable parameter β inside the
self-gate:
g(s) = s · δ(βs) .
(16)
5. Non-parametric activation functions
Intuitively, parametric activation functions have limited flexibility, resulting in mixed performance gains on average. Differently from parametric
approaches, non-parametric activation functions allow to model a larger class
of shapes (in the best case, any continuous segment), at the price of a larger
number of adaptable parameters. As stated in the introduction, these methods generally introduce a further global hyper-parameter allowing to balance
the flexibility of the function, by varying the effective number of free parameters, which can potentially grow without bound. Additionally, the methods
can be grouped depending on whether each parameter has a local or global
effect on the overall function, the former being a desirable characteristic.
In this section, we describe three state-of-the-art approaches for implementing non-parametric activation functions: APL functions in Section 5.1,
spline functions in Section 5.2, and maxout networks in Section 5.3.

9

5.1. Adaptive piecewise linear methods
An APL function, introduced in Agostinelli et al. (2014), generalizes the
SReLU function in (15) by summing multiple linear segments, where all
slopes and cut-off points are learned under the constraint that the overall
function is continuous:
g(s) = max {0, s} +

S
X

ai max {0, −s + bi } .

(17)

i=1

S is a hyper-parameter chosen by the user, while each APL is parameterized by 2S adaptable parameters {ai , bi }Si=1 . These parameters are randomly
initialized for each neuron, and can be regularized with `2 regularization,
similarly to the PELU, in order to avoid the coupling of very small linear
weights and very large ai coefficients for the APL units.
The APL unit cannot approximate any possible function. Its approximation properties are described in the following theorem.
Theorem 1 (Theorem 1, (Agostinelli et al., 2014)). The APL unit can approximate any continuous piecewise-linear function h(s), for some choice of
S and {ai , bi }Si=1 , provided that h(s) satisfies the following two conditions:
1. There exists u ∈ R such that h(s) = s, ∀s ≥ u.
= t, ∀s < v.
2. There exist two scalars, v, t ∈ R such that dh(s)
s
The previous theorem implies that any piecewise-linear function can be approximated, provided that its behavior is linear for very large, or small, s. A
possible drawback of the APL activation function is that it introduces S + 1
points of non-differentiability for each neuron, which may damage the optimization algorithm. The next class of functions solves this problem, at the
cost of a possibly larger number of parameters.
5.2. Spline activation functions
An immediate way to exploit polynomial interpolation in NNs it to build
the activation function over powers of the activation s (Piazza et al., 1992):
g(s) =

P
X
i=0

10

ai s i ,

(18)

where P is a hyper-parameter and we adapt the (P + 1) coefficients {ai }Pi=0 .
Since a polynomial of degree P can pass exactly through P + 1 points, this
polynomial activation function (PAF) can in theory approximate any smooth
function. The drawback of this approach is that each parameter ai has a
global influence on the overall shape, and the output of the function can
easily grow too large or encounter numerical problems, particularly for large
absolute values of s and large P .
An improved way to use polynomial expansions is spline interpolation,
giving rise to the spline activation function (SAF). The SAF was originally
studied in Vecci et al. (1998); Guarnieri et al. (1999), and later re-introduced
in a more modern context in Scardapane et al. (2016), following previous
advances in nonlinear filtering (Scarpiniti et al., 2013). In the sequel, we
adopt the newer formulation.
A SAF is described by a vector of T parameters, called knots, corresponding to a sampling of its y-values over an equispaced grid of T points over the
x-axis, that are symmetrically chosen around the origin with sampling step
∆x. For any other value of s, the output value of the SAF is computed
with spline interpolation over the closest knot and its P rightmost neighbors, where P is generally chosen equal to 3, giving rise to a cubic spline
interpolation scheme. Specifically, denote by k the index of the closest knot,
and by qk the vector comprising the corresponding knot and its P neighbors.
We call this vector the span. We also define a new value
j s k
s
−
,
(19)
u=
∆x
∆x
where ∆x is the user-defined sampling step. u defines a normalized abscissa
value between the k-th knot and the (k + 1)-th one. The output of the SAF
is then given by (Scarpiniti et al., 2013):
g(s) = uT Bqk ,

(20)

where the vector u collects powers of u up to the order P :

T
u = uP , uP −1 , . . . , u1 , 1 ,

(21)

and B is the spline basis matrix, which defines the properties of the interpolation scheme (as shown later in Fig. 1b). For example, the popular

11

6
SAF output

SAF output

6
4
2
0

4
2
0

2

0
Activation

2

2

(a) CR matrix

0
Activation

2

(b) B-basis matrix

Figure 1: Example of output interpolation using a SAF neuron. Knots are shown with
red markers, while the overall function is given in a light blue. (a) For a given activation,
in black, only the control points in the green background are active. (b) We use the same
control points as before, but we interpolate using the B-basis matrix (Scarpiniti et al.,
2013) instead of the CR matrix in (22). The resulting curve is smoother, but it is not
guaranteed to pass through all the control points.

Catmull-Rom basis for P = 3 is given by:


−1 3 −3 1
1  2 −5 4 −1
.
B= 
1
0
2 −1 0
0
2
0
0

(22)

The derivatives of the SAF can be computed in a similar way, both with
respect to s and with respect to qk , e.g., see Scardapane et al. (2016). A
visual example of the SAF output is given in Fig. 1.
Each knot has only a limited local influence over the output, making their
adaptation more stable. The resulting function is also smooth, and can in
fact approximate any smooth function defined over a subset of the real line
to a desired level of accuracy, provided that ∆x is chosen small enough. The
drawback is that regularizing the resulting activation functions is harder to
achieve, since `p regularization cannot be applied directly to the values of the
knots. In Guarnieri et al. (1999), this was solved by choosing a large ∆x, in
turn severely limiting the flexibility of the interpolation scheme. A different
proposal was made in Scardapane et al. (2016), where the vector q ∈ RT of
12

SAF parameters is regularized by penalizing deviations from the values at
initialization. Note that it is straightforward to initialize the SAF as any of
the known fixed activation functions described before.
5.3. Maxout networks
Differently from the other functions described up to now, the maxout
function introduced in Goodfellow et al. (2013) replaces an entire layer in
(1). In particular, for each neuron, instead of computing a single dot product
wT h to obtain the activation (where h is the input to the layer), we compute
K different products with K separate weight vectors w1 , . . . , wK and biases
b1 , . . . , bK , and take their maximum:

g(h) = max wiT h + bi ,
(23)
i=1,...,K

where the activation function is now a function of a subset of the output
of the previous layer. A NN having maxout neurons in all hidden layers is
called a maxout network, and remains an universal approximator according
to the following theorem.
Theorem 2 (Theorem 4.3, (Goodfellow et al., 2013)). Any continuous function h(·) : RN0 → RNL can be approximated arbitrarily well on a compact
domain by a maxout network with two maxout hidden units, provided K is
chosen sufficiently large.
The advantage of the maxout function is that it is extremely easy to implement using current linear algebra libraries. However, the resulting functions
have several points of non-differentiability, similarly to the APL units. In
addition, the number of resulting parameters is generally higher than with
alternative formulations. In particular, by increasing K we multiply the original number of parameters by a corresponding factor, while other approaches
contribute only linearly to this number. Additionally, we lose the possibility
of plotting the resulting activation functions, unless the input to the maxout
layer has less than 4 dimensions. An example with dimension 1 is shown in
Fig. 2.
In order to solve the smoothness problem, (Zhang et al., 2014) introduced
two smooth versions of the maxout neuron. The first one is the soft-maxout:
(K
)
X
 T
g(h) = log
exp wi h + bi
.
(24)
i=1

13

Maxout activation

2
1
0
1

2

0
1D input

2

Figure 2: An example of a maxout neuron with a one-dimensional input and K = 3. The
three linear segments are shown with a light gray, while the resulting activation is shown
with a shaded red. Note how the maxout can only generate convex shapes by definition.
However, plots cannot be made for inputs having more than three dimensions.

The second one is the `p -maxout, for a user-defined natural number p:
v
uK
uX
p
p
g(h) = t
|wiT h + bi | .
(25)
i=1

Closely related to the `p -maxout neuron is the Lp unit proposed in Gulcehre
et al. (2013). Denoting for simplicity si = wiT h + bi , the Lp unit is defined
as:
! p1
K
1 X
g(h) =
|si − ci |p
,
(26)
K i=1
where the K+1 parameters {c1 , . . . , cK , p} are all learned via back-propagation.2
If we fix ci = 0, for p going to infinity the Lp unit degenerates to a special
case of the maxout neuron:
lim g(h) = max {|si |} .

p→∞

i=1,...,K

2

(27)

In practice, p is re-parameterized as 1+log {1 + exp {p}} to guarantee that (26) defines
a proper norm.

14

6. Proposed kernel-based activation functions
In this section we describe the proposed KAF. Specifically, we model each
activation function in terms of a kernel expansion over D terms as:
g(s) =

D
X

αi κ (s, di ) ,

(28)

i=1
D
where {αi }D
i=1 are the mixing coefficients, {di }i=1 are the called the dictionary
elements, and κ(·, ·) : R × R → R is a 1D kernel function (Hofmann et al.,
2008). In kernel methods, the dictionary elements are generally selected
from the training data. In a stochastic optimization setting, this means
that D would grow linearly with the number of training iterations, unless
some proper strategy for the selection of the dictionary is implemented (Liu
et al., 2011; Van Vaerenbergh et al., 2012). To simplify our treatment, we
consider a simplified case where the dictionary elements are fixed, and we
only adapt the mixing coefficients. In particular, we sample D values over
the x-axis, uniformly around zero, similar to the SAF method, and we leave
D as a user-defined hyper-parameter. This has the additional benefit that the
resulting model is linear in its adaptable parameters, and can be efficiently
implemented for a mini-batch of training data using highly-vectorized linear
algebra routines. Note that there is a vast literature on kernel methods
with fixed dictionary elements, particularly in the field of Gaussian processes
(Snelson and Ghahramani, 2006).
The kernel function κ(·, ·) needs only respect the positive semi-definiteness
D
property, i.e., for any possible choice of {αi }D
i=1 and {di }i=1 we have that:
D X
D
X

αi αj κ (di , dj ) ≥ 0 .

(29)

i=1 j=1

For our experiments, we use the 1D Gaussian kernel defined as:

κ(s, di ) = exp −γ (s − di )2 ,

(30)

where γ ∈ R is called the kernel bandwidth, and its selection is discussed
more at length below. Other choices, such as the polynomial kernel with

15

KAF
2

0
Activation
(a) γ = 2.0

2

2

0
Activation

2

(b) γ = 0.5

2

0
Activation

2

(c) γ = 0.1

Figure 3: Examples of KAFs. In all cases we sample uniformly 20 points on the x-axis,
while the mixing coefficients are sampled from a normal distribution. The three plots show
three different choices for γ.

p ∈ N, are also possible:
κ(s, di ) = (1 + sdi )p .

(31)

By the properties of kernel methods, KAFs are equivalent to learning linear
functions over a large number of nonlinear transformations of the original activation s, without having to explicitly compute such transformations. The
Gaussian kernel has an additional benefit: thanks to its definition, the mixing coefficients have only a local effect over the shape of the output function
(where the radius depends on γ, see below), which is advantageous during
optimization. In addition, the expression in (28) with the Gaussian kernel can approximate any continuous function over a subset of the real line
(Micchelli et al., 2006). The expression resembles a one-dimensional radial
basis function network, whose universal approximation properties are also
well studied (Park and Sandberg, 1991). Below, we go more in depth over
some additional considerations for implementing our KAF model. Note that
the model has very simple derivatives for back-propagation:
∂g(s)
= κ (s, di ) ,
∂αi
D
∂g(s) X ∂κ (s, di )
=
αi
.
∂s
∂s
i=1

(32)
(33)

On the selection of the kernel bandwidth
Selecting γ is crucial for the well-behavedness of the method, by acting
indirectly on the effective number of adaptable parameters. In Fig. 3 we show
16

some examples of functions obtained by fixing D = 20, randomly sampling
the mixing coefficients, and only varying the kernel bandwidth, showing how
γ acts on the smoothness of the overall functions.
In the literature, many methods have been proposed to select the bandwidth parameter for performing kernel density estimation (Jones et al., 1996).
These methods include popular rules of thumb such as Scott (2015) or Silverman (1986).
In the problem of kernel density estimation, the abscissa corresponds to a
given dataset with an arbitrary distribution. In the proposed KAF scheme,
the abscissa are chosen according to a grid, and as such the optimal bandwidth parameter depends uniquely on the grid resolution. Instead of leaving the bandwidth parameter γ as an additional hyper-parameter, we have
empirically verified that the following rule of thumb represents a good compromise between smoothness (to allow an accurate approximation of several
initialization functions) and flexibility:
γ=

1
,
6∆2

(34)

where ∆ is the distance between the grid points. We also performed some
experiments in which γ was adapted through back-propagation, though this
did not provide any gain in accuracy.
On the initialization of the mixing coefficients
A random initialization of the mixing coefficients from a normal distribution, as in Fig. 3, provides good diversity for the optimization process.
Nonetheless, a further advantage of our scheme is that we can initialize some
(or all) of the KAFs to follow any know activation function, so as to guarantee a certain desired behavior. Specifically, denote by t = [t1 , . . . , tD ]T the
vector of desired initial KAF values corresponding to the dictionary elements
d = [d1 , . . . , dD ]T . We can initialize the mixing coefficients α = [α1 , . . . , αD ]T
using kernel ridge regression:
α = (K + εI)−1 t ,

(35)

where K ∈ RD×D is the kernel matrix computed between t and d, and we
add a diagonal term with ε > 0 to avoid degenerate solutions with very large
mixing coefficients. Two examples are shown in Fig. 4.

17

3
2
KAF

KAF

0.5
0.0

1
0

0.5
4

2

0
Activation

2

4

4

(a) tanh

2

0
Activation

2

4

(b) ELU

Figure 4: Two examples of initializing a KAF using (35), with ε = 10−6 . (a) A hyperbolic
tangent. (b) The ELU in (9). The red dots indicate the corresponding initialized values
for the mixing coefficients.

Multi-dimensional kernel activation functions
In our experiments, we also consider a two-dimensional variant of the
proposed KAF, that we denote as 2D-KAF. Roughly speaking, the 2D-KAF
acts on a pair of activation values, instead of a single one, and learns a twodimensional function to combine them. It can be seen as a generalization
of a two-dimensional maxout neuron, which is instead constrained to output
the maximum value among the two inputs.
2
Similarly to before, we construct a dictionary d ∈ RD ×2 by sampling a
uniform grid over the 2D plane, by considering D positions uniformly spaced
around 0 in both dimensions. We group the incoming activation values in
pairs (assuming that the layer has even size), and for each possible pair of
activations s = [sk , sk+1 ]T we output:
2

g (s) =

D
X

αi κ (s, di ) ,

(36)

i=1

where di is the i-th element of the dictionary, and we now have D2 adaptable
2
coefficients {αi }D
i=1 . In this case, we consider the 2D Gaussian kernel:

κ (s, di ) = exp −γ ks − di k22 ,
(37)
where we use the same rule of thumb in (34), multiplied by
18

√
2, to select

γ. The increase in parameters is counter-balanced by two factors. Firstly,
by grouping the activations we halve the size of the linear matrix in the
subsequent layer. Secondly, we generally choose a smaller D with respect to
the 1D case, i.e., we have found that values in [5, 10] are enough to provide a
good degree of flexibility. Table 1 provides a comparison of the two proposed
KAF models to the three alternative non-parametric activation functions
described before. We briefly mention here that a multidimensional variant
of the SAF was explored in Solazzi and Uncini (2000).
7. Related work
Many authors have considered ways of improving the performance of the
classical activation functions, which do not necessarily require to adapt their
shape via numerical optimization, or that require special care when implemented. For completeness, we briefly review them here before moving to the
experimental section.
As stated before, the problem of ReLU is that its gradient is zero outside
the ‘active’ regime where s ≥ 0. To solve this, the randomized leaky ReLU
(Xu et al., 2015) considers a leaky ReLU like in (8), in which during training
the parameter α is randomly sampled at every step in the uniform distribution U(l, u), and the lower/upper bounds {l, u} are selected beforehand. To
compensate with the stochasticity in training, in the test phase α is set equal
to:
l+u
,
(38)
α=
2
which is equivalent to taking the average of all possible values seen during
training. This is similar to the dropout technique (Srivastava et al., 2014),
which randomly deactivates some neurons during each step of training, and
later rescales the weights during the test phase.
More in general, several papers have developed stochastic versions of the
classical artificial neurons, whose output depend on one or more random variables sampled during their execution (Bengio et al., 2013), under the idea that
the resulting noise can help guide the optimization process towards better
minima. Notably, this provides a link between classical NNs and other probabilistic methods, such as generative networks and networks trained using
variational inference (Bengio et al., 2014; Schulman et al., 2015). The main
challenge is to design stochastic neurons that provide a simple mechanism for
back-propagating the error through the random variables, without requiring
19

20

No
Yes

Maxout

Proposed KAF
Yes

Yes

Yes**
Yes**

Yes

No

Only `2 regularization

No

Yes

Partially

Yes

Yes

No

*

Yes

Yes

Ni−1 Ni + Ni + 2SNi

Trainable weights

Size of the dictionary D

Size of the dictionary D

Number of affine maps K

Table 1: A comparison of the existing non-parametric activation functions and the proposed KAF and 2D-KAF. In our
definition, an activation function is local if each adaptable weight only affects a small portion of the output values.

Ni−1 Ni + Ni + DNi
(Ni +D2 Ni )
Ni−1 Ni +
2

KNi−1 Ni + KNi

Number of control points Q Ni−1 Ni + Ni + QNi

Number of segments S

Hyper-parameter

Maxout functions can only be plotted whenever Ni−1 ≤ 3, which is almost never the case.
Only when using the Gaussian (or similar) kernel function.

**

*

Yes

Yes

SAF

Proposed 2D-KAF

No

Smooth Locality Can use regularization Plottable

APL

Name

Noise

Value
0

1

5

0
5
Activation
(a) Original function

5

0
Activation
(b) Noise

5

Noisy activation function

1

1

1

0
5

0
5
Activation
(c) Noisy activation function

Figure 5: An example of noisy activation function. (a) Original sigmoid function (blue),
together with its hard-thresholded version (in red). (b) For any possible activation value
outside the saturated regime, we add random half-normal noise with increasing variance
and matching sign according to the algorithm in Gulcehre et al. (2016) (the shaded areas
correspond to one standard deviation). (c) Final noisy activation function computed as
in Gulcehre et al. (2016). At test time, only the expected values (represented with solid
green line) are returned.

expensive sampling procedures, and with a minimal amount of interference
over the network. As a representative example, the noisy activation functions proposed in Gulcehre et al. (2016) achieve this by combining activation
functions with ‘hard saturating’ regimes (i.e., their value is exactly zero outside a limited range) with random noise over the outputs, whose variance
increases in the regime where the function saturates to avoid problems due
to the sparse gradient terms. An example is given in Fig. 5.
Another approach is to design vector-valued activation functions to maximize parameter sharing. In the simplest case, the concatenated ReLU (Shang
et al., 2016) returns two output values by applying a ReLU function both
on s and on −s. Similarly, the order statistic network (Rennie et al., 2014)
modifies a maxout neuron by returning the input activations in sorted order,
instead of picking the highest value only. Multi-bias activation functions (Li
et al., 2016) compute several activations values by using different bias terms,
and then apply the same activation function independently over each of the
resulting values. The network-in-network (Lin et al., 2013) model is a nonparametric approach specific to convolutional neural networks, wherein the
nonlinear units are replaced with a fully connected NN.
For specific tasks of audio modeling, some authors have proposed the
use of Hermite polynomials for adapting the activation functions (Siniscalchi
et al., 2013; Siniscalchi and Salerno, 2017). Similarly to our proposed KAF,
the functions are expressed as a weighted sum of several fixed nonlinear trans21

formations of the activation values, i.e., the Hermite polynomials. However,
the nonlinear transformations are computed through the use of a recurrence
formula, thus highly increasing the computational load.
8. Experimental results
In this section we provide a comprehensive evaluation of the proposed
KAFs and 2D-KAFs when applied to several use cases. As a preliminary experiment, we begin by comparing multiple activation functions on a relatively
small classification dataset (Sensorless) in Section 8.2, where we discuss several examples of the shapes that are generally obtained by the networks and
initialization strategies. We then consider a large-scale dataset taken from
Baldi et al. (2014) in Section 8.3, where we show that two layers of KAFs
are able to significantly outperform a feedforward network with five hidden
layers, even when considering parametric activation functions and state-ofthe-art regularization techniques. In Section 8.4 we show that KAFs and
2D-KAF provide an increase in performance also when applied to convolutional layers on the CIFAR-10 dataset. Finally, we show in Section 8.5 that
they have significantly faster training and higher cumulative reward for a
set of reinforcement learning scenario using MuJoCo environments from the
OpenAI Gym3 . We provide an open-source library to replicate the experiments, with the implementation of KAFs and 2D-KAFs in three separate
frameworks, i.e., AutoGrad4 , TensorFlow5 , and PyTorch6 , which is publicly
accessible on the web7 .
8.1. Experimental setup
Unless noted otherwise, in all experiments we linearly preprocess the input features between -1 and +1, and we substitute eventual missing values
with the median values computed from the corresponding feature columns.
From the full dataset, we randomly keep a portion of the dataset for validation and another portion for test. All neural networks use a softmax
3

https://gym.openai.com/
https://github.com/HIPS/autograd
5
https://www.tensorflow.org/
6
http://pytorch.org/
7
https://github.com/ispamm/kernel-activation-functions/
4

22

activation function in their output layer, and they are trained by minimizing the average cross-entropy on the training dataset, to which we add a
small `2 -regularization term whose weight is selected in accordance to the
literature. For optimization, we use the Adam algorithm (Kingma and Ba,
2015) with mini-batches of 100 elements and default hyper-parameters. For
each epoch we compute the accuracy over the validation set, and we stop
training whenever the validation accuracy is not improving for 15 consecutive epochs. Experiments are performed using the PyTorch implementation
on a machine with an Intel Xeon E5-2620 CPU, with 16 GB of RAM and
a CUDA back-end employing an Nvidia Tesla K20c. All accuracy measures
over the test set are computed by repeating the experiments for 5 different
splits of the dataset (unless the splits are provided by the dataset itself)
and initializations of the networks. Weights of the linear layers are always
initialized using the so-called ‘Uniform He’ strategy, while additional parameters introduced by parametric and non-parametric activation functions are
initialized by following the guidelines of the original papers.
8.2. Visualizing the activation functions
We begin with an experiment on the ‘Sensorless’ dataset to investigate
whether KAFs and 2D-KAFs can indeed provide improvements in accuracy
with respect to other baselines, and for visualizing some of the common
shapes that are obtained after training. The Sensorless dataset is a standard benchmark for supervised techniques, composed of 58509 examples with
49 input features representing electric signals, that are used to predict one
among 11 different classes representing operating conditions. We partition it
using a random 15% for validation and another 15% for testing, and we use
a small regularization factor of 10−4 .
In this dataset, we found that the best performing fixed activation function is a simple hyperbolic tangent. In particular, a network with one hidden
layers of 100 neurons achieves a test accuracy of 97.75%, while the best result is obtained with a network of three hidden layers (each composed of 100
neurons), which achieves a test accuracy of 99.18%. Due to the simplicity of
the dataset, we have not found improvements here by adding more layers or
including dropout during training as in the following sections. The best performing parametric activation function is instead the PReLU, that improves
the results by obtaining a 98.48% accuracy with a single hidden layer, and
99.30% with three hidden layers.

23

1

0
Activation

1

2

0
Activation

(a)

1
Activation
(d)

2

2.5

(b)

0

0

Activation
(e)

0.0
Activation

2.5

(c)

2

1

0
1
Activation
(f)

Figure 6: Examples of 6 trained KAFs (with random initialization) on the Sensorless
dataset. On the y-axis we plot the output value of the KAF. The KAF after initialization is
shown with a dashed red, while the final KAF is shown with a solid green. The distribution
of activation values after training is shown as a reference with a light blue.

For comparison, we train several feedforward networks with KAFs in the
hidden layers, having a dictionary of D = 20 elements equispaced between
−3.0 and 3.0, and initializing the linear coefficients from a normal distribution
with mean 0 and variance 0.3. Using this setup, we already outperform all
other baselines obtaining an accuracy of 99.04% with a single hidden layer,
which improves to 99.80% when considering two hidden layers of KAFs.
Although the dataset is relatively simple, the shapes we obtain are representative of all the experiments we performed, and we provide a selection in
Fig. 6. Specifically, the initialization of the KAF is shown with a dashed red
line, while the final KAF is shown with a solid green line. For understanding the behavior of the functions, we also plot the empirical distribution of
the activations on the test set using a light blue in the background. Some
shapes are similar to common activation functions discussed in Section 3,
although they are shifted on the x-axis to correspond to the distribution of
activation values. For example Fig. 6a is similar to an ELU, while Fig. 6d
is similar to a standard saturating function. Also, while in the latter case
the final shape is somewhat determined by the initialization, the final shapes
in general tend to be independent from initialization, as in the case of Fig.
24

6a. Another common shape is that of a radial-basis function, as in Fig. 6f,
which is similar to a Gaussian function centered on the mean of the empirical
distribution. Shapes, however, can be vastly more complex than these. For
example, in Fig. 6e we show a function which acts as a standard saturating
function on the main part of the activations’ distribution, while its right-tail
tends to remove values larger than a given threshold, effectively acting as
a sort of implicit regularizer. In Fig. 6b we show a KAF without an intuitive shape, that selectively amplify (either positively or negatively) multiple
regions of its activation space. Finally, in Fig. 6c we show an interesting
pruning effect, where useless neurons correspond to activation functions that
are practically zero everywhere. This, combined with the possibility of applying `1 -regularization (Scardapane et al., 2017) allows to obtain networks
with a significant smaller number of effective parameters.
Interestingly, the shapes obtained in Fig. 6 seem to be necessary for the
high performance of the networks, and they are not an artifact of initialization. Specifically, we obtain similar accuracies (and similar output functions)
even when initializing all KAFs as close as possible to hyperbolic tangents,
following the method described in Section 6, while we obtain a vastly inferior
performance (in some cases even worse than the baseline), if we initialize the
KAFs randomly and we prevent their adaptation. This (informally) points to
the fact that their flexibility and adaptability seems to be an intrinsic component of their good performance in this experiment and in the following
sections, an aspect that we will return to in our conclusive section.
Results are also similar when using 2D-KAFs, that we initialize with
D = 10 elements on each axis using the same strategy as for KAFs. In this
scenario, they obtain a test accuracy of 98.90% with a single hidden layer,
and an accuracy of 99.84% (thus improving over the KAF) for two hidden
layers. Some examples of obtained shapes are provided in Fig. 7.
8.3. Comparisons on the SUSY benchmark
In this section, we evaluate the algorithms on a realistic large-scale use
case, the SUSY benchmark introduced in Baldi et al. (2014). The task is to
predict the presence of super symmetric particles on a simulation of a collision experiment, starting from 18 features (both low-level and high-level)
describing the simulation itself. The overall dataset is composed of five million examples, of which the last 500000 are used for test, and another 500000
for validation. The task is interesting for several reasons. Due to the nature of the data, even a tiny change in accuracy (measured in terms of area
25

Activation 2

Activation 1
Figure 7: Examples of 8 trained 2D-KAFs on the Sensorless dataset. On the x- and y-axis
we plot the two activation values (ranging from −3 to 3), while we show the output of
the KAF using a heat map, where darker colors represent larger output values and lighter
colors represent lower activation values, respectively.

under the curve, AUC) is generally statistically significant. In the original
paper, Baldi et al. (2014) showed the best AUC was obtained by a deep feedforward network having five hidden layers, with significantly better results
when compared to a shallow network. Surprisingly, Agostinelli et al. (2014)
later showed that a shallow network is in fact sufficient, so long as it uses
non-parametric activation functions (in that case, APL units).
In order to replicate these results with our proposed methods, we consider a baseline network inspired to Baldi et al. (2014), having five hidden
layers with 300 neurons each and ReLU activation functions, with dropout
applied to the last two hidden layer with probability 0.5. For comparison,
we also consider the same architecture, but we substitute ReLUs with ELU,
SELU, and PReLU functions. For SELU, we also substitute the standard
dropout with the customized version proposed in Klambauer et al. (2017).
We compare with simpler networks composed of one or two hidden layers
of 300 neurons, employing Maxout neurons (with K = 5), APL units (with
S = 3 as proposed in Agostinelli et al. (2014)), and the proposed KAFs and
2D-KAFs, following the same random initializations as the previous section.
Results, in terms of AUC and amount of trainable parameters, are given in
Table 2.
There are several clear results that emerge from the analysis of Table 2.
First of all, the use of sophisticated activation functions (such as the SELU),
or of parametric functions (such as the PReLU) can improve performance,
26

Activation function Testing AUC Trainable parameters
ReLU

0.8739(0.001)

ELU

0.8739(0.001)

SELU

0.8745(0.002)

PReLU

0.8748(0.001)

368701

Maxout (one layer)

0.8744(0.001)

17401

Maxout (two layers)

0.8744(0.002)

288301

APL (one layer)

0.8744(0.002)

7801

APL (two layers)

0.8757(0.002)

99901

KAF (one layer)

0.8756(0.001)

12001

KAF (two layers)

0.8758(0.001)

108301

2D-KAF (one layer)

0.8750(0.001)

20851

2D-KAF (two layers)

0.8764(0.002)

81151

367201

Table 2: Results of different activation functions on the SUSY benchmark. The last four
rows are the proposed KAF and 2D-KAF. Standard deviation for the AUC is given between
brackets, the best result is shown in bold, and the second best result is underlined. All
networks with fixed or parametric activation functions have five hidden layers. See the
text for a full description of the architectures.

27

in some cases even significantly. However, these improvements still require
several layers of depth, while they both fail to provide accurate results when
experimenting with shallow networks. On the other hand, all non-parametric
functions are able to achieve similar (or even superior) results, while only requiring one or two hidden layers of neurons. Among them, APL and Maxout
achieve a similar AUC with one layer, but only APL is able to benefit from
the addition of a second layer. Both KAF and 2D-KAF are able to significantly outperform all the competitors, and the overall best result is obtained
by a 2D-KAF network with two hidden layers. This is obtained with a significant reduction in the number of trainable parameters, as also described
more in depth in the following section.
8.4. Experiments with convolutive layers on CIFAR-10
Although our focus has been on feedforward networks, an interesting question is whether the superior performance exhibited by KAFs and 2D-KAFs
can also be obtained on different architectures, such as convolutional neural
networks (CNNs). To investigate this, we train several CNNs on the CIFAR10 dataset, composed of 60000 images of size 32 × 32, belonging to 10 classes.
Since our aim is only to compare different architectures for the convolutional
kernels, we train simple CNNs made by stacking convolutional ‘modules’,
each of which is composed by (a) a convolutive layer with 150 filters, with a
filter size of 5 × 5 and a stride of 1; (b) a max-pooling operation over 3 × 3
windows with stride of 2; (c) a dropout layer with probability of 0.25. We
consider CNNs with a minimum of 2 such modules and a maximum of 5,
where the output of the last dropout operation is flattened before applying a
linear projection and a softmax operation. Our training setup is equivalent
to the previous sections.
We consider different choices for the nonlinearity of the convolutional
filters, using ELU as baseline, and our proposed KAFs and 2D-KAFs. In
order to improve the gradient flow in the initial stages of training, KAFs
in this case are initialized with the KRR strategy using ELU as the target.
The results are shown in Fig. 8, where we show on the left the final test
accuracy, and on the right the number of trainable parameters of the three
architectures.
Interestingly, both KAFs and 2D-KAFs are able to get significantly better
results than the baseline, i.e., even 2 layers of convolutions are sufficient to
surpass the accuracy obtained by an equivalent 5-layered network with the
baseline activation functions. From Fig. 8b, we can see that this is obtained
28

1000000

0.84
Trainable parameters

Test accuracy

0.82
0.80
0.78
0.76

ELU
KAF
2D-KAF

0.74
2

3
4
Convolutive modules

ELU
KAF
2D-KAF

800000
600000
400000
200000

5

2

(a) Test accuracy

3
4
Convolutive modules

5

(b) Parameters

Figure 8: Results of KAF, 2D-KAF, and a baseline composed of ELU functions when
using only convolutive layers on the CIFAR-10 dataset (see the text for a full description
of the architectures). (a) Test accuracy. (b) Number of trainable parameters for the
architectures.

with a negligible increase in the number of trainable parameters for KAF, and
with a significant decrease (roughly 50%) for 2D-KAF. The reason, as before,
is that each nonlinearity for the 2D-KAF is merging information coming from
two different convolutive filters, effectively halving the number of parameters
required for the subsequent layer.
8.5. Experiments on a reinforcement learning scenario
Before concluding, we evaluate the performance of the proposed activation functions when applied to a relatively more complex reinforcement
learning scenario. In particular, we consider some representative MuJoCo
environments from the OpenAI Gym platform,8 where the task is to learn
a policy to control highly nonlinear physical systems, including pendulums
and bipedal robots. As a baseline, we use the open-source OpenAI implementation of the proximal policy optimization algorithm (Schulman et al.,
2017), that learns a policy function by alternating between gathering new
episodes of interactions with the environment, and building the policy itself by optimizing a surrogate loss function. All hyper-parameters are taken
directly from the original paper, without attempting a specific fine-tuning
8

https://github.com/openai/gym/mujoco

29

80
60
40
20

Baseline
KAF

0
0

100

200
Iteration

300

400

(a) swimmer

100000
90000
80000
70000
60000
50000

Baseline
KAF

40000
0

100

200
Iteration

300

(b) humanoidstandup

400

Average cumulative reward

100

Average cumulative reward

Average cumulative reward

120
8000
6000
4000
2000

Baseline
KAF

0
0

100

200
Iteration

300

400

(c) pendulum inverted

Figure 9: Results for the reinforcement learning experiments, in terms of average cumulative rewards. We compare the baseline algorithm to an equivalent architecture with
KAF nonlinearities. Details on the models and hyperparameters are provided in the main
discussion.

for our algorithm. The policy function for the baseline is implemented as a
NN with two hidden layers, each of which has 64 hidden neurons with tanh
nonlinearities, providing in output the mean of a Gaussian distribution that
is used to select an action. For the comparison, we keep the overall setup
fixed, but we replace the nonlinearities with KAF neurons, using the same
initialization as the preceding sections.
We plot the average cumulative reward obtained for every iteration of
the algorithms on different environments in Fig. 9. We see that the policy
networks implemented with the KAF functions consistently learn faster than
the baseline with, in several cases, a consistent improvement with respect to
the final reward.
9. Conclusive remarks
In this paper, after extensively reviewing known methods to adapt the
activation functions in a neural network, we proposed a novel family of
non-parametric functions, framed in a kernel expansion of their input value.
We showed that these functions combine several advantages of previous approaches, without introducing an excessive number of additional parameters.
Furthermore, they are smooth over their entire domain, and their operations can be implemented easily with a high degree of vectorization. Our
experiments showed that networks trained with these activations can obtain a higher accuracy than competing approaches on a number of different
benchmark and scenarios, including feedforward and convolutional neural
networks.
30

From our initial model, we made a number of design choices in this paper,
which include the use of a fixed dictionary, of the Gaussian kernel, and of a
hand-picked bandwidth for the kernel. However, many alternative choices are
possible, such as the use of dictionary selection strategies, alternative kernels
(e.g., periodic kernels), and several others. In this respect, one intriguing
aspect of the proposed activation functions is that they provide a further
link between neural networks and kernel methods, opening the door to a
large number of variations of the described framework.
Acknowledgments
The authors would like to thank the anonymous reviewers for their suggestions on how to improve the paper.
References
Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P., 2014. Learning
activation functions to improve deep neural networks. arXiv preprint
arXiv:1412.6830.
Baldi, P., Sadowski, P., Whiteson, D., 2014. Searching for exotic particles in
high-energy physics with deep learning. Nature communications 5, 4308.
Bengio, Y., Laufer, E., Alain, G., Yosinski, J., 2014. Deep generative stochastic networks trainable by backprop. In: Proc. 31st International Conference
on Machine Learning (ICML). pp. 226–234.
Bengio, Y., Léonard, N., Courville, A., 2013. Estimating or propagating
gradients through stochastic neurons for conditional computation. arXiv
preprint arXiv:1308.3432.
Bishop, C. M., 2006. Pattern recognition and machine learning. Springer
International.
Chen, C.-T., Chang, W.-D., 1996. A feedforward neural network with function shape autotuning. Neural Networks 9 (4), 627–641.
Clevert, D.-A., Unterthiner, T., Hochreiter, S., 2016. Fast and accurate deep
network learning by exponential linear units (ELUs). In: Proc. 2016 International Conference on Learning Representations (ICLR). pp. 1–14.
31

Cybenko, G., 1989. Approximation by superpositions of a sigmoidal function.
Mathematics of Control, Signals, and Systems (MCSS) 2 (4), 303–314.
Denil, M., Shakibi, B., Dinh, L., de Freitas, N., et al., 2013. Predicting parameters in deep learning. In: Advances in Neural Information Processing
Systems. pp. 2148–2156.
Glorot, X., Bengio, Y., 2010. Understanding the difficulty of training deep
feedforward neural networks. In: Aistats. Vol. 9. pp. 249–256.
Glorot, X., Bordes, A., Bengio, Y., 2011. Deep sparse rectifier neural networks. In: Aistats. Vol. 15. p. 275.
Goh, S. L., Mandic, D. P., 2003. Recurrent neural networks with trainable
amplitude of activation functions. Neural Networks 16 (8), 1095–1100.
Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep learning. MIT Press.
Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.,
2013. Maxout networks. In: Proc. 30th International Conference on Machine Learning (ICML). pp. 1–9.
Guarnieri, S., Piazza, F., Uncini, A., 1999. Multilayer feedforward networks
with adaptive spline activation function. IEEE Transactions on Neural
Networks 10 (3), 672–683.
Gulcehre, C., Cho, K., Pascanu, R., Bengio, Y., 2013. Learned-norm pooling for deep feedforward and recurrent neural networks. arXiv preprint
arXiv:1311.1780.
Gulcehre, C., Moczulski, M., Denil, M., Bengio, Y., 2016. Noisy activation
functions. In: Proc. 33rd International Conference on Machine Learning
(ICML). pp. 3059–3068.
He, K., Zhang, X., Ren, S., Sun, J., 2015. Delving deep into rectifiers:
Surpassing human-level performance on imagenet classification. In: Proc.
IEEE International Conference on Computer Vision (ICCV). pp. 1026–
1034.
Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J., 2001. Gradient flow
in recurrent nets: the difficulty of learning long-term dependencies.
32

Hofmann, T., Schölkopf, B., Smola, A. J., 2008. Kernel methods in machine
learning. The Annals of Statistics, 1171–1220.
Hornik, K., Stinchcombe, M., White, H., 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2 (5), 359–366.
Jin, X., Xu, C., Feng, J., Wei, Y., Xiong, J., Yan, S., 2016. Deep learning
with S-shaped rectified linear activation units. In: Proc. Thirtieth AAAI
Conference on Artificial Intelligence. pp. 1–7.
Jones, M. C., Marron, J. S., Sheather, S. J., 1996. A brief survey of bandwidth selection for density estimation. Journal of the American Statistical
Association 91 (433), 401–407.
Kingma, D., Ba, J., 2015. Adam: A method for stochastic optimization. In:
Proc. 3rd International Conference for Learning Representations (ICLR).
pp. 1–15.
Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S., 2017. Selfnormalizing neural networks. arXiv preprint arXiv:1706.02515.
Li, H., Ouyang, W., Wang, X., 2016. Multi-bias non-linear activation in
deep neural networks. In: Proc. 33rd International Conference on Machine
Learning (ICML). pp. 221–229.
Lin, M., Chen, Q., Yan, S., 2013. Network in network. arXiv preprint
arXiv:1312.4400.
Liu, W., Principe, J. C., Haykin, S., 2011. Kernel adaptive filtering: a comprehensive introduction. Vol. 57. John Wiley & Sons.
Maas, A. L., Hannun, A. Y., Ng, A. Y., 2013. Rectifier nonlinearities improve
neural network acoustic models. In: Proc. 30th International Conference
on Machine Learning (ICML). Vol. 30. pp. 1–6.
Micchelli, C. A., Xu, Y., Zhang, H., 2006. Universal kernels. Journal of Machine Learning Research 7 (Dec), 2651–2667.
Park, J., Sandberg, I. W., 1991. Universal approximation using radial-basisfunction networks. Neural Computation 3 (2), 246–257.

33

Piazza, F., Uncini, A., Zenobi, M., 1992. Artificial neural networks with
adaptive polynomial activation function. In: Proc. 1992 IEEE International Joint Conference on Neural Networks. pp. 1–6.
Ramachandran, P., Zoph, B., Le, Q. V., 2017. Swish: a self-gated activation
function. arXiv preprint arXiv:1710.05941.
Rennie, S. J., Goel, V., Thomas, S., 2014. Deep order statistic networks. In:
Proc. 2014 IEEE Spoken Language Technology Workshop (SLT). IEEE,
pp. 124–128.
Scardapane, S., Comminiello, D., Hussain, A., Uncini, A., 2017. Group sparse
regularization for deep neural networks. Neurocomputing 241, 81–89.
Scardapane, S., Scarpiniti, M., Comminiello, D., Uncini, A., 2016. Learning activation functions from data using cubic spline interpolation. arXiv
preprint arXiv:1605.05509.
Scarpiniti, M., Comminiello, D., Parisi, R., Uncini, A., 2013. Nonlinear spline
adaptive filtering. Signal Processing 93 (4), 772–783.
Schulman, J., Heess, N., Weber, T., Abbeel, P., 2015. Gradient estimation
using stochastic computation graphs. In: Advances in Neural Information
Processing Systems. pp. 3528–3536.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Scott, D. W., 2015. Multivariate density estimation: theory, practice, and
visualization. John Wiley & Sons.
Shang, W., Sohn, K., Almeida, D., Lee, H., 2016. Understanding and improving convolutional neural networks via concatenated rectified linear units.
In: Proc. 33rd International Conference on Machine Learning (ICML). pp.
2217–2225.
Silverman, B. W., 1986. Density estimation for statistics and data analysis.
Vol. 26. CRC press.
Siniscalchi, S. M., Li, J., Lee, C.-H., 2013. Hermitian polynomial for speaker
adaptation of connectionist speech recognition systems. IEEE Transactions
on Audio, Speech, and Language Processing 21 (10), 2152–2161.
34

Siniscalchi, S. M., Salerno, V. M., 2017. Adaptation to new microphones
using artificial neural networks with trainable activation functions. IEEE
Transactions on Neural Networks and Learning Systems 28 (8), 1959–1965.
Snelson, E., Ghahramani, Z., 2006. Sparse gaussian processes using pseudoinputs. In: Advances in neural information processing systems. pp. 1257–
1264.
Solazzi, M., Uncini, A., 2000. Artificial neural networks with adaptive multidimensional spline activation functions. In: Proc. 2000 International Joint
Conference on Neural Networks. Vol. 3. IEEE, pp. 471–476.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., Salakhutdinov,
R., 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15 (1), 1929–1958.
Trentin, E., 2001. Networks with trainable amplitude of activation functions.
Neural Networks 14 (4), 471–493.
Trottier, L., Giguère, P., Chaib-draa, B., 2016. Parametric exponential linear unit for deep convolutional neural networks. arXiv preprint
arXiv:1605.09332.
Van Vaerenbergh, S., Lázaro-Gredilla, M., Santamarı́a, I., 2012. Kernel recursive least-squares tracker for time-varying regression. IEEE Transactions
on Neural Networks and Learning Systems 23 (8), 1313–1326.
Vecci, L., Piazza, F., Uncini, A., 1998. Learning and approximation capabilities of adaptive spline activation function neural networks. Neural Networks 11 (2), 259–270.
Xu, B., Wang, N., Chen, T., Li, M., 2015. Empirical evaluation of rectified
activations in convolutional network. arXiv preprint arXiv:1505.00853.
Zhang, X., Trmal, J., Povey, D., Khudanpur, S., 2014. Improving deep neural network acoustic models using generalized maxout networks. In: Proc.
2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 215–219.

35

