Bayesian Optimization for Parameter Tuning of the XOR Neural Network
† ∗ L. Stewart, ∗ M. Stalzer
† Imperial College London
∗ California Institute of Technology

arXiv:1709.07842v2 [stat.ML] 14 Nov 2017

Abstract
When applying Machine Learning techniques to problems, one must select model parameters to ensure that the system converges
but also does not become stuck at the objective function’s local minimum. Tuning these parameters becomes a non-trivial task
for large models and it is not always apparent if the user has found the optimal parameters. We aim to automate the process of
tuning a Neural Network, (where only a limited number of parameter search attempts are available) by implementing Bayesian
Optimization. In particular, by assigning Gaussian Process Priors to the parameter space, we utilize Bayesian Optimization to tune
an Artificial Neural Network used to learn the XOR function, with the result of achieving higher prediction accuracy.

1. Introduction

1.1. Translation to a Black-Box Problem

This paper addresses the use of Bayesian Optimization for parameter tuning in Artificial Neural Networks that learn by Gradient Descent. We will focus on what are arguably the two most
important network parameters, the learning rate η, and the node
activation function hyper-parameter θ. For clarification on the
definition of θ, suppose node k in a Neural Network is a hyperbolic tanh node with hyper-parameter θ. Then the output of this
node, Ok,θ will be:
e θ 1 wi xi − e− θ 1 wi xi
Pn
Pn
e θ 1 wi xi + e −θ 1 wi xi


where x = x1 x2 · · · xn ∈ Rn and wi are the synaptic
weighting elements ( i.e. θ is the scale factor for the node function.) Whilst we have opted with α and θ as our parameters, the
ideas discussed throughout this paper generalise to any number
of chosen parameters.
Pn

Pn

Ok,θ (x) =

The selection of such parameters can be critical to a Network’s learning process. For example, a learning rate η that is
too small can stop the system from converging, whilst taking
η too high could lead to the system becoming stuck in a local
minimum of the cost function. It can often be computationally
expensive or time costly to evaluate a large set of network parameters and in this case finding the correct tuning becomes an
art in itself. We consider an automated parameter tuning technique for scenarios in which an exhaustive parameter search is
too costly to evaluate.

Email addresses: ls3914@ic.ac.uk ( † ∗ L. Stewart ),
stalzer@caltech.edu (∗ M. Stalzer )

Suppose we have a fixed Neural Network topology with ξ epochs
of a chosen learning method (e.g 10,000 epochs of Gradient
Descent Back-Propagation). Then we denote this by Nξ . We
also denote the network parameter space (the set of free parameters available to tune the network) for Nξ by P. For example, if we wish to tune the learning rate α ∈ (0.001, 1) and
the activation function hyper-parameter θ ∈ (0.001, 1), then
P = (0.001, 1) × (0.001, 1) ⊂ R2 .
With this terminology one can view the objective/cost function that is associated with the Neural Network as a function of
the parameter space for Nξ :
ΓNξ : P −→ R

(1)

The objective to optimally tune a network Nξ over a parameter
space P can now be summarised as finding p∗ ∈ P such that:
p∗ = argmin ΓNξ

(2)

p∈P

1.2. Bayesian Optimization
Bayesian Optimization is a powerful algorithmic approach to
finding the extrema of Black-Box functions that are costly to
evaluate. Bayesian Optimization techniques are some of the
most efficient approaches with respect to the number of function evaluations required [1] . Suppose we have k observations
D1:k = { x1:k , ΓNξ (x1:k ) } where xi ∈ P. The Bayesian Optimization algorithm works by using a Utility function U : P −→ R
in order to decide the next best point to evaluate. The algorithm
written in terms of the parameter optimization problem (2) can
be summarised as follows:

Algorithm 1 Bayesian Optimization
1:
2:

XOR Neural Network

for i = 1 to n do
Find x∗ = argmin U(x | D1:k )
x∈P

Sample the objective function at the chosen point
ΓNξ (x∗ )
n
o
4:
Augment the data D1:k+1 = D1:k ∪ x∗ , ΓNξ (x∗ )
5:
Update the Prior Distribution
3:

Figure 1

The target goal is to optimize α, θ over the space P = (0.001, 1)×
(0.001, 1). The Neural Network’s attributed loss function is the
Mean Square Error, the network’s topology X to be as shown in
Figure 1 and also ξ = 1000. This can all be neatly summarised
as ΓNξ = MSEX 1000 .

The algorithm terminates when i reaches n or if the Utility Function chooses the same element of P consecutively. For
more information on Bayesian Optimization see [2].
1.3. Choice of Parameter Space, Prior and Utility Function
This paper aims to optimize learning rate α and node function
hyper-parameter θ as in Section (1.3) i.e. P = (0.001, 1) ×
(0.001, 1). We will take our prior distribution (as described in
Algorithm 1) to be a Gaussian Process with a standard square
exponential kernal [3]:

For Deep Neural Networks or any Neural Network where
the number of epochs
it can become very time costly
h ξ is large,
i
to evaluate MSEX ξ (αi , θ j ) . Furthermore, to achieve a small
step size when discretizing P an exhaustive search of the Network parameters becomes time costly. In fact, for higher dimensional variants of P it becomes simply impossible to exhaustively evaluate the parameter space. To keep our proposed
solution generalised we will limit the number of parameter searches
allowed to 20.


 1
k( xi , x j ) = exp − ||xi − x j ||2
2
For speed the Julia code does not run the niave O(n3 ) ShermanMorrison-Woodbury matrix multiplication [3]. Instead it uses
a Cholesky method detailed in [3]; both methods are explained
further within the code file. For very large parameter spaces one
may want to look into using a Gaussian Process update method

of O nlog2 (n) [4].

2.1. Code Structure
The Julia code is available with this pre-print on arXiv. The
code is distributed under a Creative Commons Attribution 4.0
International Public License. If you use this code please attribute to L. Stewart and M.A. Stalzer Bayesian Optimization
for Parameter Tuning of the XOR Neural Network, 2017.

The chosen utility function is the Lower Confidence Bound
[2]:
LCB (xi ) = µ(xi ) + γσ(xi ) γ ∈ (0, 1)

The code compares Bayesian Optimization to a Random
Grid Search of P, a technique commonly used to achieve an
acceptable tuning of Neural Networks. During the Random
Grid Search process the code selects 20 random points of P
and evaluates MSEX 1000 for each point. During Bayesian Optimization, the code computes MSEX 1000 for one random element

of P say p1 and forms D1 = (p1 , MSEX 1000 (p1 ) . It then runs
the Bayesian Optimization algorithm described in Section 1.2
where γ ≡ 1.

It is worth noting that whilst this utility function is indeed sufficient for tuning the XOR Neural Network, it may be necessary
to use an IMGPO utility function (exponential convergence) for
larger scale problems [5].
2. XOR Neural Network Optimization
Consider the XOR function [6] (Exclusive Or) defined as:
XOR : Z⊕2
2 → Z2
(x, y) 7→ (x + y) mod(2)

3. Results
The results for the experiment denoted above in Section 2 are
as follows:

XOR has traditionally been a function of interest as it cannot be learned by a simple 2 layer Neural Network [7]. Instead the problem requires a 3 layer Neural Network, (hence
there will be 3 nodes affected by activation function hyperparameter tuning.) Figure 1 shows the topology of the Neural
Network required to learn the XOR function. The Neural Network consists of an input layer followed by two hidden layers
of sigmoid functions, where the output of a sigmoid node is
1P
Oθ (x) =
−θ n wi xi .
1

MSEX 1000
# Search Points
Run Time (s)

Random
Search
0.2681
20
1.5411

1+e

Table 1

2

Bayesian Opt
0.1767
6
0.6012

MSE for 20 Randomly Selected Parameter Values

MSE BO - (6 point convergance)
0.50

0.50

0.45

Mean Square Error

Mean Square Error

0.45

0.40
0.35

0.40

0.30

0.35
0.30

0.25
0.20

0.0
Sigm0.2
oid Hy0.4 0.6
per-Pa
ramete0.8
r

1.0 1.0

0.2
0.4 te
a
0.6
0.8 Learning R

0.0
Sigmo0.2 0.4
id Hyp 0.6
er-Par 0.8
amete 1.0
r

0.0

Figure 2

0.8

0.2 0.0
0.6 0.4
g Rate
Learnin

Figure 3

Where # Search Points is the number of points that each
of the processes evaluated (this will always be 20 for Random
Search as it has no convergence criterion). Figure 2 shows the
Mean Square Error plot for the 20 randomly selected elements
of P, whilst Figure 3 shows the Mean Square Error plot for the
points selected by Bayesian Optimization. The Merlot colored
plane in Figure 3 denotes the minimum MSE reached by Random Grid Search for comparative purposes.

Threshold
MSE

#Search
Points

0.190
0.185
0.177

2034
5685
N/A

Time to
Reach
Threshold (s)
121.8
454.5
N/A

Table 2

Figure 4 shows the development of Mean Square Error with
time for Bayesian Optimization and also includes Random Grid
Search for ease of error comparison.

vastly increases. When setting the threshold value to the Bayesian
Optimal (0.177), Random Grid Search did not even converge.

Development of MSE with Time

4. Concluding Remarks and Future Works

0.50
0.45

We have shown in Section 3 that Bayesian Optimization can be
successfully implemented to attain a highly accurate tuning of
the XOR Neural Network in a very fast time. There are many
avenues for future development, a few of which are briefly summarised below:

0.40
MSE

1.0

0.35
0.30
0.25
0.20

• Apply Bayesian Optimization to more complex Neural
Network topologies i.e. Recurrent or Deep Neural Nets.
To do this a more sophisticated utility function may be
required. [5].

Bayesian Optimization
Random Grid Search
0.0

0.2

0.4

0.6

0.8
Time (s)

1.0

1.2

1.4

1.6

Figure 4

• Apply to Bayesian Optimization to tuning higher dimensional variants of P. Further parameters could be added
e.g. momentum [8] and regularization rate [9]. Another
possibility is to have a different learning rate/node function hyper-parameter for each synapse/node. For both
these tasks the use of Random Embeddings may be useful
[10].

It is clear Bayesian Optimization achieved higher accuracy
whilst also running faster than Random Grid Search. This is
due to the fact that the algorithm converged to the optimal value
with respect to the LCB Metric.
This raises the question of how much more accurate is the
result obtained by the LCB Metric (i.e. is the LCB a suitable
Metric for tuning the XOR Neural Network)? To answer this
it is useful to see the time required for Random Grid Search
to achieve set a selection of threshold Mean Square Errors, including the Bayesian Optimal 0.1767. The results for this are
displayed in Table 2.

• Test a wide variety of Gaussian Process Kernals and use
a faster Kernal update method [4]. It would also be interesting to investigate the effect of using different prior
distributions in Bayesian Optimization [11].

One can see that for Random Grid Search to reach a moderately lower error of just 0.190, the consequential time cost
3

Acknowledgements
This research is funded by the Caltech SURF program and the Gordon and Betty Moore Foundation through Grant GBMF4915
to the Caltech Center for Data-Driven Discovery.
References
[1] Jonas Mockus. Application of bayesian approach to numerical methods of global and stochastic optimization. Journal of Global Optimization, 4(4):347–365,
1994.
[2] Eric Brochu, Vlad M. Cora, and Nando de Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling
and hierarchical reinforcement learning. CoRR, abs/1012.2599, 2010.
[3] Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning, volume 1. MIT press Cambridge, 2006.
[4] Sivaram Ambikasaran, Daniel Foreman-Mackey, Leslie Greengard, David W Hogg, and Michael O’Neil. Fast direct methods for gaussian processes. arXiv
preprint arXiv:1403.6015, 2014.
[5] Kenji Kawaguchi, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Bayesian optimization with exponential convergence. In Advances in Neural Information
Processing Systems, pages 2809–2817, 2015.
[6] Gerhard J. Woeginger Alexander S. Kulikov. Computer science - theory and applications. 11th International Computer Science Symposium in Russia, CSR,
2016.
[7] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990.
[8] Robert A Jacobs. Increased rates of convergence through learning rate adaptation. Neural networks, 1(4):295–307, 1988.
[9] Federico Girosi, Michael Jones, and Tomaso Poggio. Regularization theory and neural networks architectures. Neural computation, 7(2):219–269, 1995.
[10] Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. Bayesian optimization in a billion dimensions via random embeddings.
Journal of Artificial Intelligence Research, 55:361–387, 2016.
[11] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable
bayesian optimization using deep neural networks. In International Conference on Machine Learning, pages 2171–2180, 2015.

4

