1

Transforming Cooling Optimization for Green Data
Center via Deep Reinforcement Learning

arXiv:1709.05077v1 [cs.AI] 15 Sep 2017

Yuanlong Li, Yonggang Wen, Kyle Guan, and Dacheng Tao

Abstract—Cooling system plays a key role in modern data
center. Developing an optimal control policy for data center
cooling system is a challenging task. The prevailing approaches
often rely on approximated system models that are built upon
the knowledge of mechanical cooling, electrical and thermal
management, which is difficult to design and may lead to suboptimal or unstable performances. In this paper we propose
to utilize the large amount of monitoring data in data center
to optimize the control policy. To do so, we cast the cooling
control policy design into an energy cost minimization problem
with temperature constraints, and tab it into the emerging
deep reinforcement learning (DRL) framework. Specifically, we
propose an end-to-end neural control algorithm that is based
on the actor-critic framework and the deep deterministic policy
gradient (DDPG) technique. To improve the robustness of the
control algorithm, we test various DRL related optimization
techniques, such as recurrent decision making, discounted return,
different neural network architectures, and different stochastic
gradient descent algorithms, and adding additional constraints
on the output of the policy network. We evaluate the proposed
algorithms on the EnergyPlus simulation platform and on a real
data trace collected from the National Super Computing Centre
(NSCC) of Singapore. Our results show that the proposed endto-end cooling control algorithm can achieve about 10% cooling
cost saving on the simulation platform compared with a canonical
two stage optimization algorithm; and it can achieve about 13.6%
cooling energy saving on the NSCC data trace. Furthermore,
it shows high accuracy in predicting the temperature of the
racks (with mean absolute error 0.1 degree) and can control
the temperature of the data center zone close to the predefined
threshold with variation lower to 0.2 degree.

I. I NTRODUCTION
With the emergence and proliferation of services and applications such as cloud computing and social networks, data
centers play an ever important role. It is predicted that global
data center IP traffic will grow 3-fold from 2014 to 2019 with
a compound annual growth rate (CAGR) of 25 percent [1]. At
the same time, the high energy consumption of data centers is
drawing more and more attention due to economic, social, and
environmental concerns. Data center electricity consumption
in the U.S. alone is projected to increase to roughly 140
billion kilowatt-hours annually by 2020, costing $13 billion in
electricity bills and emitting nearly 100 million tons of carbon
pollution per year [2]. In this paper, we focus on one of the
Yuanlong Li and Yonggang Wen are with School of Computer Science and
Engineering, Nanyang Technological University, Nanyang Avenue, Singapore
639798. Email: {liyuanl, ygwen}@ntu.edu.sg.
K. Guan is with Bell Labs, Nokia, Holmdel, NJ 07733 USA.
Dacheng Tao is with the Centre for Quantum Computation and Intelligent
Systems and the Faculty of Engineering and Information Technology, University of Technology, Sydney, 81 Broadway Street, Ultimo, NSW, Australia.

major source of energy consumption in the data center (about
37% [3]), the cooling energy.
Cooling energy optimization involves the control of a complex cooling system, which consists of multiple components,
such as cooling tower, chiller, and ventilation system, etc. A
common practice of data center cooling system control is to
adjust the set-points, i.e., the target values of different control
variables. For example, by setting the temperature control
variable at the outlet of an air conditioner to a desired value,
the air conditioner can adjust its internal state to meet the
set-point by consuming certain amount of energy. An optimal
selection of these set-points can be challenging, as the process
relies on the knowledge of the cooling system, from thermal
dynamics to mechanics. Many existing approaches are based
on an approximated system model that often incorporates
the first-order effects of thermal, electrical, and mechanical
principles [4]–[8]. These approximated models are sometimes
either inadequate or inaccurate to capture the intricacies of
various interacting processes of data center cooling operations,
thus could lead to sub-optimal or unstable cooling controls.
Recently, the “model-free” approach has emerged as an attractive alternative. A model-free approach does not assume
any specific model of the underlying system. Instead, the
control policies are learned and derived from the massive data
collected on the system status and energy consumption [9].
This approach is especially advantageous when the complexity
of the underlying system makes an accurate system modeling
a daunting task.
Currently, the prevailing control optimization approach,
which includes a model building stage and a solving stage, can
be referred to as the two-stage (TS) approach. In comparison,
the model-free approach (also referred to as the end-to-end
or the one-stage approach) uses directly the unprocessed
and often high-dimensional input, thus omitting the model
building or feature extraction stage. One particular control
optimization frameworks to our interest is reinforcement learning (RL) [10], which enables an artificial intelligence (AI)
agent to learn better decision making by directly interacting
with environment. Algorithms that combine RL with deep
learning (DL), such as deep Q-network (DQN), have been
successfully applied to the task of training AI agent to play
video games at the human performance level with only raw
pixel inputs, thus showing the potential of the end-to-end
approach [11]. The continuous domain extension of DQN,
called deep deterministic policy gradient (DDPG), has also
shown promising results on simulated physical control tasks
[12]. However, DDPG has not been widely studied in the
context of more practical and complex control optimization,

2

such as cooling system optimization for data centers (the
subject of this paper). It is yet to be demonstrated if an end-toend approach can achieve similar or better control performance
compared with the TS approaches. In addition, DDPG is
a simulation based algorithm. Like many RL algorithms, it
requires an excessive amount of computation and possibly very
long training time. Thus it is also of interest to see how the
challenges are met. Note that Google claims that they use AI
method [13] to reduce the PUE of their data center; yet no
detailed methodology or performance evaluation results are
disclosed. In our previous work, we reviewed the data center
energy cost models [14] and the existed cooling optimization
approaches [15], and conducted several data center power
analysis and control studies [16] [17] [18]; based on which we
believe that a data driven learning based optimization method
is needed for data center energy optimization, which can be
used to achieve optimization effects with minimum human
innervations and reduce the data center management difficulty.
In this paper, we investigate the viability of applying the
end-to-end neural control algorithm for data center cooling
control optimization, and evaluate the algorithm from various
aspects. We develop a neural control optimization algorithm
adapted from the DDPG and the actor-critic architecture [12]
[19]. Our proposed algorithm is off-policy, as it uses a single
pre-collected trace to learn and improve the control policy.
Beside the standard version of the algorithm which makes
control decisions based on the current states, we develop also a
recurrent version of the algorithm which is based on the recent
states and control settings. In addition, we propose to impose a
consecutive constraint to the output of the policy network. We
also evaluate the effects of discounted return, different neural
network architectures, and different weight parameters update
algorithms in training the neural networks.
To test the proposed algorithm, we use the EnergyPlus
[20] platform to build a test case; beside the simulation
case, we also collected a real data trace from the National
Super Computing Centre (NSCC) of Singapore and tested our
algorithm on it. For the simulation test case, we control five
different set-points to achieve minimum PUE and to maintain
the temperature of the data center zone within a pre-defined
range. The results of the proposed algorithms are compared
with those generated by a standard two-stage optimal control
algorithm and the default data center zone temperature setpoint control algorithm (embedded in the simulation software).
The results indicate that the proposed algorithm not only
can successfully maintain the temperature of the data center
zone within a pre-defined range under varying workloads and
weather conditions, but also achieve lower PUE compared with
a general TS algorithm. For the real data from NSCC test case,
we focus on optimizing the airflow rate setting of the three precision cooling units (PCUs) which are used to cool 26 racks.
Our results show that the proposed algorithm can approximate
the real data with high accuracy and can output energy saving
control settings according to the cooling requirements. The
main contributions of the paper are summarized as follows.
First, we propose an end-to-end and DRL-based framework
that can be used for data center cooling control optimization.
We also propose algorithms that train the neural network with

a pre-collected data trace. This approach is well suited for
a practical data center equipped with a monitoring/sensing
system that collects data in real time.
Second, we build a test-bed with the EnergyPlus and also
collect a real data trace from NSCC Singapore to evaluate the
proposed control algorithms. Our simulation results indicate
that the proposed control algorithm can accomplish the cooling
control tasks with about 10% cooling cost saving compared
with a canonical two stage optimization algorithm. Our results
on the real data trace from NSCC show that we can achieve
about 13.6% energy saving to their major air cooling units.
Third, we gain new insights into some DRL-related optimization techniques that are unique to data center cooling
control. For instance, we show that the recurrent decision
making, which is generally ignored in existing TS approaches,
can significantly improve the performance in cooling control
optimization. We also show that adding consecutive constraints
to the output of the policy network can further enhance the
performance of the algorithm. In regard to the widely used
discounted return technique in RL research, we show that
its impact is very limited as it cannot lead to significant improvement in performance stability. Our results also show that
for the cooling control optimization, a modest neural network
architecture coupled with proper weight-update algorithm can
provide competitive and robust performance. Also our results
on the real data from NSCC show that the proposed algorithm
can approximate the real energy data and temperature readings
with high accuracy (with normalized mean absolute error less
than 10%) and can output control settings that can reduce the
energy cost while satisfying the cooling requirements. Also we
show that the trade-off between the energy cost and cooling
effects can be adjusted by varying the temperature constraint
penalty weight parameter.
In summary, we demonstrate the feasibility and effectiveness of applying end-to-end neural control algorithms to
the data center cooling optimization. The evaluation of the
performance of the proposed algorithms serves as the first
step to build intelligent data center management system that
requires minimal manually intervention. Though this work is
simulation based, it does shed new light on the application
of deep reinforcement learning (DRL) to practical data center
control optimization.
II. P ROBLEM F ORMULATION
To study the cooling control optimization problem, in this
section we utilize a simulation model to present a cooling control optimization problem formulation. The simulation model
is based on the widely adopted building energy simulation
platform EnergyPlus [20]. Although the model is largely
simplified, it does capture the major cooling dynamics and
is thus adequate for studying the cooling control optimization.
A. Simulation System Model
The model is based on a simulation example provided by
EnergyPlus. As illustrated in Fig. 1, the model consists of
two data center zones (z1 and z2 ) and the associated cooling
systems. The two data center zones are only different in size

3

•}

'DWD &HQWH

U

•}

v

v

î

í

Dy Z

^W>/dd Z

Dy Z

'LUHFW ([SDQVLRQ ';
&RROLQJ
&KLOOHU &RROLQJ

^W>/dd Z

Fig. 1. System architecture: a data center consists of two server zones, z1
and z2 , and their associated and independently operated cooling facilities. A
zone is modeled as a single point, i.e., we only model the total load of each
zone and neglect the detailed thermal distribution inside the zone. The cooling
facility consists of direct expansion (DX) system and chiller cooling system.

Fig. 2. The main components of a cooling system. The temperatures at five
outlet ports (marked as open circles) are served as the temperature set-points:
TDEC , TIEC , TCW ater , TCCoil , and Tsup .

and location. In addition, their corresponding cooling systems
are independently operated. In the following, we describe the
cooling system with a focus on identifying the state space
(parameters that characterize the system), the action space
(control variables), and the reward (optimization objectives),
while omitting the details of the facility structures and operation processes that are unrelated to problem setup (yet might
be critical parts of the overall system).
1) Data Center Model – State Space and Reward: The data
center has two server zones placed side by side, z1 and z2 ,
with each zone being a standalone server room. The two zones
are different in size ( 15.24*15.24m2 and 15.24*17.00m2 for
z1 and z2 , respectively) but similar in other aspects. The heat
in each zone is generated by IT equipment (ITE) and other
sources (such as illuminations), with ITE as the dominant heat
source. The load of the ITE is defined as α · L, where L is
the designed load per square meter and α is a load factor that
varies at different time slots. In our simulation, we use a public
trace collected from Wikimedia [21] to set α to be the same for
z1 and z2 . The heat generated by the illumination is assumed
to be a constant (per square meter) as the lights inside the data
center are on all the time. Also there is heat generated by the
human workers in the data center. This part varies according
to the work time schedule. For this work, each zone is simply
modeled as a single point heat source to be cooled. This is
less accurate compared to the computational fluid dynamics
(CFD) based thermal analysis. We leave a finer-grained model
to future work. We note that even with a finer-grained model,
the proposed framework and algorithms will remain the same,

albeit with much larger state space and action space.
In the context of RL framework, we use a tuple of workload
level and the ambient temperature to represent the state, since
both of them affect the cooling load. We use the tuple of PUE
and the temperature of the data center zone to represent the
reward. In the context of the data center cooling, PUE needs
to be minimized and the data center temperature needs to be
kept within certain range.
2) Cooling System Model: Action Space: z1 and z2 are
equipped with the same cooling facility which consists of two
different types cooling systems: direct expansion (DX) cooling
system and chilled water (chiller) cooling system. Both cooling
systems are supplied with cool water from a cooling tower, but
they use the cool water in different ways. In the DX system,
the cool water passes through coils and cool down the air flow
passing over the coils. In the chiller system, the cool water is
used first to refrigerate another water stream (chilled water),
which in turn cools down the air flow.
In our model, both systems are included as shown in Fig.
2. The intake ambient air flow is first cooled by two types of
evaporative coolers: directive (DEC) and indirect (IEC), and
then passes over the DX cooling coils and is further fed to the
data center. The cold airflow to cool the data zones can also
come from the chiller.
For the action space, the underlying control algorithm in EnergyPlus (referred to as the Kbest control algorithm) uses the
following five set-points to control the zone temperature: DEC
outlet temperature (airflow) TDEC , IEC outlet temperature
(airflow) TIEC , chilled water loop outlet temperature (water
flow) TCW ater , DX cooling coil outlet temperature (airflow)
TCCoil , and air loop outlet temperature (airflow) Tsup .
In the KBest algorithm, the adjustment of these five setpoints is based on the knowledge of the underlying system
dynamics. In our proposed learning algorithm, the same five
set-points are used as control variables, but, on the contrary,
they are learned from the pre-collected data trace. Neither
physical content of these variables nor the relationship information among them will be used in the training.
B. Problem Statement
We formulate the cooling control optimization problem as
follows. We are given a time-varying tuple of the ambient
air temperature Tamb and the load factor α. The problem is
to determine the values of five control set-points, so as to
minimize the objective function as stated in Eq. (1).
The objective function aims to strike a balance between
minimizing the PUE and preventing overheating in the server
zone. In particular, the objective function consists of two
parts: the first part is PUE (denoted as P U E ), which is to
be minimized; the second part accounts for the penalty of the
overheating (for both z1 and z2 ). The penalty function has
the form of λ ln·(1 + exp(Tzi − φ)), for i ∈ [1, 2], with λ,
Tzi , and φ denoting the scaling factor of the penalty, average
temperature of zone i, and overheating threshold, respectively.
As shown in Fig. 3, here we choose the softplus [22] function
ln(1 + exp(.)), in lieu of a much simpler but non-smooth
single-sided penalty max(0, Tz1 − φ) (denoted as SimCost,

4

P U E + λ · ln(1 + exp(Tz1 − φ)) + λ · ln(1 + exp(Tz2 − φ)),

min

TCW ater ,TCCoil ,TDEC ,TIEC ,Tsup

s.t.

(1)

LTCW ater ≤ TCW ater ≤ UTCW ater ,
LTCCoil ≤ TCCoil ≤ UTCCoil ,
LTDEC ≤ TDEC ≤ UTDEC ,
LTIEC ≤ TIEC ≤ UTIEC ,
LTsup ≤ Tsup ≤ UTsup .

5. 0

max(0, Tz1 − φ)
4. 0

ln(1 + exp(Tz1 − φ)

3. 0
2. 0
1. 0
0. 0
−6. 0

−4. 0

−2. 0

0. 0
Tz1 − φ

2. 0

4. 0

6. 0

Fig. 3. The temperature penalty functions: a straightforward penalty function
could be max(0, Tz1 − φ), which is non-smooth and is difficult to optimize;
the softplus function (ln(1 + exp(Tz1 − φ)), which is smooth and is better
for the cost function optimization.

which will be tested in Section IV), to avoid the difficulty
in the neural network training phase. We reverse the sign of
this very cost function to calculate the reward for a stateaction pair, following the convention of RL framework in
which reward functions are meant to be maximized. During
the training, we simply minimize this cost function, since
commonly the training optimization algorithms are designed
for minimization. In this formulation, we also apply the
same control setting for both z1 and z2 , which is reasonable
considering that the same workload trace is used for both z1
and z2 .
III. T HE P ROPOSED A PPROACH : N EURAL E ND - TO -E ND
C OOLING C ONTROL A LGORITHMS
In this section, we present the end-to-end cooling control
algorithms (CCA), adapted from the DDPG, which combines
the key RL techniques and methods such as deep Q-network
(DQN), deterministic policy gradient (DPG), and actor-critic
algorithm. In the following, we first provide an overview of
the related RL concepts and techniques. We then describe a
complete algorithm flow and the design of the neural networks.
A. Overview of Q-learning and Policy Gradient
Our goal is to enable an AI agent to learn an optimal
control policy from a data set that records a sequence of states,
actions taken, and rewards at discrete time steps. Within the
RL framework, this goal is achieved by using either valuebased or policy-based approaches. Central to the value-based

approaches is the Q-learning technique, which uses a Qfunction Q(s, a) to represent the maximum discounted future
reward for an action a taken at a state s. As Q(s, a) indicates
the utility (value) of choosing an action a at state s, an optimal
policy π simply chooses the action that has the highest Q value
for a state s, i.e., π = argmaxa Q(s, a). Though for discrete
state and action space (especially when the space is small), Qfunction can be represented as a table computed by Bellman
equation iterative updating, in practice it is often estimated
by a function approximator such as a neural network, like the
Deep Q-network (DQN) [11]. With policy-based approaches,
policy-gradient (PG) is an important algorithm that optimizes a
policy end-to-end by computing noisy estimates of the gradient
of the expected reward and then updating the policy in the
gradient direction. In practice, policy iterations and updates
are often carried out by a neural network, referred to as
policy network. Traditionally, PG methods have assumed a
stochastic policy, which gives a probability distribution over
actions. Recent research has proved that the stochastic policy
gradient, which is the gradient of the policy’s performance, is
equivalent to the deterministic policy gradient, thus allowing
all the machinery for stochastic policy gradients to be applied
to deterministic policy gradients [23].

When state or action space is represented by continuous
variables, a naive adaptation of DQN or PG via discretization
of state or action space often results in intractability or very
slow learning convergence (even divergence). We use DDPG
algorithm, which is essentially a hybrid method combining the
policy gradient method and the value function [12]. Specifically, DDPG is an actor-critic algorithm: it uses two neural
networks, referred to as actor network and critic network,
to approximate the policy function and the value function,
respectively. These networks compute action predictions for
the current state and generate a temporal-difference (TD) error
signal at each time step. The input of the actor network is
the current state, and the outputs are real values representing
actions chosen from a continuous action space. The critic’s
output is simply the estimated Q-value of the current state
and of the action given by the actor. The deterministic policy
gradient theorem provides the update rule for the weights of
the actor network. The critic network is updated from the
gradients obtained from the TD error signal.

5

B. Online learning vs. batch learning and Off-Policy vs. OnPolicy
We note that RL algorithms can be directly used as online
learning algorithms. This means that the control algorithm
can learn in an online manner, e.g., starting from an initial
state and adjusting itself with the input it received from the
ongoing process, either the real operation or the simulation.
However, this will be problematic for the data center cooling
task, which cannot risk erroneous settings. In this work, we
focus on the control algorithms that are pre-trained by the
offline data first, which is referred to as “batch learning”. Batch
algorithms can be further divided into two categories based on
how the training data is generated: off-policy and on-policy.
Off-policy algorithms generally employ a separate behavior
policy, which is independent of the policy being estimated,
to generate the training trace; while on-policy directly uses
control policy being estimated (in the real control practice or
more likely in a simulator) to generate training data traces.
For the case of data center simulation, the cost for simulation
time is high. Thus off-policy algorithms are easier to apply
and more suitable for our case.
In summary, we propose an off-policy control algorithm
adapted from canonical DDPG. The algorithm employs only
a single offline trace for batch learning. In the following we
introduce the details of the proposed algorithm.
C. Cooling Control Algorithm with Single Offline Trace (CCASOT)
The flowchart of the proposed Cooling Control Algorithm
with Single Offline Trace (CCASOT) is shown in Algorithm
1. For the training task, a data trace is collected (line 1),
which contains entries: state (Tamb (t), α(t)), action (TDEC (t),
TIEC (t),TCW ater (t), TCCoil (t) and Tsup (t)), and reward data
y(t) is computed by the objective function (1) based on the
observed PUE and temperature data (note here we do not
incorporate future reward, which will be discussed in Section
III-E2). All these data are prepared as time series of N
time steps, which are further divided into training data and
validation data.
Before the training starts, we first initialize two neural networks (line 2). The critic network Q(XQ |θQ ) (parameterized
by θQ ) approximates the Q-value of a state-action pairs: It
takes both state and action (combined into a single tuple XQ )
as the input, and outputs a scalar value which represents the
cost of an action a taken at a state s. The actor network
µ(Xµ |θµ ) (parameterized by θµ ) is policy network: it takes
the current state (Xµ ) and outputs a five dimensional vector
corresponding to the five control factors shown in the cooling
system model.
The training procedure is shown in lines 3-21. Here we
use standard neural network training procedure with multitraining-epochs. Within each epoch (line 5-9), each batch of
training data is used to update the weights of the neural
networks using gradient descent. The critic network is updated
by minimizing the mean square error between the output of
Q and the objective y; while the policy network is updated by
minimizing the output of Q when taking action at current state

according to the output of µ. To avoid over-fitting, we also
compute the validation error to keep track of the best weight
parameter settings for the two neural networks respectively,
as shown in lines 11-19. On important note is that for the µ
network, the validation error can be very small at the beginning
due to that at that time the Q network is not well learned. For
µ
safety, one can use a periodical re-initialization of the Eval
to
solve this problem.
Algorithm 1 CCASOT: Cooling Control Algorithm with
Single Offline Trace
1: Data preparation: Collect a single offline trace with
historical data series of the following variables: Ambient
environment temperature Tamb (t), workload level r(t),
and settings of the control factors TCW ater (t), TCCoil (t),
TDEC (t), TIEC (t), and Tsup (t), where t = 1, ..., N ,
N is the total number of time steps. Denote XQ =
{Tamb (t), r(t), TCW ater (t), TCCoil (t), TDEC (t), TIEC (t)}
and Xµ = {Tamb (t), r(t)}, where XQ is the input of
the Q-neural network Q and Xµ is the input of the
policy neural network µ. Collect also the PUE and outlet
temperature data series and compute the cooling reward
y(t) at each time step t according to objective function
Eq. (1). Divide the data into training data (used for
training the neural network) and validation data (used for
validate the neural network to find the best parameter
settings). Denote IV as the index of sample data in the
validation set.
2: Initialization: Create neural networks Q(XQ |θ Q ) and
µ(Xµ |θµ ), and randomly set the values of the weight
parameters θQ and θµ for the neural networks. Create
0
0
target network Q0 and µ0 with θQ = θQ and θµ = θµ .
Q
µ
Set initial validate error Eval
= 1e100, Eval
= 1e100 and
Qbest = N ull, µbest = N ull.
3: for Epoch = 1, ..., M axEpoch do
4:
Randomly divide the training data into batches of size
M and denote the index set of the data in the ith batch
as Ii .
5:
for i = 1, ..., N/M do
6:
Update
θQ by minimizing:
P
Q 2
7:
j∈Ii (y(j) − Q(XQ (j)|θ ) /M .
µ
8:
Update
θ by minimizing:
P
µ
Q
9:
j∈Ii Q([Xµ (j), µ(Xµ (j)|θ )]|θ )/M .
10:
end for P
0Q
11:
Eval
← j∈IV (y(j) − Q(XQ (j)|θQ )2 .
0Q
Q
12:
if Eval < Eval
then
Q
0Q
13:
Eval ← Eval
14:
Qbest = θQ
15:
end if P
0µ
16:
Eval
← j∈IV Q([Xµ (j), µ(Xµ (j)|θµ )]|θQ ).
0µ
µ
17:
if Eval < Eval
then
µ
0µ
18:
Eval ← Eval
19:
µbest = θµ
20:
end if
21: end for
22: Return: The optimal Q and µ neural network with weight
parameter settings Qbest and µbest respectively.

6

D. Neural Network Design
We setup the Q-network as follows. Denote the number of
states as ns and the number of control factors as nc . We
use a three-layer neural network (input, hidden, and output
layer) to implement Q. Specifically, with weight matrices
(1)
(2)
(1)
(2)
W(ns +nc )×nhid1 , Wnhid1 ×1 , bias vectors b1×nhid1 , b1×1 , and
input XQ , the output of Q-network is given by:
Q(XQ ) = σ((σ(XQ W (1) + b(1) ))W (2) + b(2) ),
where σ(·) is the sigmoid function; nhid1 is the number of
hidden nodes in the hidden layer of the Q-network. Note that
as the output range of the sigmoid functions is [0,1], in the
training the ground-truth of the output of Q must be scaled
into range [0, 1].
The set-up of the µ-network is as follows. Similar to Q, we
use a three-layer neural network to implement µ. With weight
(3)
(4)
(3)
(4)
matrices Wns ×nhid2 , Wnhid2 ×nc , bias vectors b1×nhid2 , b1×nc ,
and input Xµ , the output of µ is given by:
µ(Xµ ) = σ((σ(Xµ W (3) + b(3) ))W (4) + b(4) ),
where nhid2 is the number of hidden nodes in the hidden layer
of µ. Note that for each state, the output of µ-network is a
nc -dimensional vector, in which each element is distributed in
range [0,1].
During the training procedure, all input values are scaled
into range [0, 1] in the following manner:
xmin
xrange
x

= min(x);
= max(x) − min(x);
← (x − xmin )/xrange ,

where x stands for any state or control factor. The output of
the µ-network will be transformed in the following manner to
recover the real control setting xreal (used in simulation or
real practice):
xreal = xµ · xrange + xmin .
Consequently, the output of µ and the recovered true control
setting values are in valid ranges (the specific range for each
control variable after recovering), ensuring that the output
of the policy network µ automatically meets the control
constraints due to the neural network design.
E. Further Improvement of the Algorithm
In this section, we discuss two important issues in designing
the training data (input, output) pair for the Q-network and an
additional cost for the policy network. For the Q-network, in
the above listed algorithm, the training data can be presented as
a pair of traces (x(t), y(t)): x(t) represents the input to the Qnetwork, which consists of the state and actions taken at time
slot t; y(t) represents the output of the Q-network, which is
equivalent to the cost. In this case, the control setting at time
slot t depends only on the state at time slot t. In addition, the
cost to be minimized is also only the cost incurred at time
slot t. If we expand the time span for both decision making
and cost minimization, we have two different design variants –

recurrent decision making and discounted return, with respect
to the input and output of the Q-network respectively. For the
policy network, as the control settings should not vary abruptly
between two consecutive time steps, we propose an additional
cost item to improve the robustness of the policy network.
Details are shown below.
1) Recurrent Decision Making: With respect to the input
x(t) = {s(t), a(t)} of the Q-network, we can consider to
incorporate previous states and control actions into the inputs;
These additional information can affect the control decision
setting. Such approach is usually called recurrent approach
[24], which has been extensively studied in RL. We introduce
the recurrent algorithm with the CCASOT as an example. In
the standard CCASOT, the decisions are made solely based
on the current state s(t). In the recurrent version, the decision
are made based on the recent states and actions s(t − τ ), a(t −
τ ), s(t − τ + 1), a(t − τ + 1), ..., s(t − 1), a(t − 1), s(t), a(t),
where τ is the maximum number of time steps to trace back.
Making decisions with the recent states and actions, which can
be considered as a short time series, actually concedes certain
degree of the prediction of the future states of the short series.
Also, using the recent historical series as the input data can
concede certain degree of inner state inference. We denote this
recurrent version of CCASOT as CCASOT/his, which can be
trained in the same manner as CCASOT, with modified input
vector.
2) Discounted Return: With respect to the output y(t) of
the Q-network, one can use the expected return of the future
as the cost. In RL, as a general practice, the expected return is
usually defined as a discounted summation of the cost in the
future as the expected return, i.e., y(t)+ηy(t+1)+η 2 y(t+2)+
η 3 y(t + 3)..., where 0 < η < 1 is the discounted factor. In the
above proposed algorithm, we simply set y(t) as the training
cost, instead of the discounted return. The reason is that in the
general RL problems, the system state transition is determined
by the actions taken (standard Markov decision process); while
in our problem, the state transition is largely determined by
the weather and the workload fluctuations (although some
inner states are affected by the actions, which makes it a
semi-Markov process). To evaluate the effectiveness of the
discounted return, we carry out tests with discounted cost. That
is, we replace the training cost function for the Q-network, in
which the reward y is replaced by y + ηQ(Xµ0 , µ(Xµ0 )), where
Xµ0 is the next state to the current state Xµ . This version of
algorithm is denoted with a postfix “/DR”.
3) Constraints on Consecutive Set-points: In cooling control optimization, the control settings at consecutive time steps
shall not change abruptly. This is due to the fact that the
set-point is not directly controlled and each of setting values
takes certain time delay to be in effect. In such case, we can
add a constraint to limit the variability of the outputs of the
control policy in consecutive time steps. In particular, we add
the following consecutive constraint (CC) cost penalty when
updating the policy network:
νCC =

X
j∈Ii &j+1∈Ii

(xµj − xµj+1 )2 ,

7

where Ii is the i-th training batch and xµ is the output of the
policy network. This version of algorithm is denoted with a
postfix “/CC”. We carry out tests to verify its effects in the
next section.
IV. S IMULATION BASED N UMERICAL E VALUATION
R ESULTS AND A NALYSIS
In this section we present numerical evaluation results of the
proposed CCASOT. Simulations on the EnergyPlus platform
are carried out to collect the training data and evaluate the proposed algorithms. Two baseline algorithms are compared with
our proposed solutions: one is the default control algorithm
KBest from EnergyPlus, which controls the simulation with
the knowledge of the system dynamics; another is a general
TS control optimization algorithm which is trained with the
same data for the proposed approach.
A. Simulation Configurations
We use EnergyPlus to collect the training data and evaluate
different control algorithms for the following reasons. First, it
is impractical to directly test control algorithms on a real data
center due to the potential risk and the long running time.
Second, EnergyPlus, whose development is initiated by the
U.S. Department of Energy Building Technologies Office, is a
widely recognized and reliable simulation platform to model
building cooling energy consumption. Third, EnergyPlus provides the flexibility that allows simulations with user defined
algorithms, control actions, and schedules.
The simulation is configured as follows. We adopt the
original data center model provided by the EnergyPlus platform to make this simulation based study tractable, as shown
in Section II-A. We choose Singapore as the location and
select the corresponding weather file to revise the simulation
configuration file accordingly. We use a CPU loading trace
collected from the monitoring system of Wikimedia as the
workload trace for the data center model in the simulation.
The whole simulation period is one year and simulation data
are collected every 6 minutes. To avoid the potential issues of
time delay between an action and the resulting state change,
we “synchronize” the traces by aligning the state at the i-th
time step with the action and the reward at the (i + 1)-th time
step. For the simplicity of notation, the time slot index t used
in this paper is already after time slot realignment.
We use two different traces to test the algorithms. The
first trace, referred to as “D” trace, is generated by the
default KBest control algorithm of EnergyPlus, with the zone
temperature set-point set to 29o C. The second trace, referred
to as “r” trace, is collected when the control variables are
randomly selected from valid ranges for the simulation model.
That is, we select set-points according to TCW ater (t) ∈ [5, 9],
TCCoil (t) ∈ [18, 23], TDEC (t) ∈ [21, 31], TIEC (t) ∈ [21, 31],
and Tsup (t) ∈ [18, 23]. These two traces are used to test
different control algorithms for four test cases corresponding
to four different test periods. For the whole 1-year simulation
period, we select the last l% as the test period of the year,
where we set l to 15%, 25%, 35%, and 45% in four test cases.
We set λ = 1.0 in the objective function to compute the cost.

B. Algorithm Configurations
For the proposed CCASOT, the hidden layer sizes of Q and
µ network are set to 50 and 10, respectively. The optimization
algorithm to update the weight parameters is Adadelta [25].
The maximum training epoch M axEpoch is set to 100. The
training batch size is set to 1. For the CC version of the
proposed algorithm, we set the batch size to 5. For DR version
of the proposed algorithm, η is set to 0.1.
For the TS optimization algorithm, we use the following
settings. In the first stage, we use a three-layer neural network
to approximate the system dynamics (which is the same as the
Q-network of the CCASOT). The hidden layer size is set to
50. The maximum training epoch to train this neural network
is also set to 100. In the second stage, an iterative differential
evolutionary algorithm provided by Scipy [26] is used to find
the optimal solution for each test state.
For the proposed CCASOT and the TS optimization algorithms, the optimal control settings generated by these two
algorithms are further tested by simulations on the EnergyPlus
platform. That is, for each state at the testing phase, we use
the settings provided by the CCASOT or the TS algorithm
and then record the resulting state changes and rewards for
performance evaluation.

C. Comparing the proposed CCASOT to Baseline Algorithms
In this section we present the results of the average PUE and
maximum zone temperatures (during the test period), obtained
by using CCASOT, and also TS and KBest algorithms. Based
on results we further compare and evaluate the underlying
control algorithms.
Table I shows the first and second-order statistics of the
PUE and maximum zone temperatures in 10 independent
runs. For better examining these results, we also plot PUE
and the maximum zone temperature (for z1 ) distribution in
Fig. 4. We can observe the following. a) With KBest control
algorithm, the average PUE is the smallest. However, the
monitored maximum zone temperature is much higher than
that of the other algorithms (above 30o C vs. the pre-defined
set-point 29o C). This result shows that the KBest algorithm
produces a relatively coarse control granularity. b) CCASOT
algorithms can control the temperature within the constraint.
Its better performance proves that the algorithm indeed can
learn the underlying system dynamics and make reliable
decisions. c) Compared with the CCASOT, the general TS
algorithm under-performs. Since we utilize an iterative solver
at each time step, the TS algorithm approach is not only time
consuming, but also leads to slow convergence and instability.
While with the CCASOT, the trained actor network works
as a deterministic non-linear function that maps the states to
actions, thus generating much more stable results. Moreover,
the comparison between CCASOT and TS proves that the
actor network can indeed attain optimal or close-to optimal
control settings. d) The recurrent version of CCASOT has
better performance. This indicates that it is beneficial to also
take recent states and actions into consideration for decision
making. e) The proposed CCASOT/his performs the best

8

TABLE I
C OMPARISON OF THE PROPOSED CCASOT WITH THE BASELINE ALGORITHMS : KB EST AND TS. T HE RESULTS SHOW THAT THE CCASOT/ HIS
PERFORMS WELL ON BOTH THE R AND D TRACES , WITH SMALLER PUE OR BETTER MAXIMUM ZONE TEMPERATURE ( o C) ACHIEVED .

15%

25%

35%

45%

PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )

KBest
mean±std
1.3794±0.0000
30.0572±0.0000
30.0519±0.0000
1.3771±0.0000
30.0611±0.0000
30.0614±0.0000
1.3807±0.0000
30.0697±0.0000
30.0821±0.0000
1.3819±0.0000
30.0697±0.0000
30.0821±0.0000

CCASOT/r
mean±std
1.4400±0.0470
29.5385±1.0346
29.5390±1.0354
1.4222±0.0215
29.0543±0.0057
29.0544±0.0057
1.4342±0.0398
29.3834±0.9914
29.3806±0.9834
1.4294±0.0424
29.7146±1.3242
29.7101±1.3157

CCASOT/D
mean±std
1.4428±0.0418
29.0538±0.0067
29.0540±0.0066
1.4230±0.0306
29.0571±0.0044
29.0572±0.0044
1.4360±0.0436
29.0579±0.0045
29.0578±0.0045
1.4297±0.0289
29.0589±0.0025
29.0588±0.0024

CCASOT/his/r
mean±std
1.4209±0.0016
29.0561±0.0002
29.0562±0.0002
1.4112±0.0019
29.0562±0.0002
29.0563±0.0003
1.4284±0.0283
29.0573±0.0005
29.0572±0.0003
1.4170±0.0016
29.0571±0.0005
29.0571±0.0003

TABLE II
C OMPARING THE CCASOT INCORPORATED WITH CONSECUTIVE
CONSTRAINT (CC) AND DISCOUNTED RETURN (DR). A LL
TEMPERATURES ARE IN o C.

15%

25%

35%

45%

PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )

CCASOT/his/r
mean±std
1.4209±0.0016
29.0561±0.0002
29.0562±0.0002
1.4112±0.0019
29.0562±0.0002
29.0563±0.0003
1.4284±0.0283
29.0573±0.0005
29.0572±0.0003
1.4170±0.0016
29.0571±0.0005
29.0571±0.0003

CCASOT/his/D
mean±std
1.4206±0.0061
29.0585±0.0000
29.0586±0.0000
1.4158±0.0070
29.0585±0.0001
29.0586±0.0001
1.4181±0.0093
29.0597±0.0003
29.0596±0.0003
1.4200±0.0084
29.0596±0.0004
29.0595±0.0004

CCASOT/his/r/CC
mean±std
1.4196±0.0006
29.0560±0.0003
29.0561±0.0003
1.4096±0.0004
29.0561±0.0002
29.0562±0.0002
1.4166±0.0010
29.0572±0.0005
29.0572±0.0004
1.4159±0.0002
29.0574±0.0005
29.0573±0.0004

CCASOT/his/D/CC
mean±std
1.4155±0.0034
29.0585±0.0000
29.0586±0.0000
1.4081±0.0058
29.0585±0.0000
29.0586±0.0000
1.4137±0.0045
29.0599±0.0000
29.0598±0.0000
1.4159±0.0140
29.0599±0.0000
29.0598±0.0000

CCASOT/his/r/DR
mean±std
1.6181±0.1006
29.0682±0.1622
29.0546±0.1547
1.6307±0.0685
29.0223±0.0127
29.0068±0.0088
1.6001±0.0799
29.0844±0.1940
29.0753±0.2064
1.6630±0.0195
29.0314±0.0040
29.0063±0.0048

CCASOT/his/D/DR
mean±std
1.4258±0.0364
29.0585±0.0000
29.0586±0.0000
1.4288±0.0449
29.0585±0.0000
29.0586±0.0000
1.4181±0.0140
29.0599±0.0000
29.0598±0.0000
1.4214±0.0131
29.0599±0.0000
29.0598±0.0000

CCASOT/his/D
mean±std
1.4206±0.0061
29.0585±0.0000
29.0586±0.0000
1.4158±0.0070
29.0585±0.0001
29.0586±0.0001
1.4181±0.0093
29.0597±0.0003
29.0596±0.0003
1.4200±0.0084
29.0596±0.0004
29.0595±0.0004

D. Additional Methods for Further Improving the Performance of CCASOT
In this subsection we show results of CCASOT/his, incorporated with CC or DR, to verify whether these techniques
can be helpful. The results are shown in Table II.
We observe in Table II that with CC, the CCASOT performs
more stably and produces better PUE. Compared with a
general TD/r, the performance of PUE drops from 1.45 to 1.41,
equivalent to about 10% reduction of cooling cost. It should
be mentioned that although CC is useful in our problem (as
it limits the range of the control setting change), its benefit
does not necessarily extends to other problem set-ups. For the
DR, we observe that the algorithm produces unstable results,
especially for the r trace. This can be explained by the fact
that for data center temperature control the effect of a control
setting at a particular time step does not propagate too far into
the future. In other words, the effect drastically diminishes
after a certain duration. Using DR may over-emphasize the
effect of one action on the states and actions in the far future.
As such, the general practice of RL does not necessarily hold
for our solution approaches.
E. Comparing Different Neural Network Designs
In this subsection, we compare different ways to implement
the neural network. We compare the basic network design
with the four other different approaches: 1) SimCost: with
the simple definition of cost function analyzed presented in
Section II-B; 2) Drop: we add a drop-out layer with a total
number of nodes that is five times that of hidden layer;
3) Target: with the target network of DDPG; 4) NewArc:
with a different architecture in reference [27]. The results are

TS/D
mean±std
1.4555±0.0421
29.2108±0.4615
29.2088±0.4552
1.4219±0.0241
29.1985±0.3421
29.1950±0.3375
1.4426±0.0363
29.4921±0.7694
29.4876±0.7647
1.4573±0.0432
29.7679±0.7193
29.7602±0.7093

TS/his/r
mean±std
1.4537±0.0338
29.2009±0.1211
29.1907±0.1185
1.4434±0.0288
29.9581±0.7474
29.9555±0.7357
1.4484±0.0293
29.6196±0.1655
29.6024±0.1686
1.4409±0.0351
29.3090±0.2759
29.3175±0.2940

TS/his/D
mean±std
1.4295±0.0191
29.2360±0.1518
29.2310±0.1539
1.4289±0.0248
29.9856±0.5003
29.9830±0.4931
1.4293±0.0210
30.0536±0.3854
30.0383±0.3810
1.4276±0.0221
29.7497±0.4973
29.7494±0.4850

TABLE III
C OMPARING DIFFERENT NEUAL NETWORK IMPLEMENTATIONS . THE
BASIC , S IM C OST, D ROP, TARGET, AND N EWA RC . A LL TEMPERATURES
ARE IN o C.

15%

25%

35%

45%

among all the learning algorithms studied here, and it shows
competitive performance on both training traces.

TS/r
mean±std
1.4189±0.0204
31.6394±0.9517
31.6336±0.9528
1.4625±0.0414
31.5420±1.1069
31.5319±1.1014
1.4745±0.0507
31.6859±1.1578
31.6968±1.1376
1.4595±0.0552
30.9061±1.4223
30.8943±1.4151

PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )

CCASOT/r
mean±std
1.4400±0.0470
29.5385±1.0346
29.5390±1.0354
1.4222±0.0215
29.0543±0.0057
29.0544±0.0057
1.4342±0.0398
29.3834±0.9914
29.3806±0.9834
1.4294±0.0424
29.7146±1.3242
29.7101±1.3157

CCASOT/r/SimCost
mean±std
1.4603±0.0484
29.6380±1.1911
29.6394±1.1936
1.4042±0.0373
30.0447±1.5143
30.0381±1.5040
1.4461±0.0486
29.6481±1.1728
29.6434±1.1683
1.4459±0.0614
30.0390±1.5210
30.0316±1.5100

CCASOT/r/Drop
mean±std
1.4649±0.0384
30.8055±1.0116
30.7851±1.0020
1.4537±0.0589
30.5054±0.8537
30.5028±0.8470
1.4700±0.0514
30.1120±1.0948
30.0964±1.0874
1.4759±0.0219
30.7437±1.0035
30.7337±1.0039

CCASOT/r/Target
mean±std
1.4546±0.0690
30.0622±1.1306
30.0582±1.1293
1.4442±0.0514
29.1408±0.2515
29.1415±0.2520
1.4674±0.0543
29.0559±0.0060
29.0558±0.0059
1.4732±0.0473
29.3066±0.5221
29.3082±0.5205

CCASOT/r/NewArc
mean±std
1.4625±0.0551
29.0562±0.0002
29.0563±0.0002
1.4948±0.0000
30.0622±0.0000
30.0803±0.0000
1.4665±0.0000
30.0762±0.0000
30.0839±0.0000
1.4502±0.0000
30.0762±0.0000
30.0839±0.0000

shown in Table III, all with r trace. The results show that the
simple cost function causes performance degeneration, which
proves that the cost function we choose at the first place can
successfully generate optimal or near-optimal control policies.
For the other network architectures, they do not show any
significant performance improvement. The comparisons show
that for cooling control problem, implementing CCASOT with
a simple neural network design that is easy to train can produce
better and more robust performance, which is critical for
designing a practical controller. However, it should be noticed
that for problems with high-dimension of state and action
space, we may need to use a deeper (more hidden layers)
neural network to extract useful features from the input.
F. Comparing Different Training Algorithms
The above results are so far generated with the training
algorithm AdaDelta. It is also interesting to evaluate the
performance of other training algorithms. In Table IV, we
show the results for different training algorithms: AdaDelta
(Ada), rmsprop [28] (Rms), and stochastic gradient descent
(Sgd). The results show that the best training algorithm is
AdaDelta, while Rmsprop performs almost as well. This is
expected, since both algorithms adapt step size in updating
the weight parameters of the neural networks. These results
indicate that choosing a proper training algorithm is important,
even for the simple cooling control instance studied in this
paper.
G. Examples of Simulation Results
In this subsection, we present the examples of the PUE and
temperature traces obtained from our simulation. As shown
in Fig. 5, we compare the best TS variant TS/his/D, and the

9

1.60
1.55
1.50
1.45
1.40
1.35

32.5
32.0
31.5
31.0
30.5
30.0
29.5
29.0

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

(a)
1.52
1.50
1.48
1.46
1.44
1.42
1.40
1.38
1.36

32.5
32.0
31.5
31.0
30.5
30.0
29.5
29.0

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

(b)
1.55
1.50
1.45
1.40
1.35

32.5
32.0
31.5
31.0
30.5
30.0
29.5
29.0

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

(c)
1.54
1.52
1.50
1.48
1.46
1.44
1.42
1.40
1.38
1.36

32.5
32.0
31.5
31.0
30.5
30.0
29.5
29.0

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

KBest

CCASOT/r

CCASOT/D CCASOT/his CCASOT/his
/r
/D

TS/r

TS/D

TS/his/r

TS/his/D

(d)
Fig. 4. Average PUE (over the test period) and maximum zone temperature distribution achieved by different control algorithms for the test case l = 15%,
25%, 35%, and 45%, shown in (a), (b), (c), and (d), respectively. In each sub-figure, the upper part shows PUE while the lower part shows maximum zone
temperature (o C). The average values are based on 10 independent runs. The red line in each box is the median and the red square is the mean value.

10

TABLE IV
C OMPARING DIFFERENT TRAINING ALGORITHMS : A DADELTA (A DA ),
RMSPROP (R MS ), AND STOCHASTIC GRADIENT DESCENT (S GD ).

15%

25%

35%

45%

PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )
PUE
Tmax (z1 )
Tmax (z2 )

CCASOT/his/r/Ada
mean±std
1.4209±0.0016
29.0561±0.0002
29.0562±0.0002
1.4112±0.0019
29.0562±0.0002
29.0563±0.0003
1.4284±0.0283
29.0573±0.0005
29.0572±0.0003
1.4170±0.0016
29.0571±0.0005
29.0571±0.0003

CCASOT/his/D/Ada
mean±std
1.4206±0.0061
29.0585±0.0000
29.0586±0.0000
1.4158±0.0070
29.0585±0.0001
29.0586±0.0001
1.4181±0.0093
29.0597±0.0003
29.0596±0.0003
1.4200±0.0084
29.0596±0.0004
29.0595±0.0004

CCASOT/his/r/Rms
mean±std
1.4205±0.0011
29.1451±0.0000
29.1373±0.0000
1.4129±0.0051
29.7210±0.0000
29.7259±0.0000
1.4257±0.0155
29.1967±0.0000
29.1896±0.0000
1.4170±0.0017
29.0915±0.0000
29.0781±0.0000

CCASOT/his/D/Rms
mean±std
1.4330±0.0320
29.1854±0.0000
29.1800±0.0049
1.4119±0.0077
29.7210±0.0000
29.7259±0.0000
1.4154±0.0070
29.8650±0.0000
29.8516±0.0000
1.4160±0.0084
29.5037±0.0000
29.5047±0.0000

CCASOT/his/r/Sgd
mean±std
1.4711±0.0713
29.5891±0.8971
29.5803±0.8913
1.4921±0.0653
29.7210±0.0000
29.7259±0.0000
1.4637±0.0633
29.1967±0.0000
29.1896±0.0000
1.4454±0.0289
29.5597±0.9586
29.5415±0.9493

CCASOT/his/D/Sgd
mean±std
1.4499±0.0477
29.1853±0.0003
29.1792±0.0047
1.4091±0.0051
29.7210±0.0000
29.7259±0.0000
1.4281±0.0398
29.8650±0.0000
29.8516±0.0000
1.4517±0.0521
29.5037±0.0000
29.5047±0.0000

TABLE V
T HE DATA ENTRIES AND THEIR USAGE IN OPTIMIZATION OF THE AIRFLOW
RATE OF THE TARGET PCU S .

State

Action

best CCA variants CCASOT/his/D/CC with the KBest control
algorithm in EnergyPlus.
The PUE curves show that: a) The PUE curve of the KBest
algorithm can drop to a very low level (about 1.1) in the
simulation. This implies that the cooling system is almost
not running. Such a greedy cooling setting shows the advantage of the KBest algorithm, utilizing the underlying system
dynamics. b) In comparison, learning-based algorithm cannot
achieve such low PUE. Among learning-based algorithm, the
PUE curve of the CCASOT/his/D/CC is lower than that of
TS/his/D most of the time, thus shows the better performance
of CCASOT. c) From the temperature traces, we observe that
with the learning algorithms, the temperature is kept very close
to the predefined threshold of 29o C. Note that at the beginning
of the test period, the temperature of the data center zone is
high for both TS and CCASOT, due to transient caused by
the transition from the KBest algorithm and to the learning
algorithm (the settings before the test period are provided by
the KBest algorithm). However, for both TS and CCASOT,
the zone temperature soon stabilizes to predefined temperature
threshold and only fluctuates within a small range (less than
0.2o C), much smaller than that of KBest. As such, in tallying
the statistics of PUE and maximum zone temperature, we omit
the first 100 time steps of the test period. We can also note
that at certain time steps, TS fails to control the temperature
with in the pre-set range, as shown in Fig. 5 (h).
V. T EST ON R EAL DATA T RACE FROM NSCC
To further investigate the proposed algorithm, we test the
best version CCASOT/his on a data trace collected from the
National Super Computing Centre (NSCC) of Singapore and
show its performance on optimizing the energy cost while
satisfying the cooling requirements (rack intake temperature).
We focus on optimization of the air cooling system for the
computing nodes in NSCC. The 3-D model of the research
target is shown in Fig. 6. There are 26 racks in the target
system. Three precision cooling units (PCUs) supply cold air
for these racks. The PCUs supply cold air at about 20 degrees.
Cold air enters into the cold aisle and then goes through
the racks and at last returns back to the PCUs. There are
other cooling facilities installed: for racks 1-20, an additional
warm water cooling system is used to cool the CPU/GPU and
memory chips; for racks 21-26, an additional rear-door cooling
system is used. The warm water cooling system and rear-door
cooling system will not be studied here thus we omit further
details.
We try to optimize the total supply flow rate of the target
PCUs shown in Fig. 6, aiming to minimize their power

Reward

P1: total power consumption of racks 1-26;
P2: total power consumption of the other racks;
P3: total heat load recorded in warm water cooling system;
F1: water flow rate; T1: water flow supply temperature;
F2: air flow rate of other PCUs;
T2: average supply temperature of other PCUs;
F3: rear door cooling distribution units chilled water flow rate; and
T3: the average supply temperature of the target PCUs.
F4: The airflow supply rate of the target PCUs
P4: Power consumption of the target PCUs; and
T4: average rack intake temperature.

consumption while maintain the average intake temperature
of the racks (measurement point at the height of the 36U of
each rack).
To apply the proposed algorithm, we collected the related
data for the optimization goal, as shown in Table V. Note that
several measurements of the warm water cooling system and
rear-door cooling system are also used. Our experiments show
that including these readings can increase the approximation
accuracy of the Q-network. We collected these data entries for
every 3 minutes during March 1 to 15 of 2017. For these data,
we use the first 85% as the training data, and the last 15% as
the test data. With the above data, we utilize the proposed
algorithm to train the Q and µ network. In the Q network, we
tried a special design that will approximate the energy cost
and rack temperature respectively.
Experimental results show that the proposed algorithm
works well as expected. First, the results show that the
normalized mean absolute error of the Q network is smaller
than 10% (0.1 degree), as shown in Fig 7. This proves that
the network can successfully capture the system dynamics.
Second, we study how the control settings output by the
policy network change when different settings of the penalty
parameter λ (used in the objective function (1)) are used, as
shown in Fig. 8. In figure 8 (a), we show when λ changes from
0.0, 0.1, to 10.0, the control settings output by the proposed
algorithm change as the curves’ color changes from dark to
light yellow, while the real setting is shown as the blue curve.
At the same time, we show how the predicted rack temperature
changes correspondingly. We can observe that 1) With the
control setting of our proposed algorithm, we can achieve a
lower PCU supply rate (means lower power consumption),
and achieve a rack temperature curve which will not surpass
the maximum temperature of the observed real data; 2) With
higher value of λ, the predicted flow rate is increasing and the
predicted temperature curve is lower. This is reasonable as
the larger λ is, the higher “price” the temperature constraint
gains, which then causes the control setting optimization to
evolve towards even lower rack temperature direction. These
results show that our algorithm can be utilized our to generate
different control settings with different cooling requirements,
while optimize the corresponding energy cost. For example,
when we set λ = 1, we can control the rack temperature under
26.5 degree, while achieving cooling energy saving 13.6%.

11

30.2

Default
TS/D
CCASOT/His/D/CC

2.2
PUE

2.0
1.8
1.6
1.4

Default
TS/his/D
CCASOT/His/D

30.0
Temperature

2.4

29.8
29.6
29.4
29.2
29.0

1.2
0

2000

4000

6000
Time (unit: 6mins)

8000

28.8

10000

0

2000

4000

(a)

1.8
1.6
1.4

Default
TS/his/D
CCASOT/His/D

30.0
Temperature

PUE

2.0

29.8
29.6
29.4
29.2
29.0

1.2
0

5000

10000
Time (unit: 6mins)

15000

28.8

20000

0

5000

10000
Time (unit: 6mins)

(c)

15000

20000

(d)
30.2

Default
TS/D
CCASOT/His/D/CC

2.0
1.8
1.6
1.4

Default
TS/his/D
CCASOT/His/D

30.0
Temperature

2.2

PUE

12000

30.2

Default
TS/D
CCASOT/His/D/CC

2.2

1.2

29.8
29.6
29.4
29.2
29.0

0

5000

10000

15000
20000
Time (unit: 6mins)

25000

28.8

30000

0

5000

10000

(e)

15000
20000
Time (unit: 6mins)

25000

30000

(f)
31.5

Default
TS/D
CCASOT/His/D/CC

2.0
1.8
1.6
1.4

Default
TS/his/D
CCASOT/His/D

31.0
Temperature

2.2

PUE

10000

(b)

2.4

1.0

6000
8000
Time (unit: 6mins)

30.5
30.0
29.5
29.0

1.2
0

5000

10000

15000
20000
25000
Time (unit: 6mins)

30000

35000

(g)

28.5

0

5000

10000

15000
20000
25000
Time (unit: 6mins)

30000

35000

(h)

Fig. 5. Simulation result comparison of the Default algorithm, TS/D and CCASOT/his/r. For different test cases, PUE of the test period is shown in the
left and the data center temperature (for z1 ) is in the right; (a)(b): test case 15%; (c)(d): test case 25%; (e)(f): test case 35%; (g)(h): test case 45%. All
temperatures are in o C.

VI. R ELATED W ORKS
Cooling system control optimization has been extensively
studied. A significant portion of these works [4] [5] [6] [7]
[8] [9] focus on using a two stage optimization procedure to
find the optimal control setting. For the first stage, various
kinds of models are built to evaluate the policy quality, such
as approximate mathematical model [4] [5] [6] [7] [8] and
neural network model [9]. For the second stage, the control
variables are then optimized with the built model. Another area
of research for cooling control optimization is to optimize the
ice-storage system [29] [30], so that it can be used for cooling
when the electricity price is high.
In addition to the cooling system optimization surveyed
above, there exists an extensive body of works that focuses on
the ICT of the data center. For example, in [31] [32] [33], the

workload dispatch problem is studied to optimize the thermal
map in the data center to improve cooling efficiency. In [34],
the workload dispatch problem is extended to geographically
distributed data centers. Moreover, renewable energy is incorporated in [35] to reduce the cooling cost.
Reinforcement Learning deals with agents that learn to take
better actions directly from experience of interacting with the
environment. With the recent rapid advancement of RL, deep
reinforcement learning (DRL) [11] has shown its strength in
various fields. The deep Q-network (DQN) proposed in [11]
applies a neural network approximation to the Q table in
Q-learning [36]. The subsequent studies on DQN have been
focusing on improving the training stability of the framework
such as in [37] and extending the framework to solve problems
with continuous control variables [12]. Various applications of

12

19500
19000
Air flow rate (CFM)

3UHFLVLRQ FRROLQJ
XQLW 3&8

18500
18000
17500
17000
16500

0DQDJHPHQW
UDFNV
6XSHUFRPSXWLQJ
UDFNV

&ROG DLVOH

0

200

400
600
Time (step size: 3mins)

800

1000

800

1000

(a)
27.5

Power

Fig. 6. 3-D model of the target system in NSCC. There are 26 racks, which
are cooled by the PCUs. The cold air supplied by the PCUs first goes through
the raised floor to the cold aisle, and then cools the racks before returning to
the PCUs.

7.4
7.2
7.0
6.8
6.6
6.4
6.2
6.0
5.8

1e12

0

Validation Predicted Results
Validation Real Data

200

400
600
Time (step size: 3mins)

800

1000

(a)

Temperature (C)

27.0

Validation Predicted Results
Validation Real Data

26.5
26.0
25.5
0

200

400
600
Time (step size: 3mins)

800

1000

(b)
Fig. 7. The Q-network can predict the energy and rack temperature with
high accuracy. We use part of the training data to validate the accuracy of
the Q-network, the results are shown in (a) power consumption and (b) rack
temperature.

deep reinforcement learning have been proposed such as video
processing [38] and text-based game [39].
VII. C ONCLUSION
Data center powers the modern society as the infrastructure
for information storage, processing, and dissemination. At the
same time, data center consumes a formidable amount of
electricity, among which a large portion is used in cooling.
To develop an optimal control policy for the complex cooling
system for a data center is a difficult task. We propose and

Temperature (C)

27.0
26.5
26.0
25.5
25.0

0

200

400
600
Time (step size: 3mins)

(b)
Fig. 8. Test results on the NSCC data trace when λ changed from 0.0 to
10: (a) control setting (airflow rate) predicted by the proposed algorithm; (b)
the correspondingly predicted rack temperature. Darker color curves are with
smaller λ settings, and the blue curve (with ”+” marker) is the real data
trace collected in the test period. Note lower airflow rate means lower energy
consumption. When we set λ=1, we can achieve energy saving 13.6% while
the maximum rack temperature is lower than 26.5 degree.

verify an end-to-end approach for the control optimization of
the cooling system for a data center. Compared to the existed
TS optimization method, the end-to-end method can directly
optimize a policy network based on the observed historical
data, while the policy can output the optimized control settings
for any given state. Adapted from DDPG and the actor-critic
framework, our algorithm is a batch off-policy algorithm, with
which we tested various different DRL settings like recurrent
decision making, discounted return, etc.
We evaluate the proposed algorithm on the simulation
platform EnergyPlus. The simulation results show that our
method can maintain the data center temperature within the
predefined threshold while achieving low PUE, and saving
about 10% cooling energy compared to a canonical two
stage optimization approach. We also apply our method to
a real data trace collected from NSCC of Singapore, and
the results show the high approximation accuracy and the
predicted control setting can clearly reduce the cooling cost by
about 14% while maintain the rack intake temperature under
predefined threshold. The results prove that our algorithm can
successfully learn the system dynamics from the monitoring
data and can help to improve the cooling efficiency.
R EFERENCES
[1] “Cisco Global Cloud Index: Forecast and Methodology, 2014-2019,”
http://www.cisco.com/c/en/us/solutions/collateral/service-provider/
global-cloud-index-gci/Cloud Index White Paper.html,
accessed:
2015-07-01.

13

[2] “America’s data centers consuming and wasting growing amounts of
energy,” http://www.nrdc.org/energy/data-center-efficiency-assessment.
asp, accessed: 2015-07-01.
[3] “Data Centre Energy Efficiency Benchmarking - E2 Singapore,”
http://www.e2singapore.gov.sg/DATA/0/docs/Resources/
NEADCEnergyBenchmarkingSummary-FinalReport(3).pdf, accessed:
2015-07-01.
[4] J. Sun and A. Reddy, “Optimal control of building HVAC systems using complete simulation-based sequential quadratic programming (csbsqp),” Building and environment, vol. 40, no. 5, pp. 657–669, 2005.
[5] B. Ahn and J. Mitchell, “Optimal control development for chilled water
plants using a quadratic representation,” Energy and Buildings, vol. 33,
no. 4, pp. 371–378, 2001.
[6] L. Lu, W. Cai, L. Xie, S. Li, and Y. C. Soh, “Hvac system optimizationinbuilding section,” Energy and Buildings, vol. 37, no. 1, pp. 11–22, 2005.
[7] Y.-C. Chang, “A novel energy conservation methodoptimal chiller loading,” Electric Power Systems Research, vol. 69, no. 2, pp. 221–226,
2004.
[8] Z. Ma and S. Wang, “An optimal control strategy for complex building
central chilled water systems for practical and real-time applications,”
Building and Environment, vol. 44, no. 6, pp. 1188–1198, 2009.
[9] T. Chow, G. Zhang, Z. Lin, and C. Song, “Global optimization of
absorption chiller system by genetic algorithm and neural network,”
Energy and buildings, vol. 34, no. 1, pp. 103–109, 2002.
[10] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.
MIT press Cambridge, 1998, vol. 1, no. 1.
[11] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
[12] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,
D. Silver, and D. Wierstra, “Continuous control with deep reinforcement
learning,” arXiv preprint arXiv:1509.02971, 2015.
[13] “DeepMind
AI
Reduces
Google
Data
Centre
Cooling
Bill
by
40%,”
https://deepmind.com/blog/
deepmind-ai-reduces-google-data-centre-cooling-bill-40/,
accessed:
2017-03-01.
[14] M. Dayarathna, Y. Wen, and R. Fan, “Data center energy consumption modeling: A survey,” IEEE Communications Surveys & Tutorials,
vol. 18, no. 1, pp. 732–794, 2016.
[15] W. Zhang, Y. Wen, Y. W. Wong, K. C. Toh, and C.-H. Chen, “Towards
joint optimization over ict and cooling systems in data centre: A survey,”
IEEE Communications Surveys & Tutorials, vol. 18, no. 3, pp. 1596–
1616, 2016.
[16] W. Xia, Y. Wen, K.-C. Toh, and Y.-W. Wong, “Toward green data
centers as an interruptible load for grid stabilization in singapore,” IEEE
Communications Magazine, vol. 53, no. 11, pp. 192–198, 2015.
[17] Y. Li, H. Hu, Y. Wen, and J. Zhang, “Learning-based power prediction
for data centre operations via deep neural networks,” in Proceedings
of the 5th International Workshop on Energy Efficient Data Centres.
ACM, 2016, p. 6.
[18] J. Yin, P. Sun, Y. Wen, H. Gong, M. Liu, X. Li, H. You, J. Gao,
and C. Lin, “Cloud3dview: an interactive tool for cloud data center
operations,” in ACM SIGCOMM Computer Communication Review,
vol. 43, no. 4. ACM, 2013, pp. 499–500.
[19] I. Grondman, L. Busoniu, G. A. Lopes, and R. Babuska, “A survey of
actor-critic reinforcement learning: Standard and natural policy gradients,” IEEE Transactions on Systems, Man, and Cybernetics, Part C
(Applications and Reviews), vol. 42, no. 6, pp. 1291–1307, 2012.
[20] D. B. Crawley, L. K. Lawrie, F. C. Winkelmann, W. F. Buhl, Y. J.
Huang, C. O. Pedersen, R. K. Strand, R. J. Liesen, D. E. Fisher, M. J.
Witte et al., “Energyplus: creating a new-generation building energy
simulation program,” Energy and buildings, vol. 33, no. 4, pp. 319–331,
2001.
[21] “Wikimedia Grid Report,” https://ganglia.wikimedia.org/latest/, accessed: 2015-07-01.
[22] X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural
networks.” in Aistats, vol. 15, no. 106, 2011, p. 275.
[23] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller,
“Deterministic policy gradient algorithms,” in Proceedings of the 31st
International Conference on Machine Learning (ICML-14), 2014, pp.
387–395.
[24] B. Bakker, “Reinforcement learning memory,” 2002.
[25] M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXiv
preprint arXiv:1212.5701, 2012.
[26] E. Jones, T. Oliphant, P. Peterson et al., “Open source scientific tools
for python,” 2001.

[27] “Deep
Deterministic
Policy
Gradients
in
TensorFlow,”
http://pemami4911.github.io/blog/2016/08/21/ddpg-rl.html,
accessed:
2017-03-01.
[28] T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient
by a running average of its recent magnitude,” COURSERA: Neural
networks for machine learning, vol. 4, no. 2, 2012.
[29] J. E. Braun, “Reducing energy costs and peak electrical demand through
optimal control of building thermal storage,” ASHRAE transactions,
vol. 96, no. 2, pp. 876–888, 1990.
[30] C.-C. Lo, S.-H. Tsai, and B.-S. Lin, “Ice storage air-conditioning system
simulation with dynamic electricity pricing: A demand response study,”
Energies, vol. 9, no. 2, p. 113, 2016.
[31] E. Pakbaznia and M. Pedram, “Minimizing data center cooling and
server power costs,” in Proceedings of the 2009 ACM/IEEE international
symposium on Low power electronics and design. ACM, 2009, pp.
145–150.
[32] L. Li et al., “Coordinating liquid and free air cooling with workload
allocation for data center power minimization,” in 11th International
Conference on Autonomic Computing (ICAC 14). USENIX Association,
2014.
[33] A. Banerjee, T. Mukherjee, G. Varsamopoulos, and S. K. Gupta,
“Integrating cooling awareness with thermal aware workload placement
for hpc data centers,” Sustainable Computing: Informatics and Systems,
vol. 1, no. 2, pp. 134–150, 2011.
[34] C. Chen, B. He, and X. Tang, “Green-aware workload scheduling in
geographically distributed data centers,” in 2012 IEEE 4th International
Conference on Cloud Computing Technology and Science (CloudCom).
IEEE, 2012, pp. 82–89.
[35] Z. Liu, Y. Chen, C. Bash, A. Wierman, D. Gmach, Z. Wang, M. Marwah,
and C. Hyser, “Renewable and cooling aware workload management for
sustainable data centers,” in ACM SIGMETRICS Performance Evaluation Review, vol. 40, no. 1. ACM, 2012, pp. 175–186.
[36] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no.
3-4, pp. 279–292, 1992.
[37] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning
with double q-learning,” CoRR, abs/1509.06461, 2015.
[38] J. Koutnı́k, J. Schmidhuber, and F. Gomez, “Evolving deep unsupervised
convolutional networks for vision-based reinforcement learning,” in Proceedings of the 2014 Annual Conference on Genetic and Evolutionary
Computation. ACM, 2014, pp. 541–548.
[39] K. Narasimhan, T. Kulkarni, and R. Barzilay, “Language understanding
for text-based games using deep reinforcement learning,” arXiv preprint
arXiv:1506.08941, 2015.

