1

Green Data Centers: A Survey, Perspectives, and Future
Directions
Xibo Jin1,2 , Fa Zhang1 , Athanasios V. Vasilakos3 and Zhiyong Liu1
of Computing Technology, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences, Beijing, China
3 University of Western Macedonia, Greece
1,2
Email: {jinxibo, zhangfa, zyliu}@ict.ac.cn 3 vasilako@ath.forthnet.gr

arXiv:1608.00687v1 [cs.DC] 2 Aug 2016

1 Institute

Abstract—At present, a major concern regarding data centers
is their extremely high energy consumption and carbon dioxide
emissions. However, because of the over-provisioning of resources,
the utilization of existing data centers is, in fact, remarkably low,
leading to considerable energy waste. Therefore, over the past
few years, many research efforts have been devoted to increasing
efficiency for the construction of green data centers. The goal of
these efforts is to efficiently utilize available resources and to reduce
energy consumption and thermal cooling costs. In this paper, we
provide a survey of the state-of-the-art research on green data center
techniques, including energy efficiency, resource management, thermal control and green metrics. Additionally, we present a detailed
comparison of the reviewed proposals. We further discuss the key
challenges for future research and highlight some future research
issues for addressing the problem of building green data centers.
Keywords—Green data center; Energy efficiency; Resource management; Thermal control; Green metrics, monitoring and experimental
techniques

I.

I NTRODUCTION

As hosts of large-scale service applications, data centers now
play a crucial role in modern Information Technology (IT)
infrastructures. With the development of cloud computing,
data centers are growing exponentially in number and size. A
new study from International Data Corporation (IDC) predicts
that the total number of data centers deployed worldwide will
peak at 8.6 million in 2017 [1]. Consequently, the extremely
high energy consumption and carbon dioxide emissions of
data centers are becoming a major concern worldwide. The
Natural Resources Defense Council (NRDC) reported that 91
billion KWh of electricity was consumed by U.S. data centers in
2013. Continuing this trend, NRDC estimates that the electricity
usage will reach 140 billion KWh by 2020. As a result, data
center electricity consumption will cost American businesses
$13 billion annually in electricity bills and result in the emission
of nearly 100 million metric tons of carbon pollution per year
[2]. Globally, it had revealed that power requirements grew by
63% to 38 Gigawatts in 2012, up from 24 Gigawatts in 2011
[3]. Moreover, data centers will consume 8% of the worldwide
electricity supply by 2020 while this fraction was about 1.3%
in 2010 [4]. Therefore, making data centers “green” can reduce
both costs for energy consumption and carbon dioxide emissions.
Existing data centers often operate at low utilization because
of over-provisioning and fragmentation of resources [5], [6],
leading to considerable energy waste. According to a McKinsey
study in 2008 [7], the typical utilization ratio is approximately
6%. A Gartner report from 2012 [8] found the typical utilization rates to be in the 7% to 12% range, slightly better than
the result of the 2008 McKinsey study. Recently, Google Inc.
reported that they can improve the utilization of their servers
to relatively high rates of 20-40% [9]. Many studies [10], [11],

[12] have also shown that data center networks experience
high underutilization, with typical utilization ranging between
5% and 25%. What makes the matter worse is that the low
utilization of these servers and network resources also cause
the waste of other supporting infrastructure, such as power
distribution and cooling. From these data, we can see that the
present energy efficiency of data centers is extremely low and
that there is great potential for reducing energy consumption
in data centers.
Motivated by the high energy consumption and low utilization of data centers, many research efforts over the past few
years have focused on the design of green data center infrastructures and services. Generally, these approaches can be categorized into two major classes [13]: 1) those that adopt “green”
equipment in the preliminary design and building phase of the
data centers, and 2) those that increase the efficiency of the
daily expenditures incurred during the operation of the data
centers. In this paper, we focus on the latter class of methods,
i.e., those that emphasize managing the procedures for running
and operating data centers in a “green” manner. To this end,
research studies on the development of green data centers
have concentrated on the following aspects: 1) decreasing the
power consumption of data center resources, 2) increasing the
utilization of data centers, 3) controlling the thermal behavior of
data centers, and 4) developing green metrics, monitoring and
experimental techniques. From the perspective of data center
architecture, the fundamental components of data centers are
1) computing servers, 2) connection networks, and 3) cooling
equipment. Correspondingly, green data center technologies
can be applied to a individual component or to a hybrid scheme
of components.
Although obvious progress has been made in the construction of green data centers over the past decade, there is still
a large gap, and therefore a large opportunity for savings,
between the current average and the characteristics of a bestpractice green data center [14]. In this paper, we present a
survey of the current state-of-the-art research on green data
centers, as summarized in Fig. 1. Some recent surveys have
conducted on energy-aware resource allocation for cloud computing [15] or green data center networks [16]. Different from
them, our survey covers many aspects of the development of
green data centers. Our main contributions are two-fold: 1)
we discuss the key insights underlying recent strategies and
compare the relevant proposals, and 2) we note the future
research challenges and directions for various aspects of the
design of green data centers.
The remainder of this paper is organized as follows. First,
we discuss the studies on and challenges of decreasing power
consumption in Section II. We introduce efforts toward and
opportunities for the efficient utilization of data center re-

2

Green Data Center

Energy
Efficiency

Resource
Management
Virtual Machine
Assignment

Dynamic
Speed Scaling

Network Traffic
Engineering

Power-Down
Mechanism

Power Distribution
Hybrid Technology

Green
Metrics

Thermal
Control

Cooling and
Workload
Distribution

Green Metrics

TemperatureReliability Trade-off

Green Monitoring
and Experimental
Techniques

Renewable
Energy Access
Fig. 1: An overview of green data center techniques

sources in Section III. We explore studies and potential cooling
issues related to data center temperature and thermal control in
Section IV, and we describe the development of and guidelines
for green metrics, monitoring and experimental techniques in
Section V. Finally, we conclude the paper in Section VI.
II.

E NERGY E FFICIENCY

The direct approach to the development of green data centers
is to curb their energy use by employing energy efficiency
techniques. The fundamental concept is to exploit power management technology in the Information and Communication
Technology (ICT) devices of data centers. In this regard, the
research community has proposed a number of possible approaches in recent years.
A. Dynamic Speed Scaling (SS)
This scheme, which is also known as Dynamic Voltage
and Frequency Scaling (DVFS), focuses on lowering the frequency/speed of devices to save energy. Dynamic speed scaling allows for power savings as power consumption is approximately proportional to the supply voltage or the device
speed s (e.g., following the cube-root rule, s3 , or a more general
function f (s, α), where α is a constant power parameter) [17],
[18], [19]. The goal is to determine the processing speeds and
job assignment to minimize the total energy cost and guarantee
the prescribed performance constraints. Intensive research, initiated by Yao et al. [17], has been conducted in pursuit of saving
energy by speed scaling. Some efforts address the problem
of scheduling via job deadlines [20] and the optimization of
job flow times [21]. Other approaches consider the problem
of makespan minimization [22], incorporate precedence constraints defined between jobs [23] and account for affinities
between jobs and processors [24]. The research on variablespeed architectures includes single processors [17], [25], parallel
processors [26], [27], and network devices [28], [29], [35] with
preemption or nonpreemption settings [30], [31]. Some studies
also have addressed realistic speed models in which only a finite set of discrete speed levels is available [32], [33] or in which
the speeds are bounded on an interval [34]. In addition, Bampis

et al. [35] studied a heterogeneous multiprocessor preemptive
problem, in which it was assumed that each processor had
a different speed-to-power function. We can present all these
efforts in a consistent fashion by extending the standard threefield notation of [55]; we summarize the algorithmic results
obtained for dynamic speed scaling in Tables I and II, and the
symbols used are defined in Table III.
When a large-scale warehouse is considered, speed scaling is
adopted to seek solutions for optimizing the energy consumption based on the current load [56], [57], [58]. Higher speeds
allow for faster execution but also result in higher power
consumption. Gandhi et al. [56] studied the problem of finding
the optimal power allocating in terms of determining the
optimal frequencies of the servers in a server farm to minimize
mean response time after measuring the power-to-frequency
relationship of each server for a given workload. The authors
noted that use of the optimal power allocation significantly
improved the response time, by a factor of typically 1.4 and,
in some cases, as much as a factor of 5. In [57], the authors
considered how to balance the mean response time and mean
energy consumption in processor sharing scheduling when dynamic speed scaling is applied to reduce energy consumption.
They identified a scheme that provided results that were nearly
identical to the optimal objective of dynamic speed scaling,
i.e., that simultaneously minimized energy consumption and
response time. Speed scaling has also been studied in the
arbitration of power consumption and system throughput. In
[58], the authors provided a probabilistic framework in which
online decisions were made on request for admission control,
routing, and virtual machine (VM) allocation. They modeled a
unified objective function to balance the power consumption of
the servers and the system throughput for application requests.
At the network level, an Adaptive Link Rate (ALR) can be
applied on links to reduce energy consumption [59], [28], [60],
[61]. Gunaratne et al. [28] proposed a Markovian model to
reduce the energy consumption of a typical Ethernet link by
adaptively varying the link data rate in response to utilization.
The authors found that their scheme allows an Ethernet link
to operate at a lower data rate for more than 80% of the time,
yielding approximately $70 million in energy savings per year

TABLE I: Algorithmic results for speed scaling approaches based on the minimization of energy subject to deadline feasibility constraints.
Speed Mode

Environment

Problem
1|r j , d j , pmtn | E

Algorithm Type
Offline

Complexity

Approximation/Competitive Ratio

O ( n3 ) [17], O ( n2 log n ) [20]

-α α α
αα [17], 2( α−
1 ) e [25]

Online

Single processor

Continuous
speed scaling

Multiple
homogeneous
processors

1| agreeable, pmtn | E
1|r j , d j , w j = 1, non-pmtn | E
1| agreeable, non-pmtn | E
1|laminar, non-pmtn | E
1|r j , d j , non-pmtn | E

Offline

P | agreeable, w j = 1, pmtn, non-mig| E

Offline

P | agreeable, pmtn, non-mig| E
P |r j = 0, d j = d, w j = 1, M j , pmtn, non-mig| E
P |r j = 0, d j = d, pmtn, non-mig| E
P |r j , d j , w j = 1, pmtn, non-mig| E

P |r j = 0, d j = 1, w i,j , pmtn, non-mig| E

Single processor
Discrete
speed
scaling

Multiple
homogeneous
processors

Offline

P |r j = 0, d j = d, pmtn, mig| E
P |r j , d j , pmtn, mig| E

P |r j = 0, d j , pmtn, non-mig| E
P |r j , d j , pmtn, non-mig| E

Multiple
heterogeneous
processors

Offline

1

O ( n2 ) [37]
Polynomial [38]
Polynomial [17], [30]
NP-hard [30]
NP-hard

O ( n log n ) [40]
O ( n2 f ( n )) [41]
O ( n f ( n ) log K ) [42]
O ( n log n ) [26]

Online

Offline
Online
Offline
Online
Offline

O ( m f ( n ) log n ) [24]
NP-hard [24], [26]
NP-hard ( m ≥2) [26]
NP-hard
NP-hard
APX-hard [39]

P |r j , d j , non-pmtn, non-mig| E

Offline

R|r ij , dij , w ij , pmtn, mig| E
R|r ij , dij , w ij , pmtn, non-mig| E

Offline

Polynomial ( 1ǫ )
NP-hard

Offline

O ( dn log n ) [32]

1|r j , d j , pmtn | E

P |r j = 0, d j = d, M j , pmtn, non −mig| E

NP-hard

Offline

(12(1 + ǫ)) α−1 [39]
----α α α
2( α −
1 ) e [26]
αα 24α [26]
-PTAS [24], [26]
αα 24α [26], B⌈α⌉ [27]
αα 24α [26]
2(2 − m1 ) α [26], B⌈α⌉ [27]
B⌈α⌉ [27]
α α α
2( α −
1 ) e B⌈α ⌉ [27]
1

m α ( n m ) α−1 [31]

( 25 ) α−1 Beα ((1 + ǫ)(1 + wwmax )) α [39]
min
OPT + ǫ [35]
(1 + ǫ) α Beα [35]
-2 α − 1 ( α −1 ) α − 1 ( δ α −1 ) α
( δ −1)( δ α− δ ) α −1

Online
NP-hard [33]

1

4α /(2e 2 α 2 ) [36]
---QPTAS [38]
25α−4 [30]
(1 + wwmax ) α [31]
min
eα [35]
2α −1 ( 1 + ǫ ) α B

+ 1 (δ = maxi sis+i 1 ) [43]

(2∆ ) α−1 ( ∆ = maxi ( s i +1 /s i )) [33]

3

TABLE II: Algorithmic results for speed scaling approaches based on flow time, makespan, and throughput.
Speed Mode

Environment

Problem
1|r j , w j = 1, non-pmtn, E | F
1|r j , non-pmtn, E | F
1|r j , w j = 1, non-pmtn | E + F

Single processor

Continuous
speed scaling

Single processor
(speed bounded)

1|r j , w j = 1, pmtn | E + F
1|r j , pmtn | E + W F
1|r j , pmtn, E | F
1|r j , pmtn | E + F

1|r j , pmtn/non-pmtn, E |Cmax
1|r j , d j , pmtn, E |U
1|r j , d j , pmtn, E |WU
1|r j , w j = 1, pmtn | E + F
1|r j , pmtn | E + F

Algorithm
Type

Complexity
O ( n2 log L ) [44]

Offline
O ( n3 log K ) [21]
Online

Online

Offline

O ( n log n ) [22]
9 ) [49]
Pseudo-poly, O ( n6 T 9 Vmax
2
4
9
9 ) [49]
Pseudo-poly, O ( n S T Vmax

Online

Multiple
homogeneous
processors

Multiple
heterogeneous
processors
Discrete
speed scaling

Single processor

P = 2|r j , d j , pmtn, mig|U, E
P |r j , pmtn, mig| E + F
P |r j , pmtn, non-mig| E + F
P |r j , w j = 1, pmtn/non-pmtn, non-mig, E | F
P |r j , pmtn, non-mig| E + F
P |r j , w j = 1, pmtn/non-pmtn, non-mig, E |Cmax
P |r j = 0, non-pmtn, non-mig, E |Cmax

P |r j = 0, non-pmtn, non-mig, prec, E |Cmax
P |r j , d j , w j = 1, m, non-pmtn, non-mig, E |WU
P | agreeable, w j , m, non-pmtn, non-mig, E |WU
R|r j , pmtn, non-mig| E + F
R|r j , Pi ( s ) = s αi , pmtn, non-mig| E + W F
R|r j , Pi ( s ) = s αi , pmtn, non-mig| E + F
R|r ij , dij , w ij , Pi ( s ) = s αi , pmtn, non-mig, E |WU
1|r j , pmtn/non-pmtn, E |Cmax
1|r j , d j , pmtn |U, E

-(1 + ǫ) α -energy O ( ǫ1 )-approximate [44]
-√
8.3e ((3 + 5) /2) α [21]
4 [45]
O ( α2 / ln2 α) [45]
not O (1)-competitive [45]
O ( lnαα ) [46]
3 + ǫ [47]
2 + ǫ [48]
---4 [34]
α −1
2( α + 1) / ( α −
1/ ( α −1) ) [46]
( α +1 )

14-competitive ( αα + α2 4α )-competitive [50]
4-competitive ( αα + α2 4α )-competitive [34]
3-approximate O (1)-approximate [43]
3-competitive O (1)-competitive [51]
O (1) [52]
O (1) [52]
arbitrarily good [22]
(3 + ǫ) B⌈α⌉ [27]
arbitrarily good [22]

1|r j , d j , pmtn, |U, E

Offline
Multiple
homogeneous
processors
(speed bounded)

Approximation/Competitive
Ratio

Online
Offline
Online
NP-hard [22]
Offline

NP-hard
pseudo, O ( n12m +7 S2 ) [53]
O ( n2m +2V 2m +1 Sm ) [53]

O ( 1ǫ )-competitive (1 + ǫ )-speedup [54]
O ( α2 ) ( α = maxi αi ) [54]
O ( α) ( α = maxi αi ) [54]
2( Γ + 1)(1 + ǫ ) ( Γ = maxi αi ) [53]

Online
Offline
Offline
Online

2

O (log1+ α m ) [23]

O ( n log n ) [22]

-14-competitive ( ∆ α ( αα + αα 4α ) + 1)-competitive [50]

4

5

TABLE III: Definitions of symbols for dynamic speed scaling
Symbol

Definition

1
P
R
m
n
d

Single process
Homogeneous parallel (multiple) processors
Heterogeneous parallel (multiple) processors
Number of processors
Number of jobs
Number of available discrete speeds

e
α
ǫ
B⌈ α ⌉
eα
B

A mathematical constant, e ≃2.71828
A constant power parameter
An arbitrary parameter
⌈ α⌉-th Bell number
Generalized version of the Bell number Ω(( e lnα α ) α )
The largest ratio of two consecutive (non-zero)
speed levels

∆

f (n)

Time complexity of computing a maximum flow
in a layered graph with n vertices

K

The range of all possible values of speeds divided
by the desired accuracy

L

The range of all possible values of power divided
by the desired accuracy

T
Vmax
V
S

The
The
The
The

span length max j,j ′ ( d j − r j ′ )
maximum processing volume
sum of processing volumes Σ j w j
sum of weights

rj
dj
wj
w max ( w min )

Job j’s release date
Job j’s deadline
Job j’s processing volume
Maximum (Minimum) processing volume of all
jobs
r ij
Job j’s release date on processor i
Job j’s deadline on processor i
dij
w ij
Job j’s processing volume on processor i
Mj
Job j’s eligible processor set
Energy consumption
E
(W ) F
(Weight) Flow time
Makespan
Cmax
(W )U
(Weight) Throughput
pre
Precedence constraints defined between jobs
( non−) pmtn (Not) Allow jobs to be interrupted

( non−) mig

(Not) Allow jobs to be interrupted and resumed
on the same or another processor

agreeable
laminar

For any two jobs j and j′ , if r j ≤r j ′ , then d j ≤ d j ′
If r j ≤r j ′ , then d j ≥d j ′ or d j ≤r j ′

in the U.S. with only a very small increase in packet delay. Abts
et al. [60] proposed a power-optimizing mechanism for links
with the capabilities of a network with the flattened butterfly
topology. The authors scaled down the network links with a
data rate proportional to the traffic intensity while accepting a
trade-off of additional mean latency. They reported an energy
savings of approximately 42%. Wang et al. [61] presented a
rate-adaptation-based solution with the intent of approaching
network-wide energy proportionality via routing optimization.
The simulation results indicated that the scheme could achieve
up to 40% energy savings while introducing a very slight
increase in network delay.

B. Power-Down Mechanism (PDM)
In this scheme, devices are transitioned into a low-power
standby or sleep mode while they are idle to save energy. One
must determine when idle periods exist that are of sufficient
duration to outweigh the costs incurred by the scheme (such
as the cost for transitioning from the sleep state and the delay
cost) and decide when to wake the devices from the powersaving mode to satisfy the workload demand. When a single
device is considered and the transition delay is ignored, the
problem of managing power in a two-state situation (active
state and sleep state) is equivalent to the ski-rental problem.
A competition analysis of this problem was presented by [62],
[63]. Irani et al. [64] extended the study to devices with multiple states. The authors presented a deterministic algorithm
that achieved 2-competitive performance when the transition
energies were assumed to be additive. In [65], the authors
investigated the generalized case in which the state transition
energies could take
√ arbitrary values and achieved a competitiveness of 3 + 2 2. With preemptions allowed and the process
of waking up assumed to require a certain fixed amount of
energy, Baptiste et al. [67] provided an O(n5 )-time algorithm
for offline operation. For the case in which the tasks have
agreeable deadlines, i.e., later-released tasks also have later
deadlines, [68] and [69] proposed algorithms to improve the
time complexity. Given a single processor, tasks with arbitrary
processing time and unit transition costs, [68] proposed an
O(n log n ) greedy algorithm. Later, Angel et al. [69] proposed
an O(n2 ) algorithm for tasks with arbitrary processing time and
arbitrary transition costs in a single-processor environment.
For tasks with unit processing time and unit transition cost in
the multiprocessor case, the authors of [69] also provided an
O(n2 m) algorithm. In [70], Demaine et al. considered the powersaving problem in which each task must be executed within
a specified set of time intervals and proposed a (1 + 32 L c )approximation algorithm, where L c is the transition cost. In
[71], the authors studied the problem of online dynamic power
management in a three-state situation (busy state, standby state
and off state). They first considered the case in which the set of
tasks that could feasibly be scheduled on a single processor was
known in advance (packable) and, for this case, proposed a 4competitive online algorithm that used at most two processors.
The competitive ratio was improved to 3.59 for tasks with
unit processing time. For the general case of tasks with unit
processing time and multiple processors, an O(1)-competitive
algorithm was provided. In Table IV, we summarize all the
algorithmic results mentioned above regarding power-down
mechanisms; the definitions of the symbols are the same as
those given in Table III.
For large hosting centers, several studies recommend energyproportional computing [72], [73], [74], [75], [76], [77], [78], [79].
Here, “proportional” refers to the concept that the size of the
data center should be proportional to its workload demand
[80]. In [72], the authors proposed energy-conscious provisioning to concentrate the request load on a minimal active set
of servers appropriate to the current aggregate load level. The
excess servers would be transitioned into a power-saving state
to reduce the energy cost during periods of light load. These
authors reported that their system could reduce server energy
usage by 29% or more for a typical Web workload. Lin et al.
[73] investigated the energy-saving problem by dynamically
“right-sizing” the target data center by turning off servers
during lighter periods in both offline and online cases. They

6

TABLE IV: Algorithmic results for power-down mechanisms. The definitions of the symbols are the same as those given in Table III, and the
term packable means that the set of tasks that can feasibly be scheduled on a single processor is known in advance.
Environment

Task Model

Algorithm Type

rj , wj

Two states (active state and sleep
state), transition cost Lc is arbitrary

Online

--

rj , wj

Multiple states ({s0 , ..., s k }), transition
cost Lc is additive

Online

--

2 [64]

rj , wj

Multiple states ({s0 , ..., s k }), transition
cost Lc is arbitrary

Online

--

√
3 + 2 2 [65]

rj , dj , wj = 1

Two states (active state and sleep
state), transition cost Lc is arbitrary

Offline

O ( n7 ) [66],
O ( n4 ) [67]

--

r j , d j , w j , pmtn

Two states (active state and sleep
state), transition cost Lc is arbitrary

Offline

O ( n5 ) [67]

--

agreeable, w j ,
pmtn

Two states (active state and sleep
state), transition cost Lc is unit

Offline

O ( n log n) [68]

--

Single processor

Multiple
homogeneous
processors

Multiple
homogeneous
processors,
m=2
Multiple
homogeneous
processors

Complexity

Approximation/Competitive
Ratio

Transition Cost

2 [62], e/ ( e − 1) [63]

3

agreeable, w j = 1

Two states (active state and sleep
state), transition cost Lc is arbitrary

Offline

O ( n ) [68],
O ( n2 ) [69]

--

agreeable, w j ,
pmtn

Two states (active state and sleep
state), transition cost Lc is arbitrary

Offline

O ( n2 ) [69]

--

agreeable, w j = 1

Two states (active state and sleep
state), transition cost Lc is unit

Offline

O ( m2 n3 ) [68],
O ( mn2 ) [69]

--

r j , d j , w j = 1,
r j ∈N, d j ∈N

Two states (active state and sleep
state), transition cost Lc is arbitrary

Offline

O ( m5 n7 ) [70]

--

Execute in a specified
set of time intervals,
wj = 1

Two states (active state and sleep
state), transition cost Lc is arbitrary

Offline

NP-hard [70]

1 + 23 Lc [70]

r j , d j , w j = 1,
pmtn, non − mig,
packable

Three states (busy, standby and off),
transition cost Lc is arbitrary

Online

--

3.59 [71]

r j , d j , w j , pmtn,
non − mig,
packable

Three states (busy, standby and off),
transition cost Lc is arbitrary

Online

--

4 [71]

r j , d j , w j = 1,
pmtn, non − mig

Three states (busy, standby and off),
transition cost Lc is arbitrary

Online

--

modeled the cost of toggling a server back and forth between
the active and power-saving modes as a switching cost. The
authors reported a 50% energy savings for a peak-to-mean ratio
(PMR) of 5. Azar et al. [74] considered turning off machines to
save on computation costs. The authors accounted for the delay
cost as the duration required to turn the machines on or off.
They proposed a bi-objective algorithm to address the problem
of minimizing both the maximum task delay and the total cost.
On the network-device level, power-down mechanisms redirect network traffic to aggregate traffic on only a few network
devices and links while allowing idle devices to sleep for energy conservation [81]. In [75], the authors investigated poweraware network design and routing to dynamically power
on/off line cards and chassis. They modeled the minimization
of network-wide power consumption as a mixed-integer optimization problem and solved it using a branch-and-cut solver
that incorporated several practical considerations (such as the
network structure). They demonstrated that their results were
within 11% of the optimal result. Bolla et al. [76] used energyaware traffic engineering on backbone networks to manage the
standby and wakeup behavior of network devices with respect
to the IP layer. The basic concept driving their work was to
periodically reconfigure the network nodes and links to accommodate incoming traffic volumes and operational constraints
(e.g., quality of service and reconvergence times). The authors
noted that the energy savings depended on the level of traffic

O (1)-competitive [71]

load; specifically, the energy savings was approximately 40%
when the traffic load was less than 50%. Andrews et al. [77]
studied routing and scheduling in a power-down model and
captured the trade-off between energy savings and provisioning performance guarantees such as end-to-end delays. The
authors proposed scheduling algorithms for line and arbitrary
topologies. In the case of periodic scheduling, the schedule determined the active period per element within each frame and
prioritized the packets within each active period. Zhang et al.
[78] proposed GreenTE, an intra-domain energy-aware traffic
engineering mechanism that maximizes the number of links
that can be placed in sleep mode while satisfying performance
constraints (e.g., link utilization and packet delay). The authors
reported an energy savings of 27-42% for a maximum link
utilization below 50%. ElasticTree [79] dynamically adjusts the
set of active network elements to achieve a trade-off between
energy efficiency and performance. The authors used methods
including a formal model, a greedy bin-packer, a topologyaware heuristic and prediction to determine which subset of
links and switches to use. The authors reported a network
energy savings of 50%.
C. Hybrid Technology
Various studies of hybrid technology have also been reported
that have explored both speed scaling and power-down mechanisms to fully exploit the potential for energy savings. In

7

a standard speed scaling setting, there is a tendency to use
low speed levels, subject to prescribed deadline constraints.
By contrast, in combination with the power-down approach,
it can be beneficial to speed up the processing of tasks to
generate idle intervals in which devices can be transitioned
to the power-saving mode. Irani et al. [82] performed the first
theoretical analysis of reducing energy usage by combining
placing the system in a sleep state if it is idle with varying
the speed at which tasks are processed. Assuming that all
tasks could be preempted and resumed at no cost, the authors
proposed a 2-approximation offline algorithm and a constantcompetitive-ratio online algorithm. Several subsequent studies
presented algorithms that improved on these result [83], [84],
[85], [86]. In [83], the authors considered online scheduling of a
processor to manage its energy consumption using both speed
scaling and a sleep state. They proposed an algorithm called
SOA to improve the competitive ratio for energy minimization
from 22α −2 αα + 2α −1 + 2 [82] to max {4, αα + 2}, under the
assumption that the speed could be arbitrarily high. They also
showed that when the maximum speed of the processor was
bounded, their algorithm was 4-competitive for throughput
and (αα + α2 4α + 2)-competitive for energy. In [84], the authors
presented an algorithm called SqOA to further improve the
competitive ratio of SOA [83] in the unbounded-speed setting
to max {4, (2 − 1/α)α 2α −1 + 2}. The authors of [85] designed
superior approximation algorithms for the offline case. They
proposed a 4/3-approximation algorithm for general convex
power and a 137/117-approximation algorithm for power functions of the form P (s) = βsα + γ. Recently, Antoniadis et al. [86]
closed the gap in the approximation factor by presenting a fully
polynomial-time approximation scheme for the deadline-based
preemptive offline scheduling problem. In [87], the authors
extended the study of this strategy to minimize the sum of
energy and flow on a processor with one or multiple levels
of sleep states. They designed O(1)-competitive clairvoyant
and non-clairvoyant algorithms in the unlimited-speed setting
(note: the terms “clairvoyant” and “non-clairvoyant” indicate
that the sizes of the tasks are either known or not known,
respectively, when the tasks arrive). Under a maximum-speed
constraint, they enhanced their algorithm to again achieve
O(1)-competitive performance.
Chen et al. [88] proposed a framework to allow the employment of both approaches to enhance energy savings in
hosting centers. Their solutions were based on queuing models
and control theory and did not require compromise in enduser Service Level Agreements (SLAs). In [89], the authors
considered reducing the power consumed in geographically
distributed data centers. They proposed an approach that incorporated selecting the number of active servers and their service
rates to minimize the time-averaged power cost incurred when
handling delay-tolerant workloads. Liu et al. [90] developed
SleepScale, a power management tool to manage data centers
for power efficiency while fulfilling quality-of-service (QoS)
agreements. They exploited the low-power states and operating
frequency that are built into modern CPUs and proposed a
joint optimization of speed scaling and sleep-state selection to
reduce energy consumption. [91], [92] and [93] presented studies of network elements. Nedevschi et al. [91] explored the use
of hybrid technology to reduce network energy consumption
by putting network components to sleep during idle periods
and adapting the rate of network operation to the requested
workload. They showed that the effect depended on the power
profile of the network equipment and the utilization of the net-

work itself. Using real-world traffic workloads, topologies and
power constants, they reported an energy savings of 50% for
a network utilization of 10-20%, accompanied by a very slight
increase in latency. The developers of EATe [92] formulated a
method of online traffic management to spread the load across
multiple paths while minimizing energy consumption. Their
approach leveraged the rate adaptation and power-down of
links and routers to configure an energy-proportional network
hardware structure. Extensive ns-2 simulations demonstrated
that EATe could place 21% of the links in the sleep state, lower
the link energy levels to achieve 8% energy savings and put
16% of the active routers to sleep. Wang et al. [93] studied
the problem of flow scheduling and routing while meeting
flow deadlines in data center networks. They combined speed
scaling and power-down mechanisms to determine routing
paths and schedules as well as the transmission rate for each
flow.
D. Challenges and Research Issues
The studies discussed above demonstrate that energyefficient mechanisms can assist in making data centers green
and reducing energy usage. Furthermore, they reveal that there
is a trade-off between performance and energy consumption.
The aforementioned efforts also have demonstrated that a considerable amount of the energy consumed as a result of overprovisioning can be saved while still satisfying performance
constraints. However, there are still several other issues to be
addressed concerning the design of energy-efficient algorithms:
• Most studies that have been conducted on energy efficiency thus far have focused on ideal power models.
However, future investigations should consider scenarios
that more closely resemble real systems. For example,
some overhead is incurred when switching speeds because the processor must stop while the voltage is changing, and frequent changes in speed can also harm a processor’s lifetime reliability [94]. An interesting question
is how to incorporate these aspects of real systems into
performance studies.
• In classical parallel scheduling problems, the number
of servers is fixed, and the scheduler primarily decides
which task to process on which processor and determines
the speed of each processor at a given time. However,
data centers operate on a different model in which servers
can be activated and released on demand [95] and in
which it is necessary to exploit an appropriate number
of servers to process the submitted tasks. Thus, the characteristics of data centers pose new challenges and open
problems, of which the key underlying requirements are
scalability, uncertainty and efficiency.
• As mentioned earlier, the latency time and power consumption incurred in the process of rebooting means
that the effects of turning servers or switches off and on
cannot be overlooked. Moreover, these effects are more
serious at the network level because communication data
can be lost after a certain threshold latency time [96].
Fortunately, data center network architectures typically
provide multiple paths for communication among their
servers. The question of how to efficiently manage energy
consumption in data center networks is still a major concern that is worthy of investigation. The well-structured
topology of data center networks can be investigated to
address this issue. A good mechanism design should

8

minimize the effects on the network throughput, latency,
congestion and packet loss while incorporating the structure of the data center network architecture.
• Current results for multiple processors (servers) mostly
focus on environments that consist of homogeneous collections of identical processors (servers). However, many
computer architects believe that architectures based on
heterogeneous processors will dominate future architectural design [97], [98]. The primary advantage of heterogeneous architectures is that the design can include
certain processors that are specialized for particular types
of tasks, with the intent of assigning tasks to those
processors that are best suited for them [99]. It is also
natural to consider the possibility of heterogeneity in data
centers, as the various servers contained in data centers
often differ from each other in their power-performance
relationships. At this time, few results for heterogeneous
environments are extant in energy-efficient algorithm design.
We believe that addressing the above challenges and research
issues will lead to significant advances in the current level of
energy efficiency of data centers.
III.

D ATA C ENTER R ESOURCE M ANAGEMENT

Whereas the previous section discussed schemes to increase
the energy efficiency of computing and communication devices, in this section, we present prominent research works
that exploit virtualization and cloud computing to efficiently
utilize data center resources. The concept of virtualization is
in contrast to the traditional architecture, which leads to low
utilization rates and, in turn, wastes resources because servers
continue to run although they are not processing applications to
their full potential. Through virtualization, the power of servers
can be multiplexed to address many applications, and with
the help of virtualization, more than one application can be
assigned to one server. Thus, this technique enables significant
and cost-effective gains in resource utilization and associated
energy savings. A second innovation that can greatly increase
utilization rates is the development and maturation of cloud
computing. The shift to the cloud encourages economies of
scale because servers can be run on virtual platforms, which
offer higher utilization rates, and these large cloud providers
are highly incentivized to optimize the utilization of data centers and reduce energy consumption to maximize productivity.
In this section, we analyze methods that adopt these two
approaches for virtual machine assignment, network traffic
engineering, power distribution and renewable energy access
with a view to providing better energy management for green
data centers, and we consider the corresponding directions for
future research.
A. Energy-Aware Virtual Machine Assignment
Underutilization in data centers is a major cause of excessive
energy consumption. Moreover, modern servers are sufficiently
powerful to use virtualization to present the illusion of many
smaller virtual machines (VMs), each running independent
applications [100]. Therefore, optimal virtual machine management (including allocation, consolidation, and migration)
is an important technique for data centers that can facilitate
the improvement of resource utilization and the reduction of
energy consumption. The general idea is to efficiently map
VMs to servers based on resource matching and improved

performance metrics. An early study [101] extended virtualization solutions to support rich and effective policies for active
power management, which had not been done before. The
authors integrated “hard” and “soft” power states to achieve
high power savings and showed that substantial benefits could
be derived from the coordination of online methods for server
consolidation using their proposed management techniques. In
[102], Raghavendra et al. proposed a power management solution that coordinated various individual approaches, including
a virtual machine controller. Their design used techniques and
actuators to optimize power at multiple system levels across
both hardware and software. Hermenier et al. [103] proposed
Entropy, a consolidation manager that performed dynamic
consolidation of VMs based on constraint programming and
accounted for migration overhead. The authors showed that
their approach could significantly reduce the number of nodes
and the migration time compared with results obtained using
the FFD heuristic. They reported that the consumption of nodes
per hour could be reduced by 25%.
When exploiting the ability to consolidate several virtual
machines on the same physical server, a key challenge is to explicitly account for the specified QoS in the optimization problem [104], [105]. Van et al. [106] defined the condition of SLA
fulfillment and the resource management cost as a global utility
function. The authors adopted the Constraint Programming
approach to formulate and solve the considered problem. Kusic
et al. [107] considered the problem of consolidating services into
a smaller number of computing resources to achieve higher
server utilization and energy efficiency while maintaining the
desired QoS. The authors implemented a dynamic resource
provisioning framework to treat this problem as a problem of
sequential optimization and solve it using a lookahead control
scheme. They reported 22% power savings compared with a
system without dynamic control. Several other efforts have
also been directed toward the online assignment of virtual
machines to fewer servers. Beloglazov et al. [108] proposed
novel adaptive heuristics for the dynamic consolidation of VMs
based on an analysis of historical data related to VM resource
usage. They applied a modification of the Best Fit Decreasing
algorithm to perform the VM assignment, and they achieved
significant reduction of energy consumption while ensuring a
high level of adherence to the SLA. In [109], the authors considered the problem of consolidating virtual machines when the
network bandwidth demands of the VMs are dynamic. They
converted the problem into a Stochastic Bin Packing problem
and proposed an online packing algorithm.
√ They reported that
the performance ratio was within (1 + ǫ)( 2 + 1) for any ǫ > 0
and that the number of required servers was reduced by 30%
compared with the results of the HARMONIC algorithm [110].
In [111], policies that minimized the total number of occupied
servers in the system were studied. The authors presented
Greedy with sublinear Safety Stocks (GSS) and Modified GSS
(GSS-M) policies. Generally speaking, these GSS and GSS-M
policies scheduled incoming virtual machines in a manner
that greedily minimized the total number of occupied physical
servers. The authors demonstrated the asymptotic optimality
and convergence rate of their policies. However, such beneficial
consolidation also introduces an SLA violation risk. In [112], the
authors presented a risk management framework to address the
problem of consolidating virtual machines for energy efficiency
in data centers. They suggested a two-step approach to virtual
machine scheduling: 1) they solved the problem of resource
allocation for virtual machines using a stochastic program, and

9

2) they proposed an algorithm for dynamic virtual machine
consolidation at run time. Using real-world workloads, they
showed that SLA violations were reduced by a factor of four,
from 25% to 2-5%, compared with other virtual machine consolidation algorithms that do not consider the SLA violation
risk.
Table V summarizes the various approaches to energy-aware
virtual machine assignment and presents a comparison of
the common features identified among the different proposals
mentioned above. The features selected for comparison are as
follows: 1) the resources considered, 2) the techniques used
in the proposals, 3) the traces used for the experiments, 4)
the performance metrics considered, and 5) the results of the
proposals.
B. Data Center Network Traffic Engineering
Traffic engineering is a highly effective approach to making
data centers green based on different traffic patterns and network architectures [113]. Several solutions have been proposed
for achieving energy proportionality by applying traffic aggregation and virtual machine assignment techniques. In [114],
the authors demonstrated that the minimization of network
power consumption for any general data center topology is
NP-complete. They proposed a centralized network power controller program that gathered traffic data and server statistics
from all switches and servers and used this information to
perform traffic aggregation and virtual machine assignment
and migration in the target data center. They demonstrated the
optimal placement for the virtual machines and the bandwidth
savings through trace evaluation. In [115], the authors developed REsPoNse, a framework based on identifying and using
energy-critical paths to redirect traffic to allow large portions of
the network to enter a low-power state. They demonstrated that
energy savings were achieved without frequent recomputation
and deployment of new routing tables, and they showed
that REsPoNse had marginal impact on the application-level
throughput and latency. Their results yielded energy savings
of approximately 30% and 42%. Jiang et al. [116] considered
a joint design incorporating virtual machine assignment and
routing for data centers. They considered the optimization
of both energy cost and network congestion, then proposed
an offline algorithm based on a Markov chain model and
an incremental online solution for a dynamic environment.
The authors reported performance analysis results for various topologies, including clique, fat-tree, HyperX and BCube
topologies, under a spectrum of elephant and mice workloads.
Given that the bandwidth demands of different flows do not
peak at exactly the same time, the authors [117] proposed a
correlation-aware power optimization scheme that dynamically
consolidated traffic flows onto a small set of switches and links
and then shut down unused network devices. They designed a
heuristic algorithm to determine the consolidation and rate configuration, which was fundamentally based on a bin-packing
algorithm. They reported an energy savings of 46% in network
energy for a data center network with only a negligible increase
in delay. To exploit the well-structured topology of data centers,
Zhang et al. [118] adopted a hierarchical perspective to study
the task of optimizing power usage in data center networks
while guaranteeing connectivity and maximum link utilization.
They divided their discussion into two levels: 1) the core level,
for the determination of the core switches that must be powered
to carry outgoing traffic and the aggregation switches to serve

the out-pod traffic in each pod, and 2) the pod level, for
the determination of the aggregation switches that must be
active to carry intra-pod traffic. They designed a hierarchical
energy optimization algorithm and evaluated it on various
traffic patterns, including large traffic flow only, small traffic
flow only, and random traffic patterns. In [119], the authors
adopted the routing perspective to consider reducing the power
consumption of high-density data center networks while satisfying network performance requirements. They proposed a
routing algorithm that used as little network power as possible
to provide routing services, and they reported a 39% energy
savings for a link load of 20% and a reliability requirement of
2 (i.e., the threshold for the ratio of the network throughput
that the data center operator could tolerate after switch and
link elimination to the basic throughput achievable using all
switches was 2). In [120], the authors proposed a new framework to embrace the new opportunities offered by combining
certain unique features of data centers with traffic engineering.
They solved the problem in two steps: first, by allocating
virtual machines to servers to reduce the amount of traffic and
to generate favorable conditions for traffic engineering, and
then, by reducing the number of active switches and balancing
traffic flows to reduce energy consumption. The experimental
results indicated an energy savings of up to 50%. In addition to
considering the network level, the authors of [121] considered
a joint host-network energy optimization scheme involving the
simultaneous optimization of virtual machine assignment and
network flow routing. They modeled the joint optimization
problem as an integer linear program. Because this problem is
NP-complete, they proposed a series of techniques to address
the associated challenges: 1) they converted the problem to
one that considered a single optimization solution, 2) they
proposed a parallelization approach for rapid completion, and
3) they quickly identified efficient routing paths for the flows.
The authors reported that their optimization method reduced
energy consumption by 40% for a system load of 30%. By
utilizing correlation analysis in the joint power optimization
of a data center network and servers, the authors of [122]
proposed PowerNetS to achieve improved energy savings.
Their approach featured a well-designed heuristic algorithm to
incrementally perform virtual machine and traffic consolidation
with lower virtual machine migration overheads, thereby optimizing power consumption while guaranteeing the fulfillment
of network constraint. They reported that their method could
achieve 51.6% energy savings for a data center.
In Table VI, we summarize and compare the proposals mentioned above. The features presented for comparison are 1) the
traffic patterns, 2) the network architectures, 3) the techniques
used in the proposals, 4) the considered QoS specifications, and
5) the results of the proposals.
C. Power Distribution
Whereas virtual machine consolidation allows the total number of active physical servers to be reduced, power distribution
approaches have also been proposed to reduce the high energy
costs of idle systems. Driven by increasing power density, the
increasing number of IT devices, and the capability of ondemand addition and removal of IT devices, there have been
dramatic changes in how power is utilized in data centers
[123]. In general, power distribution is over-provisioned in data
centers to accommodate peaks and to allow for future expansion. However, based on the assumption that simultaneous

10

TABLE V: Comparison of the proposals regarding virtual machine assignment
Proposal

VM Resources

VirtualPower [101]

Cores on chip

No “Power”
Struggles [102]

CPU

Entropy [103]

Processing unit,
memory

Resource
management [106]

Techniques
VM consolidation, DVFS,
resource (time slice)
scheduling, server
power-down
VM consolidation, DVFS,
server power-down

Traces

Performance

Result

RUBiS tiered web service,
transactional load

Processing rate,
throughput

34% reduction in
power

Trace from enterprise
deployment

Throughput, power
budget

64% reduction in
power

VM consolidation

NASGrid benchmark
suite

Migration time

25% reduction in
nodes per hour

CPU, memory

VM provisioning, VM
packing

Multiplayer online game,
web application

Response time,
throughput

Trade-off between
SLA and energy

Power & SLA
management [107]

CPU

VM consolidation, DVFS,
server power-down

Web workload, Trade6
enterprise application

Response time

22% reduction in
power

Consolidation of
VM [108]

CPU

Dynamic consolidation of
VMs

Trace from PlanetLab

Capacity violation,
VM migration

Consolidation VMs
[109]

Bandwidth

VM consolidation

Trace from IBM Global
service and cluster

Capacity violation

Minimize occupied
servers [111]

CPU, memory

VM placement with
packing constraints

--

--

Risk management
for VMs [112]

CPU, memory

Resource allocation, VM
consolidation

Trace from CoMon project

Capacity violation

Significant
reduction in
energy
30% reduction in
required number
of servers
Reduced number
of servers
Factor of 4
reduction in SLA
violations

TABLE VI: Comparison of the proposals regarding network traffic engineering
Proposal

Network Architectures

Techniques

QoS

Result

E-commerce trace

Fat tree

Traffic aggregation, VM
placement

Bandwidth, connectivity
guarantees

55% network
power reduction

REsPoNse [115]

Media stream, web
workload

Fat tree

Energy critical-path
detection

Throughput, latency

40% network
energy savings

Joint controller
[116]

Parallel workload

Fat tree, clique, HyperX,
BCube

Route selection, VM
placement

Congestion

Data center cost
improvement

CARPO [117]

Wikipedia trace

Fat tree

Traffic consolidation,
link-rate adaptation

Packet delay

46% network
energy savings

HERO [118]

Large, small and
random traffic

Fat tree

Switching off network
switches

Connectivity, maximum
link utilization

network power
savings

PRP [119]

one-to-one,
one-to-many,
all-to-all

Fat tree, BCube

Switch and link
elimination

Throughput

39% network
energy savings

GreenDCN [120]

Communicationintensive

Fat tree

VM assignment, switch
and link engineering

Connectivity

50% network
energy savings

Joint optimization
[121]

Iperf UDP

Fat tree

VM placement, flow
routing

--

40% energy savings

PowerNetS [122]

Wikipedia, Yahoo!
and IBM traces

Fat tree

VM and flow
consolidation

Packet delay

51.6% energy
savings

Network power
controller [114]

Traffic Patterns

peak draw across all equipment will occur only rarely, power
over-subscription is intentionally exploited to improve power
utilization in modern data centers [9], [124], [126], [127]. Fan
et al. [124] were the first to study power usage at the scale
of data center workloads, and they reported the use of power
modeling for power provisioning. Their work yielded several
key findings: 1) The gap between the maximum power actually
used by large groups of servers and their aggregate theoretical
peak usage can be as large as 40% in data centers, which
suggests a significant opportunity to host additional servers
under the same power budget; 2) power capping is more
useful as a safety mechanism to prevent overload situations;
3) large groups of servers are operating near peak power

levels in certain time intervals, which suggests that power
gaps and power management techniques can be more easily
exploited at the data center level than at the rack level; and 4)
when large groups of servers are considered, frequency scaling
has the potential to be moderately effective at reducing peak
power consumption. Their evaluation results indicated energy
savings of 35-40%. Meisner et al. [125] proposed PowerNap,
an energy-conservation approach that supported an entire
system in transitioning rapidly between a high-performance
active state and a near-zero-power idle state in response to
instantaneous variations in load. Based on the PowerNap concept, the authors developed requirements and mechanisms to
eliminate idle power waste in enterprise blade servers. The

11

authors also introduced another power provisioning approach
named RAILS to be applied when PowerNap was operating
in the low-efficiency regions of current blade center power
supplies with the intent of providing high conversion efficiency
throughout the entire range. They reported that PowerNap and
RAILS could reduce the average server power consumption
by 74%. To overcome local utilization spikes that could force
the throttling of server performance to enforce safe power
budgets, Pelley et al. [126] developed mechanisms to better
utilize the installed power infrastructure. They proposed Power
Routing to centrally schedule servers dynamically across redundant power feeds, and they proved that the problem of
assigning servers to power distribution units (PDUs) was NPcomplete. The authors first optimally solved the relaxation
problem through linear programming, assuming the servers
to be assigned fractionally across feeds, and then constructed
an approximate solution to the original problem. Using traces
from production systems, they demonstrated that capital costs
could be reduced by 32% without performance degradation.
For the case in which servers are shared by virtual machines
belonging to different applications, a power budgeting solution
named VPS was presented by [127] to guarantee the quality
of service. VPS dynamically shifted power among various
distributed components as workloads and power availability
varied to efficiently utilize the power budget. The authors
combined hardware-based and software-based power control
knobs to optimize the performance. They used traces to evaluate the following performance metrics: data-center-level and
application-level power budgeting errors, application performance differentiation, and performance achieved within the
power budget. In [128], the authors explored the feasibility
of low-latency power states for enterprise server systems and
proposed an end-to-end virtualization power management solution. They implemented the capability of a low-power and
low-latency state for virtualization servers and demonstrated
this technology could overcome the traditional barriers (e.g.,
incurred latencies) to achieve substantial energy savings via
dynamic consolidation. Based on real-system evaluations, they
reported that data center power efficiency could be improved
by 30%.
In addition to power capping, several studies have introduced Uninterruptible Power Supplies (UPSs) to store energy
and reduce power budget violations [129], [130], [131]. In
[129], the authors pioneered the leveraging of a UPS in a
data center as an energy buffer (eBuff) to address the peak
power draw problem. They used the energy stored in UPS
batteries to provide energy during peak demand, resulting in a
reduction in capital expenses and operating expenses without
performance degradation. They reported that eBuff yielded
a 15-45% peak reduction, which corresponded to a 6-18%
savings in operation expenses. Because eBuff was centralized
and limited by the capacity of the UPS battery, Kontorinis et
al. [130] presented an architecture for distributed per-server
UPSs that stored energy during low-activity periods and used
this energy during power spikes. They proposed details regarding the sizing as well as technological alternatives and
approaches that managed battery charging and discharging
behavior while addressing reliability and availability concerns.
They demonstrated that their policies prolonged the duration of
UPS batteries’ usage and reduced their total cost of ownership
per server by 6.3%. In [131], the authors studied the possibility
of leveraging stored energy to address power emergencies.
Their work demonstrated that mechanisms for power capping

had performance-degrading ramifications, and they proposed
an offline theoretical framework and several online heuristics
to temporarily augment utility supplies during emergencies.
They reported that battery-based solutions 1) could cope with
emergencies of short duration, 2) supplemented existing reaction strategies to enhance efficacy for longer emergencies,
and 3) provided feasible options when other strategies (power
states, migration, etc.) did not suffice. In the case of aggressive
over-subscription, whereas the aforementioned work suggested
power capping and stored energy to reduce the risk of power
peaks, the authors of [132] suggested that the power hierarchy
should be treated as a data center resource of comparable
importance to that of other computing resources. To this end,
the authors proposed vPower, a software system to virtualize
power distribution. vPower combined computing parameters
(such as DVFS, consolidation, etc.) and energy storage devices
to establish a virtual power hierarchy for each application.
The primary advantages of vPower were that it 1) allowed
applications to specify their power needs, 2) performed admission control and assignment, 3) dynamically monitored power
usage, and 4) enforced the allocation of power for fairness and
system efficiency. The results of a power hierarchy prototype
indicated the achieved improvement in system utilization was
over 50% compared with the conservative scheduling approach
based on peak load requirements.
D. Renewable Energy Access
Expanding the use of renewable energy is another route
toward green data centers. Some data center owners have
already investigated and accepted alternative energies, such
as solar, geothermal and wind energy [133], [134]. However,
utilizing these green energies is challenging because of their
restricted, intermittent and unstable nature. Therefore, the main
question that must be addressed is how to exploit renewable
energy and overcome the associated hindrances by exploiting the unique features of data centers. Sharma et al. [135]
developed Blink to address intermittent power constraints.
Based on workload characteristics and energy constraints, Blink
determined the power state of the servers and minimized
performance degradation. Because the power varied and the
energy stored in the batteries was for short-term use, Blink’s
policy had potential benefits for distributed applications. The
authors constructed a cluster of 10 low-power motherboards
powered by an array of micro wind turbines and solar panels to
evaluate the performance under intermittent power constraints.
They demonstrated that an asymmetric load-proportional policy could increase fairness without significantly sacrificing the
cache’s hit rate to achieve a blinking version of memcached.
Considering the location restrictions of renewable energy, Akoush et al. [136] proposed a computation architecture to avoid
the necessity of expensive infrastructure investment. Their design consisted of three relevant components: 1) data centers collocated with renewable energy sources, 2) data centers linked
together over a dedicated communication network, and 3) a
software framework that supported the seamless execution and
migration of virtual machines depending on power availability.
Although they demonstrated the viability of the design in a
case study, the authors also noted the technical challenges and
inherent limitations of their approach, such as the efficiency of
virtual machine migration between data centers and suitable
workload concerns. In [137], the authors leveraged tunable
energy, such as that provided by fuel cells and gas turbines,

12

to propose a power management approach that enabled data
center operation with high performance and low overhead.
Unlike previous work, in which the workload had been forced
to track the variable power budget, their approach explored
the possibility of achieving a renewable energy supply to
follow the data center power demand. To overcome issues of
performance degradation, they proposed two adaptive load
tuning schemes, named DGR Boost and UPS Boost, based on
mature computer tuning approaches (such as DVFS and UPS
batteries). Evaluations performed on real-world data center
traces and industry data on distributed generation systems
indicated that their technique achieved a 37% improvement in
performance over existing supply-tracking-based designs and
saved 100 metric tons of carbon emissions annually, given that
the power consumption of the data center was 10MW.
If powered only by variable sources of green energy, a
data center will experience unpredictable performance for applications. The most reasonable approach is to incorporate
renewable energy into the electrical grid. In [138] and [139],
the authors considered a data center powered by a solar array
and the electrical grid. Two scheduler frameworks, named
GreenSlot and GreenHadoop, were proposed for parallel batch
jobs and data-processing jobs, respectively. Both schedulers
used historical data and weather forecasts to predict the
amount of solar energy that would be available in the future,
with the intent of maximizing the use of green energy. Because
of the inherent natural variability in the availability of solar
energy, grid energy was used to avoid deadline violations.
The results obtained using a real-world trace demonstrated
that GreenSlot could increase green energy consumption by up
to 117% compared with a conventional backfilling scheduler,
whereas GreenHadoop could increase green energy consumption by up to 31% compared with the Hadoop algorithm.
However, the inefficient and redundant load-matching activities of these algorithms can incur performance losses. iSwith
[140] explored the design trade-offs between energy utilization
and load tuning (e.g., DVFS and CPU power states) for a
data center partially powered by intermittent renewable energy.
iSwith combined optimization of the supply-side fluctuation
with minimization of the load-fluctuation-induced overhead
to mitigate performance loss. This system could reduce job
waiting time by 80% and also mitigate peak network traffic by
95% and average network traffic by 75% while still maintaining
96% renewable energy utilization.
To avoid transmitting renewable energy over long distances,
which leads to significant attenuation losses [136], several
proposed solutions have suggest dispatching traffic to multiple data centers in different geographical locations to use
the renewable energy available nearby [141], [142], [4], [143].
In [141], the authors noted that geographical traffic routing
could significantly reduce the use of brown energy if the price
of energy was dynamically determined in proportion to the
instantaneous fraction of the total energy that was brown.
Moreover, the degree of use of renewable energy was dependent on the form of the pricing model and the acceptability of
this dynamic energy pricing. In [142], the authors exploited
multiple uncorrelated wind energy sources to significantly
reduce the effects of intermittency and to achieve almost entirely green Internet-scale data centers (IDCs). They proposed
WPA, a policy that routed jobs based on the current states
of the workloads and wind power availabilities at different
data centers. The authors showed that more than 95% of the
energy consumption in IDCs could be satisfied by wind power

without delaying the processing of jobs. Similarly, Gao et al. [4]
dynamically controlled the fraction of user traffic directed to
each geographically distributed data center with the objective
of navigating the three-way trade-off among carbon footprint,
access latency and electricity cost. The authors showed that
carbon emissions could be reduced by 10% without increasing
either the mean latency or the electricity cost. However, at
present, renewable energy is more expensive to use than is
brown energy produced with fossil-based fuel. Zhang et al.
[143] considered the problem of maximizing the percentage of
renewable energy used to reduce the negative environmental
implications given certain operation budgets. They proposed a
novel middleware system that dynamically distributed requests
among different data centers. The authors formulated their
objective as a constrained optimization problem and solved it
via linear-fractional programming. The authors experimentally
demonstrated that their system could significantly increase
the use of renewable energy without violating the desired
operational cost budget, despite the intermittent nature of the
supplies of renewable energy at different locations and the time
variation in electricity prices and workload traces.
E. Future Research Directions
To reduce the energy consumption of data centers through
efficient utilization of resources, many proposals have been put
forward for the green implementation data centers. Because of
their exponential growth and underutilization, data centers still
pose several challenges, and several inadequately addressed research directions remain with regard to resource management:
• A major reason why the resource utilization of data
centers remains low is that operators are concerned about
potential QoS violations. Recently, several studies have
suggested the existence of interference between collocated applications [144], [145] when servers are virtualized and shared to improve utilization. The challenge of
virtual machine management and data center right-sizing
is to improve resource utilization while preserving QoS
guarantees, which benefits both data center operators and
users.
• In a multi-tenant data center environment, different
tenants may desire different levels of application performance. This characteristic requires heterogeneous resource management mechanisms, which will introduce
additional complexity and overhead. Moreover, tenants
possess their own resources, which may not be controlled
by the data center operators in a centralized manner. Thus
far, few studies have focused on energy-aware resource
management in multi-tenant data centers, and the task
of providing efficient polices for such scenarios requires
future exploration.
• Although traffic consolidation technologies offer considerable energy savings, they can also have severe effects
on network performance. The virtualization of network
functions is a promising approach to providing efficient
support for the deployment and management of network services. However, current implementations such as
CloudNaaS [146] do not account for energy consumption.
Finding a good balance between energy consumption and
network performance remains a challenging problem for
current data center networks.
• Most existing studies concerning power distribution are
focused on intra-data-center networking. However, geo-

13

•

graphically distributed data centers can also offer opportunities for power distribution. For example, a workload
can be preferentially scheduled to data centers with high
power capping or a greater amount of stored energy
by virtue of being located nearer to power stations. The
question of how to reduce energy consumption by combining power distribution, application performance and
workload scheduling remains an open research problem.
A demand-response mechanism is an energy-efficient
method of overcoming the instability that is characteristic
of renewable energy [141]. In SaaS data centers, such a
mechanism also causes service providers to lose users in
response to supply conditions. A promising solution is to
design beneficial pricing schemes to incentivize service
providers to reduce brown energy consumption.
IV.

T EMPERATURE AND T HERMAL C ONTROL

A large portion of the energy used in computing devices
is converted into heat. In chip design, the heat problem is
extremely important, as it can threaten to cause the chips to
fracture at extreme temperatures [147]. An ambient temperature range of 68◦ F to 75◦ F is optimal for device reliability [148].
Data centers house tens of thousands of servers and communication networks, which causes their cooling costs for heat
removal to increase exponentially with increasing energy consumption. It is reported that more than a third and sometimes
as much as one half of the total cost of a data center consists
of cooling costs [149]. As mentioned earlier, virtualizing and
consolidating servers can increase processor utilization rates.
To reduce management expenses and floor-space requirements,
operators prefer to install many multi-core-processor server
blades in a single rack-mounted enclosure. These trends toward
concentration and higher-density computing in combination
with the abovementioned issues mean that the problem of
temperature and thermal control must play a critical role in
the attempt to establish green data centers.
A. Cooling and Workload Distribution
Optimization of the delivery of cooling is one strategy for
temperature control [150]. Patel et al. [150] proposed a system
to fully control a data center environment by linking all its
attributes together, including distributed sensing, variable air
conditioning and data aggregation. In addition, they exploited
variable opening plenum tiles to assist in achieving more flexible control of the cooling distribution and in directing cooling
resources where and when needed. The authors reported an energy savings of 50%. In [151], the authors proposed TE-Shave,
a generalized framework that exploited thermal energy storage
(TES) tanks to reduce data center power. TE-Shave switched to
the TES for stored cold water or ice to supplement the chillers
and for heat exchange during peak power periods. Moreover,
TE-Shave charged the TES through the preparation of cold
water or ice by increasing the cooling power when the data
center experienced power valleys. The authors demonstrated
that their proposed framework could lead to a 28% savings in
operational expenses compared with existing work that focused
only on reducing the server-side power demand.
Another approach to addressing temperature modulation is
based on workload assignment and migration between servers
to achieve thermal balance [152], [153], [154]. In [153], the
authors presented temperature-aware workload assignment
algorithms to minimize the energy expended by the cooling

infrastructure. Their algorithms produced the recommended
theoretical heat distribution to maximize cooling efficiency and
minimize the amount of heat recirculating within a data center.
The proposed algorithms achieved cooling savings of more
than 25%. Heath et al. [154] developed Freon, a user-space
system to achieve thermal management without introducing
unnecessary performance degradation. The main policy driving
the operation of Freon was periodic temperature monitoring
and the exertion of feedback control over the utilization of
the servers by distributing requests based on load weights.
Generally speaking, Freon dynamically shifted load away from
hot servers and increased the load on other, “cool” servers.
An extended version named Freon-EC was proposed to combine energy conservation and thermal management. Freon-EC
turned off servers based on their temperatures and physical
locations in the computer room. In fact, Freon-EC turned hot
servers off and replaced them with servers from different
regions that were unaffected by the thermal threat. Through
experiments, Freon was proven to manage temperature with as
little potential throughput degradation as possible. In [155], the
authors considered the problem of temperature-aware workload distribution in geo-distributed data centers. They explored
two key aspects: 1) utilizing the geographical diversity of
temperature to reduce cooling energy consumption and 2)
exploiting the elastic nature of batch workloads to dynamically
adjust capacity allocation. The problem was formulated as a
joint optimization of request routing for interactive workloads
and capacity allocation for batch workloads, and it was solved
using a distributed m-block Alternating Direction Method of
Multipliers (ADMM) algorithm. Extensive simulations using
real-world traces revealed that their approach saved 15-20%
in cooling energy and 5-20% in overall energy cost. In addition
to these job-allocation-centric techniques, T ∗ [156] provided a
data-centric approach to reducing cooling energy costs in big
data analytics cloud computing. Files were proactively allocated in a thermal- and energy-aware manner based on knowledge of server-profile, data-semantics, and cluster information.
T ∗ proposed predictive and non-predictive temperature-aware
file allocation algorithms to reduce cooling energy costs and
ensure high data-local performance. Evaluations using onemonth-long real-world traces demonstrated a reduction of up
to 42% in cooling energy costs and a 9× better average response
time compared with techniques that were data-allocationagnostic in nature.
B. Temperature-Reliability Trade-off
Several other efforts have focused on supplying data centers with relatively warm air to reduce cooling costs, i.e.,
increasing/controlling the temperature setpoint. Studies show
that increasing the temperature setpoint by merely one degree
can yield an energy savings of 2-5% [157]. However, raising
the temperature of the server’s components can reduce the
system reliability. Based on the critical observation that hot
spots can be selectively cooled, [158] proposed a cooling-power
management mechanism to allow global data center coolers to
operate at a higher temperature while achieving the same level
of chip reliability. The authors integrated thermal models of
various components, such as the silicon in the chip, embedded
thermo-electric coolers, and air conditioners, and quantified the
expected savings in cooling power for the entire data center.
Upon evaluating over 43 applications from the SPECcup 2000
benchmark, they reported a cooling power savings of 12% on

14

average. However, there is a limited understanding of how
higher temperatures will affect data center systems. In [159],
the authors presented a multi-faceted study of temperature
management in data centers. By analyzing a large collection of
field data from different production environments, the authors
studied the influence of higher temperatures on 1) hardware
reliability, such as storage subsystem reliability, memory subsystem reliability, and server reliability as a whole, as well as 2)
server performance and power consumption. They concluded
that the operators could run their data centers hotter than they
currently were to save energy while still limiting the resultant
negative effects on system reliability and performance.
C. Open Questions and Research Directions
The continued growth of data centers is resulting in sharply
increasing levels of energy consumption required for both
powering these data centers and their thermal management.
Several methodologies have recently been introduced that offer
promising approaches to reducing the escalating energy costs
associated with meeting their cooling requirements. However,
there are still several challenges and research issues that must
be addressed regarding the temperature and thermal control of
data centers:
• Most existing studies have focused on heat dissipation and cooling load management within data centers.
Chillers are used to cool the hot water that returns
from the Computer Room Air Conditioners (CRACs) via
mechanical refrigeration cycles, and the compressors for
these chillers consume large amounts of energy [160].
Recently, various so-called free cooling technologies, such
as air-side economizing and water-side economizing,
have been adopted to reduce the dependence on mechanical chillers. Air-side economizing refers to directing
cold outside air into the data center to cool the servers
and ejecting the hot exhaust air back outside instead
of cooling and recirculating it. Water-side economizing
refers to cooling down the warmer chilled water from the
computer room using water streams. However, these free
cooling technologies heavily depend on air conditions
and geographical locations. An interesting question is
whether and how these technologies can be coupled to
address power and cooling demands. A related study
has already been performed [155], but more thorough
theoretical and practical studies are needed to explore this
possibility.
• As mentioned earlier, service providers may own geographically distributed data centers to better meet their
QoS needs, such as delay guarantees. Many of these
providers, such as Google [161], have been relying upon
fresh air for cooling to reduce the energy costs of operating their data centers. When renewable energy is
considered, the problem becomes more interesting and
sophisticated. For example, for systems that rely on airside economization, it is typically preferable to operate
in a cold-temperature environment. However, the generation of renewable energy using solar panels requires
hot, sunny days, which will adversely affect the cooling
energy consumption. Addressing this issue will require
holistic methods that simultaneously consider cooling
energy costs, the availability of renewable energy, and
performance requirements.
• In the current thermal control approaches implemented in
most data centers, the CRAC fans are operating at their

peak speed. Lowering CRAC fan speeds allows energy
consumption to be reduced. However, the challenge is
to develop CRAC fan speed controllers that can supply
a data center with only the necessary cooling demands.
The existing work of [162] represents an initial effort to
address the problem of fan speed management.

V.

G REEN M ETRICS , M ONITORING AND E XPERIMENTAL
T ECHNIQUES

In this section, we present the metrics relevant to building
a “green” data center and practical efforts to monitor and
test data centers for the optimization of energy consumption
and carbon emissions. The primary goals of developing green
metrics are to 1) quantitatively describe the degree of greenness
of a data center, 2) track and report the energy usage and
carbon emissions of a data center to increase its green efficiency,
and 3) provide a clear visualization of the efficiency of a data
center to help its operators to reduce their electricity bills and
carbon taxes. In the meantime, monitoring efforts provide data
center operators with a real-time, detailed understanding of
operation parameters, including green metrics. As a result,
operators can adopt appropriate green measures based on the
output parameters, for example, the consolidation of virtual
machines when the monitoring data indicate that the data center is experiencing low utilization. Moreover, experimentation
and simulation techniques are also critical for designing test
tools and demonstrating energy-efficient policies.

A. Green Data Center Metrics
Various metrics focused on data center energy efficiency have
arisen over the past years. We summarize the most popular
green metrics below:
1) Power Usage Effectiveness (PUE):
PUE =

Total Facility Power
IT Equipment Power

The IT equipment power is the power consumed by the servers,
network, storage, and supplemental equipment (i.e., workstations/laptops used for monitoring); the total facility power
includes the aforementioned power plus auxiliary consumption
in the data center [163].
2) Data Center infrastructure Efficiency (DCiE):
DCiE =

IT Equipment Power
1
=
PUE
Total Facility Power

This metric is simply the inverse of the PUE [163], and both
were developed by The Green Grid to measure energy efficiency in data centers. Barroso et al. [9] suggested that the
average value of the PUE is 1.83, i.e., the DCiE is 0.54.
3) Data Center energy Productivity (DCeP):
DCeP =

Use f ul Work Produced
Total Energy Consumed to Per f orm that Work

This is a metric to characterize the computing efficiency of a
data center [164]. Useful work refers to the jobs performed by
the IT equipment during the time period. The denominator is
the total energy consumed during the assessment window.

15

Set benchmarking goals

Share results

Prioritize metrics

Create follow-up
action plan

Identify required data and
develop data collection plan

Benchmark metrics and
identify potential actions

Obtain and install
monitoring equipment

Analyze data and
compute metrics

Collect data

Fig. 2: The benchmarking process for data center metrics [165]

4) Heating, Ventilation, and Air Conditioning (HVAC):
HVAC =

IT Electrical Energy
HVAC + ( Fuel + Steam + Chilled)×293

The HVAC system energy is the sum of the electrical energy
used for cooling, fan movement, and any other HVAC energy
usage such as steam or chilled water, and it can be reduced
via free cooling with air economizers and/or high-efficiency
chillers [165]. When the value of this metric is high, it suggests
a high potential to reduce HVAC energy usage.
5) Carbon Usage Effectiveness (CUE):
CUE =

Total CO2 Emissions
IT Equipment Energy

The total carbon emissions include direct greenhouse gas emissions and indirect greenhouse gas emissions from the consumption of purchased electricity, heat, and steam. The denominator
is the energy consumption of the IT equipment [166].
In addition to these metrics, many other metrics have been
proposed by The Green Grid and Lawrence Berkeley National
Laboratory that have different practical uses, such as the CEF
(Carbon Emission Factor), AEU (Air Economizer Utilization),
WEU (Water Economizer Utilization) and CSE (Cooling System
Efficiency) [165], [166]. Table VII presents the taxonomy of the
green metrics listed above. Note that the present values shown
in the table were obtained from [165], [166] if no other reference
is indicated.
Now, we focus on works that have studied how to measure
these metrics. A technical report [165] provided an approach
for obtaining the particular values of the metrics. The specific
implementation process for benchmarking a data center is
illustrated in Fig. 2. The authors of this report also provided
guidance for calculating the required data. For example, electrical energy use can be derived from meter data or utility
bills. In [169], the authors’ intent was to present the “what”
and “how” for the measurement of metrics of energy efficiency.
They attempted to introduce a cheap and effective continuous
measurement method based on the power demand profiles
recorded at various points in a facility. Their power measuring
point system is illustrated in Fig. 3. When the required data
have been obtained from the various registries, the metrics can
be calculated based on the power consumed integrated over 15minute or 1-hour periods. By using standard measuring equipment, the reliability and accuracy of the results can be guaranteed. Another work [170] adopted machine learning techniques
to study the PUE. The authors developed a neural network
framework that learned from actual operations data to predict
the PUE. They reported that the PUE was within 0.004+/-0.005,
with an error of approximately 0.4% for a PUE of 1.1. DCeP, the
metric of energy productivity, was studied in [171]. The authors
presented an approach to calculating the DCeP by designing

an experiment using a highly instrumented, high-performance
computing data center. Their evaluation results demonstrated
that the DCeP could be used to successfully distinguish among
different operational states in the data center and validated
its utility as a metric for identifying the configurations of
hardware and software that could improve energy productivity.
Unfortunately, the improvement of one specific green metric
may lead to the worsening of another metric. Therefore, holistic
frameworks for addressing all metrics have also been proposed
in recent works [172], [173]. These approaches offer a combined
visualization of all metrics to guide data center operators in
worst-metric improvement.

Main Power Input

Cooling
Cooling
System

UPS

UPS
System

Lighting

Lighting

Others

Others

IT Equipment Load

Servers

Network

Storage

Supplemental
Equipment

Fig. 3: Power measuring point registries

B. Green Monitoring and Experimental Technique
Monitoring is an effective method of detecting and proactively mitigating the failure of data centers. Moreover, we
will explore efforts that yield the tracks and traces of energy
usage, thermal emission, and power distribution for individual
devices. Based on the collected information, intelligent mechanisms can then react to increase the energy efficiency and
productivity of a data center. In [174], the authors presented a
comprehensive online monitoring service in their GreenCloud
architecture. The monitoring service monitored and collected
data concerning resource utilization, application workloads directed through virtual machine hypervisors, and power usage
using an intelligent power distribution unit. Analyzing and
mining this collected information provided a clear understanding of the data center energy usage and temperature behavior.
Furthermore, the monitoring service helped to consolidate the

16

TABLE VII: Green metrics taxonomy
Metric Definition
Total Facility Power
I T Equipment Power

PUE =

1
PUE

DCiE =

=

I T Equipment Power
Total Facility Power

DCeP =
Use f ul Work Produced
Total Energy Consumed to Per f orm that Work

HVAC =
CEF =

I T Electrical Energy
HVAC +( Fuel + Steam+ Chilled)×293

Total CO2 Emissions
Total Facility Energy

CUE = CEF× PUE =
CSE =

Total CO2 Emissions
I T Equipment Energy

Average Cooling System Power Usage
Average Cooling Load

Objective

Proposer

Ideal

Present Value

Characterize the total energy efficiency
of a data center

Green Grid, 2008

→1

1.83

Characterize the total energy efficiency
of a data center

Green Grid, 2008

→1

0.54

Characterize the IT computing
efficiency

Green Grid, 2008

Larger

866658( NormalizedTasks
)
KWhr

Characterize the energy efficiency of
the HVAC system

Lawrence Berkeley
National Laboratory, 2009

Higher

--

Assess the carbon emissions per unit of
energy used

Green Grid, 2010

Lower

0.147 (Washington)
[167]

Characterize the overall efficiency of
the cooling system

Green Grid, 2010

Lower

0.27 (Washington)

Represent the carbon emission
efficiency of IT energy use

Lawrence Berkeley
National Laboratory, 2009

Lower

0.6-0.8( KW
Ton )

AEU =

Air Economizer Hours
24 ×365

Measure the percentage of hours in a
year that an air-side economizer system
is used to provide “free cooling”

Lawrence Berkeley
National Laboratory, 2009

→100%

76% (New York)

W EU =

Water Economizer Hours
24 ×365

Measure the percentage of hours in a
year that a water-side economizer
system is used to provide “free
cooling”

Lawrence Berkeley
National Laboratory, 2009

→100%

33% =

virtual machine workload and achieve significant energy savings while guaranteeing the real-time performance required for
sensitive applications. Using a different approach, the authors
of [175] developed PowerTracer, a request-tracing strategy for
diagnosing energy inefficiencies in multi-tier service systems.
They first profiled the patterns of requests through kernel
instrumentation and then measured the server-side latency and
service time of each tier. Finally, PowerTracer collected the
resource consumption information for individual requests and
analyzed the features of each tier and each pattern. By thus
gaining insights into the root causes of energy inefficiency, the
authors could devise efficient power-saving methods for multitier applications.
Even in the initial phases of the design and development of a
data center, an effective experimental and simulation platform
enables the operator to identify and understand the design
aspects that are critical to energy and resource efficiency. EEFSim, a virtualized data center simulator for cloud computing
research, was designed and developed in [176]. The simulator
was demonstrated to be suitable for testing, as it reproduced
the behavior of a real cloud framework and the execution
information that it offered. Another advantage of this simulator
was that it could be easily used to evaluate the power performance of a system by testing different solutions and approaches
within merely a few minutes. As a result, the simulator offered
the possibility to validate different scheduling and migration
policies for their power consumption associated with virtual
machines in a cloud data center. Eliazovich et al. [177] presented
GreenCloud, a packet-level simulator for energy-aware studies
of cloud computing data centers. This simulator could be used
to capture the details of the energy consumption of various
data center components (e.g., servers, switches, and links) as
well as packet-level communication patterns. Its effectiveness
in utilizing power management schemes was demonstrated
for two-tier, three-tier, and three-tier high-speed data center
architectures. A more powerful tool for analyzing green data
center design and resource management, named GDCSim,
was proposed in [178]. This simulation tool was used for the

2900
24 ×365

[168]

iterative design of green data centers. Specifically, it captured
the interdependencies between online resource management
schemes and the physical behavior of data centers. The objective of the simulator was to generate data center configurations
for particular purposes, such as CPU sleep-state transitions and
DVFS for power management, as well as thermal behavior
characterization. In [179], the authors proposed a simulator,
named SimWare, to analyze the power consumption of servers,
cooling units, and fans as well as the effects of heat recirculation and air supply timing. SimWare considered the time
required for air to travel from the CRACs to the servers and
could evaluate user-defined job scheduling and virtual machine
management algorithms. Data center designers could use this
simulator to evaluate energy-conserving policies and to assess
mechanical design options such as server placement, airflow
management and CRAC control strategies.
C. Challenges and Design Guidelines
Currently, the PUE is a widely accepted and measured metric
in data centers, whereas the use of other green metrics is
much less widespread. There is no doubt that green efficiency
metrics will continue to be developed and drive improvements
in sustainability and environmental friendliness. However, an
effective metric to characterize the energy and resource usage
efficiency of a data center must satisfy the following design
guidelines:

• able to track the important and actionable parameters to
assist operators in making data centers more green;
• consistent, easy and cost effective to implement and
report for management purposes;
• feasible, simple and inexpensive to monitor.
Thus, developing metrics that are well suited for reducing
energy consumption is a challenging task with a significant
contribution to the construction of “green” data centers.
Moreover, the monitoring of green metrics in data centers
assists in diagnosing energy inefficiencies and in the implementation of proactive mechanisms to improve energy efficiency.

17

However, this issue has received little attention thus far, and
existing monitors have been proposed for either particular
architectures [174] or special applications [175]. The challenges
are the large number of hosted servers or virtual machines and
the availability of communication resources in data centers.
Consequently, centralized monitoring techniques suffer from
low scalability and elasticity. Monitoring techniques adopted
from cooperative monitoring [180] can be applied to overcome
these drawbacks by enabling distributed and robust monitoring
solutions for large-scale, complex data centers. The key research
question of interest is how to minimize the costs incurred while
guaranteeing monitoring accuracy.
Finally, although platforms such as GreenCloud [177] and
GDCSim [178] have provided data center energy-aware simulation environments, the primary disadvantage of these platforms is that most of them are coarse grained and focused
only on certain components or functions. For example, when
a simulation of virtual machine migration considers only CPU
resources, the results for virtual machine allocation may suffer from a failure to consider the bandwidth requirements.
Therefore, the task of designing a system-wide experimental
and simulation platform that integrates all components of a
data center, such as the CPU, memory, cache, I/O, disk and
communication network, requires further exploration.

[11]

D. Abts, M. Marty, P. Wells, P. Klausler and H. Liu. Energy proportional datacenter networks. In Proceedings of ISCA’10, pages
338-347, 2010.
[12] T. Benson, A. Akella and D. Maltz. Network traffic characteristics
of data centers in the wild. In Proceedings of IMC’10, pages 267280, 2010.
[13] S. Rivoire, M. A. Shah, P. Ranganathan, C. Kozyrakis and J. Meza.
Models and metrics to enable energy-efficiency optimizations. In
Computer, 40(12), pages 39-48, 2007.
[14] Natural Resources Defense Council. Data center efficiency assessment. Issue paper, 2014.
[15] A. Beloglazov, J. Abawajy and R. Buyya. Energy-aware resource
allocation heuristics for efficient management of data centers for
cloud computing. In Future Generation Computer Systems, 28(5),
pages 755-768, 2012.
[16] K. Bilal, S.U.R. Malik, O. Khalid, A. Hameed, E. Alvarez, V.
Wijaysekara, R. Irfan, S. Shrestha, D. Dwivedy, M. Ali, U.S.
Khan, A. Abbas, N. Jalil and S.U. Khan. A taxonomy and survey
on green data center networks. In Future Generation Computer
Systems, 36, pages 189-208, 2014.
[17] F. Yao, A. Demers, and S. Shenker. A scheduling model for
reduced CPU energy. In Proceedings of FOCS’95, pages 374–382,
1995.
[18] T. Mudge. Power: A first-class architecture design constraint.
Computer, 34(4), pages 52–58, 2001.
[19] S. Albers. Energy-efficient algorithms. In Communications of the
ACM, 53(5), pages 86-96, 2010.
[20] M. Li, A. C. Yao and F. F. Yao. Discrete and continuous minenergy schedules for variable voltage processors. In Proceedings
of the National Academy of Sciences of USA, 103(11), pages 39833987, 2006.
VI. C ONCLUSION
[21] S. Albers and H. Fujiwara. Energy-efficient algorithms for flow
time minimization. In ACM Transactions on Algorithms, 3(4):49,
The development of green data centers has garnered consid2007.
erable attention because of their financial and environmental
[22] D. P. Bunde. Power-aware scheduling for makespan and flow. In
impact. Extensive research is being conducted in an attempt
Scheduling, 12(5), pages 489-500, 2009.
[23] K. Pruhs, R. V. Stee and P. Uthaisombut. Speed scaling of tasks
to reduce energy consumption and carbon emissions. In this
with precedence constraints. In Theory of Computing Systems,
article, we present a comprehensive review of the current state43(1), pages 67-80, 2008.
of-the-art research in the area of green data center management.
[24] X. Jin, F. Zhang, Y. Song, L. Fan and Z. Liu. Energy-efficient
This survey offers significant insights into various aspects of
scheduling with time and processors eligibility restrictions. In
the problem, including energy efficiency, resource management,
Proceedings of EuroPar’13, pages 66-77, 2013.
[25] N. Bansal, T. Kimbrel and K. Pruhs. Speed scaling to manage
thermal control and green metrics. We summarize and compare
energy and temperature. In Journal of the ACM, 54(1):3, 2007.
the existing schemes and highlight the challenges and potential
[26] S. Albers, F. Müller, and S. Schmelzer. Speed scaling on parallel
directions for future research with regard to each of these
processors. In Proceedings of SPAA’07, pages 289–298, 2007.
aspects.
[27] G. Greiner, T. Nonner, and A. Souza. The bell is ringing in speedscaled multiprocessor scheduling. In Proceedings of SPAA’09,
pages 11–18, 2009.
R EFERENCES
[28] C. Gunaratne, K. Christensen, B. Nordman and S. Suen. Reducing
the energy consumption of ethernet with adaptive link rate
[1] R. L. Villars. Worldwide datacenter census and construction 2014(ALR). In IEEE Transactions on Computers, 57(4), pages 448-461,
2018 forecast: aging enterprise datacenters and the accelerating
2008.
service provider buildout. 2014.
[2] http://www.nrdc.org/energy/data-center-efficiency-assessment.asp. [29] M. Andrews, A. F. Anta, L. Zhang and W. Zhao. Routing for
power minimization in the speed scaling model. In IEEE/ACM
[3] A. Venkatraman. Global census shows datacentre power demand
Transactions on Networking, 20(1), pages 285-294, 2012.
grew 63% in 2012.
[30] A. Antoniadis and C. C. Huang. Non-preemptive speed scaling.
http://www.computerweekly.com/news/2240164589/Datacentre-power-demand-grew-63-in-2012-Global-datacentre-census.
In Scheduling, 16(4), pages 385-394, 2013.
2012.
[31] E. Bampis, A. Kononov, D. Letsios, G. Lucarelli and I. Nemparis.
[4] P.X. Gao, A.R. Curtis, B. Wong and S. Keshav. It’s not easy being
From preemptive to non-preemptive speed-scaling scheduling. In
green. In Proceedings of SIGCOMM’12, pages 211-222, 2012.
Journal of Discrete Applied Mathematics, 181, pages 11-20, 2015.
[5] A. Greenberg, J. Hamilton, D. A. Maltz and P. Patel. The cost
[32] M. Li and F. F. Yao. An efficient algorithm for computing optimal
of a cloud: research problems in data center networks. In ACM
discrete voltage schedules. In SIAM Journal on Computing, 35(3),
SIGCOMM Computer Communication Review, 39(1), pages 68-73,
pages 658-671, 2005.
2009.
[33] X. Jin, F. Zhang and Z. Liu. Discrete Min-Energy Scheduling on
[6] P. Patel-Predd. Energy-efficient ethernet. In IEEE Spectrum, pages
Restricted Parallel Processors. In Proceedings of IPDPS’13, pages
13, 2008.
2226-2229, 2013.
[7] J. M. Kaplan, W. Forrest and N. Kindler. Revolutionizing data
[34] N. Bansal, H. Chan, T. Lam and L. Lee. Scheduling for speed
center energy efficiency. Technical report, McKinsey&Company,
bounded processors. In Proceedings of ICALP’08, pages 409-420,
2008.
2008.
[8] G. James. Power, pollution, and the internet. New York Times,
[35] E. Bampis, A. Kononov, D. Letsios, G. Lucarelli and M. Sviri2012.
denko. Energy efficient scheduling and routing via randomized
[9] L. A. Barroso, J. Clidaras and U. Holzle. The data center as
rounding. In Proceedings of FSTTCS’13, pages 449-460, 2013.
a computer: an introduction to the design of warehouse-scale
[36] N. Bansal, H.-L. Chan, K. Pruhs and D. Katz. Improved Bounds
machines. Morgan&Claypool publish, 2013.
for speed scaling in devices obeying the cube-root rule. In
[10] A. Carrega, S. Singh, R. Bruschi and R. Bolla. Traffic merging for
Proceedings of ICALP’09, pages 144-155, 2009.
energy-efficient datacenter networks. In Proceedings of SPECTS’12,
[37] W. Wu, M. Li and E. Chen. Min-energy scheduling for aligned
pages 1-5, 2012.

18

[38]
[39]
[40]

[41]
[42]
[43]
[44]
[45]
[46]
[47]
[48]
[49]
[50]
[51]
[52]
[53]
[54]
[55]

[56]
[57]
[58]
[59]

[60]
[61]
[62]
[63]

jobs in accelerate model. In Theoretical Computer Science, 412(1214), pages 1122-1139, 2011.
C.-C. Huang and S. Ott. New results for non-preemptive speed
scaling. In Proceedings of MFCS’14, pages 360-371, 2014.
V. Cohen-Addad, Z. Li, C. Mathieu and I. Milis. Energy-efficient
algorithms for non-preemptive speed-scaling. In WAOA’14, 2014.
J.-J. Chen, H.-R. Hsu, K.-H. Chuang, C.-L. Yang, A.-C. Pang and
T.-W. Kuo. Multiprocessor energy-efficient scheduling with task
migration considerations. In Proceedings of ECRTS’04, pages 101108, 2004.
S. Albers, A. Antoniadis and G. Greiner. On multi-processor
speed scaling with migration: extended abstract. In Proceedings
of SPAA’11, pages 279-288, 2011.
E. Angel, E. Bampis, F. Kacem and D. Letsios. Speed scaling on
parallel processors with migration. In Proceedings of EuroPar’12,
pages 128-140, 2012.
M. Li. Approximation algorithms for variable voltage processors:
min energy, max throughput and online heuristics. In Theoretical
Computer Science, 412(32), pages 4074-4080, 2011.
K. Pruhs, P. Uthaisombut and G. Woeginger. Getting the best
response for your erg. In Proceedings of SWAT’04, pages 14-25,
2004.
N. Bansal, K. Pruhs and C. Stein. Speed scaling for weighted flow
time. In Proceedings of SODA’07, pages 805-813, 2007.
T.-W. Lam, L.-K. Lee, I.K. To and P.W. Wong. Speed scaling
functions for flow time scheduling based on active job count.
In Proceedings of ESA’08, pages 647-659, 2008.
N. Bansal, H.-L. Chan and K. Pruhs. Speed scaling with an
arbitrary power function. In Proceedings of SODA’09, pages 693701, 2009.
L.L.H. Andrew, A. Wierman and A. Tang. Optimal speed scaling
under arbitrary power functions. In ACM SIGMETRICS Performance Evaluation Review, 37(2), pages 39-41, 2009.
E. Angel, E. Bampis and V. Chau. Throughput maximization in
the speed-scaling setting. In Proceedings of STACS’14, pages 53-62,
2014.
H.-L. Chan, W.-T. Chan, T.-W. Lam, L.-K. Lee, K.-S. Mak and
P.W.H. Wong. Energy efficient online deadline scheduling. In
Proceedings of SODA’07, pages 795-804, 2007.
T.-W. Lam, L.-K. Lee, I.K.K. To and P.W.H. Wong. Energy efficient
deadline scheduling in two processor systems. In Proceedings of
ISAAC’07, pages 476-487, 2007.
T.-W. Lam, L.-K. Lee, I.K.K. To and P.W.H. Wong. Improved
multi-processor scheduling for flow time and energy. In Scheduling, 15(1), pages 105-116, 2012.
E. Angle, E. Bampis, V. Chau and N.K. Thang. Throughput
maximization in multiprocessor speed-scaling. In Proceedings of
ISAAC’14, pages 247-258, 2014.
A. Gupta, R. Krishnaswamy and K. Pruhs. Scalably scheduling power-heterogeneous processors. In Proceedings of ICALP’10,
pages 312-323, 2010.
R.L. Graham, E.L. Lawler, J.K. Lenstra and A.H.G. Rinnooy Kan.
Optimization and approximation in deterministic sequencing and
scheduling: a survey. In Annals of Discrete Mathematics, vol.5,
pages 287-326, 1979.
A. Gandhi, M. Harchol-Balter, R. Das and C. Lefurgy. Optimal
power allocation in server farms. In Proceedings of SIGMETRICS’09, pages 157-168, 2009.
A. Wierman, L. L. Andrew and A. Tang. Power-aware speed scaling in processor sharing systems. In Proceedings of INFOCOM’09,
pages 2007-2015, 2009.
F. Liu, Z. Zhou, H. Jin, B. Li, B. Li and H. Jiang. On Arbitrating the
Power-Performance Tradeoff in SaaS Clouds. In IEEE Transactions
on Parallel and Distributed Systems, 25(10), pages 2648-2658, 2014.
K. Bilal, S. U. Khan, S. A. Madani, K. Hayat, M. I. Khan, N. MinAllah, J. Kologziej, L. Wang, S. Zeadally and D. Chen. A survey
on green communications using adaptive link rate. In Journal of
Cluster Computing, 16(3), pages 575-589, 2013.
D. Abts, M. R. Marty, P. M. Wells, P. Klausler and H. Liu. Energy proportional datacenter networks. In Proceedings of ISCA’10,
pages 338-347, 2010.
L. Wang, F. Zhang, C. Hou, J. A. Aroca and Z. Liu. Incorporating
rate adaptation into green networking for future data centers. In
Proceedings of NCA’13, pages 106-109, 2013.
S. Irani and A.R. Karlin. On online computation. In Approximation
Algorithms for NP-Hard Problems, chapter 13, pages 521-564, 1997.
A.R. Karlin, M.S. Manasse, L.A. McGeoch and S. Owicki. Com-

[64]

[65]
[66]
[67]
[68]
[69]
[70]
[71]
[72]
[73]
[74]
[75]
[76]
[77]
[78]
[79]

[80]
[81]
[82]
[83]

[84]
[85]
[86]
[87]
[88]
[89]

petitive randomized algorithms for nonuniform problems. In
Algorithmica, 11(6), pages 542-571, 1994.
S. Irani, S. Shukla and R. Gupta. Online strategies for dynamic
power management in systems with multiple power-saving
states. In ACM Transaction on Embedded Computing Systems, 2(3),
pages 325-346, 2003.
J. Augustine, S. Irani and C. Swamy. Optimal power-down
strategies. In SIAM Journal on Computing, 37(5), pages 1499-1516,
2008.
P. Baptiste. Scheduling unit tasks to minimize the number of idle
periods: a polynomial time algorithm for offline dynamic power
management. In SODA’06, pages 364-367, 2006.
P. Baptiste, M. Chrobak and C. Dürr. Polynomial time algorithms
for minimum energy scheduling. In Proceedings of ESA’07, pages
136-150, 2007.
Gururaj et al.. See survey of Marek Chrobak.
http://www.cs.pitt.edu/ kirk/cs3150spring2010
/10071.ChrobakMarek.Slides.pdf, 2010.
E. Angel, E. Bampis and Vincent Chau. Low complexity scheduling algorithms minimizing the energy for tasks with agreeable
deadlines. In Discrete Applied Mathematics, 175, pages 1-10, 2014.
E.D. Demaine, M. Ghodsi, M.T. Hajiaghayi, A.S. Sayedi-Roshkhar
and M. Zadimoghaddam. Scheduling to minimize gaps and
power consumption. In Scheduling, 16(2), pages 151-160, 2013.
J.-J. Chen, M.-J. Kao, D.T. Lee, I. Rutter and D. Wagner. Online
dynamic power management with hard real-time guarantees. In
STACS’14, pages 226-238, 2014.
J. S. Chase, D. C. Anderson, P. N. Thakar, A. M. Vahdat and R. P.
Doyle. Managing energy and server resources in hosting centers.
In Proceedings of SOSP’01, pages 103-116, 2001.
M. Lin, A. Wierman, L. Andrew and E. Thereska. Dynamic
right-sizing for power-proportional data centers. In Proceedings
of INFOCOM’11, pages 1098-1106, 2011.
Y. Azar, N. Ben-Aroya, N. R. Devanur and N. Jain. Cloud
scheduling with setup cost. In Proceedings of SPAA’13, pages 298304, 2013.
J. Chabarek, J. Sommers, P. Barford, C. Estan, D. Tsiang and
S. Wright. Power awareness in network design and routing. In
Proceedings of INFOCOM’08, pages 1130-1138, 2008.
R. Bolla, R. Bruschi, A. Cianfrani and M. Listanti. Enabling
backbone networks to sleep. In IEEE Network, 25(2), pages 2631, 2011.
M. Andrews, A. F. Anta, L. Zhang and W. Zhao. Routing and
scheduling for energy and delay minimization in the powerdown
model. In Proceedings of INFOCOM’10, pages 21-25, 2010.
M. Zhang, C. Yi, B. Liu and B. Zhang. GreenTE: power-aware
traffic engineering. In Proceeding of ICNP’10, pages 21-30, 2010.
B. Heller, S. Seetharaman, P. Mahadevan, Y. Yiakoumis, P.
Sharma, S. Banerjee and N. McKeown. ElasticTree: saving energy
in data center networks. In Proceedings of NSDI’10, pages 249-264,
2010.
J. Shuja, S. A. Madani, K. Bilal, K. Hayat, S. U. Khan and S.
Sarwar. Energy-efficient data centers. In Computing, 94(12), pages
973-994, 2012.
M. Gupta and S. Singh. Greening of the internet. In Proceedings
of SIGCOMM’03, pages 19-26, 2003.
S. Irani, S. Shukla and R. Gupta. Algorithms for power savings.
In ACM Transactions on Algorithms, 2(4):41, 2007.
X. Han, T.-W. Lam, L.-K. Lee, I.K.K. To and P.W.H. Wong.
Deadline scheduling and power management for speed bounded
processors. In Theoretical Computer Science, 411(40-42), pages 35873600, 2010.
G. Kumar and S. Shannigrahi. New online algorithm for dynamic
speed scaling with sleep state. In arXiv:1303.0377, 2013.
S. Albers and A. Antoniadis. Race to idle: new algorithms for
speed scaling with a sleep state. In ACM Transactions on Algorithms 10(2):9, 2014.
A. Antoniadis, C.-C. Huang and S. Ott. A fully polynomialtime approximation scheme for speed scaling with sleep state.
In arXiv:1407.0892, 2014.
T.W. Lam, L.-K. Lee, H.-F. Ting, I.K.-K. To and P.W.H Wong.
Sleep with guilt and work faster to minimize flow plus energy.
In Proceedings of ICALP’09, pages 665-676, 2009.
Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang and N.
Gautam. Managing server energy and operation costs in hosting
centers. In Proceedings of SIGMETRICS’05, pages 303-314, 2005.
Y. Yao, L. Huang, A.B. Sharma, L. Golubchik and M.J. Neely. Data

19

[90]
[91]

[92]
[93]

[94]

[95]
[96]
[97]
[98]
[99]
[100]
[101]
[102]

[103]
[104]
[105]
[106]
[107]

[108]

[109]
[110]
[111]

[112]

centers power reduction: a two time scale approach for delay
tolerant workloads. In Proceedings of INFOCOM’12, pages 14311439, 2012.
Y. Liu, S.C. Draper and N.S. Kim. SleepScale: runtime joint speed
scaling and sleep states management for power efficient data
centers. In Proceedings of ISCA’14, pages 313-324, 2014.
S. Nedevschi, L. Popa, G. Iannaccone, S. Ratnasamy and D.
Wetherall. Reducing network energy consumption via sleeping
and rate-adaptation. In Proceedings of NSDI’08, pages 323-336,
2008.
N. Vasić and D. Kostić. Energy-aware traffic engineering. In
Proceedings of e-Energy’10, pages 169-178, 2010.
L. Wang, F. Zhang, K. Zheng, A. V. Vasilakos, S. Ren and Z. Liu.
Energy-efficient flow scheduling and routing with hard deadlines
in data centers networks. In Proceedings of ICDCS’14, pages 248257, 2014.
A.K. Coskun, R. Strong, D.M. Tullsen and T. S. Rosing. Evaluating the impact of job scheduling and power management
on processor lifetime for chip multiprocessors. In Proceedings of
SIGMETRICS’09, pages 169-180, 2009.
R. Buyya, C.S. Yeo and S. Venugopal. Market-oriented cloud
computing: vision, hype, and reality for delivering IT services as
computing utilities. In Proceedings of HPCC’08, pages 5-13, 2008.
J. F. Kurose and K. W. Ross. Computer networking: a top-down
approach. Addison-Wesley publish, 2013.
R. Kumar, D. M. Tullsen and N. P. Jouppi. Core architecture optimization for heterogeneous chip multiprocessors. In Proceedings
of PACT’06, pages 23-32, 2006.
F. A. Bower, D. J. Sorin and L. P. Cox. The impact of dynamically
heterogeneous multicore processors on thread scheduling. In
IEEE Micro, 28(3), pages 17-25, 2008.
A. Gupta, S. Im, R. Krishnaswamy, B. Moseley and K. Pruhs.
Scheduling heterogeneous processors isn’t as easy as you think.
In Proceedings of SODA’12, pages 1242-1253, 2012.
P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho,
R. Neugebauer, I. Pratt and A. Warfield. Xen and the art of
virtualization. In Proceedings of SOSP’03, pages 164-177, 2003.
R. Nathuji, K. Schwan. VirtualPower: coordinated power management in virtualized enterprise systems. In Proceedings of
SOSP’07, pages 265-278, 2007.
R. Raghavendra, P. Ranganathan, V. Talwar, Z. Wang and X.
Zhu. No “power” struggles: coordinated multi-level power management for the data center. In Proceedings of ASPLOS’08, pages
48-59, 2008.
F. Hermenier, X. Lorca, J. M. Menaud, G. Muller and J. Lawall.
Entropy: a consolidation manager for clusters. In Proceedings of
VEE’09, pages 41-50, 2009.
N. Bobroff, A. Kochut and K. Beaty. Dynamic placement of
virtual machines for managing SLA violations. In Proceedings of
IM’07, pages 119-128, 2007.
T. Wood, P. Shenoy, A. Venkataramani and M. Yousif. Blackbox and gray-box strategies for virtual machine migration. In
Proceedings of NSDI’07, pages 229-242, 2007.
H.N. Van, F.D. Tran and J.M. Menaud. SLA-aware virtual resource management for cloud infrastructures. In Proceedings of
CIT’09, pages 357-362, 2009.
D. Kusic, J. O. kephart, J. E. Hanson, N. Kandasamy and
G. Jiang. Power and performance management of virtualized
computing environments via lookahead control. Journal of Cluster
Computing, 12(1), pages 1-15, 2009.
A. Beloglazov and R. Buyya. Optimal online deterministic algorithms and adaptive heuristics for energy and performance
efficient dynamic consolidation of virtual machines in cloud data
centers. In Journal of Concurrency and Computation: Practice and
Experience, 24(3), pages 1397-1420, 2011.
M. Wang, X. Meng and L. Zhang. Consolidating virtual machines with dynamic bandwidth demand in data centers. In
Proceedings of INFOCOM’11, pages 71-75, 2011.
C. C. Lee and D. T. Lee. A simple on-line bin-packing algorithm.
In Journal of the ACM, 32(3), pages 562-572, 1985.
A.L. Stolyar and Y. Zhong. A large-scale service system with
packing constraints: minimizing the number of occupied servers.
In ACM SIGMETRICS Performance Evaluation Review, 41(1), pages
41-52, 2013.
X. Jin, F, Zhang, S. Hu and Z. Liu. Risk management for
virtual machines consolidation in data centers. In Proceedings of
Globecom’13, pages 2872-2878, 2013.

[113]

[114]
[115]
[116]
[117]
[118]
[119]
[120]

[121]

[122]
[123]
[124]
[125]
[126]
[127]
[128]

[129]
[130]

[131]

[132]
[133]
[134]
[135]
[136]
[137]

P. Mahadevan, P. Alto, S. Banerjee, P. Sharma, A. Shah and P.
Ranganathan. On energy efficiency for enterprise and data center
networks. In IEEE Communications Magazine, 49(8), pages 94-100,
2011.
R. McGeer, P. Mahadevan and S. Banerjee. On the complexity
of power minimization schemes in data center networks. In
Proceedings of Globecom’10, pages 1-5, 2010.
N. Vasić, P. Bhurat, D. Novaković, M. Canini, S. Shekhar and D.
Kostić. Identifying and using energy-critical paths. In Proceedings
of CoNEXT’11, No. 18, 2011.
J. Jiang, T. Lan, S. Ha, M. Chen and M. Chiang. Joint VM
placement and routing for data center traffic engineering. In
Proceedings of INFOCOM’12, pages 2876-2880, 2012.
X. Wang, Y. Yao, X. Wang, K. Lu and Q. Cao. CARPO:
correlation-aware power optimization in data center networks.
In Proceedings of INFOCOM’12, pages 1125-1133, 2012.
Y. Zhang and N. Ansari. HERO: hierarchical energy optimization for data center networks. In Proceedings of ICC’12, pages 29242928, 2012.
M. Xu, Y. Shang, D. Li and X. Wang. Greening data center
networks with throughput-guaranteed power-aware routing. In
Computer Networks, 57(15), pages 2880-2899, 2013.
L. Wang, F. Zhang, J.A. Aroca, A.V. Vasilakos, K. Zheng, C. Hou,
D. Li and Z. Liu. GreenDCN: a general framework for achieving
energy efficiency in data center networks. In IEEE Journal on
Selected Areas in Communications, 32(1), pages 4-15, 2014.
H. Jin, T. Cheocherngngarn, D. Levy, A. Smith, D. Pan, J. Liu and
N. Pissinou. Joint host-network optimization for energy-efficient
data center networking. In Proceedings of IPDPS’13, pages 623-634,
2013.
K. Zheng, X. Wang, L. Li and X. Wang. Joint power optimization
of data center network and servers with correlation analysis. In
Proceedings of INFOCOM’14, 2014.
N. Rasmussen. A scalable, reconfigurable, and efficient data center power distribution architecture. American Power Conversion,
White paper #129, 2009.
X. Fan, W.-D. Weber and L.A. Barroso. Power provisioning for
a warehouse-sized computer. In Proceedings of ISCA’07, pages 1323, 2007.
D. Meisner, B.T. Gold and T.F. Wenisch. PowerNap: eliminating
server idle power. In Proceedings of ASPLOS’09, pages 205-216,
2009.
S. Pelley, D. Meisner, P. Zandevakili, T.F. Wenisch and J. Underwood. Power routing: dynamic power provisioning in the data
center. In Proceedings of ASPLOS’10, pages 231-242, 2010.
H. Lim, A. Kansal and J. Liu. Power budgeting for virtualized
data centers. In Proceedings of USENIX ATC’11, 2011.
C. Isci, S. McIntosh, J. Kephart, R. Das, J. Hanson, S. Piper, R.
Wolford, T. Brey, R. Kantner, A. Ng, J. Norris, A. Traore and M.
Frissora. Agile, efficient virtualization power management with
low-latency server power states. In Proceedings of ISCA’13, pages
96-107, 2013.
S. Govindan, A. Sivasubramaniam and B. Urgaonkar. Benefits
and limitations of trapping into stored energy for datacenters. In
Proceedings of ISCA’11, pages 341-352, 2011.
V. Kontorinis, L.E. Zhang, B. Aksanli, J. Sampson, H. Homayoun, E. Pettis, D.M. Tullsen and T. S. Rosing. Managing distributed ups energy for effective power capping in data centers.
In Proceedings of ISCA’12, pages 488-499, 2012.
S. Govindan, D. Wang, A. Sivasubramaniam and B. Urgaonkar.
Leveraging stored energy for handling power emergencies in aggressively provisioned datacenters. In Proceedings of ASPLOS’12,
pages 75-86, 2012.
D. Wang, C. Ren and A. Sivasubramaniam. Virtualizing power
distribution in datacenters. In Proceedings of ISCA’13, pages 595606, 2013.
B. Stone. Google’s next frontier: renewable energy. In New York
Times, 2007.
P. Gupta. Google to use wind energy to power data centers. In
New York Times, 2010.
N. Sharma, S. Barker, D. Irwin and P. Shenoy. Blink: managing
server clusters on intermittent power. In Proceedings of ASPLOS’11, pages 185-198, 2011.
S. Akoush, R. Sohan, A. Rice, A.W. Moore and A. Hopper. Free
lunch: exploiting renewable energy for computing. In Proceedings
of HotOS’11, 2011.
C. Li, R. Zhou and T. Li. Enabling distributed generation pow-

20

[138]

[139]
[140]
[141]
[142]
[143]
[144]

[145]
[146]
[147]

[148]
[149]
[150]
[151]
[152]
[153]
[154]

[155]
[156]
[157]
[158]

[159]

[160]
[161]

ered sustainable high-performance data center. In Proceedings of
ming. http://tinyurl.com/89ros64, 2012.
HPCA’13, pages 35-46, 2013.
[162] V. Sundaralingam, P. Kumar and Y. Joshi. Server heat load based
CRAC fan controller paired with rear door heat exchanger. In
Í. Goiri, K. Le, M.E. Haque, R. Beauchea, T.D. Nguyen, J.
Proceedings of the ASME 2011 Pacific Rim Technical Conference and
Guitart, J. Torres and R. Bianchini. GreenSlot: scheduling energy
Exhibition on Packaging and Integration of Electronic and Photonic
consumption in green datacenters. In Proceedings of SC’11, pages
Systems, pages 489-496, 2011.
1-11, 2011.
[163] C. Belady, A. Rawson, D. Pfleuger and S. Cader. The green grid
Í. Goiri, K. Le, T.D. Nguyen, J. Guitart, J. Torres and R. Biandata center power efficiency metrics: PUE and DCiE. Consortium
chini. GreenHadoop: leveraging green energy in data-processing
Green Grid, 2008.
frameworks. In Proceedings of EuroSys’12, pages 57-70, 2012.
[164] D. Anderson et al.. A framework for data center energy producC. Li, A. Qouneh and T. Li. iSwitch: coordinating and optimiztivity. The Green Grid, 2008.
ing renewable energy powered server clusters. In Proceedings of
[165] P. Mathew, S. Ganguly, S. Greenberg and D. Sartor. SelfISCA’12, pages 512-523, 2012.
benchmarking guide for data centers: metrics, benchmarks, acZ. Liu, M. Lin, A. Wierman, S.H. Low and L.L.H. Andrew.
tions. The Lawrence Berkeley National Laboratory, 2009.
Greening geographical load balancing. In Proceedings of SIGMET[166] D. Azevedo et al.. Carbon usage effectiveness (CUE): A green
RICS’11, pages 233-244, 2011.
grid data center sustainability metric. The Green Grid, 2010.
Y. Gao, Z. Zeng, X. Liu and P.R. Kumar. The answer is blowing
[167] U.S. Department of Energy, Energy Information Administration.
in the wind: analysis of powering Internet data centers with wind
Appendix F. Electricity emission factors. EIA-1605, 2007.
energy. In Proceedings of INFOCOM’13, pages 520-524, 2013.
[168] 42U. WEU: Water-side economizers utilization.
Y. Zhang, Y. Wang and X. Wang. GreenWare: greening cloudhttp://www.42u.com/cooling/water-side-economizers.htm,
scale data centers to maximize the use of renewable energy. In
2011.
Proceedings of Middleware’11, pages 143-164, 2011.
[169] E. Jaureguialzo. PUE: the green grid metric for evaluating the
J. Mars, L. Tang, R. Hundt, K. Skadron and M. L. Soffa. Bubbleenergy efficiency in DC (Data Center). Measurement method
Up:increasing utilization in modern warehouse scale computers
using the power demand. In Proceedings of INTELEC’11, pages
via sensible co-locations. In Proceedings of MICRO’11, pages 2481-8, 2011.
259, 2011.
[170] J. Gao. Machine learning applications for data center optimizaH. Yang, A. Breslow, J. Mars and L. Tang. Bubble-flux: precise
tion. Google Report, 2014.
online QoS management for increased utilization in warehouse
[171] L.H. Sego, A. Márquez, A. Rawson, T. Cader, K. Fox, W.I.
scale computers. In Proceedings of ISCA’13, pages 607-618, 2013.
Gustafson JR. and C.J. Mundy. Implementing the data center enT. Benson, A. Akella, A. Shaikh and S. Sahu. CloudNaaS: a cloud
ergy productivity metric. In ACM Journal on Emerging Technologies
networking platform for enterprise applications. In Proceedings of
in Computing System, 8(4):30, 2012.
SOCC’11, No.8, 2011.
[172] Green IT Promotion Council. DPPE: holistic framework for data
K. Skadron, M.R. Stan, K. Sankaranarayanan, W. Huang, S.
center energy efficiency. 2012.
Velusamy and D. Tarjan. Temperature-aware microarchitecture:
[173] Harmonizing global metrics for data center energy efficiency. In
modeling and implementation. In ACM Transactions on ArchitecMetrics and measurements, 2014.
ture and Code Optimization, 1(1), pages 94-125, 2004.
[174] L. Liu, H. Wang, X. Liu, X. Jin, W. He, Q. Wang and Y.
R. Grundy. Recommended data center temperature & humidity.
Chen. GreenCloud:
a new architecture
for green data
center. In
Computer
Room Temperature
Humidity.htm,
http://www.avtech.com/About/Articles/AVT/NA/All/-/DD-NN-AN-TN/Recommended
Proceedings of ICAC-INDST’09, pages 29-38, 2009.
2005.
[175] G. Lu, J. Zhang, H. Wang, L. Yuan and C. Weng. PowerTracer:
C. Belady, A. Rawson, J. Pfleuger and T. Cader. The green grid
tracing requests in multi-tier services to diagnose energy ineffidata center power efficiency metrics: PUE & DCiE. Green Grid,
ciency. In Proceedings of ICAC’12, pages 97-102, 2012.
Technical Report, 2008.
C.D. Patel, C.E. Bash, R. Sharma and M. Beitelmal. Smart cooling
[176] F. Julià, J. Roldan, R. Nou, J.O. Fitó, A. Vaqué, Í. Goiri and
of data centers. In Proceedings of IPACK’03, pages 129-137, 2003.
J.L. Berral. EEFSim: energy efficiency simulator. Research Report:
W. Zheng, K. Ma and X. Wang. Exploiting thermal energy
UPC-DAC-RR-CAP-2010-15, 2010.
storage to reduce data center capital and operating expenses. In
[177] D. Kliazovich, P. Bouvry, Y. Audzevich and S.U. Khan. GreenProceedings of HPCA’14, pages 132-141, 2014.
Cloud: a packet-level simulator of energy-aware cloud computR.K. Sharma, C.L. Bash, C.D. Patel, R.J. Friedrich and J.S. Chase.
ing data centers. In Proceedings of Globecom’10, pages 1-5, 2010.
Balance of power: dynamic thermal management for Internet
[178] S.K.S. Gupta, R.R. Gilbert, A. Banerjee, Z. Abbasi, T. Mukherjee
Data centers. In IEEE Internet Computing, 9(1), pages 42-49, 2005.
and G. Varsamopoulos. GDCSim: a tool for analyzing green
J. Moore, J. Chase, P. Ranganathan and R. Sharma. Making
data center design and resource management techniques. In
scheduling “cool”: temperature-aware workload placement in
Proceedings of IGCC’11, pages 1-8, 2011.
data centers. In Proceedings of ATEC’05, pages 61-74, 2005.
[179] S. Yeo and H.-H.S. Li. SimWare: a holistic warehouse-scale
T. Heath, A.P. Centeno, P. George, L. Ramos, Y. Jaluria and
computer simulator. In Computer, 45(9), pages 48-55, 2012.
R. Bianchini. Mercury and freon: temperature emulation and
[180] K. Xu and F. Wang. Cooperative monitoring for internet data
management for server systems. In Proceedings of ASPLOS’06,
centers. In Proceedings of IPCCC’08, pages 111-118, 2008.
pages 106-116, 2006.
H. Xu, C. Feng and B. Li. Temperature aware workload management in geo-distributed datacenters. In Proceedings of ICAC’13,
pages 303-314, 2013.
R.T. Kaushik and K. Nahrstedt. T ∗ : a data-centric cooling energy costs reduction approach for big data analytics cloud. In
Proceedings of SC’12, pages 1-11, 2012.
J. Brandon. Going green in the data center: practical steps for
your sme to become more environmentally friendly. In Processor,
29(39), pages 1-30, 2007.
S. Biswas, M. Tiwari, T. Sherwood, L. Theogarajan and F.T.
Chong. Fighting fire with fire: modeling the datacenter-scale effects of targeted superlattice thermal management. In Proceedings
of ISCA’11, pages 331-340, 2011.
N. El-Sayed, I.A. Stefanovici, G. Amvrosiadis, A.A. Hwang and
B. Schroeder. Temperature management in data centers: why
some (might) like it hot. In Proceedings of SIGMETRICS’12, pages
163-174, 2012.
R. Zhou, Z. Wang, A. McReynolds, C.E. Bash, T.W. Christian
and R. Shih. Optimization and control of cooling microgrids for
data centers. In Proceedings of ITherm’12, pages 338-343, 2012.
R. Miller. Too hot for humans, but Google servers keep hum-

