Self-Awareness of Cloud Applications
Alex Iosup, Xiaoyun Zhu, Arif Merchant, Eva Kalyvianaki, Martina Maggio,
Simon Spinner, Tarek Abdelzaher, Ole Mengshoel, Sara Bouchenak

arXiv:1611.00323v1 [cs.SE] 1 Nov 2016

Note: This is an extended survey. A much shorter, revised version of this material
will be available in print, as part of a Springer book on ”Self-Aware Computing”.
The book is due to appear in 2017.

Abstract Cloud applications today deliver an increasingly larger portion of the
Information and Communication Technology (ICT) services. To address the scale,
growth, and reliability of cloud applications, self-aware management and scheduling are becoming commonplace. How are they used in practice? In this chapter,
we propose a conceptual framework for analyzing state-of-the-art self-awareness
approaches used in the context of cloud applications. We map important applications corresponding to popular and emerging application domains to this conceptual
framework, and compare the practical characteristics, benefits, and drawbacks of
self-awareness approaches. Last, we propose a roadmap for addressing open challenges in self-aware cloud and datacenter applications.
Alexandru Iosup
Delft University of Technology, the Netherlands, e-mail: A.Iosup@tudelft.nl
Xiaoyun Zhu
Futurewei Technologies, CA, USA, e-mail: xiaoyzhu@yahoo.com
Arif Merchant
Google, Inc., CA, USA, e-mail: aamerchant@google.com
Eva Kalyvianaki
Imperial College of London, UK, e-mail: Evangelia.Kalyvianaki.1@city.ac.uk
Martina Maggio
Lund University, Sweden, e-mail: mmartimay@gmail.com
Simon Spinner
University of Wuertzburg, Germany, e-mail: simon.spinner@uni-wuerzburg.de
Tarek Abdelzaher
University of Illinois at Urbana Champaign, IL, USA, e-mail: zaher@illinois.edu
Ole Mengshoel
CMU Silicon Valley at the NASA Ames Research Center, PA, USA, e-mail: ole.mengshoel@
sv.cmu.edu
Sara Bouchenak
INSA Lyon, France, e-mail: sara.bouchenak@insa-lyon.fr

1

2

Iosup et al.

1 Introduction
Cloud computing is the Information and Communication Technology (ICT) paradigm
under which services are provisioned by their users only when needed, only for as
long as needed, and with payment expected to cover only what is actually used.
Cloud users can today lease infrastructure, platform, software, and other “as a service”, from commercial clouds such as Amazon, SAP, and Google. Governments
and entire industries are building large-scale datacenters that are and will increasingly host cloud computing applications. Not only computation, but also data will be
increasingly part of cloud computing: by 2017, over three-quarters of our and business data will reside in datacenters, according to a recent IDC report [57]. Cloud
applications, often consumed by users as services, already represent over 10% of
the entire ICT market in Europe [24]. Netflix, whose users consume a large fraction
of the US and global Internet traffic, relies on ICT services from Amazon Web Services (AWS)1 . The market, which is increasing in size, diversity of applications, and
sophistication, already exceeds hundreds of millions of users and, as a consequence,
$100 billion world-wide [17]; the cloud market will likely contribute over 100 billion Euro to the European GDP, in 2020 [24]. At this scale and with this importance,
human management of IT resources is prohibitively expensive and, often, too errorprone. Thus, the use of self-awareness techniques to manage cloud applications is
increasingly more present. In this chapter, we analyze the use of self-aware in cloud
computing and its applications.
Cloud applications raise a complex management challenge, derived from the
goals of three main stakeholders: application users, application operators, and cloud
operators. Each of these stakeholders has different requirements, which are often
conflicting. For example, application users could demand that an interactive application is always responsive, even under bursty arrivals of user-issued commands.
To meet this demand, application operators could require that enough capacity is
always provided by cloud operators, yet only want to pay for what is actually consumed. Tension arises between performance and other requirements, and the cost of
operation. As a consequence, the management challenge is to optimize non-trivial
efficiency metrics and to meet complex service level agreements (SLAs), to an extent that already exceeds the capabilities of human management.
We investigate in this chapter the current state of self-awareness in cloud computing, and in particular datacenter-based cloud computing, and its applications. Our
goals are to introduce practical cases of self-awareness in datacenter applications;
to present a conceptual framework for analyzing state-of-the-art self-awareness approaches used in practice; to map already important and emerging application domains to the conceptual framework of self-awareness approaches used in practice,
and analyze similarities and differences, benefits and costs of self-awareness approaches; and to identify and analyze open challenges in self-aware cloud and datacenter applications, and propose a roadmap for advancing the state-of-the-art. The
main contribution is structured as follows.
1

Details: https://aws.amazon.com/solutions/case-studies/netflix/.

Self-Awareness of Cloud Applications

3

In Section 2, we introduce a framework for the analysis of self-awareness techniques used in cloud computing and its applications. Our framework consists of a
structured way to analyze the types of applications, of problems, and of approaches
for which self-awareness is relevant in practice. The framework also structures the
analysis of directions for future research. Although the framework is currently built
to serve the analysis of self-awareness in cloud computing and its applications, and
thus is adapted to the operational conditions in cloud computing (metrics, stakeholders, etc.), the framework could be extended to other domains. We show the
usefulness of this framework by applying it in practice, with the results presented in
the next sections.
In Section 3, we focus on eight popular or emerging application domains, described as application domains with important commercial, scientific, governance,
and other societal impact. Although any application domain is applicable, the market volume and the number of users, today or in the foreseeable future, are important
criteria for selecting the application domains for this chapter. Among the selected
applications are business applications, compute-intensive and data-intensive batch
processing, data-stream processing, online gaming, partial processing, and cyberphysical applications. Some of these applications, such as online gaming, partial
processing, and cyber-physical applications are emerging in terms of number of
users and adoption of cloud technology. We also include in this section the workloads generated by the datacenters themselves, which can be seen as overhead, but
are already consuming large amounts of resources and must meet complex, albeit
internal, SLAs.
In Section 4, we identify ten types of problems that are already addressed by
self-awareness techniques. The types of problems selected for this section include:
recovery planning, autoscaling of resources, runtime architectural reconfiguration
and load balancing, fault-tolerance in distributed systems, energy-proportionality,
workload prediction, performance isolation, diagnosis and troubleshooting, discovery of application topology, and intrusion detection and prevention. Some of these
problems, such as autoscaling, energy-proportionality, performance isolation, and
intrusion detection and prevention, have developed a new form or even appeared
specifically in the context of cloud computing.
In Section 5, we identify and analyze seven types of self-awareness approaches
used in practice: feedback control based techniques, metric optimization with constraints, machine learning based techniques, portfolio scheduling, self-aware architecture reconfiguration, and stochastic performance models. Although none of these
approaches is unique to cloud computing, their adaptation to cloud computing and
its applications is non-trivial.
In Section 6, we identify and analyze four directions for future use of selfawareness approaches for cloud computing and its applications. We focus on directions that are not only needed for practical applications, but for which we can
already envision the next research steps and that the results of this research can be
put in practice in the following 3–5 years.
Our survey of applications, problems, self-awareness approaches, and open challenges in self-awareness is by far not exhaustive. However, we study for each broad

4

Iosup et al.

Cloud Application

Self-awareness
Challenge

Self-Aware
Approach

Fig. 1 Conceptual structure of our framework for understanding the practice of using selfawareness techniques for managing and scheduling cloud applications. Each box represents a type.
See text for a description of the different line-styles.

types with existing popularity and likely future impact. Moreover, we envision that
the approach we take in this work will also be useful for studying other types.
This chapter is the result of original work by the authors, and in particular the
survey started during the Dagstuhl Seminar 15041, “Model-driven Algorithms and
Architectures for Self-Aware Computing Systems”.

2 Overview of the Framework
We propose a framework for understanding the practice of using self-awareness
techniques for managing and scheduling cloud applications. This framework, whose
conceptual structure is depicted in Figure 1, follows the structure of a natural discussion about the field, with three main questions and a format for answering them
that is conductive to surveying the field. The first is the question that triggers practitioners to select an existing self-awareness technique or to develop a new such
technique: Which cloud applications raise the challenges that self-awareness techniques are particularly good in addressing? Answering this question requires an
understanding of the nature and characteristics of self-awareness challenges that
affect cloud applications. Thus, the second question is Which are the important selfawareness challenges for cloud applications? As a third question, Which are the
self-awareness approaches that address the self-awareness challenges in this context? Last, a fourth question focuses on the future: Assuming a research horizon of
3-5 years, what are the most promising directions for future research in enabling
self-aware cloud applications?
Answering the first three questions is sufficient to yield a survey of cloud applications whose self-awareness challenges are addressed or resolved in practice by self-

Self-Awareness of Cloud Applications

5

awareness approaches (techniques, methods, best-practices, or even entire methodologies). For example, mapping all the different ¡application-challenge-approach
paths¿ can create a survey of the entire space in Figure 1. In the same figure, it is easy
to group together applications leading to the same challenge, such as the three cloud
applications depicted using dotted lines; a similar observation can be made about
challenges that can be addressed with different types of self-awareness approaches.
The survey is practical, in that a complete ¡application-challenge-approach¿ path,
such as the path depicted with thick lines in Figure 1, can be directly considered by
the practitioner.
The first three questions in our framework are also necessary. There are hundreds of application types commonly used in software engineering practice, as indicated for example by the extensive taxonomy of Forward and Lethbridge [32].
Thus, surveying without the guidance of specific applications provided by the first
question could lead to a variety of self-awareness challenges and approaches, all
with the merit of being applicable, but without much proof of use in cloud context.
Without the specific problems provided by the second question, the self-awareness
techniques could be used in a variety of cases, limited only by the creativity of the
designer and by the difficulty of proving their benefit for practical use. Thus, limiting the survey to the set of challenges that are currently addressed in the context
of cloud applications is necessary; we address this through the combined expertise
of the authors regarding the field. Last, although many self-awareness techniques
already exist, not all have yet been applied to cloud settings. Thus, surveying could
go well beyond the scope of the third question, and generic techniques that may not
work well (enough) in practice will also be surveyed.
For answering the first three questions, our framework proposes a structure to
analyze the types of applications, of problems, and of approaches for which selfawareness is relevant in practice. For each application type, we propose that each
answer should include:
1. A practical description of the application, including if possible a definition and
an analysis of the importance of the application (e.g., number of users, market
size), all expressed for the context in which the application appears in the cloud.
2. An analysis of the components that appear in typical workloads for this application, including if possible a description of how the components are structured
(e.g., multi-tier, bag of tasks, workflow).
3. A survey of typical metrics of interest for the application type, including if possible the main trade-offs when many conflicting metrics exist.
4. A list of typical self-awareness challenges (problems) affecting the deployment
of these applications in cloud settings. This list anticipates the answers to and
should be built in parallel with answering to the second question in our framework.
5. A list of typical self-awareness approaches, techniques, and methodologies (selfaware elements). This list anticipates the answers to and should be built in parallel with answering to the third question in our framework.

6

Iosup et al.

For each type of self-aware challenge (problem), we propose that each answer
should include:
1. A description of the context in which the problem appears, including an example
or another practical detail.
2. A description of the problem itself.
3. An analysis of the expected advancement in solving the problem, that selfawareness approaches may provide.
4. An analysis of the expected impact on application types, including a list of
types of cloud applications that would be (positively) affected by self-awareness
approaches alleviating or solving the problem.
For each type of self-aware approach, we propose that each answer should include:
1. A description of the self-aware approach, including if possible a contrast with
non-self-awareness techniques.
2. An analysis of the expected impact that using the self-awareness approach can
have on the application, in practice.
3. A description of the main technical details of the self-awareness approach.
4. A set of use cases in which the self-aware approach is used in practice. This last
part of the answer effectively maps popular and emerging application domains
to the set of self-awareness approaches, enabling a qualitative comparison of
practical characteristics, benefits, and drawbacks of self-awareness approaches.
The framework also structures the analysis of directions for future research. We
leverage here the collective expertise of the authors, in which important questions
are proposed by individuals and discussed by the community. By iterating this process, we believe the community can propose and refine its own most important goals.
The results of the first iteration are shared with everyone interested to help the community make progress, through the text in Section 6.
In the following sections, we show the results of our survey using the proposed
framework. In turn, we cover types of cloud applications (Section 3), types of problems (Section 4), and types of approaches (Section 5), and end with a set of open
challenges for self-aware cloud applications (Section 6).

3 Types of Applications
In this section, we present the following eight types of applications that already
benefit from the use of self-awareness techniques.
1.
2.
3.
4.
5.

Enterprise applications
Computing-intensive batch processing
Data-intensive batch processing
Data-stream processing
Workloads generated by datacenter operations

Self-Awareness of Cloud Applications

7

6. Online gaming
7. Partial and delayed processing
8. Cyber-physical applications
Each of these selected applications is already popular, or generates a significant
amount of revenue, or is critical to the operation of many businesses, or uses a
significant amount of resources, or is promising to emerge as such; often, the applications we select have a combination of these characteristics.

3.1 Enterprise Applications
Description: We reuse the definition of enterprise applications proposed by Shen
et al. [119]: ”the user-facing and backend services, generally supporting business
decisions and operations and commonly contracted under strict SLA requirements,
whose downtime or even just low performance will lead to reduced productivity,
loss of revenue, customer departure, or even legal actions. These workloads include enterprise multi-tier applications, and business-critical workloads that often
include applications in the solvency domain or other decision-making tools. Other
applications that characterize business-critical workloads are email, collaboration,
database, ERP, CRM, and management services, when used in conjunction with
other workloads.” [119]

3.1.1 Multi-Tier Enterprise Applications
Application components: Multi-tier enterprise applications refer to those webbased business applications that are architected as a collection of cooperating components, organized as multiple logical tiers. The most common three-tier architecture consists of a presentation tier, an application tier, and a database tier. The presentation tier receives client requests to the application, the application tier handles
the business logic, and in turn interacts with the database tier to obtain and store persistent data. Typical examples for such applications are Enterprise Resource Planning (ERP) or Customer Relationship typically subject to an interactive workload,
consisting of many small requests. Different types of requests often incur different loads on the system (e.g., read- vs. write-intensive transactions, compute- vs.
data-intensive workloads). The multi-tier architecture makes it challenging to implement self-awareness schemes for such applications, as there may be complex
control flows between the different tiers, and each tier may have different resource
requirements and performance bottlenecks. With the trend towards service-oriented
architectures, the different tiers are often split into different services, making the
control flow even more complex.
Metrics of interest: These include, but are not limited to, availability - measured
by the percentage of time the application service remains up and running; reliability
- is the ratio of successful requests to the total number of requests; performance

8

Iosup et al.

- characterized by such metrics as throughput (requests/sec) and request response
times; resource (CPU, memory, I/O) utilization metrics of the underlying system;
and cost - characterized by metrics related to financial and energetic cost.
Typical problems: Performance isolation and service differentiation, trading-off
between multiple metrics of interest, diagnosis and troubleshooting, dynamic load
balancing, end-to-end service level assurance, and autoscaling of resources.
Typical self-aware elements: Metric optimization [84], metric trade-off [6], machine learning [102, 129], feedback control [63, 65, 84, 114], stochastic performance
models [6, 121], and statistical estimation [78] are the main approaches researchers
have applied to help create self-aware multi-tier applications.

3.1.2 Business-Critical Applications
Application components: Business-critical workloads often include applications
that provide support for decision-making, such as Monte Carlo simulations, financial and other types of modeling applications programmed as tightly coupled parallel jobs of relatively small size, but also the regular management services described
in Section 3.1.1. It is typical for the system user to not specify the applications,
due to privacy and business secrecy. Instead, users request service expressed only
in SLA terms, e.g., number and size of virtual machines, generally provisioned for
long periods of time and operated by the user’s IT team. The current practice in the
datacenter is to require engineering confirmation for the most important provisioning and allocation decisions, especially at the initial installation of the long-running
virtual machines. Self-aware resource management and scheduling tools [125] provide advice that engineers may take into account.
Metrics of interest: Various traditional metrics, including latency and throughput, and reliability and system load. Risk-related metrics, such as the risk score [125],
which expresses the risk of significant under-performance and thus penalties paid by
the service operator to the service user, and loss of trust.
Typical problems: Reduce the risk of low performance. Use resources efficiently. Avoid system overload and unavailability.
Typical self-aware elements: Portfolio scheduling [125]. Topology-aware resource management [125]2 . Prediction of runtimes and resource occupancy. Binpacking-based optimization.

3.2 Compute-Intensive Batch Processing
Description: Compute-intensive batch processing includes workloads where computation, rather than data I/O and movement, consumes the largest part of the
runtime and of the consumed resources, and is thus the main focus of resource
2

Commercial products in this domain are scarce. Notable products include VMware’s open-source
Project Serengeti http://www.vmware.com/hadoop/serengeti

Self-Awareness of Cloud Applications

9

management and scheduling. This type of application has evolved much over the
past few decades, from select few users running large parallel jobs on supercomputers (late 1980s and early 1990s), to practically every research and engineering
lab running in multi-cluster grids many small, independent tasks [60], integrated
through scripts into mostly compute-intensive jobs (mid-1990s to today). These
bags of tasks (BoTs), which are effectively conveniently parallel implementations
of scientific and engineering applications (e.g., simulations), have emerged as a response to transitioning from the expensive supercomputers that offered high performance and availability, to commodity hardware that crashes often. Beginning
with the early 2000s, workflows of inter-dependent tasks, where dependencies are
expressed programmatically and inter-task data transfers occur through batch transfers of (typically POSIX) files, have also become increasingly more common in
practice [60, 62].

3.2.1 Compute-Intensive Batch Processing In Clusters
Application components: Workloads include bags of predominantly sequential
tasks and small-scale parallel jobs (in engineering and research labs).
Metrics of interest: Throughput, response time/bounded slowdown, makespan
for BoTs and normalized schedule length for workflows.
Typical problems: Increase throughput, reduce response time, and in particular
the (bounded) slowdown/makespan for BoTs and the (normalized) schedule length
for workflows. Balance performance and cost.
Typical self-aware elements: Traditional techniques for dynamic and adaptive scheduling and resource management. Flagship projects include Condor [124],
Globus [33], and, more recently, Mesos [55].

3.2.2 Compute-Intensive Batch Processing In and Across Datacenters
Application components: At the scale of entire datacenters and in multi-datacenter
environments, that is, on the order of 10,000 to over 100,000 machines, load is submitted by thousands of users. Workloads come from scientific computing, financial,
engineering, and other domains, and are dominated by bags of tasks of highly diverse size and resource demand [60].
Metrics of interest: User metrics are similar to those in the cluster context, but
also include aggregate measures of the fraction of deadlines and throughput goals
satisfied under extreme conditions such as large-scale failures and flash crowds.
Energy costs. Metrics interesting to datacenter managers include scalability, availability, load balance, and achievable utilization.
Typical problems: Load balancing, particularly across data centers. Handling
dynamic variations in load, both due to normal bursty behavior, time-of-day effects,
and failures. Enabling high utilization of resources without excessive impact on

10

Iosup et al.

performance and with performance isolation across the workloads many diverse
users.
Typical self-aware elements: Resource management that is aware of dynamic
loads, service-level objectives, and failure/maintenance issues. Examples include
automatic job queue reconfiguration [26], self-aware job managers [60], and largescale datacenter management systems [113, 126].

3.3 Data-Intensive Batch Processing
Description: There is a plethora of application domains including commercial applications, retail and science domains, that generate big data (large volume, high
variety, low veracity, etc.) Data-intensive batch processing involves systems to process sets of big data without an expectation of interactive data-processing sessions.
The essence of such systems is the processing by a cluster of compute nodes of
data that are typically stored on distributed storage, with intermediary results stored
in-memory or on disks local to each node. The nodes collectively execute software
that coordinates the distribution and computation of the data set across the nodes
according to the semantics of the processing. We categorise these systems into the
following two categories.

3.3.1 MapReduce-Based Data-Intensive Batch Processing
Application components: MapReduce is a popular programming model for developing and executing distributed data-intensive and compute-intensive applications
on clusters of commodity computers. A MapReduce job is an instance of a running
MapReduce program and is comprised of Map and Reduce tasks. Tasks are executed
according to the programming model, but embed functions (code) provided by the
user. High performance and fault-tolerance are two key features of typical MapReduce runtime environments. They are achieved by automatic task scheduling; data
placement, partitioning and replication; and failure detection and task re-execution.
The strength of MapReduce is in data-intensive batch processing. The MapReduce model has proven to be versatile in industry, where it is used for many Big Data
tasks including log processing, image processing, and machine learning. For example, MapReduce has been used to learn conditional probability tables of Bayesian
Networks (BNs). Both traditional parameter learning (complete data) and the classical Expectation Maximization algorithm (incomplete data) can be formulated within
the MapReduce model [8, 9].
Metrics of interest: Performance metrics such as job response time and throughput (jobs/minute), and data input and output (IOPS); reliability - measured as the
ratio of successful MapReduce job requests to the total number of requests; various
cost metrics; other low-level MapReduce metrics related to the number, length, and
status (i.e., success or failure) of MapReduce jobs and tasks.

Self-Awareness of Cloud Applications

11

Typical problems: Performance and dependability guarantees, trading-off between multiple metrics of interest. Chains and workflows of MapReduce jobs
are useful [52], but could be difficult to manage and troubleshoot. Vicissitude–
workflows of MapReduce jobs lead to diverse challenges, by stressing different system resources at different or even the same time [39]. Workloads can be dominated
by a few MapReduce jobs, used periodically or in bursts [13].
Typical self-aware elements: Performance models [115], performance management and guarantees [10], self-aware architecture reconfiguration [40].

3.3.2 Other Data-Intensive Batch Processing
Application components: Many programming models for data-intensive batch processing, and significantly different from MapReduce, exist today [92, 93, 132–134].
Such systems rely on functional, imperative, or dataflow models to express computations. For example, many systems implementing radically different programming
models compete in the batch graph-processing space [47].
Metrics of interest: Response time, recovery time, and cost. Some these systems [92, 134] are also designed to support low-latency results, to provide support
to process data and deliver results in near-real-time.
Typical problems: Handling workload variations and data recovery after machine failures. Synchonization of data state when processing spans many nodes. For
graph processing, the algorithm but also its input data set affect performance significantly, but predicting how is challenging. Auto-scaling is very challenging, due to
the possible need to transfer large state.
Typical self-aware elements: Recovering from failures. Scaling to additional
nodes to handle workload variations and arbitrary computation.

3.4 Data-Stream Processing
Description: We observe an avalanche of data continuously generated from various
sources such as sensor networks, business operations, web applications, and social
networks. There is a pressing need to process such data in real-time. For example,
several companies like Facebook and LinkedIn used to analyze their daily web logs
to better support their operations [111], but are shifting to real-time analysis.
Application Components: Data stream processing (DSP) involves the real-time
processing of data that are continuously generated from several distributed sources
at time-varying rates. Data analysis is represented via user queries that describe the
type of processing users wish to operate over source data. DSP queries are typically
represented by directed dataflow graphs where vertices correspond to operators and
directed edges indicate the flow of data among operators. Each operator typically
corresponds to certain parts of the query processing—often associated with well

12

Iosup et al.

defined semantics such as joins, aggregates, filters, etc. Finally, queries are deployed
on a cluster of nodes, referred to as data stream processing systems (DSPSs).
Metrics of interest: Primarily, query performance on a DSP is measured via
the delivery of low-latency and high-throughput results regardless of the workload
demands and their time-varying variations.
Typical problems: Dynamic workload and operator diversity (e.g., different semantics). Load balancing in fixed datacenter environments [37, 66, 128], architectural reconfiguration and performance isolation in cloud environments [12, 46, 106].
Typical self-aware elemets: Optimization models for resource allocation and
placement [66, 128] and CPU-based heuristics to trigger scale-out operations [12].

3.5 Workloads Generated by Datacenter Operations
Description: Unlike the other applications described in this section, this workload
is created by the system itself, as response to real-input workloads, in particular to
give probabilistic operational guarantees. Typical workloads here are the product
of backup, logging, checkpointing, and recovery systems. Although needed to meet
declarative specifications of the availability, durability, and recovery time requirements, these workloads cause significant reliability-related overheads that need minimization. For example, on the order of 20% of the resources are currently wasted
on failures and spent for recovery in large-scale infrastructure [23].

3.5.1 Addressing Failures in the Datacenter
Application components: Implementing data redundancy for availability and disaster tolerance results in some of the largest workloads in datacenters, especially
regarding data transfer and storage. Resources consumed may include disk and tape
capacity and bandwidth; CPU and memory for redundancy operations (such as encoding); and network bandwidth within and across datacenters.
Metrics of interest: Availability (e.g., expected fraction of time that a desired
data object will not be accessible); durability (annual data loss rates, MTTDL);
amount of data lost during failures (e.g., how far a checkpointed system will need
to rewind); recovery time ( e.g., how long it will take to recover back to a normal
operating state from a failure).
Typical problems: Recovery planning by selecting the combination of redundancy techniques to meet reliability requirements; designing a schedule of backup
operations to fit resource availability and limit interference with other workloads;
designing a schedule of recovery operations after a large-scale failure while enabling diagnosis (see Section 4.1).
Typical self-aware elements: Machine-learning-based techniques, especially
focusing on the monitored workload levels and changes, and component failure
rates. Automated designers combine multiple resilience techniques to meet reli-

Self-Awareness of Cloud Applications

13

ability and recovery requirements within resource limitations. Both deterministic and stochastic models [71, 98] are used to model the reliability provided by
the resilient system. Design methods include mathematical optimization and metaheuristic techniques [36, 69].

3.5.2 Addressing Failures at Exascale
Application components: The future exascale machines, which will exceed 1 exaflop sustained performance (so, 2-3 orders of magnitude larger than today’s Top500
machines) exacerbate the problems and pose important scale challenges to the aspects observed in Section 3.5.1. Checkpointing and other mechanisms designed for
this scale operate with waves or hierarchies of periodic or triggered operations, either partial or for the entire system, that affect CPU, memory, network, and storage
resources.
Metrics of interest: same as for Section 3.5.1; also energy and human resource
costs (e.g., does recovery require human resources?).
Typical problems: Recovery planning and general automatic recovery approaches
are the key challenge in the field, possibly aided by advanced workload prediction,
with current approaches leading to poor energy proportionality (high energetic cost).
Containment, including performance isolation is important, because correlated (e.g.,
cascading) failures can cause significant problems to other components and applications than affected by the original failure. Diagnosis and troubleshooting pose
important challenges, because at this scale applications can have over a million concurrent threads of execution and are very difficult to debug; even error identification
and reporting are important challenges in exascale systems.
Typical self-aware elements: Various methods, surveyed in a recent overview
of the field [120], among which: stochastic performance models, to trade-off recomputation of results for stored backups and checkpoints; selective checkpointing [99] and redundancy of execution [2], of all or of critical tasks, and on all or a
selection of more reliable resources, to reduce the effects of failures efficiently.

3.6 Online Gaming
Description: Hundreds of online games (OGs) entertain over 250,000,000 online
players, in a global market that generates over 30 billion Euros yearly. Massivizing,
which means to scale efficiently while meeting strict SLAs, is the biggest challenge of massively multiplayer online games (MMOGs). We consider here only the
resource management for the in-game virtual world, excluding external processes
such as gaming analytics (similar to Sections 3.3 and 3.4) and game-content generation (not standardized). There are many types of online games, among which the
most popular are online social games (OSG); First-Person Shooters (FPS) and Real-

14

Iosup et al.

Time Strategy games (RTS); and Massively Mutiplayer Online Role-Playing Games
(MMORPG).
Metrics of interest: In general, response time, cost of operation, performance
variability impact as aggregate performance penalty (own metric), time- and spacevarying reliability or availability.
Typical problems: Reduce cost. Balance cost-performance. Impact of performance variability. Unavailability at critical time or for the critical component.

3.6.1 Datacenter-based Approaches
Application components: The most popular OSGs (e.g., the FarmVille series and
Clash of Clans) have over 100 million daily active users, and hundreds of OSGs
attract over 1 million daily active users. OSGs use multi-tier web applications (described earlier) with hundreds of thousands to millions of concurrent yet short-lived
user sessions. Their populations can fluctuate significantly over time, especially during initial deployment and after their peak popularity is gone [100]. MMORPGs
(e.g., World of Warcraft, Destiny) commonly use geographically distributed clusters of servers to support (multi-)hour game sessions. FPS games, e.g., the Call
of Duty series, and RTS games, e.g., the StarCraft and DotA series, typically use
servers to run independent game instance that run for a few tens of minutes; often,
these servers are hosted by gaming-friendly datacenters.
Typical self-aware elements: Self-aware provisioning of resources from datacenters, especially in hybrid clouds, using workload prediction and modeling [94].
Cost-aware operation [95]. Portfolio scheduling [117]. Availability on demand [118].

3.6.2 Offloading of Mobile Interactive Applications
Application components: An emerging application in this space is that of mobile
games that use clouds as offloading target [21]. The structure of such an application is typically a workflow or dataflow, where tasks have inter-dependencies and
typically execute iteratively (the input-update-synchronize cycle common in game
design). Some or all tasks can be offloaded to the cloud. For example, cloud gaming applications offload all update tasks to the cloud and stream back to the mobile
device a video rendering of the current game status [83].
Metrics of interest: The general metrics, plus energy costs, and metrics relating
to more complex costs of operation (e.g., roaming and other special rates).
Typical problems: the general online gaming problems, plus focus on energy.
Typical self-aware elements: Applications may offload only (a part of) computation, data acquisition, or another resource-consuming part. Several, but not many,
feedback and reconfiguration techniques, and stochastic performance models to address which part to offload, where, and how exist [72, 83, 101, 127].

Self-Awareness of Cloud Applications

15

3.7 Partial and Delayed Processing
Description: Cloud applications have functional requirements on the computations
that they should perform and non-functional requirements on the additional properties that they should have. Some of these requirements can be expressed as incremental requirements on application behavior. For example, the accuracy of the
answer and the reliability of the operation could be bounded in stages. In partial
processing, applications can gradually downgrade user experience to avoid system
saturation.
Partial and delayed processing addresses such requirements. For example, online
shops offer end-users recommendations of similar products, which greatly increases
shopping experience and leads to higher sales—a study found an increase of 50% on
song sales when a group of users where exposed to recommendations [31]. However,
such engines are often highly demanding on computing resources [77], so using
a partial input instead of the entire database of choices could lead to acceptable
recommendations for both users and system. With delayed processing, applications
can extend their response time to cope with overload conditions [64] or perform
more efficient allocation plans in data center placement [86].
Metrics of interest: Typically, (bounded) response time: the latency experienced
by the users or the latency to deliver results in real-time data stream processing.
Also, reliability, and cost to produce the results.
Typical problems: In addition to traditional requirements, dynamic loads and
variable number of users, making dynamic resource provisioning necessary [108].
Unexpected events and potential failures, such as high load-peaks [11], software
and hardware failures [48, 96], and lack of performance isolation when workload
consolidation occurs [87].
Typical self-aware elements: For partial processing, enabling and disabling optional components on the fly [75,76], which leads to bounded response time [22,75]
and ability to address multiple failures [76]. Stochastic performance models can
help in analyzing the trade-offs of use of content vs. capacity requirements [79],
and of use of content vs. response time [97].
Stochastic performance models can also be applied to incremental processing [91] where the response is produced incrementally by successive refinements
of a preliminary answer, until the time limit is reached. In some cases, extending
the response time slightly (e.g., a few seconds every hour) is typically unnoticeable for users, but solving this optimization problem with flexible constraints can
significantly reduce the impact of overloads [64, 86].

3.8 Cyber-Physical Applications
Description: In cyber-physical system (CPS) applications [107], a computing system interacts with the physical world in some non-trivial manner. While originally
exemplified by closed embedded systems, such as industrial automation, the scope

16

Iosup et al.

of CPS applications has grown over the years to include larger open systems such
as disaster response [38], medical applications [80], energy management [58], and
vehicular control [74].
Metrics of interest: There are two often conflicting evaluation metrics for CPS
applications: safety and performance. They are typically at odds. Attaining higher
performance often requires greater coupling between components, but such coupling introduces complexity and pathways for failure propagation which may compromize safety.
Typical problems: A self-aware architecture should offer ways to meet both
safety and performance requirements despite the conflict between them. Adaptation
is needed to attain good trade-offs, as discussed in the examples below.

3.8.1 Medical Applications
Application components: Consider a future implanted smart insulin pump for a
diabetis parient. Multiple control mechanisms may be present that take measurements of activity levels from accelerometers and measurements of sugar levels in
the bloodstream to modulate insulin delivery. While inner control loops will run locally, there is an opportunity to perform some predictive optimization in the cloud
based on context derived from user location, synch’ed calendar, and other factors.
Decisions on where to execute which functionality at what time may be revisited
dynamically to adapt to different network and patient conditions, as well the current
control objectives (see Section 5.1 for details).
Metrics of interest: The safety requirement may specify that the pump shall
never overdose the patient (which could be fatal). A performance requirement may
specify that the pump should predictively adapt its output depending on the person’s
activity and food intake. The predictive aspect is key, because the human body has
too large of a response time for purely reactive (feedback) control schemes to offer
tight sugar regulation.
Typical Problems: Attaining better predictions requires exploitation of more
information. Acquisition of this additional information comes at the cost of having
to connect to other less reliable subsystems (e.g., the cloud), creating dependencies
that may act as pathways for failure propagation and hence safety violations. Hence,
a conflict manifests between performance and safety.
Typical self-aware elements: The need to reconcile safety and performance
gives rise to a new type of adaptation, where the system toggles between meeting
performance objectives and meeting safety objectives, depending on current state. In
the nominal (normal operation) state, the system optimizes performance. However,
when system approaches boundaries of safety, the objective switches to enforcement of safety, even if performance is affected. This approach is commonly referred
to as the Simplex architectural pattern [18]. For example, when the insulin pump
observes large deviations in patient’s blood sugar levels, it may switch to a simple
PID control mode based only on trusted local sensors and disconnect itself from the
less reliable cloud inputs that might be offering bad predictions.

Self-Awareness of Cloud Applications

17

3.8.2 Vehicular Applications
Application components: Vehicles are an interesting and emerging case study for
CPSs. Up until now, development has focused on making the individual more autonomous. While this trend continues, we believe there will be an increasing emphasis on (i) communication between vehicles of different degrees of autonomy;
(ii) careful sharing of CPS information resources; and (iii) system health management [14].
Metrics of interest: Safety requirements might be that a vehicle shall avoid hitting other vehicles or pedestrians; traffic signs shall be obeyed. Performance requirements could be to minimize fuel consumption and travel time.
Typical problems: Better performance may be achieved by exploiting global
information, typically from a cloud service. The service might include a database
of all traffic signs, estimates of current traffic conditions, and a schedule of traffic
signals. This information can be used to plan itineraries and driving speeds such
that fuel, trip time, and other passenger preferences are optimized. To ensure safety,
however, only reliable sensors that are local to the car should be used in making
decisions. However, these sensors have only a local view and may miss various
performance optimization opportunities.
Typical self-aware elements: To attain a good trade-off between safety and
performance, a local override mechanism is needed to take control when a safety
requirement is about to be violated. For example, if the perceived state at an approaching intersection is different from that reported by the cloud (e.g., light is red,
not green), the local override should take over and manage the car based on local
sensors only. The idea is to use the subset of most reliable information only, when
the system state is close to a safety violation boundary, while exploiting additional
less reliable sources for optimization in other states.

4 Types of Problems
In this section, we identify the following ten types of problems that affect cloud
applications, and that can benefit from the use of self-awareness techniques.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Recovery planning
Autoscaling of resources
Runtime architectural reconfiguration and load balancing
Fault-tolerance in distributed systems
Energy-proportionality and energy-efficient operation
Workload prediction
Performance isolation
Diagnosis and troubleshooting
Discovery of application topology
Intrusion detection and prevention

18

Iosup et al.

4.1 Recovery Planning
Context: Enterprise storage systems are designed to be resilient to failures, but
when a large failure occurs—for example, a datacenter level failure—recovery is
a complex process, and frequently involves some application downtime. It is important to recover the most important applications quickly; but What is the sequence of
recovery operations that will minimize the damage? [69] and What can prevent the
failure from reactivating? [120]
Problem: When an enterprise storage system experiences a large failure, critical
applications must be recovered quickly, even at the expense of additional downtime
for less important applications. Administrators are under stress, and have little time
to design the best sequence of recovery operations, and the default methods may be
far from optimal, since each failure can be different. Preventing the re-occurrence
of the failure, for example by isolating, reconfiguring, or micro-rebooting the failed
component, also requires careful planning. A self-aware recovery system that understands the failures, the criticality of the applications to be restored, and the possible
recovery options, can propose or even automatically implement a customized recovery plan.
Expected Advancement: By codifying the recovery operations, the cost of
downtime for applications, and what is needed to bring each application back up
so that future failures are also avoided, it is possible to model many possible recovery plans. An automated system can select an optimal schedule that balances
recovery targets with cost and resource waste.
Expected Impact on Application Types: A recovery planning optimizer should
be integrated with the overall mechanisms for managing backups, failures, and related workloads in datacenters (see Section 3.5.1). Clearly, the availability of resources (such as inter-datacenter bandwidth) impacts the recovery process, and
should be planned accordingly. In the other direction, the design of the backup
and restore mechanisms should be visible to the recovery planner in a way that enables introspection, modeling, and update when the mechanisms or the applications
change.

4.2 Autoscaling of Resources
Context: Many applications are subject to time-varying workloads. For instance,
workloads of internet and enterprise applications typically contain time patterns
(e.g., day vs. night, seasonal effects), long-term trends (e.g., increasing customer
base), and bursts (e.g., flashcrowds of interest for content).
Problem: As a consequence of time-varying workloads, sizing a system for the
expected peak workload is very inefficient (by orders-of-magnitude!) and may be infeasible if the workloads are hardly predictable. Therefore, a system should be able
to dynamically acquire and release resources (e.g., number of replicated VMs) as required for serving the current workload with a certain level of performance. Target

Self-Awareness of Cloud Applications

19

levels for the application performance may be specified in Service-level Agreements
(SLAs). Cloud computing provides the required flexibility to dynamically change
the amount of resources allocated to applications. However, scaling controllers in
state-of-the-art cloud platforms are following simple trigger-based approaches (e.g.,
if “utilization is above a given threshold, add one VM”), lacking knowledge about
the structure and behavior of the application. Moreover, few are able to respond to
rapid, bursty load transitions [35]. Scaling controllers when data processes require
non-trivial management, for example for big data processing (see Section 3.3) are
even more difficult to design [40].
Expected Advancement: A certain self-awareness of the system is required to
take acceptable or even optimal decisions about when to scale an application vertically or horizontally and by what amount of resources; about which part of an
application to scale if the application is comprised of multiple tiers, components or
tasks, or concurrent threads of execution; about which (part of the) application to
ensure against the risk of under-performance or failure; etc. Ideally, the self-aware
system will be able to enforce high-level objectives as specified in SLAs (e.g., endto-end response times and maximum risk of SLA breaches).
Expected Impact on Application Types: Autoscaling affects resource provisioning in both multi-tier enterprise applications (Section 3.1), and componentbased applications such as data-intensive batch processing (Section 3.3), data stream
processing (Section 3.4), and datacenter-based online gaming (Section 3.6.1). This
allows the auto-scaled applications to meet their quality of service goals in spite of
increased workload demands.

4.3 Runtime Architectural Reconfiguration and Load Balancing
Context: For many of the applications we describe in Section 3, the conditions they
are operating under can change at runtime. These include workloads from users or
other systems interacting with the application, resource usage pattens of the application itself, component failures from the hosting platform, as well as competing
demands from other applications sharing the same infrastructure.
Problem: In contrast to autoscaling, which keeps the same architecture while
changing the scale of the system, architectural reconfiguration is a family of techniques that reconfigure some architectural components of the overall system. When
workload changes occur, the current architecture of the system may become obsolete. For example, even if individual virtual clusters can auto-scale, the overall architecture of how the virtual clusters are deployed on the physical infrastructure also
need self-aware capabilities. A self-aware architecture should reconfigure and manage its components, for example by resizing its queues and changing their scheduling policies, or by changing the paths for sharing loads between different queues.
Another common problem in managing a cluster of resources (e.g., servers or
storage devices) is how to automatically balance the load across the cluster. This
involves migrating workloads from one server or device to another quickly and

20

Iosup et al.

without any downtime to the applications. Load blancing can also happen among
multiple clusters running similar or different workloads.
A further complication can arise in data centers with thousands of hosts and
services where some services have affinity or anti-affinity constraints. For example,
a service and its backup service should not reside on the same host (anti-affinity),
while it is perferred to have a user interface service and its corresponding DB service
on the same host (affinity). It is a challenge to maximize the utilization of the servers
while providing a high-quality user experience and not violating these constraints.
Expected Advancement: Runtime reconfiguration and load balancing typically
require solving an online optimization problem, whose objective involves specific
performance or availability metrics. First, for each type of applications, such metrics need to be identified, collected, and calculated in real time. Second, we need to
develop techniques for quantifying the cost of each reconfiguration step (e.g., an architectural change or a VM migration) and for weighing the cost against the benefit
of the reconfiguration. Thrid, we need frameworks for dealing with the fundamental
tradeoff between faster response and stability. Finally, advances on load rebalancing
optimization under affinity or anti-affinity constraints are expected, especially with
the ever increasing scale and complexity of such problems.
Expected Impact on Application Types: Runtime reconfiguration benefits especially applications who are not negatively impacted by the duration and other
costs of reconfiguration. Among such applications are batch processing applications, either compute-intensive (Section 3.2) or data-intensive (Section 3.3), and
some data-stream processing applications (Section 3.4).

4.4 Fault-Tolerance in Distributed Systems
Context: Due to ever-increasing scale and complexity, hardware and software faults
(which lead to errors, which may lead to a failure) in cloud computing infrastructures are the norm rather than the exception [5, 43]. This is why many from the
application classes introduced in Section 3 include fault-tolerance techniques, such
as replication, early in their design [49].
Problem: Failures in cloud infrastructures are often correlated in time and
space [34, 131], which means they a failure may affect tens to hundreds of nodes,
or even entire datacenters. Therefore, it may be economically inefficient for the service provider to provision enough spare capacity for dealing with all failures in a
satisfactory manner. When correlated failures occur, the service may saturate, i.e.,
it can no longer serve users in a timely manner. This in turn leads to dissatisfied
users, that may abandon the service, thus incurring long-term revenue loss to the service provider. Note that the saturated service causes infrastructure overload, which
by itself may trigger additional failures [15], thus aggravating the initial situation.
Hence, a mechanism is required to deal with rare, cascading failures, that feature
temporary capacity shortage. The main problem is to maintain bounded response
times in the presence of failure, while wasting an acceptable amount of resources

Self-Awareness of Cloud Applications

21

(today, about 20% of the entire capacity, but the goal for exascale systems is to waste
under 2% [120]). The problem of fault handling also includes a component about
preventing fault re-occurrence, which includes elements of diagnosis, troubleshooting, isolation, and (micro-)rebooting. [120]
Expected Advancement: Advances are expected in the use of control theory
with the brownout approach, in smart load-balancing using knowledge gained with
control, in self-checking and self-diagnosing, and in self-reconfiguration and in
smart decisions about micro-reboots. Using these techniques, the applications perform better at hiding faults from the user, as measured in the number of timeouts a
user would observe.
Expected Impact on Application Types: This problem affects request-response
applications (Section 3.7).

4.5 Energy-Proportionality and Energy-Efficient Operation
Context: A problem tightly related to autoscaling is one of energy proportionality.
Workloads in many applications are becoming more data-centric. In other words,
the data volume, and not the algorithmic complexity, is becoming the primary contributor to resource consumption. Single pass algorithms are used on most streaming
data, and their complexity is largely linear in the data size. Moving data across machines is therefore very expensive, compared to the cost of data processing.
Problem: It is hard to design systems where resources operate at capacity all the
time. Necessarily, some resources will not be fully utilized. Solution such as autoscaling could become prohibitively expensive if they involve frequent data movement. On the other hand, in the absence of autoscaling, some machines will not be
fully utilized. This operating mode exposes a problem with most current data center
hardware; namely, energy proportionality (or, rather, lack thereof). A server that is
only 30% utilized may be using 80% of the energy needed at full load. One needs
to design solutions where energy consumption shrinks proportionally to load.
Expected Advancement: Attaining energy proportionality in data-centric applications is challenging. It requires algorithms that minimize unnecessary data movement, while performing autoscaling. These algorithms must amortize cost of data
movement over time [82]. The latter may require a prediction of future data access
patterns [68].
Expected Impact on Application Types: Energy proportionality will benefit
both data-centric multi-tier enterprise applications and stream processing applications by allowing them to operate in a more energy efficient manner while minimizing the need for data movement.

22

Iosup et al.

4.6 Workload Prediction
Context: The increased volume of data involved in modern cloud applications suggests that initial data placement will play a big role in application performance and
energy consumption. Improper placement will create future load imbalance (e.g.,
if many popular items are collocated) or needlessly increase energy consumption
(e.g., if infrequently accessed items are placed together with some frequently accessed ones thus preventing machines from being turned off). There is a chapter
“Online Workload Forecasting” that discusses this issues in further detail, from both
a time-series and a machine learning perspective.
Problem: In (nearly) stateless services, such as serving requests for static Web
pages, a load balancer can rectify imbalance problems simply by distributing future
requests in a more equitable fashion. In applications where moving data is costly, it
is harder to predict computing load because such load has a substantial data affinity.
Hence, data placement dictates where computation runs. Getting the placement right
in the first place becomes important. This motivates techniques for predicting future
access patterns to data items at the time these items first enter the system and are
stored [68].
Expected Advancement: Self-awareness techniques that can understand and
represent efficiently the state of the system. Collecting and summarizing monitoring data at the scales expected for cloud computing infrastructures and workloads is
challenging, yet needed. Proper data access pattern prediction techniques will minimize the need for moving data unnecessarily, and hence improve both performance
and energy consumption of data centers. For example, data predicted to be of no
further interest could be moved proactively to servers that operate in more aggressive energy saving modes, or are in places that are harder to cool, hence saving
energy. Similarly, data predicted to be popular could be partitioned among a sufficient number of servers, reducing the chances of developing hotspots and needing to
relocate some of the data to other machines. To conclude, increasing awareness for
the lowest possible cost is an important trade-off that remains largely unexplored.
Expected Impact on Application Types: Most cloud computing applications
will benefit from some form of workload prediction.

4.7 Performance Isolation
Context: Support for multi-tenancy is an important feature of clouds. For example,
SaaS offerings are typically implemented by multi-tenant application architectures.
Multi-tenancy means that different tenants from separate organizations are sharing
the same application instance and see their own tenant-specific view of the data
and functionality. Thus, the operator of a SaaS provider can increase the efficiency
compared to running separate application instances.
Problem: The tenants of a cloud service may, unwillingly or even willingly,
affect the operation of the system and thus each other. If one tenant exceeds its

Self-Awareness of Cloud Applications

23

shared portion, or if the services are oversubscribed and the rightful tenants access
the service simultaneously, the performance as experienced by the other tenants can
fluctuate or even depreciate significantly. Likely because of (lack of) performance
isolation, SaaS, but also PaaS and IaaS clouds, can experience high variability in the
performance of their service [61].
Expected Advancement: To ensure performance isolation in such disruptive
scenarios, per-request admission control is required, that automatically throttles
users exceeding their quota to avoid breaking the SLAs of other tenants.
Expected Impact on Application Types: Multi-tier enterprise applications in a
cloud environment need both performance isolation among different application instances (e.g., on IaaS) [84, 103] or different tenants of the same application instance
(e.g., on SaaS) [78].

4.8 Diagnosis and Troubleshooting
Context: When an application or service goes down or fails to reach the service
level objective (SLO) regarding its end-to-end performance, one needs to engage in
the process of diagnosis and troubleshooting.
Problem: The key problem is to identify the faulty component that has caused the
failure, or the associated performance bottleneck that has led to the service degradation. This can be challenging due to the increasingly more complex and distributed
nature of modern applications, their growing space of configurations, the typically
time-varying workload demands, and the applications’ dependency on a variety of
hardware resources, such as processors, memory, storage and network I/O devices,
as well as software resources, such as locks, threads, and connection pools.
Expected Advancement: Traditionally, maintenance personnel, system administrators or datacenter operators perform diagnosis and troubleshooting manually,
using a combination of logs, performance charts, best practices menus, and their domain knowledge, which is time consuming and error prone. With the utilization of
self-awareness techniques, we should build management services that can automatically determine the likely causes of failures or performance problems [73,129,130].
Expected Impact on Application Types: Applications from enterprise multitier systems to networked cyber-physical front-ends can benefit from automatic diagnosis and troubleshooting, resulting in shorter durations of failures or servicelevel violations, and reduced cost in management and operations. Prior work addresses system health management, including diagnostics and prognostics capabilities [88, 110, 112], application troubleshooting [73], and troubleshooting uncoordinated self-aware managers [54].

24

Iosup et al.

4.9 Discovery of Application Topology
Context: Automatic discovery of application topology or runtime architecture is a
required feature for any mature application performance monitoring or management
solutions, according to the Gartner APM Conceptual Framework [109].
Problem: The problem here is to automate the process of identifying the relationship and dependency among individual application components at runtime, as
well as how they are mapped to the physical infrastructure (e.g., servers), with no or
only minimum input from human operators.
Expected Advancement: To solve the above problem, one needs to implement
real-time, fine-grained tracing of individual transactions as they traverse through the
execution paths of the application. Such monitoring solutions can be passive [3] or
require intrumentation at the kernel, middleware, or application level [7]. Statistical
techniques for inferring correlations or discovering dependencies are often needed.
Expected Impact on Application Types: Having access to an accurate application topology can help diagnose or debug performance degradations and discover
hidden performance bottlenecks in multi-tier enterprise applications (Section 3.1.1)
during their operation [3, 7] or help identify potential root causes of observed failures through event correlation.

4.10 Intrusion Detection and Prevention
Context: In cloud environments, different applications coming from diverse organizations may share the same physical resources. Depending on the cloud service
model (IaaS, PaaS, or SaaS), the data-center owner has different levels of control
over the executed application and their configuration. Vulnerabilities in the infrastructure software (e.g., hypervisors) or in shared services (e.g., storage services) can
be exploited to widen an attack from any application to any other virtual machine in
the same infrastructure. For instance, attackers may rent virtual machines (in case
of public clouds) or exploit vulnerabilities of applications (e.g., private website) to
get access to sensitive data (e.g., e-commerce system with credit card data) in other
virtual machines in the same cloud environment.
Problem: The detection and prevention of attacks in a cloud environment requires a classification of cloud workloads (either using application or network
probes) into benign and malicious ones. False positives can result in unnecessary
actions countering attacks, which may have negative effects on the applications.
Expected Advancement: A self-aware system automatically learns to distinguish between benign and malicious workloads and can filter out false positives.
Furthermore, the system is able to react to attacks and adapt itself to ensure its
self-protection capabilities. For more details on quantifying the self-protection capabilities of self-aware systems we refer the reader to Chapter 6.1.
Expected Impact on Application Types: All application types running in a
cloud are potential targets for attacks. Prior work in this area is aimed at enabling

Self-Awareness of Cloud Applications

25

self-healing capabilities of systems [42, 123] and automatic learning of workload
anomalies [59].

5 Types of Approaches
In this section, we identify and analyze the following seven types of self-awareness
approaches used in practice to address the problems identified in the previous section.
1.
2.
3.
4.
5.
6.
7.

Feedback control-based techniques
Metric optimization with constraints
Machine learning-based techniques
Portfolio scheduling
Self-aware architecture reconfiguration
Stochastic performance models
Other approaches

5.1 Feedback Control-Based Techniques
Description: Control theory is a branch of mathematics that studies how to influence the behavior of dynamical systems [81]. Based on a formal model of the target
system in the form of equations (for time-based control) or automaton (for eventbased control), control theory provides principles for how to synthesize a controller
that would regulate the behavior of the system and obtain prescribed properties.
Expected Impact: Although control theory was invented to deal with physical
systems, the same principles have been successfully applied to many different application domains in computing systems [53], including resource allocation [65,84,85],
application performance via bounded response times [64, 75], reliability [28], and
fault tolerance [12, 76], stream processing [66], and big data [27].
Details: A controller should provide the following properties [20, 28]:
• Setpoint Tracking. A setpoint refers to the goals to be achieved. For example, a
system is considered responsive when its user-perceived latency is sub-second.
• Transient behavior. This concerns how the setpoint is reached by the system.
• Robustness. The controller should be able to cope with inaccurate measurements,
delayed data, or other uncertainties not captured in the system model.
• Disturbance rejection. The closed-loop system should reach its goal in spite of
other disturbing actions happening simultaneously in the system.
These properties are often translated into the corresponding control properties of
stability, no overshooting, quick settling time and robustness to model errors and
disturbances [30].

26

Iosup et al.

Use Cases: There have been many published studies of developing self-awareness
capabilities using feedback control based techniques.
One broad application area has been resource management in computing systems. For example, in [65], an approach is described to optimally configure the
CPU resource entitlement of virtualized multi-tier web applications. The approach
applies the Kalman Filter to predict CPU resource utilizationof the application components in the next interval based on previous observations. The controller selfadapts to changing workload conditions by self-configuring its parameters and capturing the resource coupling within a multi-tier application. In follow-up work [63],
a control-based approach is presented to assign CPU resource shares of virtualized
web server applications. This paper emphasizes the CPU allocation around periods
of workload changes and uses H∞ filters to minimize the maximum controller error.
More recently [114], a control-theoretic approach was described to provide performance, dependability and cost guarantees for online cloud services, with timevarying workloads. The approach is validated through case studies and extensive
experiments with online services hosted on Amazon EC2. One case study demonstrates SLA guarantees for a cluster-based multi-tier e-commerce service. In [84],
application managers automatically learn a quantitative model that correlates applevel performance with resource utilizations and use control theory to derive the
optimal resource control settings (limits, reservations) for individual virtual machines such that the multi-tier application can achieve its performance target. Finally, in [85], the authors propose a formal way to automate core allocation based
on deadline metrics from the application. The approach provides formal guarantees
on the application behavior in spite of external disturbances like additional load on
the machine. The approach has also been generalized and automated [29], applying
it to other actuation mechanisms and other metrics such as energy and size of compressed videos. This shows the versatility of feedback control as an approach for
building self-aware and self-adaptive systems.
Another broad application area for feedback control is cyber-physical systems [116]
(see Section 3.8). Feedback control is central in smart grids [25], intelligent transportation [135], and modern critical care units [67] to name a few. Parameters and
offsets of local loops might be obtained from remote repositories [51]. The criticality of these applications typically requires continued, correct operation even in the
presence of connectivity problems to the cloud, bad data, and other failures, which
often leads to joint investigation of control and safety [104].

5.2 Metric Optimization with Constraints
Description: System design, configuration, and management decisions often come
down to a choice between various options. Self-aware systems can reason about the
impact of different choices, but that still leaves the question of which choice best
achieves the system goal. Optimization techniques require first a formal specification of the system objective and the constraints under which the system operates,

Self-Awareness of Cloud Applications

27

and then a solution that attempts to optimize the objective. Solution methods can be
exact or approximate, depending upon the situation, but the explicit specification of
goals is essential.
Expected Impact: Using optimization techniques for system decisions has several advantages [70]. First, it guarantees clarity: system decisions often come down
to choosing between conflicting goals, and specifying an objective forces the designer to explicitly choose how much weight to assign to them. Secondly, when
approximate solution techniques are used. It enables a quantitative evaluation of
how far the results are from the optimal. Thirdly, it allows use of the enormous toolbox of optimization techniques that already exists, allowing the system designer to
focus on the design aspects.
Details: System designer sometimes use ad hoc heuristics to make system decisions, on the basis that specifying there are conflicting, sometimes intangible requirements that are hard to quantify, and the heuristics produce results that are “good
enough”. However, it is then unclear what “good enough” means, or if the heuristics
in fact achieve it. Expressing the trade-offs into a common currency – whether execution time, throughput, monetary cost, or a utility function – enables a definition of
goodness. A wide variety of optimization techniques can be applied, depending on
the formulation and requirements: mathematical optimizations, such as linear programming or mixed integer programming; constraint programming, if only a feasible solution is required; meta-heuristic techniques such as simulated annealing or
genetic algorithms, when the optimization problem cannot be solved exactly in the
available time; or other approximate optimization techniques that bound the error in
the resulting solution.
Use Cases: Examples of the use of optimization in systems include Maestro [90],
where online optimization was used with feedback-control to provide performance
differentiation between applications in a disk array, and Janus [4], where offline optimization was used to determine allocation of flash resources to workloads. Metaheuristic techniques have been used for finding the sequence of recovery operations
to use after a failure to minimize the cost of the downtime [69] and for designing a
redundancy configuration for large enterprise storage systems, in order to minimize
the overall cost of the system including operating overheads and potential downtime
costs.
Another example from the commercial world is the VMware Distributed Resource Scheduler (DRS) [45], a widely used feature in the VMware vCenter management software. DRS manages a set of virtual machines running on a cluster
of physical hosts and performs dynamic load balancing to avoid hot spots and improve application performance, by solving an online optimization problem with hillcliming heuristics and by taking into account the cost/benefit tradeoff of each move.
Finally, this approach can be applied to the applications described in Section 3.4.
For example, the SQPR query planner [66] allocates physical resources of heterogeneous clusters to data stream processing queries. SQPR models query admission,
allocation and stream reuse as a single constrained optimization problem and solves
an approximate version to achieve scalability. The SQPR approach monitors the resource utilization across the cluster and performance progress of running queries to

28

Iosup et al.

decide on queries’ placement and allocation. SQPR adapts to operating conditions
through continuous monitoring and modelling of cluster and queries performance.
SQPR uses an off-the-shelf optimization solver for optimal solutions.

5.3 Machine Learning-Based Techniques
Description: Machine learning is the science of using data to “uncover an underlying process” [1]. More specifically, it involves designing, implementing, and validating a set of algorithms that can extract insights from data regarding the relationships
among objects and events, often captured in the form of statistical models.
Expected Impact: Besides the successful application of machine learning to
financial applications, e-commerce, and medical applications, there has been a great
deal of research in the past decade on leveraging machine learning based techniques
to creating self-awareness for business critical applications (Sec. 3.1).
Details: Supervised learning, unsupervised learning, and reinforcement learning
are the three main types of machine learning approaches. They are commonly applied to perform clusting, classification, and prediction, using a variety of statistical
models, including decision trees, regression models, neural networks, Bayesian networks, or support vector machines.
A system that utilizes machine learning typically has the following components:
• Sensors. Software or hardware modules that measure and collect metrics of interest for the target system or process.
• Preprocessor. Raw data collected from systems are rarely perfect. It is not uncommon to have missing data from certain components or during certain periods
of time, or data corrupted during collection. Such data need to be “cleaned up”
or aggregated before being fed into an analysis engine.
• Analyzer. This is where statistical learning algorithms are being run, on top of the
collected data, to extract relationships, and to built models that can represent the
learned behavior in a concise form.
• Reporter/Predictor. This is where the insights gained from data are presented
to human operators in the forms of alerts, charts, or dashboards. Alternatively,
the model learned can be used to generate predictions for the target system or
metrics, or to recommend remediation actions.
Use Cases: In [16], Tree-Augmented Bayesian Network (TAN) models are
learned on top of instrumentation data collected from a three-tier Internet service
to automatically identify the top system-level metrics that have likely contributed to
the observed violation of service level objectives (SLOs).
In [102], an autoscaling system employs reinforcement learning to automatically
learn the performance behavior of a multi-tier application when instances of an elastic tier are added or removed, and then uses the knowledge to scale the application
horizontally as its workload demand varies over time.

Self-Awareness of Cloud Applications

29

5.4 Portfolio Scheduling
Description: Traditional scheduling policies are designed a specific workload, and
sometimes even for specific applications within a workload; in practice, reuse of old
and adoption of new scheduling policies happens rarely. Instead, portfolio scheduling, which is a self-aware and self-expressive technique, considers a set (portfolio)
of scheduling policies, from which it selects at the appropriate moments (e.g., periodically) the policy which promises the best results for the current and expected
conditions. In this way, the portfolio scheduler combines the strengths of each individual policy in its portfolio, which also means that designers of scheduling policies
can focus on simpler policies that do not need to address every type of workload.
Expected Impact: Portfolio schedulers promise to deliver performance at least
as good as any of their constituent policies, without poor performance when the
workload changes. In this context, performance includes traditional performance
metrics, such as application response time and system utilization, and non-traditional
metrics, such as availability, cost-performance efficiency [19], and risk of SLA violations [125], etc.
Details: The concept of portfolio scheduling derives from economics, where
stock brokers can use portfolio theory to select policies for managing their stocks, to
balance risks and rewards. For cloud applications, portfolio scheduling uses the following four-stage iterative process. In the configuration stage, the portfolio scheduler is equipped with a set of scheduling policies. Then, the portfolio scheduler
goes through a selection stage, which results in the selection of an optimal policy;
through an application stage, which results in applying the optimal policy to the
current scheduling problem (queue) and in monitoring the results; and through a
reflection stage, where stale policies are possibly eliminated and the portfolio can
compare its operation relative to the goals of the system. The next cycle can be triggered periodically or, if enough resources allow for timely completion of stages,
whenever an even can lead to system reconfiguration (e.g., at the arrival of a new
request in the system).
Use Cases: Portfolio scheduling has been used for business-critical applications,
with a focus on reducing the risk of SLA violations [125]. The selection stage is
simulation-based, online, and applied after each arrival of a job and periodically. The
optimal policy, from the policies included in the portfolio, is then applied until the
next selection occurs. The results obtained by applying this portfolio scheduler on
the workloads of a real cloud provider indicate that this portfolio scheduler is better
than its constituent policies, but also that the initial configuration of the portfolio is
very important.
Portfolio scheduling has been used for onling gaming applications, but only
tested under realistic, yet laboratory conditions [117]. Here, the portfolio is configured with various typical online scheduling policies, but also with an optimal solver
of a linear integer problem; in the selection stage, the portfolio is given a limited
amount of time to decide, so the optimal solver is stopped if it exceeds the allocated
runtime.

30

Iosup et al.

5.5 Self-Aware Architecture Reconfiguration
Description: Self-aware architecture reconfiguration is a family of techniques that
can re-configure at runtime the architecture of the system or of the application.
Expected Impact: The generality of this approach is an advantage, but also
makes the actual impact difficult to predict in advance. By affecting the essence
of the entire system, this approach can affect every metric and virtually every application. When applied to applications, this approach can change the operational
characteristics of the application. This approach addresses the runtime architectural
reconfiguration problem.
Details: By monitoring the environment and the (queued) workload, by predicting the performance of the system or application, and by reflecting on the overall
goals of the system or application, self-aware architecture reconfiguration leads to
changing of operational characteristics (queue size and policy, structure of application components), but also the way in which the system or application operate (e.g.,
how queues share load).
Use Cases: Koala-C [26], which services the Dutch research cloud DAS4, creates a system with multiple queues, which can be instructed to shed load to only one
other or a group of other queues. Each queue is aimed to run jobs of a specific runtime (job size). If a job scheduled by a queue does not finish in the time allocated
for that queue (exceeds the job size for that queue), it is stopped and moved to a
queue of larger size. The approach does not require any prior knowledge about the
input workload. Instead, each job is submitted to the queue(s) servicing the shortest job; larger jobs traverses progressively the chain of queues, with probabilistic
guarantees in terms of performance and wasted resources [50]. (Queues can also be
equipped with autoscaling mechanisms.) Another use case [56] is that of the cloud
provider Blue Yonder: through a parameterized application performance model, a
set of modeled adaptation points, and an adaptation model, the Blue Yonder applications can adjust their resources to workload changes and can share resources
between customers.

5.6 Stochastic Performance Models
Description: Stochastic performance models (e.g., Queueing Networks) enable the
prediction of the expected system performance in terms of throughput, response
time and utilization for a given workload. In a self-aware system, these models are
automatically created and maintained in the learning phase.
Expected Impact: In contrast to machine-learning or feedback control based
techniques, they promise the ability to predict also the performance for previously
unseen workloads or configurations which are significantly different to the current operating point. Stochastic performance models have been applied to dynamically reconfigure enterprise multi-tier applications for auto-scaling [121] and performance isolation [78].

Self-Awareness of Cloud Applications

31

Details: Classic queueing networks model the computing at processing resources
(e.g., CPU, hard disk, network). Furthermore, extended formalisms, such as Layered Queueing Networks or Queueing Petri nets, are able to capture the influence
of passive resources (e.g., software or memory resources). In Chapter 4.1, a more
extensive discussion of stochastic performance models can be found.
Use Cases: In [78], the authors describe a request admission controller for multitenant applications (e.g., SaaS application) that enables performance isolation and
SLA differentiation between tenants. Using a combination of statistical estimation
of resource demands and operational analysis from queueing theory, the admission
control is able to determine the current resource usage of individual tenants and
determine which requests are admitted and which are delayed.
In [121], a controller for vertical CPU scaling of virtualized applications is presented based on a Layered Queueing Network. The model is dynamically parameterized based on monitoring data from the system and also captures contention
effects due to hypervisor scheduling. By using the performance model, oscillating
reconfigurations are avoided and the parameters can be estimated automatically at
runtime.

5.7 Other Approaches
A number of other approaches have been developed by related fields, such as design
science [105]. Although these approaches do not share the entire spectrum of characteristics of, for example, control-based systems, they can be seen as proto-selfawareness approaches. We enumerate in the following several of these approaches.
Other approaches for auto-scaling: For data-intensive batch processing, which
typically has large state, auto-scalers such as Amazon Elastic MapReduce and
FAWKES [40] use various types of system feedback to decide on scaling. Amazon’s approach is based on the topology-aware S3 storage. Closer to application semantics, FAWKES considers various types of VMs, including VMs that store data
transiently or permanently. These approaches consider the MapReduce programming model, which has not been designed to enable interactive processing (e.g., for
fast decision making) or tightly coupled data items (e.g., for graph processing).
In the area of data stream processing, systems such as Apache S4 and Storm
exploit intra-query parallelization to scale out operators and eventually handle demanding resource requirements per operator. There are two main challenges in operators’ parallelization to support a scalable DSPS, i.e., how to handle operators’
state and when to scale out/in. Existing research has shown that simple heuristics to
detect violations on resource-based thresholds can be used to horizontally scale out
to additional stream processing operators to handle excess load [12, 46]. When handling operator state, most systems focus on the parellization of stateless operators.
In [12,106] different generic approaches to manage the parallelization of both stateful and stateless operators are introduced. In [27] an advanced approach to handle
operators state alongside specialized programming primitives is discussed.

32

Iosup et al.

Other approaches for runtime architectural reconfiguration and load balancing: Better load balancing within a cluster of compute nodes [45] or storage
nodes [44], or between clusters [26], can lead to avoiding hot spots and to improving the performance of many types of applications, including compute-intensive scientific applications (Section 3.2), business-critical applications (Section 3.1), and
batch-processing applications (Section 3.2). Reconfiguring the system partitions,
not only in scale but also in the way they interact, can greatly benefit batch workloads where long and short jobs can coexist [26]. A genetic algorithm for load rebalancing (GALR) was proposed to deal with affinity or anti-affinity constraints [122].
In experiments running the PlanetLab dataset in a cluster up to 300 hosts, GALR
performs close to optimality in load rebalancing, within 4-5 seconds.
Other approaches for fault-tolerance in distributed systems: Experimental
results demonstrated that using brownout [75] and smart load-balancing [22] application can tolerate more replica failures and that the novel load-balancing algorithms improve the number of requests served with optional content, and thus the
revenue of the provider by up to 5%, with high statistical significance [76]. Various
self-testing and self-diagnostic techniques exist [41] (in fact, SMART is a commercial technology for hard-disks), but self-repair through self-reconfiguration and
self-rebooting is still a largely open field.

6 Open Challenges for Self-Aware Cloud Applications
In this section we focus on identifying and analyzing directions for future research
in developing self-aware cloud applications. We ask questions that we find challenging, yet promising, that is, whose answers we foresee being put in practice in the
next 3-5 years.
To what extent are self-awareness techniques necessary for cloud applications?
Currently, non-self-aware techniques are prevalent in the management of increasingly large cloud datacenters and their applications. Uncertainty about the need and
possible gains limit adoption of of self-awareness techniques for this setting. Anecdotal evidence such as that gathered in this chapter and the existing small controlled
experiments conducted by scientists provide some evidence of the benefit of using
self-awareness techniques, but on their own still cannot provide the needed strong
evidence. Instead, inspired by the evolution of related fields, it would be beneficial to collect and share many operational traces from real-world deployments into
an open-access Trace Archive, and to provide a layer of fundamental understanding and knowledge by analyzing and sharing quantitative information such as the
frequency of self-aware decisions and metrics about their impact. Large-scale experimental comparisons of self-aware and non-self-aware techniques, published as
both technical material and open-access data, could provide a useful complement
and further the acquisition of fundamental knowledge.
How can self-aware computing and communications enable or improve upon
emerging applications for increasingly capable mobile devices — including smart

Self-Awareness of Cloud Applications

33

phones, wearable devices, and IoT devices? Today’s cloud architecture, which relies on the use of a “dumb pipe” sitting between it and a smart edge including increasingly capable mobile devices, is likely to be a productive area for future research [89]. In particular, there is a fundamental limitation in how much power a
wearable or hand-held device can consume before it becomes “too hot to wear”
or “too hot to handle.” Consequently, off-loading to an external computing platform, currently a cloud, becomes attractive. However, given the current Internet and
Telecom system architectures, data from apps need to cross a dumb pipe before it
reaches the cloud. This can be a severely limiting factor, for example when it comes
to supporting emerging applications that need low-latency, high-bandwidth communications. Examples of such emerging applications include rich media (including
video), unmanned vehicles, and gaming.
How can self-aware computing learn and maintain knowledge of itself if it is subject to frequent releases? The development of many popular Internet applications
(e.g., Facebook, Netflix) is characterized by very short release cycles (e.g., daily).
The term DevOps is often used to describe approaches and processes to align development and operation of software systems with the goal of continuous delivery of
new versions with new or changed functionality. As a result, such software systems
are often in transient phases where different versions of software run in parallel.
How can self-aware computing learn the characteristics of the workloads and
update the allocation of their resources in a distributed manner with global coordination for efficient use of data center resources? Current cloud data centers span
thousands of physical servers and host tens of thousands of virtual machines running different workloads. Managing such large systems while satisfying individual
workload performance requirements and making efficient use of cluster resources
is an open challenge. Current scalable approaches use either distributed scheduling [55] or optimistic resource allocation using shared state [113]. However, such
approaches cannot guarantee global scheduling optimality with respect to multiple
(possibly competing) goals across all resources and workloads in the data center.
Can self-aware computing offer techniques for fully automated root cause analysis (RCA) that can be applied to a variety of management operations in the cloud?
Over the past decade, the wide deployment of monitoring solutions in data centers and access to real time telemetry have advanced the art in diagnosis and troubleshooting. At the same time, automated RCA still remains an unsolved puzzle
for operators. This is mainly due to the large number of hardware (compute, storage, networking) and software (OS, hypervisor, container, middleware, application)
components that can potentially contribute to an observed failure or performance
drop, and the complex interactions among them. Furthermore, typical statistical
analysis and learning approaches discover correlations not causality between different metrics or events, and hence can only provide hints for the real root cause.

34

Iosup et al.

7 Conclusion
Cloud computing and its applications are already an important branch of ICT, with
interesting benefits and challenges. Due to sheer scale, but also to the increasing
sophistication of both their stakeholders and their infrastructure, clouds and their
applications are increasingly relying on self-aware management techniques. In this
chapter we have proposed a systematic framework to explain existing self-awareness
approaches, and to facilitate the analysis of self-awareness in the future.
Our framework proposes a structured way to analyze the types of self-awareness
approaches used in practice, in cloud computing and its applications. The framework
focuses on the types of applications, of problems, and of approaches relevant to selfawareness. The framework also structures the discussion and the analysis of open
challenges.
In this chapter, we have used the framework to analyze seven types of selfawareness approaches used in practice: feedback control based techniques, metric optimization with constraints, machine learning based techniques, portfolio
scheduling, self-aware architecture reconfiguration, and stochastic performance models. We conduct a systematic survey of self-awareness techniques that spans over
100 contemporary references. We focus on eight types of applications, among which
most are already established, whereas applications such as online gaming, partial
processing, and cyber-physical applications are still emerging. We analyze ten types
of traditional and novel problems. Novel, we focus on problems that have developed beyond their traditional scope or even emerged altogether in the space of cloud
computing, such as autoscaling, energy-proportionality, performance isolation, and
intrusion detection and prevention. We also identify four open challenges for selfawareness in cloud computing and its applications.
The future of this work is in facilitating, for the authors and for the self-awareness
community at large, work addressing the open challenges. We also hope the framework will be used for analyzing other self-awareness approaches, new problems,
and new applications.

References
1. Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. Learning From Data.
AMLBook, 2012.
2. Orna Agmon Ben-Yehuda, Assaf Schuster, Artyom Sharov, Mark Silberstein, and Alexandru
Iosup. Expert: Pareto-efficient task replication on grids and a cloud. In IPDPS’12, 2012.
3. Marcos Aguilera, Jeffrey C. Mogul, Janet L. Wiener, and Patrick Reynolds. Performance
debugging for distributed systems of black boxes. In SOSP, 2003.
4. Christoph Albrecht, Arif Merchant, Murray Stokely, Muhammad Waliji, François Labelle,
et al. Janus: Optimal flash provisioning for cloud storage workloads. In USENIX ATC, 2013.
5. Luiz Andre Barroso and Urs Hölzle. The Datacenter as a Computer: An Introduction to the
Design of Warehouse-Scale Machines. Morgan & Claypool, 2009.
6. Jean Arnaud and Sara Bouchenak. Performance and Dependability in Service Computing,
chapter Performance, Availability and Cost of Self-Adaptive Internet Services. IGI, 2011.

Self-Awareness of Cloud Applications

35

7. Paul Barham, Austin Donnelly, Rebecca Isaacs, and Richard Mortier. Using Magpie for
request extraction and workload modelling. In OSDI, 2004.
8. Aniruddha Basak, Irina Brinster, Xianheng Ma, and Ole J. Mengshoel. Accelerating
Bayesian network parameter learning using Hadoop and MapReduce. In Proc. of BigMine’12, 2012.
9. Aniruddha Basak, Irina Brinster, and Ole J. Mengshoel. MapReduce for Bayesian network
parameter learning using the EM algorithm. In Proc. of Big Learning: Algorithms, Systems
and Tools, 2012.
10. Mihaly Berekmeri, Damian Serrano, Sara Bouchenak, Nicolas Marchand, and Bogdan Robu.
A control approach for performance of big data systems. In IFAC World Congress, 2014.
11. Peter Bodik, Armando Fox, Michael J. Franklin, Michael I. Jordan, and David A. Patterson.
Characterizing, modeling, and generating workload spikes for stateful services. In SOCC,
2010.
12. Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. Integrating scale out and fault tolerance in stream processing using operator state management.
In SIGMOD, 2013.
13. Yanpei Chen, Sara Alspaugh, and Randy H. Katz. Interactive analytical processing in big
data systems: A cross-industry study of mapreduce workloads. PVLDB, 5(12):1802–1813,
2012.
14. Arthur Choi, Adnan Darwiche, Lu Zheng, and Ole J. Mengshoel. A tutorial on Bayesian
networks for system health management. In A. Srivastava and J. Han, editors, Data Mining in Systems Health Management: Detection, Diagnostics, and Prognostics. Chapman and
Hall/CRC Press, 2011.
15. Edward Chuah, Arshad Jhumka, Sai Narasimhamurthy, John Hammond, James C. Browne,
and Bill Barth. Linking resource usage anomalies with system failures from cluster log data.
In SRDS, 2013.
16. Ira Cohen, Moises Goldszmidt, Terence Kelly, Julie Simons, and Jeff Chase. Correlating
instrumentation data to system states: A building block for automated diagnosis and control.
In OSDI, 2004.
17. Louis Columbus. Roundup of cloud computing forecasts and market estimates, 2015. Forbes
Tech Report, 2015.
18. Tanya L. Crenshaw, Elsa Gunter, C.L. Robinson, Lui Sha, and P.R. Kumar. The simplex
reference model: Limiting fault-propagation due to unreliable components in cyber-physical
system architectures. In RTSS, 2007.
19. Kefeng Deng, Junqiang Song, Kaijun Ren, and Alexandru Iosup. Exploring portfolio
scheduling for long-term execution of scientific workloads in iaas clouds. In SC, 2013.
20. Yixin Diao, Joseph L. Hellerstein, Sujay Parekh, Rean Griffith, Gail E. Kaiser, and Dan
Phung. A control theory foundation for self-managing computing systems. IEEE J. on
Selected Areas in Communications, 23(12):2213–2222, 2006.
21. Hoang T. Dinh, Chonho Lee, Dusit Niyato, and Ping Wang. A survey of mobile cloud
computing: architecture, applications, and approaches. Wirel. Commun. Mob. Comput.,
13(18):1587–1611, 2011.
22. Jonas Dürango, Manfred Dellkrantz, Martina Maggio, Cristian Klein, Alessandro Vittorio Papadopoulos, et al. Control-theoretical load-balancing for cloud applications with
brownout. In CDC, 2014.
23. E.N. Elnohazy et al. System resilience at extreme scale. White paper. Defense Advanced
Research Project Agency (DARPA) report, 2009.
24. European Commission. Uptake of cloud in europe. Final Report. Digital Agenda for Europe
report. Publications Office of the European Union, Luxembourg, 2014.
25. Xi Fang, Satyajayant Misra, Guoliang Xue, and Dejun Yang. Smart grid; the new and improved power grid: A survey. Communications Surveys Tutorials, IEEE, 14(4):944–980,
2012.
26. Lipu Fei, Bogdan Ghit, Alexandru Iosup, and Dick H. J. Epema. KOALA-C: A task allocator
for integrated multicluster and multicloud environments. In CLUSTER, 2014.

36

Iosup et al.

27. Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch.
Making state explicit for imperative big data processing. In USENIX ATC, 2014.
28. Antonio Filieri, Carlo Ghezzi, Alberto Leva, and Martina Maggio. Self-adaptive software
meets control theory: A preliminary approach supporting reliability requirements. In ASE,
2011.
29. Antonio Filieri, Henry Hoffmann, and Martina Maggio. Automated design of self-adaptive
software with control-theoretical formal guarantees. In ICSE, 2014.
30. Antonio Filieri, Martina Maggio, Konstantinos Angelopoulos, Nicolás D’Ippolito, Ilias
Gerostathopoulos, et al. Software engineering meets control theory. In SEAMS, 2015.
31. Daniel Fleder, Kartik Hosanagar, and Andreas Buja. Recommender systems and their effects
on consumers: the fragmentation debate. In EC, 2010.
32. Andrew Forward and Timothy C. Lethbridge. A taxonomy of software types to facilitate
search and evidence-based software engineering. In Conference of the Centre for Advanced
Studies on Collaborative Research, page 14, 2008.
33. Ian T. Foster. Globus online: Accelerating and democratizing science through cloud-based
services. IEEE Internet Computing, 15(3):70–73, 2011.
34. Matthieu Gallet, Nezih Yigitbasi, Bahman Javadi, Derrick Kondo, Alexandru Iosup, and Dick
H. J. Epema. A model for space-correlated failures in large-scale distributed systems. In
Euro-Par, 2010.
35. Anshul Gandhi, Mor Harchol-Balter, Ram Raghunathan, and Michael A. Kozuch. Autoscale:
Dynamic, robust capacity management for multi-tier data centers. ACM Trans. Comput. Syst.,
30(4):14, 2012.
36. Shravan Gaonkar, Kimberly Keeton, Arif Merchant, and William H. Sanders. Designing
dependable storage solutions for shared application environments. IEEE Trans. Dependable
Secur. Comput., 7(4):366–380, 2010.
37. Bugra Gedik, Henrique Andrade, Kun-Lung Wu, Philip S. Yu, and Myungcheol Doo. Spade:
The system s declarative stream processing engine. In SIGMOD, 2008.
38. Erol Gelenbe, Gökçe Görbil, and Fang-Jing Wu. Emergency cyber-physical-human systems.
In ICCCN, 2012.
39. Bogdan Ghit, Mihai Capota, Tim Hegeman, Jan Hidders, Dick H. J. Epema, and Alexandru
Iosup. V for vicissitude: The challenge of scaling complex big data workflows. In CCGrid,
2014.
40. Bogdan Ghit, Nezih Yigitbasi, Alexandru Iosup, and Dick H. J. Epema. Balanced resource
allocations across multiple dynamic mapreduce clusters. In SIGMETRICS, 2014.
41. Roberto Gioiosa, Gokcen Kestor, and Darren J. Kerbyson. Online monitoring systems for
performance fault detection. Parallel Processing Letters, 24(4), 2014.
42. Ashvin Goel, Wu-chang Feng, Wu-chi Feng, and David Maier. Automatic high-performance
reconstruction and recovery. Comput. Netw., 51(5):1361–1377, 2007.
43. Qiang Guan and Song Fu. Adaptive anomaly identification by exploring metric subspace in
cloud computing infrastructures. In SRDS, 2013.
44. Ajay Gulati, Chethan Kumar, Irfan Ahmad, and Karan Kumar. BASIL: Automated IO load
balancing across storage devices. In FAST, 2010.
45. Ajay Gulati, Ganesha Shanmuganathan, Anne Holler, Carl Waldspurger, Minwen Ji, and
Xiaoyun Zhu. VMware Distributed Resource Management: Design, implementation, and
lessons learned. VMware Technical Journal, 1(1), 2012.
46. Vincenzo Gulisano, Ricardo Jimenez-Peris, Marta Patino-Martinez, Claudio Soriente, and
Patrick Valduriez. Streamcloud: An elastic and scalable data streaming system. IEEE Trans.
Parallel Distrib. Syst., 23(12):2351–2365, 2012.
47. Yong Guo, Marcin Biczak, Ana Lucia Varbanescu, Alexandru Iosup, Claudio Martella, and
Theodore L. Willke. How well do graph-processing platforms perform? An empirical performance evaluation and analysis. In IPDPS, 2014.
48. Zhenyu Guo, Sean McDirmid, Mao Yang, Li Zhuang, Pu Zhang, et al. Failure recovery:
when the cure is worse than the disease. In HotOS, 2013.
49. James Hamilton. On designing and deploying internet-scale services. In LISA, 2007.

Self-Awareness of Cloud Applications

37

50. Mor Harchol-Balter. Task assignment with unknown duration. J. ACM, 49(2):260–288,
2002.
51. John Hatcliff, Andrew King, Insup Lee, Alasdair Macdonald, Anura Fernando, et al. Rationale and architecture principles for medical application platforms. In ICCPS, 2012.
52. Tim Hegeman, Bogdan Ghit, Mihai Capota, Jan Hidders, Dick H. J. Epema, and Alexandru
Iosup. The btworld use case for big data analytics: Description, mapreduce logical workflow,
and empirical evaluation. In IEEE International Conference on Big Data, 2013.
53. Joseph L. Hellerstein, Yixin Diao, Sujay Parekh, and Dawn M. Tilbury. Feedback Control of
Computing Systems. John Wiley & Sons, 2004.
54. Jin Heo and Tarek Abdelzaher. Adaptguard: Guarding adaptive systems from instability. In
ICAC, pages 77–86, 2009.
55. Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, et al.
Mesos: A platform for fine-grained resource sharing in the data center. In NSDI, 2011.
56. Nikolaus Huber, Jürgen Walter, Manuel Bähr, and Samuel Kounev. Model-based Autonomic
and Performance-aware System Adaptation in Heterogeneous Resource Environments: A
Case Study. In ICCAC, 2015.
57. IDC. Worldwide and regional public it cloud services: 2013-2017 forecast. IDC Tech Report.
[Online] Available: www.idc.com/getdoc.jsp?containerId=251730, 2013.
58. Marija D. Ilić, Le Xie, Usman A. Khan, and José M. F. Moura. Modeling of future cyberphysical energy systems for distributed sensing and control. Trans. Sys. Man Cyber. Part A,
40(4):825–838, 2010.
59. Kenneth L. Ingham, Anil Somayaji, John Burge, and Stephanie Forrest. Learning dfa representations of http for protecting web applications. Comput. Netw., 51(5):1239–1255, 2007.
60. Alexandru Iosup and Dick H. J. Epema. Grid computing workloads. IEEE Internet Computing, 15(2):19–26, 2011.
61. Alexandru Iosup, Nezih Yigitbasi, and Dick H. J. Epema. On the performance variability of
production cloud services. In CCGrid, 2011.
62. Gideon Juve, Ann L. Chervenak, Ewa Deelman, Shishir Bharathi, Gaurang Mehta, and Karan
Vahi. Characterizing and profiling scientific workflows. Future Generation Comp. Syst.,
29(3):682–692, 2013.
63. Evangelia Kalyvianaki and Themistoklis Charalambous. A Min-Max framework for CPU
resource provisioning in virtualized servers using H-infinity Filters. In CDC, 2010.
64. Evangelia Kalyvianaki, Themistoklis Charalambous, Marco Fiscato, and Peter Pietzuch.
Overload Management in Data Stream Processing Systems with Latency Guarantees. In
Feedback Computing, 2012.
65. Evangelia Kalyvianaki, Themistoklis Charalambous, and Steven Hand. Self-Adaptive and
self-configured CPU resource provisioning for virtualized servers using Kalman Filters. In
ICAC, 2009.
66. Evangelia Kalyvianaki, Wolfram Wiesemann, Quang Hieu Vu, Daniel Kuhn, and Peter Pietzuch. Sqpr: Stream query planning with reuse. In ICDE, 2011.
67. Woochul Kang, Lui Sha, Richard B. Berlin, and Julian M. Goldman. The design of safe
networked supervisory medical systems using organ-centric hierarchical control architecture.
Biomedical and Health Informatics, IEEE Journal of, 19(3):1077–1086, 2015.
68. Rini T. Kaushik, Tarek Abdelzaher, Ryota Egashira, and Klara Nahrstedt. Predictive data
and energy management in greenhdfs. In IGCC, pages 1–9, 2011.
69. Kimberly Keeton, Dirk Beyer, Ernesto Brau, Arif Merchant, Cipriano Santos, and Alex
Zhang. On the road to recovery: Restoring data after disasters. In EuroSys, 2006.
70. Kimberly Keeton, Terence Kelly, Arif Merchant, Cipriano A. Santos, Janet L. Wiener, Xiaoyun Zhu, and Dirk Beyer. Don’t settle for less than the best: Use optimization to make
decisions. In HotOS, 2007.
71. Kimberly Keeton and Arif Merchant. A framework for evaluating storage system dependability. In DSN, 2004.
72. Roelof Kemp, Nicholas Palmer, Thilo Kielmann, and Henri E. Bal. Cuckoo: A computation
offloading framework for smartphones. In MobiCASE, 2010.

38

Iosup et al.

73. Mohammad M. H. Khan, Hieu Khac Le, Hossein Ahmadi, Tarek F. Abdelzaher, and Jiawei
Han. Troubleshooting interactive complexity bugs in wireless sensor networks using data
mining techniques. ACM Trans. Sen. Netw., 10(2):31:1–31:35, 2014.
74. Junsung Kim, Hyoseung Kim, Karthik Lakshmanan, and Ragunathan Rajkumar. Parallel
scheduling for cyber-physical systems: Analysis and case study on a self-driving car. In
ICCPS, 2013.
75. Cristian Klein, Martina Maggio, Karl-Erik Årzén, and Francisco Hernández-Rodriguez.
Brownout: Building more robust cloud applications. In ICSE, 2014.
76. Cristian Klein, Alessandro V. Papadopoulos, Manfred Dellkrantz, Jonas Durango, Martina
Maggio, et al. Improving cloud service resilience using brownout-aware load-balancing. In
SDRS, 2014.
77. Joseph A. Konstan and John Riedl. Recommended to you. IEEE Spectum, 2012.
78. Rouven Krebs, Simon Spinner, Nadia Ahmed, and Samuel Kounev. Resource usage control
in multi-tenant applications. In CCGrid, 2014.
79. Ioan Doré Landau, Yoan D Landau, and Gianluca Zito. Digital control systems: design,
identification and implementation. Springer, 2006.
80. Insup Lee and Oleg Sokolsky. Medical cyber physical systems. In DAC, 2010.
81. William S. Levine. The control handbook. The electrical engineering handbook series. CRC
Press New York, 1996.
82. Shen Li, Shiguang Wang, Fan Yang, Shaohan Hu, Fatemeh Saremi, and Tarek Abdelzaher.
Proteus: Power proportional memory cache cluster in data centers. In ICDCS, pages 73–82,
2013.
83. Yao Liu, Shaoxuan Wang, and Sujit Dey. Content-aware modeling and enhancing user experience in cloud mobile rendering and streaming. IEEE J. Emerg. Sel. Topics Circuits Syst.,
4(1):43–56, 2014.
84. Lei Lu, Xiaoyun Zhu, Rean Griffith, Pradeep Padala, Aashish Parikh, Parth Shar, and Evgenia
Smirni. Application-driven dynamic vertical scaling of virtual machines in resource pools.
In NOMS, 2014.
85. Martina Maggio, Henry Hoffmann, Marco D. Santambrogio, Anant Agarwal, and Alberto
Leva. Controlling software applications via resource allocation within the heartbeats framework. In CDC, 2010.
86. Luo Mai, Evangelia Kalyvianaki, and Paolo Costa. Exploiting time-malleability in cloudbased batch processing systems. In LADIS, 2013.
87. Jason Mars, Lingjia Tang, Robert Hundt, Kevin Skadron, and Mary Lou Soffa. Bubble-up:
increasing utilization in modern warehouse scale computers via sensible co-nolocations. In
MICRO, 2011.
88. Ole J. Mengshoel, Mark Chavira, Keith Cascio, Scott Poll, Adnan Darwiche, et al. Probabilistic model-based diagnosis: An electrical power system case study. IEEE Trans. on
Systems, Man and Cybernetics, Part A: Systems and Humans, 40(5):874–885, 2010.
89. Ole J. Mengshoel, Bob Iannucci, and Abe Ishihara. Mobile computing: Challenges and
opportunities for autonomy and feedback. In Feedback Computing’13, 2013.
90. Arif Merchant, Mustafa Uysal, Pradeep Padala, Xiaoyun Zhu, Sharad Singhal, and Kang G.
Shin. Maestro: quality-of-service in large disk arrays. In ICAC, 2011.
91. Mohamed F. Mokbel, Xiaopeing Xiong, and Walid G. Aref. Sina: Scalable incremental
processing of continuous queries in spatio-temporal databases. In SIGMOD, 2004.
92. Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martı́n
Abadi. Naiad: A timely dataflow system. In SOSP, pages 439–455, 2013.
93. Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy, and Steven Hand. Ciel: a universal execution engine for distributed data-flow
computing. In NSDI, 2011.
94. Vlad Nae, Alexandru Iosup, and Radu Prodan. Dynamic resource provisioning in massively
multiplayer online games. IEEE Trans. Parallel Distrib. Syst., 22(3):380–395, 2011.
95. Vlad Nae, Radu Prodan, Alexandru Iosup, and Thomas Fahringer. A new business model for
massively multiplayer online games. In ICPE, 2011.

Self-Awareness of Cloud Applications

39

96. Meiyappan Nagappan, Aaron Peeler, and Mladen Vouk. Modeling cloud failure data: a case
study of the virtual computing lab. In SECLOUD, 2011.
97. Fiona Fui-Hoon Nah. A study on tolerable waiting time: how long are web users willing to
wait? Behaviour and Information Technology, 23(3), 2004.
98. Syouji Nakamura and Toshio Nakagawa. Stochastic Reliability Modeling, Optimization and
Applications. World Scientific Publishing Company, 2009.
99. Bogdan Nicolae and Franck Cappello. Blobcr: Virtual disk based checkpoint-restart for HPC
applications on iaas clouds. J. Parallel Distrib. Comput., 73(5):698–711, 2013.
100. Alexandru-Corneliu Olteanu, Alexandru Iosup, and Nicolae Tapus. Towards a workload
model for online social applications. In ICPE, 2013.
101. Alexandru-Corneliu Olteanu, Nicolae Tapus, and Alexandru Iosup. Extending the capabilities of mobile devices for online social applications through cloud offloading. In CCGrid,
2013.
102. Pradeep Padala, Anne Holler, Lei Lu, Xiaoyun Zhu, Aashish Parikh, and Madhuri Yechuri.
Scaling of cloud applications using machine learning. VMware Technical Journal, 2014.
103. Pradeep Padala, Kai-Yuan Hou, Kang Shin, Xiaoyun Zhu, Mustafa Uysal, Zhijui Wang,
Sharad Singhal, and Arif Merchant. Automated control of multiple virtualized resources.
In Eurosys, 2009.
104. Miroslav Pajic, Rahul Mangharam, Oleg Sokolsky, David Arney, Julian M. Goldman, and
Insup Lee. Model-driven safety analysis of closed-loop medical systems. IEEE Trans. Industrial Informatics, 10(1):3–16, 2014.
105. Ken Peffers, Tuure Tuunanen, Marcus A. Rothenberger, and Samir Chatterjee. A design science research methodology for information systems research. J. of Management Information
Systems, 24(3):45–77, 2008.
106. Zhengping Qian, Yong He, Chunzhi Su, Zhuojie Wu, Hongyu Zhu, et al. Timestream: Reliable stream computation in the cloud. In EuroSys, 2013.
107. Ragunathan Rajkumar, Insup Lee, Lui Sha, and John Stankovic. Cyber-physical systems:
The next computing revolution. In DAC, 2010.
108. Charles Reiss, Alexey Tumanov, Gregory R. Ganger, Randy H. Katz, and Michael A.
Kozuch. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In SOCC,
2012.
109. Gartner Research. Keep the Five Functional Dimensions of APM Distinct, 2010.
110. Brian Ricks and Ole J. Mengshoel. Diagnosis for uncertain, dynamic, and hybrid domains
using bayesian networks and arithmetic circuits. International Journal on Approximate Reasoning, 55(5):1207–1234, 2014.
111. Matthew A. Russell. Mining the Social Web: Analyzing Data from Facebook, Twitter,
LinkedIn, and Other Social Media Sites. O’Reilly Media, Inc., 1st edition, 2011.
112. Johann Schumann, K. Y. Rozier, T. Reinbacher, Ole J. Mengshoel, T. Mbaya, and C. Ippolito. Real-time, on-board, hardware-supported sensor and software health management
for unmanned aerial systems. In Annual Conf. Prognostics and Health Management Society,
2013.
113. Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. Omega:
Flexible, scalable schedulers for large compute clusters. In EuroSys, 2013.
114. Damián Serrano, Sara Bouchenak, Yousn Kouki, Thomas Ledoux, Jonathan Lejeune, et al.
Towards QoS-oriented SLA guarantees for online cloud services. In CCGrid, 2013.
115. Damián Serrano, Sara Bouchenak, Yousri Kouki, Frederico Alvares de Oliveira Jr, Thomas
Ledoux, et al. SLA guarantees for cloud services. Future Generation Computer Systems,
2015.
116. Lui Sha, Sathish Gopalakrishnan, Xue Liu, and Qixin Wang. Cyber-physical systems: A new
frontier. In SUTC, pages 1–9, 2008.
117. Siqi Shen, Kefeng Deng, Alexandru Iosup, and Dick H. J. Epema. Scheduling jobs in the
cloud using on-demand and reserved instances. In Euro-Par, 2013.
118. Siqi Shen, Alexandru Iosup, Assaf Israel, Walfredo Cirne, Danny Raz, and Dick H. J. Epema.
An availability-on-demand mechanism for datacenters. In CCGRID, 2015.

40

Iosup et al.

119. Siqi Shen, Vincent van Beek, and Alexandru Iosup. Statistical characterization of businesscritical workloads hosted in cloud datacenters. In CCGRID, 2015.
120. Snir et al. Noaddressing failures in exascale computing. IJHPCA, 28(2):129–173, 2014.
121. Simon Spinner, Samuel Kounev, Xiaoyun Zhu, Lei Lu, Mustafa Uysal, Anne Holler, and
Rean Griffith. Runtime vertical scaling of virtualized applications via online model estimation. In SASO, 2014.
122. Priya K. Sundararajan, Eugen Feller, Julien Forgeat, and Ole J. Mengshoel. A constrained
genetic algorithm for rebalancing of services in cloud data centers. In CLOUD, 2015.
123. Morton Swimmer. Using the danger model of immune systems for distributed defense in
modern data networks. Comput. Netw., 51(5):1315–1333, 2007.
124. Douglas Thain, Todd Tannenbaum, and Miron Livny. Distributed computing in practice: the
condor experience. Concurrency - Practice and Experience, 17(2-4):323–356, 2005.
125. Vincent van Beek, Jesse Donkervliet, Tim Hegeman, Stefan Hugtenburg, and Alexandru Iosup. Menmos: Self-expressive management of business-critical workloads in virtualized datacenters. IEEE Computer, 48(7):46–54, 2015.
126. Abhishek Verma, Luis Pedrosa, Madhukar R. Korupolu, David Oppenheimer, Eric Tune, and
John Wilkes. Large-scale cluster management at Google with Borg. In EuroSys, 2015.
127. Qiushi Wang and Katinka Wolter. Reducing task completion time in mobile offloading systems through online adaptive local restart. In ICPE, 2015.
128. Joel Wolf, Nikhil Bansal, Kirsten Hildrum, Sujay Parekh, Deepak Rajan, et al. Soda: An optimizing scheduler for large-scale stream-based distributed computer systems. In Middleware,
2008.
129. Pengcheng Xiong, Calton Pu, Xiaoyun Zhu, and Rean Griffith. vPerfGuard: An automated
model-driven framework for application performance diagnosis in consolidated cloud environments. In ICPE, 2013.
130. Yong Yang, Lu Su, Mohammad Khan, Michael Lemay, Tarek Abdelzaher, and Jiawei Han.
Power-based diagnosis of node silence in remote high-end sensing systems. ACM Trans. Sen.
Netw., 11(2):33:1–33:33, 2014.
131. Nezih Yigitbasi, Matthieu Gallet, Derrick Kondo, Alexandru Iosup, and Dick H. J. Epema.
Analysis and modeling of time-correlated failures in large-scale distributed systems. In
GRID, 2010.
132. Yuan Yu, Michael Isard, Dennis Fetterly, Mihai Budiu, Úlfar Erlingsson, Pradeep Kumar
Gunda, and Jon Currey. Dryadlinq: A system for general-purpose distributed data-parallel
computing using a high-level language. In OSDI, pages 1–14, 2008.
133. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, et al. Resilient
distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI,
pages 2–2, 2012.
134. Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica.
Discretized streams: Fault-tolerant streaming computation at scale. In SOSP, pages 423–438,
2013.
135. Junping Zhang, Fei-Yue Wang, Kunfeng Wang, Wei-Hua Lin, Xin Xu, and Cheng Chen.
Data-driven intelligent transportation systems: A survey. Intelligent Transportation Systems,
IEEE Transactions on, 12(4):1624–1639, 2011.
Acknowledgements This work is partially supported by the Dutch STW/NWO Veni personal
grant @large(#11881) and Vidi personal grant MagnaData, by the Dutch national program COMMIT and COMMissioner sub-project, by the Dutch KIEM project KIESA, and by a generous ERO
gift from Oracle, by the European FP7 research project AMADEOS Grant Agreement 610535 on
Systems of Systems, by the Swedish Research Council (VR) for the projects “Cloud Control” and
“Power and temperature control for large-scale computing infrastructures”, and through the LCCC
Linnaeus and ELLIIT Excellence Centers.

