D EEP CAS: A Deep Reinforcement Learning Algorithm for
Control-Aware Scheduling

arXiv:1803.02998v1 [cs.SY] 8 Mar 2018

Burak Demirel, Arunselvan Ramaswamy, Daniel E. Quevedo and Holger Karl
Abstract— We consider networked control systems consisting
of multiple independent closed-loop control subsystems, operating over a shared communication network. Such systems are
ubiquitous in cyber-physical systems, Internet of Things, and
large-scale industrial systems. In many large-scale settings, the
size of the communication network is smaller than the size of
the system. In consequence, scheduling issues arise. The main
contribution of this paper is to develop a deep reinforcement
learning-based control-aware scheduling (D EEP CAS) algorithm
to tackle these issues. We use the following (optimal) design
strategy: First, we synthesize an optimal controller for each
subsystem; next, we design learning algorithm that adapts to
the chosen subsystem (plant) and controller. As a consequence
of this adaptation, our algorithm finds a schedule that minimizes the control loss. We present empirical results to show
that D EEP CAS finds schedules with better performance than
periodic ones. Finally, we illustrate that our algorithm can be
used for scheduling and resource allocation in more general
networked control settings than the above-mentioned one.
Index Terms— Deep Learning; Reinforcement Learning; Optimal Control; Networked Control Systems

I. I NTRODUCTION
Today’s cyber-physical systems (CPS), Internet of Things
(IoT), large-scale industrial systems, and myriad real-world
systems try to integrate traditional control systems with artificial intelligence to improve the overall system performance
and reliability. Examples of such systems include smart grids,
smart cities, city-wide vehicular traffic control, industrial
process optimization, among others. In recent years, artificial
intelligence (AI) has seen a resurgence as an effective
model-free solution to many problems, including optimal
control problems arising in the aforementioned systems. This
resurgence is partly owing to the advances in computational
capacities and the advent of deep neural networks for function approximation and feature extraction. Oftentimes, the
use of reinforcement learning algorithms or AI in conjunction
with traditional controllers reduces the complexity of system
design while boosting efficiency.
The large size of systems poses tremendous challenges
in resource allocation. In applications involving feedback,
this challenge is exaggerated since resource allocation is
required to be “control-aware”, i.e., it should reduce control
loss. Typical resources including communication channels
and computational resources do not scale with system size.
B. Demirel, A. Ramaswamy, D. E. Quevedo, and H. Karl
are with Paderborn University, Warburger Straße 100, 33098,
Paderborn,
Germany
burak.demirel@protonmail.com
arunselvan.ramaswamy@upb.de dquevedo@ieee.org.
holger.karl@upb.de.
A. Ramaswamy was funded by the German Research Foundation (DFG)
- 315248657.

DeepCAS

Smart Sensor
Kalman
Filter (I)

Plant

Control
Algorithm

Kalman
Filter (II)

Feedback Control Loop (i)

Controller
Control
Algorithm

Net work
with M available
channels

···

Kalman
Filter (II)

Fig. 1.
Networked control system (NCS) that consists of N control
subsystems closed over a shared communication network (with M available
channel).

In large-scale systems, controllers often rely on information
collected from various sensors to make intelligent decisions.
Hence, efficient information dispersion is essential for decision making over communication networks to be effective.
As noted earlier, this is a hard problem since the number of
communication channels available is much smaller than what
is ideally required to transfer data from sensors to controllers.
In this paper, we present a deep reinforcement learningbased scheduling algorithm for the sensor-to-controller communication in large-scale networked control systems. To
illustrate our ideas, we use the system architecture illustrated
in Fig. 1. Later, we will show that our ideas can be used for
scheduling in more general architectures.
Fig. 1 is a simplified representation of (some) CPS and
IoT systems. The system consists of N independent control
subsystems that communicate over a shared communication
network, which contains M channels. We assume that M 
N (M is much smaller than N ) and, in particular, that
each channel can be used to serve any communication
request. Each control subsystem consists of a smart sensor, a
controller, and a plant to be controlled. The feedback loops
are closed over this resource-constrained communication
network. For more details on the system model, we refer
the reader to § II.
D EEP CAS, our deep reinforcement learning-based modelfree scheduling algorithm, decides which of the N subsystems are allocated to the M channels in a given time

instant. Further, it adapts to the controller at hand and finds
a schedule that minimizes the control loss. We present an
implementation wherein D EEP CAS obtains feedbacks (i.e.,
rewards) from smart sensors for making decisions. At each
time instant, each smart sensor computes the estimate of the
subsystem’s state, using Kalman Filter (I), and transmits it to
the corresponding controller; see Fig. 1. The controller runs
Kalman Filter (II) to estimate the subsystem’s state in the
absence of transmissions. In addition to Kalman Filter (I), a
smart sensor also implements a copy of Kalman Filter (II)
and the control algorithm. This way the smart sensor knows
the state estimate used by the controller at every time instant.
Several scheduling approaches for control systems have
been proposed in the literature to determine the access order
of different sensors and/or actuators; see [1] and references
therein. A widely considered approach is to employ periodic
schedules [2]–[5]. However, finding such schedules for control applications may not be easy since both specific period
and sequence need to be found. Further, periodic sequences
may not be even optimal in which case searching for them
is futile. With a few exceptions (e.g., [6] and [7]), the
determination of optimal schedules indeed requires solving a
mixed-integer quadratic program, which is computationally
infeasible for all but very small systems.
Our contribution is in the development of a deep reinforcement learning-based control-aware scheduling (D EEP CAS)
algorithm. At its heart lies the Deep Q-Network (DQN), a
modern variant of Q learning, introduced in [8]. In addition to
being readily scalable, D EEP CAS is completely model-free.
To optimize the overall control performance, we propose
the following (optimal) design of control and scheduling:
In the first step, we design an optimal controller for each
independent subsystem. As discussed in [9], under limited
communication, the control loss has two components: (a)
best possible control loss (b) error due to intermittent transmissions. If M = N , then (b) vanishes. Since we are in
setting of M  N , the goal of the scheduler is to minimize
(b). To this end, we first construct a Markov decision process
(MDP). The state space of this MDP is the difference in state
estimates of all controllers and sensors (obtainable from the
smart sensors). Its reward is the negative of the previously
mentioned (b). Since we are using DQN (reinforcement
learning) to solve this MDP, we do not need the knowledge
of transition probabilities. Since the goal of DQN is to find a
scheduling strategy that maximizes reward, it naturally aims
at minimizing (b). For more details on the MDP, the reader is
referred to § III. We present empirical results showing that
D EEP CAS finds schedules that have lower control losses
than traditional hand-tuned heuristics.
The organization of this paper is as follows: Section II
introduces models and assumptions for subsystems, sensors,
controllers, and network. This section also presents the
primary objective of the paper and how to reach this objective
via solving control design and control-aware scheduling
problems separately. Section III presents an MDP associated
with control-aware scheduling problems and proposes a
deep reinforcement learning algorithm to solve this MDP

efficiently. In Section IV, numerical examples are used to
illustrate the power of our AI-based scheduling algorithm.
Section V gives concluding remarks and directions for future work.
II. N ETWORKED C ONTROL S YSTEM : M ODEL ,
A SSUMPTIONS , AND G OALS
A. Model for each subsystem
As illustrated in Figure 1, our networked control system
consists of N independent closed-loop subsystems. The
feedback loop within each subsystem (plant) is closed over a
shared communication network. For 1 ≤ i ≤ N , subsystem
i is described by
(i)

(i)

(i)

(i)

(1)

xt+1 = A(i) xt + B (i) ut + wt ,

where A(i) and B (i) are matrices of appropriate dimensions,
(i)
(i)
xt ∈ Rni is the subsystem i’s state, ut ∈ Rmi is the
(i)
ni
control input, and wt ∈ R is zero-mean i.i.d. Gaussian
noise with covariance matrix W (i) . The initial state of
(i)
subsystem i, x0 , is assumed to be a Gaussian random vector
(i)
(i)
with mean x̄0 and covariance matrix X0 .
At a given time t, we assume that only noisy output
measurements are available. We thus have:
(i)

(i)

(i)

(2)

yt = C (i) xt + vt ,
(i)

where vt ∈ Rpi is zero-mean i.i.d. Gaussian noise with
(i)
(i)
covariance matrix V (i) . All noise sources, wt and vt , are
(i)
independent of the initial conditions x0 .
B. Control architecture and loss function
The dynamics of each subsystem is a stochastic linear
time-invariant (LTI) system given by (1). Further, each subsystem is independently controlled although dependencies do
arise from sharing a communication network. Subsystem i
(i)
has a smart sensor which samples the subsystem’s output yt
and computes an estimate of the subsystem’s state. This value
is then sent to the associated controller, provided a channel
is allocated to it by D EEP CAS. If the controller obtains
a new state estimate from the sensor, then it calculates a
control command based on this state estimate. Otherwise, it
calculates a control command based on its own estimate of
the subsystem’s state.
The control actions and scheduling decisions (of D EEP CAS) are taken to minimize the total control loss given by
J=

N
X

(3)

J (i) ,

i=1

where J (i) is the control loss of subsystem i and is given by

(i)> (i) (i)
J (i) = E xT Qf xT
+

T−1
X

(i)>
(i)
xt Q(i) xt

+

(i)>
(i)
ut R(i) ut



, (4)

t=0
(i)

where Q(i) and Qf are positive semi-definite matrices and
R(i) is positive definite.

C. Smart sensors and pre-processing units
Within our setting, the primary role of a smart sensor
is to take measurements of a subsystem’s output. Also, it
plays a vital role in helping D EEP CAS with scheduling
decisions. It is from the smart sensors that D EEP CAS gets
all the necessary feedback information for scheduling. For
these tasks, each smart sensor employs two Kalman filters:
(1) Kalman Filter (I) is used to estimate the subsystem’s
state, (2) a copy of Kalman Filter (II) is used to estimate
the subsystem’s state as perceived by the controller. Note
that the controller employs Kalman Filter (II). Below, we
discuss them in detail.
Kalman filter (I): The sensor employs a standard Kalman
(i)s
(i)s
filter to compute the state estimate x̂t|t and covariance Pt|t
recursively as
(i)s
x̂t|t−1
(i)s
Pt|t−1

=
=

(i)s
(i)
A x̂t−1|t−1 + B (i) ut−1
(i)s
A(i) Pt−1|t−1 A(i)> + W (i)
(i)


−1
(i)s
(i)s
Kt = Pt|t−1 C (i)> C (i) Pt|t−1 C (i)> + V (i)


(i)s
(i)s
(i)
(i)
(i)s
x̂t|t = x̂t|t−1 + Kt yt − C x̂t|t−1


(i)s
(i)
(i)s
Pt|t = I − Kt C (i) Pt|t−1 ,
(i)s

(i)

(i)s

(i)c

(i)

(i)

(i)

with x̂0|−1 = x̄0 .
D. Goal: minimizing the control loss
Under the assumptions presented in § II, the certainty
equivalent controller is still optimal; see [9] for details. The
total control loss in (4) has two components: (a) best possible
control loss (b) error due to intermittent communications.
Hence, the problem of minimizing control loss has two separate components: (i) designing the best (optimal) controller
for each subsystem and (ii) scheduling in a control-aware
manner.
Component I: Controller design. The controller in feedback
(i)
loop i takes the following control action, ut , at time t:
(i)

(i)

(i)c

(7)

ut = −Lt x̂t|t ,
(i)c

where x̂t|t is the state estimate used by the controller,
(i)

(i)

(i)

Lt = (B (i)> St+1 B (i) + R(i) )−1 B (i)> St+1 A(i)
and

(i)
St
(i)

(8)

is recursively computed as
(i)

(i)

St = A(i)> St+1 A(i) + Q(i) − A(i)> St+1 B (i)
(i)

(i)

(i)s

(i)c

J

(i)

=

(i)> (i) (i)
x̄0 S0 x̄0 +Tr

(i) (i) 
S0 X0 +

T−1
X

(i)

Tr St+1 W (i)



t=0

+

T−1
X

(i)s (i) 

Tr Pt|t Γt

t=0
(i)
where Γt ,
(i)s
(i)c
x̂t|t − x̂t|t .

+

T−1
X

h
i
(i)> (i) (i)
E et|t Γt et|t , (10)

t=0
(i)>
(i)
(i)
(i)
Lt (B (i)> St+1 B (i) + R(i) )Lt and et|t
(i)
Note that et|t is the communication error

,

in
subsystem i. Recall that there are N subsystems and M
(i)
communication channels. If M = N , then et|t = 0 for all
t ≥ 0 and 1 ≤ i ≤ N .
Component II: Control-aware scheduling. The main aim of
the scheduling algorithm is to help minimize J of (3). To
this end, one must minimize
i
h
(i)> (i) (i)
E et|t Γt et|t

(11)

t=0

x̂t|t−1 = A(i) x̂t−1|t−1 + B (i) ut−1 ,
(5)
( (i)s
x̂t|t
if the MMSE estimate received ,
(i)c
(6)
x̂t|t =
(i)c
x̂t|t−1 otherwise ,
(i)c

(i)

loop i have communicated. Otherwise, x̂t|t is the state
estimate obtained from Kalman Filter (II). The minimum
value of the control loss of subsystem i is given by

T−1
X

starting from x̂0|−1 = x̄0 and P0|−1 = X0 .
Kalman filter (II): The controller runs a minimum mean
square error (MMSE) estimator to compute estimates of the
subsystem’s state as follows:
(i)c

(i)

with initial values SN = Qf . Let x̂t|t be the state estimate
of Kalman Filter (I), as employed by the sensor. We have
(i)c
(i)s
x̂t|t = x̂t|t when the sensor and controller of the feedback

× (B (i)> St+1 B (i) + R(i) )−1 B (i)> St+1 A(i) , (9)

of (10) for every 1 ≤ i ≤ N . Note that T in (11) is the
control horizon. At any time t, the scheduler decides which
M among the N subsystems may communicate. Note that
(i)
et|t = 0 when a communication channel is assigned to
subsystem i at time t.
In the following section, we present a deep reinforcement learning algorithm for control-aware scheduling called
D EEP CAS. D EEP CAS communicates only with the smart
sensors. At every time instant, sensors are told if they can
transmit to their associated controllers. Then, the sensors
provide feedback on the scheduling decision for that stage.
Note that we do not consider the overhead involved in
providing feedback.
III. D EEP REINFORCEMENT LEARNING FOR
CONTROL - AWARE SCHEDULING
As stated earlier, at the heart of our D EEP CAS lies the
DQN. The DQN is a modern variant of Q-learning that
effectively counters Bellman’s curse of dimensionality. Essentially, DQN or Q-learning finds a solution to an associated
Markov decision process (MDP) in an iterative model-free
manner. Before proceeding, let us recall the definition of an
MDP. For a more detailed exposition, the reader is referred
to [10]. An MDP, M, is given by the following tuple
(S, A, P, r, γ), where
S is the state-space of M.
A is the set of actions that can be taken.
P is the transition probability, i.e., P (s, s0 ; a) is the probability of transitioning to state s0 when action a is taken
at state s.
r is the one stage reward function, i.e., r(s, a) is the
reward when action a is taken at state s.

γ is the discount factor and ∈ [0, 1].
Below, we state the MDP Md associated with our problem.
S: The state space S consists of all possible augmented
error vectors. Hence, state vector st at time t is given
(1)
(N )
by (et|t , . . . , et|t ).
A: Action space is {S | S ⊂ {1, 2, . . . , N }, |S| = M }.
Hence, the cardinality (size), |A|, of the action space
N
equals M
.
N
P
(i) (i) (i)
r: At time t, the reward r(t) is given by −
et|t Γt et|t .
i=1

γ: Although it would seem natural to use γ = 1, we use
0 < γ < 1 since it hastens the rate of convergence.
Note that the scheduler (D EEP CAS) takes action just before
time t and receives rewards just after time t, based on
transmissions at time t. Also, note that D EEP CAS only gets
non-zero rewards from non-transmitting sensors. D EEP CAS
is model-free. Hence, it does not need to know transition
probabilities.
Let us suppose we use a reinforcement learning algorithm,
such as Q-learning, to solve Md . Since the learning algorithm will find policies that minimize the future expected
cumulative rewards, we expect to find policies that minimize
scheduling effects on the entire system. This is a consequence
of our above definition of reward r. In the following section,
we provide a brief overview of Q-learning and DQN, the
reinforcement learning algorithm at the heart of our D EEP CAS. Simply put, D EEP CAS is a DQN solving the above
defined MDP Md .
D EEP CAS. At any time t0 , the scheduler (D EEP CAS) is
interested in maximizing the following expected discounted
future reward:
"T −1
#
X
t−t0
R(t0 ) := E
γ
r(t) ,
t=t0

where r(t) is the single stage cost given by −

N
P
i=1

(i) (i) (i)

et|t Γt et|t .

Q-learning is a useful methodology to solve such problems.
It is based on finding the following Q-factor for every stateaction pair:
∗

Q (s, a) := max E [Rt | st = s, at = a, π] ,
π

where π is a policy that maps states to actions. The algorithm
itself is based on the Bellman equation:
Q∗ (s, a) = Es0 ∼E [r + γ 0max 0 Q∗ (s0 , a0 ) | s, a] .
a ∈U (s )

Since our state space is continuous, we use a deep neural
network (DNN) for function approximation. Specifically, we
try to find good approximations of the Q-factors iteratively.
In other words, the neural network takes as input state s
and outputs Q(s, a, θ) for every possible action a, such that
Q(s, a, θ) ≈ Q∗ (s, a). This deep function approximator, with
weights θ, is referred to as a Deep Q-Network. The Deep QNetwork is trained by minimizing a time-varying sequence
of loss functions Lt (θt ) given by



Lt (θt ) = 1/2 Es,a∼ρ(s,a) (Q(s, a, θt ) − yt )2 ,

where yt := Es0 ∼E [r + γ maxa0 Q(s0 , a0 , θt−1 ) | s, a] is the
expected cost-to-go based on the latest update of the weights;
ρ is the behavior distribution. Training the neural network
involves finding θ∗ , which minimizes the loss functions.
Since the algorithm is online, training is done in conjunction with scheduling. At time t, after feedback (reward) is
received, one gradient descent step can be performed using
the following gradient term:

∇θt Lt (θt ) = Es,a∼ρ(·);s∼E Q(s, a; θt ) − r


0 0
− γ max
Q(s , a , θt−1 ) ∇θt Q(s, a; θt ) . (12)
0
a

To make the algorithm implementable, we use a sample rather than find the above expectation, when updating
weights. At each time, we pick actions using the -greedy
approach. Specifically, we pick a random action with probability , and we pick a greedy action with probability
1 − . This constitutes the previously mentioned behavior
distribution ρ, which governs how actions are picked. Note
that a greedy action at at time t is one that maximizes
Q(st , a; θ). Initially it is desirable to explore, hence  is set
to 1. Once the algorithm has gained some experience, it is
better to exploit this experience. To accomplish this, we use
an attenuating .
Although we train our DNN in an online manner, we
do not perform a gradient descent step using (12), since it
can lead to poor learning. Instead, we store the previous
K experiences (st , at , rt , st+1 ), t0 − K + 1 ≤ t ≤ t0 , in
an experience replay memory D. In other words, at time
t, D EEP CAS stores (st , at , rt , st+1 ) into D whose size is
K. When it comes to training the neural network at time
t, it performs a single mini-batch gradient descent step.
The mini-batch (of gradients) is randomly sampled from
the aforementioned experience replay D. The idea of using
experience replay memory, to overcome biases and to have
a stabilizing effect on algorithms, was introduced in [8].
DQN for control-aware scheduling
1: Initialize the replay memory D to capacity K.
2: Initialize the weights, θ, of the Q-Network.
3: for the entire duration do
4:
With probability  select a random action at .
5:
With probability 1 −  pick at that maximizes
Q(st , a, θ).
6:
Execute action at to obtain reward rt and observe
st+1 .
7:
Store (st , at , rt , st+1 ) in D.
8:
Sample random mini-batch transitions from D.
9:
Corresponding to (sj , aj , rj , sj+1 ), set
yj := rj + γ max
Q(sj+1 , a0 ; θ).
0
a

10:

Perform a gradient descent step with loss given by
(yj − Q(sj , aj ; θ))2 .

100

We are now ready to present our numerical results. Recall
that a DQN is at the heart of our D EEP CAS, which uses
a deep neural network to approximate Q-factors. Before
proceeding, we specify the algorithm parameters. The input
to the neural network is the appended error vector. The
hidden layer consists of 1024 rectifier units. The output layer
is a fully connected
linear layer with a single output for each

N
of the M
actions. The discount factor γ in our Q-learning
algorithm is 0.95. The size of the experience replay buffer
is fixed at 20000. The exploration parameter  is initialized
to 1, then attenuated to 0.001 at the rate of 0.9. For training
the neural network, we use the optimizer ADAM [11] with
a learning rate of e−4 and a decay of 0.001. Note that we
used the same set of parameters for all of the experiments
presented below.
Experiment 1: N=3, M=1, and T=500. For our first experiment, we used D EEP CAS to schedule one channel for
a system with three subsystems, over a control horizon of
length 500. We considered three second-order single-inputsingle-output (SISO) subsystems consisting of one stable
(subsystem 2) and two unstable subsystems (subsystems 1
and 3). If there were three channels, then there would be
no scheduling problem and the total optimal control loss
J would be 13.8487. Since there is a single channel, one
expects a solution to the scheduling problem to allocate it
to subsystems 1 and 3 for a more substantial fraction of the
time, as compared to subsystem 2. This expectation is fair
since subsystems 1 and 3 are unstable while subsystem 2 is
stable. Once trained, D EEP CAS indeed allocates the channel
to subsystem 1 for 52% of the time, to subsystem 2 for 12%
of the time, and to subsystem 3 for 36% of the time.
We train D EEP CAS continuously over many epochs. Each
epoch corresponds to a single run of the control problem with
horizon 500. At the start of each epoch, the initial conditions
for the control problem are chosen as explained in § II. Fig. 2
illustrates the learning progress of D EEP CAS. The abscissa
axis of the graph represents the epoch number while the
ordinate axis represents the average control loss. The plot is
obtained by taking the mean of 30 Monte Carlo runs. Since
DQN is randomly initialized, scheduling decisions are poor
at the beginning, and the average control loss is high. As
learning proceeds, the decisions taken improve. After only
10 epochs, D EEP CAS converges to a scheduling strategy
with an associated control loss of around 21.
Comparison to periodic scheduling. Traditionally, the problem of scheduling for control systems is solved by using
control theoretic heuristics to find periodic schedules. We
do the following to compare the performance of D EEP CAS
to such techniques. For every fixed n ∈ {2, . . . , 11}, we
perform an exhaustive search (among 3n sequences) to find
a scheduling sequence, of length n, which minimizes the
control loss. The results are listed in Table I1 . As illustrated
in Table I, this methodology yields a minimum control loss
1 We stopped at n = 11 since exhaustive search is extremely timeconsuming and impractical beyond that.

Empirical average cost

IV. N UMERICAL RESULTS

80
60
40
20
0

5

10

15

20

25

30

Epochs
Fig. 2.
The convergence of the empirical average control loss for
D EEP CAS governing a networked system having three subsystems and one
communication channel.
TABLE I
O PTIMAL P ERIODIC S CHEDULES FOR THE T HREE S UBSYSTEMS
Length n
2
3
4
5
6
7
8
9
10
11

Sequence
{1, 3}
{1, 3, 2}
{1, 3, 1, 2}
{1, 3, 1, 3, 2}
{1, 3, 1, 3, 1, 2}
{1, 3, 1, 3, 1, 3, 2}
{1, 3, 1, 3, 1, 3, 1, 2}
{1, 3, 1, 3, 1, 3, 1, 3, 2}
{1, 3, 1, 3, 1, 3, 1, 3, 1, 2}
{1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 2}

Total cost J
29.0510
25.7798
25.2363
23.6020
23.7801
22.9636
23.2944
22.8112
23.1738
22.8601

of J = 22.8112 for n = 9. In comparison, D EEP CAS finds a
scheduling strategy with an associated control loss of 21.15.
We conclude that, in addition to being faster, D EEP CAS does
not need any system specification and can schedule efficiently
for very long control horizons.
Another approach to scheduling is to solve an associated mixed-integer quadratic program (MIQP); see [12]
for details. Solving a MIQP accurately is computationally
infeasible for all but small system sizes.
Experiment 2: N=6, M=3, and T=500. For our second
experiment, we train D EEP CAS to schedule three channels
for a system with six second-order SISO subsystems. If
N = M , then the total control loss equals 18.234. As before,
learning is done continuously over many epochs. Fig. 3
illustrates the learning progress of D EEP CAS in scheduling
three channels among six subsystems. The abscissa and
ordinate axes are as before. As evidenced in the figure,
D EEP CAS quickly finds schedules with an associated control
loss of around 20. Due to the difference in scale, the control
loss in Fig. 3 seems to have high variance as compared to
Fig. 2.
We are unable to compare the results of Experiment 2
with any traditional heuristics. This is because traditional
heuristics do not extend to the system size and control
horizon considered here. Further, performing an exhaustive
search for finding periodic schedules is not possible
n since
the number of possibilities are in the order of 63 = 20n ,
where n is the period-length.

Empirical average cost

Empirical average cost

24

23

22

21

20

0

10

20

30

40

50

Epochs

Fig. 3.
The convergence of the empirical average control loss for
D EEP CAS governing a networked system having six subsystems and three
communication channels.

Scheduling in general control settings. The systems considered hitherto have independent subsystems. This facilitates
the splitting of the total control cost into two components;
see (10). The one-stage reward in our algorithm is the
negative of the error due to lack of communication defined
in (11). However, in general multi-agent settings, the previously mentioned splitting may not be possible.
Now, we discuss an extension to our algorithm which
will allow for scheduling in these general scenarios. For
such an extension, we observe that one merely needs to
provide a recipe for obtaining the state-space and reward in
the definition of Md (associated MDP). Since we want our
scheduler to reduce the control loss, it seems that a natural
choice for reward is the negative of the one-stage control
cost. Regarding state-space, one may need to consider the
state (or its estimate) control-action pairs, say (xt , ut ), of the
system. It may also be necessary to append the pairs (xt , ut )
of all the subsystems. The expected control cost associated
with taking action while in some state does not change with
time. This allows for a consistent definition of Md . It may
be noted that there may be other ways to define an associated
MDP.
To show that the above extension is viable, we repeated
Experiments 1 and 2 with negative of the one-stage control
cost as the reward. The results of the modified experiments
are very similar to the original ones. Due to constraints in
space, we only present Fig. 4 associated with the modification of Experiment 1.
V. C ONCLUSIONS
This paper considered the problem of scheduling the
sensor-to-controller communication in a networked control
system, consisting of a multitude of independent control
subsystems, in the presence of scarce communication resources. For this reason, we developed D EEP CAS, i.e., a
reinforcement learning-based control-aware scheduling algorithm. This algorithm is model-free and scalable, and
it outperforms scheduling heuristics tailored for feedback
control applications. Specifically, we compared our algorithm
with periodic schedules. Also, we briefly discussed how
D EEP CAS could be extended to a networked control system,

100
80
60
40
20
0

10

20

30

40

50

Epochs
Fig. 4.
The convergence of the empirical average control loss for
D EEP CAS governing a networked system having three subsystems and one
communication channel (reward as −J).

containing coupled subsystems.
As stated earlier, this paper does not address the problem
associated with overheads. An exciting future direction could
be the development of sophisticated reinforcement learning
algorithm which can reduce overhead. Addressing the aforementioned issues may lead to the associated problem of large
discrete action spaces. The algorithms thus developed would
need to take this into account. Lastly, it is also interesting to
consider imperfect communication scenarios.
R EFERENCES
[1] P. Park, S. C. Ergen, C. Fischione, C. Lu, and K. H. Johansson,
“Wireless network design for control systems: a survey,” IEEE Communications Surveys & Tutorials, 2018.
[2] H. Rehbinder and M. Sanfridson, “Scheduling of a limited communication channel for optimal control,” Automatica, vol. 40, no. 3, pp.
491–500, March 2004.
[3] D. Hristu-Varsakelis and L. Zhang, “LQG control of networked control
systems,” International Journal of Control, vol. 81, no. 8, pp. 1266–
1280, 2008.
[4] L. Shi, P. Cheng, and J. Chen, “Optimal periodic sensor scheduling
with limited resources,” IEEE Transactions on Automatic Control,
vol. 56, no. 9, pp. 2190–2195, 2011.
[5] L. Orihuela, A. Barreiro, F. Gómez-Estern, and F. R. Rubio, “Periodicity of Kalman-based scheduled filters,” IEEE Transactions on
Automatic Control, vol. 50, no. 10, pp. 2672–2676, 2014.
[6] L. Zhao, W. Zhang, J. Hu, A. Abate, and C. J. Tomlin, “On the optimal
solutions of the infinite-horizon linear sensor scheduling problem,”
IEEE Transactions on Automatic Control, vol. 59, no. 10, pp. 2825–
2830, March 2014.
[7] Y. Mo, E. Garone, and B. Sinopoli, “On infinite-horizon sensor
scheduling,” Systems & Control Letters, vol. 67, pp. 65–70, May 2014.
[8] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou,
D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement
learning,” in NIPS Deep Learning Workshop, 2013.
[9] B. Demirel, A. S. Leong, V. Gupta, and D. E. Quevedo, “Trade-offs
in stochastic event-triggered control,” arXiv:1708.02756, 2017.
[10] D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming.
Athena Scientific, 1996.
[11] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceeding of the 3rd International Conference for Learning
Representations, 2015.
[12] T. Charalambous, A. Ozcelikkale, M. Zanon, P. Falcone, and
H. Wymeersch, “On the resource allocation problem in wireless networked control systems,” in Proceedings of the 57th IEEE Conference
on on Decision and Control, 2017.

