Echo State Learning for Wireless Virtual Reality
Resource Allocation in UAV-enabled LTE-U Networks
Mingzhe Chen∗, Walid Saad†, and Changchuan Yin∗
∗ Beijing Laboratory of Advanced Information Network,

Beijing University of Posts and Telecommunications, Beijing, China 100876, Emails: chenmingzhe@bupt.edu.cn, ccyin@ieee.org.

† Wireless@VT, Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA, Email: walids@vt.edu.

arXiv:1708.00921v1 [] 2 Aug 2017

Abstract—In this paper, the problem of resource management
is studied for a network of wireless virtual reality (VR) users
communicating using an unmanned aerial vehicle (UAV)-enabled
LTE-U network. In the studied model, the UAVs act as VR control
centers that collect tracking information from the VR users over the
wireless uplink and, then, send the constructed VR images to the
VR users over an LTE-U downlink. Therefore, resource allocation
in such a UAV-enabled LTE-U network must jointly consider the
uplink and downlink links over both licensed and unlicensed bands.
In such a VR setting, the UAVs can dynamically adjust the image
quality and format of each VR image to change the data size of each
VR image, then meet the delay requirement. Therefore, resource
allocation must also take into account the image quality and format.
This VR-centric resource allocation problem is formulated as a
noncooperative game that enables a joint allocation of licensed and
unlicensed spectrum bands, as well as a dynamic adaptation of VR
image quality and format. To solve this game, a learning algorithm
based on the machine learning tools of echo state networks (ESNs)
with leaky integrator neurons is proposed. Unlike conventional ESN
based learning algorithms that are suitable for discrete-time systems,
the proposed algorithm can dynamically adjust the update speed
of the ESN’s state and, hence, it can enable the UAVs to learn the
continuous dynamics of their associated VR users. Simulation results
show that the proposed algorithm achieves up to 14% and 27.1%
gains in terms of total VR QoE for all users compared to Q-learning
using LTE-U and Q-learning using LTE.

I. I NTRODUCTION

Virtual reality (VR) services are viewed as one of the primary use cases of tomorrow’s 5G cellular networks [1]. Indeed,
wireless VR services will enable users to experience immersive
services, without the constraints imposed by conventional wired
VR systems. To enable the users to experience immersive VR
applications without any constraints, VR systems can be operated
using wireless networking technologies [1]. However, deploying
VR services over wireless cellular systems, such as LTE and
beyond, faces many challenges [1] that range from providing
reliable data transmission to optimizing tracking and resource
allocation.
Some recent works such as in [1]–[5] have studied a number
of problems related to wireless VR. In [1] and [2], the authors
provided a qualitative summary of the challenges of VR over
wireless. However, these works rely on simple toy examples, and
do not provide a rigorous analytical treatment of the wireless
networking challenges of VR. The authors in [3] proposed a
channel access scheme using the unlicensed band for wireless
multi-user VR system. The work in [4] investigated the use of
VR devices to control an unmanned drone with 2.4 GHz Wi-Fi
grid. However, the works in [3] and [4] only focus on how to use
WiFi technology to transmit VR images and do not consider the
resource allocation problem over downlink and uplink, licensed
and unlicensed bands for VR over cellular networks. In [5],
we proposed a wireless VR model that captures the tracking

accuracy, processing delay, and transmission delay and introduced
a machine learning algorithm for resource allocation. However,
our work in [5] is restricted to classical LTE networks in which
the VR devices and static ground base stations can access only
a single, licensed band. However, relying only on the licensed
band may not allow the cellular base stations to meet the high
data rate demands of the VR users. One potential solution is
to exploit the use of LTE over unlicensed (LTE-U) [6] bands
Moreover, in dense urban environments, the presence of high rise
buildings and obstacles, can present a major challenge for meeting
the delay needs for VR transmissions by using conventional,
terrestrial base stations. Instead, a flying base station carried by an
unmanned aerial vehicle (UAV) [7] that can move to an optimal
location to establish communication links with little blockage can
be considered to provide better connectivity and assist any ground
network. Moreover, all of the existing works such as in [1]–[5]
ignore the image quality and image format [8] in the improvement
of the delay of the VR users. Indeed, as the VR image quality and
format change, the data size of each image will be varied and,
hence, wireless base stations will need to dynamically adjust the
image’ quality and format to effectively service the users.
The main contribution of this paper is to introduce a novel
framework for enabling VR applications over UAV-based LTE-U
networks. To the best of our knowledge, this is the first work
that jointly considers licensed and unlicensed spectrum resource
allocation, image format and quality allocation for VR users
over UAV-based LTE-U networks. Hence, our key contributions
include:
•

•

•

For the considered VR applications over a wireless LTE-U
network, we develop an effective resource allocation scheme
that allocates licensed and unlicensed resource blocks over
uplink and downlink and adjusts the image quality and
format for each VR user’s image. The goal of this scheme
is to maximize the quality-of-experience (QoE) of all users
while meeting the delay requirement of the VR system. We
formulate the problem as a noncooperative game in which
the players are the UAVs. Each player seeks to find an
optimal resource allocation scheme to optimize the users’
QoE while meeting the delay requirement.
To solve this game, we propose a learning algorithm based
on echo state networks (ESNs) with leaky integrator neurons
[9]. This ESN-based algorithm can effectively find a Nash
equilibrium of the game. The proposed algorithm enables
the UAVs to learn the continuous dynamics of their associated VR users hence allowing adaptation to environmental
dynamics.
Simulation results show that the proposed algorithm can,
respectively, achieve 14% and 27.1% gains in terms of total

A. VR Image Transmission
UAV
Tracking
Information

VR image

WAP

WiFi User

Licensed uplink

Licensed downlink

Uplink resource block

Unlicensed downlink
Downlink resource block

Fig. 1. A network of immersive VR application.

Equirectangular

Cube

Pyramid

Fig. 2. The formats of a VR image.

VR QoE for all users compared to Q-learning using LTE-U
network and Q-learning using LTE network.
The rest of this paper is organized as follows. The system
model and problem formulation are presented in Section II. The
ESN-based resource allocation algorithm is proposed in Section
III. In Section IV, numerical simulation results are presented and
analyzed. Finally, conclusions are drawn in Section V.
II. S YSTEM M ODEL AND P ROBLEM F ORMULATION
Consider the downlink and uplink transmission of an LTE-U
network composed of a set B of B unmanned aerial vehicles
(UAVs) and W ground WiFi access points (WAPs). The UAVs
are deployed to act as flying LTE-U base stations to serve a
set U of U wireless VR users. Here, we consider dual-mode
UAVs that are able to access both the licensed and unlicensed
bands. The licensed band is used for both downlink and uplink
transmission while the unlicensed band is only used for the downlink transmission (due to the more stringent downlink traffic).
The downlink is used to transmit the VR images displayed on
each user’s VR device while the uplink is used to transmit the
tracking information that is used to construct a user’s VR image
as shown in Fig. 1. Here, the tracking information includes a
user’s orientation and location information. In our model, the
UAVs adopt an orthogonal frequency division multiple access
(OFDMA) technique over the licensed band and transmit over
uplink resource blocks and downlink resource blocks. Our LTEU model uses a time division duplexing (TDD) mode with duty
cycle method [6]. Under this method, LTE-U and WiFi users
will, separately, use the time slots on the unlicensed band. In
particular, UAVs will use a fraction ϑ of time for VR transmission
and will be muted for 1 − ϑ time which is allocated to the WiFi
transmission links.
The coverage of each UAV is considered to be a circular area
with radius r and each UAV only allocates resource blocks to the
users located in its coverage range. Each user can only connect
to a single UAV. Note that, in our model, we consider static users
that move only within confined areas, as is the case for typical
VR entertainment applications, such as immersive videos.

The data size of each VR image depends on the quality and
the format of each VR image. The quality of a VR image can
be captured by the resolution of a VR image (720p, 1080p, etc.).
When the resolution of each VR image increases, the number of
pixels that each UAV needs to construct the VR image will also
increase. We use a set L = {l1 , l2 , . . . , lK } to represent the level
of VR image quality where li ∈ L indicates that the quality of a
VR image is at level i and lK represents the highest quality of a
VR image [10] (lK > li , i = 1, 2, . . . , K −1). In VR, the standard
format of a VR image is equirectangular [8]. However, to reduce
its data size, a VR image can be transformed from the standard
equirectangular format into either cubic or pyramidal formats as
shown in Fig. 2. Using the cube or pyramid formats can reduce,
respectively, 25% and 80% of data size compared to using the
equirectangular format [8]. However, reducing the data size of a
VR image will decrease the QoE of each user. This is due to the
fact that as the data size of a VR image decreases, the number of
the pixels used to construct a VR image decreases [8] and, hence,
the users may not be able to see the entire 360◦ VR image. We
use a set F = {f1 , f2 , f3 } to represent the formats of a VR
image. Here, f1 to f3 , respectively, represent the pyramid, cube,
and standard formats. Based on the above formulation, for user i,
the data size of a VR image is defined as mi (l, f ), l ∈ L, f ∈ F .
Note that mi (lK , f3 ) is the maximum data size of a VR image
and m (l1 , f1 ) is the minimum data size of a VR image. The QoE
m(l,f )
of user i can be given by Qi (l, f ) = m(l
. Here, when the
K ,f3 )
quality of a VR image increases, the QoE of a user will increase,
which corresponds to the fact that watching a high definition video
is more comfortable than watching a standard definition video.
B. WiFi data rate analysis
For the considered WiFi system, a standard carrier sense multiple access with collision avoidance (CSMA/CA) protocol will be
used along with its corresponding RTS/CTS access mechanism.
The WAPs will use a CSMA/CA scheme with binary slotted
exponential backoff. We assume that one LTE time slot consists
of TW WiFi time slots. Based on the duty cycle mechanism,
the UAVs can occupy a fraction ϑ of TW time slots over the
unlicensed band while the WiFi users can occupy a fraction
(1 − ϑ) of TW time slots. Thus, the per WiFi user rate will be:
R (N ) (1 − ϑ)
,
(1)
N
where R (N ) is the saturation capacity of N WiFi users sharing
the same unlicensed band which is given by [6]. Given the rate
requirement of each WiFi user ξ, the fraction of the unlicensed
band time slot allocated to the LTE-U users can be given by
ϑ ≤ 1 − N ξ/R(N ).
Rw =

C. Delay Model
In the studied system, VR images are transmitted from the
UAVs to the users while tracking information is transmitted from
the users to the UAVs. Hence, the transmission delay over the
uplink and downlink will directly affect the users’ QoE and the
UAVs must guarantee the transmission delay for each user. In
consequence, we must consider the transmission delay in this
model. Next, we first introduce a probabilistic UAV channel
model and, then, we formulate the transmission delay of each

VR user. In the considered UAV channel model, probabilistic lineof-sight (LoS) and non-line-of-sight (NLoS) links are considered
as in [11]. NLoS links experience higher attenuations than LoS
links due to the shadowing and diffraction loss [11]. The LoS
and NLoS path loss of UAV k transmitting a VR image to user
i over the licensed resource blocks can be given by (in dB) [12]:




4πdij f
4πdij f
LoS
L
NLoS
L
gij
= 20 log
+ ηLoS
, gij
= 20 log
+ ηNLoS
,
c
c
where 20 log (dij f 4π/c) is the free space path loss with dij being
the distance between user i and UAV j, f being the carrier
L
L
frequency, and c being the speed of light. Here, ηLoS
and ηNLoS
represent, respectively, additional attenuation factors due to the
LoS/NLoS connections over the licensed resource blocks. The
LoS probability is given by [12]:

LoS
= (1 + X exp (−Y [φij − X]))−1 ,
(2)
Pr gij

where X and Y are constants that depend on the environment and
φij = sin−1 (hj /dij ) is the elevation angle. Then, the average
path loss from UAV j to user i will be [12]:


NLoS
NLoS
LoS
LoS
,
(3)
× gij
× gij
+ Pr gij
ḡij = Pr gij


LoS
NLoS
. The downlink rate of user i
= 1 − Pr gij
where Pr gij
associated with UAV j on the licensed resource blocks is:
Q
 X
cLij q ij =
qij,k Blog2 (1 + γij,k ),

(4)

k=1

where q ij = [qij,1 , . . . , qij,Q ] is the vector of resource blocks
that UAV j allocates to user i with qij,k ∈ {0, 1}. Here,
qij,k = 1 indicates that resource block k is allocated to user i.
γij,k =

σ2 +

k
PB ḡij
P

l∈Rk ,l6=j

k
PB ḡil

is the downlink signal-to-interference-

where v ij = [vij,1 , . . . , vij,V ] is the vector of resource
blocks that UAV j allocates to user i with vij,k ∈ {1, 0}.
u
γij,k
=

σ2 +

PU hk
P ij

l∈U k ,l6=j

PU hk
il

is the SINR between user i and UAV

j over resource block k with U k being the set of users that use
uplink resource blocks k and PU is each user’s transmit power.
Let A be the data size of the tracking information. For user i,
the total delay that consists of downlink and uplink transmission
delays that can be given by:
A
m (li , fi )
+
,
(7)
Dij (fi , li , sij , v ij , oij ) =
cij (sij , eij ) cij (v ij )
where the first term is the time that UAV j needs to transmit a
VR image to user i and the second term is the time that user i
needs to transmit the tracking information to UAV j.
D. Problem Formulation
Given the defined system model, our goal is to develop an
effective resource allocation scheme to allocate resource blocks,
image quality level, and image format to maximize the QoE
of all users while meeting the delay requirement of the VR
system. However, this problem must jointly account for the
coupled problems of user association and resource allocation.
Moreover, the delay of each user depends on the decisions of
all UAVs, due to interference and user association. Therefore,
we
as a noncooperative game [13] G =
i
h formulate this problem
. In this game, the players are the UAVs
B, {Aj }j∈B , {ūj }
j∈B
in set B, Aj represents the action set of each UAV j, and ūj
of each UAV j. Here, each action aj =
is the utility function

q j , v j , ej , lj , f j , consists
h of: (i) downlink
i licensed resource

allocation vector q j = q 1j , q 2j , . . . , q Uj j with Uj being the
number of the users associated
with UAV j, (ii) uplink resource

allocation vector v j = v 1j , v 2j , . . . , v Uj j , (iii) downlink un
licensed resource allocation vector ej = e1j , e2j , . . . ,eUj j ,
(iv) image quality allocation vector lj = l1j , . . . , lUjj , and
(v) image format allocation vector f j = f1j , . . . , fUj j . Here,
lij ∈ L, fij ∈ F , respectively, represent the image quality and
ϑ
format of user i. eij ∈ M where M = ϑ, ϑ2 , ..., M
is a finite
set of M -level fractions of UAV j’s total time that UAV j can
be used to transmit VR images. q j , v j and ej must satisfy:

plus-noise ratio (SINR) between UAV j and user i over licensed
resource block k. Rk represents the set of UAVs that use downlink
resource block k, B is the bandwidth of each resource block, PB
is the transmit power of UAV j which is assumed to be equal for
all UAVs, and σ 2 is the variance of the Gaussian noise. For UAV
k transmitting a VR image to user i using the unlicensed band,
the LoS and NLoS path loss will be (in dB) [12]:




4πdij f
4πdij f
LoS
U
NLoS
U
gij
= 20 log
+ ηLoS
, gij
= 20 log
+ ηNLoS
,
c
c
X
X
q
=
[1,
1,
.
.
.
,
1]
,
v ij = [1, 1, . . . , 1] ,
(8)
U
U
ij
where ηLoS and ηNLoS represent additional attenuation factors
i∈U
i∈U
j
j
due to, respectively, the LoS and NLoS connections over the
unlicensed band. The LoS probability and the average path loss
X
can be, respectively, calculated using a method that is similar to
eij ≤ 1, eij ∈ M,
(9)
(2) and (3). Therefore, the downlink rate of user i associated with
i∈Uj
UAV j over the unlicensed band is:

where (8) indicates that each UAV will allocate all of its resource
U
(5) blocks to the associated users. We assume that each UAV j
cU
ij (eij ) = eij ϑBu log2 1 + γij,k ,
where Bu is the bandwidth of the unlicensedP
band, eij is the chooses only one action at each time slot t. The utility function
fraction of ϑ over the unlicensed band with
of each UAV j can be given by:
i eij = 1, and
X
U
γij,k
is the SINR between UAV j and user i over the unlicensed
Qi (lij , fij )1{Dij (fij ,lij ,q ,vij ,eij )≤γD} , (10)
uj (aj , a−j ) =
ij
band. The total
rate of user i associated with UAV j

 downlink
i∈U
j
is cij q ij , eij = cLij q ij + cU
ij (eij ). The uplink rate of user i
where γD represents the maximal tolerable delay for each VR user
associated with UAV j can be given by:
(maximum
supported by the VR system being used), aj ∈ Aj
V
X

represents
an
action of UAV j and a−j is the actions of all UAVs
u
(6)
vij,k Blog2 1 + γij,k ,
cij (v ij ) =
other
than
UAV
j. In this game, the QoE and delay of all users
k=1

are analyzed over a period T and, hence, the utility function is:
T
1X
ūj (aj , a−j ) =
uj (aj , a−j ) .
(11)
T t=1
T
P
Let πj,aij = T1 1{aj,t =aij } = Pr (aj,t = aij ) be the probability
t=1

of UAV j using action aij . Here, aj,t is the action that UAV j
uses at time t and aj,t
i UAV j adopts action
h = aij indicates that
aij at time t. π j = πj,a1j , . . . , πj,aAj j is the action selection
strategy of UAV j with Aj being the number of actions of UAV
j. Based on the definition of the strategy, the utility function in
(11) can be rewritten as follows:
ūj (aj , a−j ) =

X

a∈A

uj (aj , a−j )

Y

k∈B

πk,ak

!

,

(12)

where a ∈ A with A being the action set of all UAVs. (12)
actually captures the average utility value of each UAV j over a
period T .
Hence, for this model, our objective is to solve the proposed
resource allocation game. One suitable solution is the mixedstrategy Nash equilibrium (NE), formally defined as follows[13]:
A mixed strategy profile π ∗ = (π∗1 , . . . , π ∗B ) = π ∗j , π ∗−j is a
mixed-strategy Nash equilibrium if, ∀j ∈ B and π j , we have:


(13)
ūj π ∗j , π ∗−j ≥ ūj π j , π∗−j ,
Q
P
πk,ak is the expected
uj (aj , a−j )
where ūj (π j , π−j ) =
a∈A

k∈B

utility of UAV j when it selects the mixed strategy π j . For our
game, the mixed-strategy NE for the UAVs represents a solution
of the game at which each UAV j can maximize the QoE for
its associated users while meeting the delay requirement of VR
system, given the actions of its opponents.
III. E CHO S TATE N ETWORKS FOR S ELF -O RGANIZING
R ESOURCE A LLOCATION

Next, we introduce a reinforcement learning (RL) algorithm
based on the neural network tools from echo state networks
with leaky integrator neurons [9] that can be used to solve
this game and find a mixed-strategy NE [13]. Traditional RL
algorithms such as Q-learning [14] typically use a Q-table to
record the utility values resulting from different actions and states.
However, in our model, each action consists of five components
which, if used in Q-learning, can result in an exponential increase
in the number of actions and its corresponding utility values.
Moreover, the update of Q-learning needs to traverse the Q-table
to find the update position which will significantly decrease the
update speed. Therefore, using a limited-size matrix to record
all of the needed utility values as done in Q-learning may not
be practical. In contrast, the proposed RL algorithm will use
echo state networks as an approximation method to record the
relationship between the actions, states and utility values and,
hence, it can record all of the needed actions, states and utility
values. Compared to the ESN algorithms in [5] and [6] that are
used for discrete-time systems, the proposed ESN with leaky
integrator neurons can dynamically adjust the update speed of
the ESN’s state and, hence, it can enable the UAVs to learn the
continuous dynamics of their associated VR users.

TABLE I
ESN- BASED L EARNING A LGORITHM FOR R ESOURCE A LLOCATION
Inputs: Mixed strategies xj,t
out
Initialize: W in
j , W j , W j , and y j = 0. Set π j,t uniformly
for each time t do.
(a) Estimate the value of the utility values ûj,t based on (16).
if rand(.) < ε
(b) Select action randomly.
else
(c) Choose action aj = arg max ûj,t (aj ).
aj ∈Aj

end if
(d) Broadcast the index of the mixed strategy to other UAVs.
(e) Receive the index of the mixed strategy as input xj,t .
(f) Perform action to calculate the actual value of utility ûj,t .
(g) Set the mixed strategy π j,t based on (14)
(h) Update the dynamic reservoir state based on (15).
(i) Update the output weight matrix based on (17).
end for
Output: Prediction y j,t

A. ESN Components
For each UAV, the proposed ESN algorithm consists of four
components: (1) inputs, (2) ESN model, (3) actions, and (4)
output, which are specified as follows:
 action of UAV j is the vector aj =
 • Actions: Each
q j , v j , ej , lj , f j that jointly considers the downlink licensed
resource allocation, uplink licensed resource allocation, downlink
unlicensed resource allocation, image quality allocation, and
image format allocation.
• Input: The input of our ESN is defined as a vector xj,t =
T
[x1 , · · · , xB ] where xj represents the index of the strategy of
UAV j at time t. To guarantee that any action always has a nonzero probability to be chosen, the ε-greedy exploration in [6]
is used. This mechanism is also used to harmonize the tradeoff
between exploitation and exploration. Therefore, the probability
of UAV j taking a certain action aj can be given by:
(
1 − ε + |Aεj | , arg max ûj,t (aj ) ,
aj ∈Aj
Pr (aj ) =
(14)
ε
|Aj | , otherwise,
P
uj (aj , a−j )π−j,a−j represents the exwhere ûj,t (aj ) =
a−j ∈A−j

pected utility of a P
UAV j with respect to the actions of other
UAVs, π−j,a−j = aj ∈Aj π (aj , a−j ) represents the marginal
probability
distribution over the action set of UAV j, and A−j =
Q
A
k6=j,k∈B k denotes the set of actions other than UAV j.
• Output: The output of the ESN model at time
 t is a vector
of utility values y j,t = yj1,t , yj2,t , . . . , yj|Aj |,t where yji,t is
the estimated value of utility ûj,t (aji ) due to action i of UAV j.
As time elapses,
 finally converge to the
 the ESN output, y j , will
utility ûj = ûj (aj1 ) , . . . , ûj aj|Aj | .
• ESN Model: An ESN model is a learning architecture that
can find the relationship between the input xj,t and output
y j,t , thus building the relationship between the UAVs’ actions,
strategies, and utility values. Mathematically, the ESN model
Aj ×(Nw +B+1)
consists of the output weight matrix W out
, the
j ∈R
in
Nw ×B+1
, and the recurrent matrix
input weight matrix W j ∈ R
W j ∈ RNw ×Nw with Nw being the number of the units.
B. ESN-Based Learning Algorithm for Resource Allocation
Next, we introduce the proposed ESN-based learning algorithm
to find a mixed strategy NE. The ESN model can store historical
ESN information such as input that can be used to find a fast

where C is a time constant, z is the leaking decay rate, δ
x
−e−x
is the step size, and f (x) = eex +e
−x is the tanh function. In
fact, (15) captures the difference between conventional ESN such
as in [5] and [6] and the proposed ESN with leaky integrator
neurons. When δ decreases, the ESN can be used for learning
the continuous dynamics of VR users such as data rate and data
size of a VR image. (15) shows that the reservoir state consists
of the ESN input at time t and the past dynamic reservoir state
that includes the historical reservoir states and ESN inputs. Thus,
the reservoir state can store the mixed strategy from time 0 to
time t. Using the reservoir state with the output weight matrix,
the proposed ESN algorithm can estimate the utility value ûj .
The estimation of ûj is:


(16)
y j,t = W out
j,t µj,t ; xj,t ,
where W out
j,t is the output weight matrix at time slot t. To enable
the ESN to predict the utility value ûj using reservoir state µj,t
and input xj,t , we must train the output matrix W out
using a
j
linear gradient descent approach, which can be given by:
= W out
ji,t +
i of W out
j,t ,

λ (ûji,t −

yji,t ) µT
j,t ,

Parameter
PB
PU
σ
RTS
Fu
C
CTS
FC

Value
15 dBm
20 dBm
-94 dBm
352 µs
20 Mbit
0.44
304 µs
2 Gbit

IV. S IMULATION R ESULTS
In our simulations, a circular network area having a radius
r = 500 m is considered with U = 20 uniformly distributed
users and B = 5 uniformly distributed UAVs. The HTC VR
device is considered in our simulations [15] and, hence, the rate
requirement for VR transmission is set to 51.2 Mbits/s. The
requirement of total delay that consists of the downlink and uplink
transmission delay is 20 ms [16]. Other simulation parameters
are listed in Table II. We compare our approach with: a) ESN
algorithm in [5], b) Q-learning algorithm within an LTE-U, and
c) Q-learning algorithm applied to an LTE system. All statistical
results are averaged over 5000 independent runs.

Value
5, 5
1.2, 23
50 µs
1500 byte
304 µs
16 µs
4 Mbps
1, 20

Parameter
M
z
ς
X
δ
K
Y
N

Value
5
0.04
20 dB
11.9
1
3
0.13
8

ESN-based learning algorithm
Traditional ESN algorithm
Q-learning algorithm in LTE-U
Q-learning algorithm in LTE

25
24
23
22
21
20
19
18
1

2

3

4

5

6

Number of UAVs

Fig. 3. Average delay of each user vs. number of UAVs.

(17)

where
λ is the learning rate, and ûji,t
is the actual utility value. Here, ûji,t is estimated by the utility
value resulting from the actions performed by each UAV during
each time slot t. Based on the above formulation, the distributed
ESN-based learning algorithm performed by every UAV j is
summarized in Table I.
In this algorithm, each UAV can approximate the relationship
between the actions, strategies, and utility values. During each
iteration, the ESN-based algorithm can record the the strategies
that each UAV uses and the corresponding utility values ûj . By
scaling the input weight matrix W in
j and input xj , the ESN-based
algorithm can satisfy the convergence conditions of [6, Theorem
2]. As time elapses, each UAV j’s output resulting from action i
will converge to a final value ûji . At this convergence point, the
output results ûj indicates that the proposed algorithm converges
to a mixed strategy-NE. This is due to the fact when ûj (aj ) is
maximum, Pr (aj ) will be maximum (Pr (aj ) = 1 − ε + |Aεj | ).
The strategy depends on ûj (aj ). When the proposed learning
algorithm finds the optimal ûj (aj ), the strategy will be optimal
and, hence, ūj (aj , a−j ) will be maximized.

Parameter
V, S
U , ηU
ηLoS
NLoS
DIFS
E [S]
ACK
SIFS
ξ
L , ηL
ηLoS
NLoS

26

8
ESN-based learning algorithm
Traditional ESN algorithm
Q-learning algorithm in LTE-U
Q-learning algorithm in LTE

7

Total VR QoE for all users

W out
ji,t+1
out
W ji,t is row

TABLE II
SYSTEM PARAMETERS

Average delay for each user (ms)

converging process from the initial state to the mixed-strategy
NE. The historical ESN information is stored in the reservoir
state which can be given by:

µj,t = (1 − δCz) µj,t−1 + δCf W j µj,t−1 + W in
j xj,t , (15)

6

5

4

3

2
1

2

3

4

5

6

Number of SBSs

Fig. 4. Total VR QoE of all users vs. number of UAVs.

Fig. 3 shows how the average delay per user changes as the
number of UAVs varies. From Fig. 3, we can see that as the
number of UAVs increases, the average delay of all considered
algorithms starts by first rapidly decreasing and then the decrease
becomes slower. This is due to the fact that, as the number of
UAVs increases, each user will have more choices of UAVs to
connect to, and, hence, the distance between the UAVs and the
users decreases. However, as the number of UAVs keeps increasing, the interference from the UAVs to the users increases. Fig.
3 also shows that when the delay reaches the delay requirement,
the delay resulting from the proposed algorithm remains constant
as the number of UAVs increases. This is due to the fact that our
purpose is to maximize the QoE utility value while meeting the
delay requirement of the VR system and, hence, once the delay
requirement is met, the proposed scheme will no longer need to
further reduce it.
In Fig. 4, we show how the total VR QoE for all users changes
as the number of UAVs varies. From Fig. 4, we can see that, as
the number of UAVs increases, the total QoE utility values of
all considered algorithms will increase. This is due to the fact
that, as the number of UAVs increases, the number of the users

40.1% gain in terms of the number of the users that meet the
delay requirement. This is due to the fact the proposed algorithm
use the ESN to record all the needed utility values to estimate
the future utility values and allocate the unlicensed band to the
users so as to increase the data rate of each user.

5
Proposed ESN algorithm
Traditional ESN algorithm
Q-learning algorithm in LTE-U
Q-learning algorithm in LTE
Convergent point

Total utility value of each UAV

4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
5

10

15

20

25

Number of iterations

Percentage of users meeting their delay requirements (%)

Fig. 5. Convergence of the proposed algorithm and Q-learning. Here, we use the
ESN algorithm in [6] as a benchmark.
80
ESN-based learning algorithm
Traditional ESN algorithm
Q-learning algorithm in LTE-U
Q-learning algorithm in LTE

70

60

V. C ONCLUSION
In this paper, we have developed a novel framework that uses
flying UAV-enabled networks to provide service for VR users in
an LTE-U system. We have proposed a novel resource allocation
framework for optimizing the users’ QoE while meeting the delay
requirement for each user. We have formulated this problem as a
noncooperative game between the UAVs and we have developed a
novel algorithm based on the machine learning tools of echo state
networks with leaky integrator neurons. The proposed algorithm
can dynamically adjust the update speed of the ESN’s state and,
hence, it can enable the UAVs to learn the continuous dynamics
of their associated VR users. Simulation results have shown that
the proposed approach yields significant gains in terms of total
QoE utilities for all users compared to conventional approaches.
R EFERENCES

50

40

30

20

10
1

2

3

4

5

6

Number of UAVs

Fig. 6. Percentage of users meeting their delay requirements vs. number of UAVs.

located in each UAV’s coverage decreases and, hence, each UAV
can allocate more resource to its associated users. Fig. 4 also
shows that, for the case with 6 UAVs, the proposed algorithm
can achieve, respectively, up to 14% and 27.1% gains in terms of
total VR QoE for all users. These gains stem from the fact that
the proposed algorithm can use the leaky integrator neurons to
adjust the learning speed of ESNs. Hence, it enables the UAVs
to learn the utility values resulting from the dynamical users and
use the already learned utility value to estimate the new utility
value that must be learned.
Fig. 5 shows the number of iterations needed till convergence
for all considered algorithms. In this figure, we can see that, as
time elapses, the QoE utilities for both considered algorithms will
increase until convergence to their final values. Fig. 5 also shows
that the proposed algorithm yields up to 11.3% and 17.4% gains in
terms of the number of the iterations needed to reach convergence
compared to traditional ESN algorithm and Q-learning. This
implies that the proposed algorithm can use the ESN with the
leaky integrator neurons to adjust the learning speed and, hence
it can accurately approximate the relationship between the actions,
strategies, and utility values.
Fig. 6 shows how the percentage of users meeting their delay
requirements changes as the number of UAVs varies. From Fig.
6, we can see that the percentage of VR users meeting their delay
requirements increases, as the number of UAVs increases. This
stems from the fact that as the number of UAVs increases, the
users have more UAV choices and the distances from the UAVs to
the users decrease and, hence, the data rate of each user increases.
Fig. 6 also shows that the proposed algorithm can achieve up to

[1] E. Baştuğ, M. Bennis, M. Médard, and M. Debbah, “Towards interconnected
virtual reality: Opportunities, challenges and enablers,” IEEE Communications Magazine, vol. 55, no. 6, pp. 110–117, June 2017.
[2] AT&T, “Enabling mobile augmented and virtual reality with 5G networks,”
http://about.att.com/innovationblog/foundry_ar_vr.
[3] J. Ahn, Y. Yong Kim, and R. Y. Kim, “Delay oriented VR mode WLAN
for efficient wireless multi-user virtual reality device,” in Proc. of IEEE
International Conference on Consumer Electronics, Las Vegas, NV, USA,
March 2017.
[4] C. Wu, Z. Wang, and S. Yang, “Drone streaming with Wi-Fi grid aggregation
for virtual tour,” available online: arxiv.org/abs/1605.09486, May 2016.
[5] M. Chen, W. Saad, and C. Yin, “Virtual reality over wireless networks:
Quality-of-service model and learning-based resource management,” available online: arxiv.org/abs/1703.04209, Mar. 2017.
[6] M. Chen, W. Saad, and C. Yin, “Echo state networks for self-organizing
resource allocation in LTE-U with uplink-downlink decoupling,” IEEE
Transactions on Wireless Communications, vol. 16, no. 1, January 2017.
[7] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Unmanned aerial
vehicle with underlaid device-to-device communications: Performance and
tradeoffs,” IEEE Transactions on Wireless Communications, vol. 15, no. 6,
pp. 3949–3963, June 2016.
[8] D. Pio, “Facebook engineering,” https://www.facebook.com/Engineering/
videos/10153781047207200/.
[9] H. Jaeger, “The echo state approach to analysing and training recurrent
neural networks-with an erratum note,” German National Research Center
for Information Technology GMD Technical Report, vol. 148, no. 34, pp.
13, 2001.
[10] C. Charles, B. Mark, L. Ajay, M. Sean, M. Paul, and W. Kevin, “Virtual and
augmented reality-How do they affect the current service delivery and home
and network architectures?,” https://www.arris.com/globalassets/resources/
white-papers/exec-paper_virtual-and-augmented-reality_final.pdf.
[11] M. Chen, M. Mozaffari, W. Saad, C. Yin, M. Debbah, and C. S. Hong,
“Caching in the sky: Proactive deployment of cache-enabled unmanned
aerial vehicles for optimized quality-of-experience,” IEEE Journal on
Selected Areas in Communications, vol. 35, no. 5, pp. 1046–1061, May
2017.
[12] A. Al-Hourani, S. Kandeepan, and A. Jamalipour, “Modeling air-to-ground
path loss for low altitude platforms in urban environments,” in Proc.
of IEEE Global Communications Conference (GLOBECOM), Austin, TX,
USA, December 2014.
[13] Z. Han, D. Niyato, W. Saad, T. Basar, and A. HjÃ¸rungnes, Game Theory in
Wireless and Communication Networks: Theory, Models, and Applications,
Cambridge University Press, 2012.
[14] M. Bennis and D. Niyato, “A Q-learning based approach to interference
avoidance in self-organized femtocell networks,” in Proc. of IEEE Global
Commun. Conference (GLOBECOM) Workshop on Femtocell Networks,
Miami, FL, USA, Dec. 2010.
[15] HTC, “HTC Vive,” https://www.vive.com/us/.
[16] M. Abrash, “What VR could, should, and almost certainly will be
within two years,” https://media.steampowered.com/apps/abrashblog/Abrash
%20Dev%20Days%202014.pdf.

