Echo State Transfer Learning for Data Correlation
Aware Resource Allocation in Wireless Virtual Reality
Mingzhe Chen∗, Walid Saad†, Changchuan Yin∗ , and Mérouane Debbah‡

∗ Beijing Key Laboratory of Network System Architecture and Convergence,
Beijing University of Posts and Telecommunications, Beijing, China 100876, Emails: chenmingzhe@bupt.edu.cn, ccyin@ieee.org.
† Wireless@VT, Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA, Email: walids@vt.edu.
‡ Mathematical and Algorithmic Sciences Lab, Huawei France R & D, Paris, France,
Email: merouane.debbah@huawei.com.

arXiv:1711.09173v1 [cs.IT] 25 Nov 2017

Abstract—In this paper, the problem of data correlation-aware
resource management is studied for a network of wireless virtual
reality (VR) users communicating over cloud-based small cell networks (SCNs). In the studied model, small base stations (SBSs)
with limited computational resources act as VR control centers that
collect the tracking information from VR users over the cellular
uplink and send them to the VR users over the downlink. In such
a setting, VR users may send or request correlated or similar data
(panoramic images and tracking data). This potential spatial data
correlation can be factored into the resource allocation problem
to reduce the traffic load in both uplink and downlink. This VR
resource allocation problem is formulated as a noncooperative game
that allows jointly optimizing the computational and spectrum
resources, while being cognizant of the data correlation. To solve
this game, a transfer learning algorithm based on the machine
learning framework of echo state networks (ESNs) is proposed.
Unlike conventional reinforcement learning algorithms that must be
executed each time the environment changes, the proposed algorithm
can intelligently transfer information on the learned utility, across
time, to rapidly adapt to environmental dynamics due to factors
such as changes in the users’ content or data correlation. Simulation
results show that the proposed algorithm achieves up to 16.7% and
18.2% gains in terms of delay compared to the Q-learning with data
correlation and Q-learning without data correlation. The results also
show that the proposed algorithm has a faster convergence time than
Q-learning and can guarantee low delays.

I. I NTRODUCTION

Virtual reality (VR) can enable users to virtually hike the
Grand Canyon or engage in a secret adventure as a video game
hero without leaving their room. However, due to the wired
connections of conventional VR devices, the users can only
take a restricted set of actions which, in turn, limits the VR
application space. To enable immersive VR applications, VR
systems can be operated using wireless networking technologies
[1]. However, operating VR devices over wireless small cell
networks (SCNs) faces many challenges [1] that include effective
image compression, tracking, and low-latency computation and
communication.
The existing literature has studied a number of problems
related to wireless VR such as in [1]–[4]. The authors in [1]
exposed the future challenges of VR systems over a wireless
network. However, this work is a qualitative survey that does
not provide any rigorous wireless VR model. In [2], a channel
access scheme for wireless multi-user VR system is proposed.
The authors in [3] proposed an alternate current magnetic fieldbased tracking system to track the position and orientation of a

This work was supported in part by the National Natural Science Foundation
of China under Grants 61671086 and 6162910, by the U.S. National Science
Foundation under Grants IIS-1633363, CNS-1460316, and CNS-1617896 and
by the ERC Starting Grant 305123 MORE (Advanced Mathematical Tools for
Complex Network Engineering).

VR user’s head. However, the recent works in [2] and [3] do not
develop a VR model that can capture all factors of VR QoS and
they only analyze a single VR metric such as delay or tracking
accuracy. In [4], we proposed a wireless VR model that captures
the tracking accuracy, processing delay, and transmission delay
and proposed a machine learning based algorithm to solve the
resource allocation problem. However, the work in [4] focuses
only on spectrum allocation that ignores the correlation between
the data of the VR users. Indeed, the VR data (tracking data or
VR image data) pertaining to different users can be potentially
correlated. For example, when the VR users are watching a
football game with different perspectives, the cloud will need to
only transmit one 360◦ image to the SBS, then the SBS can rotate
the image and transmit it to different users. In this case, the use
of data correlation to reduce the traffic load in the transmission
of tracking information and VR images can improve the delay.
The main contribution of this paper is a novel framework for
enabling VR applications over wireless cellular networks. To
the best of our knowledge, this is the first work that jointly
considers data correlation, spectrum resource allocation, and
computational resource allocation for VR over cellular networks.
In this regard, our key contributions include:
• We propose a novel VR model to jointly capture the downlink and uplink transmission delay, backhaul transmission
delay, and computational time thus effectively quantifying
the VR delay for all users in a wireless VR network.
• For the considered wireless VR applications, we analyze
the allocation of resource blocks jointly over, the uplink
and downlink, along with the allocation of computational
resources in the uplink. We formulate the problem as a
noncooperative game in which the players are the small
base stations (SBSs). Each player seeks to find an optimal
resource allocation scheme to optimize a utility function that
captures the VR delay.
• To find a Nash equilibrium of this game, we propose a
transfer learning algorithm based on echo state networks
(ESNs) [5]. The proposed algorithm can intelligently transfer
information on the learned utility across time, and, hence,
allow adaptation to environmental dynamics due to factors
such as changes in the users’ data correlation.
• Simulation results show that the proposed algorithm can,
respectively, yield 16.7% and 18.2% gains in terms of delay
compared to Q-learning with data correlation and Q-learning
without data correlation.
II. S YSTEM M ODEL AND P ROBLEM F ORMULATION
Consider the downlink and uplink transmissions of an SCN
servicing a set U of U wireless VR users and a set B of B

SBSs. Here, the downlink is used to transmit the VR images
displayed on each user’s VR device while the uplink is used to
transmit the tracking information that is used to determine each
VR user’s location and orientation. The SBSs are connected to a
cloud via capacity-constrained backhaul links and the SBSs serve
their users using the cellular band. VF represents the maximum
backhaul transmission rate for all users. Here, we focus on
entertainment VR applications such as watching immersive videos
and playing immersive games.
In our model, the SBSs adopt an orthogonal frequency division
multiple access (OFDMA) technique and transmit over a set of V
of V uplink resource blocks and a set of S of S downlink resource
blocks. The coverage of each SBS is a circular area with radius r
and each SBS only allocates resource blocks to the users located
in its coverage range. We also assume that the resource blocks of
each SBS will all be allocated to the associated users.
A. Data Correlation Model
1) Downlink Data Correlation Model: In VR wireless networks, multiple VR users may play the same immersive game
with different locations and orientations. In this case, the cloud
can exploit the data correlation between the users that are playing
the same immersive game to reduce the traffic load of backhaul
links. For example, when the users are watching the same immersive sports game, the cloud can extract the difference between the
VR images of these users and will need to only transmit to an
SBS the data that is unique to each user. However, when the VR
users are playing different immersive games, the data correlation
between the users is low and, hence, the cloud needs to transmit
entire VR images to the associated VR users. In order to define the
data correlation of VR images, we first assume that the number
of pixels that user i needs to construct the VR images is Ni
and the number of different pixels between any pair of users i
and k is Nik . Here, Nik is calculated by the cloud using image
processing methods such as motion search [6]. Then, the data
correlation between user i and user k can be defined as follows:
Nik
φik =
,
(1)
Ni + Nk
where Nk is the number of pixels that user k needs to construct
the VR images during a period. Indeed, (1) captures the difference
between the images of users i and j. From (1), we can see that,
when user i and user k are associated with the same SBS, the
cloud needs to only transmit Ni + Nj − (Ni + Nj ) φij pixels to
that SBS.
2) Uplink Data Correlation Model: In the uplink, the users
must transmit the tracking information to the SBSs. The tracking
information is collected by the sensors placed at a VR user’s
headset or near the VR user. It has been shown that, for commonly
used data-gathering applications, the data source can be modeled
as a Gaussian field [7]. The uplink data is collected by the sensors
and, hence, the uplink data can be assumed to follow the Gaussian
distribution. We can assume that the tracking data, Xi , collected
by each VR user i is a Gaussian random variable with mean µi
and variance σi2 . In wireless VR, observations from proximal VR
devices are often correlated due to the dense deployment density.
Hence, we consider the power exponential model [8] to capture
the spatial correlation of VR tracking data. Here, the covariance
σij between user i and user j separated by distance dij is:
α
(2)
σ = cov (X , X ) = σ σ e−dij /κ ,
ij

i

j

i j

where α and κ capture the significance of distance variation on
data correlation.
B. Delay Model
In our model, the VR images are transmitted from the cloud to
the SBSs then to the users. The tracking information is transmitted
from the users to the SBSs and processed at each corresponding
SBS. In this case, the backhaul links are only used for VR image
transmission and the transmission rate of each VR image from the
cloud to the SBS can be given as VF i = VUF [9]. Here, we assume
that the backhaul transmission rate of each user is equal and we do
not consider the optimization of the backhaul transmission. In a
VR model, we need to capture the VR transmission requirements
such as high data rate, low delay, and accurate tracking and,
hence, we consider the transmission delay as the main VR QoS
metric of interest. The downlink rate of user i associated with
SBS j is:
S
X
cij (sij ) =
sij,k Blog2 (1 + γij,k ),
(3)
k=1

where sij = [sij,1 , . . . , sij,S ] is the vector of resource blocks
that SBS j allocates to user i with sij,k ∈ {1, 0}. Here,
sij,k = 1 indicates that resource block k is allocated to user i.
γij,k =

N02 +

PB hk
P ij

l∈Rk ,l6=j

PB hk
il

is the signal-to-interference-plus-noise

ratio (SINR) between user i and SBS j over resource block k. Rk
represents the set of the SBSs that use downlink resource block k,
B is the bandwidth of each subcarrier, PB is the transmit power
of SBS j which is assumed to be equal for all SBSs, N02 is the
k −β
variance of the Gaussian noise and hkij = gij
pij is the path loss
k
between user i and SBS j over resource block with gij
is the
Rayleigh fading parameter, dij is the distance between user i and
SBS j, and β is the path loss exponent. Based on (1) and (3), the
downlink transmission delay at time slot t is:
)
Li (φmax
) Li (φmax
i
i
,
(4)
+
Dij (Li (φmax
) , sij ) =
i
cij (sij )
VF i
where Li (φmax
) is the data that user i needs to construct a VR
i
image during a period and φmax
= max (φik ) is the maxii
k∈Uj ,k6=i

mum downlink data correlation between user i and other users
associated with SBS j. Finding the maximum data correlation
allows minimizing the downlink transmission data transmitted in
the downlink and that will be used construct a VR image. Here,
the first term is the transmission time from SBS j to user i and
the second term is the transmission time from the cloud to SBS
j. We assume that PU is the transmit power of each user which is
assumed to be equal for all users. The bandwidth of each uplink
resource block is also B. In this case, the uplink rate of each user
i associated with SBS j is:
V
X

u
,
(5)
cij (v ij ) =
vij,k Blog2 1 + γij,k
k=1

where v ij = [vij,1 , . . . , vij,V ] is the vector of resource
blocks that SBS j allocates to user i with vij,k ∈ {1, 0}.
u
γij,k
=

σ2 +

PU hk
P ij

l∈U k ,l6=j

PU hk
il

is the SINR between user i and SBS

j over resource block k with U k represents the set of users that
use uplink resource blocks k. In this case, the uplink transmission

K (σmax )

delay can be given by ciij (vi ij ) where Ki is the data that needs to
be transmitted and σimax = max (σik ) is the maximum uplink
k∈Uj ,k6=i

data correlation between user i and other SBS j’s associated
users. Similarly, finding the maximum data correlation allows
minimizing the uplink transmission data that SBS j uses to
determine user i’s location and orientation.
In the uplink, the tracking information can be directly processed by the SBSs that have limited computational power. The
computational resource of each SBS, c, represents its ability to
compute the tracking data. Each SBS j will allocate the total
computational power to the associated users and, hence, mij is
used to represent
Pthe computational power that SBS j allocates
to user i with
i∈Uj mij = m. Uj represents the set of the
users associated with SBS j. The computation time of SBS
j
K (σimax )
that processes the tracking data collected by user i is i mij
and the total uplink delay can be given by:
Ki (σimax ) Ki (σimax )
u
,
(6)
+
Dij
(Ki (σimax ), v ij , mij ) =
cij (v ij )
mij
where the first term is the transmission time from user i to SBS
j and the second term is the computation time for user i’ data.
The computation time depends on the computational resources
that SBS j allocates to each user that will affect the uplink delay.
C. Utility Function Model
In order to jointly consider the transmission delay in both
uplink and downlink, we introduce a method based on the
framework of multi-attribute utility theory [10] to construct an
appropriate utility function to capture transmission delay in both
uplink and downlink. We first introduce the utility functions of
transmission delay in uplink and downlink, separately. Then, we
formulate the utility function based on [10].
The utility function of downlink transmission delay is constructed based on the normalization of downlink transmission
delay, which can be given by:
D̄ij (Li (φmax
) , sij ) =
i
(
Dij,max −Dij (Li (φmax
),sij )
i
, Dij (Li (φmax
), sij ) ≥ γD ,
i
Dij,max −γD
(7)
1,
Dij (Li (φmax
),
sij ) < γD ,
i
where γD is the maximal tolerable delay for each VR user (maximum supported by the VR system being used) and Dij,max =
max (Dij (Li (0) , sij )) is the maximum transmission delay. From
sij

(7), we can see that, when the downlink transmission delay is
smaller than γD , the utility value will remain at 1. This is due to
the fact when the delay meets the system requirement, the network
will encourage the SBSs to reallocate the resource blocks to other
users. The utility function for the uplink transmission is:
u
D̄ij
(Ki (σimax ) , v ij , mij ) =
( u
u
Dij,max−Dij
(Ki (σimax ),vij ,mij )
u
Dij,max
−γD

1,

u
u
, Dij
(Ki (σimax ), v ij , mij ) ≥ γD
,
u
max
u
Dij (Ki (σi ), v ij , mij ) < γD
,

(8)

u
where γD
is the maximal tolerable delay for the
u
VR tracking information transmission and Dij,max
=
u
max Dij (Ki (0) , v ij , mij ) is the maximal uplink delay.
v ij ,mij

Based on (7) and (8), the total utility function that captures both
downlink and uplink delay for user i associated with SBS j is:
Uij (sij , v ij , mij ) =
u
D̄ij (Li (φmax
) , sij ) D̄ij
(Ki (σimax ) , v ij , mij ) .
(9)
i

) and Ki (σimax ) are determined by the user assoHere, Li (φmax
i
ciation scheme. To capture the gain that stems from the allocation
of the resource blocks and the computational capabilities, we state
the following result:
Theorem 1. The utility gain of user i’s delay due to an increase
in the amount of allocated resource blocks and computational
resources is:
i) The gain that stems from an increase in the allocated uplink
resource blocks, ∆Uij , is given by:

∆Uij





1

u
f
,
cij (∆v ij ) ≫ cij (v ij ) ,

D̄

ij
v
c


 ij ( ij ) 

cij (∆v ij )
u
=
, cij (∆v ij ) ≪ cij (v ij ) ,
fD̄ij
2



cij (vij )


cij (∆v ij )


u
, else,
 fD̄ij
2
cij (v ij ) +cij (v ij )cij (∆v ij )

(10)



K (σmax )x
max
u (x) = D̄ij (Li (φ
where fD̄ij
) , sij ) Dui i −γ u .
i
ij,max
D
ii) The gain that stems from the increase in the number of
downlink resource blocks allocated to user i, ∆Uij , is:



1

, cij (∆sij ) ≫ cij (sij ) ,
f


 D̄ij  cij (sij ) 
cij (∆sij )
(11)
fD̄ij c (s )2 , cij (∆sij ) ≪ cij (sij ) ,
∆Uij =


 ij ij


cij (∆sij )

fD̄ij c (s )2 +c (s )c (∆s ) , else,
ij
ij
ij
ij
ij
ij


Li (φmax
)x
u
i
(Ki (σimax ), v ij , mij ) Dij,max
where fD̄ij(x) = D̄ij
−γD .
iii) The gain that stems from the increase in the amount of
computational resources, ∆m, allocated to user i, ∆Uij , is:
∆Uij =

D̄ij (Li (φmax
) , sij )
i

!
Ki (σimax )∆m

. (12)
u
u
Dij,max
−γD
(mij(mij+∆m))

Proof. The details of the proof are found in [11]

From Theorem 1, we can see that the allocation of spectrum
and computational resources jointly determines the delay utility.
Indeed, Theorem 1 will provide guidance for the SBSs when they
select actions in the learning algorithm proposed in Section III.
D. Problem Formulation
Given the defined system model, our goal is to develop an
effective resource allocation scheme that allocates resource blocks
and computational power to maximize the utility functions of all
users. However, the maximization problem depends not only on
the allocation of resource blocks and computational resources
but also on the user associations. Moreover, the utility value
of each SBS depends not only on its own choice of resource
allocation scheme but also on the remaining SBSs’ schemes.
In addition, the data correlation among the users varies as the
period changes, which will affect the resource allocation and
user association.
In this case, we firsti formulate a noncooperative
h
. In this game, the players
game G = R, {Aj }j∈R , {Uj }
j∈R
are the SBSs, Aj represents the action set of each SBS j,
and Uj is the utility function of each SBS j. The action aj
of SBS
 j consists of: (i)
 downlink resource allocation vector
sj = s1j , s2j , . . . , sUj j , (ii) uplink resource allocation vector
resource
v j = v 1j , v 2j , . . . , v Uj j , and (iii) computational

allocation vector mj = m1j , m2j , . . . , mUj j . Here, mij ∈
c 2c
M, i ∈ Uj where M = M
, M , ..., c is a finite set of M level

fractions of SBS j’s total computational resource mj . We assume
that each SBS j adopts one action at each time slot t. Then, the
utility function of each SBS j can be given by:
T
1 XX
uj (aj , a−j ) =
Uij,t (sij , v ij , cij ),
(13)
T t=1
i∈Uj

where aj ∈ Aj is an action of SBS j and a−j denotes
the action profile of all SBSs other than SBS j. Indeed, (13)
captures the average utility value of each SBS j. Let πj,aij =
T
P
1
1{aj,t =aij } = Pr (aj,t = aij ) be the probability of SBS j
T
t=1

using action aij . aj,t represents the action that SBS j uses at
time t and ahj,t = aij denotes thati SBS j adopts action aij at
time t. π j = πj,a1j , . . . , πj,a|A |j is the action selection mixed
j
strategy of SBS j with |Aj | being the number of actions of SBS
j. Based on the definition of the strategy, the utility function in
(13) is given by:
!
T
Y
X
1X
uj (aj , a−j )=
Uj (aj , a−j ) πj,aj , (14)
Uj,t (aj , a−j ) =
T t=1
j∈B
a∈A

where a ∈ A with A being the action set of all SBSs.
One suitable solution for this game is the mixed-strategy Nash
equilibrium (NE), formally defined as follows [12]:
 A mixed
strategy profile π ∗ = (π∗1 , . . . , π ∗B ) = π ∗j , π∗−j is a mixedstrategy Nash equilibrium if, ∀j ∈ R and πj , we have:


(15)
uj π ∗j , π ∗−j ≥ uj π j , π∗−j ,
Q
P
πj,aj is the exUj (aj , a−j )
where uj (π n , π−n ) =
a∈A

j∈B

pected utility of SBS j when it selects the mixed strategy πj .For
our game, the mixed-strategy NE for the SBSs represents a
solution of the game at which each SBS j can minimize the
delay for its associated users, given the actions of its opponents.
III. E CHO S TATE N ETWORKS FOR S ELF -O RGANIZING
R ESOURCE A LLOCATION
Next, we introduce a transfer reinforcement learning (RL)
algorithm that can be used to find an NE of the VR game. To
satisfy the delay requirement for the VR transmission, we propose
a transfer RL algorithm based on the neural networks framework
of ESNs [16]. Traditional RL algorithms such as Q-learning
typically rely on a Q-table to record the utility value. However,
as the number of players and actions increases, the number of
utility values that the Q-table needs to include will increase
exponentially and, hence, the Q-table may not be able to record
all of the needed utility values. However, the proposed algorithm
uses a utility function approximation method to record the utility
value and, hence, it can be used for large networks and large
utility spaces. Moreover, a dynamic network in which the users’
computational resources and data correlation may change across
the time, traditional RL algorithms need to be executed each
time the network changes. However, the proposed ESN transfer
RL algorithm can find the relationship of the utility functions
when the environment changes. After learning this relationship,
the proposed algorithm can use the historic learning result to find
a mixed strategy NE.
The proposed transfer RL algorithm consists of two components: (i) ESN-based RL algorithm and (ii) ESN-based transfer
learning algorithm. The ESN-based RL algorithm is based on

our work in [4], and, thus, here, we just introduce the ESN-based
transfer learning algorithm.
We first assume that, before the users’ state information
changes, the mixed strategy, action, and utility of each SBS j are
π j , aj and ûj (aj , a−j ), while the strategy, action, and utility of
SBS j, after the users’ state information changes, are π j , aj , and
û′j (aj , a−j ). Since the number of users associated with SBS j is
unchanged, the action and mixed strategy sets of SBS j will not
change when the users’ state information changes. In this case,
the proposed ESN-based transfer learning algorithm is used to
find the relationship between ûj (aj , a−j ) and û′j (aj , a−j ) when
SBS j only knows ûj (aj , a−j ). This means that the proposed
algorithm can transfer the information from the already learned
utility ûj (aj , a−j ) to the new utility û′j (aj , a−j ) that must be
learned. The ESN-based transfer learning algorithm of each SBS
j consists of three components: (a) input, (b) output, and (c) ESN
model, which are given by:
• Input: The ESN-based transfer learning algorithm takes the
strategies of the SBSs and the action of SBS j uses at time t as
T
input which is given by x′t,j = [π1 , · · · , πB , aj,t ] .
• Output: The output of the ESN-based transfer learning
algorithm at time t is the deviation of the utility values when
′
the users’ information changes yj,t
= û′j (aj,t ) − ûj (aj,t ).
• ESN Model: An ESN model is used to find the relationship
′
between the input x′t,j and output yt,j
. The ESN model consists
′
out
of the output weight matrix W j ∈ R1×Nw and the dynamic
′
reservoir containing the input weight matrix W jin ∈ RNw ×B+1 ,
and the recurrent matrix W ′j ∈ RNw ×Nw with Nw being
the number of the dynamic reservoir units. Here, the dynamic
reservoir is used to store historic ESN information that includes
input, reservoir state, and output. This information is used to build
the relationship between the input and output. The update process
of the dynamic reservoir will be given by:


′
µ′j,t = f W ′j µ′j,t−1 + W jin x′j,t .
(16)
x

−x

−e
where f (x) = eex +e
−x is the tanh function. Based on the dynamic reservoir state, the ESN-based transfer learning algorithm
will combine with the output weight matrix to approximate the
deviation of the utility value, which can be given by:
′

out ′
′
yj,t
= W j,t
µj,t ,

where
′

(17)

′

out
W j,t

out
W j,t+1
′

is the output weight matrix at time slot t.
 ′T
′
out
′
µ j,t , (18)
= W j,t
+ λ′ û′j (aj,t ) − ûj (aj,t ) − yj,t

where λ is the learning rate, and û′j,t is the actual deviation
between two utility values. In this case, the ESN-based transfer
learning algorithm can find the relationship between the utility
functions when the users’ state information changes and, hence,
reduce the iterations of the RL algorithm to learn the new utility
values. The proposed, distributed ESN-based learning algorithm
performed by each SBS j is summarized in Table I. The proposed algorithm is guaranteed to converge to an NE and this
convergence follows from [4].
IV. S IMULATION R ESULTS
For our simulations, we consider an SCN deployed within
a circular area with radius r = 100 m. U = 25 users and
B = 4 SBSs are uniformly distributed in this SCN area. The rate
requirement of VR transmission is 25.32 Mbit/s [4]. The detailed

TABLE II
SYSTEM PARAMETERS

Parameter
F
B
Nw
Nv
m
α

Value
1000
2 MHz
1000
6
5
2

Value
20 dBm
5, 5
-95 dBm
0.03, 0.3
30 m
100 Gbit/s

ESN-based transfer learning algorithm
Q-learning algorithm with data correlation
Q-learning without data correlation
ESN-based learning algorithm

26

Average delay of each serviced user (ms)

Parameter
PB
S, V
σ2
λ, λ′
rB
VF

24

22

20

18

16

14
1

2

3

4

5

6

Number of SBSs

Fig. 1. Average delay of each user vs. number of SBSs.

parameters are listed in Table III. For comparison purposes, we
use ESN algorithm and a baseline Q-learning algorithm in [4].
Fig. 1 shows how the average delay per user changes with
the number of SBSs. Fig. 1 shows that, as the number of SBSs
increases, the average delay of all algorithms decreases, then
increases. This is due to the fact that, as the number of SBSs
increases, the number of users located in each SBS’s coverage
decreases and, hence, the average delay decreases. However, as
the number of SBSs keeps increasing, the interference will also
increase. Fig. 1 also shows that our algorithm achieves up to
16.7% and 18.2% gains in terms of average delay compared to
the Q-learning with data correlation and Q-learning without data
correlation for 6 SBSs. This is due to the fact that our algorithm
can transfer information across time. From Fig. 1, we can also
see that the deviation between Q-learning algorithms decreases
as the number of SBSs changes. In fact, as the number of SBSs
increases, the number of users associated with each SBS decreases
and, hence, the data correlation of users decreases. Fig. 1 also
shows that the delay gain of the proposed algorithm is small
compared with ESN algorithm. However, the proposed algorithm
can converge much faster as shown in Fig. 2.
Fig. 2 shows the number of iterations needed till convergence
for the proposed approach, ESN algorithm, and Q-learning with
data correlation when the users’ information changes. In this

ESN-based transfer learning algorithm
ESN-based learning algorithm
Q-learning with data correlation
Convergent point

6

5

Delay utility of each SBS

TABLE I
ESN- BASED L EARNING A LGORITHM FOR RESOURCE A LLOCATION
Inputs: xj,t and x′j,t
′ in
′ out
out
′
′
Initialize: W in
j , W j , W j , W j , W j , W j , y j = 0, and yj = 0.
for each time t do.
(a) Estimate the value of the utility function ûj,t based on (17).
if t == 1
(b) Set the mixed strategy π j,t uniformly.
else
(c) Set the mixed strategy π j,t based on the ε-greedy exploration.
end if
(d) Broadcast the index of the mixed strategy to other SBSs.
(e) Receive the index of the mixed strategy as input xj,t .
(f) Perform an action based on the mixed strategy.
(g) Use the index of the mixed strategies and action as input x′j,t .
′ .
(h) Estimate the value of the difference of utility function yj,t
(i) Update the dynamic reservoir state µj,t .
′
(j) Update the output weight matrix W out
j based on yj,t .
end for

4

3

2

1

0
5

10

15

20

25

30

35

40

45

50

55

Number of iterations (10 2 )

Fig. 2. Convergence of the proposed algorithm and Q-learning.

figure, we can see that, as time elapses, the delay utilities for
all considered algorithms increase until convergence to their final
values. Fig. 2 also shows that the proposed algorithm achieves,
respectively, 22.5% and 36% gains in terms of the number of the
iterations needed to reach convergence compared to ESN algorithm and Q-learning. This implies that the proposed algorithm
can apply the already learned utility value to the new utility value
that must be learned as the users’ information changes.
V. C ONCLUSION
In this paper, we have proposed a novel resource allocation
framework for optimizing delay for wireless VR services with
data correlation. We have formulated the problem as a noncooperative game and proposed a novel transfer learning algorithm based
on echo state networks to solve the game. The proposed learning
algorithm can use the existing learning result to directly find
the optimal resource allocation when the users’ state information
changes and, hence, can quickly converge to a mixed-strategy
NE. Simulation results have shown that the proposed algorithm
has a faster convergence time than Q-learning and guarantees low
delays for VR services.
R EFERENCES
[1] E. Baştuğ, M. Bennis, M. Médard, and M. Debbah, “Towards interconnected
virtual reality: Opportunities, challenges and enablers,” IEEE Communications Magazine, vol. 55, no. 6, pp. 110–117, Jan. 2017.
[2] J. Ahn, Y. Yong Kim, and R. Y. Kim, “Delay oriented VR mode WLAN
for efficient wireless multi-user virtual reality device,” in Proc. of IEEE
International Conference on Consumer Electronics, Las Vegas, NV, USA,
March 2017.
[3] M. Singh and B. Jung, “High-definition wireless personal area tracking using
AC magnetic field for virtual reality,” in Proc. of IEEE Virtual Reality, Los
Angeles, California, USA, March 2017.
[4] M. Chen, W. Saad, and C. Yin, “Virtual reality over wireless networks:
Quality-of-service model and learning-based resource management,” available online: arxiv.org/abs/1703.04209, Mar. 2017.
[5] M. Chen, U. Challita, W. Saad, C. Yin, and M. Debbah, “Machine
learning for wireless networks with artificial intelligence: A tutorial on
neural networks,” available online: arxiv.org/abs/1710.02913, Oct. 2017.
[6] J. F. Yang, S. C. Chang, and C. Y. Chen, “Computation reduction for motion
search in low rate video coders,” IEEE Transactions on Circuits and Systems
for Video Technology, vol. 12, no. 10, pp. 948–951, Oct. 2002.
[7] N. Cressie and C. K. Wikle, Statistics for spatio-temporal data, John Wiley
& Sons, 2015.
[8] M. C. Vuran, O. B. Akan, and I. F. Akyildiz, “Spatio-temporal correlation:
theory and applications for wireless sensor networks,” Computer Networks,
vol. 45, no. 3, pp. 245–259, June 2004.
[9] M. Chen, M. Mozaffari, W. Saad, C. Yin, M. Debbah, and C. S. Hong,
“Caching in the sky: Proactive deployment of cache-enabled unmanned
aerial vehicles for optimized quality-of-experience,” IEEE Journal on
Selected Areas on Communications (JSAC), Special Issue on Human-InThe-Loop Mobile Networks, to appear, 2017.
[10] A. E. Abbas, “Constructing multiattribute utility functions for decision
analysis,” INFORMS Tutorials in Operations Research, pp. 62–98, Oct.
2010.
[11] M. Chen, W. Saad, C. Yin, and M. Debbah, “Echo state transfer learning for
data correlation aware resource allocation in wireless virtual reality,” available online: http://resume.walid-saad.com/pdf/extendedasilomarpaper.pdf,
May 2017.
[12] G. Bacci, S. Lasaulce, W. Saad, and L. Sanguinetti, “Game theory for
networks: A tutorial on game-theoretic tools for emerging signal processing
applications,” IEEE Signal Processing Magazine, vol. 33, no. 1, pp. 94–119,
Jan. 2016.

