Deep Reinforcement Learning Based Power control for Wireless Multicast Systems
Ramkumar Raghu Pratheek Upadhyaya Mahadesh Panju Vaneet Aggarwal Vinod Sharma



1]Ramkumar Raghu 
1]Pratheek Upadhyaya 
1]Mahadesh Panju 
1,2]Vaneet Aggarwal 
1]Vinod Sharma  
[1]Indian Institute of Science, Bangalore, INDIA. {mahadesh,ramkumar,vinod}@iisc.ac.in
[2]Purdue University, West Lafayette IN, USA.  vaneet@purdue.edu

Deep Reinforcement Learning Based Power control for Wireless Multicast Systems
    [
    Received: date / Accepted: date
==============================================================================

 
We consider a multicast scheme recently proposed for a wireless downlink in <cit.>. It was shown earlier that power control can significantly improve its performance. However for this system, obtaining optimal power control is intractable because of a very large state space. Therefore in this paper we use deep reinforcement learning where we use function approximation of the Q-function via a deep neural network. We show that optimal power control can be learnt for reasonably large systems via this approach. The average power constraint is ensured via a Lagrange multiplier, which is  also learnt. Finally, we demonstrate that a slight modification of the learning algorithm allows the optimal control to track the time varying system statistics.
 


Multicasting, Deep Reinforcement Learning, Quality of Service, Power Control, Dynamics Tracking.



§ INTRODUCTION


Wireless networks are being constantly refined to cater for seamless delivery of huge amount of data to the end users. With increased user generated contents and proliferation of social networking sites, almost 78% of mobile data traffic is expected to be due to mobile videos <cit.>. Also, the requested  traffic for these contents is ridden with redundant requests <cit.>.    
Thus, multicasting is a natural way to address these requests. 

A multicast queue with network coding is studied in <cit.> with infinite library of files. The case of broadcast systems with one server transmitting to multiple users is studied in <cit.>. Both of these works study a slotted system. Some recent works <cit.> use coded caching to achieve multicast. This approach uses local information in the user caches to decode the coded transmission and provides improvement in throughput by increasing the effective number of files transferred per transmission. This throughput may get reduced in a practical scenario due to queueing delays at the basestation/server. <cit.> addresses these issues, analyses the queuing delays and compares it with an alternate coded scheme with LRU caches (CDLS) which provides improvement over the coded schemes in <cit.>. A more recent work in this direction, provides alternate multicast schemes and analyses queueing delays for such multicast systems <cit.>. The authors show that a simple multicast scheme, can have significant gains over the schemes in <cit.>, <cit.> in high traffic regime. 

We further study the multicast scheme proposed in <cit.> in this paper.  This multicast queue merges the requests for a given file from different users, arriving during the waiting time of the initial requests. The merged requests are then served simultaneously.  The gains achieved by this simple multicast scheme, however, are quickly lost in wireless channels due to fading. <cit.> addresses this issue and also provides several multicast queueing schemes to improve the average user delays. Also, it shows that these schemes combined with an optimal power control policy under average power constraint, can provide significant reduction in delays. 

The power control policy proposed in <cit.>, though provides improved delays, has following limitations:


  * The algorithm to get the policy is not scalable with the number of users and the number of states of the channel gains.

  * The policy doesn't adapt to the changing system statistics, which in turn depends on the policy.
           

These systems are often conveniently modelled as a Markov Decision Process, but with large state and action spaces. Obtaining transition probabilities and the optimal policy for such large MDPs is not feasible. Reinforcement learning, particularly, Deep reinforcement learning comes as a natural tool to address such problems <cit.>. Reinforcement learning has the added advantage that it can be used even when the transition probabilities are not available. However, large state/action space can still be an issue. Using function approximation via deep neural networks can provide significant gains  since the Q-values of different state-action pairs can be interpolated even if that state-action pair has never or rarely occurred in the past. Several, deep reinforcement learning techniques such as Deep Q-Network <cit.>, Trust Region Policy Optimization (TRPO) <cit.>, Proximal Policy Gradient (PPO) <cit.> etc. have been successfully applied to several large state-space dynamical systems such as Atari <cit.>, AlphaGo <cit.> etc. DQN is one of the first Deep-RL methods based on value iteration, usually employing ϵ-greedy exploration to learn the optimal policy. TRPO and PPO are policy gradient based methods that employ stochastic gradient descent over policy space to obtain the optimal value function. Policy-Gradient methods often suffer from high variance in sample estimates and poor sample efficiency <cit.>. Value iteration based deep RL methods, like DQN, have been theoretically shown to have competitive performance <cit.>, specifically due to sample efficiency of experience replay <cit.>.  
	
In addition to the above mentioned trade-offs, a constrained stochastic optimization problem, as considered in this paper, further adds to the complexity of the problem. A modification of TRPO for constrained optimization is Constrained Policy Optimization <cit.>. But, this too suffers from the high estimator variance issue. Work in <cit.> considers a multi-timescale approach for constrained DeepRL problems, as considered in this paper. However, <cit.> does not track the system statistics and hence cannot be applied in practical systems. Thus we propose a constrained optimization variant of DQN based on multi-timescale stochastic gradient descent <cit.>. We have preferred DQN in this work, as the Target network and Replay memory used in the DQN reduce the estimator variance and finally achieve the global minimum empirical risk <cit.>. 
	 
	The major contributions of this paper are:

	
  * Proposing two modifications to DQN, to accommodate constraints and system adaptations. We call this Adaptive Constrained DQN (AC-DQN).
	
  * Unlike DQN, constrained DQN can be applied to the multicast systems with constraints, as in <cit.>, to learn the optimal power control policy, online. The constraints can be met by using a Lagrange multiplier. The appropriate Lagrange multiplier is also learnt via a two time scale stochastic gradient descent. The proposed method meets the average power constraint while achieving the global optima as achieved by the static policy proposed in <cit.>. 
	
  * We demonstrate the scalability of our algorithms with system size (number. of users, arrival rate, complex fading).
	
  * We show that AC-DQN can track the changes in the dynamics of the system, e.g., change of rate of arrival over the time of a day, and achieve optimal performance. 

 




Our algorithms work equally well when we replace DQN with its improvements such as DDQN <cit.>. In fact we have run our simulations with DDQN variant of AC-DQN and have achieved similar performance. Next, we describe some more related works to this paper:

Power control in Multicast Systems: Power control in multicast systems has been studied in <cit.>. In <cit.>, optimal power allocation is made to achieve the ergodic capacity (defined as the boundary of the region of all achievable rates in a fading channel with arbitrarily small error probability) while maintaining minimum rate requirements at users and average power constraints.   Authors use water-filling to achieve the optimal policy. In <cit.>, the authors minimize a utility function via linear programming, under SINR constraints at the users and transmit power constraints at the transmitter. Both <cit.> derive an optimal power control policy for delivery to all the users, whereas this paper considers delivery to a random subset of users. In <cit.>, each packet has a deadline and packets not received by the end of the slot are discarded. The authors use dynamic programming to obtain the optimal policies.

Deep Reinforcement Learning (DeepRL) in Wireless Multicast systems: The ability of DeepRL to handle large state-space dynamic systems is being exploited in various multicast wireless systems/networks. In <cit.>, authors look at resource allocation problem in unicast and broadcast. The DeepRL agent learns and selects power and frequency for each channel to improve rate, under some latency constraints. Authors, like in our work, introduce constraints via Lagrange multiplier.   However, the agent doesn't learn the Lagrange multiplier.   Thus, the agent also does not adapt if the system dynamics changes as the  Lagrange constant in the reward is fixed for a given dynamics,   and the learning rate decays with time. Another work, <cit.>, applies unconstrained deep reinforcement learning to multiple transmitters for a proportionally fair scheduling policy by adjusting individual transmit powers. Some studies <cit.> have applied DeepRL to control power for anti-jamming systems.  

Rest of the paper is organised as follows. Section <ref> explains the system model and motivates the problem. Section <ref> presents the proposed DeepRL algorithm AC-DQN. Section <ref> demonstrates our algorithms via simulations and Section <ref> concludes the paper. 

§ SYSTEM MODEL
 

We consider a system with one server transmitting files from a fixed finite library to a set of users (Figure <ref>). We denote the set of users by ℒ = {1,2,⋯, L} and the set of files by ℳ = {1,2,⋯, M}. We assume that M >> L. The request process for file i from user j is a Poisson process of rate λ_ij which is independent of the request processes of other files from user j and also from other users. The total arrival rate is λ=∑_i,jλ_ij. The requests of a file from each user are queued at the server till the user successfully receives the file. All the files are of length F bits. The server transmits at a fixed rate, R bits/sec. Thus, the transmission time for each file is T = F/R. 
 
The channels between the server and the users experience time varying fading. The channel gain of each user is assumed to be constant during transmission of a file. The channel gain for the j^th user at the t^th transmission, is represented by H_j(t). Each H_j(t) takes values in a finite set and form an independent identically distributed (i.i.d) sequence in time, as in <cit.>. The  channel gains of different users are independent of each other and may have different distributions. Let H={H_1, ⋯, H_L}.
 


 More details of the system are described in the following subsections as follows. Section <ref> describes the basic Multicast queue proposed in <cit.>. The scheduling scheme to mitigate the effects of fading studied in <cit.> are also presented. In Sections <ref> and <ref> we summarise  the results from <cit.> which show that using power control can further improve the performance and the algorithm used to obtain the optimal power policy.  We will see that this algorithm is not scalable. Then in Section <ref>  we provide the MDP of the power control problem. In Section <ref> we will present the scalable  DeepRL  solution for this formuation. 
 
 


 §.§ Multicast Queue

 
For scheduling of transmission at the server, we consider the multicast queue studied in <cit.>. In this system, the requests for different files from different users are queued in a single queue, called the multicast queue. In this queue, the requests for file i from all users are merged and considered as a single request. The requested file and the users requesting it, is denoted by (i,𝕃_i). A new request for file i, from user j is merged with the corresponding entry 𝕃_i, if it exists. Else, it is appended to the tail of the queue. Service/transmission of file i, serves all the users in 𝕃_i, possibly with errors due to channel fading.

The random subset of users served by the multicast queue at the t^th transmission, is denoted by the random binary vector, V(t)={V_1(t), ⋯, V_L(t)}, where V_j(t)=1 implies that the user j has requested the file being transmitted; otherwise, V_j(t) = 0. From [Theorem 1, <cit.>], V(t) has a unique stationary distribution. 

It was shown in <cit.> that the above multicast queue performs much better than the multicast queues proposed in literature before. The main difference compared to previous multicast schemes is that in this scheme, all requests of all the users for a given file are merged together over time. One direct consequence of this is that the queue length at the base station does not exceed M. Thus the delay is bounded for all traffic rates. In fact the mean delays are often better than the coded caching schemes proposed in the literature, as well, for most of the traffic conditions. However, in a fading scenario, where the different users have independent fading, the performance of this scheme can significantly deteriorate because of multiple retransmissions required to successfully transmit to all the users needed. Thus, in <cit.>, multiple retransmission schemes were proposed and compared to recover the performance of the system. The following scheme was among the best. It not only (almost) minimizes the overall mean delays of the system, it also is fair to different users in the sense, that the users with good channel gains do not suffer due to users with bad channel gains.
 
Single queue with loop-back (1-LB): The Multicast queue is serviced from head to tail. When a file is transmitted, some of the users will receive the file successfully and some users may receive the file with errors. In the case of unsuccessful reception by some users, the file is retransmitted. A maximum of N (1 ≤ N ≤∞) transmission attempts are made. If there are some users who did not receive the file within N transmission attempts, the request (tuple (i,𝕃_i) with 𝕃_i,   now modified to contain only   the set of users who have not received the file i successfully) is fed back to the queue. If there is another pending request in the queue for the same file (a request for the file which came during the transmission of the current transmission), it is merged with the existing request. Otherwise, a new request for the same file with unsuccessful users is inserted at the tail of the queue.

It was further shown in <cit.> that  choosing the transmit power based on the channel gains, can further improve the system performance.


 §.§ Average Power Constraint


Depending on the value of H(t) and V(t) at time t, the server chooses transmit power P_t, based on a power control policy P_t=π(H(t),V(t)). Choosing a good power control policy is the topic of this paper. 

	The state, S_t  of the system at time t is (H(t), V(t)). Let P_S_t be the power chosen by a policy for state S_t and R(S_t, P_S_t) be the number of successful transmissions for the selected power P_S_t, during the t^th service.

For a fixed transmission rate C and for a given  channel gain H(t) of users, the transmit power requirement P_req (from Shannon's Formula) for user j is  (assuming file length is long enough) 

    P_req(j,S_t)=N_g/H_j^2(t)(2^C/B-1),
	      
where, B is the bandwidth and N_g is the Gaussian noise power at receiver j. Thus the reward for the chosen power control policy, during t^th transmission is given by,

    R(S_t,P_S_t)=∑_j=1^LV_j,S_t 1_{P_S_t>P_req(j,S_t)}(t),

where V_j,S_t = 1 if the user j has requested the file in service and V_j,S_t = 0 otherwise. We now describe the Mesh Adaptive Direct Search (MADS) power control policy.
 


 §.§ MADS Power control policy


  The power control policy in <cit.> is derived from the following optimization problem,

    {P_1,⋯,P_K}max∑_k=1^Kq_k R_k


    s.t. ∑_k=1^Kq_k P_k≤P and P_k≥ 0, k=1,⋯,K,

where, P is the average power constraint, K is the total number of states, P_k is the power chosen by the policy in state k, q_k is the stationary distribution of state k ∈{1,⋯,K} and R_k is the reward for state k, as defined in (<ref>) with S_t=k. This is a non-convex optimization problem. Mesh Adaptive Direct Search (MADS) <cit.> is used in <cit.> to solve this constrained optimization problem and obtain the power control policy. Though MADS achieves global optimum, it is not scalable as its computational complexity is very high.

	The state space and action space of this problem can be very high even for a moderate number of users and channel gains, e.g., a system with L users and G channel gain states, has 𝒪(2^L G^L) states. Therefore, in this paper we propose a deep reinforcement learning framework. This not only provides optimal solution for a reasonably large system but does so without knowing the arrival rates and channel gain statistics. In addition, we will be able to provide an optimal solution even when the arrival and channel gain statistics change with time. 



 §.§ MDP Formulation


The above system can be formulated into a finite state, action Markov Decision Process denoted by tuple (𝕊, 𝔸, r, P, γ): (state space, action space, reward, transition probability, discount factor), where, transition probability P(S_t+1|S_0,P_0, ..., S_t, P_t)=P(S_t+1|S_t, P_t), policy π  chooses power P_t ∼π (.|S_t) in state S_t and the instantaneous reward r_t=R(S_t,P_t).
  
The action-value function <cit.> for this discounted MDP for policy π is

    Q^π   (s,a)=𝔼[∑_t=0^∞γ^t r_t|S_0=s,P_0=a].

The optimal Q-function, Q^* is given by Q^*(s,a)=πmax Q^π(s,a) and satisfies the optimality relation,


    Q^*(s,a)=r(s,a)+a'max γ𝔼[Q^*(s',a')],
  
where, s' is sampled with distribution P(.|s,a). If we know the optimal Q-function (Q^*), we can compute the optimal policy via π(s)=a'max Q^*(s,a). We know the transition matrix of this system and hence can compute the Q-function. But the state space is very large even for a small number of users, rendering the computations infeasible.  Thus, we use a parametric function approximation of the Q function via Deep neural networks and use DeepRL algorithms to get the optimal Q^*.  

Further, to introduce the constraint in the MDP formulation, we look at the policies achieving 

    Q^*(s,a)=π:C_P≤Pmax Q^π(s,a)

where

    C_P=𝔼[T→∞lim∑_t=0^T P_t/T]
 
is the long term average power. We use the Lagrange method for constrained MDPs <cit.> to achieve the optimal policy. In this method, the instantaneous reward is modified as 

    r_t=R(S_t,P_t)-β P_t,
    
where, β is the Lagrange constant achieving optimal Q^* while maintaining, C_P ≤P. Choosing β wrongly will provide the optimal policy with average power constraint different from P.  


§ DEEP REINFORCEMENT LEARNING BASED POWER CONTROL POLICY


In this section, we describe the Deep-Q-Network (DQN) <cit.> based power control. First we describe the DQN algorithm. We then propose a variant of DQN for constrained problems, where in, we use a Lagrange multiplier to take care of the average power constraint. We use multi-timescale stochastic gradient descent approach to learn the Lagrange multiplier, to obtain the right average power constraint. Finally, we change the learning step size from decreasing to a constant so that the optimal power control can track the time varying system statistics. 


 §.§ Deep Q Networks

DQN is a popular Deep Reinforcement learning algorithm to handle large state-space MDPs with unknown/complex dynamics, P(S_t+1|S_t, P_t). The DQN is a Value Iteration based method, where the action-value function is approximated by a Neural Network. Though there are several follow up works providing improvements over this algorithm <cit.>, we use this algorithm owing to its simplicity. We will show that DQN itself is able to provide us the optimal solution and tracking. These improvements may further improve the performance in terms of sample efficiency, estimator variance etc. The DQN algorithm is given in Algorithm <ref>. Earlier attempts in combining nonlinear function approximators such as neural networks and RL were unsuccessful due to instabilities caused by 1) correlated training samples, 2) drastic change in policy with small change in function approximation, and 3) correlation between the training function and approximated function <cit.>. Success of DQN is attributed to addressing these issues with two key ingredients of the algorithm: Experience Replay Memory 𝕄 and Target Network, Q_θ^*. The replay memory stores the transitions of an MDP, specifically the tuple, (S_t, P_t, r_t, S_t+1). The algorithm then samples, uniformly, a random minibatch of transitions from the memory. This removes correlation  between the data and smoothens the data distribution change with iteration.  The target network and randomly sampled mini-batch from the memory 𝕄, form the training set for training the Q_θ Network, at every epoch. This random sampling provides i.i.d samples for performing stochastic gradient descent with loss 

    L^π_θ_Q = 1/n∑_j=1^n(Y_j-Q_θ(S_j, A_j))^2

 where, Y_i=r_i+γ a'maxQ_θ^*(S_i,a')). The iterates {θ_t} are given by: 
 

    θ_t+1←θ_t - η_1(t) ∇_θ L^π_θ_Q,

where η_1(t) satisfies:

    ∑_t=0^∞η_1(t)=∞, ∑_t=0^∞η_1^2(t)<∞, η_1(t)≥ 0.

The weights of the target network Q^* are held constant for T_target epochs, thereby controlling any drastic change in policy and reducing correlation between Q and Q^*. This can be seen as a Risk Minimization problem in nonparametric-regression with regression function Q_θ^* and risk L^π_θ_t_Q. Readers are referred to <cit.> for elaborate analysis of DQN.  Theorem 4.4 in <cit.> provides a proof of convergence and the rate of convergence using non-parametric regression bounds, when sparse ReLU networks are used, under certain smoothness assumptions on the reward function and the dynamics.    
  
   

  







  


 §.§ Adaptive Constrained DQN (AC-DQN)

The DQN algorithm is meant for unconstrained optimization. Since our problem has an average power constraint of P, we consider the instantaneous reward in (<ref>), with a Lagrange multiplier β. The long term constraint depends on the Lagrange multiplier and can be quite sensitive to it. Thus, we design our algorithm, AC-DQN, to learn the appropriate β. We will see later, that this will enable us to further modify our algorithm to track the changing statistics of the  channel gains and arrival statistics. The AC-DQN algorithm is given in Algorithm <ref>. Here, we use multi-timescale SGD as in <cit.>. In this approach, in addition to the SGD on Q_θ, using minibatch, we use a stochastic gradient descent on the Lagrange constant, β as 


    β_t+1←β_t + η_2(t) ∇_β L_P^π_θ,

where ∇_β L_P^π_θ = C_P(S_t)-P. Since the expectation in (<ref>) is not available to us, we take C_P(S_t)=∑_i=t-T_W^t P_i(S_i)/T_W, where T_W is the finite horizon window.



Additionally η_1 and η_2 are required to follow <cit.>:

    ∑_i=1^∞η_1(i)=∑_i=1^∞   η_2(i)=∞,
    ∑_i=1^∞η_1^2(i)+η_2^2(i)   <∞, η_2(i)/η_1(i)→ 0.
        


  

Tracking with AC-DQN: Tracking of system statistics is essential, to achieve optimal power control in a non-stationary system. In multi-time scale stochastic gradient descent, such as AC-DQN, step sizes η_1(t) and η_2(t) can be fixed to enable tracking. If η_2<<η_1, then the Lagrange multiplier changes much more slowly than the Q-function. Then the two timescale theory (see, e.g., <cit.>), will allow the Lagrange multiplier to adapt slowly to the changing system statistics but at the same time provide average power control. The solution will reach in a neighbourhood of the optimal point.

Although the convergence of this modified algorithm is not proved yet (even for the unconstrained DQN, convergence has been proved only recently  in <cit.>), our simulations will show that the resulting algorithm tracks the optimal solution in the time varying scenario.

The time varying scenario in our setup results due to change in the request arrival statistics from the users and changing channel gain statistics due to motion of the users.


§ SIMULATION RESULTS AND DISCUSSION


In this section, we demonstrate the Deep Learning methods for power control proposed in this paper. We compare performances of AC-DQN and MADS Power control policies. Though MADS provides optimal solutions for small system sizes, it is not scalable. We show that the Deep Learning algorithm, AC-DQN, indeed achieves the global optimum obtained by MADS algorithm, while being scalable with the system size (number of users). We further demonstrate that AC-DQN algorithm tracks the changing system dynamics and obtains the optimal policy, adaptively. We use Keras libraries <cit.> in Python for implementation of our algorithms and our system is implemented in MATLAB.

We consider two systems, one with 4 users and compare AC-DQN and MADS; the other with 20 users, showing performance of AC-DQN. MADS is not able to provide a solution for the second system since the space and time complexity of MADS increase exponentially with the number of users. In all the examples, we split the users in two equal sized groups, one group has good channel statistics and the other bad channel statistics. In both the systems, we compare all the algorithms with a constant power control policy, where the transmit power P_t is fixed to P_t=P, to indicate the gain due to power control. The system and algorithm parameters, used for the simulations are as follows:  



  §.§.§ Small User Case
 Number of users, L=4, Catalog Size M= 100, File Size F=10MB, Transmission rate C=10MB/s, Bandwidth B=10MHz, Channel Gains, ∼ Uniform([0.1 0.2 0.3]) for two users with bad channel statistics and ∼ Uniform([0.7 0.8 0.9]) for two users with good channel statistics, File Popularity : Uniform, (Zipf exponent = 0). Average Power Constraint  P=7, Simulation time= 10^5 mutlicast transmissions. 




  §.§.§ Large User Case
 System Parameters: Power Transmit Levels = 20 (1 to 50), L=20, M = 100, F=10MB, C=10MB/s, Channel Gains : Exponentially distributed. (∼ exp(0.1) for bad channel, ∼ exp(1.0) for good channel), R=10MB/s, B=10MHz, P=7, File Popularity : Zipf distribution with (Zipf exponent = 1). Simulation time: 10^5 mutlicast transmissions. In both the cases we set the noise power as N_g=1.



  §.§.§ Hyperparameters

We consider fully connected neural networks  with two hidden layers for all the function approximations considered in the algorithms.   Input layer nodes are assumed to be 2L and the output layer nodes is equal to 20, the number of transmit power levels. Each output represents the Q value for a particular action. The action space is restricted to be finite, as DQN converges only with finite action spaces.  We use two hidden layers for the neural network, with 128 and 64 nodes, and ReLU activation function is chosen, respectively. The other parameters are as follows: Replay memory size |𝕄|=30000,
γ= 0.9, ϵ_0=1.0, ϵ_decay= 0.98, ϵ_t = ϵ_0 (0.98)^t, η_1= 0.001, η_1^decay=0.00001, η_2=.0001, η_2^decay=0.00001, Mini-batch Size (n)=64, T_target=100, and T_W=200. 

Achieving Global Optima (AC-DQN vs MADS):
  We use the system setting of small user case, specified above.  We run the system for the average power constraint P=7, with exponential arrivals of rate 0.4 to 4.0. Figure <ref> shows a comparison of sojourn times of Constant Power Policy, P_t=P, MADS and AC-DQN. Further, Figure <ref> shows convergence of average power to P for AC-DQN. We see from Figure <ref> that AC-DQN achieves the global optimum achieved by MADS, while maintaining the average power constraint.




AC-DQN performance in a Scaled Network:
To show the scalability of AC-DQN, we simulate  the relatively complex system mentioned in large user case, above. We run the simulation for P=7. We see in Figure <ref> that the AC-DQN gives, drastic improvement (around  50 percent) over constant power case. AC-DQN achieves this while maintaining the average power, by learning the Lagrange constant as seen in Figure <ref>. Figure <ref> shows the convergence of average power of AC-DQN to the average power constraint, P, for arrival rate of 1.0 requests per sec in the same simulation run.









AC-DQN Tracking Simulations:
In this section we show via simulations the tracking capabilities of AC-DQN. We show this for large user case with average power constraint, P=7. We fix η_1 = 0.001 and η_2 = 0.00003. As explained previously, this is important for detecting the change in the environment dynamics faster. In this simulation, we vary the arrival rate at every six hours over a period of 24 hours. This captures the real world scenario where the request traffic to the base station varies with time of the day. To make the learning harder for our algorithm, we make these changes abruptly at every six hours. Specifically we use arrival rates λ= 0.4, 0.8, 0.2, 1.0 for 1st, 2nd, 3rd and 4th six hour period, respectively. We plot the AC-DQN performance for P=7 in Figure <ref>. We calculate the mean sojourn time and average power using a moving average window of size 1000 samples. We observe that for each arrival rate in this simulation, the AC-DQN achieves the corresponding stationary mean sojourn time performance. For instance for λ=0.8 and P=7, the values in Figure <ref> and Figure <ref> are comparable. It is important to note that this performance is achieved while maintaining the average power constraint as can be seen in Figure <ref>. The effect of fixing the learning rates is seen in the small oscillations of average power around P=7 in Figure <ref>. This is the oscillation in a small neighborhood around the optimal average power. Smaller the step size, lesser the oscillations. 

Next, we demonstrate the importance of constant step sizes for η_1 and η_2, and the inability of decaying step sizes to track the changing system statistics. We consider a system where the arrival rates change over a period of 48 hours. We fix λ=1.0 for first 24 hours, then fix λ=0.6, 0.5, 0.4, 0.8 for four consecutive 6 hours intervals. This change in time period is just to illustrate the tracking ability in a more emphatic manner. It will be clear in the previous time frame also but will require more simulation time. We fix P=5. We run the AC-DQN algorithm for this system with: 1) decaying η_1 and η_2 satisfying (<ref>) and 2) constant step sizes, η_1=0.001 and η_2=0.00003. Rest of the parameters remain same as in the large user case. We see in Figure <ref> that the AC-DQN with constant step-size almost always outperforms the decaying step size. Specifically, after the first 24 hours the delay reduction is nearly 50 percent for constant step-size. The reason for this is evident from Figures <ref> and  <ref>. We see in Figure <ref> that the AC-DQN with constant step-size learns the Lagrange constant through out the simulation time, whereas, the AC-DQN decaying step size is unable to learn the  Lagrange constant after the first 24 hours. As can be seen in Figure <ref>, this affects the average power achieved by the AC-DQN with decaying step size. While constant step size maintains the average power constraint of P=5, the average power achieved by the decaying step-size AC-DQN drops to 4. Hence, the decaying step-size AC-DQN suffers suboptimal utilization of available power.  Thus in practical systems only constant step-size AC-DQN will be capable of adapting to the changing system statistics.            















 










Discussion: We see from the simulations that the DeepRL techniques can achieve global optimal performance while providing scalability with system size. Our two-timescale approach,  AC-DQN, extends this to systems with constrained control. Though we have demonstrated this on a system with a single constraint, AC-DQN can very well be extended to systems with multiple constraints. In such systems each constraint is associated with a Lagrange constant. Each Lagrange constant adds an additional SGD step to the AC-DQN algorithm. For a stationary system, it is enough that the step-sizes satisfy multi-timescale criterion similar to (<ref>), see <cit.>. However, if AC-DQN is used in systems with changing system statistics the step sizes shall be kept constant. The step sizes shall be fixed as per the tolerance requirement for a given constraint (e.g., in our system the tolerance could be P±Δ P. In other words, Δ P is the allowed deviation from the constraint P). Lesser the tolerance, lesser the step-size. However, fixing the step-sizes too small may make the algorithm too slow to track the changes in system statistics. Hence, choosing the step sizes is a trade-off between the tolerance of the constraint and the required algorithmic agility to track the system changes.



§ CONCLUSION


We have considered a multicast downlink in a wireless network. Fading of different links to users causes significant reduction in the performance of the system. However, appropriate  change in the scheduling policies and power control can mitigate most of the losses.  However, obtaining optimal power control for this system is computationally very hard. We show that using Deep Reinforcement Learning, we can obtain optimal power control, online, even when the system statistics are unknown. We use a recently developed version  of Q learning, Deep Q Network to learn the Q-function of the system via function approximation. Furthermore, we modify the algorithm to satisfy our constraints and also to make the optimal policy track the  time varying system statistics. DDQN variant of AC-DQN provides similar performance. 

One interesting extension of this work would be adding the caches at the user nodes and learning the optimal caching policy along with the power control using DeepRL. Future works may also consider applying AC-DQN to multiple-base-station scenarios for interference mitigation. 
IEEEtran

