t)] E A121 the actions for all UEs. Qt(S,A) and r(S, A) are respectively the Q-value and reward for the state-action pair (S, A) E where r(S, A) = The update rule of Q-value can be expressed as Q0(S,A) = M, for all A A121 and SES, Qt+1(S,A) = (9.20) Qt(S, if A(t) # A or S(t) # S, r(S, A) min Q,(S(t+1),A'), otherwise, 9 Access Control and Handoff Policy Design for RAN Slicing Information interaction Agent Take actions: Environment Change BS or/and NS q-table Cooperation 1. Serving UE ID NS 3/ q-table 2. BS load Handoff Cooperation policy 3. NS load 4. Available Cooperation resource q-table Get reward: handoff cost Figure 9.7 The framework of LESS-DL. where S(t) and A(t) are respectively the state and the action vector at time t, M is a large constant for initialization, and B(O < B < 1) is the discount factor. The target BS and NS selection policy is to choose the NS-BS pair with the smallest Q-value with respect to e-greedy policy. However, applying traditional Q-learning to solving our problem requires a large action space (i.e. which takes a long time to converge. Moreover, it requires all UEs to make handoff decisions simultaneously, which is unrealistic. Thus, to overcome these issues, we develop a distributed learning algorithm, LESS-DL, shown in Figure 9.7 to select target BS and NS for each individual UE. The main idea of LESS-DL is that each UE only needs to maintain a reduced Q-table where the action space is composed of its own actions without distinguishing the actions from other UEs. 9.2.3.1 q-Value Update Policy Denote by qi-table the reduced Q-table maintained by UE: i, and (i) (S, aj) the Q-value of UE i at time t with the state-action pair (S, aj). For convenience, we use q-value and Q-value to denote the value in the reduced and original Q-table respectively. Using a similar idea of [17],qqu (S, aj) can be updated as q.ou (S,a) = M, for all a E Aand SES, (S,aj) = q(6) ((S,a), if aj(t) # a or S(t) # S, (9.21) ai),ri(S,ai)+minqt(a,s(S,A))} otherwise. By using this update method, we can get the redu