t Handoff Policy Design for RAN Slicing 9.2.2 Multi-Agent Reinforcement Learning Based Handoff Framework In this section, we first formulate the handoff decision problem as a multi-agent reinforce- ment learning (RL) model, and then propose an intelligent handoff mechanism based on the learning model, LESS. Once the handoff trigger condition for a UE is met, it should choose an appropriate serv- ing NS and BS in order to maintain the desired QoS. We model this target BS and NS selection problem as a multi-agent RL consisting of four main elements: agents, states, actions, and reward. In detail, each UE is an agent to make handoff decisions. The states are defined as the available bandwidth levels of NSs. Let s(c(t) denote the available bandwidth level of NS via BS k at time t after discretization. The environment state can be written as S(t) = at time t. An action means selecting both target BS and NS when a handoff occurs. In detail, we denote by aj(t) = (x;(t),y(t)) the action taken by UE i at time t, where x{(t) and yi(t) is the target BS and NS respectively. If y{ (t) E N, the action denotes deploying a new NS. Let A be the action space for a UE, and thus the system action space for all UEs is AIV. The reward denoted by ri(S(t),a(t)) is the handoff cost for UE i in state S(t) E S with action a(t) E A at time t, which can be expressed as ri(S(t),a(t))= (9.19) CNS-BS, if y{(t) E N. Our objective is to minimize the total long-term handoff cost i(t)) by designing an intelligent handoff mechanism. Traditional Q-learning algorithm [16] is widely used to get the optimal solution to RL problems. However, considering that the system action space IAI is very large, Q-learning algorithm needs a long time to converge. To address this issue, we propose the distributed learning based LESS handoff mechanism in the following. 9.2.3 LESS Algorithm for Target BS and NS Selection Q-learning is a simple yet effective algorithm for solving RL problems, and it can be briefly described as follows. Denote by vector A = [a1((t), aw((